[DISCUSSION] Mirror implementation with isar-bootstrap

[DISCUSSION] Mirror implementation with isar-bootstrap

[Claudius Heine]

Hi,

since isar-bootstrap is currently missing support to change the mirror 
of the apt repositories via the bitbake configuration, I would like to 
start the discussion about how to implement this the best way.

AFAIK apt doesn't has a native support for mirror selection that could 
be simply adapted to our use-case. Debian itself uses http redirection 
to find the closest remote server (http://httpredir.debian.org/) while 
apt also supports a 'mirror' method like this:

     deb mirror://mirrors.ubuntu.com/mirrors.txt precise main

Both of those methods aren't really what we are looking for.

What we want is being able to overwrite official mirror URIs with our 
own. OEs mechanism to deal with this is the PREMIRRORS variable [1].
It basically contains a regex and a replace string separated by a 
whitespace where each of those sets is separated by a new line.

We could do something similar like this:

   PREMIRRORS="http://ftp.debian.org/ http://internal.mirror/ \n
               ..."

And that would work as long as the internal mirror mirrors exactly the 
same content as ftp.debian.org.

If this is not the case, then things are getting more difficult.
My first question is: Can we assume the internal mirrors are 1:1 copies 
of the external ones?

If not then here are some other solutions:

   PREMIRRORS="http://ftp.debian.org/ stretch http://internal.mirror/ \n
               ..."

This way we could select additionally the distribution suite. It's just 
a bit more flexible but maybe not enough? We could add components and 
what not here as well. Maybe we also need to split between 'deb' and 
'deb-src' mirrors?

With those previous examples PREMIRRORS are operating on the global 
aggregated source list. So maybe specifying which file they needed to be 
applied could be better:

   PREMIRRORS="conf/distro/stretch.list http://ftp.debian.org 
http://internal.mirror \n
               ..."

That might limit the effect of those regexes to lower the risk of 
accidentally overwriting anything else.

I currently don't know how flexible we need to design this, so that is 
why I am asking the community.

Thanks,
Claudius


[1] 
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#var-PREMIRRORS


-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [DISCUSSION] Mirror implementation with isar-bootstrap

[Baurzhan Ismagulov]

Hello Claudius,

On Mon, Apr 09, 2018 at 02:03:49PM +0200, Claudius Heine wrote:
> AFAIK apt doesn't has a native support for mirror selection that could be
> simply adapted to our use-case. Debian itself uses http redirection to find
> the closest remote server (http://httpredir.debian.org/)

Which we used to have in Isar master and which failed sporadically with some
4xx errors, so we've reverted to f.d.o.


> What we want is being able to overwrite official mirror URIs with our own.
> OEs mechanism to deal with this is the PREMIRRORS variable [1].
> It basically contains a regex and a replace string separated by a whitespace
> where each of those sets is separated by a new line.
> 
> We could do something similar like this:
> 
>   PREMIRRORS="http://ftp.debian.org/ http://internal.mirror/ \n
>               ..."

I usually prefer to define vars in local.conf and sed
's/##DEBIANMIRROR##/$DEBIANMIRROR/' debian-stretch.list.in >
debian-stretch.list, so that the user can see what is being used. Patching on
the fly is too magic for me :) . That said, since upstream does it in that way
and people are familiar with it, I'm fine with it.


> If this is not the case, then things are getting more difficult.
> My first question is: Can we assume the internal mirrors are 1:1 copies of
> the external ones?

I think projects would usually be able to satisfy whatever Isar expects. I'd
assume that an internal mirror to use has all files that are needed for an
Isar-based project. (Depending on what you mean by "1:1": It has to contain all
files needed, not all available in Debian, and possibly in other paths, not
necessarily where f.d.o has them -- that is handled by Packages.gz).


> If not then here are some other solutions:
> 
>   PREMIRRORS="http://ftp.debian.org/ stretch http://internal.mirror/ \n
>               ..."
> 
> This way we could select additionally the distribution suite. It's just a
> bit more flexible but maybe not enough? We could add components and what not
> here as well. Maybe we also need to split between 'deb' and 'deb-src'
> mirrors?

If this is needed at this time, my suggestion would be matching the whole
sources.list line or even the file as you suggest below. That said, I'd be
reluctant to do that if that would break the upstream PREMIRRORS syntax as
people know it.


> With those previous examples PREMIRRORS are operating on the global
> aggregated source list. So maybe specifying which file they needed to be
> applied could be better:
> 
>   PREMIRRORS="conf/distro/stretch.list http://ftp.debian.org
> http://internal.mirror \n
>               ..."
> 
> That might limit the effect of those regexes to lower the risk of
> accidentally overwriting anything else.
> 
> I currently don't know how flexible we need to design this, so that is why I
> am asking the community.

If this is not needed at this time, I'd suggest that we look at your initial
implementation and keep it simple, stupid till it becomes necessary. Then we
would have the exact requirements.


With kind regards,
Baurzhan.

Re: [DISCUSSION] Mirror implementation with isar-bootstrap

[Claudius Heine]

Hi,

On 2018-04-12 16:12, Baurzhan Ismagulov wrote:
> Hello Claudius,
> 
> On Mon, Apr 09, 2018 at 02:03:49PM +0200, Claudius Heine wrote:
>> AFAIK apt doesn't has a native support for mirror selection that could be
>> simply adapted to our use-case. Debian itself uses http redirection to find
>> the closest remote server (http://httpredir.debian.org/)
> 
> Which we used to have in Isar master and which failed sporadically with some
> 4xx errors, so we've reverted to f.d.o.

I didn't suggest changing the mirror here, its just a short report about 
my research in Debian/Ubuntu-native solutions.

>> What we want is being able to overwrite official mirror URIs with our own.
>> OEs mechanism to deal with this is the PREMIRRORS variable [1].
>> It basically contains a regex and a replace string separated by a whitespace
>> where each of those sets is separated by a new line.
>>
>> We could do something similar like this:
>>
>>    PREMIRRORS="http://ftp.debian.org/ http://internal.mirror/ \n
>>                ..."
> 
> I usually prefer to define vars in local.conf and sed
> 's/##DEBIANMIRROR##/$DEBIANMIRROR/' debian-stretch.list.in >
> debian-stretch.list, so that the user can see what is being used. Patching on
> the fly is too magic for me :) . That said, since upstream does it in that way
> and people are familiar with it, I'm fine with it.
Your method would also have the disadvantage of being much less flexible 
if multiple repositories are used. (Convenient support for multiple 
repositories is one of my reasons for doing all this debootstrapizations.)

>> If this is not the case, then things are getting more difficult.
>> My first question is: Can we assume the internal mirrors are 1:1 copies of
>> the external ones?
> 
> I think projects would usually be able to satisfy whatever Isar expects. I'd
> assume that an internal mirror to use has all files that are needed for an
> Isar-based project. (Depending on what you mean by "1:1": It has to contain all
> files needed, not all available in Debian, and possibly in other paths, not
> necessarily where f.d.o has them -- that is handled by Packages.gz).

What I meant is that such a search and replace method might accidentally 
overwrite Mirrors you didn't want to change.

For example if you use Debian stretch and jessie repos on the same 
image. The reason doing so might be because you want to use specific 
packages from stretch, but have all other files from jessie.
For some reason your internal mirror just provides a mirror of the 
stretch distribution. Now its difficult to overwrite just that one entry 
where stretch is used and leave the jessie one in place.

I disregarded those scenarios with the current patch, but it should be 
possible to change it in a later version if the need arise.

>> If not then here are some other solutions:
>>
>>    PREMIRRORS="http://ftp.debian.org/ stretch http://internal.mirror/ \n
>>                ..."
>>
>> This way we could select additionally the distribution suite. It's just a
>> bit more flexible but maybe not enough? We could add components and what not
>> here as well. Maybe we also need to split between 'deb' and 'deb-src'
>> mirrors?
> 
> If this is needed at this time, my suggestion would be matching the whole
> sources.list line or even the file as you suggest below. That said, I'd be
> reluctant to do that if that would break the upstream PREMIRRORS syntax as
> people know it.

No. Matching a whole source list line is to complicated, matching just 
some parts of it makes things easier. I would call myself reasonable 
experienced with regexes, but that format and how it is understood is a 
bit too much:

   deb [ option1=value1 option2=value2 ] uri suite [component1] 
[component2] [...]
   deb-src [ option1=value1 option2=value2 ] uri suite [component1] 
[component2] [...]

If we want full flexibility, then having a "simpler" matching syntax 
would help:

   PREMIRRORS += "type:deb option:arch=amd64 src:ftp\.(\S+\.)?debian.org 
suite:jessie component:main component:contrib trg:internal.server \n"

That would then match all of those entries:

   deb [ arch=amd64 ] ftp.de.debian.org jessie main contrib
   deb [ arch=amd64 ] ftp.de.debian.org jessie contrib main

and not:

   deb ftp.de.debian.org jessie main contrib
   deb [ arch=armhf ] ftp.de.debian.org jessie main contrib
   deb [ arch=amd64 ] ftp.de.debian.org stretch main contrib
   deb [ arch=amd64 ] ftp.de.debian.org jessie main contrib non-free

Doing the same with one regex is much more painful.

> 
> 
>> With those previous examples PREMIRRORS are operating on the global
>> aggregated source list. So maybe specifying which file they needed to be
>> applied could be better:
>>
>>    PREMIRRORS="conf/distro/stretch.list http://ftp.debian.org
>> http://internal.mirror \n
>>                ..."
>>
>> That might limit the effect of those regexes to lower the risk of
>> accidentally overwriting anything else.
>>
>> I currently don't know how flexible we need to design this, so that is why I
>> am asking the community.
> 
> If this is not needed at this time, I'd suggest that we look at your initial
> implementation and keep it simple, stupid till it becomes necessary. Then we
> would have the exact requirements.

I agree.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [DISCUSSION] Mirror implementation with isar-bootstrap

[Baurzhan Ismagulov]

On Thu, Apr 12, 2018 at 05:13:00PM +0200, Claudius Heine wrote:
> I didn't suggest changing the mirror here, its just a short report about my
> research in Debian/Ubuntu-native solutions.

Sorry for the confusion. Yes, I got your point. Then, I shared our past
experience with httpredir, just to inform people.


> Your method would also have the disadvantage of being much less flexible if
> multiple repositories are used. (Convenient support for multiple
> repositories is one of my reasons for doing all this debootstrapizations.)

With multiple repos, multiple vars would be necessary. Whether it's
inconvenient is a matter of taste. For me, overriding an additive templace in a
config is easier to understand than sedding on the fly without leaving a trail
for the user what exactly has been done; but as already mentioned, mimicking
the upstream has other advantages. So, no objection here.

Before I dig into the upstream docs: Your patch implements the same syntax as
the upstream PREMIRRORS, no?


> What I meant is that such a search and replace method might accidentally
> overwrite Mirrors you didn't want to change.
> 
> For example if you use Debian stretch and jessie repos on the same image.
> The reason doing so might be because you want to use specific packages from
> stretch, but have all other files from jessie.
> For some reason your internal mirror just provides a mirror of the stretch
> distribution. Now its difficult to overwrite just that one entry where
> stretch is used and leave the jessie one in place.
> 
> I disregarded those scenarios with the current patch, but it should be
> possible to change it in a later version if the need arise.

Ah, now I see. I think this is good enough for now.


> No. Matching a whole source list line is to complicated, matching just some
> parts of it makes things easier. I would call myself reasonable experienced
> with regexes, but that format and how it is understood is a bit too much:
...
> Doing the same with one regex is much more painful.

Ok.


With kind regards,
Baurzhan.

Re: [DISCUSSION] Mirror implementation with isar-bootstrap

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 1181 bytes --]

Hi,

On Fri, 2018-04-13 at 12:33 +0200, Baurzhan Ismagulov wrote:
> Before I dig into the upstream docs: Your patch implements the same
> syntax as
> the upstream PREMIRRORS, no?

I tried to. There might be subtle differences, since its a new
implementation.

For instance I do a regex search. OE might do a match instead. That
means my implementation is compatible with OEs, but you don't need to
specify a regex for the whole string. I can change that, but then the
protocol has to be specified in every regex and replace string. I
preferred not do do that. With my implementation you can do that, but
you don't need to.

Simply put, you can use OEs style of PREMIRROS in DISTRO_APT_PREMIRROS,
but you can also use the shorted style version I described in the patch
notes.

regards,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [DISCUSSION] Mirror implementation with isar-bootstrap

[Baurzhan Ismagulov]

On Fri, Apr 13, 2018 at 12:56:55PM +0200, Claudius Heine wrote:
> > Before I dig into the upstream docs: Your patch implements the same
> > syntax as
> > the upstream PREMIRRORS, no?
> 
> I tried to. There might be subtle differences, since its a new
> implementation.
> 
> For instance I do a regex search. OE might do a match instead. That
> means my implementation is compatible with OEs, but you don't need to
> specify a regex for the whole string. I can change that, but then the
> protocol has to be specified in every regex and replace string. I
> preferred not do do that. With my implementation you can do that, but
> you don't need to.
> 
> Simply put, you can use OEs style of PREMIRROS in DISTRO_APT_PREMIRROS,
> but you can also use the shorted style version I described in the patch
> notes.

Ok, I see. The idea is to avoid bad surprises for people using PREMIRRORs in
the way they know from Yocto.

With kind regards,
Baurzhan.