Re: [DISCUSSION] Mirror implementation with isar-bootstrap

[Claudius Heine <claudius.heine.ext@siemens.com>]

Hi,

On 2018-04-12 16:12, Baurzhan Ismagulov wrote:
> Hello Claudius,
> 
> On Mon, Apr 09, 2018 at 02:03:49PM +0200, Claudius Heine wrote:
>> AFAIK apt doesn't has a native support for mirror selection that could be
>> simply adapted to our use-case. Debian itself uses http redirection to find
>> the closest remote server (http://httpredir.debian.org/)
> 
> Which we used to have in Isar master and which failed sporadically with some
> 4xx errors, so we've reverted to f.d.o.

I didn't suggest changing the mirror here, its just a short report about 
my research in Debian/Ubuntu-native solutions.

>> What we want is being able to overwrite official mirror URIs with our own.
>> OEs mechanism to deal with this is the PREMIRRORS variable [1].
>> It basically contains a regex and a replace string separated by a whitespace
>> where each of those sets is separated by a new line.
>>
>> We could do something similar like this:
>>
>>    PREMIRRORS="http://ftp.debian.org/ http://internal.mirror/ \n
>>                ..."
> 
> I usually prefer to define vars in local.conf and sed
> 's/##DEBIANMIRROR##/$DEBIANMIRROR/' debian-stretch.list.in >
> debian-stretch.list, so that the user can see what is being used. Patching on
> the fly is too magic for me :) . That said, since upstream does it in that way
> and people are familiar with it, I'm fine with it.
Your method would also have the disadvantage of being much less flexible 
if multiple repositories are used. (Convenient support for multiple 
repositories is one of my reasons for doing all this debootstrapizations.)

>> If this is not the case, then things are getting more difficult.
>> My first question is: Can we assume the internal mirrors are 1:1 copies of
>> the external ones?
> 
> I think projects would usually be able to satisfy whatever Isar expects. I'd
> assume that an internal mirror to use has all files that are needed for an
> Isar-based project. (Depending on what you mean by "1:1": It has to contain all
> files needed, not all available in Debian, and possibly in other paths, not
> necessarily where f.d.o has them -- that is handled by Packages.gz).

What I meant is that such a search and replace method might accidentally 
overwrite Mirrors you didn't want to change.

For example if you use Debian stretch and jessie repos on the same 
image. The reason doing so might be because you want to use specific 
packages from stretch, but have all other files from jessie.
For some reason your internal mirror just provides a mirror of the 
stretch distribution. Now its difficult to overwrite just that one entry 
where stretch is used and leave the jessie one in place.

I disregarded those scenarios with the current patch, but it should be 
possible to change it in a later version if the need arise.

>> If not then here are some other solutions:
>>
>>    PREMIRRORS="http://ftp.debian.org/ stretch http://internal.mirror/ \n
>>                ..."
>>
>> This way we could select additionally the distribution suite. It's just a
>> bit more flexible but maybe not enough? We could add components and what not
>> here as well. Maybe we also need to split between 'deb' and 'deb-src'
>> mirrors?
> 
> If this is needed at this time, my suggestion would be matching the whole
> sources.list line or even the file as you suggest below. That said, I'd be
> reluctant to do that if that would break the upstream PREMIRRORS syntax as
> people know it.

No. Matching a whole source list line is to complicated, matching just 
some parts of it makes things easier. I would call myself reasonable 
experienced with regexes, but that format and how it is understood is a 
bit too much:

   deb [ option1=value1 option2=value2 ] uri suite [component1] 
[component2] [...]
   deb-src [ option1=value1 option2=value2 ] uri suite [component1] 
[component2] [...]

If we want full flexibility, then having a "simpler" matching syntax 
would help:

   PREMIRRORS += "type:deb option:arch=amd64 src:ftp\.(\S+\.)?debian.org 
suite:jessie component:main component:contrib trg:internal.server \n"

That would then match all of those entries:

   deb [ arch=amd64 ] ftp.de.debian.org jessie main contrib
   deb [ arch=amd64 ] ftp.de.debian.org jessie contrib main

and not:

   deb ftp.de.debian.org jessie main contrib
   deb [ arch=armhf ] ftp.de.debian.org jessie main contrib
   deb [ arch=amd64 ] ftp.de.debian.org stretch main contrib
   deb [ arch=amd64 ] ftp.de.debian.org jessie main contrib non-free

Doing the same with one regex is much more painful.

> 
> 
>> With those previous examples PREMIRRORS are operating on the global
>> aggregated source list. So maybe specifying which file they needed to be
>> applied could be better:
>>
>>    PREMIRRORS="conf/distro/stretch.list http://ftp.debian.org
>> http://internal.mirror \n
>>                ..."
>>
>> That might limit the effect of those regexes to lower the risk of
>> accidentally overwriting anything else.
>>
>> I currently don't know how flexible we need to design this, so that is why I
>> am asking the community.
> 
> If this is not needed at this time, I'd suggest that we look at your initial
> implementation and keep it simple, stupid till it becomes necessary. Then we
> would have the exact requirements.

I agree.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de