Re: Multi repo support

[Alexander Smirnov <asmirnov@ilbers.de>]

Hi,

On 02/02/2018 04:26 PM, Claudius Heine wrote:
> On Fri, 2018-02-02 at 15:28 +0300, Alexander Smirnov wrote:
>> Hi again!
>>
>>> Hi,
>>>
>>> On Fri, 2018-02-02 at 00:52 +0300, Alexander Smirnov wrote:
>>>> Hi!
>>>>
>>>> Claudius Heine <ch@denx.de> 2 февраля 2018 г. 0:07:49 написал:
>>>>
>>>>> Hi Alex,
>>>>>
>>>>> On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov wrote:
>>>>>>
>>>>>> On 02/01/2018 09:51 PM, Claudius Heine wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Thu, 2018-02-01 at 19:34 +0100, Henning Schild wrote:
>>>>>>>> Am Thu, 1 Feb 2018 16:16:58 +0100
>>>>>>>> schrieb "[ext] Henning Schild" <henning.schild@siemens.co
>>>>>>>> m>:
>>>>>>>>
>>>>>>>>> Am Thu, 1 Feb 2018 15:54:26 +0100
>>>>>>>>> schrieb "[ext] Claudius Heine" <claudius.heine.ext@siem
>>>>>>>>> ens.
>>>>>>>>> com>
>>>>>>>>> :
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I would like to start the discussion about how to
>>>>>>>>>> best
>>>>>>>>>> implement
>>>>>>>>>> muti repository support in isar.
>>>>>>>>>>
>>>>>>>>>> Does someone already has some ideas or even something
>>>>>>>>>> in
>>>>>>>>>> the
>>>>>>>>>> pipeline for this?
>>>>>>>>>>
>>>>>>>>>> If not then I do have an idea that was outlined
>>>>>>>>>> together
>>>>>>>>>> with
>>>>>>>>>> Jan:
>>>>>>>>>>
>>>>>>>>>> Adding and configuring apt repositories should be
>>>>>>>>>> done
>>>>>>>>>> via
>>>>>>>>>> config
>>>>>>>>>> files. It should be possible to define own
>>>>>>>>>> multiconfigs
>>>>>>>>>> while
>>>>>>>>>> including multiconfigs from other layers. These
>>>>>>>>>> configs
>>>>>>>>>> then
>>>>>>>>>> append
>>>>>>>>>> filepaths to a global variable.
>>>>>>>>>>
>>>>>>>>>> Every file that is added this way contains
>>>>>>>>>> 'sources.list'
>>>>>>>>>> compatible repository definitions. So one repo each
>>>>>>>>>> line.
>>>>>>>>>>
>>>>>>>>>> For every line in those files a repository entry for
>>>>>>>>>> multistrap.conf
>>>>>>>>>> is created. Here we might need some more complex code
>>>>>>>>>> to
>>>>>>>>>> convert
>>>>>>>>>> such a apt repo tripel to the right format multistrap
>>>>>>>>>> expects.
>>>>>>>>>> But
>>>>>>>>>> by using the 'sources.lists' format we would be
>>>>>>>>>> independent
>>>>>>>>>> of
>>>>>>>>>> multistrap and become more future proof.
>>>>>>>>
>>>>>>>> We will also need a way to tell apt the priorities of
>>>>>>>> these
>>>>>>>> repos.
>>>>>>>> Multistrap just adds them and apt-get installs packages
>>>>>>>> according
>>>>>>>> to
>>>>>>>> its default behavior.
>>>>>>>> That means that a package with the same name and version
>>>>>>>> will
>>>>>>>> get
>>>>>>>> picked from a "random" repo. Overlays will need to make
>>>>>>>> sure
>>>>>>>> to
>>>>>>>> always
>>>>>>>> have a greater version or we will need apt configuration.
>>>>>>>> https://wiki.debian.org/AptPreferences
>>>>>>>> The big question here is how/whether multistrap will
>>>>>>>> handle
>>>>>>>> apt
>>>>>>>> preference.
>>>>>>>>
>>>>>>>> Alex already ran into that when he wanted to modify
>>>>>>>> "hello".
>>>>>>>> Renaming
>>>>>>>> packages - like Alex suggested - is not the way to go.
>>>>>>>> Because
>>>>>>>> the
>>>>>>>> ones
>>>>>>>> we do overlay could be deps of packages we do not
>>>>>>>> overlay.
>>>>>>
>>>>>> The version collision could be avoided by using epoch, and if
>>>>>> I
>>>>>> understand it correctly - this is the preferable way to
>>>>>> assign
>>>>>> apt
>>>>>> priority.
>>>>>>
>>>>>> https://www.debian.org/doc/debian-policy/#s-f-version
>>>>>
>>>>> Where did you get that epoch is the preferable way to assign
>>>>> apt
>>>>> priority?
>>>>
>>>> Not the apt, just specific package. Hmm, I'm a bit incorrectly
>>>> called
>>>> my
>>>> pov. :-) I see epoch as an easy way to patch existing package
>>>> from
>>>> upstream. You don't care about upstream one's version even if it
>>>> goes
>>>> forward, until your epoch is higher - your package will be
>>>> installed.
>>>> Of
>>>> course it doesn't mean that upstream epoch couldn't be changed
>>>> too,
>>>> but
>>>> this happens quite rare.
>>>>
>>>>>
>>>>>   From you link:
>>>>>
>>>>>       epoch
>>>>>           This is a single (generally small) unsigned integer.
>>>>> It may
>>>>> be
>>>>> omitted, in which case zero is assumed. If it is omitted then
>>>>> the
>>>>> upstream_version may not contain any colons.
>>>>>           It is provided to allow mistakes in the version
>>>>> numbers of
>>>>> older versions of a package, and also a package’s previous
>>>>> version
>>>>> numbering schemes, to be left behind.
>>>>>
>>>>> I see epoch just as a possibility to fix mistakes or if
>>>>> upstream
>>>>> changed their versioning scheme, not use it use it generally
>>>>> for
>>>>> specifying package priorities. Using it that way sound more
>>>>> like a
>>>>> hack
>>>>> to me. Because what happens if debian raises the epoch of a
>>>>> package?
>>>>> Then you would have to raise yours again? Sounds like playing
>>>>> poker.
>>>>
>>>> Could you please describe the case when you have two repositories
>>>> which
>>>> contain package with the same name-version but different content?
>>>
>>> With apt-preferences we could prefer every package from our own
>>> repo
>>> with the same name AFAIK, but this would make the original package
>>> no
>>> longer install-able beside that.
>>> And that is a limitation of debian/dpkg and we can do nothing about
>>> it
>>> in isar. This problem it not ours to solve.
>>>
>>> Ideally debian would introduce package namespaces or similar. But
>>> that
>>> is completely out of isars scope. On our side we could only try
>>> some
>>> hacks to avoid name collision, but I would rather spend time
>>> solving
>>> other problems in isar than that, since just choosing a different
>>> name
>>> is pretty easy. Multi-repo or reproducible builds are much more
>>> important issues than trying to play cat and mouse with package
>>> names.
>>>
>>
>> I see this. But what is the practical usecase to have the same
>> packages
>> in several repositories?
> 
> Where did this question comes from? It does not really make sense to

 From the text above, but I found that it wasn't your message, sorry :-)

...
That means that a package with the same name and version will get
picked from a "random" repo.
...

> have exactly the same package in multiple different repositories. But
> that is AFAIK not related to any of this.
> 

So if it's not the case, then it's Ok for me.

> If you have changed some existing debian package put it into your own
> repo, maybe change the version of the package to contain something like
> "*~isar0" similar to how ubuntu does it. Then set a apt-preference like
> this:
> 
>      Package: *
>      Pin: release c=isar
>      Pin-Priority: 650
> 
> or similar. And apt would prefer to use packages from the isar repo
> instead of upstream if there is a name collision. Even if the version
> of the package in isar is lower than the one upstream.

Do you think it's possible to handle the following scenario:

1. You pick current glibc from upstream distro XYZ, let's say v1.0.
2. Patch it and put glibc-v1.0-isar to local repo.
3. Wait for some time until glibc is updated in upstream, let's say to v1.1.
4. Due to glibc-v1.0-isar is set to be preferred, it's going to be 
installed. But glibc is the core library in distro, and lots of packages 
in XYZ now require glibc-v1.1. So you won't be able to generate image 
anymore, it will fail with unmet dependency.

Alex

> 
> Cheers,
> Claudius
> 
> 
>>
>> Alex
>>
>>>>> Telling apt to prefer packages with specific versions or from
>>>>> specific
>>>>> repositories via apt-preferences sounds more like the right
>>>>> tool
>>>>> for
>>>>> the job.
>>>>>
>>>>> Cheers,
>>>>> Claudius
>>>>>
>>>>>>
>>>>>> Regarding renaming 'hello' there was a bit different case,
>>>>>> our
>>>>>> 'hello'
>>>>>> has nothing common with the upstream one, moreover I've
>>>>>> already
>>>>>> modified
>>>>>> it to use 'libhello'. So this is not the case when we patch
>>>>>> the
>>>>>> upstream
>>>>>> package. So I decided to rename it, to avoid confusion.
>>>>>>
>>>>>> In case of patching the upstream package, we could just add
>>>>>> epoch
>>>>>> to
>>>>>> version, so our package will be always in first prio.
>>>>>>
>>>>>> Also I'm not sure that when there are 2 packages with the
>>>>>> same
>>>>>> names
>>>>>> and
>>>>>> versions available in source.list - is a good practice. In
>>>>>> this
>>>>>> case
>>>>>> you
>>>>>> have no way to easily check if your rootfs contains correct
>>>>>> package,
>>>>>> 'dpkg -l' won't be enough.
>>>>>>
>>>>>> Alex
>>>>>>
>>>>>>> Thanks, yes that is a very good point.
>>>>>>>
>>>>>>> I would prefer the apt preferences settings. Maybe handle
>>>>>>> it
>>>>>>> similar to
>>>>>>> how I proposed the multi-repo support. A global variable
>>>>>>> where
>>>>>>> apt
>>>>>>> preference file paths are appended.
>>>>>>>
>>>>>>> If multistrap cannot support them with a reasonable complex
>>>>>>> conversion
>>>>>>> script then it might be time to say multistrap goodbye.
>>>>>>>
>>>>>>> Claudius
>>>>>>>
>>>>>
>>>>> --
>>>>> DENX Software Engineering GmbH,      Managing Director:
>>>>> Wolfgang
>>>>> Denk
>>>>> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
>>>>> Germany
>>>>> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@d
>>>>> enx.
>>>>> de
>>>>>
>>>>>               PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19
>>>>> 9808
>>>>> B153
>>>>>                                 Keyserver: hkp://pool.sks-
>>>>> keyservers.net
>>>>
>>>>
>>
>>