Re: Multi repo support

[Alexander Smirnov <asmirnov@ilbers.de>]

On 02/02/2018 05:15 PM, Claudius Heine wrote:
> On Fri, 2018-02-02 at 16:48 +0300, Alexander Smirnov wrote:
>> Hi,
>>
>> On 02/02/2018 04:26 PM, Claudius Heine wrote:
>>> On Fri, 2018-02-02 at 15:28 +0300, Alexander Smirnov wrote:
>>>> Hi again!
>>>>
>>>>> Hi,
>>>>>
>>>>> On Fri, 2018-02-02 at 00:52 +0300, Alexander Smirnov wrote:
>>>>>> Hi!
>>>>>>
>>>>>> Claudius Heine <ch@denx.de> 2 февраля 2018 г. 0:07:49
>>>>>> написал:
>>>>>>
>>>>>>> Hi Alex,
>>>>>>>
>>>>>>> On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov wrote:
>>>>>>>>
>>>>>>>> On 02/01/2018 09:51 PM, Claudius Heine wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On Thu, 2018-02-01 at 19:34 +0100, Henning Schild
>>>>>>>>> wrote:
>>>>>>>>>> Am Thu, 1 Feb 2018 16:16:58 +0100
>>>>>>>>>> schrieb "[ext] Henning Schild" <henning.schild@siemen
>>>>>>>>>> s.co
>>>>>>>>>> m>:
>>>>>>>>>>
>>>>>>>>>>> Am Thu, 1 Feb 2018 15:54:26 +0100
>>>>>>>>>>> schrieb "[ext] Claudius Heine" <claudius.heine.ext@
>>>>>>>>>>> siem
>>>>>>>>>>> ens.
>>>>>>>>>>> com>
>>>>>>>>>>> :
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I would like to start the discussion about how to
>>>>>>>>>>>> best
>>>>>>>>>>>> implement
>>>>>>>>>>>> muti repository support in isar.
>>>>>>>>>>>>
>>>>>>>>>>>> Does someone already has some ideas or even
>>>>>>>>>>>> something
>>>>>>>>>>>> in
>>>>>>>>>>>> the
>>>>>>>>>>>> pipeline for this?
>>>>>>>>>>>>
>>>>>>>>>>>> If not then I do have an idea that was outlined
>>>>>>>>>>>> together
>>>>>>>>>>>> with
>>>>>>>>>>>> Jan:
>>>>>>>>>>>>
>>>>>>>>>>>> Adding and configuring apt repositories should be
>>>>>>>>>>>> done
>>>>>>>>>>>> via
>>>>>>>>>>>> config
>>>>>>>>>>>> files. It should be possible to define own
>>>>>>>>>>>> multiconfigs
>>>>>>>>>>>> while
>>>>>>>>>>>> including multiconfigs from other layers. These
>>>>>>>>>>>> configs
>>>>>>>>>>>> then
>>>>>>>>>>>> append
>>>>>>>>>>>> filepaths to a global variable.
>>>>>>>>>>>>
>>>>>>>>>>>> Every file that is added this way contains
>>>>>>>>>>>> 'sources.list'
>>>>>>>>>>>> compatible repository definitions. So one repo
>>>>>>>>>>>> each
>>>>>>>>>>>> line.
>>>>>>>>>>>>
>>>>>>>>>>>> For every line in those files a repository entry
>>>>>>>>>>>> for
>>>>>>>>>>>> multistrap.conf
>>>>>>>>>>>> is created. Here we might need some more complex
>>>>>>>>>>>> code
>>>>>>>>>>>> to
>>>>>>>>>>>> convert
>>>>>>>>>>>> such a apt repo tripel to the right format
>>>>>>>>>>>> multistrap
>>>>>>>>>>>> expects.
>>>>>>>>>>>> But
>>>>>>>>>>>> by using the 'sources.lists' format we would be
>>>>>>>>>>>> independent
>>>>>>>>>>>> of
>>>>>>>>>>>> multistrap and become more future proof.
>>>>>>>>>>
>>>>>>>>>> We will also need a way to tell apt the priorities of
>>>>>>>>>> these
>>>>>>>>>> repos.
>>>>>>>>>> Multistrap just adds them and apt-get installs
>>>>>>>>>> packages
>>>>>>>>>> according
>>>>>>>>>> to
>>>>>>>>>> its default behavior.
>>>>>>>>>> That means that a package with the same name and
>>>>>>>>>> version
>>>>>>>>>> will
>>>>>>>>>> get
>>>>>>>>>> picked from a "random" repo. Overlays will need to
>>>>>>>>>> make
>>>>>>>>>> sure
>>>>>>>>>> to
>>>>>>>>>> always
>>>>>>>>>> have a greater version or we will need apt
>>>>>>>>>> configuration.
>>>>>>>>>> https://wiki.debian.org/AptPreferences
>>>>>>>>>> The big question here is how/whether multistrap will
>>>>>>>>>> handle
>>>>>>>>>> apt
>>>>>>>>>> preference.
>>>>>>>>>>
>>>>>>>>>> Alex already ran into that when he wanted to modify
>>>>>>>>>> "hello".
>>>>>>>>>> Renaming
>>>>>>>>>> packages - like Alex suggested - is not the way to
>>>>>>>>>> go.
>>>>>>>>>> Because
>>>>>>>>>> the
>>>>>>>>>> ones
>>>>>>>>>> we do overlay could be deps of packages we do not
>>>>>>>>>> overlay.
>>>>>>>>
>>>>>>>> The version collision could be avoided by using epoch,
>>>>>>>> and if
>>>>>>>> I
>>>>>>>> understand it correctly - this is the preferable way to
>>>>>>>> assign
>>>>>>>> apt
>>>>>>>> priority.
>>>>>>>>
>>>>>>>> https://www.debian.org/doc/debian-policy/#s-f-version
>>>>>>>
>>>>>>> Where did you get that epoch is the preferable way to
>>>>>>> assign
>>>>>>> apt
>>>>>>> priority?
>>>>>>
>>>>>> Not the apt, just specific package. Hmm, I'm a bit
>>>>>> incorrectly
>>>>>> called
>>>>>> my
>>>>>> pov. :-) I see epoch as an easy way to patch existing package
>>>>>> from
>>>>>> upstream. You don't care about upstream one's version even if
>>>>>> it
>>>>>> goes
>>>>>> forward, until your epoch is higher - your package will be
>>>>>> installed.
>>>>>> Of
>>>>>> course it doesn't mean that upstream epoch couldn't be
>>>>>> changed
>>>>>> too,
>>>>>> but
>>>>>> this happens quite rare.
>>>>>>
>>>>>>>
>>>>>>>    From you link:
>>>>>>>
>>>>>>>        epoch
>>>>>>>            This is a single (generally small) unsigned
>>>>>>> integer.
>>>>>>> It may
>>>>>>> be
>>>>>>> omitted, in which case zero is assumed. If it is omitted
>>>>>>> then
>>>>>>> the
>>>>>>> upstream_version may not contain any colons.
>>>>>>>            It is provided to allow mistakes in the version
>>>>>>> numbers of
>>>>>>> older versions of a package, and also a package’s previous
>>>>>>> version
>>>>>>> numbering schemes, to be left behind.
>>>>>>>
>>>>>>> I see epoch just as a possibility to fix mistakes or if
>>>>>>> upstream
>>>>>>> changed their versioning scheme, not use it use it
>>>>>>> generally
>>>>>>> for
>>>>>>> specifying package priorities. Using it that way sound more
>>>>>>> like a
>>>>>>> hack
>>>>>>> to me. Because what happens if debian raises the epoch of a
>>>>>>> package?
>>>>>>> Then you would have to raise yours again? Sounds like
>>>>>>> playing
>>>>>>> poker.
>>>>>>
>>>>>> Could you please describe the case when you have two
>>>>>> repositories
>>>>>> which
>>>>>> contain package with the same name-version but different
>>>>>> content?
>>>>>
>>>>> With apt-preferences we could prefer every package from our own
>>>>> repo
>>>>> with the same name AFAIK, but this would make the original
>>>>> package
>>>>> no
>>>>> longer install-able beside that.
>>>>> And that is a limitation of debian/dpkg and we can do nothing
>>>>> about
>>>>> it
>>>>> in isar. This problem it not ours to solve.
>>>>>
>>>>> Ideally debian would introduce package namespaces or similar.
>>>>> But
>>>>> that
>>>>> is completely out of isars scope. On our side we could only try
>>>>> some
>>>>> hacks to avoid name collision, but I would rather spend time
>>>>> solving
>>>>> other problems in isar than that, since just choosing a
>>>>> different
>>>>> name
>>>>> is pretty easy. Multi-repo or reproducible builds are much more
>>>>> important issues than trying to play cat and mouse with package
>>>>> names.
>>>>>
>>>>
>>>> I see this. But what is the practical usecase to have the same
>>>> packages
>>>> in several repositories?
>>>
>>> Where did this question comes from? It does not really make sense
>>> to
>>
>>   From the text above, but I found that it wasn't your message, sorry
>> :-)
>>
>> ...
>> That means that a package with the same name and version will get
>> picked from a "random" repo.
>> ...
>>
>>> have exactly the same package in multiple different repositories.
>>> But
>>> that is AFAIK not related to any of this.
>>>
>>
>> So if it's not the case, then it's Ok for me.
>>
>>> If you have changed some existing debian package put it into your
>>> own
>>> repo, maybe change the version of the package to contain something
>>> like
>>> "*~isar0" similar to how ubuntu does it. Then set a apt-preference
>>> like
>>> this:
>>>
>>>       Package: *
>>>       Pin: release c=isar
>>>       Pin-Priority: 650
>>>
>>> or similar. And apt would prefer to use packages from the isar repo
>>> instead of upstream if there is a name collision. Even if the
>>> version
>>> of the package in isar is lower than the one upstream.
>>
>> Do you think it's possible to handle the following scenario:
>>
>> 1. You pick current glibc from upstream distro XYZ, let's say v1.0.
>> 2. Patch it and put glibc-v1.0-isar to local repo.
>> 3. Wait for some time until glibc is updated in upstream, let's say
>> to v1.1.
>> 4. Due to glibc-v1.0-isar is set to be preferred, it's going to be
>> installed. But glibc is the core library in distro, and lots of
>> packages
>> in XYZ now require glibc-v1.1. So you won't be able to generate
>> image
>> anymore, it will fail with unmet dependency.
> 
> Why do we need to handle this?

If you use upstream sources and prioritize apts, this problem likely 
will occur. So repo prioritizing is then unpredictable feature.

> 
> It would be great if image generation fails in this case and not cause
> the generation of a broken image.
> 
> Otherwise that is the reason we should have reproducible builds, so
> that we can still create the old image.

If you don't work with upstream and use only local repositories, why 
can't you just drop duplicated packages from them?

Alex

>>>>>>> Telling apt to prefer packages with specific versions or
>>>>>>> from
>>>>>>> specific
>>>>>>> repositories via apt-preferences sounds more like the right
>>>>>>> tool
>>>>>>> for
>>>>>>> the job.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Claudius
>>>>>>>
>>>>>>>>
>>>>>>>> Regarding renaming 'hello' there was a bit different
>>>>>>>> case,
>>>>>>>> our
>>>>>>>> 'hello'
>>>>>>>> has nothing common with the upstream one, moreover I've
>>>>>>>> already
>>>>>>>> modified
>>>>>>>> it to use 'libhello'. So this is not the case when we
>>>>>>>> patch
>>>>>>>> the
>>>>>>>> upstream
>>>>>>>> package. So I decided to rename it, to avoid confusion.
>>>>>>>>
>>>>>>>> In case of patching the upstream package, we could just
>>>>>>>> add
>>>>>>>> epoch
>>>>>>>> to
>>>>>>>> version, so our package will be always in first prio.
>>>>>>>>
>>>>>>>> Also I'm not sure that when there are 2 packages with the
>>>>>>>> same
>>>>>>>> names
>>>>>>>> and
>>>>>>>> versions available in source.list - is a good practice.
>>>>>>>> In
>>>>>>>> this
>>>>>>>> case
>>>>>>>> you
>>>>>>>> have no way to easily check if your rootfs contains
>>>>>>>> correct
>>>>>>>> package,
>>>>>>>> 'dpkg -l' won't be enough.
>>>>>>>>
>>>>>>>> Alex
>>>>>>>>
>>>>>>>>> Thanks, yes that is a very good point.
>>>>>>>>>
>>>>>>>>> I would prefer the apt preferences settings. Maybe
>>>>>>>>> handle
>>>>>>>>> it
>>>>>>>>> similar to
>>>>>>>>> how I proposed the multi-repo support. A global
>>>>>>>>> variable
>>>>>>>>> where
>>>>>>>>> apt
>>>>>>>>> preference file paths are appended.
>>>>>>>>>
>>>>>>>>> If multistrap cannot support them with a reasonable
>>>>>>>>> complex
>>>>>>>>> conversion
>>>>>>>>> script then it might be time to say multistrap goodbye.
>>>>>>>>>
>>>>>>>>> Claudius
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> DENX Software Engineering GmbH,      Managing Director:
>>>>>>> Wolfgang
>>>>>>> Denk
>>>>>>> HRB 165235 Munich, Office: Kirchenstr.5, D-82194
>>>>>>> Groebenzell,
>>>>>>> Germany
>>>>>>> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email:
>>>>>>> ch@d
>>>>>>> enx.
>>>>>>> de
>>>>>>>
>>>>>>>                PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173
>>>>>>> CB19
>>>>>>> 9808
>>>>>>> B153
>>>>>>>                                  Keyserver: hkp://pool.sks-
>>>>>>> keyservers.net