Re: Multi repo support

[Alexander Smirnov <asmirnov@ilbers.de>]

Hi again!

> Hi,
> 
> On Fri, 2018-02-02 at 00:52 +0300, Alexander Smirnov wrote:
>> Hi!
>>
>> Claudius Heine <ch@denx.de> 2 февраля 2018 г. 0:07:49 написал:
>>
>>> Hi Alex,
>>>
>>> On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov wrote:
>>>>
>>>> On 02/01/2018 09:51 PM, Claudius Heine wrote:
>>>>> Hi,
>>>>>
>>>>> On Thu, 2018-02-01 at 19:34 +0100, Henning Schild wrote:
>>>>>> Am Thu, 1 Feb 2018 16:16:58 +0100
>>>>>> schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
>>>>>>
>>>>>>> Am Thu, 1 Feb 2018 15:54:26 +0100
>>>>>>> schrieb "[ext] Claudius Heine" <claudius.heine.ext@siemens.
>>>>>>> com>
>>>>>>> :
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I would like to start the discussion about how to best
>>>>>>>> implement
>>>>>>>> muti repository support in isar.
>>>>>>>>
>>>>>>>> Does someone already has some ideas or even something in
>>>>>>>> the
>>>>>>>> pipeline for this?
>>>>>>>>
>>>>>>>> If not then I do have an idea that was outlined together
>>>>>>>> with
>>>>>>>> Jan:
>>>>>>>>
>>>>>>>> Adding and configuring apt repositories should be done
>>>>>>>> via
>>>>>>>> config
>>>>>>>> files. It should be possible to define own multiconfigs
>>>>>>>> while
>>>>>>>> including multiconfigs from other layers. These configs
>>>>>>>> then
>>>>>>>> append
>>>>>>>> filepaths to a global variable.
>>>>>>>>
>>>>>>>> Every file that is added this way contains 'sources.list'
>>>>>>>> compatible repository definitions. So one repo each line.
>>>>>>>>
>>>>>>>> For every line in those files a repository entry for
>>>>>>>> multistrap.conf
>>>>>>>> is created. Here we might need some more complex code to
>>>>>>>> convert
>>>>>>>> such a apt repo tripel to the right format multistrap
>>>>>>>> expects.
>>>>>>>> But
>>>>>>>> by using the 'sources.lists' format we would be
>>>>>>>> independent
>>>>>>>> of
>>>>>>>> multistrap and become more future proof.
>>>>>>
>>>>>> We will also need a way to tell apt the priorities of these
>>>>>> repos.
>>>>>> Multistrap just adds them and apt-get installs packages
>>>>>> according
>>>>>> to
>>>>>> its default behavior.
>>>>>> That means that a package with the same name and version will
>>>>>> get
>>>>>> picked from a "random" repo. Overlays will need to make sure
>>>>>> to
>>>>>> always
>>>>>> have a greater version or we will need apt configuration.
>>>>>> https://wiki.debian.org/AptPreferences
>>>>>> The big question here is how/whether multistrap will handle
>>>>>> apt
>>>>>> preference.
>>>>>>
>>>>>> Alex already ran into that when he wanted to modify "hello".
>>>>>> Renaming
>>>>>> packages - like Alex suggested - is not the way to go.
>>>>>> Because
>>>>>> the
>>>>>> ones
>>>>>> we do overlay could be deps of packages we do not overlay.
>>>>
>>>> The version collision could be avoided by using epoch, and if I
>>>> understand it correctly - this is the preferable way to assign
>>>> apt
>>>> priority.
>>>>
>>>> https://www.debian.org/doc/debian-policy/#s-f-version
>>>
>>> Where did you get that epoch is the preferable way to assign apt
>>> priority?
>>
>> Not the apt, just specific package. Hmm, I'm a bit incorrectly called
>> my
>> pov. :-) I see epoch as an easy way to patch existing package from
>> upstream. You don't care about upstream one's version even if it
>> goes
>> forward, until your epoch is higher - your package will be installed.
>> Of
>> course it doesn't mean that upstream epoch couldn't be changed too,
>> but
>> this happens quite rare.
>>
>>>
>>>  From you link:
>>>
>>>      epoch
>>>          This is a single (generally small) unsigned integer. It may
>>> be
>>> omitted, in which case zero is assumed. If it is omitted then the
>>> upstream_version may not contain any colons.
>>>          It is provided to allow mistakes in the version numbers of
>>> older versions of a package, and also a package’s previous version
>>> numbering schemes, to be left behind.
>>>
>>> I see epoch just as a possibility to fix mistakes or if upstream
>>> changed their versioning scheme, not use it use it generally for
>>> specifying package priorities. Using it that way sound more like a
>>> hack
>>> to me. Because what happens if debian raises the epoch of a
>>> package?
>>> Then you would have to raise yours again? Sounds like playing
>>> poker.
>>
>> Could you please describe the case when you have two repositories
>> which
>> contain package with the same name-version but different content?
> 
> With apt-preferences we could prefer every package from our own repo
> with the same name AFAIK, but this would make the original package no
> longer install-able beside that.
> And that is a limitation of debian/dpkg and we can do nothing about it
> in isar. This problem it not ours to solve.
> 
> Ideally debian would introduce package namespaces or similar. But that
> is completely out of isars scope. On our side we could only try some
> hacks to avoid name collision, but I would rather spend time solving
> other problems in isar than that, since just choosing a different name
> is pretty easy. Multi-repo or reproducible builds are much more
> important issues than trying to play cat and mouse with package names.
> 

I see this. But what is the practical usecase to have the same packages 
in several repositories?

Alex

>>> Telling apt to prefer packages with specific versions or from
>>> specific
>>> repositories via apt-preferences sounds more like the right tool
>>> for
>>> the job.
>>>
>>> Cheers,
>>> Claudius
>>>
>>>>
>>>> Regarding renaming 'hello' there was a bit different case, our
>>>> 'hello'
>>>> has nothing common with the upstream one, moreover I've already
>>>> modified
>>>> it to use 'libhello'. So this is not the case when we patch the
>>>> upstream
>>>> package. So I decided to rename it, to avoid confusion.
>>>>
>>>> In case of patching the upstream package, we could just add epoch
>>>> to
>>>> version, so our package will be always in first prio.
>>>>
>>>> Also I'm not sure that when there are 2 packages with the same
>>>> names
>>>> and
>>>> versions available in source.list - is a good practice. In this
>>>> case
>>>> you
>>>> have no way to easily check if your rootfs contains correct
>>>> package,
>>>> 'dpkg -l' won't be enough.
>>>>
>>>> Alex
>>>>
>>>>> Thanks, yes that is a very good point.
>>>>>
>>>>> I would prefer the apt preferences settings. Maybe handle it
>>>>> similar to
>>>>> how I proposed the multi-repo support. A global variable where
>>>>> apt
>>>>> preference file paths are appended.
>>>>>
>>>>> If multistrap cannot support them with a reasonable complex
>>>>> conversion
>>>>> script then it might be time to say multistrap goodbye.
>>>>>
>>>>> Claudius
>>>>>
>>>
>>> --
>>> DENX Software Engineering GmbH,      Managing Director: Wolfgang
>>> Denk
>>> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
>>> Germany
>>> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.
>>> de
>>>
>>>              PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808
>>> B153
>>>                                Keyserver: hkp://pool.sks-
>>> keyservers.net
>>
>>

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov