Re: Multi repo support

[Claudius Heine <ch@denx.de>]

[-- Attachment #1: Type: text/plain, Size: 7923 bytes --]

Hi,

On Fri, 2018-02-02 at 00:52 +0300, Alexander Smirnov wrote:
> Hi!
> 
> Claudius Heine <ch@denx.de> 2 февраля 2018 г. 0:07:49 написал:
> 
> > Hi Alex,
> > 
> > On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov wrote:
> > > 
> > > On 02/01/2018 09:51 PM, Claudius Heine wrote:
> > > > Hi,
> > > > 
> > > > On Thu, 2018-02-01 at 19:34 +0100, Henning Schild wrote:
> > > > > Am Thu, 1 Feb 2018 16:16:58 +0100
> > > > > schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
> > > > > 
> > > > > > Am Thu, 1 Feb 2018 15:54:26 +0100
> > > > > > schrieb "[ext] Claudius Heine" <claudius.heine.ext@siemens.
> > > > > > com>
> > > > > > :
> > > > > > 
> > > > > > > Hi,
> > > > > > > 
> > > > > > > I would like to start the discussion about how to best
> > > > > > > implement
> > > > > > > muti repository support in isar.
> > > > > > > 
> > > > > > > Does someone already has some ideas or even something in
> > > > > > > the
> > > > > > > pipeline for this?
> > > > > > > 
> > > > > > > If not then I do have an idea that was outlined together
> > > > > > > with
> > > > > > > Jan:
> > > > > > > 
> > > > > > > Adding and configuring apt repositories should be done
> > > > > > > via
> > > > > > > config
> > > > > > > files. It should be possible to define own multiconfigs
> > > > > > > while
> > > > > > > including multiconfigs from other layers. These configs
> > > > > > > then
> > > > > > > append
> > > > > > > filepaths to a global variable.
> > > > > > > 
> > > > > > > Every file that is added this way contains 'sources.list'
> > > > > > > compatible repository definitions. So one repo each line.
> > > > > > > 
> > > > > > > For every line in those files a repository entry for
> > > > > > > multistrap.conf
> > > > > > > is created. Here we might need some more complex code to
> > > > > > > convert
> > > > > > > such a apt repo tripel to the right format multistrap
> > > > > > > expects.
> > > > > > > But
> > > > > > > by using the 'sources.lists' format we would be
> > > > > > > independent
> > > > > > > of
> > > > > > > multistrap and become more future proof.
> > > > > 
> > > > > We will also need a way to tell apt the priorities of these
> > > > > repos.
> > > > > Multistrap just adds them and apt-get installs packages
> > > > > according
> > > > > to
> > > > > its default behavior.
> > > > > That means that a package with the same name and version will
> > > > > get
> > > > > picked from a "random" repo. Overlays will need to make sure
> > > > > to
> > > > > always
> > > > > have a greater version or we will need apt configuration.
> > > > > https://wiki.debian.org/AptPreferences
> > > > > The big question here is how/whether multistrap will handle
> > > > > apt
> > > > > preference.
> > > > > 
> > > > > Alex already ran into that when he wanted to modify "hello".
> > > > > Renaming
> > > > > packages - like Alex suggested - is not the way to go.
> > > > > Because
> > > > > the
> > > > > ones
> > > > > we do overlay could be deps of packages we do not overlay.
> > > 
> > > The version collision could be avoided by using epoch, and if I
> > > understand it correctly - this is the preferable way to assign
> > > apt
> > > priority.
> > > 
> > > https://www.debian.org/doc/debian-policy/#s-f-version
> > 
> > Where did you get that epoch is the preferable way to assign apt
> > priority?
> 
> Not the apt, just specific package. Hmm, I'm a bit incorrectly called
> my 
> pov. :-) I see epoch as an easy way to patch existing package from 
> upstream. You don't care about upstream one's version even if it
> goes 
> forward, until your epoch is higher - your package will be installed.
> Of 
> course it doesn't mean that upstream epoch couldn't be changed too,
> but 
> this happens quite rare.
> 
> > 
> > From you link:
> > 
> >     epoch
> >         This is a single (generally small) unsigned integer. It may
> > be
> > omitted, in which case zero is assumed. If it is omitted then the
> > upstream_version may not contain any colons.
> >         It is provided to allow mistakes in the version numbers of
> > older versions of a package, and also a package’s previous version
> > numbering schemes, to be left behind.
> > 
> > I see epoch just as a possibility to fix mistakes or if upstream
> > changed their versioning scheme, not use it use it generally for
> > specifying package priorities. Using it that way sound more like a
> > hack
> > to me. Because what happens if debian raises the epoch of a
> > package?
> > Then you would have to raise yours again? Sounds like playing
> > poker.
> 
> Could you please describe the case when you have two repositories
> which 
> contain package with the same name-version but different content?

With apt-preferences we could prefer every package from our own repo
with the same name AFAIK, but this would make the original package no
longer install-able beside that.
And that is a limitation of debian/dpkg and we can do nothing about it
in isar. This problem it not ours to solve.

Ideally debian would introduce package namespaces or similar. But that
is completely out of isars scope. On our side we could only try some
hacks to avoid name collision, but I would rather spend time solving
other problems in isar than that, since just choosing a different name
is pretty easy. Multi-repo or reproducible builds are much more
important issues than trying to play cat and mouse with package names.

Cheers,
Claudius

> 
> Alex
> 
> > 
> > Telling apt to prefer packages with specific versions or from
> > specific
> > repositories via apt-preferences sounds more like the right tool
> > for
> > the job.
> > 
> > Cheers,
> > Claudius
> > 
> > > 
> > > Regarding renaming 'hello' there was a bit different case, our
> > > 'hello'
> > > has nothing common with the upstream one, moreover I've already
> > > modified
> > > it to use 'libhello'. So this is not the case when we patch the
> > > upstream
> > > package. So I decided to rename it, to avoid confusion.
> > > 
> > > In case of patching the upstream package, we could just add epoch
> > > to
> > > version, so our package will be always in first prio.
> > > 
> > > Also I'm not sure that when there are 2 packages with the same
> > > names
> > > and
> > > versions available in source.list - is a good practice. In this
> > > case
> > > you
> > > have no way to easily check if your rootfs contains correct
> > > package,
> > > 'dpkg -l' won't be enough.
> > > 
> > > Alex
> > > 
> > > > Thanks, yes that is a very good point.
> > > > 
> > > > I would prefer the apt preferences settings. Maybe handle it
> > > > similar to
> > > > how I proposed the multi-repo support. A global variable where
> > > > apt
> > > > preference file paths are appended.
> > > > 
> > > > If multistrap cannot support them with a reasonable complex
> > > > conversion
> > > > script then it might be time to say multistrap goodbye.
> > > > 
> > > > Claudius
> > > > 
> > 
> > --
> > DENX Software Engineering GmbH,      Managing Director: Wolfgang
> > Denk
> > HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> > Germany
> > Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.
> > de
> > 
> >             PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808
> > B153
> >                               Keyserver: hkp://pool.sks-
> > keyservers.net
> 
> 
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]