Re: Multi repo support

[Claudius Heine <ch@denx.de>]

[-- Attachment #1: Type: text/plain, Size: 13294 bytes --]

On Fri, 2018-02-02 at 16:48 +0300, Alexander Smirnov wrote:
> Hi,
> 
> On 02/02/2018 04:26 PM, Claudius Heine wrote:
> > On Fri, 2018-02-02 at 15:28 +0300, Alexander Smirnov wrote:
> > > Hi again!
> > > 
> > > > Hi,
> > > > 
> > > > On Fri, 2018-02-02 at 00:52 +0300, Alexander Smirnov wrote:
> > > > > Hi!
> > > > > 
> > > > > Claudius Heine <ch@denx.de> 2 февраля 2018 г. 0:07:49
> > > > > написал:
> > > > > 
> > > > > > Hi Alex,
> > > > > > 
> > > > > > On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov wrote:
> > > > > > > 
> > > > > > > On 02/01/2018 09:51 PM, Claudius Heine wrote:
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > On Thu, 2018-02-01 at 19:34 +0100, Henning Schild
> > > > > > > > wrote:
> > > > > > > > > Am Thu, 1 Feb 2018 16:16:58 +0100
> > > > > > > > > schrieb "[ext] Henning Schild" <henning.schild@siemen
> > > > > > > > > s.co
> > > > > > > > > m>:
> > > > > > > > > 
> > > > > > > > > > Am Thu, 1 Feb 2018 15:54:26 +0100
> > > > > > > > > > schrieb "[ext] Claudius Heine" <claudius.heine.ext@
> > > > > > > > > > siem
> > > > > > > > > > ens.
> > > > > > > > > > com>
> > > > > > > > > > :
> > > > > > > > > > 
> > > > > > > > > > > Hi,
> > > > > > > > > > > 
> > > > > > > > > > > I would like to start the discussion about how to
> > > > > > > > > > > best
> > > > > > > > > > > implement
> > > > > > > > > > > muti repository support in isar.
> > > > > > > > > > > 
> > > > > > > > > > > Does someone already has some ideas or even
> > > > > > > > > > > something
> > > > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > pipeline for this?
> > > > > > > > > > > 
> > > > > > > > > > > If not then I do have an idea that was outlined
> > > > > > > > > > > together
> > > > > > > > > > > with
> > > > > > > > > > > Jan:
> > > > > > > > > > > 
> > > > > > > > > > > Adding and configuring apt repositories should be
> > > > > > > > > > > done
> > > > > > > > > > > via
> > > > > > > > > > > config
> > > > > > > > > > > files. It should be possible to define own
> > > > > > > > > > > multiconfigs
> > > > > > > > > > > while
> > > > > > > > > > > including multiconfigs from other layers. These
> > > > > > > > > > > configs
> > > > > > > > > > > then
> > > > > > > > > > > append
> > > > > > > > > > > filepaths to a global variable.
> > > > > > > > > > > 
> > > > > > > > > > > Every file that is added this way contains
> > > > > > > > > > > 'sources.list'
> > > > > > > > > > > compatible repository definitions. So one repo
> > > > > > > > > > > each
> > > > > > > > > > > line.
> > > > > > > > > > > 
> > > > > > > > > > > For every line in those files a repository entry
> > > > > > > > > > > for
> > > > > > > > > > > multistrap.conf
> > > > > > > > > > > is created. Here we might need some more complex
> > > > > > > > > > > code
> > > > > > > > > > > to
> > > > > > > > > > > convert
> > > > > > > > > > > such a apt repo tripel to the right format
> > > > > > > > > > > multistrap
> > > > > > > > > > > expects.
> > > > > > > > > > > But
> > > > > > > > > > > by using the 'sources.lists' format we would be
> > > > > > > > > > > independent
> > > > > > > > > > > of
> > > > > > > > > > > multistrap and become more future proof.
> > > > > > > > > 
> > > > > > > > > We will also need a way to tell apt the priorities of
> > > > > > > > > these
> > > > > > > > > repos.
> > > > > > > > > Multistrap just adds them and apt-get installs
> > > > > > > > > packages
> > > > > > > > > according
> > > > > > > > > to
> > > > > > > > > its default behavior.
> > > > > > > > > That means that a package with the same name and
> > > > > > > > > version
> > > > > > > > > will
> > > > > > > > > get
> > > > > > > > > picked from a "random" repo. Overlays will need to
> > > > > > > > > make
> > > > > > > > > sure
> > > > > > > > > to
> > > > > > > > > always
> > > > > > > > > have a greater version or we will need apt
> > > > > > > > > configuration.
> > > > > > > > > https://wiki.debian.org/AptPreferences
> > > > > > > > > The big question here is how/whether multistrap will
> > > > > > > > > handle
> > > > > > > > > apt
> > > > > > > > > preference.
> > > > > > > > > 
> > > > > > > > > Alex already ran into that when he wanted to modify
> > > > > > > > > "hello".
> > > > > > > > > Renaming
> > > > > > > > > packages - like Alex suggested - is not the way to
> > > > > > > > > go.
> > > > > > > > > Because
> > > > > > > > > the
> > > > > > > > > ones
> > > > > > > > > we do overlay could be deps of packages we do not
> > > > > > > > > overlay.
> > > > > > > 
> > > > > > > The version collision could be avoided by using epoch,
> > > > > > > and if
> > > > > > > I
> > > > > > > understand it correctly - this is the preferable way to
> > > > > > > assign
> > > > > > > apt
> > > > > > > priority.
> > > > > > > 
> > > > > > > https://www.debian.org/doc/debian-policy/#s-f-version
> > > > > > 
> > > > > > Where did you get that epoch is the preferable way to
> > > > > > assign
> > > > > > apt
> > > > > > priority?
> > > > > 
> > > > > Not the apt, just specific package. Hmm, I'm a bit
> > > > > incorrectly
> > > > > called
> > > > > my
> > > > > pov. :-) I see epoch as an easy way to patch existing package
> > > > > from
> > > > > upstream. You don't care about upstream one's version even if
> > > > > it
> > > > > goes
> > > > > forward, until your epoch is higher - your package will be
> > > > > installed.
> > > > > Of
> > > > > course it doesn't mean that upstream epoch couldn't be
> > > > > changed
> > > > > too,
> > > > > but
> > > > > this happens quite rare.
> > > > > 
> > > > > > 
> > > > > >   From you link:
> > > > > > 
> > > > > >       epoch
> > > > > >           This is a single (generally small) unsigned
> > > > > > integer.
> > > > > > It may
> > > > > > be
> > > > > > omitted, in which case zero is assumed. If it is omitted
> > > > > > then
> > > > > > the
> > > > > > upstream_version may not contain any colons.
> > > > > >           It is provided to allow mistakes in the version
> > > > > > numbers of
> > > > > > older versions of a package, and also a package’s previous
> > > > > > version
> > > > > > numbering schemes, to be left behind.
> > > > > > 
> > > > > > I see epoch just as a possibility to fix mistakes or if
> > > > > > upstream
> > > > > > changed their versioning scheme, not use it use it
> > > > > > generally
> > > > > > for
> > > > > > specifying package priorities. Using it that way sound more
> > > > > > like a
> > > > > > hack
> > > > > > to me. Because what happens if debian raises the epoch of a
> > > > > > package?
> > > > > > Then you would have to raise yours again? Sounds like
> > > > > > playing
> > > > > > poker.
> > > > > 
> > > > > Could you please describe the case when you have two
> > > > > repositories
> > > > > which
> > > > > contain package with the same name-version but different
> > > > > content?
> > > > 
> > > > With apt-preferences we could prefer every package from our own
> > > > repo
> > > > with the same name AFAIK, but this would make the original
> > > > package
> > > > no
> > > > longer install-able beside that.
> > > > And that is a limitation of debian/dpkg and we can do nothing
> > > > about
> > > > it
> > > > in isar. This problem it not ours to solve.
> > > > 
> > > > Ideally debian would introduce package namespaces or similar.
> > > > But
> > > > that
> > > > is completely out of isars scope. On our side we could only try
> > > > some
> > > > hacks to avoid name collision, but I would rather spend time
> > > > solving
> > > > other problems in isar than that, since just choosing a
> > > > different
> > > > name
> > > > is pretty easy. Multi-repo or reproducible builds are much more
> > > > important issues than trying to play cat and mouse with package
> > > > names.
> > > > 
> > > 
> > > I see this. But what is the practical usecase to have the same
> > > packages
> > > in several repositories?
> > 
> > Where did this question comes from? It does not really make sense
> > to
> 
>  From the text above, but I found that it wasn't your message, sorry
> :-)
> 
> ...
> That means that a package with the same name and version will get
> picked from a "random" repo.
> ...
> 
> > have exactly the same package in multiple different repositories.
> > But
> > that is AFAIK not related to any of this.
> > 
> 
> So if it's not the case, then it's Ok for me.
> 
> > If you have changed some existing debian package put it into your
> > own
> > repo, maybe change the version of the package to contain something
> > like
> > "*~isar0" similar to how ubuntu does it. Then set a apt-preference
> > like
> > this:
> > 
> >      Package: *
> >      Pin: release c=isar
> >      Pin-Priority: 650
> > 
> > or similar. And apt would prefer to use packages from the isar repo
> > instead of upstream if there is a name collision. Even if the
> > version
> > of the package in isar is lower than the one upstream.
> 
> Do you think it's possible to handle the following scenario:
> 
> 1. You pick current glibc from upstream distro XYZ, let's say v1.0.
> 2. Patch it and put glibc-v1.0-isar to local repo.
> 3. Wait for some time until glibc is updated in upstream, let's say
> to v1.1.
> 4. Due to glibc-v1.0-isar is set to be preferred, it's going to be 
> installed. But glibc is the core library in distro, and lots of
> packages 
> in XYZ now require glibc-v1.1. So you won't be able to generate
> image 
> anymore, it will fail with unmet dependency.

Why do we need to handle this?

It would be great if image generation fails in this case and not cause
the generation of a broken image.

Otherwise that is the reason we should have reproducible builds, so
that we can still create the old image.

Cheers,
Claudius

> 
> Alex
> 
> > 
> > Cheers,
> > Claudius
> > 
> > 
> > > 
> > > Alex
> > > 
> > > > > > Telling apt to prefer packages with specific versions or
> > > > > > from
> > > > > > specific
> > > > > > repositories via apt-preferences sounds more like the right
> > > > > > tool
> > > > > > for
> > > > > > the job.
> > > > > > 
> > > > > > Cheers,
> > > > > > Claudius
> > > > > > 
> > > > > > > 
> > > > > > > Regarding renaming 'hello' there was a bit different
> > > > > > > case,
> > > > > > > our
> > > > > > > 'hello'
> > > > > > > has nothing common with the upstream one, moreover I've
> > > > > > > already
> > > > > > > modified
> > > > > > > it to use 'libhello'. So this is not the case when we
> > > > > > > patch
> > > > > > > the
> > > > > > > upstream
> > > > > > > package. So I decided to rename it, to avoid confusion.
> > > > > > > 
> > > > > > > In case of patching the upstream package, we could just
> > > > > > > add
> > > > > > > epoch
> > > > > > > to
> > > > > > > version, so our package will be always in first prio.
> > > > > > > 
> > > > > > > Also I'm not sure that when there are 2 packages with the
> > > > > > > same
> > > > > > > names
> > > > > > > and
> > > > > > > versions available in source.list - is a good practice.
> > > > > > > In
> > > > > > > this
> > > > > > > case
> > > > > > > you
> > > > > > > have no way to easily check if your rootfs contains
> > > > > > > correct
> > > > > > > package,
> > > > > > > 'dpkg -l' won't be enough.
> > > > > > > 
> > > > > > > Alex
> > > > > > > 
> > > > > > > > Thanks, yes that is a very good point.
> > > > > > > > 
> > > > > > > > I would prefer the apt preferences settings. Maybe
> > > > > > > > handle
> > > > > > > > it
> > > > > > > > similar to
> > > > > > > > how I proposed the multi-repo support. A global
> > > > > > > > variable
> > > > > > > > where
> > > > > > > > apt
> > > > > > > > preference file paths are appended.
> > > > > > > > 
> > > > > > > > If multistrap cannot support them with a reasonable
> > > > > > > > complex
> > > > > > > > conversion
> > > > > > > > script then it might be time to say multistrap goodbye.
> > > > > > > > 
> > > > > > > > Claudius
> > > > > > > > 
> > > > > > 
> > > > > > --
> > > > > > DENX Software Engineering GmbH,      Managing Director:
> > > > > > Wolfgang
> > > > > > Denk
> > > > > > HRB 165235 Munich, Office: Kirchenstr.5, D-82194
> > > > > > Groebenzell,
> > > > > > Germany
> > > > > > Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: 
> > > > > > ch@d
> > > > > > enx.
> > > > > > de
> > > > > > 
> > > > > >               PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173
> > > > > > CB19
> > > > > > 9808
> > > > > > B153
> > > > > >                                 Keyserver: hkp://pool.sks-
> > > > > > keyservers.net
> > > > > 
> > > > > 
> > > 
> > > 
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]