Re: Multi repo support

[Claudius Heine <ch@denx.de>]

[-- Attachment #1: Type: text/plain, Size: 17276 bytes --]

On Fri, 2018-02-02 at 17:36 +0300, Alexander Smirnov wrote:
> On 02/02/2018 05:15 PM, Claudius Heine wrote:
> > On Fri, 2018-02-02 at 16:48 +0300, Alexander Smirnov wrote:
> > > Hi,
> > > 
> > > On 02/02/2018 04:26 PM, Claudius Heine wrote:
> > > > On Fri, 2018-02-02 at 15:28 +0300, Alexander Smirnov wrote:
> > > > > Hi again!
> > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > On Fri, 2018-02-02 at 00:52 +0300, Alexander Smirnov wrote:
> > > > > > > Hi!
> > > > > > > 
> > > > > > > Claudius Heine <ch@denx.de> 2 февраля 2018 г. 0:07:49
> > > > > > > написал:
> > > > > > > 
> > > > > > > > Hi Alex,
> > > > > > > > 
> > > > > > > > On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov
> > > > > > > > wrote:
> > > > > > > > > 
> > > > > > > > > On 02/01/2018 09:51 PM, Claudius Heine wrote:
> > > > > > > > > > Hi,
> > > > > > > > > > 
> > > > > > > > > > On Thu, 2018-02-01 at 19:34 +0100, Henning Schild
> > > > > > > > > > wrote:
> > > > > > > > > > > Am Thu, 1 Feb 2018 16:16:58 +0100
> > > > > > > > > > > schrieb "[ext] Henning Schild" <henning.schild@si
> > > > > > > > > > > emen
> > > > > > > > > > > s.co
> > > > > > > > > > > m>:
> > > > > > > > > > > 
> > > > > > > > > > > > Am Thu, 1 Feb 2018 15:54:26 +0100
> > > > > > > > > > > > schrieb "[ext] Claudius Heine"
> > > > > > > > > > > > <claudius.heine.ext@
> > > > > > > > > > > > siem
> > > > > > > > > > > > ens.
> > > > > > > > > > > > com>
> > > > > > > > > > > > :
> > > > > > > > > > > > 
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I would like to start the discussion about
> > > > > > > > > > > > > how to
> > > > > > > > > > > > > best
> > > > > > > > > > > > > implement
> > > > > > > > > > > > > muti repository support in isar.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Does someone already has some ideas or even
> > > > > > > > > > > > > something
> > > > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > pipeline for this?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > If not then I do have an idea that was
> > > > > > > > > > > > > outlined
> > > > > > > > > > > > > together
> > > > > > > > > > > > > with
> > > > > > > > > > > > > Jan:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Adding and configuring apt repositories
> > > > > > > > > > > > > should be
> > > > > > > > > > > > > done
> > > > > > > > > > > > > via
> > > > > > > > > > > > > config
> > > > > > > > > > > > > files. It should be possible to define own
> > > > > > > > > > > > > multiconfigs
> > > > > > > > > > > > > while
> > > > > > > > > > > > > including multiconfigs from other layers.
> > > > > > > > > > > > > These
> > > > > > > > > > > > > configs
> > > > > > > > > > > > > then
> > > > > > > > > > > > > append
> > > > > > > > > > > > > filepaths to a global variable.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Every file that is added this way contains
> > > > > > > > > > > > > 'sources.list'
> > > > > > > > > > > > > compatible repository definitions. So one
> > > > > > > > > > > > > repo
> > > > > > > > > > > > > each
> > > > > > > > > > > > > line.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > For every line in those files a repository
> > > > > > > > > > > > > entry
> > > > > > > > > > > > > for
> > > > > > > > > > > > > multistrap.conf
> > > > > > > > > > > > > is created. Here we might need some more
> > > > > > > > > > > > > complex
> > > > > > > > > > > > > code
> > > > > > > > > > > > > to
> > > > > > > > > > > > > convert
> > > > > > > > > > > > > such a apt repo tripel to the right format
> > > > > > > > > > > > > multistrap
> > > > > > > > > > > > > expects.
> > > > > > > > > > > > > But
> > > > > > > > > > > > > by using the 'sources.lists' format we would
> > > > > > > > > > > > > be
> > > > > > > > > > > > > independent
> > > > > > > > > > > > > of
> > > > > > > > > > > > > multistrap and become more future proof.
> > > > > > > > > > > 
> > > > > > > > > > > We will also need a way to tell apt the
> > > > > > > > > > > priorities of
> > > > > > > > > > > these
> > > > > > > > > > > repos.
> > > > > > > > > > > Multistrap just adds them and apt-get installs
> > > > > > > > > > > packages
> > > > > > > > > > > according
> > > > > > > > > > > to
> > > > > > > > > > > its default behavior.
> > > > > > > > > > > That means that a package with the same name and
> > > > > > > > > > > version
> > > > > > > > > > > will
> > > > > > > > > > > get
> > > > > > > > > > > picked from a "random" repo. Overlays will need
> > > > > > > > > > > to
> > > > > > > > > > > make
> > > > > > > > > > > sure
> > > > > > > > > > > to
> > > > > > > > > > > always
> > > > > > > > > > > have a greater version or we will need apt
> > > > > > > > > > > configuration.
> > > > > > > > > > > https://wiki.debian.org/AptPreferences
> > > > > > > > > > > The big question here is how/whether multistrap
> > > > > > > > > > > will
> > > > > > > > > > > handle
> > > > > > > > > > > apt
> > > > > > > > > > > preference.
> > > > > > > > > > > 
> > > > > > > > > > > Alex already ran into that when he wanted to
> > > > > > > > > > > modify
> > > > > > > > > > > "hello".
> > > > > > > > > > > Renaming
> > > > > > > > > > > packages - like Alex suggested - is not the way
> > > > > > > > > > > to
> > > > > > > > > > > go.
> > > > > > > > > > > Because
> > > > > > > > > > > the
> > > > > > > > > > > ones
> > > > > > > > > > > we do overlay could be deps of packages we do not
> > > > > > > > > > > overlay.
> > > > > > > > > 
> > > > > > > > > The version collision could be avoided by using
> > > > > > > > > epoch,
> > > > > > > > > and if
> > > > > > > > > I
> > > > > > > > > understand it correctly - this is the preferable way
> > > > > > > > > to
> > > > > > > > > assign
> > > > > > > > > apt
> > > > > > > > > priority.
> > > > > > > > > 
> > > > > > > > > https://www.debian.org/doc/debian-policy/#s-f-version
> > > > > > > > 
> > > > > > > > Where did you get that epoch is the preferable way to
> > > > > > > > assign
> > > > > > > > apt
> > > > > > > > priority?
> > > > > > > 
> > > > > > > Not the apt, just specific package. Hmm, I'm a bit
> > > > > > > incorrectly
> > > > > > > called
> > > > > > > my
> > > > > > > pov. :-) I see epoch as an easy way to patch existing
> > > > > > > package
> > > > > > > from
> > > > > > > upstream. You don't care about upstream one's version
> > > > > > > even if
> > > > > > > it
> > > > > > > goes
> > > > > > > forward, until your epoch is higher - your package will
> > > > > > > be
> > > > > > > installed.
> > > > > > > Of
> > > > > > > course it doesn't mean that upstream epoch couldn't be
> > > > > > > changed
> > > > > > > too,
> > > > > > > but
> > > > > > > this happens quite rare.
> > > > > > > 
> > > > > > > > 
> > > > > > > >    From you link:
> > > > > > > > 
> > > > > > > >        epoch
> > > > > > > >            This is a single (generally small) unsigned
> > > > > > > > integer.
> > > > > > > > It may
> > > > > > > > be
> > > > > > > > omitted, in which case zero is assumed. If it is
> > > > > > > > omitted
> > > > > > > > then
> > > > > > > > the
> > > > > > > > upstream_version may not contain any colons.
> > > > > > > >            It is provided to allow mistakes in the
> > > > > > > > version
> > > > > > > > numbers of
> > > > > > > > older versions of a package, and also a package’s
> > > > > > > > previous
> > > > > > > > version
> > > > > > > > numbering schemes, to be left behind.
> > > > > > > > 
> > > > > > > > I see epoch just as a possibility to fix mistakes or if
> > > > > > > > upstream
> > > > > > > > changed their versioning scheme, not use it use it
> > > > > > > > generally
> > > > > > > > for
> > > > > > > > specifying package priorities. Using it that way sound
> > > > > > > > more
> > > > > > > > like a
> > > > > > > > hack
> > > > > > > > to me. Because what happens if debian raises the epoch
> > > > > > > > of a
> > > > > > > > package?
> > > > > > > > Then you would have to raise yours again? Sounds like
> > > > > > > > playing
> > > > > > > > poker.
> > > > > > > 
> > > > > > > Could you please describe the case when you have two
> > > > > > > repositories
> > > > > > > which
> > > > > > > contain package with the same name-version but different
> > > > > > > content?
> > > > > > 
> > > > > > With apt-preferences we could prefer every package from our
> > > > > > own
> > > > > > repo
> > > > > > with the same name AFAIK, but this would make the original
> > > > > > package
> > > > > > no
> > > > > > longer install-able beside that.
> > > > > > And that is a limitation of debian/dpkg and we can do
> > > > > > nothing
> > > > > > about
> > > > > > it
> > > > > > in isar. This problem it not ours to solve.
> > > > > > 
> > > > > > Ideally debian would introduce package namespaces or
> > > > > > similar.
> > > > > > But
> > > > > > that
> > > > > > is completely out of isars scope. On our side we could only
> > > > > > try
> > > > > > some
> > > > > > hacks to avoid name collision, but I would rather spend
> > > > > > time
> > > > > > solving
> > > > > > other problems in isar than that, since just choosing a
> > > > > > different
> > > > > > name
> > > > > > is pretty easy. Multi-repo or reproducible builds are much
> > > > > > more
> > > > > > important issues than trying to play cat and mouse with
> > > > > > package
> > > > > > names.
> > > > > > 
> > > > > 
> > > > > I see this. But what is the practical usecase to have the
> > > > > same
> > > > > packages
> > > > > in several repositories?
> > > > 
> > > > Where did this question comes from? It does not really make
> > > > sense
> > > > to
> > > 
> > >   From the text above, but I found that it wasn't your message,
> > > sorry
> > > :-)
> > > 
> > > ...
> > > That means that a package with the same name and version will get
> > > picked from a "random" repo.
> > > ...
> > > 
> > > > have exactly the same package in multiple different
> > > > repositories.
> > > > But
> > > > that is AFAIK not related to any of this.
> > > > 
> > > 
> > > So if it's not the case, then it's Ok for me.
> > > 
> > > > If you have changed some existing debian package put it into
> > > > your
> > > > own
> > > > repo, maybe change the version of the package to contain
> > > > something
> > > > like
> > > > "*~isar0" similar to how ubuntu does it. Then set a apt-
> > > > preference
> > > > like
> > > > this:
> > > > 
> > > >       Package: *
> > > >       Pin: release c=isar
> > > >       Pin-Priority: 650
> > > > 
> > > > or similar. And apt would prefer to use packages from the isar
> > > > repo
> > > > instead of upstream if there is a name collision. Even if the
> > > > version
> > > > of the package in isar is lower than the one upstream.
> > > 
> > > Do you think it's possible to handle the following scenario:
> > > 
> > > 1. You pick current glibc from upstream distro XYZ, let's say
> > > v1.0.
> > > 2. Patch it and put glibc-v1.0-isar to local repo.
> > > 3. Wait for some time until glibc is updated in upstream, let's
> > > say
> > > to v1.1.
> > > 4. Due to glibc-v1.0-isar is set to be preferred, it's going to
> > > be
> > > installed. But glibc is the core library in distro, and lots of
> > > packages
> > > in XYZ now require glibc-v1.1. So you won't be able to generate
> > > image
> > > anymore, it will fail with unmet dependency.
> > 
> > Why do we need to handle this?
> 
> If you use upstream sources and prioritize apts, this problem likely 
> will occur. So repo prioritizing is then unpredictable feature.

Yes, but that is predictable unpredictable.
If you want predictability don't mix new upstream and old own stuff.

You need to have the reproducibility data from that one point in time
where you own stuff and upstream has worked together and was tested
against each other. Otherwise build fails and you need to update your
code.

To make it absolutely clear, the only method I see to solve this with
isar is implementing reproducibility. And then this is no longer a
problem.

> 
> > 
> > It would be great if image generation fails in this case and not
> > cause
> > the generation of a broken image.
> > 
> > Otherwise that is the reason we should have reproducible builds, so
> > that we can still create the old image.
> 
> If you don't work with upstream and use only local repositories, why 
> can't you just drop duplicated packages from them?

That completely depends on how you are implementing reproducible
builds.

If reproducibilty is implemented the way I suggested, then the local
upstream mirror would have an entry in the package index for the other
package but would not store the package data itself, since it was not
downloaded while building the image. The apt preference entry would
still be needed.

So I don't see prioritizing as an unpredictable feature, since you
either have to make sure that the custom stuff is up to date or that
you have reproducible data from that time when your custom stuff was up
to date.

IMO updating/fetching stuff from upstream should not be the default
only the fallback. This way the dev would always has the reproducible
data that has worked for him once available.

Cheers,
Claudius

> 
> Alex
> 
> > > > > > > > Telling apt to prefer packages with specific versions
> > > > > > > > or
> > > > > > > > from
> > > > > > > > specific
> > > > > > > > repositories via apt-preferences sounds more like the
> > > > > > > > right
> > > > > > > > tool
> > > > > > > > for
> > > > > > > > the job.
> > > > > > > > 
> > > > > > > > Cheers,
> > > > > > > > Claudius
> > > > > > > > 
> > > > > > > > > 
> > > > > > > > > Regarding renaming 'hello' there was a bit different
> > > > > > > > > case,
> > > > > > > > > our
> > > > > > > > > 'hello'
> > > > > > > > > has nothing common with the upstream one, moreover
> > > > > > > > > I've
> > > > > > > > > already
> > > > > > > > > modified
> > > > > > > > > it to use 'libhello'. So this is not the case when we
> > > > > > > > > patch
> > > > > > > > > the
> > > > > > > > > upstream
> > > > > > > > > package. So I decided to rename it, to avoid
> > > > > > > > > confusion.
> > > > > > > > > 
> > > > > > > > > In case of patching the upstream package, we could
> > > > > > > > > just
> > > > > > > > > add
> > > > > > > > > epoch
> > > > > > > > > to
> > > > > > > > > version, so our package will be always in first prio.
> > > > > > > > > 
> > > > > > > > > Also I'm not sure that when there are 2 packages with
> > > > > > > > > the
> > > > > > > > > same
> > > > > > > > > names
> > > > > > > > > and
> > > > > > > > > versions available in source.list - is a good
> > > > > > > > > practice.
> > > > > > > > > In
> > > > > > > > > this
> > > > > > > > > case
> > > > > > > > > you
> > > > > > > > > have no way to easily check if your rootfs contains
> > > > > > > > > correct
> > > > > > > > > package,
> > > > > > > > > 'dpkg -l' won't be enough.
> > > > > > > > > 
> > > > > > > > > Alex
> > > > > > > > > 
> > > > > > > > > > Thanks, yes that is a very good point.
> > > > > > > > > > 
> > > > > > > > > > I would prefer the apt preferences settings. Maybe
> > > > > > > > > > handle
> > > > > > > > > > it
> > > > > > > > > > similar to
> > > > > > > > > > how I proposed the multi-repo support. A global
> > > > > > > > > > variable
> > > > > > > > > > where
> > > > > > > > > > apt
> > > > > > > > > > preference file paths are appended.
> > > > > > > > > > 
> > > > > > > > > > If multistrap cannot support them with a reasonable
> > > > > > > > > > complex
> > > > > > > > > > conversion
> > > > > > > > > > script then it might be time to say multistrap
> > > > > > > > > > goodbye.
> > > > > > > > > > 
> > > > > > > > > > Claudius
> > > > > > > > > > 
> > > > > > > > 
> > > > > > > > --
> > > > > > > > DENX Software Engineering GmbH,      Managing Director:
> > > > > > > > Wolfgang
> > > > > > > > Denk
> > > > > > > > HRB 165235 Munich, Office: Kirchenstr.5, D-82194
> > > > > > > > Groebenzell,
> > > > > > > > Germany
> > > > > > > > Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80
> > > > > > > > Email:
> > > > > > > > ch@d
> > > > > > > > enx.
> > > > > > > > de
> > > > > > > > 
> > > > > > > >                PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1
> > > > > > > > 1173
> > > > > > > > CB19
> > > > > > > > 9808
> > > > > > > > B153
> > > > > > > >                                  Keyserver:
> > > > > > > > hkp://pool.sks-
> > > > > > > > keyservers.net
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]