Re: Multi repo support

[Claudius Heine <ch@denx.de>]

[-- Attachment #1: Type: text/plain, Size: 10216 bytes --]

On Fri, 2018-02-02 at 15:28 +0300, Alexander Smirnov wrote:
> Hi again!
> 
> > Hi,
> > 
> > On Fri, 2018-02-02 at 00:52 +0300, Alexander Smirnov wrote:
> > > Hi!
> > > 
> > > Claudius Heine <ch@denx.de> 2 февраля 2018 г. 0:07:49 написал:
> > > 
> > > > Hi Alex,
> > > > 
> > > > On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov wrote:
> > > > > 
> > > > > On 02/01/2018 09:51 PM, Claudius Heine wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > On Thu, 2018-02-01 at 19:34 +0100, Henning Schild wrote:
> > > > > > > Am Thu, 1 Feb 2018 16:16:58 +0100
> > > > > > > schrieb "[ext] Henning Schild" <henning.schild@siemens.co
> > > > > > > m>:
> > > > > > > 
> > > > > > > > Am Thu, 1 Feb 2018 15:54:26 +0100
> > > > > > > > schrieb "[ext] Claudius Heine" <claudius.heine.ext@siem
> > > > > > > > ens.
> > > > > > > > com>
> > > > > > > > :
> > > > > > > > 
> > > > > > > > > Hi,
> > > > > > > > > 
> > > > > > > > > I would like to start the discussion about how to
> > > > > > > > > best
> > > > > > > > > implement
> > > > > > > > > muti repository support in isar.
> > > > > > > > > 
> > > > > > > > > Does someone already has some ideas or even something
> > > > > > > > > in
> > > > > > > > > the
> > > > > > > > > pipeline for this?
> > > > > > > > > 
> > > > > > > > > If not then I do have an idea that was outlined
> > > > > > > > > together
> > > > > > > > > with
> > > > > > > > > Jan:
> > > > > > > > > 
> > > > > > > > > Adding and configuring apt repositories should be
> > > > > > > > > done
> > > > > > > > > via
> > > > > > > > > config
> > > > > > > > > files. It should be possible to define own
> > > > > > > > > multiconfigs
> > > > > > > > > while
> > > > > > > > > including multiconfigs from other layers. These
> > > > > > > > > configs
> > > > > > > > > then
> > > > > > > > > append
> > > > > > > > > filepaths to a global variable.
> > > > > > > > > 
> > > > > > > > > Every file that is added this way contains
> > > > > > > > > 'sources.list'
> > > > > > > > > compatible repository definitions. So one repo each
> > > > > > > > > line.
> > > > > > > > > 
> > > > > > > > > For every line in those files a repository entry for
> > > > > > > > > multistrap.conf
> > > > > > > > > is created. Here we might need some more complex code
> > > > > > > > > to
> > > > > > > > > convert
> > > > > > > > > such a apt repo tripel to the right format multistrap
> > > > > > > > > expects.
> > > > > > > > > But
> > > > > > > > > by using the 'sources.lists' format we would be
> > > > > > > > > independent
> > > > > > > > > of
> > > > > > > > > multistrap and become more future proof.
> > > > > > > 
> > > > > > > We will also need a way to tell apt the priorities of
> > > > > > > these
> > > > > > > repos.
> > > > > > > Multistrap just adds them and apt-get installs packages
> > > > > > > according
> > > > > > > to
> > > > > > > its default behavior.
> > > > > > > That means that a package with the same name and version
> > > > > > > will
> > > > > > > get
> > > > > > > picked from a "random" repo. Overlays will need to make
> > > > > > > sure
> > > > > > > to
> > > > > > > always
> > > > > > > have a greater version or we will need apt configuration.
> > > > > > > https://wiki.debian.org/AptPreferences
> > > > > > > The big question here is how/whether multistrap will
> > > > > > > handle
> > > > > > > apt
> > > > > > > preference.
> > > > > > > 
> > > > > > > Alex already ran into that when he wanted to modify
> > > > > > > "hello".
> > > > > > > Renaming
> > > > > > > packages - like Alex suggested - is not the way to go.
> > > > > > > Because
> > > > > > > the
> > > > > > > ones
> > > > > > > we do overlay could be deps of packages we do not
> > > > > > > overlay.
> > > > > 
> > > > > The version collision could be avoided by using epoch, and if
> > > > > I
> > > > > understand it correctly - this is the preferable way to
> > > > > assign
> > > > > apt
> > > > > priority.
> > > > > 
> > > > > https://www.debian.org/doc/debian-policy/#s-f-version
> > > > 
> > > > Where did you get that epoch is the preferable way to assign
> > > > apt
> > > > priority?
> > > 
> > > Not the apt, just specific package. Hmm, I'm a bit incorrectly
> > > called
> > > my
> > > pov. :-) I see epoch as an easy way to patch existing package
> > > from
> > > upstream. You don't care about upstream one's version even if it
> > > goes
> > > forward, until your epoch is higher - your package will be
> > > installed.
> > > Of
> > > course it doesn't mean that upstream epoch couldn't be changed
> > > too,
> > > but
> > > this happens quite rare.
> > > 
> > > > 
> > > >  From you link:
> > > > 
> > > >      epoch
> > > >          This is a single (generally small) unsigned integer.
> > > > It may
> > > > be
> > > > omitted, in which case zero is assumed. If it is omitted then
> > > > the
> > > > upstream_version may not contain any colons.
> > > >          It is provided to allow mistakes in the version
> > > > numbers of
> > > > older versions of a package, and also a package’s previous
> > > > version
> > > > numbering schemes, to be left behind.
> > > > 
> > > > I see epoch just as a possibility to fix mistakes or if
> > > > upstream
> > > > changed their versioning scheme, not use it use it generally
> > > > for
> > > > specifying package priorities. Using it that way sound more
> > > > like a
> > > > hack
> > > > to me. Because what happens if debian raises the epoch of a
> > > > package?
> > > > Then you would have to raise yours again? Sounds like playing
> > > > poker.
> > > 
> > > Could you please describe the case when you have two repositories
> > > which
> > > contain package with the same name-version but different content?
> > 
> > With apt-preferences we could prefer every package from our own
> > repo
> > with the same name AFAIK, but this would make the original package
> > no
> > longer install-able beside that.
> > And that is a limitation of debian/dpkg and we can do nothing about
> > it
> > in isar. This problem it not ours to solve.
> > 
> > Ideally debian would introduce package namespaces or similar. But
> > that
> > is completely out of isars scope. On our side we could only try
> > some
> > hacks to avoid name collision, but I would rather spend time
> > solving
> > other problems in isar than that, since just choosing a different
> > name
> > is pretty easy. Multi-repo or reproducible builds are much more
> > important issues than trying to play cat and mouse with package
> > names.
> > 
> 
> I see this. But what is the practical usecase to have the same
> packages 
> in several repositories?

Where did this question comes from? It does not really make sense to
have exactly the same package in multiple different repositories. But
that is AFAIK not related to any of this.

If you have changed some existing debian package put it into your own
repo, maybe change the version of the package to contain something like
"*~isar0" similar to how ubuntu does it. Then set a apt-preference like
this:

    Package: *
    Pin: release c=isar
    Pin-Priority: 650

or similar. And apt would prefer to use packages from the isar repo
instead of upstream if there is a name collision. Even if the version
of the package in isar is lower than the one upstream.

Cheers,
Claudius


> 
> Alex
> 
> > > > Telling apt to prefer packages with specific versions or from
> > > > specific
> > > > repositories via apt-preferences sounds more like the right
> > > > tool
> > > > for
> > > > the job.
> > > > 
> > > > Cheers,
> > > > Claudius
> > > > 
> > > > > 
> > > > > Regarding renaming 'hello' there was a bit different case,
> > > > > our
> > > > > 'hello'
> > > > > has nothing common with the upstream one, moreover I've
> > > > > already
> > > > > modified
> > > > > it to use 'libhello'. So this is not the case when we patch
> > > > > the
> > > > > upstream
> > > > > package. So I decided to rename it, to avoid confusion.
> > > > > 
> > > > > In case of patching the upstream package, we could just add
> > > > > epoch
> > > > > to
> > > > > version, so our package will be always in first prio.
> > > > > 
> > > > > Also I'm not sure that when there are 2 packages with the
> > > > > same
> > > > > names
> > > > > and
> > > > > versions available in source.list - is a good practice. In
> > > > > this
> > > > > case
> > > > > you
> > > > > have no way to easily check if your rootfs contains correct
> > > > > package,
> > > > > 'dpkg -l' won't be enough.
> > > > > 
> > > > > Alex
> > > > > 
> > > > > > Thanks, yes that is a very good point.
> > > > > > 
> > > > > > I would prefer the apt preferences settings. Maybe handle
> > > > > > it
> > > > > > similar to
> > > > > > how I proposed the multi-repo support. A global variable
> > > > > > where
> > > > > > apt
> > > > > > preference file paths are appended.
> > > > > > 
> > > > > > If multistrap cannot support them with a reasonable complex
> > > > > > conversion
> > > > > > script then it might be time to say multistrap goodbye.
> > > > > > 
> > > > > > Claudius
> > > > > > 
> > > > 
> > > > --
> > > > DENX Software Engineering GmbH,      Managing Director:
> > > > Wolfgang
> > > > Denk
> > > > HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> > > > Germany
> > > > Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@d
> > > > enx.
> > > > de
> > > > 
> > > >              PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19
> > > > 9808
> > > > B153
> > > >                                Keyserver: hkp://pool.sks-
> > > > keyservers.net
> > > 
> > > 
> 
> 
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]