Re: base-apt work in progress, RFC

[Henning Schild <henning.schild@siemens.com>]

Am Wed, 23 Oct 2019 22:28:24 +0200
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> On Tue, Oct 22, 2019 at 05:43:59PM +0200, Henning Schild wrote:
> > i now have a working version that changed a few things around the
> > implementation of base-apt, to make sure it will also work when some
> > rootfs clear their package cache etc.  
> 
> Ok, good to know.
> 
> 
> > So far base-apt mixes together packages from several rootfss. If
> > these rootfss had different versions of a package the latest
> > version of that package would win. See
> > meta/classes/image-cache-extension.bbclass, we are mixing the image
> > with the two buildchroots. The preferences and sources in these
> > three rootfs can lead to such mixing. When this base-apt gets used
> > for a rebuild such newer packages can leak to rootfs where they
> > were not in the first run.  
> 
> Do I understand correctly, you'd like to cover e.g. the use case
> below?
> 
> * multiconfig card1-arm64-buster uses buster with pkg1 1.0-10.
> 
> * multiconfig card2-armhf-buster uses buster with an older pkg1 1.0-9.

Isar does support preferences and different debian mirrors. And it
supports to control what is used for which rootfs. So we happily
download packages from "random" mirrors, random "components" etc. In
the end we put everything into the base-apt under the
BASE_DISTRO_CODENAME into component "main".

So two images that are based on buster, but with different preferences,
components or mirros will happily "share" packages they probably should
not.
Say you have a buster 10.0 which uses snapshot.debian.org (or a
base-apt from a few weeks ago), that is your product version 1.0. For
your devel version you use bleeding edge (upstream mirror). And maybe
the components also differ ...

The bleeding edge will win and your 1.0 will get those packages when
rebuilding from base-apt.

> This is a relatively complex policy issue. I'd suggest to start with
> the simplest case first, then add features on top of that.
> 
> 
> > I see two general directions to solve that:
> > 1. align the sources and preferences across the 4-tuple image,
> > buildchroots, sdk ... and have a base-apt for every image-4-tuple  
> 
> Could you please give an example of two such tuples? I don't see how
> that differs from the image-MACHINE-DISTRO-ARCH case.

Not that kind of tuple, sorry. Every image comes with 2 buildchroots
and an sdk ... up to 4 rootfs related to one "image".

> > 2. have a base-apt for every rootfs-PN-PV  
> 
> For the example above, would that mean the following?
> 
> base-apt/debian-buster-pkg1-1.0-10/...
> base-apt/debian-buster-pkg1-1.0-9/...
> 

Every image, and its up to three sibbling rootfss.

base-apt/image1-rootfs/
base-apt/image1-sdk-rootfs/
base-apt/image1-buildchroot-target-rootfs/
base-apt/image1-buildchroot-host-rootfs/
base-apt/image2-rootfs/
...

> > 3. if reprepro has deduplication accross distros/suites possibly
> > have all "base-apts" in one repo, and make every instance a "suite"
> >  so instead of "buster" think
> >  "isar-image-base-MACHINE-DISTRO-DISTRO_ARCH_1.0"
> > Now if they in the end are all buster and use the same file, we only
> > need to store it once and can still store differences. Same for
> > sources, where this even works cross-arch.
> > 
> > I think 1 is not a good idea, isar currently allows sources and
> > preferences to differ and it is probably a wanted feature.  
> 
> I'd suggest to start cache_base_repo without preferences and add the
> feature later. And when we add it, keep the cache_base_repo tooling
> separate from the bootstrap tooling. I think the users would like to
> use a certain codename (e.g., buster) with some explicit pinning
> (e.g., pkg1 1.0-9). That could be described statically in e.g. a json
> file, rather than resolving dynamically via preferences. After you've
> cache_base_repo-ed that, apt would get the version in base-apt,
> making preferences unnecessary.

Isar has prefs and sources knobs. It is a feature currently not
properly working with base-apt. Your rebuild might "work" but might not
really use the same packages.

We are at the state where base-apt does this "unwanted" mixing. Since i
am trying to "fix" issues in base-apt, the question is how to deal with
this one.

Agreed, prefs do not make any sense once there is only one mirror and
only one version each of the selection of packages.

> > 2 is "clear" but we have a lot of those repos, which might be hard
> > to handle, if we think people pull updates from there  
> 
> I'm not in favor of that, either.
> 
> 
> > 3 is currently me preferred version if deduplication is in place, i
> > think even without it  
> 
> Deduplication is possible with the Debian repository format. reprepro
> supports it. I think the first step should be exactly how Debian does
> it, i.e., BASE_DISTRO/dists/CODENAME, with different DISTRO_ARCH
> packages in BASE_DISTRO/pool/*/*/*.deb, without preferences.

I guess the real point with partially mirroring debian+optional other
mirrors at a certain day is that you probably should invent your own
CODENAME and must not use the debian one any longer.

And i guess that would be "arch+machine+distro+imagePN".

> 
> > I am not sure whether the solution to that problem should be part of
> > the first patch-series. It is not a new problem ...
> > In fact i will probably sent my current state without the "deep tree
> > download", which would also be something on top.  
> 
> If it is not worse than the current implementation (which I hope is
> the case) and passes the CI, I'd like to see it.

No the whole "issue" is already existing but probably should be fixed.
Or align the rootfs 4-tuple and ditch multiconfig.

Henning

> 
> With kind regards,
> Baurzhan.
>