base-apt work in progress, RFC

base-apt work in progress, RFC

[Henning Schild]

Hi all,

i now have a working version that changed a few things around the
implementation of base-apt, to make sure it will also work when some
rootfs clear their package cache etc.

major changes:
 - do not rely on rootfs package cache anymore
 - support .dsc source packages
 - include full debootstrap via a common mechanism
 - full download caching support in DL_DIR, .deb + .dsc + tar
 - base-apt building now is the first task in the build chain, not the
   last
 - many cleanups


planned changes on top:
 - deep tree download
  - sibbling packages
  - source packages
  - build deps
  - while more goto -3

But now i hit a major issue that i do not know how to best deal with,
it is connected to my best friend feature ... multiconfig. But in fact
it could also show in single config builds, because they have up to 4
rootfs as well.

So far base-apt mixes together packages from several rootfss. If these
rootfss had different versions of a package the latest version of that
package would win. See meta/classes/image-cache-extension.bbclass, we
are mixing the image with the two buildchroots.
The preferences and sources in these three rootfs can lead to such
mixing. When this base-apt gets used for a rebuild such newer packages
can leak to rootfs where they were not in the first run.

I see two general directions to solve that:
1. align the sources and preferences across the 4-tuple image,
buildchroots, sdk ... and have a base-apt for every image-4-tuple

2. have a base-apt for every rootfs-PN-PV

3. if reprepro has deduplication accross distros/suites possibly have
all "base-apts" in one repo, and make every instance a "suite"
 so instead of "buster" think
 "isar-image-base-MACHINE-DISTRO-DISTRO_ARCH_1.0"
Now if they in the end are all buster and use the same file, we only
need to store it once and can still store differences. Same for
sources, where this even works cross-arch.

I think 1 is not a good idea, isar currently allows sources and
preferences to differ and it is probably a wanted feature.

2 is "clear" but we have a lot of those repos, which might be hard to
handle, if we think people pull updates from there

3 is currently me preferred version if deduplication is in place, i
think even without it

I am not sure whether the solution to that problem should be part of
the first patch-series. It is not a new problem ...
In fact i will probably sent my current state without the "deep tree
download", which would also be something on top.

Henning

Re: base-apt work in progress, RFC

[Baurzhan Ismagulov]

On Tue, Oct 22, 2019 at 05:43:59PM +0200, Henning Schild wrote:
> i now have a working version that changed a few things around the
> implementation of base-apt, to make sure it will also work when some
> rootfs clear their package cache etc.

Ok, good to know.


> So far base-apt mixes together packages from several rootfss. If these
> rootfss had different versions of a package the latest version of that
> package would win. See meta/classes/image-cache-extension.bbclass, we
> are mixing the image with the two buildchroots.
> The preferences and sources in these three rootfs can lead to such
> mixing. When this base-apt gets used for a rebuild such newer packages
> can leak to rootfs where they were not in the first run.

Do I understand correctly, you'd like to cover e.g. the use case below?

* multiconfig card1-arm64-buster uses buster with pkg1 1.0-10.

* multiconfig card2-armhf-buster uses buster with an older pkg1 1.0-9.

This is a relatively complex policy issue. I'd suggest to start with the
simplest case first, then add features on top of that.


> I see two general directions to solve that:
> 1. align the sources and preferences across the 4-tuple image,
> buildchroots, sdk ... and have a base-apt for every image-4-tuple

Could you please give an example of two such tuples? I don't see how that
differs from the image-MACHINE-DISTRO-ARCH case.


> 2. have a base-apt for every rootfs-PN-PV

For the example above, would that mean the following?

base-apt/debian-buster-pkg1-1.0-10/...
base-apt/debian-buster-pkg1-1.0-9/...


> 3. if reprepro has deduplication accross distros/suites possibly have
> all "base-apts" in one repo, and make every instance a "suite"
>  so instead of "buster" think
>  "isar-image-base-MACHINE-DISTRO-DISTRO_ARCH_1.0"
> Now if they in the end are all buster and use the same file, we only
> need to store it once and can still store differences. Same for
> sources, where this even works cross-arch.
> 
> I think 1 is not a good idea, isar currently allows sources and
> preferences to differ and it is probably a wanted feature.

I'd suggest to start cache_base_repo without preferences and add the feature
later. And when we add it, keep the cache_base_repo tooling separate from the
bootstrap tooling. I think the users would like to use a certain codename
(e.g., buster) with some explicit pinning (e.g., pkg1 1.0-9). That could be
described statically in e.g. a json file, rather than resolving dynamically via
preferences. After you've cache_base_repo-ed that, apt would get the version in
base-apt, making preferences unnecessary.


> 2 is "clear" but we have a lot of those repos, which might be hard to
> handle, if we think people pull updates from there

I'm not in favor of that, either.


> 3 is currently me preferred version if deduplication is in place, i
> think even without it

Deduplication is possible with the Debian repository format. reprepro supports
it. I think the first step should be exactly how Debian does it, i.e.,
BASE_DISTRO/dists/CODENAME, with different DISTRO_ARCH packages in
BASE_DISTRO/pool/*/*/*.deb, without preferences.


> I am not sure whether the solution to that problem should be part of
> the first patch-series. It is not a new problem ...
> In fact i will probably sent my current state without the "deep tree
> download", which would also be something on top.

If it is not worse than the current implementation (which I hope is the case)
and passes the CI, I'd like to see it.


With kind regards,
Baurzhan.

Re: base-apt work in progress, RFC

[Henning Schild]

Am Wed, 23 Oct 2019 22:28:24 +0200
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> On Tue, Oct 22, 2019 at 05:43:59PM +0200, Henning Schild wrote:
> > i now have a working version that changed a few things around the
> > implementation of base-apt, to make sure it will also work when some
> > rootfs clear their package cache etc.  
> 
> Ok, good to know.
> 
> 
> > So far base-apt mixes together packages from several rootfss. If
> > these rootfss had different versions of a package the latest
> > version of that package would win. See
> > meta/classes/image-cache-extension.bbclass, we are mixing the image
> > with the two buildchroots. The preferences and sources in these
> > three rootfs can lead to such mixing. When this base-apt gets used
> > for a rebuild such newer packages can leak to rootfs where they
> > were not in the first run.  
> 
> Do I understand correctly, you'd like to cover e.g. the use case
> below?
> 
> * multiconfig card1-arm64-buster uses buster with pkg1 1.0-10.
> 
> * multiconfig card2-armhf-buster uses buster with an older pkg1 1.0-9.

Isar does support preferences and different debian mirrors. And it
supports to control what is used for which rootfs. So we happily
download packages from "random" mirrors, random "components" etc. In
the end we put everything into the base-apt under the
BASE_DISTRO_CODENAME into component "main".

So two images that are based on buster, but with different preferences,
components or mirros will happily "share" packages they probably should
not.
Say you have a buster 10.0 which uses snapshot.debian.org (or a
base-apt from a few weeks ago), that is your product version 1.0. For
your devel version you use bleeding edge (upstream mirror). And maybe
the components also differ ...

The bleeding edge will win and your 1.0 will get those packages when
rebuilding from base-apt.

> This is a relatively complex policy issue. I'd suggest to start with
> the simplest case first, then add features on top of that.
> 
> 
> > I see two general directions to solve that:
> > 1. align the sources and preferences across the 4-tuple image,
> > buildchroots, sdk ... and have a base-apt for every image-4-tuple  
> 
> Could you please give an example of two such tuples? I don't see how
> that differs from the image-MACHINE-DISTRO-ARCH case.

Not that kind of tuple, sorry. Every image comes with 2 buildchroots
and an sdk ... up to 4 rootfs related to one "image".

> > 2. have a base-apt for every rootfs-PN-PV  
> 
> For the example above, would that mean the following?
> 
> base-apt/debian-buster-pkg1-1.0-10/...
> base-apt/debian-buster-pkg1-1.0-9/...
> 

Every image, and its up to three sibbling rootfss.

base-apt/image1-rootfs/
base-apt/image1-sdk-rootfs/
base-apt/image1-buildchroot-target-rootfs/
base-apt/image1-buildchroot-host-rootfs/
base-apt/image2-rootfs/
...

> > 3. if reprepro has deduplication accross distros/suites possibly
> > have all "base-apts" in one repo, and make every instance a "suite"
> >  so instead of "buster" think
> >  "isar-image-base-MACHINE-DISTRO-DISTRO_ARCH_1.0"
> > Now if they in the end are all buster and use the same file, we only
> > need to store it once and can still store differences. Same for
> > sources, where this even works cross-arch.
> > 
> > I think 1 is not a good idea, isar currently allows sources and
> > preferences to differ and it is probably a wanted feature.  
> 
> I'd suggest to start cache_base_repo without preferences and add the
> feature later. And when we add it, keep the cache_base_repo tooling
> separate from the bootstrap tooling. I think the users would like to
> use a certain codename (e.g., buster) with some explicit pinning
> (e.g., pkg1 1.0-9). That could be described statically in e.g. a json
> file, rather than resolving dynamically via preferences. After you've
> cache_base_repo-ed that, apt would get the version in base-apt,
> making preferences unnecessary.

Isar has prefs and sources knobs. It is a feature currently not
properly working with base-apt. Your rebuild might "work" but might not
really use the same packages.

We are at the state where base-apt does this "unwanted" mixing. Since i
am trying to "fix" issues in base-apt, the question is how to deal with
this one.

Agreed, prefs do not make any sense once there is only one mirror and
only one version each of the selection of packages.

> > 2 is "clear" but we have a lot of those repos, which might be hard
> > to handle, if we think people pull updates from there  
> 
> I'm not in favor of that, either.
> 
> 
> > 3 is currently me preferred version if deduplication is in place, i
> > think even without it  
> 
> Deduplication is possible with the Debian repository format. reprepro
> supports it. I think the first step should be exactly how Debian does
> it, i.e., BASE_DISTRO/dists/CODENAME, with different DISTRO_ARCH
> packages in BASE_DISTRO/pool/*/*/*.deb, without preferences.

I guess the real point with partially mirroring debian+optional other
mirrors at a certain day is that you probably should invent your own
CODENAME and must not use the debian one any longer.

And i guess that would be "arch+machine+distro+imagePN".

> 
> > I am not sure whether the solution to that problem should be part of
> > the first patch-series. It is not a new problem ...
> > In fact i will probably sent my current state without the "deep tree
> > download", which would also be something on top.  
> 
> If it is not worse than the current implementation (which I hope is
> the case) and passes the CI, I'd like to see it.

No the whole "issue" is already existing but probably should be fixed.
Or align the rootfs 4-tuple and ditch multiconfig.

Henning

> 
> With kind regards,
> Baurzhan.
>