Re: base-apt work in progress, RFC

[Baurzhan Ismagulov <ibr@radix50.net>]

On Tue, Oct 22, 2019 at 05:43:59PM +0200, Henning Schild wrote:
> i now have a working version that changed a few things around the
> implementation of base-apt, to make sure it will also work when some
> rootfs clear their package cache etc.

Ok, good to know.


> So far base-apt mixes together packages from several rootfss. If these
> rootfss had different versions of a package the latest version of that
> package would win. See meta/classes/image-cache-extension.bbclass, we
> are mixing the image with the two buildchroots.
> The preferences and sources in these three rootfs can lead to such
> mixing. When this base-apt gets used for a rebuild such newer packages
> can leak to rootfs where they were not in the first run.

Do I understand correctly, you'd like to cover e.g. the use case below?

* multiconfig card1-arm64-buster uses buster with pkg1 1.0-10.

* multiconfig card2-armhf-buster uses buster with an older pkg1 1.0-9.

This is a relatively complex policy issue. I'd suggest to start with the
simplest case first, then add features on top of that.


> I see two general directions to solve that:
> 1. align the sources and preferences across the 4-tuple image,
> buildchroots, sdk ... and have a base-apt for every image-4-tuple

Could you please give an example of two such tuples? I don't see how that
differs from the image-MACHINE-DISTRO-ARCH case.


> 2. have a base-apt for every rootfs-PN-PV

For the example above, would that mean the following?

base-apt/debian-buster-pkg1-1.0-10/...
base-apt/debian-buster-pkg1-1.0-9/...


> 3. if reprepro has deduplication accross distros/suites possibly have
> all "base-apts" in one repo, and make every instance a "suite"
>  so instead of "buster" think
>  "isar-image-base-MACHINE-DISTRO-DISTRO_ARCH_1.0"
> Now if they in the end are all buster and use the same file, we only
> need to store it once and can still store differences. Same for
> sources, where this even works cross-arch.
> 
> I think 1 is not a good idea, isar currently allows sources and
> preferences to differ and it is probably a wanted feature.

I'd suggest to start cache_base_repo without preferences and add the feature
later. And when we add it, keep the cache_base_repo tooling separate from the
bootstrap tooling. I think the users would like to use a certain codename
(e.g., buster) with some explicit pinning (e.g., pkg1 1.0-9). That could be
described statically in e.g. a json file, rather than resolving dynamically via
preferences. After you've cache_base_repo-ed that, apt would get the version in
base-apt, making preferences unnecessary.


> 2 is "clear" but we have a lot of those repos, which might be hard to
> handle, if we think people pull updates from there

I'm not in favor of that, either.


> 3 is currently me preferred version if deduplication is in place, i
> think even without it

Deduplication is possible with the Debian repository format. reprepro supports
it. I think the first step should be exactly how Debian does it, i.e.,
BASE_DISTRO/dists/CODENAME, with different DISTRO_ARCH packages in
BASE_DISTRO/pool/*/*/*.deb, without preferences.


> I am not sure whether the solution to that problem should be part of
> the first patch-series. It is not a new problem ...
> In fact i will probably sent my current state without the "deep tree
> download", which would also be something on top.

If it is not worse than the current implementation (which I hope is the case)
and passes the CI, I'd like to see it.


With kind regards,
Baurzhan.