Re: [Discussion]: Metadata to consolidate and rebuild base-apt from distributed CI builds

[Baurzhan Ismagulov <ibr@radix50.net>]

On Mon, Mar 07, 2022 at 12:53:26PM +0530, vijai kumar wrote:
> If there is already a branch for this activity with some initial
> implementations, can you please point to it?

v2: https://groups.google.com/g/isar-users/c/65lRtw4EU_8/m/_O2hIRPBAgAJ

v3 WIP: https://github.com/ilbers/isar/tree/baseapt_v3/10


> It is for recreating a repo for a given project from some kind of
> manifest. This way we could
> avoid pushing the repos between multiple CI runners.
> 
> The current goal is to create a single repo from multiple projects. We
> might have multiple projects running parallel in different CI runners,
> the idea is to create a single repo from all those builds without the
> need to push data around. So, some kind of project manifest.
> 
> These manifests then can be used to create a single repo. Instead of
> copying over all the debs to a single location and trigger creation of
> base-apt.

Ok, so it's about creating One Canonical Base-Apt (for a project / department /
business unit / company).


> We thought about a few options,
> 1. Gather the download urls for all the packages we download. This
> could be our metadata.
> 2. Club all manifests at the end of build. (Buildchroot,
> isar-bootstrap & image rootfs) to recreate a master list of packages
> used in the build.
> 
> 2 has some disadvantages. We have to probe buildchroot after image
> build to get a complete package list. Even then it doesnot capture the
> packages that are removed.
> 1 seems like a solution, It would have to be injected as part of the
> build, like how we injected downloading debs. The advantage it brings
> is that we don't necessarily need the apt sources information to
> recreate the repo. A simple wget would do. There is also a risk of
> urls  becoming obsolete.
> 
> There could be better solutions, maybe our new way of creating
> base-apt might help in creating metadata in a cleaner way.

I agree that post-build collection has some limitations -- that was the
motivation for us to make a small step towards a more Debian-like repo
management. The patchset implements the approach #1. We use python-apt to
determine what we need upfront. The debootstrap part works in v2. Build-deps
part is about to be finished in v3. We download immediately, but updating to
output only is easy -- that is in fact one of our requirements.

We don't address package removal and recursive fetching (rebuilding the whole
base-apt exclusively from local files) in this step. Implementing
https://wiki.debian.org/HelmutGrohne/rebootstrap with Isar would be cool (and
we need at least parts of that logic for certain use cases), but we have to see
whether we need other stuff like Build-Depends support before that.


With kind regards,
Baurzhan.