Re: [Discussion]: Metadata to consolidate and rebuild base-apt from distributed CI builds

[Henning Schild <henning.schild@siemens.com>]

Hey Vijai,

Am Tue, 22 Feb 2022 16:04:36 +0530
schrieb vijai kumar <vijaikumar.kanagarajan@gmail.com>:

> Problem:
> --------
> We could have several CI jobs that are running in parallel in
> different nodes. One might want to consolidate and build a base-apt
> from the debs/deb-srcs of all these builds.

Can you go into more detail. I do not yet get the problem.

It seems like you want to save compute time by sharing pre-built
artifacts via some common storage. The sstate can do that very well, we
are using shared folders for on-prem runners, s3 for AWS and sstate
mirrors for population of "new empty runners" and "partial result
delivery" of failed jobs and to sync on-prem with s3.

isar is a tool to build images, not distros or repos or packages. While
it can do all of that using it for such things can get tricky and isar
was not designed for such cases. Meaning "base-apt" is not meant to be
your cache to build many images from ... it is meant to be the cache
for exactly one ... and sharing can cause problems.

sstate would detect false sharing, say a package recipe for some reason
uses a machine-conf variable. multiconfig or base-apt sharing would
make you run into that bug, while sstate would likely not.

So if it is about build time i suggest you have a look at sstate and the
not yet upstreamed python helper scripts for sharing/eviction i can
point you to in case you do not find it yourself.

Henning

> What's possible:
> ---------------
> With the current state of ISAR, the below is possible.
> 
> 1. Run all the jobs in parallel in separate CI runners
> 2. Collect all the debs and deb-srcs from those builds and push to a
> common file server.
> 3. Download the debs and deb-srcs and create a repo out of it in the
> final CI step,
> 4. Upload the base-apt to the server.
> 
> This has some disadvantages, we need to move all those
> data(deb/debsrcs), this increases time and cost.
> 
> What's needed:
> --------------
> The idea is to have a simple meta-data that can be used by repo
> generation tools to recreate the repo.
> 
> Why manifest cannot be used:
> ----------------------------
> Manifest does not serve this particular need. Below are the
> shortcomings of image manifest,
> 1. Does not have details about removed packages(eg localepurge)
> 2. Manifest of buildchroot would not have details about the package
> dependencies/imager installs at the time of generation(i.e.
> postprocess)
> 
> Some ideas:
> -----------
> There were a couple of ideas,
> 1. To use an external script to create a manifest of the
> downloads/{deb, debsrc} folder and try to download the packages using
> that manifest and appropriate sourceslist in the final runner.
> 2. To use "apt --print-uris" + "debootstrap --keep-debootstrap-dir" to
> create a metadata with complete url to the package. Later wget can be
> used to download those from the web.
> 
> We are wondering if we could discuss and derive a solution for this
> here in ISAR itself instead of opting for some local scripts in
> downstream layers.
> 
> Thanks,
> Vijai Kumar K