Re: [Discussion]: Metadata to consolidate and rebuild base-apt from distributed CI builds

[vijai kumar <vijaikumar.kanagarajan@gmail.com>]

Hi Henning,

On Tue, Feb 22, 2022 at 8:01 PM Henning Schild
<henning.schild@siemens.com> wrote:
>
> Hey Vijai,
>
> Am Tue, 22 Feb 2022 16:04:36 +0530
> schrieb vijai kumar <vijaikumar.kanagarajan@gmail.com>:
>
> > Problem:
> > --------
> > We could have several CI jobs that are running in parallel in
> > different nodes. One might want to consolidate and build a base-apt
> > from the debs/deb-srcs of all these builds.
>
> Can you go into more detail. I do not yet get the problem.

runner 1(Germany) -> Building de0 nano
runner 2(India) -> Building qemuarm
runner 3(US) -> Building qemuamd64


All these builds are running in different servers.
If we wanted to create a single base-apt from all these servers, then
we need to copy over their deb/debsrcs/base-apt to a common server and
then
create a consolidated repo.

This involves moving around this data.

The problem can be avoided if we have a single metadata produced by
all these builds which would have details of all the packages the
build used.
Basically a manifest of the build. This manifest can be later used to
recreate the repo which can be hosted later on for these jobs.

Having metadata and recreating repo is one way. There might be other
ways as well.

That is where we thought about the --print-uris option of apt. It
basically gives you the complete URL to the package which we can
download using wget.
A manifest containing all the packages ever used by the build with its
complete url. It could easily be used for several purposes, like as
clearing input,
repo regeneration etc.

I don't think sstate can help here. I might be wrong though.

Thanks,
Vijai Kumar K

>
> It seems like you want to save compute time by sharing pre-built
> artifacts via some common storage. The sstate can do that very well, we
> are using shared folders for on-prem runners, s3 for AWS and sstate
> mirrors for population of "new empty runners" and "partial result
> delivery" of failed jobs and to sync on-prem with s3.
>
> isar is a tool to build images, not distros or repos or packages. While
> it can do all of that using it for such things can get tricky and isar
> was not designed for such cases. Meaning "base-apt" is not meant to be
> your cache to build many images from ... it is meant to be the cache
> for exactly one ... and sharing can cause problems.
>
> sstate would detect false sharing, say a package recipe for some reason
> uses a machine-conf variable. multiconfig or base-apt sharing would
> make you run into that bug, while sstate would likely not.
>
> So if it is about build time i suggest you have a look at sstate and the
> not yet upstreamed python helper scripts for sharing/eviction i can
> point you to in case you do not find it yourself.
>
> Henning
>
> > What's possible:
> > ---------------
> > With the current state of ISAR, the below is possible.
> >
> > 1. Run all the jobs in parallel in separate CI runners
> > 2. Collect all the debs and deb-srcs from those builds and push to a
> > common file server.
> > 3. Download the debs and deb-srcs and create a repo out of it in the
> > final CI step,
> > 4. Upload the base-apt to the server.
> >
> > This has some disadvantages, we need to move all those
> > data(deb/debsrcs), this increases time and cost.
> >
> > What's needed:
> > --------------
> > The idea is to have a simple meta-data that can be used by repo
> > generation tools to recreate the repo.
> >
> > Why manifest cannot be used:
> > ----------------------------
> > Manifest does not serve this particular need. Below are the
> > shortcomings of image manifest,
> > 1. Does not have details about removed packages(eg localepurge)
> > 2. Manifest of buildchroot would not have details about the package
> > dependencies/imager installs at the time of generation(i.e.
> > postprocess)
> >
> > Some ideas:
> > -----------
> > There were a couple of ideas,
> > 1. To use an external script to create a manifest of the
> > downloads/{deb, debsrc} folder and try to download the packages using
> > that manifest and appropriate sourceslist in the final runner.
> > 2. To use "apt --print-uris" + "debootstrap --keep-debootstrap-dir" to
> > create a metadata with complete url to the package. Later wget can be
> > used to download those from the web.
> >
> > We are wondering if we could discuss and derive a solution for this
> > here in ISAR itself instead of opting for some local scripts in
> > downstream layers.
> >
> > Thanks,
> > Vijai Kumar K
>
> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20220222153136.08432cb3%40md1za8fc.ad001.siemens.net.