Re: [Discussion]: Metadata to consolidate and rebuild base-apt from distributed CI builds

[vijai kumar <vijaikumar.kanagarajan@gmail.com>]

On Fri, Feb 25, 2022 at 10:57 PM Jan Kiszka <jan.kiszka@siemens.com> wrote:
>
> On 24.02.22 16:42, Henning Schild wrote:
> > Am Thu, 24 Feb 2022 18:50:50 +0530
> > schrieb vijai kumar <vijaikumar.kanagarajan@gmail.com>:
> >
> >> Hi Henning,
> >>
> >> On Tue, Feb 22, 2022 at 8:01 PM Henning Schild
> >> <henning.schild@siemens.com> wrote:
> >>>
> >>> Hey Vijai,
> >>>
> >>> Am Tue, 22 Feb 2022 16:04:36 +0530
> >>> schrieb vijai kumar <vijaikumar.kanagarajan@gmail.com>:
> >>>
> >>>> Problem:
> >>>> --------
> >>>> We could have several CI jobs that are running in parallel in
> >>>> different nodes. One might want to consolidate and build a
> >>>> base-apt from the debs/deb-srcs of all these builds.
> >>>
> >>> Can you go into more detail. I do not yet get the problem.
> >>
> >> runner 1(Germany) -> Building de0 nano
> >> runner 2(India) -> Building qemuarm
> >> runner 3(US) -> Building qemuamd64
> >>
> >>
> >> All these builds are running in different servers.
> >> If we wanted to create a single base-apt from all these servers, then
> >> we need to copy over their deb/debsrcs/base-apt to a common server and
> >> then
> >> create a consolidated repo.
> >
> > But why would you want to do that? I mean i get why you would want to
> > store all in the same location, but not why it should be one repo.
> > Maybe to save some space on sources and arch all .. but hey there are
> > ways of deduplcating on filesystem or block level.
> > You are just risking a weird local "all" package not being so "all"
> > after all ... false sharing.
>
> We want to auto-build a single, "offline" capable repo from the BoM
> accumulated from those builds of all possible targets. And that in a way
> that does not require pushing large artifacts between the build stages,
> ideally only those BoM lists.


If we are in agreement then we can think about how to achieve this.
There are changes coming in soon, so the implementation should take
that into consideration.

I am not sure if the caching part is reworked. If so having an idea on
the design would definitely help;

Maybe ISAR maintainers can clarify on this.

Thanks,
Vijai Kumar K

>
> >
> >> This involves moving around this data.
> >
> > Yes, if it one central storage place. No matter if it is one "repo" or
> > many "repos" in i.e. folders.
> >
> >> The problem can be avoided if we have a single metadata produced by
> >> all these builds which would have details of all the packages the
> >> build used.
> >> Basically a manifest of the build. This manifest can be later used to
> >> recreate the repo which can be hosted later on for these jobs.
> >
> > We have a manifest for "image content" which already is fed into
> > clearing, it is a bill of materials an nothing else, it can not
> > be used to rebuild.
> > Even if you had all metadata you need to store sources and binaries
> > somewhere reliable, whether that is central or distributed is another
> > story.
> > Pointers to anything on the internet (including all debian repos) will
> > at some point stop working. So if "exact rebuilding" in a "far away
> > future" is what you want, mirroring is what you will need.
>
> Exactly, this mirror is supposed to be generated, and that shortly after
> the individual builds succeeded (in a common pipeline stage). That can
> fail as any build can fail if a referenced version picked up during
> bootstrap got dropped while building an image.
>
> > Partial mirroring based on base-apt even with sources will be shaky and
> > you will find yourself digging in snapshots again. But it will work.
>
> Yes, it works for us (you should know ;)).
>
> Jan
>
> --
> Siemens AG, Technology
> Competence Center Embedded Linux