Re: [Discussion]: Metadata to consolidate and rebuild base-apt from distributed CI builds

[Jan Kiszka <jan.kiszka@siemens.com>]

On 24.02.22 16:42, Henning Schild wrote:
> Am Thu, 24 Feb 2022 18:50:50 +0530
> schrieb vijai kumar <vijaikumar.kanagarajan@gmail.com>:
> 
>> Hi Henning,
>>
>> On Tue, Feb 22, 2022 at 8:01 PM Henning Schild
>> <henning.schild@siemens.com> wrote:
>>>
>>> Hey Vijai,
>>>
>>> Am Tue, 22 Feb 2022 16:04:36 +0530
>>> schrieb vijai kumar <vijaikumar.kanagarajan@gmail.com>:
>>>  
>>>> Problem:
>>>> --------
>>>> We could have several CI jobs that are running in parallel in
>>>> different nodes. One might want to consolidate and build a
>>>> base-apt from the debs/deb-srcs of all these builds.  
>>>
>>> Can you go into more detail. I do not yet get the problem.  
>>
>> runner 1(Germany) -> Building de0 nano
>> runner 2(India) -> Building qemuarm
>> runner 3(US) -> Building qemuamd64
>>
>>
>> All these builds are running in different servers.
>> If we wanted to create a single base-apt from all these servers, then
>> we need to copy over their deb/debsrcs/base-apt to a common server and
>> then
>> create a consolidated repo.
> 
> But why would you want to do that? I mean i get why you would want to
> store all in the same location, but not why it should be one repo.
> Maybe to save some space on sources and arch all .. but hey there are
> ways of deduplcating on filesystem or block level.
> You are just risking a weird local "all" package not being so "all"
> after all ... false sharing.

We want to auto-build a single, "offline" capable repo from the BoM
accumulated from those builds of all possible targets. And that in a way
that does not require pushing large artifacts between the build stages,
ideally only those BoM lists.

> 
>> This involves moving around this data.
> 
> Yes, if it one central storage place. No matter if it is one "repo" or
> many "repos" in i.e. folders.
> 
>> The problem can be avoided if we have a single metadata produced by
>> all these builds which would have details of all the packages the
>> build used.
>> Basically a manifest of the build. This manifest can be later used to
>> recreate the repo which can be hosted later on for these jobs.
> 
> We have a manifest for "image content" which already is fed into
> clearing, it is a bill of materials an nothing else, it can not
> be used to rebuild.
> Even if you had all metadata you need to store sources and binaries
> somewhere reliable, whether that is central or distributed is another
> story.
> Pointers to anything on the internet (including all debian repos) will
> at some point stop working. So if "exact rebuilding" in a "far away
> future" is what you want, mirroring is what you will need.

Exactly, this mirror is supposed to be generated, and that shortly after
the individual builds succeeded (in a common pipeline stage). That can
fail as any build can fail if a referenced version picked up during
bootstrap got dropped while building an image.

> Partial mirroring based on base-apt even with sources will be shaky and
> you will find yourself digging in snapshots again. But it will work.

Yes, it works for us (you should know ;)).

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux