Re: [Discussion]: Metadata to consolidate and rebuild base-apt from distributed CI builds

[vijai kumar <vijaikumar.kanagarajan@gmail.com>]

On Fri, Mar 4, 2022 at 3:33 PM Baurzhan Ismagulov <ibr@radix50.net> wrote:
>
> On Thu, Mar 03, 2022 at 07:15:40PM +0530, vijai kumar wrote:
> > If we are in agreement then we can think about how to achieve this.
> > There are changes coming in soon, so the implementation should take
> > that into consideration.
> >
> > I am not sure if the caching part is reworked. If so having an idea on
> > the design would definitely help;
>
> Thanks Vijai for the discussion. In short, we've already started further
> base-apt improvement due to a number of reasons, e.g.:

If there is already a branch for this activity with some initial
implementations, can you please point to it?

>
> * Strict usage of base-apt for debootstrap and build-dep to ensure base-apt
>   correctness in any build.
>
> * Pluggability of debootstrap, which is necessary for multistrapping, sudo
>   removal, and maintainability.
>
> * We need to know which PN-PV is satisfiable from which location (base-apt,
>   isar-apt, bitbake) in order to use Debian Build-Depends in bitbake.
>
> python-apt provides the necessary functionality. After we have the above, more
> necessary use cases become possible. E.g., storing and reusing built packages
> in per-layer apt repos.
>
> We also want to have parallel building. For us, it comes more from the CI side,
> as we have 3 h for fast and 10 h for full testsuite on the latest inexpensive
> hardware. The first step would be to parallelize the testcases with storing of
> intermediate results in a shared location. The second step would be extending
> that to individual bitbake tasks. Maybe icecc would be good enough to cover
> either or both, we have to test.
>
> Regarding your implementation proposal, I think that could be done. However,
> I'd like to better understand the motivation first. Is it e.g. creating a
> canonical repo for a given project? That would be easier to implement on top of
> the above.

It is for recreating a repo for a given project from some kind of
manifest. This way we could
avoid pushing the repos between multiple CI runners.

The current goal is to create a single repo from multiple projects. We
might have multiple projects running parallel in different CI runners,
the idea is to create a single repo from all those builds without the
need to push data around. So, some kind of project manifest.

These manifests then can be used to create a single repo. Instead of
copying over all the debs to a single location and trigger creation of
base-apt.

>
> Regarding downloading time -- we had tested full local Debian mirrors and
> didn't see any performance improvement of CI jobs. We haven't dug deeper, maybe
> we have some parallelization killers in Isar.
>
> Regarding the central repo for remote building sites -- in my experience, it is
> very slow, our customers end up installing local replication servers.
>
> We aim at full Debian support, be it packages, repos, or images. Debian, being
> a binary server / desktop distribution and not a source-based development kit,
> has a number of inflexibilities such as sudo, versioning, rules, etc.; we would
> like to work towards more developer friendliness here. Bitbake and Yocto
> contribute much here, and we would like to find a good working solution.
>
> That is why we welcome this use case and would like to work on that after
> understanding the details. Jan told me you already had some implementations for
> this. You also mention time and costs. Could you please share the concept
> behind the work so far, and which time and costs you mean? Then we could
> proceed step by step while having the big picture in mind.

We thought about a few options,
1. Gather the download urls for all the packages we download. This
could be our metadata.
2. Club all manifests at the end of build. (Buildchroot,
isar-bootstrap & image rootfs) to recreate a master list of packages
used in the build.

2 has some disadvantages. We have to probe buildchroot after image
build to get a complete package list. Even then it doesnot capture the
packages that are removed.
1 seems like a solution, It would have to be injected as part of the
build, like how we injected downloading debs. The advantage it brings
is that we don't necessarily need the apt sources information to
recreate the repo. A simple wget would do. There is also a risk of
urls  becoming obsolete.

There could be better solutions, maybe our new way of creating
base-apt might help in creating metadata in a cleaner way.

Thanks,
Vijai Kumar K


>
> With kind regards,
> Baurzhan.
>
> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/YiHj1KTffbhLxPl5%40ilbers.de.