[Discussion]: Metadata to consolidate and rebuild base-apt from distributed CI builds

[vijai kumar <vijaikumar.kanagarajan@gmail.com>]

Problem:
--------
We could have several CI jobs that are running in parallel in different nodes.
One might want to consolidate and build a base-apt from the
debs/deb-srcs of all these builds.

What's possible:
---------------
With the current state of ISAR, the below is possible.

1. Run all the jobs in parallel in separate CI runners
2. Collect all the debs and deb-srcs from those builds and push to a
common file server.
3. Download the debs and deb-srcs and create a repo out of it in the
final CI step,
4. Upload the base-apt to the server.

This has some disadvantages, we need to move all those
data(deb/debsrcs), this increases time and cost.

What's needed:
--------------
The idea is to have a simple meta-data that can be used by repo
generation tools to recreate the repo.

Why manifest cannot be used:
----------------------------
Manifest does not serve this particular need. Below are the
shortcomings of image manifest,
1. Does not have details about removed packages(eg localepurge)
2. Manifest of buildchroot would not have details about the package
dependencies/imager installs at the time of generation(i.e.
postprocess)

Some ideas:
-----------
There were a couple of ideas,
1. To use an external script to create a manifest of the
downloads/{deb, debsrc} folder and try to download the packages using
that manifest and appropriate sourceslist in the final runner.
2. To use "apt --print-uris" + "debootstrap --keep-debootstrap-dir" to
create a metadata with complete url to the package. Later wget can be
used to download those from the web.

We are wondering if we could discuss and derive a solution for this
here in ISAR itself instead of opting for some local scripts in
downstream layers.

Thanks,
Vijai Kumar K