Re: [PATCH 0/5] support creation of a full repo for offline/reproducible builds

[Alexander Smirnov <asmirnov@ilbers.de>]

On 02/05/2018 10:18 AM, Jan Kiszka wrote:
> On 2018-02-04 18:54, Cedric_Hombourger@mentor.com wrote:
>> From: Cedric Hombourger <Cedric_Hombourger@mentor.com>
>>
>> The package repository created by isar using reprepro only includes packages by isar.
>> To support offline/reproducible builds, this changeset adds a do_populate task to
>> augment the repo with packages used during the build. The task may be used against
>> the buildchroot and images recipes. It should be noted that isar currently assumes
>> that the base distribution will provide both an -updates and security feed. This is
>> certainly true for Debian but may not be the case for other distributions or when
>> when using our own feed.
> 
> The automatic addition of update and security feeds is more of a
> workaround until we have multi-repo support like Claudius is working on.
> I guess we can then drop this and just have repo lists for the different
> Debian versions with multiple entries.
> 
>>
>> Some rework may be needed if the isar-apt changes get merged first. Conceptually the
>> implementation may not change much (as far as I can tell!)
>>
>> Please review and let me know if any rework is required.
> 
> There is indeed quite some overlap with what Alex and I were discussing
> yesterday at FOSDEM. However, also looking at these patches, we need to
> do some homework first. As you correctly stated in patch 1, there is the
> issue that we pull package list twice at different times: first for the
> buildchroot and then again for the image. That needs to go first so that
> we end up with a consistent build. Also, all those duplications in logic
> between the two chroot setup recipes are killing us.
> 
> So I would propose the following roadmap:
> 
> - consolidate chroot building into a common class that both buildchroot
>    and image recipes derive from, while doing that
>     - generate multistrap.conf (e.g. "cat <<EOF") instead of doing sed
>       all over the place

I'd propose here to start with buildchroot/image recipes clean up. I 
mean there are lots of files in:

  - meta/recipes-devtools/buildchroot/files
  - meta-isar/recipes-core/images/files

Until these files will be unified, it makes no sense to implement common 
class - you anyway have to keep several sets of these files per 
image/buildchroot. That's why I proposed some time ago to drop 
multistrap hooks and implement their actions inside bitbake recipe, but 
this idea wasn't found support here.

> - build up a single apt cache that all chroot builders use

In other words, there should be the only one component/recipe who 
fetches the upstream packages.

There is still open question, how to fetch all the packages: in a single 
step or during the build.

I want to have an option similar to Yocto's one: -c fetchall to have 
possibility to fetch all the necessary artifacts and do not depend from 
network anymore.

> - derive the mirror repo list after the first installation from that
>    cache which will then contain ALL required packages in the right
>    versions
> - describe the workflow (doc/) how to generate that mirror and how to
>    use it in succeeding builds
> 
> Makes sense?
> 
> We should than clarify who will work on what.
> 
> BTW, the very first step is sorting out all the other patches and
> applying them to next. For that, Alex plans to first update bitbake and
> then go through what is pending (and still merges fine).

My wish will be to keep an eye on this list and synchronize your 
activities to avoid overlaps.

Alex