Re: [PATCH 0/5] support creation of a full repo for offline/reproducible builds

[Benedikt Niedermayr <benbrenson89@googlemail.com>]

Am 05.02.2018 um 08:18 schrieb Jan Kiszka:
> On 2018-02-04 18:54, Cedric_Hombourger@mentor.com wrote:
>> From: Cedric Hombourger <Cedric_Hombourger@mentor.com>
>>
>> The package repository created by isar using reprepro only includes packages by isar.
>> To support offline/reproducible builds, this changeset adds a do_populate task to
>> augment the repo with packages used during the build. The task may be used against
>> the buildchroot and images recipes. It should be noted that isar currently assumes
>> that the base distribution will provide both an -updates and security feed. This is
>> certainly true for Debian but may not be the case for other distributions or when
>> when using our own feed.
> The automatic addition of update and security feeds is more of a
> workaround until we have multi-repo support like Claudius is working on.
> I guess we can then drop this and just have repo lists for the different
> Debian versions with multiple entries.
>
>> Some rework may be needed if the isar-apt changes get merged first. Conceptually the
>> implementation may not change much (as far as I can tell!)
>>
>> Please review and let me know if any rework is required.
> There is indeed quite some overlap with what Alex and I were discussing
> yesterday at FOSDEM. However, also looking at these patches, we need to
> do some homework first. As you correctly stated in patch 1, there is the
> issue that we pull package list twice at different times: first for the
> buildchroot and then again for the image. That needs to go first so that
> we end up with a consistent build. Also, all those duplications in logic
> between the two chroot setup recipes are killing us.
>
> So I would propose the following roadmap:
>
> - consolidate chroot building into a common class that both buildchroot
>    and image recipes derive from, while doing that
>     - generate multistrap.conf (e.g. "cat <<EOF") instead of doing sed
>       all over the place
> - build up a single apt cache that all chroot builders use
> - derive the mirror repo list after the first installation from that
>    cache which will then contain ALL required packages in the right
>    versions
> - describe the workflow (doc/) how to generate that mirror and how to
>    use it in succeeding builds
>
> Makes sense?
>
> We should than clarify who will work on what.
>
> BTW, the very first step is sorting out all the other patches and
> applying them to next. For that, Alex plans to first update bitbake and
> then go through what is pending (and still merges fine).
>
> Jan
>

Hi,

> - consolidate chroot building into a common class that both buildchroot
>    and image recipes derive from, while doing that
Yes, sounds good.

> - generate multistrap.conf (e.g. "cat <<EOF") instead of doing sed
>       all over the place
Maybe creating a new task like do_multistrap_conf() written in Python 
may help here. Python could import the "configparser" module

for creating the config file? AFAIK the configparser module is included 
in standart python installation, so no further host system dependencies.

> - build up a single apt cache that all chroot builders use
> - describe the workflow (doc/) how to generate that mirror and how to
>    use it in succeeding builds

So we will separate building the apt cache from the Isar build process 
itself? Means, the user will setup the Mirror manually?

> - derive the mirror repo list after the first installation from that
>    cache which will then contain ALL required packages in the right
>    versions
Can you explain that in more detail?


Benni