Re: [PATCH 0/5] support creation of a full repo for offline/reproducible builds

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2018-02-05 11:26, Alexander Smirnov wrote:
> On 02/05/2018 10:18 AM, Jan Kiszka wrote:
>> On 2018-02-04 18:54, Cedric_Hombourger@mentor.com wrote:
>>> From: Cedric Hombourger <Cedric_Hombourger@mentor.com>
>>>
>>> The package repository created by isar using reprepro only includes
>>> packages by isar.
>>> To support offline/reproducible builds, this changeset adds a
>>> do_populate task to
>>> augment the repo with packages used during the build. The task may be
>>> used against
>>> the buildchroot and images recipes. It should be noted that isar
>>> currently assumes
>>> that the base distribution will provide both an -updates and security
>>> feed. This is
>>> certainly true for Debian but may not be the case for other
>>> distributions or when
>>> when using our own feed.
>>
>> The automatic addition of update and security feeds is more of a
>> workaround until we have multi-repo support like Claudius is working on.
>> I guess we can then drop this and just have repo lists for the different
>> Debian versions with multiple entries.
>>
>>>
>>> Some rework may be needed if the isar-apt changes get merged first.
>>> Conceptually the
>>> implementation may not change much (as far as I can tell!)
>>>
>>> Please review and let me know if any rework is required.
>>
>> There is indeed quite some overlap with what Alex and I were discussing
>> yesterday at FOSDEM. However, also looking at these patches, we need to
>> do some homework first. As you correctly stated in patch 1, there is the
>> issue that we pull package list twice at different times: first for the
>> buildchroot and then again for the image. That needs to go first so that
>> we end up with a consistent build. Also, all those duplications in logic
>> between the two chroot setup recipes are killing us.
>>
>> So I would propose the following roadmap:
>>
>> - consolidate chroot building into a common class that both buildchroot
>>    and image recipes derive from, while doing that
>>     - generate multistrap.conf (e.g. "cat <<EOF") instead of doing sed
>>       all over the place
> 
> I'd propose here to start with buildchroot/image recipes clean up. I
> mean there are lots of files in:
> 
>  - meta/recipes-devtools/buildchroot/files
>  - meta-isar/recipes-core/images/files
> 
> Until these files will be unified, it makes no sense to implement common
> class - you anyway have to keep several sets of these files per
> image/buildchroot. That's why I proposed some time ago to drop
> multistrap hooks and implement their actions inside bitbake recipe, but
> this idea wasn't found support here.

Did you start that effort back then? I don't remember if there was some
code already or just the idea.

In any case, the similarity is too high to keep on with the two
instances. Let's start this now.

> 
>> - build up a single apt cache that all chroot builders use
> 
> In other words, there should be the only one component/recipe who
> fetches the upstream packages.
> 
> There is still open question, how to fetch all the packages: in a single
> step or during the build.

I think - as long as we fetch the package list only once - we can get
away for now with fetching the actual deb files when they are needed.

> 
> I want to have an option similar to Yocto's one: -c fetchall to have
> possibility to fetch all the necessary artifacts and do not depend from
> network anymore.

Let's look into this as an add-on feature later, I would say. I you see
that our implementation is making that needlessly harder, raise your
hand, though.

> 
>> - derive the mirror repo list after the first installation from that
>>    cache which will then contain ALL required packages in the right
>>    versions
>> - describe the workflow (doc/) how to generate that mirror and how to
>>    use it in succeeding builds
>>
>> Makes sense?
>>
>> We should than clarify who will work on what.
>>
>> BTW, the very first step is sorting out all the other patches and
>> applying them to next. For that, Alex plans to first update bitbake and
>> then go through what is pending (and still merges fine).
> 
> My wish will be to keep an eye on this list and synchronize your
> activities to avoid overlaps.
> 

Right, mine as well.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux