Re: [RFC][PATCH 0/6] Isar build reproducibility

[Alexander Smirnov <asmirnov@ilbers.de>]

Hi Jan,

On 01/02/2018 07:09 PM, Jan Kiszka wrote:
> Hi Alex,
> 
> On 2018-01-02 15:57, Alexander Smirnov wrote:
>> Hello all,
>>
>> this series proposes the way how build reproducibility could be implemented
>> in Isar. General idea is to get the list of all the necessary packages for
>> build, fetch them and create local repo, that will be used for further
>> builds/
>>
>> Briefly speeking, it works like the following:
>>
>> 1. User sets the list of images that should be 'reproducible' in
>>     BASE_APT_IMAGES variable in local.conf file.
> 
> What speaks against making every image reproducible? I strongly assume
> this is the common case. There should be a way to disable that, and that
> could be local.conf controlled.

I assume, that there could be lots of overlays used for building custom 
product. So I'm not sure if it makes sense to include all the image 
recipes from all the overlays.

For example, in Yocto, there is image sato, which provides graphical UI 
and adds lots of tasks to build pipeline. In my experience I don't 
remember anybody who used it in commercial products, so for they "sato" 
reproducibility will be excessive.

> 
> Is there a technical reason the core needs to know the name of the
> reproducible image? Or can be control this on a per-build basis,
> including all images of that build?

Yes, unfortunately it's a technical reason, at least I haven't found any 
other way for now. Bitbake uses "waterfall one direction" dependencies 
processing. Roughly speaking, if you imagine all the deps as tree with 
head on the top, bitbake could derive only deps in the bottom for each 
node in the tree.

              isar-image-base
                /       \
             hello    example-raw
               \        /
              buildchroot
                  |
               base-apt


So, if your recipe, for example 'hello.bb' is building, you can't get 
within it the information who exactly added him to build pipeline, what 
is the final target/image bitbake is building etc...

That's why I need to know the top nodes of the dependency tree to get 
the list of all the packages being used.

> 
> And will all images of the same bitbake run (thinking of multiconfig...)
> share also the same base-apt repo, or is that per image?
> 
>>
>> 2. Based on the list of images above, Isar will derive all the run-time and
>>     build dependencies for these images.
>>
>> 3. Using multistrap, Isar will fetch the list of packages and create base-apt
>>     local repository.
>>
>> 4. Now buildchroot and image root filesystems are generated using base-apt.
>>
>> Some notes:
>>
>> 1. base-apt repository is mounted to buildchroot, so Isar packages are
>>     able to install necessary deps via apt-get.
> 
> This does not your provide the packages a way to feed their own build
> results into the same repo, does it? I'm referring to the issue that
> Christian described
> (https://groups.google.com/forum/#!msg/isar-users/efer3RF989o/r-zfUNNDAgAJ)
> and which I ran into as well meanwhile.

The idea of base-apt - to be a partial mirror of upstream Debian. In my 
understanding, for all the build results Isar apt should be used.

>>
>> 2. bitbake events are used to clean up buildchroot. I haven't found another
>>     way how base-apt could be unmounted. So it's mounted once before any
>>     package starts building and unmounted by bitbake event:
>>     bb.event.BuildCompleted.
> 
> What is the implication of that? At least to me (without detailed
> understanding of bitbake), this remains unclear.

No implications. Bitbake has various events that are generated at 
certain points of operations:

https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#events

These events for example are used in Yocto to generate build statistics, 
in toaster UI.

The key problem here is to implement the following model:

  - buildchroot:do_prepare (mount /proc, /dev, base-apt)
  - build custom packages
  - buildchroot:do_cleanup (unmount everything)

I could add dependency from "buildchroot:do_prepare" to any package 
being build, it's not a problem.

But I can't add dependency from "buildchroot:do_cleanup". This task 
should be executed when all the packages are built, but the list of 
packages being build is generated dynamically, while bitbake supports 
only static deps.

One way could be add "do_cleanup" to image recipe, because it has build 
dependency from all the recipes, but this will work only if you build 
image. The command:

  $ bitbake example-raw

will not call "do_cleanup" in this case.

Alex

> 
>>
>> This series works good with latest next. Any comments are welcome.
>>
>> Happy New Year 2018! :-)
> 
> Same to you.
> 
> Jan
>