Re: [RFC][PATCH 0/6] Isar build reproducibility

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2018-01-02 17:58, Alexander Smirnov wrote:
> Hi Jan,
> 
> On 01/02/2018 07:09 PM, Jan Kiszka wrote:
>> Hi Alex,
>>
>> On 2018-01-02 15:57, Alexander Smirnov wrote:
>>> Hello all,
>>>
>>> this series proposes the way how build reproducibility could be
>>> implemented
>>> in Isar. General idea is to get the list of all the necessary
>>> packages for
>>> build, fetch them and create local repo, that will be used for further
>>> builds/
>>>
>>> Briefly speeking, it works like the following:
>>>
>>> 1. User sets the list of images that should be 'reproducible' in
>>>     BASE_APT_IMAGES variable in local.conf file.
>>
>> What speaks against making every image reproducible? I strongly assume
>> this is the common case. There should be a way to disable that, and that
>> could be local.conf controlled.
> 
> I assume, that there could be lots of overlays used for building custom
> product. So I'm not sure if it makes sense to include all the image
> recipes from all the overlays.
> 
> For example, in Yocto, there is image sato, which provides graphical UI
> and adds lots of tasks to build pipeline. In my experience I don't
> remember anybody who used it in commercial products, so for they "sato"
> reproducibility will be excessive.

What are the costs again? Please specify, at latest when you provide
some (user) documentation for this new feature.

> 
>>
>> Is there a technical reason the core needs to know the name of the
>> reproducible image? Or can be control this on a per-build basis,
>> including all images of that build?
> 
> Yes, unfortunately it's a technical reason, at least I haven't found any
> other way for now. Bitbake uses "waterfall one direction" dependencies
> processing. Roughly speaking, if you imagine all the deps as tree with
> head on the top, bitbake could derive only deps in the bottom for each
> node in the tree.
> 
>              isar-image-base
>                /       \
>             hello    example-raw
>               \        /
>              buildchroot
>                  |
>               base-apt
> 
> 
> So, if your recipe, for example 'hello.bb' is building, you can't get
> within it the information who exactly added him to build pipeline, what
> is the final target/image bitbake is building etc...
> 
> That's why I need to know the top nodes of the dependency tree to get
> the list of all the packages being used.

The top nodes are what is provided to bitbake as targets, no? Can't we
access that information?

> 
>>
>> And will all images of the same bitbake run (thinking of multiconfig...)
>> share also the same base-apt repo, or is that per image?
>>
>>>
>>> 2. Based on the list of images above, Isar will derive all the
>>> run-time and
>>>     build dependencies for these images.
>>>
>>> 3. Using multistrap, Isar will fetch the list of packages and create
>>> base-apt
>>>     local repository.
>>>
>>> 4. Now buildchroot and image root filesystems are generated using
>>> base-apt.
>>>
>>> Some notes:
>>>
>>> 1. base-apt repository is mounted to buildchroot, so Isar packages are
>>>     able to install necessary deps via apt-get.
>>
>> This does not your provide the packages a way to feed their own build
>> results into the same repo, does it? I'm referring to the issue that
>> Christian described
>> (https://groups.google.com/forum/#!msg/isar-users/efer3RF989o/r-zfUNNDAgAJ)
>>
>> and which I ran into as well meanwhile.
> 
> The idea of base-apt - to be a partial mirror of upstream Debian. In my
> understanding, for all the build results Isar apt should be used.

So, recipes would then deploy their debs into base-apt, making them
immediately available to dependent recipes?

> 
>>>
>>> 2. bitbake events are used to clean up buildchroot. I haven't found
>>> another
>>>     way how base-apt could be unmounted. So it's mounted once before any
>>>     package starts building and unmounted by bitbake event:
>>>     bb.event.BuildCompleted.
>>
>> What is the implication of that? At least to me (without detailed
>> understanding of bitbake), this remains unclear.
> 
> No implications. Bitbake has various events that are generated at
> certain points of operations:
> 
> https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#events
> 
> 
> These events for example are used in Yocto to generate build statistics,
> in toaster UI.
> 
> The key problem here is to implement the following model:
> 
>  - buildchroot:do_prepare (mount /proc, /dev, base-apt)
>  - build custom packages
>  - buildchroot:do_cleanup (unmount everything)
> 
> I could add dependency from "buildchroot:do_prepare" to any package
> being build, it's not a problem.
> 
> But I can't add dependency from "buildchroot:do_cleanup". This task
> should be executed when all the packages are built, but the list of
> packages being build is generated dynamically, while bitbake supports
> only static deps.
> 
> One way could be add "do_cleanup" to image recipe, because it has build
> dependency from all the recipes, but this will work only if you build
> image. The command:
> 
>  $ bitbake example-raw
> 
> will not call "do_cleanup" in this case.

Is there no "bitbake is exiting" event that you could hook up to? Then
the cleanups would happen at the very end of everything.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux