Re: [RFC][PATCH 0/6] Isar build reproducibility

[Alexander Smirnov <asmirnov@ilbers.de>]

Hi all,

On 01/03/2018 04:49 PM, Henning Schild wrote:
> Am Tue, 2 Jan 2018 17:57:38 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Hello all,
>>
>> this series proposes the way how build reproducibility could be
>> implemented in Isar. General idea is to get the list of all the
>> necessary packages for build, fetch them and create local repo, that
>> will be used for further builds/
>>
>> Briefly speeking, it works like the following:
>>
>> 1. User sets the list of images that should be 'reproducible' in
>>     BASE_APT_IMAGES variable in local.conf file.
> 
> I am with Jan here, i would prefer an all or nothing approach.
> 

Still thinking if it's possible to implement this with bitbake. The key 
problem is that Isar doesn't have static list of Debian dependencies, 
i.e. it needs to fetch source code and parse 'debian/conrtol' to get 
list of required packages to install.

So some kind of the following chain should be implemented:

1. Identify image recipes.
2. Get list of packages in IMAGE_INSTALL for all images.
3. Make do_fetch/do_unpack and parse 'debian/control' for packages in 
the list above.

In this case base-apt recipe should depend from *all* the available images.

Anonymous tasks can't help to derive dependencies, because package's 
deps could be derived after 'do_unpack' only. In theory, we could define 
some bitbake variable like: DEBIAN_DEPENDENCY that contains the same 
data as in 'debian/control'. Moreover it could make sense to generate 
'debian/control' on the fly, like Henning does it for 'example-raw'.

Having variable DEBIAN_DEPENDENCY makes possible to get list of packages 
that are possibly used in whole current Isar tree by using anonymous 
task in bbclass.

Opinions? :-)

>> 2. Based on the list of images above, Isar will derive all the
>> run-time and build dependencies for these images.
>>
>> 3. Using multistrap, Isar will fetch the list of packages and create
>> base-apt local repository.
>>
>> 4. Now buildchroot and image root filesystems are generated using
>> base-apt.
> 
> Patch5 makes base-apt and Isar the only repos for a rootfs/buildchroot.
> How do images, not using BASE_APT, get packages that are not cached?
> 

In my opinion, in case of base-apt, image should not use upstream 
anymore, otherwise build reproducibility can't be guaranteed. apt will 
try to fetch the freshest package.

So if new package appears in the build, the base-apt should be 
updated/regenerated.

Alex

>> Some notes:
>>
>> 1. base-apt repository is mounted to buildchroot, so Isar packages are
>>     able to install necessary deps via apt-get.
>>
>> 2. bitbake events are used to clean up buildchroot. I haven't found
>> another way how base-apt could be unmounted. So it's mounted once
>> before any package starts building and unmounted by bitbake event:
>>     bb.event.BuildCompleted.
>>
>> This series works good with latest next. Any comments are welcome.
>>
>> Happy New Year 2018! :-)
> 
> Same from me!
> 
> Henning
> 
>> With best regards,
>> Alex
>>
>> Alexander Smirnov (6):
>>    base-apt: Introduce fetching upstream apt
>>    base-apt: Add to pipeline
>>    buildchroot: Switch to base-apt
>>    buildchroot: Add mount/umount for 'base-apt'
>>    image: Switch to base-apt
>>    base-apt: Add possibility to reuse
>>
>>   meta-isar/conf/local.conf.sample                   |  8 ++
>>   .../recipes-core/images/files/multistrap.conf.in   | 18 +---
>>   meta-isar/recipes-core/images/isar-image-base.bb   |  5 +-
>>   meta/classes/dpkg-base.bbclass                     | 16 +++-
>>   meta/classes/image.bbclass                         | 13 ++-
>>   meta/classes/isar-events.bbclass                   | 21 +++++
>>   meta/conf/isar-bitbake.conf                        |  2 +
>>   meta/recipes-devtools/base-apt/base-apt.bb         | 97
>> ++++++++++++++++++++++ .../base-apt/files/distributions.in
>> |  3 + .../base-apt/files/multistrap.conf.in              | 28 +++++++
>>   meta/recipes-devtools/buildchroot/buildchroot.bb   | 28 ++++++-
>>   .../buildchroot/files/configscript.sh              |  1 -
>>   .../buildchroot/files/multistrap.conf.in           | 18 +---
>>   13 files changed, 217 insertions(+), 41 deletions(-)
>>   create mode 100644 meta/classes/isar-events.bbclass
>>   create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb
>>   create mode 100644
>> meta/recipes-devtools/base-apt/files/distributions.in create mode
>> 100644 meta/recipes-devtools/base-apt/files/multistrap.conf.in
>>
>