Re: Reproducibility of builds

[Alexander Smirnov <asmirnov@ilbers.de>]

On 08/28/2017 02:27 PM, Claudius Heine wrote:
> Hi,
> 
> On 08/21/2017 01:23 PM, [ext] Claudius Heine wrote:
>> Hi,
>>
>> On 08/03/2017 10:13 AM, Claudius Heine wrote:
>>> Hi,
>>>
>>> am I right that Isar supports or should support reproducible root 
>>> file system build?
>>>
>>> If I understand correctly, when multistrap is called, it fetches 
>>> always the latest version of all packages from the debian repository 
>>> mirrors. Am I mistaken or is this feature still on the roadmap?
>>>
>>> I that is on the roadmap, how are you thinking of solving this issue?
>>>
>>> The openembedded way would be to seperate the fetch and 'install' 
>>> step and first download all packages into the DL_DIR and then use 
>>> them from there. Maybe we could create this pipeline:
>>>
>>> dpkg-binary Recipe:
>>>
>>> fetch deb file into downloads -> insert into local repository
>>>
>>> dpkg-source Recipe:
>>>
>>> fetch sources into downloads -> build packages -> insert into local 
>>> repository
>>>
>>> image Recipe:
>>>
>>> fetch all required packages into downloads -> insert all of them into 
>>> the local repository -> create root fs using only the local repository
>>>
>>> Multistrap provides a '--source-dir DIR' parameter, that stores all 
>>> installed packages into a directory. So if we would use that as a 
>>> fetcher, then we would create a temporary rootfs just to get all 
>>> required packages for the project.
>>>
>>> Are there other possible solutions for this?
>>
>> The problem with this solution is that its not possible to create 
>> multiple images with different sets of packages that share the version 
>> of the all the common packages.
>>
>> An alternative solution is to employ a repository cacher that caches 
>> the 'Packages.gz' of the first request. This way it would also be 
>> faster then running multistrap one additional time just to fetch all 
>> required packages.
>>
>> Maybe apt-cacher-ng or something similar can be used for this.
>> However I am currently not sure how this can be integrated into the 
>> current build process. Some ideas? Maybe implementing a simple repo 
>> caching proxy that is integrated into isar?
>>
>> The repository cacher is likely a daemon running in parallel to 
>> multistrap and fetches everything to the DL_DIR that is requested by 
>> it. Maybe provide a 'clean_package_cache' task, that deletes the 
>> cached 'Packages.gz', causing the next root fs build to use new 
>> package versions.
>>
>> I would really like to hear some feedback on this.
> 
> In our meeting today, it was discussed that we should collect all 
> requirements for this feature and discuss possible implementation ideas 
> based on those requirements.
> 
> Here are some requirements from my side:
> 
>      1 If multiple different images with some common set of packages are
>        build with one bitbake call, then all images should contain
>        exactly the same version of every package that it has in common
>        with any of the other images.
> 
>      2 The resulting image should only depend on the build environment
>        and isar metadata, not on the point in time it is build.
>        This means if the environment, including the downloads directory,
>        is complete (for instance by an earlier build of the image), every
>        following build of this image recipe should result in exactly the
>        same packages installed on this image.
> 
>      3 Binary and source packages should be part of the archival process.
>        Source packages are useful in case some package needs to be
>        patched at a later date. Binary packages are useful, because
>        building them from source packages is currently not 100%
>        reproducible in Debian upstream. [1]
> 
>      4 For development, it should be possible to easily reset the
>        environment, triggering an upgrade of the packages on the next
>        image build.
> 
>      5 Deployable in CI environments. What those are exactly should be
>        further discussed. Here are some:
> 
>        5.1 Possibility to use a download cache, that is not bound to only
>            one product/image/environment
> 
>        5.2 More than one build at the same time in one environment should
>            be possible
> 
>      6 Efficiency: The reproducibility feature should be time and
>        resource efficient as possible. E.g. Process should only fetch and
>        store the required files.
> 
>      7 Outputs a description file with the name and version of every
>        package deployed/used in the image/environment.
> 
> To 5: Since I don't have much experience with CI systems, requirements 
> mentioned here might not be correct.
> 
> Any comment or requirement additions are welcome.

Thank you for the requirements, they quite good describe your usecase. 
Unfortunately, ATM I don't know all the capabilities of 
multistrap/debootstrap, so could not propose too much.

In general, I think there could be following solutions:

  - Create local apt cache with specified packages versions.
  - Patch multistrap to add capabilities to specify package versions.
  - Add hook to multistrap hooks (for example, in configscript.sh), that 
will re-install desired package versions via apt-get.

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov