Re: Reproducibility of builds

[Alexander Smirnov <asmirnov@ilbers.de>]

On 09/05/2017 01:38 PM, Jan Kiszka wrote:
> On 2017-09-05 12:05, Alexander Smirnov wrote:
>>
>>
>> On 08/28/2017 02:27 PM, Claudius Heine wrote:
>>> Hi,
>>>
>>> On 08/21/2017 01:23 PM, [ext] Claudius Heine wrote:
>>>> Hi,
>>>>
>>>> On 08/03/2017 10:13 AM, Claudius Heine wrote:
>>>>> Hi,
>>>>>
>>>>> am I right that Isar supports or should support reproducible root
>>>>> file system build?
>>>>>
>>>>> If I understand correctly, when multistrap is called, it fetches
>>>>> always the latest version of all packages from the debian repository
>>>>> mirrors. Am I mistaken or is this feature still on the roadmap?
>>>>>
>>>>> I that is on the roadmap, how are you thinking of solving this issue?
>>>>>
>>>>> The openembedded way would be to seperate the fetch and 'install'
>>>>> step and first download all packages into the DL_DIR and then use
>>>>> them from there. Maybe we could create this pipeline:
>>>>>
>>>>> dpkg-binary Recipe:
>>>>>
>>>>> fetch deb file into downloads -> insert into local repository
>>>>>
>>>>> dpkg-source Recipe:
>>>>>
>>>>> fetch sources into downloads -> build packages -> insert into local
>>>>> repository
>>>>>
>>>>> image Recipe:
>>>>>
>>>>> fetch all required packages into downloads -> insert all of them
>>>>> into the local repository -> create root fs using only the local
>>>>> repository
>>>>>
>>>>> Multistrap provides a '--source-dir DIR' parameter, that stores all
>>>>> installed packages into a directory. So if we would use that as a
>>>>> fetcher, then we would create a temporary rootfs just to get all
>>>>> required packages for the project.
>>>>>
>>>>> Are there other possible solutions for this?
>>>>
>>>> The problem with this solution is that its not possible to create
>>>> multiple images with different sets of packages that share the
>>>> version of the all the common packages.
>>>>
>>>> An alternative solution is to employ a repository cacher that caches
>>>> the 'Packages.gz' of the first request. This way it would also be
>>>> faster then running multistrap one additional time just to fetch all
>>>> required packages.
>>>>
>>>> Maybe apt-cacher-ng or something similar can be used for this.
>>>> However I am currently not sure how this can be integrated into the
>>>> current build process. Some ideas? Maybe implementing a simple repo
>>>> caching proxy that is integrated into isar?
>>>>
>>>> The repository cacher is likely a daemon running in parallel to
>>>> multistrap and fetches everything to the DL_DIR that is requested by
>>>> it. Maybe provide a 'clean_package_cache' task, that deletes the
>>>> cached 'Packages.gz', causing the next root fs build to use new
>>>> package versions.
>>>>
>>>> I would really like to hear some feedback on this.
>>>
>>> In our meeting today, it was discussed that we should collect all
>>> requirements for this feature and discuss possible implementation
>>> ideas based on those requirements.
>>>
>>> Here are some requirements from my side:
>>>
>>>       1 If multiple different images with some common set of packages are
>>>         build with one bitbake call, then all images should contain
>>>         exactly the same version of every package that it has in common
>>>         with any of the other images.
>>>
>>>       2 The resulting image should only depend on the build environment
>>>         and isar metadata, not on the point in time it is build.
>>>         This means if the environment, including the downloads directory,
>>>         is complete (for instance by an earlier build of the image), every
>>>         following build of this image recipe should result in exactly the
>>>         same packages installed on this image.
>>>
>>>       3 Binary and source packages should be part of the archival process.
>>>         Source packages are useful in case some package needs to be
>>>         patched at a later date. Binary packages are useful, because
>>>         building them from source packages is currently not 100%
>>>         reproducible in Debian upstream. [1]
>>>
>>>       4 For development, it should be possible to easily reset the
>>>         environment, triggering an upgrade of the packages on the next
>>>         image build.
>>>
>>>       5 Deployable in CI environments. What those are exactly should be
>>>         further discussed. Here are some:
>>>
>>>         5.1 Possibility to use a download cache, that is not bound to only
>>>             one product/image/environment
>>>
>>>         5.2 More than one build at the same time in one environment should
>>>             be possible
>>>
>>>       6 Efficiency: The reproducibility feature should be time and
>>>         resource efficient as possible. E.g. Process should only fetch and
>>>         store the required files.
>>>
>>>       7 Outputs a description file with the name and version of every
>>>         package deployed/used in the image/environment.
>>>
>>> To 5: Since I don't have much experience with CI systems, requirements
>>> mentioned here might not be correct.
>>>
>>> Any comment or requirement additions are welcome.
>>
>> Thank you for the requirements, they quite good describe your usecase.
>> Unfortunately, ATM I don't know all the capabilities of
>> multistrap/debootstrap, so could not propose too much.
> 
> Then I guess that needs to be explored further before we can decide
> which patch to go. Who could contribute to this?

I think the one who will investigate the opportunities, also should 
implement this. It's a complete feature, so I can't handle it right now 
due to load with another features.

Alex

> 
>>
>> In general, I think there could be following solutions:
>>
>>   - Create local apt cache with specified packages versions.
>>   - Patch multistrap to add capabilities to specify package versions.
>>   - Add hook to multistrap hooks (for example, in configscript.sh), that
>> will re-install desired package versions via apt-get.
>>
>