Re: Reproducibility of builds

[Claudius Heine <claudius.heine.ext@siemens.com>]

Hi,

On 08/03/2017 10:13 AM, Claudius Heine wrote:
> Hi,
> 
> am I right that Isar supports or should support reproducible root file 
> system build?
> 
> If I understand correctly, when multistrap is called, it fetches always 
> the latest version of all packages from the debian repository mirrors. 
> Am I mistaken or is this feature still on the roadmap?
> 
> I that is on the roadmap, how are you thinking of solving this issue?
> 
> The openembedded way would be to seperate the fetch and 'install' step 
> and first download all packages into the DL_DIR and then use them from 
> there. Maybe we could create this pipeline:
> 
> dpkg-binary Recipe:
> 
> fetch deb file into downloads -> insert into local repository
> 
> dpkg-source Recipe:
> 
> fetch sources into downloads -> build packages -> insert into local 
> repository
> 
> image Recipe:
> 
> fetch all required packages into downloads -> insert all of them into 
> the local repository -> create root fs using only the local repository
> 
> Multistrap provides a '--source-dir DIR' parameter, that stores all 
> installed packages into a directory. So if we would use that as a 
> fetcher, then we would create a temporary rootfs just to get all 
> required packages for the project.
> 
> Are there other possible solutions for this?

The problem with this solution is that its not possible to create 
multiple images with different sets of packages that share the version 
of the all the common packages.

An alternative solution is to employ a repository cacher that caches the 
'Packages.gz' of the first request. This way it would also be faster 
then running multistrap one additional time just to fetch all required 
packages.

Maybe apt-cacher-ng or something similar can be used for this.
However I am currently not sure how this can be integrated into the 
current build process. Some ideas? Maybe implementing a simple repo 
caching proxy that is integrated into isar?

The repository cacher is likely a daemon running in parallel to 
multistrap and fetches everything to the DL_DIR that is requested by it. 
Maybe provide a 'clean_package_cache' task, that deletes the cached 
'Packages.gz', causing the next root fs build to use new package versions.

I would really like to hear some feedback on this.

Cheers,
Claudius