Re: Reproducibility of builds

[Claudius Heine <claudius.heine.ext@siemens.com>]

Hi Christian,

On 11/14/2017 05:04 PM, [ext] Christian Storm wrote:
> Hi,
> 
> since I'm very interested in this feature, I'd like to resume this
> discussion and to eventually come to an agreed upon proposal on how
> to implement it. So, without further ado, here are my thoughts on
> the subject:
> 
> Regardless of the concrete technical implementation, I guess we can
> agree on the need for a local cache/repository/store in which the Debian
> binary packages plus their sources have to be stored since one may not
> rely on the availability of those files online for eternity.
> 
> These files in this cache/repository/store are the union of the Debian
> binary packages installed in the resulting image plus their sources as
> well as those installed in the buildchroot plus their sources.
> The latter is required to be able to rebuild Debian packages built from
> source with the same compiler version, libraries, -dev packages, etc. pp.
> 
> Having the cache/repository/store at hand, there should be a mechanism
> to prime Isar with it, i.e., Isar should only and exclusively use Debian
> binary packages and sources from this cache/repository/store.
> This is again, irrespective of the technical implementation, be it via
> a repository cache or other means like git, a proxy server or whatsoever.
> 
> Granted, if one changes, e.g, IMAGE_INSTALL_append, the build fails but
> does so rightfully as the set of packages is modified, resulting in a
> new version/epoch (=set of Debian packages plus their sources). So,
> there should be a convenient "interface" provided by Isar to maintain
> the cache/repository/store. For example, one may want to have different
> versions/epochs that may correspond to particular versions (git sha) of
> the Isar layer. Or one wants to later add a Debian package plus its
> source (which is automatically fetched), resulting in a new
> version/epoch etc.
> 
> The remaining question is how to fill the cache/repository/store. In
> order to have a consistent version/epoch (=set of Debian packages plus
> their sources), there should not be duplicate packages in it, i.e., the
> same Debian package but with different versions.
> This could currently happen because there is a "window of vulnerability":
> multistrap is run twice, once for isar-image-base.bb and once for
> buildchroot.bb. In between those two runs, the Debian mirror used could
> get updated, resulting in a different version of the Debian package
> being installed in buildchroot than in the resulting image.
> This is an inherent problem of relying on the Debian way of distributing
> packages as one cannot a priori control what particular package versions
> one gets: In contrast to, e.g., Yocto where the particular package
> versions are specified in the recipes, this does not hold for Isar as
> the particular package versions are defined by the Debian mirror used,
> hence, one gets "injected" the particular package versions.
> So, what's required to reduce the "window of vulnerability" and to have
> a consistent cache/repository/store for a particular version/epoch is to
> make a snapshot-type download of the required packages. For this, of
> course, one needs to know the concrete set of packages. This list could
> be delivered by a "package trace" Isar run since not only multistrap
> does install packages but sprinkled apt-get install commands do as well.
> Thereafter, knowing the list, the snapshot-type download can happen,
> hopefully resulting in a consistent cache/repository/store.
> 
> 
> So, what do you think?

I agree with your formulation of the problem here.

Simple tracing of installed packages will have the problem you 
described, that its possible that different versions of a package are 
installed into buildchroot and image. So this trace needs to be cleaned 
up and then based on that the whole process has to be started again to 
create a consistent package list between buildchroot and image. This 
doubles the build time in the trivial implementation.

With my suggestion of using a caching proxy, this could be solved 
without any additional overhead.

I do have other ideas to do this, but that would restructure most of isar.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de