Re: Reproducibility of builds

["[ext] Christian Storm" <christian.storm@siemens.com>]

> > since I'm very interested in this feature, I'd like to resume this
> > discussion and to eventually come to an agreed upon proposal on how
> > to implement it. So, without further ado, here are my thoughts on
> > the subject:
> > 
> > Regardless of the concrete technical implementation, I guess we can
> > agree on the need for a local cache/repository/store in which the Debian
> > binary packages plus their sources have to be stored since one may not
> > rely on the availability of those files online for eternity.
> > 
> > These files in this cache/repository/store are the union of the Debian
> > binary packages installed in the resulting image plus their sources as
> > well as those installed in the buildchroot plus their sources.
> > The latter is required to be able to rebuild Debian packages built from
> > source with the same compiler version, libraries, -dev packages, etc. pp.
> > 
> > Having the cache/repository/store at hand, there should be a mechanism
> > to prime Isar with it, i.e., Isar should only and exclusively use Debian
> > binary packages and sources from this cache/repository/store.
> > This is again, irrespective of the technical implementation, be it via
> > a repository cache or other means like git, a proxy server or whatsoever.
> > 
> > Granted, if one changes, e.g, IMAGE_INSTALL_append, the build fails but
> > does so rightfully as the set of packages is modified, resulting in a
> > new version/epoch (=set of Debian packages plus their sources). So,
> > there should be a convenient "interface" provided by Isar to maintain
> > the cache/repository/store. For example, one may want to have different
> > versions/epochs that may correspond to particular versions (git sha) of
> > the Isar layer. Or one wants to later add a Debian package plus its
> > source (which is automatically fetched), resulting in a new
> > version/epoch etc.
> > 
> > The remaining question is how to fill the cache/repository/store. In
> > order to have a consistent version/epoch (=set of Debian packages plus
> > their sources), there should not be duplicate packages in it, i.e., the
> > same Debian package but with different versions.
> > This could currently happen because there is a "window of vulnerability":
> > multistrap is run twice, once for isar-image-base.bb and once for
> > buildchroot.bb. In between those two runs, the Debian mirror used could
> > get updated, resulting in a different version of the Debian package
> > being installed in buildchroot than in the resulting image.
> > This is an inherent problem of relying on the Debian way of distributing
> > packages as one cannot a priori control what particular package versions
> > one gets: In contrast to, e.g., Yocto where the particular package
> > versions are specified in the recipes, this does not hold for Isar as
> > the particular package versions are defined by the Debian mirror used,
> > hence, one gets "injected" the particular package versions.
> > So, what's required to reduce the "window of vulnerability" and to have
> > a consistent cache/repository/store for a particular version/epoch is to
> > make a snapshot-type download of the required packages. For this, of
> > course, one needs to know the concrete set of packages. This list could
> > be delivered by a "package trace" Isar run since not only multistrap
> > does install packages but sprinkled apt-get install commands do as well.
> > Thereafter, knowing the list, the snapshot-type download can happen,
> > hopefully resulting in a consistent cache/repository/store.
> > 
> > 
> > So, what do you think?
> 
> I agree with your formulation of the problem here.
> 
> Simple tracing of installed packages will have the problem you 
> described, that its possible that different versions of a package are 
> installed into buildchroot and image. So this trace needs to be cleaned 
> up and then based on that the whole process has to be started again to 
> create a consistent package list between buildchroot and image. This 
> doubles the build time in the trivial implementation.

Sure, there's no free lunch here :)
I'd rather strive for a good solution and avoid trivial implementations
to make lunch as close to free as it gets, to stay in the picture.


> With my suggestion of using a caching proxy, this could be solved 
> without any additional overhead.

Could be the case, what are the drawbacks? What proxy do you propose to
use? Maybe I missed something on the proxy suggestion.. Could you
please elaborate on this?


> I do have other ideas to do this, but that would restructure most of isar.

Well, at least speaking for myself, I'd like to hear those as I consider
this feature to be essential. Choice in solutions is always good :)



Kind regards,
   Christian

-- 
Dr. Christian Storm
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Otto-Hahn-Ring 6, 81739 M�nchen, Germany