Re: Reproducibility of builds

[Christian Storm <christian.storm@siemens.com>]

Hi,

since I'm very interested in this feature, I'd like to resume this
discussion and to eventually come to an agreed upon proposal on how
to implement it. So, without further ado, here are my thoughts on
the subject:

Regardless of the concrete technical implementation, I guess we can
agree on the need for a local cache/repository/store in which the Debian
binary packages plus their sources have to be stored since one may not
rely on the availability of those files online for eternity.

These files in this cache/repository/store are the union of the Debian
binary packages installed in the resulting image plus their sources as
well as those installed in the buildchroot plus their sources.
The latter is required to be able to rebuild Debian packages built from
source with the same compiler version, libraries, -dev packages, etc. pp.

Having the cache/repository/store at hand, there should be a mechanism
to prime Isar with it, i.e., Isar should only and exclusively use Debian
binary packages and sources from this cache/repository/store.
This is again, irrespective of the technical implementation, be it via
a repository cache or other means like git, a proxy server or whatsoever.

Granted, if one changes, e.g, IMAGE_INSTALL_append, the build fails but
does so rightfully as the set of packages is modified, resulting in a
new version/epoch (=set of Debian packages plus their sources). So,
there should be a convenient "interface" provided by Isar to maintain
the cache/repository/store. For example, one may want to have different
versions/epochs that may correspond to particular versions (git sha) of
the Isar layer. Or one wants to later add a Debian package plus its
source (which is automatically fetched), resulting in a new
version/epoch etc.

The remaining question is how to fill the cache/repository/store. In
order to have a consistent version/epoch (=set of Debian packages plus
their sources), there should not be duplicate packages in it, i.e., the
same Debian package but with different versions.
This could currently happen because there is a "window of vulnerability":
multistrap is run twice, once for isar-image-base.bb and once for
buildchroot.bb. In between those two runs, the Debian mirror used could
get updated, resulting in a different version of the Debian package
being installed in buildchroot than in the resulting image.
This is an inherent problem of relying on the Debian way of distributing
packages as one cannot a priori control what particular package versions
one gets: In contrast to, e.g., Yocto where the particular package
versions are specified in the recipes, this does not hold for Isar as
the particular package versions are defined by the Debian mirror used,
hence, one gets "injected" the particular package versions.
So, what's required to reduce the "window of vulnerability" and to have
a consistent cache/repository/store for a particular version/epoch is to
make a snapshot-type download of the required packages. For this, of
course, one needs to know the concrete set of packages. This list could
be delivered by a "package trace" Isar run since not only multistrap
does install packages but sprinkled apt-get install commands do as well.
Thereafter, knowing the list, the snapshot-type download can happen,
hopefully resulting in a consistent cache/repository/store.


So, what do you think?


Besten Gru�,
   Christian

-- 
Dr. Christian Storm
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Otto-Hahn-Ring 6, 81739 M�nchen, Germany