Re: Reproducibility of builds

[Baurzhan Ismagulov <ibr@radix50.net>]

Hello Claudius,

thanks much for sharing the concept and the requirements!

Let's check whether I understand your concept correctly. Assume we have minimal
stretch binaries + hello source. Your concept provides for:

1. Start bitbake, which:
   1. Downloads debs to be installed and adds them into a local apt repo.
   2. Fetches hello sources, builds them, and copies the deb to the local apt
      repo.
   3. Bootstraps Debian and hello binary debs from the local apt repo.

Please correct me if I got anything incorrectly.

I like this workflow. This is what is used in Isar's predecessor. There, it is
implemented manually using debmirror.

The reason why Isar installs Debian packages from apt repos in Internet is to
give first-time users a setup working OOTB. If they start developing a product,
they are expected to create their own apt repos. See e.g.
http://events.linuxfoundation.org/sites/events/files/slides/isar-elce-2016_1.pdf,
slides 19 ("Debian apt" repo) and 26 ("Create repos for all components:
Debian...").

That is why I was originally thinking about a tool that would support this
manual workflow. After pondering on your proposal, I think it makes sense. I'd
like to see the following features:

* The functionality is implemented in a standalone tool usable manually or from
  bitbake.

* The functionality is implemented based on dry-run output of
  {deboot,multi,...}strap.

* The feature can be turned off in Isar's local configuration.

* The tool supports initial mirroring as well as an update. This should also be
  controllable in Isar's local config.

What I don't like is the implementation via a http proxy. IMHO, it's too
indirect for the task (why bother with dynamic proxying if the list of packages
is defined statically in a given apt repo). It supports only one of apt's six
fetch methods (such as https, file, ssh, etc., see sources.list(5), more could
be defined in the future or in special environments). The implementation is
going to be complex, since it needs to distinguish between different build
process chains in the same environment (two bitbakes running in a single
docker).

It should be trivial to get a list of packages from multistrap. The same
functionality is available in debootstrap, when we move to it. Mirroring could
be done by an existing or a new tool. The latter may be a step to identify
requirements and get experience with the workflow before integrating the
functionality into the former (possibly upon feedback from Debian community).

Archiving of the apt repo is a CM issue outside of Isar. For reproducing older
versions, it should be managed in an SCM (e.g., git). Synchronization between
the right product and apt repo revisions is also outside Isar and could be
solved e.g. with kas. Or, one goes hard-core and commits apt stuff into the
product repo. In the future, we might come with a better solution for archiving
and version pinning; at this stage I'd like to utilize existing Debian means
first before going further. The details of the pinning concept would be
affected by bitbake debian/control backend implementation.

Similarly, at this stage I don't address advanced issues like sharing modified
and non-modified apt repos, which could be implemented by a KISS
jessie/Packages and myjessie/Packages with the shared pool. If we have many of
them in practice (which I doubt), we could still return to the issue.


Some comments below.

On Thu, Aug 03, 2017 at 10:13:12AM +0200, Claudius Heine wrote:
> am I right that Isar supports or should support reproducible root file
> system build?

Yes, this is possible outside of Isar. We wish that Isar makes that easier.


> If I understand correctly, when multistrap is called, it fetches always the
> latest version of all packages from the debian repository mirrors. Am I
> mistaken or is this feature still on the roadmap?

In the sense I interpret your wording, yes, multistrap always fetches the
latest version of all packages from the Debian repo.

That said, for a given repo, there is only one version of every package,
defined in the Packages file. It is the latest for that repo. Given a URI in
Internet, multistrap always fetches its "latest" (and the only) Packages and
installs the "latest" (and the only) package versions.


With kind regards,
Baurzhan.