Re: Idea for implementing reproducible builds

[Baurzhan Ismagulov <ibr@radix50.net>]

Hello Claudius,

On Tue, May 22, 2018 at 01:55:21PM +0200, Claudius Heine wrote:
> I am still working on reproducible builds and here is my current idea to
> solve this.
> 
> Simple put: Mount the /var/cache/apt/archives of the images and buildchroot
> to the isar-bootstrap root file system and then create a tarball of it. This
> way we have a tarball of the build just after debootstrap + upgrade with the
> one 'apt update' step done, but without any other changes to it and all used
> packages already in the apt package cache.
> 
> When restoring just skip most of the isar-bootstrap steps and extract the
> tarball instead, since the packages are available in the package cache and
> the package index is not updated it will use the packages from the cache.
> 
> This way we would side step the obstacle to make debootstrap reproducible by
> just using its product while the reset of the process can be redone by isar.

Thanks for sharing.

As I understand it:

1. The user runs bitbake isar-image-base, which
   1. Debootstraps a rootfs
   2. Tars it
   3. Unpacks the tar into buildchroot/rootfs and isar-image-base/rootfs
2. The user adds the tarball to the product repo

Is this correct?


In this scenario:

* Step 1: How does bitbake decide whether to debootstrap or use the tarball?

* Step 2: If I have the following repo, where should the tar file be located
  and versioned?

  myrepo
  - meta
  - meta-isar
  - product1
  - product2

* If two products built from one repo have non-identical rootfses, what does
  the tarball contain?

* What is the user supposed to do if he wants to update the tar to the current
  upstream, fully or in part?


Considering our existing use cases, I'd suggest a couple of changes to your
concept.


Let's abbreviate our copy of Debian artifacts as "debian-mirror" (be it in form
of a tarball or anything else).


I see the following use cases:

U1. debian-mirror doesn't exist. Create debian-mirror from upstream.

U2.1. debian-mirror is versioned, e.g. in git.

U2.2. Use debian-mirror for buildchroot/rootfs and isar-image-base/rootfs.

U2.3. Don't use upstream for building buildchroot/rootfs and
      isar-image-base/rootfs.

U3.1. debian-mirror exists. Update all packages from upstream into
      debian-mirror.

U3.2. debian-mirror exists. Update chosen packages from upstream into
      debian-mirror. E.g., openssl, optionally its dependencies, optionally its
      dependents.

U3.3. debian-mirror exists and is used by two products. One product has to be
      updated. The other one will be updated later. For product 1, update
      chosen or all packages from upstream to debian-mirror. Product 2 should
      still use the old packages.

U3.4. Remove packages not used in any previous commit.


Given those, I'd suggest using debs as versioned entities instead of the rootfs
tarball.

Create an apt repo with dpkg-scanpackages and dpkg-scansources and use it to
debootstrap buildchroot and isar-image-base.

This would address U2.3 and U3.3. This has been tested in practice, works
well, and is in my opinion the best way to solve the problem.

With versioned tarballs, an update of a single package would make the whole
tarball change. This makes the history unreadable, wastes disk space, and many
tools (including git) have problems with big files.


>From the UX perspective, I'd prefer to separate building images from preparing
debian-mirror if possible. A separate command / task / bitbake run with a var
set / unset, etc. E.g., bitbake -C createmirror isar-image-base, bitbake -C
updatemirror isar-image-base, etc.


Please include user documentation when you provide patches.


I think that in the end, we'll have to providing specialized infrastructure for
managing debian-mirror(s). It may or may not be based on anything from
https://wiki.debian.org/DebianRepository/Setup?action=show&redirect=HowToSetupADebianRepository
.


With kind regards,
Baurzhan.