Re: [RFC PATCH 0/3] Reproducible build

[Claudius Heine <ch@denx.de>]

[-- Attachment #1: Type: text/plain, Size: 3603 bytes --]

Hi Maxim.

On Wed, 2018-05-23 at 16:30 +0200, Maxim Yu. Osipov wrote:
> Hi Claudius,
> 
> I've looked through discussion thread.
> 
> As far as I understood with the proposed approach we don't have
> the ability to reproduce this tarball - it contains some unversioned 
> snapshot of isar-bootstrap rootfs, containing unversioned snapshot
> of 
> debian's packages cache used to create rootfs. It's fine if you just 
> want to reproduce locally the current build from the scratch in your 
> sandbox by avoiding debootstrap stage (fetching again packages, etc).
> 
> Do you have another use-case scenario in mind?
> 
> F.e. to share this tarball with other developers (linked to
> particular 
> version of isar tree) so they can fully reproduce the build?
> 
> If yes, how do you plan to version/manage such growing list of
> tarballs? 
> As it was mentioned in the discussion, upgrading one package from
> debian 
> repo will result to other tarball.

My focus to tackle the reproducibility was two fold:

  1. Output the build input in some form
  2. Allow the build input to be used in the subsequent builds while
     allowing customization e.g. fixes to isar packages.

I choose the form of the output to be just one tarball because its
pretty simple to move it around. To backup it somewhere you can extract
it if you like. Maybe put it into a OSTree repo or just create
incremental tarballs as I described before. How this tarball is
versioned, how and where it is stored was not in the scope of my work
and belongs to the backup mechanism chosen by the users IMO. We can
just point to the files that need backuping. That is how OE does it as
well with the downloads directory. They just don't pack it together,
but they also don't have to worry about a root fs with permissions.

Now to the choice of the 'build input'. Of course it would be great to
make the debootstrap step reproducible as well, but that means I have
to create a repository with the packages. Creating a new repository
from the packages in the cache means that later adding new packages,
that were not part of the cache isn't possible, since they aren't part
of the repository.

My earlier suggestion (months ago) was using a manual controllable
repository proxy. That could be in the form of a http webserver or
proxy. This proxy would fetch and store the index and packages from
upstream. That is serious implementation effort, however.

apt_cacher_ng and aptly are missing important features, that would need
to be implemented there. So we could try to interest those projects to
this and propose patches. And maybe we should do that. Aplty looks
really nice, but I don't know how they stand of creating partial
repositories where the index contains entries that aren't available in
the repo itself.

So to solve this locally in isar was just using what debootstrap
outputs + all the used packages from the cache as the base for the next
build. I agree that this currently isn't the nicest solution possible,
but its pretty simple to implement and could be expanded upon. If this
isn't enough then I would need to look into aptly for example to create
a more extensive solution.

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]