Re: [RFC PATCH 0/3] Reproducible build

[Baurzhan Ismagulov <ibr@radix50.net>]

Hello Claudius,

On Fri, May 25, 2018 at 07:04:53PM +0200, Claudius Heine wrote:
> - Idea 0: Store tarball of debootstrap output with filled apt cache and use
>   that to restore isar-bootstrap.
> - Idea 1: Generate a repository from the cache and use that for the next
>   debootstrap run.
> - Idea 2: Like idea 1 but with aptly. And then use aptly to manage packages.
> - Idea 3: Create a whole repo mirror with aptly or similar and strip unused
>   packages later.
> - Idea 4: Create a whole repo mirror with aptly or similar and import used
>   package into a new repo.
> - Idea 5: Implementing a 'caching proxy' feature in aptly.
> - Idea 6: Implementing a caching proxy feature in isar.

Thanks for summarizing, this makes it easier to communicate.


Some general points first:

* I'm ok with a partial implementation that goes in the right direction.

* I'd really like to see user docs, also in RFC, because UX is a part of the
  design. It shows what use cases the change covers and how it does that.


Regarding the implementation, I think idea 1 is the right way to go. Today, we
operate with pure Debian inputs -- packages and metadata -- to build our
outputs. Debian inputs are what we should store.


> Because of the contra arguments 'whole local mirror' and 'different apt
> repo urls are used' I would got for 0 and 5.

Idea 1 is very similar to your current implementation and is achievable with
dpkg-scanpackages and debootstrapping.

I'm not proposing the whole mirror, just the packages you debootstrap +
dpkg-scanpackages.

Our actual problem is:

1. Getting the list of packages we need.

2. Fetching and managing them locally.

Proxying is a quick approach to avoid solving the problem rather than
addressing it. Also, it wouldn't support all Debian's fetch methods.


> Critique 1: Similar to my 'simple solution' but adds the creation of an
> additional repository to it. -> higher complexity
>     Pro: debootstrap process is done on every build.
>     Con: Different apt repo urls are used.
>            For me that is a no-go, because that means the configuration
>            is different between the initial and subsequent builds.

IIUC, this is also the case with your current implementation. You build without
or with ISAR_BOOTSTRAP_TARBALL. This could be changed to building with or
without e.g. ISAR_BOOTSTRAP_SOURCE containing a complete sources.list line.


>          How to add new packages later? (maybe like partial update?)

With the tarball, you suggest deleting and starting from scratch for now. For
the first step, I'd suggest to limit the usage to that. That is possible with
idea 1, too.

In the future, we'd need some tool. FWIW, I'm currently not aware of a tool
that does both (1) and (2) above or is sufficiently suitable for that. So, I
think we should work with Debian to get introspection on debootstrap and
apt-get and work on the tool for (2). Cooperating with some project would be
nice, but isn't a requirement for me.


>          How to handle multiple repos?
>            => map all repos from initial run to the local one.

Currently, you suggest to use multiple tarballs. With idea 1, you could provide
multiple directories.

FWIW, Alex's implementation [1] did (1) and (2) in a Debian way in a single
repo, without duplication.


>               And then what? => cannot be reverted, loss of information

It doesn't have to be reverted. Maintaining that manually would be
time-consuming, but that is what people are forced to do today anyway. The
feature would ease that burden till partial mirror management is implemented.


So, I'd be happy to review patches with:

1. Package directory as output, dpkg-scanpackages, and debootstrap on every
   build.

2. User docs giving an example of initial usage, versioning, and one update
   (e.g., deleting everything and starting from scratch). I agree that we don't
   impose versioning tools on our users, but we should demonstrate one simple
   working setup, even if with binary artifacts in git.


References:

1. https://groups.google.com/forum/#!topic/isar-users/QQUsVmSaAGk


With kind regards,
Baurzhan.