Re: Reproducibility of builds

[Claudius Heine <claudius.heine.ext@siemens.com>]

Hi Baurzhan,

On 09/18/2017 05:05 PM, Baurzhan Ismagulov wrote:
> What I don't like is the implementation via a http proxy. IMHO, it's too
> indirect for the task (why bother with dynamic proxying if the list of packages
> is defined statically in a given apt repo).

Only if someone bothers to create a separate debian mirror repository 
for every product. It uses much more resources. It would be much easier 
to have a global package cache and a project local package index for it. 
IMO that would be only possible with a caching repo proxy.

> It supports only one of apt's six
> fetch methods (such as https, file, ssh, etc., see sources.list(5), more could
> be defined in the future or in special environments).

At first. I started with a http proxy it the easiest to implement. Its 
always possible to add additional functionality to the proxy to support 
other fetch methods if necessary. But IMO that not really that important.

> The implementation is
> going to be complex, since it needs to distinguish between different build
> process chains in the same environment (two bitbakes running in a single
> docker).

Why? We have more then one port available, so we can run more then one 
proxy simultaneously for each build. My current implementation just 
chooses a free port and makes it available to the calling process.

> It should be trivial to get a list of packages from multistrap. The same
> functionality is available in debootstrap, when we move to it.

The problem is we still need to use apt in the buildchroot to install 
additional build dependencies for each recipe. Those are not part of 
what multistrap/debootstrap lists out. But since it would go through the 
http proxy it would be part of the static package cache.
	
> Mirroring could
> be done by an existing or a new tool. The latter may be a step to identify
> requirements and get experience with the workflow before integrating the
> functionality into the former (possibly upon feedback from Debian community).

As I said, I don't see the sense in creating a full debian mirror for 
every project. And partial mirrors are difficult to create because of 
multistrap/debootstrap (in case of the buildchroot) don't know about 
every package that is added to the image.

> 
> Archiving of the apt repo is a CM issue outside of Isar. For reproducing older
> versions, it should be managed in an SCM (e.g., git).

That should be possible. Just archive the package index in a git repo 
and the packages in a git lfs repo.

> Synchronization between
> the right product and apt repo revisions is also outside Isar and could be
> solved e.g. with kas.

Never said that it is. But isar is responsible for providing ways to 
import/export some kind of package list into a build.

> Or, one goes hard-core and commits apt stuff into the
> product repo.

That might depend on your 'product' definition, but for me product is 
not a image. So products can have varying package versions, while images 
obviously doesn't. So committing them together with products makes no 
sense to me. But committing them together with the final image with a 
reference of the used refspec of the product repository makes more sense.

> In the future, we might come with a better solution for archiving
> and version pinning; at this stage I'd like to utilize existing Debian means
> first before going further. The details of the pinning concept would be
> affected by bitbake debian/control backend implementation.

I said nothing about pining, because IMO package updates etc. should 
still be possible on the target if wanted. But we should be able to 
recreate images at least from a package list. So apt package pinning is 
just a different solution for a different problem. If you mean pinning 
just in the bootstraping phase, then yes, that would be nice. But I 
don't know how that can solve the buildchroot problem. Also since the 
package index contains just one version of each package, I don't see how 
it would be possible to pin them to an older version at this stage, 
because those would no longer be available in the index and *bootstrap 
would not know where to fetch them.

AFAIK Debian currently has no convenient means for solving these issues yet.

I used apt-cacher-ng when I worked with elbe, but setting that up for 
every project separately is a big hassle. I want a easy solutions where 
this stuff is done inside of the normal bitbake process, where not every 
developer has to wire up her own process of building root file systems.
Because if they have to build their own most developers don't care about 
it or it becomes impossible to recreate images because a couple of 
unknown software packages in some unknown version and with unknown 
configuration are necessary for it.

So its important to use normal upstream package mirrors and have a 
process in place inside bitbake that cares about these issues transparently.

IMO its important to not have to many options and unneeded complexity. 
So reproducibility should be the default and everyone is free to 
update/clear the package index manually via a single bitbake task.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de