Re: Reproducibility of builds

["[ext] Christian Storm" <christian.storm@siemens.com>]

> > [...]
> > > With my suggestion of using a caching proxy, this could be solved 
> > > without any additional overhead.
> > 
> > Could be the case, what are the drawbacks?
> 
> More complexity and stuff to implement. Also maybe download speed.
> 
> >  What proxy do you propose to use?
> 
> I was at first going with my own standalone proxy implementation in
> pure stdlib python, so that it could be completely integrated into
> isar. 

Why not hooking this into the fetcher(s) so that it's integrated rather
than a standalone thing? As a bonus, you'll have full control on this
from the Isar core/code. I think the main invention here is the code
that does the consistent version/epoch guarantee anyway...


> I had a very simple solution ready rather quickly, but it was
> only synchronous and as such could only handle one connection at a
> time. Instead of just throwing more threads at it, I wanted to go the
> asyncio route. Sadly the python stdlib does not provide a http
> implementation for asyncio. I wasn't clear how to proceed from here
> further (aiohttp dependency or minimal own http implementation).

Ah, OK. Wouldn't this account for premature optimization? :)


> The other idea is to just use a ready made apt caching proxy like apt-
> cache-ng. But here I am unsure if its flexible enough to use in our
> case. Starting it multiple times in parallel with different ports for
> different caches and only user privileges might be possible but I
> suspect that seperating the pool and the dists folder (pool should go
> to DL_DIR while dists is part of the TMP_DIR) could be more difficult.

I would consider on the bonus side for this that we don't have to
develop/maintain a custom solution, given that it suits our purposes of
course...


> >  Maybe I missed something on the proxy suggestion.. Could you
> > please elaborate on this?
> 
> As for the integration the basic idea was that for taged bitbake tasks
> the proxy is started and sets the *_PROXY environment variables. This
> should be doable with some mods to the base.bbclass and some external
> python scripts.
> 
> > 
> > 
> > > I do have other ideas to do this, but that would restructure most
> > > of isar.
> > 
> > Well, at least speaking for myself, I'd like to hear those as I
> > consider
> > this feature to be essential. Choice in solutions is always good :)
> > 
> 
> One idea that I got when I first investigated isar, was trying to be oe
> compatible as much as possible. So using this idea would solve the
> reproducable builds as well:
> 
> Basically implementing debootstrap with bitbake recipes that are
> created virtually on runtime by downloading and parsing the
> 'dists/*/*/*/Packages.gz' file.

Those virtual recipes then will have to be serialized as they contain
the version number of the package, right?


> I suppose it should be possible to fetch the Packages file at an early
> parsing step in a bitbake build, if its not already preset, and fill
> the bitbake data store with recipe definitions that fetch those binary
> deb packages, have the appropriate dependencies and install them into
> the root file system.

Yes, or do a 'download-only' step prior to building as it's available
on Yocto. 


> However, this idea is still in the brain storming phase.
> 
> Since that would involve a very big redesign I don't think its feasible
> currently.

Sounds interesting, at least for me...



Kind regards,
   Christian

-- 
Dr. Christian Storm
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Otto-Hahn-Ring 6, 81739 M�nchen, Germany