Re: [RFC PATCH 0/3] Reproducible build

[Claudius Heine <claudius.heine.ext@siemens.com>]

Hi,

On 2018-06-06 11:17, [ext] Claudius Heine wrote:
> Hi,
> 
> On 2018-06-05 12:42, [ext] Claudius Heine wrote:
>> Hi,
>>
>> On 2018-06-04 18:05, [ext] Claudius Heine wrote:
>>> I will try to post a small graphic about this soon.
>>
>> I attached the design Jan and me came up with yesterday.
>>
>> Hopefully its understandable enough even with all those arrows 
>> pointing around. I tried to vary the arrow lines and heads to signify 
>> dependencies/execution order, using of repositories and deploying 
>> packages to the repositories.
>>
>> The main point about this is that it will work with apt preferences 
>> repository pinning and that we will try to move all installed packages 
>> to the local cache after every step.
>>
>> I highlighted the new components in the diagram as green and the 
>> changed components as light green.
>>
>> If this is the way we want to go then I would try to get a RFC 
>> patchset started.
> 
> Unfortunately 'apt-move' has some bugs that make usage with debootstrap 
> near impossible without some patches.
> 
> The required patches to solve this issue were posted to the debian 
> mailinglist 5 years ago, so I guess that will not be merged in the near 
> future [1]. This means apt-move is pretty much dead.
> 
> So we have to find a different solution or need to implement it ourselves.

I did some further research into the different solutions for creating 
apt repositories:

  0. apt-move
     - Would fit our usage very well, but this is effectively dead and
       cannot be used, because of the unfixed bugs and some missing
       features.
  1. reprepro
     - We are using in in isar already, but development there seems to be
       a bit stale [1].
     - Fetching the corresponding source packages would need to be
       implemented by us.
  2. apt-ftparchive or dpkg-scanpackages
     - Can be used to generate Package and Release files, but does not
       care about creating the pool directory structure
     - We could implement the missing functionality as well as fetching
       the corresponding source packages ourselves.
  3. mini_dinstall
     - Implementation of a repository management daemon supporting the
       official maintenance cycle.
     - Submitting of package to be added to the repository is done with
       .changes files, that are generated by building source packages.
       We would have to generate those files ourselves. But I think
       that this might get awkward to implement around.
  4. aptly
     - Implemented in Go. I am a bit reluctant of using Go projects,
       because my experiences have shown that Go applications have the
       tendency to not care much about reproducibility and stable
       maintenance. (Importing code directly from github urls without
       even enforcing commitids???)
  5. pulp
     - Focused on RPM and Puppet, has DEB support with the Note:
       "WARNING: There may be bugs."

I looked at other products as well, but most are just unmaintained for a 
long time or doesn't solve our issues (mrepo, local-apt-repository).

If we don't find a solution that does not require any implementation 
ourselves, then I would prefer implementing the missing functionality in 
python instead of shell, with the hope that it becomes easier to maintain.

Claudius

[1] https://tracker.debian.org/pkg/reprepro


-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de