Re: [RFC PATCH 0/3] Reproducible build

[Claudius Heine <claudius.heine.ext@siemens.com>]

Hi,

On 2018-06-11 15:51, [ext] Claudius Heine wrote:
> Hi,
> 
> On 2018-06-11 10:45, [ext] Claudius Heine wrote:
>> Hi Maxim,
>>
>> On 2018-06-07 10:08, Maxim Yu. Osipov wrote:
>>> Hi Claudius,
>>>
>>> As far as I understood, 'apt-move' doesn't fit your requirements.
>>
>> The documented functionality of apt-move would fit the requirements, 
>> but since its no longer maintained and has bugs that makes it 
>> incompatible with debootstrap, it cannot be used.
>>
>>> Nevertheless, diagram illustrating the approach you propose would be 
>>> be very helpful.
>>>
>>> As for diagram you attached to email below, honestly, it's rather 
>>> difficult to understand it - too many arrows :(. It's worth to 
>>> additionally describe the steps in text (and put corresponding 
>>> numbers on the picture).
>>
>> Yes I know its a bit overwhelming with all those arrows. It was much 
>> worse before and this is the result of much simplifications.
>>
>> But either way, since we cannot use apt-move I have to investigate 
>> what is possible with reprepro and change the diagram accordingly.
> 
> reprepro does provide many nice features that might be interesting to 
> use. For instance the 'gensnapshot' command. This caused me to further 
> redesign this approach as the diagram attached shows.
> 
> I know this kind of diagram does not 100% the UML meaning, but I could 
> not find a diagram type that really fits to what I want to show here.
> 
> This time I try to explain a bit more about what those arrows mean and 
> how to read this diagram.
> 
> Dashed lines symbolizes dependencies between steps or 'packages' (in 
> this case I mean those 'isar-bootstrap', 'buildchroot', 'recipes' and 
> 'image'. So if they are followed from the 'debootstrap download' in the 
> 'isar-bootstrap' package upwards, then this is the execution order. When 
> the 'package import' component is reached the next 'package' in the 
> dependency graph are started. This is 'buildchroot' or 'image'. Normally 
> that is the buildchroot. Only if there are no recieps around that need a 
> buildchroot it could be skipped. That means the last step to be executed 
> is the 'finished' one.
> 
> The arrows going out from those database things, mean that packages are 
> used from their. Those arrows have the same arrow head as the dependency 
> arrows. In this diagram I made the lines of the arrows that are going 
> out from 'local partial upstream mirror' a bit thicker to better 
> differentiate them from the other ones.
> 
> The arrows that are going to 'to local mirror', 'to isar repository or 
> 'create local snapshot' nodes have a slightly different arrow head. 
> These arrows mean that there are debian packages that are added to repos.
> 
> The basic idea it that after every step that installs or upgrades 
> packages to the rootfs those packages are added to the local partial 
> mirror. This can be done by first doing a download-only step adding it 
> to the repo and then doing an install from the repo or the cache.
> 
> I also changed how the repo is build. The current idea it to have one 
> repository with 2 components. One component for all upstream packages 
> and one for all isar built packages. When the snapshot is generated in 
> the end it will create one containing both components.
> 
> Maybe we should start adding the build time to all files in the deploy 
> directory. This way we could add this to the name of the snapshot as 
> well, so the association between those is made clear.
> 
> What do you think?

This solution has one issue I can currently think of. It doesn't solve this:

 >>> U3.4. Remove packages not used in any previous commit.
 >>
 >> I am currently not sure what you mean by that. Why would there be 
packages
 >> that aren't used in any previous commits?
 >
 > Bad wording, I meant just "remove unused packages".

We could solve that by always creating a fresh repository and add the 
repository from the old build as primary source for the current one.

However, this is getting even more complex and I might need more 
arrows... :/

It would be simple if we didn't need to combine this with updatability 
of selected packages and automated fetching of new packages.

Cheers,
Claudius

> 
> best regards,
> Claudius
> 
>>
>> Claudius
>>
>>>
>>> Kind regards,
>>> Maxim.
>>>
>>> On 06/05/2018 12:42 PM, Claudius Heine wrote:
>>>> Hi,
>>>>
>>>> On 2018-06-04 18:05, [ext] Claudius Heine wrote:
>>>>> I will try to post a small graphic about this soon.
>>>>
>>>> I attached the design Jan and me came up with yesterday.
>>>>
>>>> Hopefully its understandable enough even with all those arrows 
>>>> pointing around. I tried to vary the arrow lines and heads to 
>>>> signify dependencies/execution order, using of repositories and 
>>>> deploying packages to the repositories.
>>>>
>>>> The main point about this is that it will work with apt preferences 
>>>> repository pinning and that we will try to move all installed 
>>>> packages to the local cache after every step.
>>>>
>>>> I highlighted the new components in the diagram as green and the 
>>>> changed components as light green.
>>>>
>>>> If this is the way we want to go then I would try to get a RFC 
>>>> patchset started.
>>>>
>>>> Cheers,
>>>> Claudius
>>>>
>>>
>>>
>>
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de