Re: [RFC PATCH 0/3] Reproducible build

["Maxim Yu. Osipov" <mosipov@ilbers.de>]

Hi Claudius,

On 06/14/2018 10:50 AM, Claudius Heine wrote:
> Hi,
> 
> On 2018-06-11 15:51, [ext] Claudius Heine wrote:
>> Hi,
>>
>> On 2018-06-11 10:45, [ext] Claudius Heine wrote:
>>> Hi Maxim,
>>>
>>> On 2018-06-07 10:08, Maxim Yu. Osipov wrote:
>>>> Hi Claudius,
>>>>
>>>> As far as I understood, 'apt-move' doesn't fit your requirements.
>>>
>>> The documented functionality of apt-move would fit the requirements, 
>>> but since its no longer maintained and has bugs that makes it 
>>> incompatible with debootstrap, it cannot be used.
>>>
>>>> Nevertheless, diagram illustrating the approach you propose would be 
>>>> be very helpful.
>>>>
>>>> As for diagram you attached to email below, honestly, it's rather 
>>>> difficult to understand it - too many arrows :(. It's worth to 
>>>> additionally describe the steps in text (and put corresponding 
>>>> numbers on the picture).
>>>
>>> Yes I know its a bit overwhelming with all those arrows. It was much 
>>> worse before and this is the result of much simplifications.
>>>
>>> But either way, since we cannot use apt-move I have to investigate 
>>> what is possible with reprepro and change the diagram accordingly.
>>
>> reprepro does provide many nice features that might be interesting to 
>> use. For instance the 'gensnapshot' command. This caused me to further 
>> redesign this approach as the diagram attached shows.
>>
>> I know this kind of diagram does not 100% the UML meaning, but I could 
>> not find a diagram type that really fits to what I want to show here.
>>
>> This time I try to explain a bit more about what those arrows mean and 
>> how to read this diagram.
>>
>> Dashed lines symbolizes dependencies between steps or 'packages' (in 
>> this case I mean those 'isar-bootstrap', 'buildchroot', 'recipes' and 
>> 'image'. So if they are followed from the 'debootstrap download' in 
>> the 'isar-bootstrap' package upwards, then this is the execution 
>> order. When the 'package import' component is reached the next 
>> 'package' in the dependency graph are started. This is 'buildchroot' 
>> or 'image'. Normally that is the buildchroot. Only if there are no 
>> recieps around that need a buildchroot it could be skipped. That means 
>> the last step to be executed is the 'finished' one.
>>
>> The arrows going out from those database things, mean that packages 
>> are used from their. Those arrows have the same arrow head as the 
>> dependency arrows. In this diagram I made the lines of the arrows that 
>> are going out from 'local partial upstream mirror' a bit thicker to 
>> better differentiate them from the other ones.
>>
>> The arrows that are going to 'to local mirror', 'to isar repository or 
>> 'create local snapshot' nodes have a slightly different arrow head. 
>> These arrows mean that there are debian packages that are added to repos.
>>
>> The basic idea it that after every step that installs or upgrades 
>> packages to the rootfs those packages are added to the local partial 
>> mirror. This can be done by first doing a download-only step adding it 
>> to the repo and then doing an install from the repo or the cache.
>>
>> I also changed how the repo is build. The current idea it to have one 
>> repository with 2 components. One component for all upstream packages 
>> and one for all isar built packages. When the snapshot is generated in 
>> the end it will create one containing both components.
>>
>> Maybe we should start adding the build time to all files in the deploy 
>> directory. This way we could add this to the name of the snapshot as 
>> well, so the association between those is made clear.
>>
>> What do you think?
> 
> This solution has one issue I can currently think of. It doesn't solve 
> this:
> 
>  >>> U3.4. Remove packages not used in any previous commit.
>  >>
>  >> I am currently not sure what you mean by that. Why would there be 
> packages
>  >> that aren't used in any previous commits?
>  >
>  > Bad wording, I meant just "remove unused packages".
> 
> We could solve that by always creating a fresh repository and add the 
> repository from the old build as primary source for the current one.
> 
> However, this is getting even more complex and I might need more 
> arrows... :/
> 
> It would be simple if we didn't need to combine this with updatability 
> of selected packages and automated fetching of new packages.

I agree with you - we may drop this use case as this is a kind of 
overkill which makes design more complex.

I need some clarification on your diagram on the box 'apt upgrade'. The 
comment to 'apt upgrade' states "prefers local mirror over upstream".

Do I understand correctly that as soon as we created our local partial 
mirror it will be only updated as a side effect of installation of build 
dependencies (buildchroot or package) or if the package listed in 
image's IMAGE_PREINSTALL is not in the local repo?

So we don't call 'apt upgrade' over upstream apt repos anymore, right?

It would be nice for understanding of how local mirroring works is you 
describe the case when we need a package which is not present in our 
local mirror (just imagine the case that we stick for a long time to our 
local mirror and we add a new package to IMAGE_PREINSTALL which is not 
in our local mirror and this package depends on the updated versions of 
upstream packages).

Kind regards,
Maxim.

> Cheers,
> Claudius
> 
>>
>> best regards,
>> Claudius
>>
>>>
>>> Claudius
>>>
>>>>
>>>> Kind regards,
>>>> Maxim.
>>>>
>>>> On 06/05/2018 12:42 PM, Claudius Heine wrote:
>>>>> Hi,
>>>>>
>>>>> On 2018-06-04 18:05, [ext] Claudius Heine wrote:
>>>>>> I will try to post a small graphic about this soon.
>>>>>
>>>>> I attached the design Jan and me came up with yesterday.
>>>>>
>>>>> Hopefully its understandable enough even with all those arrows 
>>>>> pointing around. I tried to vary the arrow lines and heads to 
>>>>> signify dependencies/execution order, using of repositories and 
>>>>> deploying packages to the repositories.
>>>>>
>>>>> The main point about this is that it will work with apt preferences 
>>>>> repository pinning and that we will try to move all installed 
>>>>> packages to the local cache after every step.
>>>>>
>>>>> I highlighted the new components in the diagram as green and the 
>>>>> changed components as light green.
>>>>>
>>>>> If this is the way we want to go then I would try to get a RFC 
>>>>> patchset started.
>>>>>
>>>>> Cheers,
>>>>> Claudius
>>>>>
>>>>
>>>>
>>>
>>
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov