From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6558372643829972992 X-Received: by 2002:a19:d919:: with SMTP id q25-v6mr858303lfg.14.1529468422367; Tue, 19 Jun 2018 21:20:22 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:9256:: with SMTP id v22-v6ls168462ljg.4.gmail; Tue, 19 Jun 2018 21:20:21 -0700 (PDT) X-Google-Smtp-Source: ADUXVKL1CPl/EcfugnnUwJnzuweh9aP61L+9cSdzEUcEefkJ1BGlcKFVdUOdz1nMQmEvGNd77OQ/ X-Received: by 2002:a2e:8556:: with SMTP id u22-v6mr950707ljj.19.1529468421822; Tue, 19 Jun 2018 21:20:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529468421; cv=none; d=google.com; s=arc-20160816; b=wWEvzK03a/oeaH3GiwGGcxXND6O56hO2LXj4n2gyDl+59LPMAM0HVAc28Z/NuzmjmJ dxVTQYOjCCaITbnhpsTxn2j/JFNqIzlMOQOux/8Yev9RoEtpmFKo9+pzhM2H2Oo3XLCD TEeXcZ/AujiqHvARRYTLDVFF/KcZO14jLXtJeQxqi3zsGg/1bVWO3zVwty78LnGcTY9v aAmyykXrMzLzBUJjUNZRzIH+SKpAA/LgU74nFfwdn9qhjK5bAnydCvYfJ/lY4RHqYpas J7I9o1x2rAY6lkFRvaejAhMvCaOEdOo88PKTbMZAsPHxRgzaSilAHAIdBHv2hCMc2nAm szeg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:organization:from:references:cc:to :subject:arc-authentication-results; bh=Ec3fUx99+uJXrtzR7rp3NUzUBKnJDstM+TvKYKwsYFM=; b=ZmqfJD/WfeCnY7BHZmWU+kly4FarLRrv4G+t0nA8tyneDDTvA/FRAvsf4j0/GuhiYp oOX+zgT1cXpx/A7EolyMc7R5oU78ciMr7Sc+b/LIsSBV9rdTWOEdPfDsew9jmSt9LVnF o4awCmSV/yvKwCl+O1hE0otEbtJ9QmNc1qX/VN1xzntoeLpLknsebGI08uiolQ7nlrX7 D7f2wuZvZ9/MKiPj2q2W+ZVZWvBU2X7XZVaIDDMWKhfx5UT3tEu/zA/PKa/Te7rn8NO5 SgceakozXTPXiAJghnaS5RtEyn4uHPjIOpMv93/jP5NRtV5EZx8QJq37b3PB8vOiGZP7 p+gg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of mosipov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=mosipov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id l24-v6si30246lja.1.2018.06.19.21.20.21 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Jun 2018 21:20:21 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of mosipov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of mosipov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=mosipov@ilbers.de Received: from [192.168.50.163] (d51a48a80.access.telenet.be [81.164.138.128]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w5K4KIWg022011 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 20 Jun 2018 06:20:20 +0200 Subject: Re: [RFC PATCH 0/3] Reproducible build To: Claudius Heine , isar-users@googlegroups.com Cc: Silvano Cirujano Cuesta References: <3467a5ec-182e-8c9a-cd19-7ad898323be7@siemens.com> <20180523063206.29180-1-claudius.heine.ext@siemens.com> <20180524180027.09b7b880@md1pvb1c.ad001.siemens.net> <3a6032fee718de6cf44fff4e8051a8c7a89a6471.camel@denx.de> <20180604113736.GD5657@yssyq.radix50.net> <3431e051-cdc2-1ba6-8d8f-c426679c6954@siemens.com> <1e8bda03-70e6-93bf-6a30-29740164e83d@siemens.com> <348ee070-a530-b594-b8dc-2d85a4fdc083@siemens.com> From: "Maxim Yu. Osipov" Organization: ilbers GmbH Message-ID: <383e7f38-82f9-cc3c-dfec-f974d0e0bfc0@ilbers.de> Date: Wed, 20 Jun 2018 06:20:18 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: s1zsXIBIAg/Y Hi Claudius, On 06/14/2018 10:50 AM, Claudius Heine wrote: > Hi, > > On 2018-06-11 15:51, [ext] Claudius Heine wrote: >> Hi, >> >> On 2018-06-11 10:45, [ext] Claudius Heine wrote: >>> Hi Maxim, >>> >>> On 2018-06-07 10:08, Maxim Yu. Osipov wrote: >>>> Hi Claudius, >>>> >>>> As far as I understood, 'apt-move' doesn't fit your requirements. >>> >>> The documented functionality of apt-move would fit the requirements, >>> but since its no longer maintained and has bugs that makes it >>> incompatible with debootstrap, it cannot be used. >>> >>>> Nevertheless, diagram illustrating the approach you propose would be >>>> be very helpful. >>>> >>>> As for diagram you attached to email below, honestly, it's rather >>>> difficult to understand it - too many arrows :(. It's worth to >>>> additionally describe the steps in text (and put corresponding >>>> numbers on the picture). >>> >>> Yes I know its a bit overwhelming with all those arrows. It was much >>> worse before and this is the result of much simplifications. >>> >>> But either way, since we cannot use apt-move I have to investigate >>> what is possible with reprepro and change the diagram accordingly. >> >> reprepro does provide many nice features that might be interesting to >> use. For instance the 'gensnapshot' command. This caused me to further >> redesign this approach as the diagram attached shows. >> >> I know this kind of diagram does not 100% the UML meaning, but I could >> not find a diagram type that really fits to what I want to show here. >> >> This time I try to explain a bit more about what those arrows mean and >> how to read this diagram. >> >> Dashed lines symbolizes dependencies between steps or 'packages' (in >> this case I mean those 'isar-bootstrap', 'buildchroot', 'recipes' and >> 'image'. So if they are followed from the 'debootstrap download' in >> the 'isar-bootstrap' package upwards, then this is the execution >> order. When the 'package import' component is reached the next >> 'package' in the dependency graph are started. This is 'buildchroot' >> or 'image'. Normally that is the buildchroot. Only if there are no >> recieps around that need a buildchroot it could be skipped. That means >> the last step to be executed is the 'finished' one. >> >> The arrows going out from those database things, mean that packages >> are used from their. Those arrows have the same arrow head as the >> dependency arrows. In this diagram I made the lines of the arrows that >> are going out from 'local partial upstream mirror' a bit thicker to >> better differentiate them from the other ones. >> >> The arrows that are going to 'to local mirror', 'to isar repository or >> 'create local snapshot' nodes have a slightly different arrow head. >> These arrows mean that there are debian packages that are added to repos. >> >> The basic idea it that after every step that installs or upgrades >> packages to the rootfs those packages are added to the local partial >> mirror. This can be done by first doing a download-only step adding it >> to the repo and then doing an install from the repo or the cache. >> >> I also changed how the repo is build. The current idea it to have one >> repository with 2 components. One component for all upstream packages >> and one for all isar built packages. When the snapshot is generated in >> the end it will create one containing both components. >> >> Maybe we should start adding the build time to all files in the deploy >> directory. This way we could add this to the name of the snapshot as >> well, so the association between those is made clear. >> >> What do you think? > > This solution has one issue I can currently think of. It doesn't solve > this: > > >>> U3.4. Remove packages not used in any previous commit. > >> > >> I am currently not sure what you mean by that. Why would there be > packages > >> that aren't used in any previous commits? > > > > Bad wording, I meant just "remove unused packages". > > We could solve that by always creating a fresh repository and add the > repository from the old build as primary source for the current one. > > However, this is getting even more complex and I might need more > arrows... :/ > > It would be simple if we didn't need to combine this with updatability > of selected packages and automated fetching of new packages. I agree with you - we may drop this use case as this is a kind of overkill which makes design more complex. I need some clarification on your diagram on the box 'apt upgrade'. The comment to 'apt upgrade' states "prefers local mirror over upstream". Do I understand correctly that as soon as we created our local partial mirror it will be only updated as a side effect of installation of build dependencies (buildchroot or package) or if the package listed in image's IMAGE_PREINSTALL is not in the local repo? So we don't call 'apt upgrade' over upstream apt repos anymore, right? It would be nice for understanding of how local mirroring works is you describe the case when we need a package which is not present in our local mirror (just imagine the case that we stick for a long time to our local mirror and we add a new package to IMAGE_PREINSTALL which is not in our local mirror and this package depends on the updated versions of upstream packages). Kind regards, Maxim. > Cheers, > Claudius > >> >> best regards, >> Claudius >> >>> >>> Claudius >>> >>>> >>>> Kind regards, >>>> Maxim. >>>> >>>> On 06/05/2018 12:42 PM, Claudius Heine wrote: >>>>> Hi, >>>>> >>>>> On 2018-06-04 18:05, [ext] Claudius Heine wrote: >>>>>> I will try to post a small graphic about this soon. >>>>> >>>>> I attached the design Jan and me came up with yesterday. >>>>> >>>>> Hopefully its understandable enough even with all those arrows >>>>> pointing around. I tried to vary the arrow lines and heads to >>>>> signify dependencies/execution order, using of repositories and >>>>> deploying packages to the repositories. >>>>> >>>>> The main point about this is that it will work with apt preferences >>>>> repository pinning and that we will try to move all installed >>>>> packages to the local cache after every step. >>>>> >>>>> I highlighted the new components in the diagram as green and the >>>>> changed components as light green. >>>>> >>>>> If this is the way we want to go then I would try to get a RFC >>>>> patchset started. >>>>> >>>>> Cheers, >>>>> Claudius >>>>> >>>> >>>> >>> >> > -- Maxim Osipov ilbers GmbH Maria-Merian-Str. 8 85521 Ottobrunn Germany +49 (151) 6517 6917 mosipov@ilbers.de http://ilbers.de/ Commercial register Munich, HRB 214197 General Manager: Baurzhan Ismagulov