From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6558372643829972992 X-Received: by 2002:adf:f3cd:: with SMTP id g13-v6mr112785wrp.8.1528966213978; Thu, 14 Jun 2018 01:50:13 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:4641:: with SMTP id j1-v6ls2231363wrs.3.gmail; Thu, 14 Jun 2018 01:50:13 -0700 (PDT) X-Google-Smtp-Source: ADUXVKI0pVatY/OTLKSvBmbam1fjm6CeGt91xC3VcmichOqPCUUzkgQM1w4Czv5N3I5mmZR2pNMi X-Received: by 2002:a5d:454b:: with SMTP id p11-v6mr117360wrr.6.1528966213596; Thu, 14 Jun 2018 01:50:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528966213; cv=none; d=google.com; s=arc-20160816; b=pDOVoAmP35nsLmEVWmR8LboiqlGyV6/DNGXQlJGvm6Ri3yzz0fUS4VRWkWAbNbzQFL mKGYXmw5eN3TLPwr5FI3MNfvwACvBNRE+1IYF6Y6IRyg4f81sufoODkFCXI6KYszCGMv RnrH6edfuSdH/NMYGegKLBBdryJ2bzcpFxSYieRw/fwzEYqgC1lqMnUV0YDojUsxMPCK /Plw2gU9EERCj+UVOwwaeHGOAeFAo2Mrt8wQ5a5rqVK8GODlO5EeMgEiKxJ2OBuymzQ1 qxdH4vxShx7jsvJvicVTJKTlvOA35VnLkIYoVDmlSVfDA+nqfKyPwPvRZlIBK2RIltj5 CUkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:references:cc:to:from:subject :arc-authentication-results; bh=qyNsWetyp8KvEvgS8IhSd47/3Y4EWDKmSqZ4xUncVaw=; b=A+Gpr0JXsPdBwJxlldz2ExqE9jsT0o/1iRriW90MFaztc1G+N5duzazuxh1r3C5+We CswjdOLWz6RHxjRFJvwtuaB83tC3WljQq+wYm1xOLj/hP90Ie02KwzRQlUgQ13HvotCr EmKD9kORnDLbDQ1e8jSCVB9+3wx3I6qMQwAPs32bLTr+F7TptXf/pjgGGe+9l5xN+H09 zAZkcrQtuzqRYP4EyZ4jFG9gHlewysyiBwvXFmv9zZf5MexynNbaGe3BpwR4kIgEiGuP fMqc/AtOOxerusaWDoNGv8NGl+MntcQ1LQ61Il8cw/JVLcXIocKIEMpgiNWbFQrOj/R9 dIcA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id o4-v6si142459wrm.1.2018.06.14.01.50.13 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Jun 2018 01:50:13 -0700 (PDT) Received-SPF: pass (google.com: domain of claudius.heine.ext@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id w5E8oCTE025462 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Jun 2018 10:50:12 +0200 Received: from [139.25.69.69] (linux-ses-ext02.ppmd.siemens.net [139.25.69.69]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id w5E8oDhu028545; Thu, 14 Jun 2018 10:50:13 +0200 Subject: Re: [RFC PATCH 0/3] Reproducible build From: Claudius Heine To: "Maxim Yu. Osipov" , isar-users@googlegroups.com Cc: Silvano Cirujano Cuesta References: <3467a5ec-182e-8c9a-cd19-7ad898323be7@siemens.com> <20180523063206.29180-1-claudius.heine.ext@siemens.com> <20180524180027.09b7b880@md1pvb1c.ad001.siemens.net> <3a6032fee718de6cf44fff4e8051a8c7a89a6471.camel@denx.de> <20180604113736.GD5657@yssyq.radix50.net> <3431e051-cdc2-1ba6-8d8f-c426679c6954@siemens.com> <1e8bda03-70e6-93bf-6a30-29740164e83d@siemens.com> <348ee070-a530-b594-b8dc-2d85a4fdc083@siemens.com> Message-ID: Date: Thu, 14 Jun 2018 10:50:13 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: <348ee070-a530-b594-b8dc-2d85a4fdc083@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: XrCM6WL0mCEC Hi, On 2018-06-11 15:51, [ext] Claudius Heine wrote: > Hi, > > On 2018-06-11 10:45, [ext] Claudius Heine wrote: >> Hi Maxim, >> >> On 2018-06-07 10:08, Maxim Yu. Osipov wrote: >>> Hi Claudius, >>> >>> As far as I understood, 'apt-move' doesn't fit your requirements. >> >> The documented functionality of apt-move would fit the requirements, >> but since its no longer maintained and has bugs that makes it >> incompatible with debootstrap, it cannot be used. >> >>> Nevertheless, diagram illustrating the approach you propose would be >>> be very helpful. >>> >>> As for diagram you attached to email below, honestly, it's rather >>> difficult to understand it - too many arrows :(. It's worth to >>> additionally describe the steps in text (and put corresponding >>> numbers on the picture). >> >> Yes I know its a bit overwhelming with all those arrows. It was much >> worse before and this is the result of much simplifications. >> >> But either way, since we cannot use apt-move I have to investigate >> what is possible with reprepro and change the diagram accordingly. > > reprepro does provide many nice features that might be interesting to > use. For instance the 'gensnapshot' command. This caused me to further > redesign this approach as the diagram attached shows. > > I know this kind of diagram does not 100% the UML meaning, but I could > not find a diagram type that really fits to what I want to show here. > > This time I try to explain a bit more about what those arrows mean and > how to read this diagram. > > Dashed lines symbolizes dependencies between steps or 'packages' (in > this case I mean those 'isar-bootstrap', 'buildchroot', 'recipes' and > 'image'. So if they are followed from the 'debootstrap download' in the > 'isar-bootstrap' package upwards, then this is the execution order. When > the 'package import' component is reached the next 'package' in the > dependency graph are started. This is 'buildchroot' or 'image'. Normally > that is the buildchroot. Only if there are no recieps around that need a > buildchroot it could be skipped. That means the last step to be executed > is the 'finished' one. > > The arrows going out from those database things, mean that packages are > used from their. Those arrows have the same arrow head as the dependency > arrows. In this diagram I made the lines of the arrows that are going > out from 'local partial upstream mirror' a bit thicker to better > differentiate them from the other ones. > > The arrows that are going to 'to local mirror', 'to isar repository or > 'create local snapshot' nodes have a slightly different arrow head. > These arrows mean that there are debian packages that are added to repos. > > The basic idea it that after every step that installs or upgrades > packages to the rootfs those packages are added to the local partial > mirror. This can be done by first doing a download-only step adding it > to the repo and then doing an install from the repo or the cache. > > I also changed how the repo is build. The current idea it to have one > repository with 2 components. One component for all upstream packages > and one for all isar built packages. When the snapshot is generated in > the end it will create one containing both components. > > Maybe we should start adding the build time to all files in the deploy > directory. This way we could add this to the name of the snapshot as > well, so the association between those is made clear. > > What do you think? This solution has one issue I can currently think of. It doesn't solve this: >>> U3.4. Remove packages not used in any previous commit. >> >> I am currently not sure what you mean by that. Why would there be packages >> that aren't used in any previous commits? > > Bad wording, I meant just "remove unused packages". We could solve that by always creating a fresh repository and add the repository from the old build as primary source for the current one. However, this is getting even more complex and I might need more arrows... :/ It would be simple if we didn't need to combine this with updatability of selected packages and automated fetching of new packages. Cheers, Claudius > > best regards, > Claudius > >> >> Claudius >> >>> >>> Kind regards, >>> Maxim. >>> >>> On 06/05/2018 12:42 PM, Claudius Heine wrote: >>>> Hi, >>>> >>>> On 2018-06-04 18:05, [ext] Claudius Heine wrote: >>>>> I will try to post a small graphic about this soon. >>>> >>>> I attached the design Jan and me came up with yesterday. >>>> >>>> Hopefully its understandable enough even with all those arrows >>>> pointing around. I tried to vary the arrow lines and heads to >>>> signify dependencies/execution order, using of repositories and >>>> deploying packages to the repositories. >>>> >>>> The main point about this is that it will work with apt preferences >>>> repository pinning and that we will try to move all installed >>>> packages to the local cache after every step. >>>> >>>> I highlighted the new components in the diagram as green and the >>>> changed components as light green. >>>> >>>> If this is the way we want to go then I would try to get a RFC >>>> patchset started. >>>> >>>> Cheers, >>>> Claudius >>>> >>> >>> >> > -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de