From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6558372643829972992 X-Received: by 2002:a1c:6c10:: with SMTP id h16-v6mr88358wmc.18.1529482335288; Wed, 20 Jun 2018 01:12:15 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:f117:: with SMTP id p23-v6ls579226wmh.4.canary-gmail; Wed, 20 Jun 2018 01:12:14 -0700 (PDT) X-Google-Smtp-Source: ADUXVKKNMKLzBL5l5D58SlYKsglGKzfjQD2i7L6f4Ir1CdOxmZ3orjqRTIxIFPcaIou2TloScoOc X-Received: by 2002:a1c:2d54:: with SMTP id t81-v6mr93430wmt.3.1529482334885; Wed, 20 Jun 2018 01:12:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529482334; cv=none; d=google.com; s=arc-20160816; b=wyT+zBbl7nY7BhbIj9RbTJWu9leFUlZJAy6CpbayI/NmLF9qioT6eLqt39ZtbgjBte Rfkuuxi7CvmO3Br04mCDJgzgetk/bmRtmi96dmOH9k7aBVdga55j8VDUYzbI84RYv6Hy EmrHBkf9JYod6/Byd49LBXqjK6o3RCX5zz4ErRyqd6D6U5cuZ8EFLTEHONc36LYp0ORw +SSuai5klrU6VRMRf4uWfn2mGLwLPePT18zegwJsCFhAWol8x0+W24pf9o8GtD/dyUfv P+mpLXDVq0sAQ1n24Kmg7jgFSZce7g8Isy9XuRnRYLk3/BJ+Zg1Xgd8mLG/AgfEawN53 lBzg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=KHU9GRgtblNyqHJK8q/PzvWmXYRaRr+QMG8oOVxSJqA=; b=gLTD/AE8TunsqsgFDTQcl7yd9uKbmXA0SIg0yJCO68sCFcp2CoOVp00ZXJ2+J6DhTi wBO9zG0JYYhSnaCKRPHTyYUEDGZjbIQ5DSffQscqY8S4U/pi4xNLnrXB5AVP1NbR8xfb KS75Xo+meluVpJz5fbhbzcl0a3IHXfByo385n8iuljzKflu2MsZFp95ydXVDQuE9IEzu 5G7EkE7ns8bbgEdB6AonGWpjKTJKhaTHw7BnBOBE4sq+oytm4Ud3WfbdngBklQW2ww3q R1Eny3rfNi35YcYMv/SSjH8Dgp10bIB7kvbZxX/TjILoYDZkI8dy4vNo48o3ggeaSquL trwQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id c18-v6si39931wrq.5.2018.06.20.01.12.14 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 20 Jun 2018 01:12:14 -0700 (PDT) Received-SPF: pass (google.com: domain of claudius.heine.ext@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id w5K8CEq8007171 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 20 Jun 2018 10:12:14 +0200 Received: from [139.25.69.69] (linux-ses-ext02.ppmd.siemens.net [139.25.69.69]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id w5K8CEo6032343; Wed, 20 Jun 2018 10:12:14 +0200 Subject: Re: [RFC PATCH 0/3] Reproducible build To: "Maxim Yu. Osipov" , isar-users@googlegroups.com Cc: Silvano Cirujano Cuesta References: <3467a5ec-182e-8c9a-cd19-7ad898323be7@siemens.com> <20180523063206.29180-1-claudius.heine.ext@siemens.com> <20180524180027.09b7b880@md1pvb1c.ad001.siemens.net> <3a6032fee718de6cf44fff4e8051a8c7a89a6471.camel@denx.de> <20180604113736.GD5657@yssyq.radix50.net> <3431e051-cdc2-1ba6-8d8f-c426679c6954@siemens.com> <1e8bda03-70e6-93bf-6a30-29740164e83d@siemens.com> <348ee070-a530-b594-b8dc-2d85a4fdc083@siemens.com> <383e7f38-82f9-cc3c-dfec-f974d0e0bfc0@ilbers.de> From: Claudius Heine Message-ID: Date: Wed, 20 Jun 2018 10:12:14 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: <383e7f38-82f9-cc3c-dfec-f974d0e0bfc0@ilbers.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: JLv+wtanQk4g Hi Maxim, On 2018-06-20 06:20, Maxim Yu. Osipov wrote: > Hi Claudius, > > On 06/14/2018 10:50 AM, Claudius Heine wrote: >> Hi, >> >> On 2018-06-11 15:51, [ext] Claudius Heine wrote: >>> Hi, >>> >>> On 2018-06-11 10:45, [ext] Claudius Heine wrote: >>>> Hi Maxim, >>>> >>>> On 2018-06-07 10:08, Maxim Yu. Osipov wrote: >>>>> Hi Claudius, >>>>> >>>>> As far as I understood, 'apt-move' doesn't fit your requirements. >>>> >>>> The documented functionality of apt-move would fit the requirements, >>>> but since its no longer maintained and has bugs that makes it >>>> incompatible with debootstrap, it cannot be used. >>>> >>>>> Nevertheless, diagram illustrating the approach you propose would >>>>> be be very helpful. >>>>> >>>>> As for diagram you attached to email below, honestly, it's rather >>>>> difficult to understand it - too many arrows :(. It's worth to >>>>> additionally describe the steps in text (and put corresponding >>>>> numbers on the picture). >>>> >>>> Yes I know its a bit overwhelming with all those arrows. It was much >>>> worse before and this is the result of much simplifications. >>>> >>>> But either way, since we cannot use apt-move I have to investigate >>>> what is possible with reprepro and change the diagram accordingly. >>> >>> reprepro does provide many nice features that might be interesting to >>> use. For instance the 'gensnapshot' command. This caused me to >>> further redesign this approach as the diagram attached shows. >>> >>> I know this kind of diagram does not 100% the UML meaning, but I >>> could not find a diagram type that really fits to what I want to show >>> here. >>> >>> This time I try to explain a bit more about what those arrows mean >>> and how to read this diagram. >>> >>> Dashed lines symbolizes dependencies between steps or 'packages' (in >>> this case I mean those 'isar-bootstrap', 'buildchroot', 'recipes' and >>> 'image'. So if they are followed from the 'debootstrap download' in >>> the 'isar-bootstrap' package upwards, then this is the execution >>> order. When the 'package import' component is reached the next >>> 'package' in the dependency graph are started. This is 'buildchroot' >>> or 'image'. Normally that is the buildchroot. Only if there are no >>> recieps around that need a buildchroot it could be skipped. That >>> means the last step to be executed is the 'finished' one. >>> >>> The arrows going out from those database things, mean that packages >>> are used from their. Those arrows have the same arrow head as the >>> dependency arrows. In this diagram I made the lines of the arrows >>> that are going out from 'local partial upstream mirror' a bit thicker >>> to better differentiate them from the other ones. >>> >>> The arrows that are going to 'to local mirror', 'to isar repository >>> or 'create local snapshot' nodes have a slightly different arrow >>> head. These arrows mean that there are debian packages that are added >>> to repos. >>> >>> The basic idea it that after every step that installs or upgrades >>> packages to the rootfs those packages are added to the local partial >>> mirror. This can be done by first doing a download-only step adding >>> it to the repo and then doing an install from the repo or the cache. >>> >>> I also changed how the repo is build. The current idea it to have one >>> repository with 2 components. One component for all upstream packages >>> and one for all isar built packages. When the snapshot is generated >>> in the end it will create one containing both components. >>> >>> Maybe we should start adding the build time to all files in the >>> deploy directory. This way we could add this to the name of the >>> snapshot as well, so the association between those is made clear. >>> >>> What do you think? >> >> This solution has one issue I can currently think of. It doesn't solve >> this: >> >>  >>> U3.4. Remove packages not used in any previous commit. >>  >> >>  >> I am currently not sure what you mean by that. Why would there be >> packages >>  >> that aren't used in any previous commits? >>  > >>  > Bad wording, I meant just "remove unused packages". >> >> We could solve that by always creating a fresh repository and add the >> repository from the old build as primary source for the current one. >> >> However, this is getting even more complex and I might need more >> arrows... :/ >> >> It would be simple if we didn't need to combine this with updatability >> of selected packages and automated fetching of new packages. > > I agree with you - we may drop this use case as this is a kind of > overkill which makes design more complex. > > I need some clarification on your diagram on the box 'apt upgrade'. The > comment to 'apt upgrade' states "prefers local mirror over upstream". > > Do I understand correctly that as soon as we created our local partial > mirror it will be only updated as a side effect of installation of build > dependencies (buildchroot or package) or if the package listed in > image's IMAGE_PREINSTALL is not in the local repo? > > So we don't call 'apt upgrade' over upstream apt repos anymore, right? So the basic idea is that the local partial mirror is given a priority of 1001 via apt preferences. So calling an unrestricted `apt update` and `apt upgrade` can be done anytime in the build. `apt upgrade` would prefer installing any package from the local partial mirror over any upstream distribution even if upstream has a more recent version of that package. Only packages that are not available in the local partial mirror would be fetched from upstream, since it has to get them from somewhere. After that though this new package is added to the local partial mirror, so any builds after that would fetch the same version of the package from the local partial mirror instead of the upstream. > It would be nice for understanding of how local mirroring works is you > describe the case when we need a package which is not present in our > local mirror (just imagine the case that we stick for a long time to our > local mirror and we add a new package to IMAGE_PREINSTALL which is not > in our local mirror and this package depends on the updated versions of > upstream packages). Ok, I try to describe some scenarios that might help understand what I imagine: 1. Adding a package to an old build - Add the package to the IMAGE_PREINSTALL list - Start build - Everything until the image recipe runs the same, since all packages that are used by isar-bootstrap and buildchroot are the same and can be fetched from the local partial mirror. Only difference is that the image contains an updated index of the upstream repositories. - In the image recipe a new package is installed, that is not available in the local partial mirror. That image is then fetched from the upstream mirror and installed. - That could cause a conflict with the old versions of the rest of the system and hopefully lead to an error, requiring manual steps to add an old but compatible package to be added to the local partial mirror by hand. I don't think this kind of stuff can be prevented or automated by isar reasonably. - The newly installed package is installed to the local partial mirror, making it available for subsequent builds. 2. Removing a package from an old build This requires some changes of the current system. A new 'local partial mirror repo' needs to be created at every build and filled with packages needed for that specific build. This way no unused packages find their way into the repo. An old local partial mirror repo will used as an immutable package repository with the 1001 priority. This approach borrows from concepts like copy-on-write and functional programming. - Remove the package from the IMAGE_PREINSTALL list - Start build - Everything is done like the build before while using the local partial mirror repository from the build before as a base. The removed package is not installed to the image or added to the fresh local partial mirror repo. 3. Updating a package from an old build The goal here is to get an updated version of a package to the image and by extension to the local partial mirror repo. To solve this depends of where this package is originating. If its a specific version the developer wants, it might be better to download that package add add it to isar via a recipe. Since the apt-isar repo, containing the packages 'build' by isar, should have a higher priority than the local partial mirror repository, the specific version would be preferred over any version from upstream or the local partial mirror repo. To solve this, we might need to implement some helper scripts outside of bitbake build. Similar to the `bitbake-layers` script. If the developer generally wants to use *some* version of the package that is currently used by upstream, pinning rules could be used for one run and then disabled again, just to make sure that those packages are now in the local partial mirror repo. The other solution for both of those is to modify the local partial mirror outside of the bitbake build. This is the way I showed in the diagram as well. I am a bit hesitant to do that tough, since it modifies something that is used for reproducible builds outside of the build. Maybe if we have the changes I described in point 2, we could just add those packages to the newly created local partial mirror cache, but then this needs to be part of the build as well. This would result in these priorities: 1. local partial mirror repo from previous build: 1001 2. local partial mirror repo from current build: 1002 3. isar apt repo for current isar built packages: 1003 So what I meant was adding the updated packages to 2 before the build is started. And I am hesitant about adding packages to 1, because that modifies stuff from a previous build. If updating certain packages from upstream should be done within the build, then it could be done between setting the sources.list entry and apt-preferences. This way those packages will be fetched from upstream if its more up-to-date there and then added to the local partial mirror cache. In any-case tough this could lead to a 'local partial mirror repo' that uses a mix of many different upstream repos snapshots, where it can be obscure as to *why* specific versions are used. The first 2 points behave similar when adding/removing build or runtime dependencies, then they might just apply to the buildchroot as well. regards, Claudius > > Kind regards, > Maxim. > >> Cheers, >> Claudius >> >>> >>> best regards, >>> Claudius >>> >>>> >>>> Claudius >>>> >>>>> >>>>> Kind regards, >>>>> Maxim. >>>>> >>>>> On 06/05/2018 12:42 PM, Claudius Heine wrote: >>>>>> Hi, >>>>>> >>>>>> On 2018-06-04 18:05, [ext] Claudius Heine wrote: >>>>>>> I will try to post a small graphic about this soon. >>>>>> >>>>>> I attached the design Jan and me came up with yesterday. >>>>>> >>>>>> Hopefully its understandable enough even with all those arrows >>>>>> pointing around. I tried to vary the arrow lines and heads to >>>>>> signify dependencies/execution order, using of repositories and >>>>>> deploying packages to the repositories. >>>>>> >>>>>> The main point about this is that it will work with apt >>>>>> preferences repository pinning and that we will try to move all >>>>>> installed packages to the local cache after every step. >>>>>> >>>>>> I highlighted the new components in the diagram as green and the >>>>>> changed components as light green. >>>>>> >>>>>> If this is the way we want to go then I would try to get a RFC >>>>>> patchset started. >>>>>> >>>>>> Cheers, >>>>>> Claudius >>>>>> >>>>> >>>>> >>>> >>> >> > > -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de