From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6506467811408216064 X-Received: by 10.223.152.182 with SMTP id w51mr745267wrb.10.1515483961601; Mon, 08 Jan 2018 23:46:01 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.223.156.130 with SMTP id d2ls552329wre.13.gmail; Mon, 08 Jan 2018 23:46:01 -0800 (PST) X-Google-Smtp-Source: ACJfBovGgqeQN+VkToEt3lyiWK6r4q9Vss15vkogblV+z1uYaGYvPv9DWvIdaFt/dz/TP6F0FAqb X-Received: by 10.28.106.18 with SMTP id f18mr1511459wmc.30.1515483961211; Mon, 08 Jan 2018 23:46:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515483961; cv=none; d=google.com; s=arc-20160816; b=SN866+2kiKxsSU2IGI1n3R8ZFjHj/zKy8iW0yacCydTHPqeYRSYrcdRe9dLvIUlFi1 D7nAz+MFBMDVTGAzDJdYmAtL66gC45p11GmbQLgQijUsaZ+rSvC3sb7kuyD8Ah9QIOBK KC7FWXZ1ka9tYbxiIuNt4dAxe+LUbYSpjEEH5tUe5vR7k9jjhTNp3TG6gDcVU5jGVYpi RmFeQaLDMvoRYpBGR9mjcngMBVXAa7F2NjRK1o4NJwyzrU/jFW1Stp8L/mjWwTF4fS+U PsuFTYtRXCy0BvJMGd6Gn8yIlj7QxYUDl4I4Xr/VPt7ekRlpgR+t+rUSzKMyjLiUT0Ki aZgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=lJGYahMCiG6LH2OuARb42dSc2M2BxXPPPzdL84AAX6s=; b=JraO6Woaq4K1gwpWCVNgj/Z169D0vUb2KfYPDk5G+UVXZsa2QxwZXANInfv38C7V6x O4SDuSuLK86mjFDO8Xm9di2pqlFgoySGAtUAc5O44mJk3YwW8geqbqfLKgf/s4aOBbPI HhvNJ5YJWTypvsGckr6aCl9qL6WVWMTzkVpD6i5etpogu30vrHxcxhdafzfIjJ1wKjxi 1Kjo0VSwEISdfmPWmju4wst2LagWF5PhQTqdzHsXkG6fIwJyj7a7iITsbVK+fSAzZY+B 8DS6gXogwwL+AUL43jE/eft9OQrbVoc63T4pNeArGPfH6f1Q+7/WG8T7hAWLxIOUgexa /CbQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id h76si1455079wmd.1.2018.01.08.23.46.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 08 Jan 2018 23:46:01 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w097jvcg010452 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 9 Jan 2018 08:45:59 +0100 Subject: Re: [RFC][PATCH 0/6] Isar build reproducibility To: Henning Schild Cc: isar-users@googlegroups.com References: <20180102145744.21814-1-asmirnov@ilbers.de> <20180103144945.017b062a@mmd1pvb1c.ad001.siemens.net> From: Alexander Smirnov Message-ID: <7953b59a-6055-03a7-3917-b85c19f37cd8@ilbers.de> Date: Tue, 9 Jan 2018 10:45:52 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <20180103144945.017b062a@mmd1pvb1c.ad001.siemens.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: Y8z85ZlOkTnV Hi all, On 01/03/2018 04:49 PM, Henning Schild wrote: > Am Tue, 2 Jan 2018 17:57:38 +0300 > schrieb Alexander Smirnov : > >> Hello all, >> >> this series proposes the way how build reproducibility could be >> implemented in Isar. General idea is to get the list of all the >> necessary packages for build, fetch them and create local repo, that >> will be used for further builds/ >> >> Briefly speeking, it works like the following: >> >> 1. User sets the list of images that should be 'reproducible' in >> BASE_APT_IMAGES variable in local.conf file. > > I am with Jan here, i would prefer an all or nothing approach. > Still thinking if it's possible to implement this with bitbake. The key problem is that Isar doesn't have static list of Debian dependencies, i.e. it needs to fetch source code and parse 'debian/conrtol' to get list of required packages to install. So some kind of the following chain should be implemented: 1. Identify image recipes. 2. Get list of packages in IMAGE_INSTALL for all images. 3. Make do_fetch/do_unpack and parse 'debian/control' for packages in the list above. In this case base-apt recipe should depend from *all* the available images. Anonymous tasks can't help to derive dependencies, because package's deps could be derived after 'do_unpack' only. In theory, we could define some bitbake variable like: DEBIAN_DEPENDENCY that contains the same data as in 'debian/control'. Moreover it could make sense to generate 'debian/control' on the fly, like Henning does it for 'example-raw'. Having variable DEBIAN_DEPENDENCY makes possible to get list of packages that are possibly used in whole current Isar tree by using anonymous task in bbclass. Opinions? :-) >> 2. Based on the list of images above, Isar will derive all the >> run-time and build dependencies for these images. >> >> 3. Using multistrap, Isar will fetch the list of packages and create >> base-apt local repository. >> >> 4. Now buildchroot and image root filesystems are generated using >> base-apt. > > Patch5 makes base-apt and Isar the only repos for a rootfs/buildchroot. > How do images, not using BASE_APT, get packages that are not cached? > In my opinion, in case of base-apt, image should not use upstream anymore, otherwise build reproducibility can't be guaranteed. apt will try to fetch the freshest package. So if new package appears in the build, the base-apt should be updated/regenerated. Alex >> Some notes: >> >> 1. base-apt repository is mounted to buildchroot, so Isar packages are >> able to install necessary deps via apt-get. >> >> 2. bitbake events are used to clean up buildchroot. I haven't found >> another way how base-apt could be unmounted. So it's mounted once >> before any package starts building and unmounted by bitbake event: >> bb.event.BuildCompleted. >> >> This series works good with latest next. Any comments are welcome. >> >> Happy New Year 2018! :-) > > Same from me! > > Henning > >> With best regards, >> Alex >> >> Alexander Smirnov (6): >> base-apt: Introduce fetching upstream apt >> base-apt: Add to pipeline >> buildchroot: Switch to base-apt >> buildchroot: Add mount/umount for 'base-apt' >> image: Switch to base-apt >> base-apt: Add possibility to reuse >> >> meta-isar/conf/local.conf.sample | 8 ++ >> .../recipes-core/images/files/multistrap.conf.in | 18 +--- >> meta-isar/recipes-core/images/isar-image-base.bb | 5 +- >> meta/classes/dpkg-base.bbclass | 16 +++- >> meta/classes/image.bbclass | 13 ++- >> meta/classes/isar-events.bbclass | 21 +++++ >> meta/conf/isar-bitbake.conf | 2 + >> meta/recipes-devtools/base-apt/base-apt.bb | 97 >> ++++++++++++++++++++++ .../base-apt/files/distributions.in >> | 3 + .../base-apt/files/multistrap.conf.in | 28 +++++++ >> meta/recipes-devtools/buildchroot/buildchroot.bb | 28 ++++++- >> .../buildchroot/files/configscript.sh | 1 - >> .../buildchroot/files/multistrap.conf.in | 18 +--- >> 13 files changed, 217 insertions(+), 41 deletions(-) >> create mode 100644 meta/classes/isar-events.bbclass >> create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb >> create mode 100644 >> meta/recipes-devtools/base-apt/files/distributions.in create mode >> 100644 meta/recipes-devtools/base-apt/files/multistrap.conf.in >> >