From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6509751388101148672 X-Received: by 10.25.18.85 with SMTP id h82mr1073681lfi.4.1515763770881; Fri, 12 Jan 2018 05:29:30 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.25.94.19 with SMTP id s19ls74642lfb.0.gmail; Fri, 12 Jan 2018 05:29:30 -0800 (PST) X-Google-Smtp-Source: ACJfBouia+RclZ1yVi4TECc43OOe41pTvKjCUFp3+tclPewVDLUAUJPlRuewg3wI5uak90jyFSjd X-Received: by 10.25.207.75 with SMTP id f72mr185768lfg.28.1515763770067; Fri, 12 Jan 2018 05:29:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515763770; cv=none; d=google.com; s=arc-20160816; b=crPPbcHmSS+Nldq2Kytp+1AkdWWICV43lt4WhLWMe6dtdEOcqdK8GCW3FKOuNe3Xfy clwRCyB0cx0kHDhmIPrQTXMMY82lG09/0CET7JHFAwGVWiYEbMK6zQbZLr6tHSE/qUXz 3+dY34zB0/IfMZUN+bMOk684Of2FyRfa7YdIcFkRSAjdjk0vmiWWquCg3yJE1Q14PAwq SkqCgP5zGAhpr36G2FddTuezSmVFm/6xQUizWitRBfBP3RRh19OqLcheLDfluVgA9734 AWYh9ihebd4JZcmZSOAxgruhDLwePtullqc53itiGMZ++wlvf/KbOMl113w5V2WwDknH OMgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=yGi2EaUnDySCfGtI7elyv1JxqSDN7NBoTwKXaZvUMZg=; b=zWtOMcGzQYxK9+6NXLcIUrNVCtHL8I5I7zUldiu3Oe50WDK6Y8+EXzQhj6x2Nc0jj6 4DPi4vGRCO6AzXSFtCendO5Xr/sHnyqXocxlmApbGGhTtsrFVs1SdY4GjH0k7KSSO8Ox iIeC1reeXiLx2v+H94DdfXmtSOwNeyh6cjVPrQVvmiLB4uFOROE1yESIlLIyWSWbUQmt AaR3UhpQFVVcJdCrX/RhRjC4yiNrJfGvzZmvXpp0g5c3qNyShy5i5hmLitJW/uGfsOuM 4srEbZ5LeYzPpyXPHASYiTxfsc1Dh1me9FzKPSRD8NVrxwVZ2vEN1oko27f25Ahey0+y T+LQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id a26si344552ljb.3.2018.01.12.05.29.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 12 Jan 2018 05:29:29 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w0CDTP4G029456 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 12 Jan 2018 14:29:27 +0100 Subject: Re: [RFC v2][PATCH 2/3] build-rep: Add helper class To: Henning Schild Cc: isar-users@googlegroups.com References: <20180111111939.25667-1-asmirnov@ilbers.de> <20180111111939.25667-3-asmirnov@ilbers.de> <20180112133205.2d2d6615@mmd1pvb1c.ad001.siemens.net> From: Alexander Smirnov Message-ID: Date: Fri, 12 Jan 2018 16:29:20 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <20180112133205.2d2d6615@mmd1pvb1c.ad001.siemens.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: 7w/H2YY5UGn+ On 01/12/2018 03:32 PM, Henning Schild wrote: > Am Thu, 11 Jan 2018 14:19:38 +0300 > schrieb Alexander Smirnov : > >> Add class that helps to implement build reproducibility. It implements >> anonymous function that will get all the Debian dependencies that are >> needed for current Isar tree. >> >> Until build reproducibility will be fully implemented, it's disabled >> by default. To enable it just set ISAR_BUILD_REP to "1" in local.conf. >> >> Signed-off-by: Alexander Smirnov >> --- >> meta-isar/conf/local.conf.sample | 6 +++++ >> meta-isar/recipes-app/hello/hello.bb | 2 ++ >> meta/classes/build-rep.bbclass | 32 >> ++++++++++++++++++++++++ >> meta/classes/dpkg-base.bbclass | 2 ++ >> meta/classes/image.bbclass | 2 ++ >> meta/recipes-devtools/buildchroot/buildchroot.bb | 2 ++ 6 files >> changed, 46 insertions(+) create mode 100644 >> meta/classes/build-rep.bbclass >> >> diff --git a/meta-isar/conf/local.conf.sample >> b/meta-isar/conf/local.conf.sample index 660958f..45b8995 100644 >> --- a/meta-isar/conf/local.conf.sample >> +++ b/meta-isar/conf/local.conf.sample >> @@ -162,3 +162,9 @@ BB_NUMBER_THREADS = "4" >> # >> # Number of attempts to try to get reprepro lock for access to apt >> cache REPREPRO_LOCK_ATTEMPTS = "16" >> + >> +# Isar build reproducibility feature creates local repository which >> contains +# copies for all the upstream Debian packages that could be >> used to build +# your Isar tree. So fetching them once will guarantee >> that all the next Isar +# builds will be identically. >> +ISAR_BUILD_REP ?= "0" >> diff --git a/meta-isar/recipes-app/hello/hello.bb >> b/meta-isar/recipes-app/hello/hello.bb index 44b8bc3..fafda2e 100644 >> --- a/meta-isar/recipes-app/hello/hello.bb >> +++ b/meta-isar/recipes-app/hello/hello.bb >> @@ -16,3 +16,5 @@ SRCREV = "ad7065ecc4840cc436bfcdac427386dbba4ea719" >> SRC_DIR = "git" >> >> inherit dpkg >> + >> +DEBIAN_DEPENDS = "debhelper (>= 9), autotools-dev" > > If i understand it correctly, this approach is an absolute NoGo. > > DEBIAN_DEPENDS as it is used today are runtime deps of pre-built > packages. Here you include build-deps of a package that Isar needs to > build. And you do so by introducing a copy of a string that is already > included in the sources /debian/-folder. In fact you would have to > put build and runtime deps into that one variable. And what about > packages where the content depends on DISTRO_ARCH and friends? > > I think we first need a working solution of two "inherit dpkg" packages > actually depending on each-other. After that we will need a real build > of the Image to fill the cache. That is the only reliable way to make > sure that the repo will contain the runtime and the build deps of all > packages. More "sed"-guesswork is not going to do that trick and is > only getting us 95% of what we need. And maintaining copies of stuff > that is in /debian/ should not be considered. Let's look at this realistically: 1. The upstream Debian should be fetched via single operation. If you want to populate the cache during image build - you can't guarantee that all the packages were built in the same environment. As Christoph wrote, at any time the upstream apt could be updated. So the buildchroot and image filesystems should be generated using local static apt, otherwise it makes no sense to claim reproducibility. 2. Regarding 'debian/control' duplications. According to the idea, that 'base-apt' should be generated first of all, in bitbake terms it means that: *.bb:do_build() should depends on base-apt:do_build(). On the other hand, base-apt:do_build() should already know the full list of all deps for *.bb. So for now I see two ways how to implement this: - Duplicate 'debian/control' in recipe and share this variable with base-apt (like it's done in this series). - Manually parse 'debian/control'. But in this case base-apt should depend on *.bb:do_unpack. Due to bitbake is 'statically configured' tool, base-apt should know the list of all the recipes, what is also impossible. In previous series I've tried to implement this by defining a list of images in some global variable. 2. Question how to get the list of build and runtime dependencies is still open, I've already asked about this in previous mail. So, the goal is to have the full list of packages that should be fetched. 'debian/control' contains very specific format, so I can't feed this line to multistrap/apt-get as it is. I see the following solutions: - Implement custom tool to do this (apt/dpkg uses standard perl libs which contains all this stuff). - Manually parse 'debian/control' - Drop the requirement about single upstream fetch. Alex > > Henning > >> diff --git a/meta/classes/build-rep.bbclass >> b/meta/classes/build-rep.bbclass new file mode 100644 >> index 0000000..ede5a93 >> --- /dev/null >> +++ b/meta/classes/build-rep.bbclass >> @@ -0,0 +1,32 @@ >> +# This software is a part of ISAR. >> +# Copyright (C) 2017 Siemens AG >> + >> +python __anonymous() { >> + rep = d.getVar('ISAR_BUILD_REP', True) or "0" >> + if rep == "0": >> + return >> + >> + depsdir = d.getVar('BASE_APT_DIR', True) >> + if depsdir is None: >> + return >> + >> + depsdir += '/deps/' >> + >> + pn = d.getVar('PN', True) >> + >> + if not os.path.exists(depsdir): >> + os.makedirs(depsdir, exist_ok=True) >> + >> + if d.getVar('DEBIAN_DEPENDS', True): >> + with open(depsdir + '/' + pn + '.depends', 'w') as the_file: >> + the_file.write(d.getVar('DEBIAN_DEPENDS', True)) >> + the_file.close >> + elif d.getVar('BUILDCHROOT_PREINSTALL', True): >> + with open(depsdir + '/' + pn + '.preinst', 'w') as the_file: >> + the_file.write(d.getVar('BUILDCHROOT_PREINSTALL', True)) >> + the_file.close >> + elif d.getVar('IMAGE_PREINSTALL', True): >> + with open(depsdir + '/' + pn + '.preinst', 'w') as the_file: >> + the_file.write(d.getVar('IMAGE_PREINSTALL', True)) >> + the_file.close >> +} >> diff --git a/meta/classes/dpkg-base.bbclass >> b/meta/classes/dpkg-base.bbclass index 4d220da..bf66e78 100644 >> --- a/meta/classes/dpkg-base.bbclass >> +++ b/meta/classes/dpkg-base.bbclass >> @@ -3,6 +3,8 @@ >> >> DEBIAN_DEPENDS ?= "" >> >> +inherit build-rep >> + >> # Add dependency from buildchroot creation >> do_build[depends] = "buildchroot:do_build" >> >> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >> index e2cb01b..67f5af8 100644 >> --- a/meta/classes/image.bbclass >> +++ b/meta/classes/image.bbclass >> @@ -5,6 +5,8 @@ IMAGE_INSTALL ?= "" >> IMAGE_TYPE ?= "ext4-img" >> IMAGE_ROOTFS = "${WORKDIR}/rootfs" >> >> +inherit build-rep >> + >> def get_image_name(d, name_link): >> S = d.getVar("IMAGE_ROOTFS", True) >> path_link = os.path.join(S, name_link) >> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb >> b/meta/recipes-devtools/buildchroot/buildchroot.bb index >> 51f9d5d..da18231 100644 --- >> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++ >> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -16,6 +16,8 @@ >> SRC_URI = "file://multistrap.conf.in \ file://build.sh" >> PV = "1.0" >> >> +inherit build-rep >> + >> BUILDCHROOT_PREINSTALL ?= "gcc \ >> make \ >> build-essential \ > -- With best regards, Alexander Smirnov ilbers GmbH Baierbrunner Str. 28c D-81379 Munich +49 (89) 122 67 24-0 http://ilbers.de/ Commercial register Munich, HRB 214197 General manager: Baurzhan Ismagulov