From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6509751388101148672 X-Received: by 10.223.162.136 with SMTP id s8mr3006041wra.11.1515774328265; Fri, 12 Jan 2018 08:25:28 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.223.195.143 with SMTP id p15ls1198420wrf.11.gmail; Fri, 12 Jan 2018 08:25:27 -0800 (PST) X-Google-Smtp-Source: ACJfBouOHSFcScQs8n+WGwfTeMWz4buJVtplwnqudPB78cCBm+Ut7STWqJTIP7CSRv2dZekC46Ep X-Received: by 10.28.91.79 with SMTP id p76mr216601wmb.19.1515774327767; Fri, 12 Jan 2018 08:25:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515774327; cv=none; d=google.com; s=arc-20160816; b=AWhX/R8hGQLRDMDAEmyNhgEpgd6w/mPjGuj0hWsHiExav/1N+VC3Y16JVo3b4yetXg 5w6xdC9GEt/zpGNyH/Hy37ZuFVvMZuJfA8qBzEkWyM8AYCepUj9fQn73q0kN8BZVCS+B 6tSXb5kuJBTxbQ+tObB9qlwiqnd5uxJISXs0ORY5gytKKZIdJqkz2Y4fjOA7rupj5cJs QUthpDQ5YsXLqXJS1xq56BC0TK8EBaQJN/X9L1vkwULRwKGMPNn956NoFfETwNrQN8Qj cyNVglwrZN/O7F1qZ6N+AOyFz5F+PzpX7mPDp0G2rWCIoIaE7Vn8NQWL2CmkkE2HUqoU wwJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:arc-authentication-results; bh=zwJ2dMio3+7cHRZYVEYwK9oiZcXp9hb/4PD84Vy+aWQ=; b=DLiWIrrxqkxbI48b/0MKmKbItOzo1bwt4cQfgcu1xtG44bYsz1a/kP/6sigRlX+gog NHmWGVPRupwqSu9ImQ9QFXWrTg3WMWuI0pdOPD02gDGvtOL5sY5LcgEw+lQM6I/HS2+V 7HSHuMVRZVAThvJy8dqfNt7Z3K/vLk3O7lIhWujK80z75TF4sW7D2jOFIsVurHsjFj+i WqVCXDlVOdIUZz1lxxz9cJf28iDr5VwoL1yLYGtLpy8j6X6Og3R+6VSAFxMQ2IefSt8m uqoshylvHJ8FDcMhkZhzL8FkxWhoJ76bfMfufw6CKJN0ljkRXNLXcqBWmmMk7BZGiYkX LZQg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id s81si456842wmd.2.2018.01.12.08.25.27 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 12 Jan 2018 08:25:27 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Received: from mail3.siemens.de (mail3.siemens.de [139.25.208.14]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id w0CGPRZG003226 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Jan 2018 17:25:27 +0100 Received: from mmd1pvb1c.ad001.siemens.net (md1pvb1c.ad001.siemens.net [139.25.68.40] (may be forged)) by mail3.siemens.de (8.15.2/8.15.2) with ESMTP id w0CGPR9A009155; Fri, 12 Jan 2018 17:25:27 +0100 Date: Fri, 12 Jan 2018 17:25:26 +0100 From: Henning Schild To: Alexander Smirnov Cc: Subject: Re: [RFC v2][PATCH 2/3] build-rep: Add helper class Message-ID: <20180112172526.19bd23a2@mmd1pvb1c.ad001.siemens.net> In-Reply-To: References: <20180111111939.25667-1-asmirnov@ilbers.de> <20180111111939.25667-3-asmirnov@ilbers.de> <20180112133205.2d2d6615@mmd1pvb1c.ad001.siemens.net> X-Mailer: Claws Mail 3.15.0-dirty (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: ocloUaeHLI7y Am Fri, 12 Jan 2018 16:29:20 +0300 schrieb Alexander Smirnov : > On 01/12/2018 03:32 PM, Henning Schild wrote: > > Am Thu, 11 Jan 2018 14:19:38 +0300 > > schrieb Alexander Smirnov : > > > >> Add class that helps to implement build reproducibility. It > >> implements anonymous function that will get all the Debian > >> dependencies that are needed for current Isar tree. > >> > >> Until build reproducibility will be fully implemented, it's > >> disabled by default. To enable it just set ISAR_BUILD_REP to "1" > >> in local.conf. > >> > >> Signed-off-by: Alexander Smirnov > >> --- > >> meta-isar/conf/local.conf.sample | 6 +++++ > >> meta-isar/recipes-app/hello/hello.bb | 2 ++ > >> meta/classes/build-rep.bbclass | 32 > >> ++++++++++++++++++++++++ > >> meta/classes/dpkg-base.bbclass | 2 ++ > >> meta/classes/image.bbclass | 2 ++ > >> meta/recipes-devtools/buildchroot/buildchroot.bb | 2 ++ 6 files > >> changed, 46 insertions(+) create mode 100644 > >> meta/classes/build-rep.bbclass > >> > >> diff --git a/meta-isar/conf/local.conf.sample > >> b/meta-isar/conf/local.conf.sample index 660958f..45b8995 100644 > >> --- a/meta-isar/conf/local.conf.sample > >> +++ b/meta-isar/conf/local.conf.sample > >> @@ -162,3 +162,9 @@ BB_NUMBER_THREADS = "4" > >> # > >> # Number of attempts to try to get reprepro lock for access to > >> apt cache REPREPRO_LOCK_ATTEMPTS = "16" > >> + > >> +# Isar build reproducibility feature creates local repository > >> which contains +# copies for all the upstream Debian packages that > >> could be used to build +# your Isar tree. So fetching them once > >> will guarantee that all the next Isar +# builds will be > >> identically. +ISAR_BUILD_REP ?= "0" > >> diff --git a/meta-isar/recipes-app/hello/hello.bb > >> b/meta-isar/recipes-app/hello/hello.bb index 44b8bc3..fafda2e > >> 100644 --- a/meta-isar/recipes-app/hello/hello.bb > >> +++ b/meta-isar/recipes-app/hello/hello.bb > >> @@ -16,3 +16,5 @@ SRCREV = > >> "ad7065ecc4840cc436bfcdac427386dbba4ea719" SRC_DIR = "git" > >> > >> inherit dpkg > >> + > >> +DEBIAN_DEPENDS = "debhelper (>= 9), autotools-dev" > > > > If i understand it correctly, this approach is an absolute NoGo. > > > > DEBIAN_DEPENDS as it is used today are runtime deps of pre-built > > packages. Here you include build-deps of a package that Isar needs > > to build. And you do so by introducing a copy of a string that is > > already included in the sources /debian/-folder. In fact you would > > have to put build and runtime deps into that one variable. And what > > about packages where the content depends on DISTRO_ARCH and friends? > > > > I think we first need a working solution of two "inherit dpkg" > > packages actually depending on each-other. After that we will need > > a real build of the Image to fill the cache. That is the only > > reliable way to make sure that the repo will contain the runtime > > and the build deps of all packages. More "sed"-guesswork is not > > going to do that trick and is only getting us 95% of what we need. > > And maintaining copies of stuff that is in /debian/ should not be > > considered. > > Let's look at this realistically: > > 1. The upstream Debian should be fetched via single operation. If you > want to populate the cache during image build - you can't guarantee > that all the packages were built in the same environment. As > Christoph wrote, at any time the upstream apt could be updated. So > the buildchroot and image filesystems should be generated using local > static apt, otherwise it makes no sense to claim reproducibility. If the versions change on the way we should end up with two versions of one package in the cache. When we build from that again we will not get what we got the first time because image and buildchroot will probably take the latest version, while in the first run they got different versions. I think that is fine and we could detect that and declare the first build as "failed because of upstream change". You will just have to do that over and over again until you find a 30min window where the mirror did not change, i suspect you will hardly ever hit the case where the mirror changed between the multistraps. > 2. Regarding 'debian/control' duplications. According to the idea, > that 'base-apt' should be generated first of all, in bitbake terms it > means that: INHO that idea is wrong, it should be generated as a side-product of an actual build. If you want to get this right you will end up reimplementing Debian internals and keep running after them. That does not sound like a promising way to go. > *.bb:do_build() should depends on base-apt:do_build(). > > On the other hand, base-apt:do_build() should already know the full > list of all deps for *.bb. So for now I see two ways how to implement > this: > > - Duplicate 'debian/control' in recipe and share this variable with > base-apt (like it's done in this series). > > - Manually parse 'debian/control'. But in this case base-apt should > depend on *.bb:do_unpack. Due to bitbake is 'statically configured' > tool, base-apt should know the list of all the recipes, what is also > impossible. In previous series I've tried to implement this by > defining a list of images in some global variable. "Manuall parse" is a NoGo, just look at the multiple broken versions of perl code guessing the build-deps ... emulating something that debian itself just gets right. > 2. Question how to get the list of build and runtime dependencies is > still open, I've already asked about this in previous mail. So, the > goal is to have the full list of packages that should be fetched. > 'debian/control' contains very specific format, so I can't feed this > line to multistrap/apt-get as it is. I see the following solutions: > - Implement custom tool to do this (apt/dpkg uses standard perl > libs which contains all this stuff). > - Manually parse 'debian/control' > - Drop the requirement about single upstream fetch. And again the infamous "manual parse" and even a "custom tool" ... No. The repo should come out of what the multiple "multistraps" and "apt-get installs" actually fetch when they are allowed to fetch from the outside. If you refer to that with "Drop the requirement about ..." i fully agree. Henning > Alex > > > > > Henning > > > >> diff --git a/meta/classes/build-rep.bbclass > >> b/meta/classes/build-rep.bbclass new file mode 100644 > >> index 0000000..ede5a93 > >> --- /dev/null > >> +++ b/meta/classes/build-rep.bbclass > >> @@ -0,0 +1,32 @@ > >> +# This software is a part of ISAR. > >> +# Copyright (C) 2017 Siemens AG > >> + > >> +python __anonymous() { > >> + rep = d.getVar('ISAR_BUILD_REP', True) or "0" > >> + if rep == "0": > >> + return > >> + > >> + depsdir = d.getVar('BASE_APT_DIR', True) > >> + if depsdir is None: > >> + return > >> + > >> + depsdir += '/deps/' > >> + > >> + pn = d.getVar('PN', True) > >> + > >> + if not os.path.exists(depsdir): > >> + os.makedirs(depsdir, exist_ok=True) > >> + > >> + if d.getVar('DEBIAN_DEPENDS', True): > >> + with open(depsdir + '/' + pn + '.depends', 'w') as > >> the_file: > >> + the_file.write(d.getVar('DEBIAN_DEPENDS', True)) > >> + the_file.close > >> + elif d.getVar('BUILDCHROOT_PREINSTALL', True): > >> + with open(depsdir + '/' + pn + '.preinst', 'w') as > >> the_file: > >> + the_file.write(d.getVar('BUILDCHROOT_PREINSTALL', > >> True)) > >> + the_file.close > >> + elif d.getVar('IMAGE_PREINSTALL', True): > >> + with open(depsdir + '/' + pn + '.preinst', 'w') as > >> the_file: > >> + the_file.write(d.getVar('IMAGE_PREINSTALL', True)) > >> + the_file.close > >> +} > >> diff --git a/meta/classes/dpkg-base.bbclass > >> b/meta/classes/dpkg-base.bbclass index 4d220da..bf66e78 100644 > >> --- a/meta/classes/dpkg-base.bbclass > >> +++ b/meta/classes/dpkg-base.bbclass > >> @@ -3,6 +3,8 @@ > >> > >> DEBIAN_DEPENDS ?= "" > >> > >> +inherit build-rep > >> + > >> # Add dependency from buildchroot creation > >> do_build[depends] = "buildchroot:do_build" > >> > >> diff --git a/meta/classes/image.bbclass > >> b/meta/classes/image.bbclass index e2cb01b..67f5af8 100644 > >> --- a/meta/classes/image.bbclass > >> +++ b/meta/classes/image.bbclass > >> @@ -5,6 +5,8 @@ IMAGE_INSTALL ?= "" > >> IMAGE_TYPE ?= "ext4-img" > >> IMAGE_ROOTFS = "${WORKDIR}/rootfs" > >> > >> +inherit build-rep > >> + > >> def get_image_name(d, name_link): > >> S = d.getVar("IMAGE_ROOTFS", True) > >> path_link = os.path.join(S, name_link) > >> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb > >> b/meta/recipes-devtools/buildchroot/buildchroot.bb index > >> 51f9d5d..da18231 100644 --- > >> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++ > >> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -16,6 +16,8 > >> @@ SRC_URI = "file://multistrap.conf.in \ file://build.sh" > >> PV = "1.0" > >> > >> +inherit build-rep > >> + > >> BUILDCHROOT_PREINSTALL ?= "gcc \ > >> make \ > >> build-essential \ > > >