From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6506467811408216064 X-Received: by 10.28.1.194 with SMTP id 185mr4294668wmb.12.1514912847190; Tue, 02 Jan 2018 09:07:27 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.223.156.134 with SMTP id d6ls9360113wre.10.gmail; Tue, 02 Jan 2018 09:07:26 -0800 (PST) X-Google-Smtp-Source: ACJfBotF7GHBppWvty2XK3rVtPKQr1pj7LbGZvjy+vvFpsbdfcshJkcGs0lj0Z69D87+3VYdLNNP X-Received: by 10.223.169.166 with SMTP id b35mr2066320wrd.16.1514912846881; Tue, 02 Jan 2018 09:07:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514912846; cv=none; d=google.com; s=arc-20160816; b=Bq7xBqdKA3AP6fa1WtzkQvtQMz7uFRTMvJun5V2RoeRIGMZWO5eZXOVY46LyYG1B60 s5CHCpqur/f+lLn6vd88xfaXsWeXEjAxXwq3pL/clXKMDN6ZhzdpAF5U5eVNYBWPqCBT FlCHgHft0PHFsKdJzxe6G+Rkc/S8UaVTYeA/vBAKuTLByi2ucVmofIxWDO6scpcFPw9Y 63Ioyfn9Q7+WY2BUlGzsDbqHHN8DvRHGT+TUgTlXoS8ivGXZkBPQW84wFvESpWD8vznD YDdSOYrsbWsV7fosT8uNdN2bU05fmLE+k/VgpieJ7pwNoVcEMi6g+pY1ibzsmGbafuu4 1Qvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=zRyf1HIosnbNHTOo/G817h94BgEE9s4fDF60V5Mj5Q4=; b=yoBsazCPQIO1IclrxhTP1qIDe88oSz0dGwTY4nbWDFqOtJ/2oUrclDUXu+c5dFC3IZ SaGVvO8X/EnsD3Jxnb+vOjv/TmTjR3t8BJY0Rh9LvQI1NX2rArej00FwVXiwPHZqeyB4 qDlfmGTpSQt3Be6Lm90SpsSrnjBo8wcxuOw5H6Xmrsqi9OEkvItXdANa9RLbU5piJQks mWuOKlN7s9bGWFIrK3Ync2dZag/jxCdiq3y7zDyTh7W+oh7FfcrGgul8GDfNJ9ACR3Vi 8I98MAAXCLSGxtQ5sTshOBhq8QVw+BAbiLcjcxb3PHrHv54JFG1QpvSpy88nw/yoeEAU Zr0w== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id m3si187201wrm.5.2018.01.02.09.07.26 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 09:07:26 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail2.siemens.de (mail2.siemens.de [139.25.208.11]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id w02H7Qx5003955 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Jan 2018 18:07:26 +0100 Received: from md1f2u6c.ww002.siemens.net (md1q0hnc.ad001.siemens.net [139.25.68.37] (may be forged)) by mail2.siemens.de (8.15.2/8.15.2) with ESMTP id w02H7QXQ030120; Tue, 2 Jan 2018 18:07:26 +0100 Subject: Re: [RFC][PATCH 0/6] Isar build reproducibility To: Alexander Smirnov , isar-users@googlegroups.com References: <20180102145744.21814-1-asmirnov@ilbers.de> <349d8da8-bdfe-633e-0ee7-06b2de68adba@ilbers.de> From: Jan Kiszka Message-ID: Date: Tue, 2 Jan 2018 18:07:26 +0100 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <349d8da8-bdfe-633e-0ee7-06b2de68adba@ilbers.de> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: IcQUug34Z4uB On 2018-01-02 17:58, Alexander Smirnov wrote: > Hi Jan, > > On 01/02/2018 07:09 PM, Jan Kiszka wrote: >> Hi Alex, >> >> On 2018-01-02 15:57, Alexander Smirnov wrote: >>> Hello all, >>> >>> this series proposes the way how build reproducibility could be >>> implemented >>> in Isar. General idea is to get the list of all the necessary >>> packages for >>> build, fetch them and create local repo, that will be used for further >>> builds/ >>> >>> Briefly speeking, it works like the following: >>> >>> 1. User sets the list of images that should be 'reproducible' in >>>     BASE_APT_IMAGES variable in local.conf file. >> >> What speaks against making every image reproducible? I strongly assume >> this is the common case. There should be a way to disable that, and that >> could be local.conf controlled. > > I assume, that there could be lots of overlays used for building custom > product. So I'm not sure if it makes sense to include all the image > recipes from all the overlays. > > For example, in Yocto, there is image sato, which provides graphical UI > and adds lots of tasks to build pipeline. In my experience I don't > remember anybody who used it in commercial products, so for they "sato" > reproducibility will be excessive. What are the costs again? Please specify, at latest when you provide some (user) documentation for this new feature. > >> >> Is there a technical reason the core needs to know the name of the >> reproducible image? Or can be control this on a per-build basis, >> including all images of that build? > > Yes, unfortunately it's a technical reason, at least I haven't found any > other way for now. Bitbake uses "waterfall one direction" dependencies > processing. Roughly speaking, if you imagine all the deps as tree with > head on the top, bitbake could derive only deps in the bottom for each > node in the tree. > >              isar-image-base >                /       \ >             hello    example-raw >               \        / >              buildchroot >                  | >               base-apt > > > So, if your recipe, for example 'hello.bb' is building, you can't get > within it the information who exactly added him to build pipeline, what > is the final target/image bitbake is building etc... > > That's why I need to know the top nodes of the dependency tree to get > the list of all the packages being used. The top nodes are what is provided to bitbake as targets, no? Can't we access that information? > >> >> And will all images of the same bitbake run (thinking of multiconfig...) >> share also the same base-apt repo, or is that per image? >> >>> >>> 2. Based on the list of images above, Isar will derive all the >>> run-time and >>>     build dependencies for these images. >>> >>> 3. Using multistrap, Isar will fetch the list of packages and create >>> base-apt >>>     local repository. >>> >>> 4. Now buildchroot and image root filesystems are generated using >>> base-apt. >>> >>> Some notes: >>> >>> 1. base-apt repository is mounted to buildchroot, so Isar packages are >>>     able to install necessary deps via apt-get. >> >> This does not your provide the packages a way to feed their own build >> results into the same repo, does it? I'm referring to the issue that >> Christian described >> (https://groups.google.com/forum/#!msg/isar-users/efer3RF989o/r-zfUNNDAgAJ) >> >> and which I ran into as well meanwhile. > > The idea of base-apt - to be a partial mirror of upstream Debian. In my > understanding, for all the build results Isar apt should be used. So, recipes would then deploy their debs into base-apt, making them immediately available to dependent recipes? > >>> >>> 2. bitbake events are used to clean up buildchroot. I haven't found >>> another >>>     way how base-apt could be unmounted. So it's mounted once before any >>>     package starts building and unmounted by bitbake event: >>>     bb.event.BuildCompleted. >> >> What is the implication of that? At least to me (without detailed >> understanding of bitbake), this remains unclear. > > No implications. Bitbake has various events that are generated at > certain points of operations: > > https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#events > > > These events for example are used in Yocto to generate build statistics, > in toaster UI. > > The key problem here is to implement the following model: > >  - buildchroot:do_prepare (mount /proc, /dev, base-apt) >  - build custom packages >  - buildchroot:do_cleanup (unmount everything) > > I could add dependency from "buildchroot:do_prepare" to any package > being build, it's not a problem. > > But I can't add dependency from "buildchroot:do_cleanup". This task > should be executed when all the packages are built, but the list of > packages being build is generated dynamically, while bitbake supports > only static deps. > > One way could be add "do_cleanup" to image recipe, because it has build > dependency from all the recipes, but this will work only if you build > image. The command: > >  $ bitbake example-raw > > will not call "do_cleanup" in this case. Is there no "bitbake is exiting" event that you could hook up to? Then the cleanups would happen at the very end of everything. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux