From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6506467811408216064 X-Received: by 10.28.135.82 with SMTP id j79mr3855235wmd.7.1514912345743; Tue, 02 Jan 2018 08:59:05 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.216.197 with SMTP id p188ls1574795wmg.6.gmail; Tue, 02 Jan 2018 08:59:05 -0800 (PST) X-Google-Smtp-Source: ACJfBoshxRjcpyquOFApJ8qp44MOUlH6nGJOjwp4IsVD4rnYQ5SoBrRvgZ9G5mcGhmOiAKpFXEhA X-Received: by 10.28.216.18 with SMTP id p18mr4263155wmg.21.1514912345329; Tue, 02 Jan 2018 08:59:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514912345; cv=none; d=google.com; s=arc-20160816; b=dtDizrpGYBUu/c0ssNRHuPhnIO3A/kfZriVVq/i93vGI08FhNm2etYIcYqIPeXCadZ EHeWpbpLV7agTqs0fapBWZ94NqUCutV2BsHfiKvPfuEULTHRfmKg/2OhOH0f6ntUM37H xTPohnnhldoatAMSpHuDzI/ZUv12o5MY911x8hBCDNFlJMy4PCR88isUgE74cL6eNtLi ICUhTIjqQvYSFwFlVdGzzdfpavKhWVmUpvkVmuMFpKRnNEa2qp0Mn0IWgDQH6SuQWTCL 1keuhYqma4yrGKK2y7N/3O8e8NYl/P7aAOGg7Zsqc3M0Df3aiz3uwb8RBI99AHdM8T2+ dZxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=ubZ0p1Xh6wlUdTnfZJXikm9pinK/DOsQ/CtE8fvrCUU=; b=OaykgrK/+z9cdyFqTZW9smU1csLqp6jOn9P1thA/kjcceWymaUft3AkUP+QNvAFSJH 1DAvIvAUHN9nLK6hUw8vNXp2MqW93E6NZhW81VBC+o2sn+KbMW/KRQZdoJI+TZgAtOCF /crHI+cz+UtS//KvDOjdCBnnsnjkfJcaYQ6OJjq3jCQUOzdRpbZ2q0ORowQVZ3+BzPQJ f0p7FNJ0HWuPWTfNOcLbb1izEmvkwhI/U8ibSKInvls8w01p8Xg9LOdyRiwThaGRgCaD +/5G2pPbWrdTKjJF3ufK00J6xNZVlsZ5fpVY9JwjLKxTqwMVxc8caHK964OPFhEsT2xq 5niA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id x5si10347wrb.0.2018.01.02.08.59.05 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 08:59:05 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w02Gx1vi017103 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 2 Jan 2018 17:59:03 +0100 Subject: Re: [RFC][PATCH 0/6] Isar build reproducibility To: Jan Kiszka , isar-users@googlegroups.com References: <20180102145744.21814-1-asmirnov@ilbers.de> From: Alexander Smirnov Message-ID: <349d8da8-bdfe-633e-0ee7-06b2de68adba@ilbers.de> Date: Tue, 2 Jan 2018 19:58:56 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: x/VR/s7d1eFM Hi Jan, On 01/02/2018 07:09 PM, Jan Kiszka wrote: > Hi Alex, > > On 2018-01-02 15:57, Alexander Smirnov wrote: >> Hello all, >> >> this series proposes the way how build reproducibility could be implemented >> in Isar. General idea is to get the list of all the necessary packages for >> build, fetch them and create local repo, that will be used for further >> builds/ >> >> Briefly speeking, it works like the following: >> >> 1. User sets the list of images that should be 'reproducible' in >> BASE_APT_IMAGES variable in local.conf file. > > What speaks against making every image reproducible? I strongly assume > this is the common case. There should be a way to disable that, and that > could be local.conf controlled. I assume, that there could be lots of overlays used for building custom product. So I'm not sure if it makes sense to include all the image recipes from all the overlays. For example, in Yocto, there is image sato, which provides graphical UI and adds lots of tasks to build pipeline. In my experience I don't remember anybody who used it in commercial products, so for they "sato" reproducibility will be excessive. > > Is there a technical reason the core needs to know the name of the > reproducible image? Or can be control this on a per-build basis, > including all images of that build? Yes, unfortunately it's a technical reason, at least I haven't found any other way for now. Bitbake uses "waterfall one direction" dependencies processing. Roughly speaking, if you imagine all the deps as tree with head on the top, bitbake could derive only deps in the bottom for each node in the tree. isar-image-base / \ hello example-raw \ / buildchroot | base-apt So, if your recipe, for example 'hello.bb' is building, you can't get within it the information who exactly added him to build pipeline, what is the final target/image bitbake is building etc... That's why I need to know the top nodes of the dependency tree to get the list of all the packages being used. > > And will all images of the same bitbake run (thinking of multiconfig...) > share also the same base-apt repo, or is that per image? > >> >> 2. Based on the list of images above, Isar will derive all the run-time and >> build dependencies for these images. >> >> 3. Using multistrap, Isar will fetch the list of packages and create base-apt >> local repository. >> >> 4. Now buildchroot and image root filesystems are generated using base-apt. >> >> Some notes: >> >> 1. base-apt repository is mounted to buildchroot, so Isar packages are >> able to install necessary deps via apt-get. > > This does not your provide the packages a way to feed their own build > results into the same repo, does it? I'm referring to the issue that > Christian described > (https://groups.google.com/forum/#!msg/isar-users/efer3RF989o/r-zfUNNDAgAJ) > and which I ran into as well meanwhile. The idea of base-apt - to be a partial mirror of upstream Debian. In my understanding, for all the build results Isar apt should be used. >> >> 2. bitbake events are used to clean up buildchroot. I haven't found another >> way how base-apt could be unmounted. So it's mounted once before any >> package starts building and unmounted by bitbake event: >> bb.event.BuildCompleted. > > What is the implication of that? At least to me (without detailed > understanding of bitbake), this remains unclear. No implications. Bitbake has various events that are generated at certain points of operations: https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#events These events for example are used in Yocto to generate build statistics, in toaster UI. The key problem here is to implement the following model: - buildchroot:do_prepare (mount /proc, /dev, base-apt) - build custom packages - buildchroot:do_cleanup (unmount everything) I could add dependency from "buildchroot:do_prepare" to any package being build, it's not a problem. But I can't add dependency from "buildchroot:do_cleanup". This task should be executed when all the packages are built, but the list of packages being build is generated dynamically, while bitbake supports only static deps. One way could be add "do_cleanup" to image recipe, because it has build dependency from all the recipes, but this will work only if you build image. The command: $ bitbake example-raw will not call "do_cleanup" in this case. Alex > >> >> This series works good with latest next. Any comments are welcome. >> >> Happy New Year 2018! :-) > > Same to you. > > Jan >