From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6558372643829972992 X-Received: by 2002:a1c:ec42:: with SMTP id k63-v6mr334660wmh.29.1527028353384; Tue, 22 May 2018 15:32:33 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:ea0e:: with SMTP id i14-v6ls2572384wmh.4.gmail; Tue, 22 May 2018 15:32:32 -0700 (PDT) X-Google-Smtp-Source: AB8JxZr6ZXcUjAg80J6xemB84Rt5z+sTCluoh0bUtFYXA3zS/VCtW3XGf3smDPD40Dt65n/3baP3 X-Received: by 2002:a1c:f109:: with SMTP id p9-v6mr347881wmh.11.1527028352896; Tue, 22 May 2018 15:32:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1527028352; cv=none; d=google.com; s=arc-20160816; b=d9m/8DGuTaOfXUoBMqVj00xfCGL2GncrX1OCB0rItYq9HdWuw8HHocqplwVkMWBMLn 6/ykJRhOBVzrbWXz4AsJx7PThFH9CBsWeF/e1QUVMPHgCMgQz9l//ZCee/319M67qO5a e4aVIQNULCxT+9FpgLu3iOqIgAxoJIQpklmVl6mkI8N4gXfyqYhbaqq7RKi0NuNw6Vpa g8L+UFyQCNWlNBVrBE24gHQLHxQqI6yAL1qoxJLTy90DmOISBWLcKYfnggaeboehWNcG UCNlhivAb8Dl4eItwLplEVQuGFnPwxM2cC8CmQ8iN4j3syxDuYLldSPQPYqMMXTr2bX8 z3cg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:to:from:date :arc-authentication-results; bh=QjlDLAjbpMGvYXAId+T4T9iI1pHoGXQ8c8AkwRnNhfA=; b=CsEmGeCLnxqpmGJW2o4ZO9Ihci9EzAti5ZG8MgjIrBAVdghIDmOcupIrVwBoamxhWX BFU+Beda+5mubPZmS10fhdvKIaNKP11U8AXjcXRRMERj55ioHOcPHmkZidRU+yFZcMxu zBEotUbgwVuoa5Y7PTFAl/7GCmi7G7mzP0Ur55v2oHYDqvSabRlsnP89cUcSwwnw92qF DQAofCO37ABngdrV6/IA37ZAf3IvDJhsajUAjmEjdY9nWzdy/xk68wKvUZ3EgPndUps1 pBFAKYsARvatweML7YGrE9ZAviAsc1lufvuMNeuc+xA4DB5AHo/pGu77fbEuiODA4fYz 3kiQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 85.214.62.211 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id m11-v6si692966wrn.3.2018.05.22.15.32.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 22 May 2018 15:32:32 -0700 (PDT) Received-SPF: neutral (google.com: 85.214.62.211 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 85.214.62.211 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Received: from yssyq.radix50.net (dslb-002-207-018-151.002.207.pools.vodafone-ip.de [2.207.18.151]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w4MMWUqa005148 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 23 May 2018 00:32:31 +0200 Received: from yssyq.radix50.net (localhost [127.0.0.1]) by yssyq.radix50.net (8.14.4/8.14.4/Debian-8) with ESMTP id w4MMWO0l016304 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 23 May 2018 00:32:24 +0200 Received: (from ibr@localhost) by yssyq.radix50.net (8.14.4/8.14.4/Submit) id w4MMWOv9016303 for isar-users@googlegroups.com; Wed, 23 May 2018 00:32:24 +0200 Date: Wed, 23 May 2018 00:32:24 +0200 From: Baurzhan Ismagulov To: isar-users Subject: Re: Idea for implementing reproducible builds Message-ID: <20180522223224.GE5882@yssyq.radix50.net> Mail-Followup-To: isar-users References: <3467a5ec-182e-8c9a-cd19-7ad898323be7@siemens.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3467a5ec-182e-8c9a-cd19-7ad898323be7@siemens.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-TUID: uDcvytBhD4II Hello Claudius, On Tue, May 22, 2018 at 01:55:21PM +0200, Claudius Heine wrote: > I am still working on reproducible builds and here is my current idea to > solve this. > > Simple put: Mount the /var/cache/apt/archives of the images and buildchroot > to the isar-bootstrap root file system and then create a tarball of it. This > way we have a tarball of the build just after debootstrap + upgrade with the > one 'apt update' step done, but without any other changes to it and all used > packages already in the apt package cache. > > When restoring just skip most of the isar-bootstrap steps and extract the > tarball instead, since the packages are available in the package cache and > the package index is not updated it will use the packages from the cache. > > This way we would side step the obstacle to make debootstrap reproducible by > just using its product while the reset of the process can be redone by isar. Thanks for sharing. As I understand it: 1. The user runs bitbake isar-image-base, which 1. Debootstraps a rootfs 2. Tars it 3. Unpacks the tar into buildchroot/rootfs and isar-image-base/rootfs 2. The user adds the tarball to the product repo Is this correct? In this scenario: * Step 1: How does bitbake decide whether to debootstrap or use the tarball? * Step 2: If I have the following repo, where should the tar file be located and versioned? myrepo - meta - meta-isar - product1 - product2 * If two products built from one repo have non-identical rootfses, what does the tarball contain? * What is the user supposed to do if he wants to update the tar to the current upstream, fully or in part? Considering our existing use cases, I'd suggest a couple of changes to your concept. Let's abbreviate our copy of Debian artifacts as "debian-mirror" (be it in form of a tarball or anything else). I see the following use cases: U1. debian-mirror doesn't exist. Create debian-mirror from upstream. U2.1. debian-mirror is versioned, e.g. in git. U2.2. Use debian-mirror for buildchroot/rootfs and isar-image-base/rootfs. U2.3. Don't use upstream for building buildchroot/rootfs and isar-image-base/rootfs. U3.1. debian-mirror exists. Update all packages from upstream into debian-mirror. U3.2. debian-mirror exists. Update chosen packages from upstream into debian-mirror. E.g., openssl, optionally its dependencies, optionally its dependents. U3.3. debian-mirror exists and is used by two products. One product has to be updated. The other one will be updated later. For product 1, update chosen or all packages from upstream to debian-mirror. Product 2 should still use the old packages. U3.4. Remove packages not used in any previous commit. Given those, I'd suggest using debs as versioned entities instead of the rootfs tarball. Create an apt repo with dpkg-scanpackages and dpkg-scansources and use it to debootstrap buildchroot and isar-image-base. This would address U2.3 and U3.3. This has been tested in practice, works well, and is in my opinion the best way to solve the problem. With versioned tarballs, an update of a single package would make the whole tarball change. This makes the history unreadable, wastes disk space, and many tools (including git) have problems with big files. >>From the UX perspective, I'd prefer to separate building images from preparing debian-mirror if possible. A separate command / task / bitbake run with a var set / unset, etc. E.g., bitbake -C createmirror isar-image-base, bitbake -C updatemirror isar-image-base, etc. Please include user documentation when you provide patches. I think that in the end, we'll have to providing specialized infrastructure for managing debian-mirror(s). It may or may not be based on anything from https://wiki.debian.org/DebianRepository/Setup?action=show&redirect=HowToSetupADebianRepository . With kind regards, Baurzhan.