From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6558372643829972992 X-Received: by 2002:a50:90fd:: with SMTP id d58-v6mr2118086eda.12.1527088823121; Wed, 23 May 2018 08:20:23 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a50:c9c2:: with SMTP id c2-v6ls10966054edi.4.gmail; Wed, 23 May 2018 08:20:22 -0700 (PDT) X-Google-Smtp-Source: AB8JxZqNbC2TPZmZSbBsVApMi6OAIYyXIs6FVa4SzNxsndZGAV/uWfU00BTNqXSBEhig/c87xbo5 X-Received: by 2002:a50:a8e3:: with SMTP id k90-v6mr543552edc.10.1527088822743; Wed, 23 May 2018 08:20:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1527088822; cv=none; d=google.com; s=arc-20160816; b=O3wRiSWBjU4SmXFEnHsx5Z3bopcrRFhT2D0Qof6oPAc/SRGLbniQU4Tw8rl6MlMKoe 7sQAEtc/n+7Ib6tCpmkWaVYL/HjrLbCDLFE0APupOMmJtDKZDcbYKd5MqrURZZHDeAij 3IShg6WBXLUKdMFaW0s55LE86Jr6V0j8vLeSQ9rMjDCJTyZPav9nYNTzreMAGgCoXhp+ zk3O+u+hCapLPp3T2TCqGRkeniFBOSmNHQAagFcUVQHZcSAn4tqd9+vJ8sb/B1OGd73u eDhfk4WPo1xSLl1qkcITshsjVqcsfQ7TWnj912VDM8y9uP8SGkEcrqUtdWnn0VEF5LA+ i7AQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:date:to:from:subject:message-id :arc-authentication-results; bh=11bQ7ratYr2lEdOc6sUEsA//gXSLJMhHEITob/xltl8=; b=CdzfFtHQKJnrS1nqu/1di8nh+0sXyQFtbSTCnwqyUbPAJUDrRnREg0YlAgx73CN376 5WkVgj6xiXNvIvy/wJj5o6akVN5SWsFAHdT3YM3GdBBO7EpbAp3ho3yBFuDqVSqmJzO1 QJyYN5MSOXmJDfBDzFMwiBlY71Fp3ZU59P7dgpuhzSt4bXuaUnwT2KmXuL6MgYdenVc+ sNv8q9d5P6FIzS5OKvirWqIatNCJe0g5BNXjCdLXz06fBr9HwKfYTDb64nrRED8pj+Kv r+PHnVUd6IKDPO6hmCv1xsVCg1gbT5GPq7WrJIGmmK13V+SkzKyaxgaJCNfoZTo0nk7R acMA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 2001:a60:0:28:0:1:25:1 is neither permitted nor denied by best guess record for domain of ch@denx.de) smtp.mailfrom=ch@denx.de Return-Path: Received: from mail-out.m-online.net (mail-out.m-online.net. [2001:a60:0:28:0:1:25:1]) by gmr-mx.google.com with ESMTPS id u9-v6si905664edp.3.2018.05.23.08.20.22 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 May 2018 08:20:22 -0700 (PDT) Received-SPF: neutral (google.com: 2001:a60:0:28:0:1:25:1 is neither permitted nor denied by best guess record for domain of ch@denx.de) client-ip=2001:a60:0:28:0:1:25:1; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 2001:a60:0:28:0:1:25:1 is neither permitted nor denied by best guess record for domain of ch@denx.de) smtp.mailfrom=ch@denx.de Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 40rbmk34LSz1qwHM; Wed, 23 May 2018 17:20:22 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 40rbmk2l85z1qrX4; Wed, 23 May 2018 17:20:22 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id jldIrkSRdLSi; Wed, 23 May 2018 17:20:19 +0200 (CEST) X-Auth-Info: SnVFxj9OlYM1f0N3eQWmLBLrtXFkpJilIH/Ad76dvn0= Received: from Orrorin (p578a821c.dip0.t-ipconnect.de [87.138.130.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPSA; Wed, 23 May 2018 17:20:19 +0200 (CEST) Message-ID: <183090ee660c27b2d5eec664da28da5ae6f77285.camel@denx.de> Subject: Re: [RFC PATCH 0/3] Reproducible build From: Claudius Heine To: "Maxim Yu. Osipov" , claudius.heine.ext@siemens.com, isar-users@googlegroups.com Date: Wed, 23 May 2018 17:20:11 +0200 In-Reply-To: References: <3467a5ec-182e-8c9a-cd19-7ad898323be7@siemens.com> <20180523063206.29180-1-claudius.heine.ext@siemens.com> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-Q6al08sMOmw2TRqRLx3H" X-Mailer: Evolution 3.28.2 Mime-Version: 1.0 X-TUID: SQqNcCdCXHcd --=-Q6al08sMOmw2TRqRLx3H Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Maxim. On Wed, 2018-05-23 at 16:30 +0200, Maxim Yu. Osipov wrote: > Hi Claudius, >=20 > I've looked through discussion thread. >=20 > As far as I understood with the proposed approach we don't have > the ability to reproduce this tarball - it contains some unversioned=20 > snapshot of isar-bootstrap rootfs, containing unversioned snapshot > of=20 > debian's packages cache used to create rootfs. It's fine if you just=20 > want to reproduce locally the current build from the scratch in your=20 > sandbox by avoiding debootstrap stage (fetching again packages, etc). >=20 > Do you have another use-case scenario in mind? >=20 > F.e. to share this tarball with other developers (linked to > particular=20 > version of isar tree) so they can fully reproduce the build? >=20 > If yes, how do you plan to version/manage such growing list of > tarballs?=20 > As it was mentioned in the discussion, upgrading one package from > debian=20 > repo will result to other tarball. My focus to tackle the reproducibility was two fold: 1. Output the build input in some form 2. Allow the build input to be used in the subsequent builds while allowing customization e.g. fixes to isar packages. I choose the form of the output to be just one tarball because its pretty simple to move it around. To backup it somewhere you can extract it if you like. Maybe put it into a OSTree repo or just create incremental tarballs as I described before. How this tarball is versioned, how and where it is stored was not in the scope of my work and belongs to the backup mechanism chosen by the users IMO. We can just point to the files that need backuping. That is how OE does it as well with the downloads directory. They just don't pack it together, but they also don't have to worry about a root fs with permissions. Now to the choice of the 'build input'. Of course it would be great to make the debootstrap step reproducible as well, but that means I have to create a repository with the packages. Creating a new repository from the packages in the cache means that later adding new packages, that were not part of the cache isn't possible, since they aren't part of the repository. My earlier suggestion (months ago) was using a manual controllable repository proxy. That could be in the form of a http webserver or proxy. This proxy would fetch and store the index and packages from upstream. That is serious implementation effort, however. apt_cacher_ng and aptly are missing important features, that would need to be implemented there. So we could try to interest those projects to this and propose patches. And maybe we should do that. Aplty looks really nice, but I don't know how they stand of creating partial repositories where the index contains entries that aren't available in the repo itself. So to solve this locally in isar was just using what debootstrap outputs + all the used packages from the cache as the base for the next build. I agree that this currently isn't the nicest solution possible, but its pretty simple to implement and could be expanded upon. If this isn't enough then I would need to look into aptly for example to create a more extensive solution. Claudius --=20 DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153 Keyserver: hkp://pool.sks-keyservers.net --=-Q6al08sMOmw2TRqRLx3H Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEb/LlnwDGvCgx2GTBEXPLGZgIsVMFAlsFhqsACgkQEXPLGZgI sVN8vg/+I/yPtx4S32iTOg35D63PB0R1akJdbk7Yb784Hf4ZYB1qfhpt40OtsWMC X8917HLNymcLTlFyJ7BgOUfS5wKTkePmWHL4+HnaTDrj1BRcjWex88r+ffCe1JER z7pbcVkkgyETTPn3tob5ILnNMUNMrgPdEXt0sX6HdgZfUWaJwhJKeD2FA1NU92qq bDLq643kg4xjz3moPmsjWrXumwn1OUFuV6r551dX9kg4iRiqwruXDuYx00lt2p07 zfJgHTT7PptJMMafb4xQQWy7mTFoBPDDghfxlj+K3AkGYaNuHnYOslqJSNpiy/Ju gWhSEgCGtUL3pK0OotYMTrJrI5R1jtg4wDoAiRmDnYt+ZCO5I3aVC+Z7/N7Zw6gF gwUDYU8y/QamcEpxRaRPC94cYAxcPgr0X7NzJPYLRjYwhT/EHAMzhe4zIGQuwIak HWpI6W1GpX085n9eBfiSmXup/imc3vPb5qboRP7lbKq3GnJ62cEUVOSpDHMdS3F4 vdR7Aa29cHXcgTegkY7nfZkN3AKmPVFibNx9oIBdgzM/M9WXHyl5eviQxlPfvX+H 9iZ1rBTmJNL0C73jKl2OnTftVeALK3/jZPoi4ZU82ZindZETftv9d4v5dMuuymKi G0TyAYHQG0Fo+b6VU46dESUfS9642Dp8/2Skf81vx2y9u34ogV4= =6Y7V -----END PGP SIGNATURE----- --=-Q6al08sMOmw2TRqRLx3H--