From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6449958519840964608 X-Received: by 10.25.150.148 with SMTP id y142mr209193lfd.33.1512034326782; Thu, 30 Nov 2017 01:32:06 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.46.5.3 with SMTP id 3ls2365119ljf.13.gmail; Thu, 30 Nov 2017 01:32:06 -0800 (PST) X-Google-Smtp-Source: AGs4zMYE/emulPYmBDz04RGMUIsO94O4wRvinapG+jIxpz278uQnkMJIkk1u26BPfSjyHA8DNIYU X-Received: by 10.46.101.14 with SMTP id z14mr241343ljb.26.1512034326430; Thu, 30 Nov 2017 01:32:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1512034326; cv=none; d=google.com; s=arc-20160816; b=nu1b3vKjcJUyRyMEe9+yeVAKJ5HosMn8w00TZ9zu+L75ep18YSQKJzIUfUEJSVL27y uyGv5wi7WZ3W06EjPxDow4m6IVXLqNYWwYTZsP2iCp9UxPwwW7xnh3T+hes6lRrDSoHC W7IZv7zbNMmSYTxdOh9R6mUje9ApPy2osXWY+pxmb2PQV6aePCneYeRQ+ao1Zfvj8BFG uiDBCLu9FnCi9RbY66EFWGkXwSPqAK6clXssoXxz4Ox55r7uhg67eb6swqkefSeYzim5 xgDIJm6RRj51dAqysAEmP/+8N8hqP0rqgDqyAx75ER1Hdp2fACb9+bXwM7onvB6C545o 2x9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:mime-version:user-agent:date:message-id:openpgp:from :references:to:subject:arc-authentication-results; bh=V49HZfxtxN6KXs6aplCvyda7dQYMaGG1J3zlTcUUmds=; b=xpLAOATY1j0aAJQQRFBF7U+QqePOimdcjHSHt++GFyGCklesWiFXe7jejB4j8NfPSg bAZC5KodgYci8VYn2h2/4dPqBjxtdDY7lGDKARnsvGmBz1hjL9HhnsLMs2aTTiRyk+xW t6UlcWSxekEFPL7VQyvZDutI8tP0nba1gZRSr/Sim/5HlOzz8j+++lLuOqHPUsHRcAp/ WGTPezmJsbX44r8LmudwixcxUs+XAVZq5tx5Mf2nSq9INQ2or1Hg+0lopUEra+07bNTt sWsw64q7y2B5XjVxPuQsWLd5BYlCOny8Xb9IdDm3ZIBLCz+g5Du9MPbRMwa0BuEo76+R n4BQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 212.18.0.10 is neither permitted nor denied by best guess record for domain of ch@denx.de) smtp.mailfrom=ch@denx.de Return-Path: Received: from mail-out.m-online.net (mail-out.m-online.net. [212.18.0.10]) by gmr-mx.google.com with ESMTPS id 64si138701ljz.5.2017.11.30.01.32.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Nov 2017 01:32:06 -0800 (PST) Received-SPF: neutral (google.com: 212.18.0.10 is neither permitted nor denied by best guess record for domain of ch@denx.de) client-ip=212.18.0.10; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 212.18.0.10 is neither permitted nor denied by best guess record for domain of ch@denx.de) smtp.mailfrom=ch@denx.de Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 3ynXH954jDz1qvyy; Thu, 30 Nov 2017 10:32:05 +0100 (CET) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 3ynXH94VlQz1qqlD; Thu, 30 Nov 2017 10:32:05 +0100 (CET) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id lgdkaCM_Lif0; Thu, 30 Nov 2017 10:32:04 +0100 (CET) X-Auth-Info: 4zlbHwAQLdoPzE+WoEE4kXlJs4/sAUCNixagxyfuRG0= Received: from deneb.denx.de (p578adb1c.dip0.t-ipconnect.de [87.138.219.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPSA; Thu, 30 Nov 2017 10:32:04 +0100 (CET) Subject: Re: Reproducibility of builds To: Alexander Smirnov , isar-users@googlegroups.com References: <20171120083322.mz7uiz7nbgesf2qp@MD1KR9XC.ww002.siemens.net> <0508e061-441f-396b-98df-ab6226aa04cd@denx.de> From: Claudius Heine Openpgp: id=6FF2E59F00C6BC2831D864C11173CB199808B153; url=http://pool.sks-keyservers.net/pks/lookup?op=get&search=0x1173CB199808B153 Message-ID: Date: Thu, 30 Nov 2017 10:31:59 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="HBrxooro00vA9xBjbAi0ONlW7HLkilL9A" X-TUID: 9QAERudsg2K9 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --HBrxooro00vA9xBjbAi0ONlW7HLkilL9A Content-Type: multipart/mixed; boundary="8UhQTHuTWW4PQapfWth8Kp13opAGQvAKu"; protected-headers="v1" From: Claudius Heine To: Alexander Smirnov , isar-users@googlegroups.com Message-ID: Subject: Re: Reproducibility of builds References: <20171120083322.mz7uiz7nbgesf2qp@MD1KR9XC.ww002.siemens.net> <0508e061-441f-396b-98df-ab6226aa04cd@denx.de> In-Reply-To: --8UhQTHuTWW4PQapfWth8Kp13opAGQvAKu Content-Type: text/plain; charset=utf-8 Content-Language: en-MW Content-Transfer-Encoding: quoted-printable Hi Alex, On 11/29/2017 07:53 PM, Alexander Smirnov wrote: > Hi everybody, >=20 > I've started working on this topic and here I'd like to share my vision= =2E > At the moment I've implemented simple PoC in my branch > 'asmirnov/build_rep'. >=20 > What it does: >=20 > 1. There is new recipe: base-apt. It provides task which: >=20 > =C2=A0- Fetches packages from origin Debian apt to local folder using > deboostrap. > =C2=A0- Put these packages via 'reprepro' to local repository called 'b= ase-apt'. >=20 > 2. Buildchroot uses 'base-apt' to generate rootfs. >=20 > 3. Isar image uses 'base-apt' and 'isar' repos to generate rootfs. >=20 >=20 >=20 > What are the key benefits of this approach: >=20 > 1. Download session for upstream packages is performed in a single step= =2E >=20 > 2. You could use your local 'versioned' apt repository instead of > downloading origin packages. >=20 > 3. Having local apt repository managed by 'reprepro' provides us > possibility to implement version pinning. Reprepro provides lots of > things like: > =C2=A0- Get package name. > =C2=A0- Get package version. > =C2=A0- Remove specific package from repo. > =C2=A0- Add single package to repo. >=20 > So in general, if we have know which package version we want to have, w= e > need to get binary with this version and put it to 'base-apt'. >=20 >=20 >=20 > Which issues I see at the moment: >=20 > 1. The key issue for me the list of packages for 'base-apt'. So before > 'base-apt' task is executed, we should prepare full list of packages > that will be used by: > =C2=A0- buildchroot (BUILDCHROOT_PREINSTALL). > =C2=A0- packages to build (their build deps). > =C2=A0- image (IMAGE_PREINSTALL). Maybe try to do this flexible, because it should be also possible for example to generate lxc images that are deployed to the final target in the same isar run. Also as Jan said, deduplication of packages so maybe try to fetch those packages in the DL_DIR first, so that rebuilding is possible without internet access, no tmp_dir, a populated DL_DIR and a package+version list of sorts. Then have a task that copies those packages into the tmp_dir into a repo within the tmp_dir and install from there. This way the DL_DIR would only contain one instance of every packages and the repo in the tmp_dir only have a copy (or maybe even just a symlink). Archiving the DL_DIR would in this case be enough to build different sets of images. If you can solve this, than this solution looks promising. Cheers, Claudius --=20 DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153 Keyserver: hkp://pool.sks-keyservers.net --8UhQTHuTWW4PQapfWth8Kp13opAGQvAKu-- --HBrxooro00vA9xBjbAi0ONlW7HLkilL9A Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEb/LlnwDGvCgx2GTBEXPLGZgIsVMFAlof0BAACgkQEXPLGZgI sVPKCQ//e8NKmtAzGjukR8wMO+MEuHKDUG5xvNT75ejNYEpkr/E1bY3dXzIgRQHE WOSWKo7B3zzkhSmzA/jjXKECLq8GzEfQynO6AIbhnOUaME6EFQLUOXmZgwRRQnyT Y+FpGubDHM7s2ylqduMqc7q7G9CqkfMX9wytjVd/yEl8DxnW0+SehlpGXoasWrb+ VcvomRKq+7RDbv8uMWm7bhK70hH2NP1/huzXfuIFsT1Np3t68NBjDC3khtnCT8Kt CLsS7Yz7QKqLuyBkc3m6x8xHJd+0th1m6I4XHXlvpxjpXGfR2ugey+2xwH8ouZld 240hlVlAyJo3ehCzXAv70EBdCrCWq1pFGCrEehrZZOHqigeHxnO6CbeG6TEl5Kjf hoA7mwpP0HIeLA13BxAhAb2Xd4sqcB1OwWfPQKy2vUt/g5YSut26ZkliP1TzV77W d5BX+ZqTncsnvD+0RwfLLBLvCH5F8FIfM5R2G1jDF9kppCGvS1qd+Xzpy6MeZi5K vWx97FAZ8KexuHVhja99ExjeWRrfwUM8p+59+Egu3mupD1PKzBqnhJk/ZmnBV19W PSSSwdYB1Qf0PhK+0Xbw9doq8gQ2pR3UnXantLaaRY5KBdkU01RUR9zTznaf9Ak3 zidbobJO8QSp1iBENNJBLnEkMhShq+32dVSi0jDr+zgpcfUqBg0= =xuHd -----END PGP SIGNATURE----- --HBrxooro00vA9xBjbAi0ONlW7HLkilL9A--