From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6449958519840964608 X-Received: by 10.25.77.17 with SMTP id a17mr128906lfb.34.1510942502625; Fri, 17 Nov 2017 10:15:02 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.46.67.136 with SMTP id z8ls769180lje.2.gmail; Fri, 17 Nov 2017 10:15:01 -0800 (PST) X-Google-Smtp-Source: AGs4zMZ5DOYZ8JMC5o5YMj8SEuYiP4xAy66dniYPSKD7JGl4drcTZi6VN9qve28agXMMLzEv0R0O X-Received: by 10.46.5.147 with SMTP id 141mr147979ljf.44.1510942501676; Fri, 17 Nov 2017 10:15:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510942501; cv=none; d=google.com; s=arc-20160816; b=0CRzhBuJV5iVWoBGvkNF750jKslQZh03a7ImQN0R30gy683O4mTjmIQhZra/toxceR hvKPG1YJisIjLcrJRWOu3s72bIcxaBlBwMUSM9erDkU4Y3R2tgp1IHIxBtWjk47gBe4D HxM8SW45QVdUBBi/TDhLh+w2DsRYCLTfjAUeDcBMMeD3i2TffAiWjtd+8C4xhwwsiOGe MjcUC1eh9uGEDX+JaYcc2r0T2xfEALV8hLA5Fxy857GOJ1ecPocuDkpbqIThSvxEsgRS riW6vvaDPZc4ukMu6QOuQOUq04E7SPQnkXk7ZDicQxvIS1hPs3FSCCwQ5noK/UubcYTK 4ycw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:date:to:from:subject:message-id :arc-authentication-results; bh=PzvZmIk/JAeQqE+mQeIt5cY7pL/ggUf5EnjUGIDL7vk=; b=VG5ocuh1aY8tzvct6Al614qoq+rnJYKNqaOiJ3LJGMb+mDzkM6Dug7UZ94aCsoBUbs go5qKwCUVTkH78Vpu513B3JAAgLN/gze1ILISu4Lw7DcQmFMXhJjLdDo37BA2SxSEfPL 1agBysSpy1gtJAogUnVqeALhptH8PnuylvkMAM+cLKh3phJplsF9ijiERToyzhgOqAHQ UUOiWM0jNTbHJDVH5Id4Hf8X/2ycdhOf/TKHWVdFoLY7rZIt1LgIi3O505Jo3OoCYgIY 9MxVwzR0r3Rl+8aThqu10jATNFxs/MBnZLr0U3xc86UtLxaKC48E7W361SkW3xKJ4aUU CB+Q== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 212.18.0.9 is neither permitted nor denied by best guess record for domain of ch@denx.de) smtp.mailfrom=ch@denx.de Return-Path: Received: from mail-out.m-online.net (mail-out.m-online.net. [212.18.0.9]) by gmr-mx.google.com with ESMTPS id q27si327905lfd.0.2017.11.17.10.15.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 17 Nov 2017 10:15:01 -0800 (PST) Received-SPF: neutral (google.com: 212.18.0.9 is neither permitted nor denied by best guess record for domain of ch@denx.de) client-ip=212.18.0.9; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 212.18.0.9 is neither permitted nor denied by best guess record for domain of ch@denx.de) smtp.mailfrom=ch@denx.de Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 3ydmVY0SwTz1qsTY; Fri, 17 Nov 2017 19:15:01 +0100 (CET) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 3ydmVY0GlYz1wdNb; Fri, 17 Nov 2017 19:15:01 +0100 (CET) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id U_tqHCWXnmet; Fri, 17 Nov 2017 19:14:59 +0100 (CET) X-Auth-Info: 6nmVg0PAHOx3IrC/oJvwkmqPKMFfHfjEANAi0MMxrxA= Received: from Orrorin.lan (dslb-084-056-059-158.084.056.pools.vodafone-ip.de [84.56.59.158]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPSA; Fri, 17 Nov 2017 19:14:59 +0100 (CET) Message-ID: <1510942495.3306.107.camel@denx.de> Subject: Re: Reproducibility of builds From: Claudius Heine To: "[ext] Christian Storm" , isar-users@googlegroups.com Date: Fri, 17 Nov 2017 19:14:55 +0100 In-Reply-To: <20171117165358.zyl7jjsu3rxutyod@MD1KR9XC.ww002.siemens.net> References: <20171117165358.zyl7jjsu3rxutyod@MD1KR9XC.ww002.siemens.net> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-hjG83H5H3r8eQ7Xj2FjO" X-Mailer: Evolution 3.26.2 Mime-Version: 1.0 X-TUID: znjZTO+uNxDR --=-hjG83H5H3r8eQ7Xj2FjO Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, On Fri, 2017-11-17 at 17:53 +0100, [ext] Christian Storm wrote: > > > since I'm very interested in this feature, I'd like to resume > > > this > > > discussion and to eventually come to an agreed upon proposal on > > > how > > > to implement it. So, without further ado, here are my thoughts on > > > the subject: > > >=20 > > > Regardless of the concrete technical implementation, I guess we > > > can > > > agree on the need for a local cache/repository/store in which the > > > Debian > > > binary packages plus their sources have to be stored since one > > > may not > > > rely on the availability of those files online for eternity. > > >=20 > > > These files in this cache/repository/store are the union of the > > > Debian > > > binary packages installed in the resulting image plus their > > > sources as > > > well as those installed in the buildchroot plus their sources. > > > The latter is required to be able to rebuild Debian packages > > > built from > > > source with the same compiler version, libraries, -dev packages, > > > etc. pp. > > >=20 > > > Having the cache/repository/store at hand, there should be a > > > mechanism > > > to prime Isar with it, i.e., Isar should only and exclusively use > > > Debian > > > binary packages and sources from this cache/repository/store. > > > This is again, irrespective of the technical implementation, be > > > it via > > > a repository cache or other means like git, a proxy server or > > > whatsoever. > > >=20 > > > Granted, if one changes, e.g, IMAGE_INSTALL_append, the build > > > fails but > > > does so rightfully as the set of packages is modified, resulting > > > in a > > > new version/epoch (=3Dset of Debian packages plus their sources). > > > So, > > > there should be a convenient "interface" provided by Isar to > > > maintain > > > the cache/repository/store. For example, one may want to have > > > different > > > versions/epochs that may correspond to particular versions (git > > > sha) of > > > the Isar layer. Or one wants to later add a Debian package plus > > > its > > > source (which is automatically fetched), resulting in a new > > > version/epoch etc. > > >=20 > > > The remaining question is how to fill the cache/repository/store. > > > In > > > order to have a consistent version/epoch (=3Dset of Debian packages > > > plus > > > their sources), there should not be duplicate packages in it, > > > i.e., the > > > same Debian package but with different versions. > > > This could currently happen because there is a "window of > > > vulnerability": > > > multistrap is run twice, once for isar-image-base.bb and once for > > > buildchroot.bb. In between those two runs, the Debian mirror used > > > could > > > get updated, resulting in a different version of the Debian > > > package > > > being installed in buildchroot than in the resulting image. > > > This is an inherent problem of relying on the Debian way of > > > distributing > > > packages as one cannot a priori control what particular package > > > versions > > > one gets: In contrast to, e.g., Yocto where the particular > > > package > > > versions are specified in the recipes, this does not hold for > > > Isar as > > > the particular package versions are defined by the Debian mirror > > > used, > > > hence, one gets "injected" the particular package versions. > > > So, what's required to reduce the "window of vulnerability" and > > > to have > > > a consistent cache/repository/store for a particular > > > version/epoch is to > > > make a snapshot-type download of the required packages. For this, > > > of > > > course, one needs to know the concrete set of packages. This list > > > could > > > be delivered by a "package trace" Isar run since not only > > > multistrap > > > does install packages but sprinkled apt-get install commands do > > > as well. > > > Thereafter, knowing the list, the snapshot-type download can > > > happen, > > > hopefully resulting in a consistent cache/repository/store. > > >=20 > > >=20 > > > So, what do you think? > >=20 > > I agree with your formulation of the problem here. > >=20 > > Simple tracing of installed packages will have the problem you=20 > > described, that its possible that different versions of a package > > are=20 > > installed into buildchroot and image. So this trace needs to be > > cleaned=20 > > up and then based on that the whole process has to be started again > > to=20 > > create a consistent package list between buildchroot and image. > > This=20 > > doubles the build time in the trivial implementation. >=20 > Sure, there's no free lunch here :) > I'd rather strive for a good solution and avoid trivial > implementations > to make lunch as close to free as it gets, to stay in the picture. >=20 >=20 > > With my suggestion of using a caching proxy, this could be solved=20 > > without any additional overhead. >=20 > Could be the case, what are the drawbacks? More complexity and stuff to implement. Also maybe download speed. > What proxy do you propose to > use? I was at first going with my own standalone proxy implementation in pure stdlib python, so that it could be completely integrated into isar. I had a very simple solution ready rather quickly, but it was only synchronous and as such could only handle one connection at a time. Instead of just throwing more threads at it, I wanted to go the asyncio route. Sadly the python stdlib does not provide a http implementation for asyncio. I wasn't clear how to proceed from here further (aiohttp dependency or minimal own http implementation). The other idea is to just use a ready made apt caching proxy like apt- cache-ng. But here I am unsure if its flexible enough to use in our case. Starting it multiple times in parallel with different ports for different caches and only user privileges might be possible but I suspect that seperating the pool and the dists folder (pool should go to DL_DIR while dists is part of the TMP_DIR) could be more difficult. > Maybe I missed something on the proxy suggestion.. Could you > please elaborate on this? As for the integration the basic idea was that for taged bitbake tasks the proxy is started and sets the *_PROXY environment variables. This should be doable with some mods to the base.bbclass and some external python scripts. >=20 >=20 > > I do have other ideas to do this, but that would restructure most > > of isar. >=20 > Well, at least speaking for myself, I'd like to hear those as I > consider > this feature to be essential. Choice in solutions is always good :) >=20 One idea that I got when I first investigated isar, was trying to be oe compatible as much as possible. So using this idea would solve the reproducable builds as well: Basically implementing debootstrap with bitbake recipes that are created virtually on runtime by downloading and parsing the 'dists/*/*/*/Packages.gz' file. I suppose it should be possible to fetch the Packages file at an early parsing step in a bitbake build, if its not already preset, and fill the bitbake data store with recipe definitions that fetch those binary deb packages, have the appropriate dependencies and install them into the root file system. However, this idea is still in the brain storming phase. Since that would involve a very big redesign I don't think its feasible currently. Cheers, Claudius --=20 DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153 Keyserver: hkp://pool.sks-keyservers.net --=-hjG83H5H3r8eQ7Xj2FjO Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEb/LlnwDGvCgx2GTBEXPLGZgIsVMFAloPJx8ACgkQEXPLGZgI sVMiSRAAp4eDJFEsvget0xxrsQJu4tDUfLBVB+rnEZgTqiBYE3H2hH9T0ELXocSE 1YVRuiDgEIyXTQEY7DNDQxkfDJrG0JJnryjX4fi6vX/gw0EazEyQpxVbyQnwqi1i uJOACE6eq8PXukb8WdnvN1tgls9vcQn5cavckOYa8DU6+pOvkBr5pkDk+LsIb6P1 vp5HIPTLSw8aHBHbt99aKilSUvbNkz1xEH5KNrsOoUdOqtil2Fc2EtmxCjRkf8zU vAHxHRrSXyOa6UK4PtaKrzuVurKSU0qDE6mssHYlkZvXMcAA66R8GPbfqvnis9Qt ue2+xRDG6cWsUK3TPOQbOsWT2GH7DB1xcCttg95rKEBTgGWq7wDieyo5OJOoeXn4 MCnJAsh02h7hzfysrXIZUNbz5RYVDMLSedfQltU880UbOQeL1wDemp8JkjEfIGkG Vxk8Rwm1NyMqWOIFJ9YEU42bpsGbSZ70MSk2AhxHh0ZvQEvSpRfMtPD/Y7yK1sEm tDodvv60gN3Y2CyeJyLfNca/xiGwn2Ql21YhS9dvEflo2hF3QsJLljj+rmADjg2m sUjF9zAXE5IzYFp9g4m8P7YIz28PAJz1x+jlxPTLL6D7jJN3WAFHFjSGO6UnOqkU baVNut49rN0gXZs+OI3Q+XysSlCy3PSzdSrK/6D7a6LZkQLChnQ= =QvZq -----END PGP SIGNATURE----- --=-hjG83H5H3r8eQ7Xj2FjO--