From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6449958519840964608 X-Received: by 10.25.198.195 with SMTP id w186mr338464lff.10.1511169383993; Mon, 20 Nov 2017 01:16:23 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.25.44.204 with SMTP id s195ls183173lfs.11.gmail; Mon, 20 Nov 2017 01:16:23 -0800 (PST) X-Google-Smtp-Source: AGs4zMYUb76le7Kq6jwQn0E8xkFUHUdK5iB7ObleCf6XkqhlYPSgWM3BateRVj/imTv/wwr/7Wm8 X-Received: by 10.46.99.14 with SMTP id x14mr447246ljb.12.1511169383470; Mon, 20 Nov 2017 01:16:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511169383; cv=none; d=google.com; s=arc-20160816; b=NFEaIKzTtlUtwh21Sy/hPOtYilGe6v7doD16FNy1kdN67Xi7L5KzKbOBttxMJ6pE2F tf2jQtWQogOHcq/GK9EH1yRCKrf9a/bMo7KW6a6laO/c+uvQzk+UA2jIV/RP3HEAskgs 9O43qqaxIPiSbo8MqKJ57kIpx0/V0+LegG02uL/qP7g6DfNc0ftyKJg1l+2pHFaucdHN ENCrtbrdUJS5awlBw/n6WYLdZcBKvdHjjKoetshtsF0up/VwpMk7nl25Ii0nGJdODlnK wxLEDFiyQo1s8I+0+OAZ5ZIgj53fdMqzs2xyrx8e09looq01DgG2XQy/Nc0xBoHE6WCw z/kQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:mime-version:user-agent:date:message-id:openpgp:from :references:to:subject:arc-authentication-results; bh=0XSYVcyQwhsbISvx5Wrn/7OjQ/UE16XFVh/NLXdJGrM=; b=jdTWCJNXMRDUIcNWb3TEQLZfeEhdtjrC3xL0b+MSAomSxWZMkcRoY6LUxfJsl977ET vkRNvxygRMtR9y5m5znrYbJS6tswPyUTiOsl6i/NyBjhfqpiucTowlxVGTA8Ms4mKtjw HIQV7NKzXA36AJHDCWni1QvCGH969wMF84jfD8ne4lCOZAqvTLsJqsjU6ud8LGvnQpEy X8wSXPStmEGjvOQW+FUdAb2jkaEHQE8QtODsBtcVJ5jTh0cI7wfV1cNNmYQaqEtFInPO WvJQaeEkIsUgF0wjISG4E6E521xOuQXDno5FSqdjyEQ2aQpou20cG8o1TFVxiqI4JocS GsGw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 212.18.0.9 is neither permitted nor denied by best guess record for domain of ch@denx.de) smtp.mailfrom=ch@denx.de Return-Path: Received: from mail-out.m-online.net (mail-out.m-online.net. [212.18.0.9]) by gmr-mx.google.com with ESMTPS id o75si605818lfc.2.2017.11.20.01.16.22 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 20 Nov 2017 01:16:23 -0800 (PST) Received-SPF: neutral (google.com: 212.18.0.9 is neither permitted nor denied by best guess record for domain of ch@denx.de) client-ip=212.18.0.9; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 212.18.0.9 is neither permitted nor denied by best guess record for domain of ch@denx.de) smtp.mailfrom=ch@denx.de Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 3ygNPf4DQyz1qs0x for ; Mon, 20 Nov 2017 10:16:22 +0100 (CET) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 3ygNPf3sHJz1qqkP for ; Mon, 20 Nov 2017 10:16:22 +0100 (CET) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id cR7ZJsQMDPqc for ; Mon, 20 Nov 2017 10:16:20 +0100 (CET) X-Auth-Info: Z2t6dmZgIbLLVrLjxFDRvva4/g9HS++hl+PuQunPYtc= Received: from leda.denx.de (p578adb1c.dip0.t-ipconnect.de [87.138.219.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPSA for ; Mon, 20 Nov 2017 10:16:20 +0100 (CET) Subject: Re: Reproducibility of builds To: isar-users@googlegroups.com References: <20171120083322.mz7uiz7nbgesf2qp@MD1KR9XC.ww002.siemens.net> From: Claudius Heine Openpgp: id=6FF2E59F00C6BC2831D864C11173CB199808B153; url=http://pool.sks-keyservers.net/pks/lookup?op=get&search=0x1173CB199808B153 Message-ID: <0508e061-441f-396b-98df-ab6226aa04cd@denx.de> Date: Mon, 20 Nov 2017 10:16:15 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <20171120083322.mz7uiz7nbgesf2qp@MD1KR9XC.ww002.siemens.net> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="udGDQ1f5dR399BcNJBVUCjL3oSuJCmca8" X-TUID: 0v6LZDHNs0LS This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --udGDQ1f5dR399BcNJBVUCjL3oSuJCmca8 Content-Type: multipart/mixed; boundary="7XLLa5p4U2FpB5DJob5BtGPDbVQ8Crjcr"; protected-headers="v1" From: Claudius Heine To: isar-users@googlegroups.com Message-ID: <0508e061-441f-396b-98df-ab6226aa04cd@denx.de> Subject: Re: Reproducibility of builds References: <20171120083322.mz7uiz7nbgesf2qp@MD1KR9XC.ww002.siemens.net> In-Reply-To: <20171120083322.mz7uiz7nbgesf2qp@MD1KR9XC.ww002.siemens.net> --7XLLa5p4U2FpB5DJob5BtGPDbVQ8Crjcr Content-Type: text/plain; charset=utf-8 Content-Language: en-MW Content-Transfer-Encoding: quoted-printable Hi Christian, On 20.11.2017 09:33, [ext] Christian Storm wrote: >>> [...] >>>> With my suggestion of using a caching proxy, this could be solved=20 >>>> without any additional overhead. >>> >>> Could be the case, what are the drawbacks? >> >> More complexity and stuff to implement. Also maybe download speed. >> >>> What proxy do you propose to use? >> >> I was at first going with my own standalone proxy implementation in >> pure stdlib python, so that it could be completely integrated into >> isar.=20 >=20 > Why not hooking this into the fetcher(s) so that it's integrated rather= > than a standalone thing? The bitbake fetcher is not the only step that downloads stuff in isar. There is also multistrap and possible 'apt-get install' calls within a chroot environment. I was going to integrate it into isar at some point, but first I wanted to have a working proof of concept without bitbake in between to be easily testable. Then integrate it tightly into isar later. > As a bonus, you'll have full control on this > from the Isar core/code. I think the main invention here is the code > that does the consistent version/epoch guarantee anyway... Hmm... My hope is that this will be solved by itself, by splitting 'dists' and 'pool'. >=20 >=20 >> I had a very simple solution ready rather quickly, but it was >> only synchronous and as such could only handle one connection at a >> time. Instead of just throwing more threads at it, I wanted to go the >> asyncio route. Sadly the python stdlib does not provide a http >> implementation for asyncio. I wasn't clear how to proceed from here >> further (aiohttp dependency or minimal own http implementation). >=20 > Ah, OK. Wouldn't this account for premature optimization? :) Handling more than one connection in parallel should be possible IMO. Going from one to two is harder then from two to n (n>2). So I was lucky, in a sense, to discover at that early point in implementation that this is harder to do than expected. >> The other idea is to just use a ready made apt caching proxy like apt-= >> cache-ng. But here I am unsure if its flexible enough to use in our >> case. Starting it multiple times in parallel with different ports for >> different caches and only user privileges might be possible but I >> suspect that seperating the pool and the dists folder (pool should go >> to DL_DIR while dists is part of the TMP_DIR) could be more difficult.= >=20 > I would consider on the bonus side for this that we don't have to > develop/maintain a custom solution, given that it suits our purposes of= > course... Agree. But if it only 'sort of' suits our purpose, we might need to write wrapper code around its short comings and maintain that. >>> Maybe I missed something on the proxy suggestion.. Could you >>> please elaborate on this? >> >> As for the integration the basic idea was that for taged bitbake tasks= >> the proxy is started and sets the *_PROXY environment variables. This >> should be doable with some mods to the base.bbclass and some external >> python scripts. >> >>> >>> >>>> I do have other ideas to do this, but that would restructure most >>>> of isar. >>> >>> Well, at least speaking for myself, I'd like to hear those as I >>> consider >>> this feature to be essential. Choice in solutions is always good :) >>> >> >> One idea that I got when I first investigated isar, was trying to be o= e >> compatible as much as possible. So using this idea would solve the >> reproducable builds as well: >> >> Basically implementing debootstrap with bitbake recipes that are >> created virtually on runtime by downloading and parsing the >> 'dists/*/*/*/Packages.gz' file. >=20 > Those virtual recipes then will have to be serialized as they contain > the version number of the package, right? I'm not sure if I understand your point correctly. I don't think the recipes needs to be written down as a file somewhere. We might have to take a look at the parsing part of bitbake, were the recipe data store is filled. So were the deserialization happens from '*.bb' to entry in ds. Here we just take one or more Debian package lists with some additional information, like the repo url and fill the ds with generated recipes. >> I suppose it should be possible to fetch the Packages file at an early= >> parsing step in a bitbake build, if its not already preset, and fill >> the bitbake data store with recipe definitions that fetch those binary= >> deb packages, have the appropriate dependencies and install them into >> the root file system. >=20 > Yes, or do a 'download-only' step prior to building as it's available > on Yocto.=20 Not sure if that is possible. Task execution is done after all those recipes are parsed and dependencies are resolved. To add virtual packages ourselves we need to do that before any task is triggered. So fetching the 'Packages.gz' file needs to be very early outside of what recipes normally do. I suspect that this is possible by using bitbake event handlers [1]. >> However, this idea is still in the brain storming phase. >> >> Since that would involve a very big redesign I don't think its feasibl= e >> currently. >=20 > Sounds interesting, at least for me... Thanks. Claudius [1] https://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user= -manual.html#events --=20 DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153 Keyserver: hkp://pool.sks-keyservers.net --7XLLa5p4U2FpB5DJob5BtGPDbVQ8Crjcr-- --udGDQ1f5dR399BcNJBVUCjL3oSuJCmca8 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEb/LlnwDGvCgx2GTBEXPLGZgIsVMFAloSnV8ACgkQEXPLGZgI sVOMXhAAnOcr8GaOkyCZ6s0+qbES7rfJ/4KQlycmLTDLuv0H0xxqq7i1/Cdg4rjl E57AfSn+2ZiiCgO4/9kz+zjZ7knRtlv9pkgTM/CWgBh757dhpikRGFGQemZjEgnZ Om86M4PA810C4/ac1v5obr/vw7ZNf5DXx/hORTNeS1LErOu97A9KEIILDcMMW04q 5P0YX7s4lyl899AkCEjJmudLZSXi+8jphMduijltMA6YrpelG0vav98bo3mcS7b6 CiCIO7vyxH490EKppXBs11zkaCcN/lCIYdytBtx4n4jHd2GQ2PP2J3PcLtZfWheY BCdYGGf7rjd9ICvgfcAHdfq25vS0RvotSBo0ECN0vheG+5HBGwGLEF/LelVrxH1b cmrZMoaNQCFktOy5AcSZL19950olm+3/J1AU1gEy8cEy2oi0ycKxx7XkN9qUjlbk XN4LznU8vEH7xDeIs0rIygyWKH1SfqcnpPYREaNtqBvi30Bx4mi4KGGLvOtXxinM ve797YHckF1a1AbhJ/BuUa6y1iiNgfrL4aSFWxi4p9He7suzghAazacuN62KV9iN 00631ToQmG1Ne3UBrtellQGXZIAnUhGzQIb4h9qG1GrCZxJBBOYOEFQxf3U/W845 osovK6K7RlTY3gdncM4uGd7MTXrqYyDeyOXjNFcFwoGRR8fBh4w= =uCxK -----END PGP SIGNATURE----- --udGDQ1f5dR399BcNJBVUCjL3oSuJCmca8--