From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6449958519840964608 X-Received: by 10.46.5.3 with SMTP id 3mr91284ljf.7.1511166882086; Mon, 20 Nov 2017 00:34:42 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.46.20.8 with SMTP id u8ls1316907ljd.9.gmail; Mon, 20 Nov 2017 00:34:41 -0800 (PST) X-Google-Smtp-Source: AGs4zMYo8BeMD+i0Z6SOE0ZW1OcwkSTamCZIm4Xj63aB+U7QZ70tgUWQmsQGT+iaYLqaS8JvQ4Rk X-Received: by 10.25.31.13 with SMTP id f13mr329888lff.16.1511166881776; Mon, 20 Nov 2017 00:34:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511166881; cv=none; d=google.com; s=arc-20160816; b=nM/5jJT5ScoQ1omSMLN19UX9k6L0ToSBKB8odwT3L96UaRGiDYm4eZFiMJrSEuY8OV A3IbHNuIn6g/N/ecryTMDqrUXAaifdYb9yrhEOflbY1OFlNl3u6BUKLY/I5xApPeRFYG XrbMUU48E3G8eB97fsSlp7IfjVyQyvUPSThz/GJ0QGy8arQ0g2JFs0Eliij3+uRpuUq6 NjqFH3D7/mvw2URgXcJYyhqqNHWHoa0sfKqPVYZMRbcmF7ERN1zq7++VC69rCRAPeEka AetfobLHAOt6w60NuaHuXU0F8FJSQ/bIT+eEiCAqotWH9dYcEBG2P7+Wza938clIIOce Wl2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:in-reply-to:content-transfer-encoding :content-disposition:mime-version:mail-followup-to:message-id :subject:to:from:date:arc-authentication-results; bh=lLZAT7AFRetZA+9miy1gGeQK/OHJJmWfk4oGadIyDoY=; b=Pip7jcU8mL8PpopX3HJfQg3W2lL8WMnHCf09yZFp88fOvirKujiez+Cl3gbEsPZMpM rItCG96u1NtU4PttipE8GFtYbv1X5rhfQ+Nb/UhIXkQ0AOnSk9fMmqtCwvqhIw8eNU/N 7yfBznWJOTiePtUu3rYykz2VRolClvSplmW+gOu875CTNggvWSgBrzxS1xCRAEFuRos/ 9J8xHa/SQq7h4mNP8FOQ6uuL7/4jK1xG3HpgfRWBj6a1C+Oya7hUWAmSsZZ1V1gKPF0/ EnryyNUEz7FWPd6rdKFyuJV4LbECwYtZJLRBCdB6QbXWL5IIyPoZFiEns5j/TweRcxd6 fOuw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of christian.storm@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=christian.storm@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id 198si120362ljj.1.2017.11.20.00.34.41 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 20 Nov 2017 00:34:41 -0800 (PST) Received-SPF: pass (google.com: domain of christian.storm@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of christian.storm@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=christian.storm@siemens.com Received: from mail3.siemens.de (mail3.siemens.de [139.25.208.14]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id vAK8YebY005016 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Mon, 20 Nov 2017 09:34:40 +0100 Received: from localhost ([139.25.69.251]) by mail3.siemens.de (8.15.2/8.15.2) with ESMTPS id vAK8YeGo011047 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 20 Nov 2017 09:34:40 +0100 Date: Mon, 20 Nov 2017 09:33:22 +0100 From: "[ext] Christian Storm" To: isar-users@googlegroups.com Subject: Re: Reproducibility of builds Message-ID: <20171120083322.mz7uiz7nbgesf2qp@MD1KR9XC.ww002.siemens.net> Mail-Followup-To: isar-users@googlegroups.com MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1510942495.3306.107.camel@denx.de> User-Agent: Mutt/20170113 (1.7.2) X-TUID: UwLoiarjuQbf > > [...] > > > With my suggestion of using a caching proxy, this could be solved > > > without any additional overhead. > > > > Could be the case, what are the drawbacks? > > More complexity and stuff to implement. Also maybe download speed. > > > What proxy do you propose to use? > > I was at first going with my own standalone proxy implementation in > pure stdlib python, so that it could be completely integrated into > isar. Why not hooking this into the fetcher(s) so that it's integrated rather than a standalone thing? As a bonus, you'll have full control on this from the Isar core/code. I think the main invention here is the code that does the consistent version/epoch guarantee anyway... > I had a very simple solution ready rather quickly, but it was > only synchronous and as such could only handle one connection at a > time. Instead of just throwing more threads at it, I wanted to go the > asyncio route. Sadly the python stdlib does not provide a http > implementation for asyncio. I wasn't clear how to proceed from here > further (aiohttp dependency or minimal own http implementation). Ah, OK. Wouldn't this account for premature optimization? :) > The other idea is to just use a ready made apt caching proxy like apt- > cache-ng. But here I am unsure if its flexible enough to use in our > case. Starting it multiple times in parallel with different ports for > different caches and only user privileges might be possible but I > suspect that seperating the pool and the dists folder (pool should go > to DL_DIR while dists is part of the TMP_DIR) could be more difficult. I would consider on the bonus side for this that we don't have to develop/maintain a custom solution, given that it suits our purposes of course... > > Maybe I missed something on the proxy suggestion.. Could you > > please elaborate on this? > > As for the integration the basic idea was that for taged bitbake tasks > the proxy is started and sets the *_PROXY environment variables. This > should be doable with some mods to the base.bbclass and some external > python scripts. > > > > > > > > I do have other ideas to do this, but that would restructure most > > > of isar. > > > > Well, at least speaking for myself, I'd like to hear those as I > > consider > > this feature to be essential. Choice in solutions is always good :) > > > > One idea that I got when I first investigated isar, was trying to be oe > compatible as much as possible. So using this idea would solve the > reproducable builds as well: > > Basically implementing debootstrap with bitbake recipes that are > created virtually on runtime by downloading and parsing the > 'dists/*/*/*/Packages.gz' file. Those virtual recipes then will have to be serialized as they contain the version number of the package, right? > I suppose it should be possible to fetch the Packages file at an early > parsing step in a bitbake build, if its not already preset, and fill > the bitbake data store with recipe definitions that fetch those binary > deb packages, have the appropriate dependencies and install them into > the root file system. Yes, or do a 'download-only' step prior to building as it's available on Yocto. > However, this idea is still in the brain storming phase. > > Since that would involve a very big redesign I don't think its feasible > currently. Sounds interesting, at least for me... Kind regards, Christian -- Dr. Christian Storm Siemens AG, Corporate Technology, CT RDA ITP SES-DE Otto-Hahn-Ring 6, 81739 M�nchen, Germany