From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6449958519840964608 X-Received: by 10.223.136.137 with SMTP id f9mr628259wrf.0.1510937714038; Fri, 17 Nov 2017 08:55:14 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.232.150 with SMTP id f22ls976415wmi.8.gmail; Fri, 17 Nov 2017 08:55:13 -0800 (PST) X-Google-Smtp-Source: AGs4zMboQ5/R5xNMhWCd1F+sVauTm+XBBk1gRegNMM0BWg2osmXFrBggSv1HR7uSoJK67Myca7I8 X-Received: by 10.28.114.6 with SMTP id n6mr626893wmc.15.1510937713789; Fri, 17 Nov 2017 08:55:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510937713; cv=none; d=google.com; s=arc-20160816; b=ZFNPd+uFbViUSQ3Y/zW3tms0Qafu+iAoi9r4NqEk4StwIvNBaTlg21u2mqLtoyxsVg pAtlq28mQf1pKbCD/oAjM2J0SugI9/5Db0oV2H+NFieuXLCXCSedme2O2ga5bk+b9J5o ni0UuPIlELF6/+XfbFu+9PD202nURqYEZ1noR9EdkRjn5B2Tmo7a20XUxM9hcVizwmBe ISWZtLbGa3R2u1Vuo9MHHC/tIgD6zmrFsz1UPAthTtVAV3zaZZkRjmZmZgLtcrTt+lyL xgUZvtQ2ar81mDl5g0ThEdFpTOKLcT8ThR7puq/vzNwcok8Xg6eJIkm44eYIevCTcPTT PxLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:in-reply-to:content-transfer-encoding :content-disposition:mime-version:mail-followup-to:message-id :subject:to:from:date:arc-authentication-results; bh=BL+CN8QwkuICL+IPWWCjrytNKttpnIblcF+ZiJ3Z7d0=; b=nq1gYBefmlq7+r3QhNzBchRDQXAhbm6ZPMZScVEsqZBahGSsK1t/i1lbiM4AwQKU0I 9Qpac3R52e/D5KxsZwQWrFJQTlMQdlHA3NcWo/TlpOlJQXseWBO9RF9q+dCmx6HVwfay nmvIjar+TY+6Y/791uRBUkTfJptrto//FoUuh3FyIJ/3wChzruqdQX2HnZMbP1lPXtbU XDw3cJVBP8AOdL/r634t+3D7eiOuwcXyn25lBS9TWgTT2Vk6jiVKUJAVCbgisVA9yUTv C7BvAfxRGHIaTPkeR21nvutTk3YIL+bZ81u9Co+ocGNDGVSrrNEHOUX9BAADyM+uD013 Nhvw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of christian.storm@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=christian.storm@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id i136si307135wmd.2.2017.11.17.08.55.13 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 17 Nov 2017 08:55:13 -0800 (PST) Received-SPF: pass (google.com: domain of christian.storm@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of christian.storm@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=christian.storm@siemens.com Received: from mail2.siemens.de (mail2.siemens.de [139.25.208.11]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id vAHGtDrU013031 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Fri, 17 Nov 2017 17:55:13 +0100 Received: from localhost ([139.25.69.251]) by mail2.siemens.de (8.15.2/8.15.2) with ESMTPS id vAHGtD2P022462 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 17 Nov 2017 17:55:13 +0100 Date: Fri, 17 Nov 2017 17:53:58 +0100 From: "[ext] Christian Storm" To: isar-users@googlegroups.com Subject: Re: Reproducibility of builds Message-ID: <20171117165358.zyl7jjsu3rxutyod@MD1KR9XC.ww002.siemens.net> Mail-Followup-To: isar-users@googlegroups.com MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <6196aaa3-763a-7d5a-9971-d864d0943970@siemens.com> User-Agent: Mutt/20170113 (1.7.2) X-TUID: UdUvwuxhc/fM > > since I'm very interested in this feature, I'd like to resume this > > discussion and to eventually come to an agreed upon proposal on how > > to implement it. So, without further ado, here are my thoughts on > > the subject: > > > > Regardless of the concrete technical implementation, I guess we can > > agree on the need for a local cache/repository/store in which the Debian > > binary packages plus their sources have to be stored since one may not > > rely on the availability of those files online for eternity. > > > > These files in this cache/repository/store are the union of the Debian > > binary packages installed in the resulting image plus their sources as > > well as those installed in the buildchroot plus their sources. > > The latter is required to be able to rebuild Debian packages built from > > source with the same compiler version, libraries, -dev packages, etc. pp. > > > > Having the cache/repository/store at hand, there should be a mechanism > > to prime Isar with it, i.e., Isar should only and exclusively use Debian > > binary packages and sources from this cache/repository/store. > > This is again, irrespective of the technical implementation, be it via > > a repository cache or other means like git, a proxy server or whatsoever. > > > > Granted, if one changes, e.g, IMAGE_INSTALL_append, the build fails but > > does so rightfully as the set of packages is modified, resulting in a > > new version/epoch (=set of Debian packages plus their sources). So, > > there should be a convenient "interface" provided by Isar to maintain > > the cache/repository/store. For example, one may want to have different > > versions/epochs that may correspond to particular versions (git sha) of > > the Isar layer. Or one wants to later add a Debian package plus its > > source (which is automatically fetched), resulting in a new > > version/epoch etc. > > > > The remaining question is how to fill the cache/repository/store. In > > order to have a consistent version/epoch (=set of Debian packages plus > > their sources), there should not be duplicate packages in it, i.e., the > > same Debian package but with different versions. > > This could currently happen because there is a "window of vulnerability": > > multistrap is run twice, once for isar-image-base.bb and once for > > buildchroot.bb. In between those two runs, the Debian mirror used could > > get updated, resulting in a different version of the Debian package > > being installed in buildchroot than in the resulting image. > > This is an inherent problem of relying on the Debian way of distributing > > packages as one cannot a priori control what particular package versions > > one gets: In contrast to, e.g., Yocto where the particular package > > versions are specified in the recipes, this does not hold for Isar as > > the particular package versions are defined by the Debian mirror used, > > hence, one gets "injected" the particular package versions. > > So, what's required to reduce the "window of vulnerability" and to have > > a consistent cache/repository/store for a particular version/epoch is to > > make a snapshot-type download of the required packages. For this, of > > course, one needs to know the concrete set of packages. This list could > > be delivered by a "package trace" Isar run since not only multistrap > > does install packages but sprinkled apt-get install commands do as well. > > Thereafter, knowing the list, the snapshot-type download can happen, > > hopefully resulting in a consistent cache/repository/store. > > > > > > So, what do you think? > > I agree with your formulation of the problem here. > > Simple tracing of installed packages will have the problem you > described, that its possible that different versions of a package are > installed into buildchroot and image. So this trace needs to be cleaned > up and then based on that the whole process has to be started again to > create a consistent package list between buildchroot and image. This > doubles the build time in the trivial implementation. Sure, there's no free lunch here :) I'd rather strive for a good solution and avoid trivial implementations to make lunch as close to free as it gets, to stay in the picture. > With my suggestion of using a caching proxy, this could be solved > without any additional overhead. Could be the case, what are the drawbacks? What proxy do you propose to use? Maybe I missed something on the proxy suggestion.. Could you please elaborate on this? > I do have other ideas to do this, but that would restructure most of isar. Well, at least speaking for myself, I'd like to hear those as I consider this feature to be essential. Choice in solutions is always good :) Kind regards, Christian -- Dr. Christian Storm Siemens AG, Corporate Technology, CT RDA ITP SES-DE Otto-Hahn-Ring 6, 81739 M�nchen, Germany