From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6449958519840964608 X-Received: by 10.28.156.76 with SMTP id f73mr442184wme.4.1510675534451; Tue, 14 Nov 2017 08:05:34 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.223.152.120 with SMTP id v111ls2613690wrb.1.gmail; Tue, 14 Nov 2017 08:05:34 -0800 (PST) X-Google-Smtp-Source: AGs4zMZXn0PckcT4cbDKIP6/RRedpUgsKOFCwH8TUuaz4G6WqaNd4Jd8ThkcasZ9c2CfYA4r3aRt X-Received: by 10.28.129.17 with SMTP id c17mr1162922wmd.18.1510675534136; Tue, 14 Nov 2017 08:05:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510675534; cv=none; d=google.com; s=arc-20160816; b=T22U7DdrqJ855FHHKwxuWdV1WMzBQQbtaInPjAFVDUTZhz1zI/XVd1jkjLpOL89yLP r/TxO2F0CANXTHmQse06k6VIAxbXduh87eO3aAXrTWMfVZEO/FybL0su0obyXFhaUzFG IJMOvqbgtOktd0f7MQmShXYtFeiDYxAjpD8AFMhdUe1n23+aiA73cxVxOaErvwO7BjfD nM3lHUqEJxkoySuC2MKd+zvw7UBE6Hj1gbZRm0elX37QPMrzsrjj2sbaG/MDSKpWfUxH qyIzdr+UlipRn2jo99Qzo3JJo+2cTu2Pi3NXHaifNnuGzWoKkBRaYhlsNgbyenIApk5J pR2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:in-reply-to:content-transfer-encoding :content-disposition:mime-version:mail-followup-to:message-id :subject:to:from:date:arc-authentication-results; bh=nnKKIkr8KgHvXqeOusE/z6mfhNxSWNL+CTltISZghtw=; b=cuQBNQAumTLzJ68ommWJ5WnF8Dc3PZk/j+10cFMjgW0SARtMF0jATpU+t2zRTBCXEM mOHuw/vqlOje6WiU8PpZGRAkfZJPPbmFQWGkyQMnPGQtUKUlSh918K7vkgiNoYkNpKcb dD5Z2FRmMuT/jTfh3IohDpZPl1BNa448NJsh9qbzsFoumQE7HEyRQnfRdi9duPR/wrmw Ea2UPjVCKgBZOX6maIVKlMSuDp9saZRuxaj0nirWQyzUvh9eBbovORC53VUdBj1yJGUt CS0kA24eW8O8w+5GO+HuY24wHSihuivR652XMLdGE3HAFP2XqrrjYgBLTt4n8MGyeTGZ 5wVA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of christian.storm@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=christian.storm@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id t37si303592wrc.3.2017.11.14.08.05.34 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 Nov 2017 08:05:34 -0800 (PST) Received-SPF: pass (google.com: domain of christian.storm@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of christian.storm@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=christian.storm@siemens.com Received: from mail2.siemens.de (mail2.siemens.de [139.25.208.11]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id vAEG5XK5022421 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 14 Nov 2017 17:05:33 +0100 Received: from localhost ([139.25.69.251]) by mail2.siemens.de (8.15.2/8.15.2) with ESMTPS id vAEG5Xrg005819 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 14 Nov 2017 17:05:33 +0100 Date: Tue, 14 Nov 2017 17:04:27 +0100 From: Christian Storm To: isar-users@googlegroups.com Subject: Re: Reproducibility of builds Message-ID: <20171114160427.xzkzqwc322ih5265@MD1KR9XC.ww002.siemens.net> Mail-Followup-To: isar-users@googlegroups.com MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/20170113 (1.7.2) X-TUID: zvFh5WR+LDof Hi, since I'm very interested in this feature, I'd like to resume this discussion and to eventually come to an agreed upon proposal on how to implement it. So, without further ado, here are my thoughts on the subject: Regardless of the concrete technical implementation, I guess we can agree on the need for a local cache/repository/store in which the Debian binary packages plus their sources have to be stored since one may not rely on the availability of those files online for eternity. These files in this cache/repository/store are the union of the Debian binary packages installed in the resulting image plus their sources as well as those installed in the buildchroot plus their sources. The latter is required to be able to rebuild Debian packages built from source with the same compiler version, libraries, -dev packages, etc. pp. Having the cache/repository/store at hand, there should be a mechanism to prime Isar with it, i.e., Isar should only and exclusively use Debian binary packages and sources from this cache/repository/store. This is again, irrespective of the technical implementation, be it via a repository cache or other means like git, a proxy server or whatsoever. Granted, if one changes, e.g, IMAGE_INSTALL_append, the build fails but does so rightfully as the set of packages is modified, resulting in a new version/epoch (=set of Debian packages plus their sources). So, there should be a convenient "interface" provided by Isar to maintain the cache/repository/store. For example, one may want to have different versions/epochs that may correspond to particular versions (git sha) of the Isar layer. Or one wants to later add a Debian package plus its source (which is automatically fetched), resulting in a new version/epoch etc. The remaining question is how to fill the cache/repository/store. In order to have a consistent version/epoch (=set of Debian packages plus their sources), there should not be duplicate packages in it, i.e., the same Debian package but with different versions. This could currently happen because there is a "window of vulnerability": multistrap is run twice, once for isar-image-base.bb and once for buildchroot.bb. In between those two runs, the Debian mirror used could get updated, resulting in a different version of the Debian package being installed in buildchroot than in the resulting image. This is an inherent problem of relying on the Debian way of distributing packages as one cannot a priori control what particular package versions one gets: In contrast to, e.g., Yocto where the particular package versions are specified in the recipes, this does not hold for Isar as the particular package versions are defined by the Debian mirror used, hence, one gets "injected" the particular package versions. So, what's required to reduce the "window of vulnerability" and to have a consistent cache/repository/store for a particular version/epoch is to make a snapshot-type download of the required packages. For this, of course, one needs to know the concrete set of packages. This list could be delivered by a "package trace" Isar run since not only multistrap does install packages but sprinkled apt-get install commands do as well. Thereafter, knowing the list, the snapshot-type download can happen, hopefully resulting in a consistent cache/repository/store. So, what do you think? Besten Gru�, Christian -- Dr. Christian Storm Siemens AG, Corporate Technology, CT RDA ITP SES-DE Otto-Hahn-Ring 6, 81739 M�nchen, Germany