From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6449958519840964608 X-Received: by 10.46.31.2 with SMTP id f2mr2778131ljf.31.1505747161961; Mon, 18 Sep 2017 08:06:01 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.25.17.160 with SMTP id 32ls409017lfr.3.gmail; Mon, 18 Sep 2017 08:06:01 -0700 (PDT) X-Google-Smtp-Source: AOwi7QDIbAZlbKoV6stJ3/1ZcEAJDE8ahTYER3kYNvdmGYzDV+njMnQ+YTFPWT7wo8q1piEne7Gn X-Received: by 10.25.19.201 with SMTP id 70mr875165lft.35.1505747161384; Mon, 18 Sep 2017 08:06:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1505747161; cv=none; d=google.com; s=arc-20160816; b=fJdSICF7dgdVK+FgQzYLWxTtG/RxSuTPuFwmFnTEMWM9u0HgLdapRuF4fBe15CHHkj c6DtqJz7fQBugaeOzR1CtQg/ohvy7TVbU8kbpUJ6O2pdncatdEktc4I8wFKX3VaB4P8n gnXnuPYpamTyqbnAdwtUJN7kjWYA+lHvcafjSwHgB5hyz3CzgPG4UX4pO4pteIVJzq5W jnHehpfn0VFUd2L8TT70AyAe/IXa/zcZdVZ6dFhyU3DJY0tcAJ69ebjBlz1I5pMkpEW8 p7qW33tD3xSjHLqUkd9i++S9AtGdbAp0wbfKoh2447TUlDDARSiMSwFI3Aw2Wyz8EErH lIkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:to:from:date :arc-authentication-results; bh=gPHNcNp07sRCavn+r2vhwo4tNeDQDLvxtAeHOusbVX0=; b=h3ZLP9isXLI1TAQYNMOZPimZAM5aprAfyO8uToykXDG3BFKcPzbHgMBizYQHhygyEU JNKSWtiQChqP3KFynzur2SGfb9VEETErPsqlfGaYdcvBbUsuoMUJ7Mpb91A/jeGvtTLs 4SiE9QIXSjb9pK7uzUBXy9z59mPQf0iZXuowUHVVVaHqdVzLF29ZaIy5Zb0Kdg8upW+A /eMk+4kxKsYjphFUqpHLW7c0uWunVatuW0jmAp3ErGL6CN/W7GbJ/eoyBhi4B0QgcywP 6ZNr0Tcy4VYEz+97EaI3aIK142Tq5fYOe/M5RciU17nkkP98oLS4H3g5e35VsMG0YBLv FVDg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 85.214.62.211 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id 191si500463wmn.2.2017.09.18.08.06.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 18 Sep 2017 08:06:01 -0700 (PDT) Received-SPF: neutral (google.com: 85.214.62.211 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 85.214.62.211 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Received: from yssyq.radix50.net (p2E51BCA7.dip0.t-ipconnect.de [46.81.188.167]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id v8IF5w8M006502 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 18 Sep 2017 17:06:00 +0200 Received: from yssyq.radix50.net (localhost [127.0.0.1]) by yssyq.radix50.net (8.14.4/8.14.4/Debian-8) with ESMTP id v8IF5w6g015431 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 18 Sep 2017 17:05:58 +0200 Received: (from ibr@localhost) by yssyq.radix50.net (8.14.4/8.14.4/Submit) id v8IF5wor015430 for isar-users@googlegroups.com; Mon, 18 Sep 2017 17:05:58 +0200 Date: Mon, 18 Sep 2017 17:05:58 +0200 From: Baurzhan Ismagulov To: isar-users@googlegroups.com Subject: Re: Reproducibility of builds Message-ID: <20170918150558.GB3848@yssyq.radix50.net> Mail-Followup-To: isar-users@googlegroups.com References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-TUID: 64b5YTvdi/DK Hello Claudius, thanks much for sharing the concept and the requirements! Let's check whether I understand your concept correctly. Assume we have minimal stretch binaries + hello source. Your concept provides for: 1. Start bitbake, which: 1. Downloads debs to be installed and adds them into a local apt repo. 2. Fetches hello sources, builds them, and copies the deb to the local apt repo. 3. Bootstraps Debian and hello binary debs from the local apt repo. Please correct me if I got anything incorrectly. I like this workflow. This is what is used in Isar's predecessor. There, it is implemented manually using debmirror. The reason why Isar installs Debian packages from apt repos in Internet is to give first-time users a setup working OOTB. If they start developing a product, they are expected to create their own apt repos. See e.g. http://events.linuxfoundation.org/sites/events/files/slides/isar-elce-2016_1.pdf, slides 19 ("Debian apt" repo) and 26 ("Create repos for all components: Debian..."). That is why I was originally thinking about a tool that would support this manual workflow. After pondering on your proposal, I think it makes sense. I'd like to see the following features: * The functionality is implemented in a standalone tool usable manually or from bitbake. * The functionality is implemented based on dry-run output of {deboot,multi,...}strap. * The feature can be turned off in Isar's local configuration. * The tool supports initial mirroring as well as an update. This should also be controllable in Isar's local config. What I don't like is the implementation via a http proxy. IMHO, it's too indirect for the task (why bother with dynamic proxying if the list of packages is defined statically in a given apt repo). It supports only one of apt's six fetch methods (such as https, file, ssh, etc., see sources.list(5), more could be defined in the future or in special environments). The implementation is going to be complex, since it needs to distinguish between different build process chains in the same environment (two bitbakes running in a single docker). It should be trivial to get a list of packages from multistrap. The same functionality is available in debootstrap, when we move to it. Mirroring could be done by an existing or a new tool. The latter may be a step to identify requirements and get experience with the workflow before integrating the functionality into the former (possibly upon feedback from Debian community). Archiving of the apt repo is a CM issue outside of Isar. For reproducing older versions, it should be managed in an SCM (e.g., git). Synchronization between the right product and apt repo revisions is also outside Isar and could be solved e.g. with kas. Or, one goes hard-core and commits apt stuff into the product repo. In the future, we might come with a better solution for archiving and version pinning; at this stage I'd like to utilize existing Debian means first before going further. The details of the pinning concept would be affected by bitbake debian/control backend implementation. Similarly, at this stage I don't address advanced issues like sharing modified and non-modified apt repos, which could be implemented by a KISS jessie/Packages and myjessie/Packages with the shared pool. If we have many of them in practice (which I doubt), we could still return to the issue. Some comments below. On Thu, Aug 03, 2017 at 10:13:12AM +0200, Claudius Heine wrote: > am I right that Isar supports or should support reproducible root file > system build? Yes, this is possible outside of Isar. We wish that Isar makes that easier. > If I understand correctly, when multistrap is called, it fetches always the > latest version of all packages from the debian repository mirrors. Am I > mistaken or is this feature still on the roadmap? In the sense I interpret your wording, yes, multistrap always fetches the latest version of all packages from the Debian repo. That said, for a given repo, there is only one version of every package, defined in the Packages file. It is the latest for that repo. Given a URI in Internet, multistrap always fetches its "latest" (and the only) Packages and installs the "latest" (and the only) package versions. With kind regards, Baurzhan.