From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6558372643829972992 X-Received: by 2002:a2e:98c8:: with SMTP id s8-v6mr32190ljj.14.1528112261780; Mon, 04 Jun 2018 04:37:41 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a19:7014:: with SMTP id h20-v6ls650229lfc.6.gmail; Mon, 04 Jun 2018 04:37:40 -0700 (PDT) X-Google-Smtp-Source: ADUXVKL9hRg7UgqKnS1nUsnVdP9qVGBO/i7lS2J5rL/fgYNM4KvcIEmu6Gvo2NAsh2J0phqDzFNo X-Received: by 2002:a19:1659:: with SMTP id m86-v6mr822998lfi.39.1528112260717; Mon, 04 Jun 2018 04:37:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528112260; cv=none; d=google.com; s=arc-20160816; b=bB3zEXecgu2hZHmebMylYwQd0WmjRpNH5zPMQMyvB4+0x6r50u2d/ywJ3Z3DmVTXOn OIVBLMBjs6RG2LC2zXdk/cmZ3LCdGWT74hQA2pfaCqf0S4w1AdUXq19vMOVUbcNBKZHE yipyduTF4NyS38RygkVo6SYGDoszU1Dz8wv6cVPfblOTNu2/sWQVR+V8i7DM34hABLUP tgDoRCiYIHJI/r2mZVyOgxUXb1VOYdGKaHGy+LojCG/awVab+mHl8p2roYfNVQ4R/iRD u9CTqCi1sr80HCBL46jMeqoFC33dnJLN5TRiZpJIw7iq2X09NNwtB3Sw3KGAlaQy/CN/ gaSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:to:from:date :arc-authentication-results; bh=Ggaiw1l2sDQEjQMw6pEM1BnJTq35CpDcxGUZ6XNaolw=; b=vnFQifAynSDNAc5j5x/MGY4TeoQ81vq7Oz4mn1KMfAxkC2v6cmSpuSkCPmyGzxQbbH UZriOLRGySkKtq3JchJ9joXiacMNN0ag2i0WK5bz3kOXUQvjzj76xg+F5YjTPJnkRcR6 oXA8uFska3MJ8l58KBqb4VMZ3ImRqqc8d0GNc2EAnhjHnKuOezw1F8ETQGQTFcNK8Y1X xzHDlHAFGH7AMQNxD+BwH3qZePPZiB09t1XMwKGXNaTYM7bL+2LXVd/NGaCeMVMcCZtY 04Fgx7c/C6N0Y8k3h7FW8VspcuI8SFvDzEuiA2YuG5qD9fHShUwghuFACvPsnBg8cWkV vdXA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 85.214.62.211 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id j127-v6si1793609lfe.4.2018.06.04.04.37.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 04 Jun 2018 04:37:40 -0700 (PDT) Received-SPF: neutral (google.com: 85.214.62.211 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 85.214.62.211 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Received: from yssyq.radix50.net (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w54Bbbvx028711 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 4 Jun 2018 13:37:39 +0200 Received: from yssyq.radix50.net (localhost [127.0.0.1]) by yssyq.radix50.net (8.14.4/8.14.4/Debian-8) with ESMTP id w54Bbbqc016825 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 4 Jun 2018 13:37:37 +0200 Received: (from ibr@localhost) by yssyq.radix50.net (8.14.4/8.14.4/Submit) id w54Bbb8D016824 for isar-users@googlegroups.com; Mon, 4 Jun 2018 13:37:37 +0200 Date: Mon, 4 Jun 2018 13:37:36 +0200 From: Baurzhan Ismagulov To: isar-users@googlegroups.com Subject: Re: [RFC PATCH 0/3] Reproducible build Message-ID: <20180604113736.GD5657@yssyq.radix50.net> Mail-Followup-To: isar-users@googlegroups.com References: <3467a5ec-182e-8c9a-cd19-7ad898323be7@siemens.com> <20180523063206.29180-1-claudius.heine.ext@siemens.com> <20180524180027.09b7b880@md1pvb1c.ad001.siemens.net> <3a6032fee718de6cf44fff4e8051a8c7a89a6471.camel@denx.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3a6032fee718de6cf44fff4e8051a8c7a89a6471.camel@denx.de> User-Agent: Mutt/1.5.23 (2014-03-12) X-TUID: SUBNm0ualmPb Hello Claudius, On Fri, May 25, 2018 at 07:04:53PM +0200, Claudius Heine wrote: > - Idea 0: Store tarball of debootstrap output with filled apt cache and use > that to restore isar-bootstrap. > - Idea 1: Generate a repository from the cache and use that for the next > debootstrap run. > - Idea 2: Like idea 1 but with aptly. And then use aptly to manage packages. > - Idea 3: Create a whole repo mirror with aptly or similar and strip unused > packages later. > - Idea 4: Create a whole repo mirror with aptly or similar and import used > package into a new repo. > - Idea 5: Implementing a 'caching proxy' feature in aptly. > - Idea 6: Implementing a caching proxy feature in isar. Thanks for summarizing, this makes it easier to communicate. Some general points first: * I'm ok with a partial implementation that goes in the right direction. * I'd really like to see user docs, also in RFC, because UX is a part of the design. It shows what use cases the change covers and how it does that. Regarding the implementation, I think idea 1 is the right way to go. Today, we operate with pure Debian inputs -- packages and metadata -- to build our outputs. Debian inputs are what we should store. > Because of the contra arguments 'whole local mirror' and 'different apt > repo urls are used' I would got for 0 and 5. Idea 1 is very similar to your current implementation and is achievable with dpkg-scanpackages and debootstrapping. I'm not proposing the whole mirror, just the packages you debootstrap + dpkg-scanpackages. Our actual problem is: 1. Getting the list of packages we need. 2. Fetching and managing them locally. Proxying is a quick approach to avoid solving the problem rather than addressing it. Also, it wouldn't support all Debian's fetch methods. > Critique 1: Similar to my 'simple solution' but adds the creation of an > additional repository to it. -> higher complexity > Pro: debootstrap process is done on every build. > Con: Different apt repo urls are used. > For me that is a no-go, because that means the configuration > is different between the initial and subsequent builds. IIUC, this is also the case with your current implementation. You build without or with ISAR_BOOTSTRAP_TARBALL. This could be changed to building with or without e.g. ISAR_BOOTSTRAP_SOURCE containing a complete sources.list line. > How to add new packages later? (maybe like partial update?) With the tarball, you suggest deleting and starting from scratch for now. For the first step, I'd suggest to limit the usage to that. That is possible with idea 1, too. In the future, we'd need some tool. FWIW, I'm currently not aware of a tool that does both (1) and (2) above or is sufficiently suitable for that. So, I think we should work with Debian to get introspection on debootstrap and apt-get and work on the tool for (2). Cooperating with some project would be nice, but isn't a requirement for me. > How to handle multiple repos? > => map all repos from initial run to the local one. Currently, you suggest to use multiple tarballs. With idea 1, you could provide multiple directories. FWIW, Alex's implementation [1] did (1) and (2) in a Debian way in a single repo, without duplication. > And then what? => cannot be reverted, loss of information It doesn't have to be reverted. Maintaining that manually would be time-consuming, but that is what people are forced to do today anyway. The feature would ease that burden till partial mirror management is implemented. So, I'd be happy to review patches with: 1. Package directory as output, dpkg-scanpackages, and debootstrap on every build. 2. User docs giving an example of initial usage, versioning, and one update (e.g., deleting everything and starting from scratch). I agree that we don't impose versioning tools on our users, but we should demonstrate one simple working setup, even if with binary artifacts in git. References: 1. https://groups.google.com/forum/#!topic/isar-users/QQUsVmSaAGk With kind regards, Baurzhan.