From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6449958519840964608 X-Received: by 10.25.38.140 with SMTP id m134mr84089lfm.2.1505811327446; Tue, 19 Sep 2017 01:55:27 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.216.71 with SMTP id p68ls1357933wmg.2.gmail; Tue, 19 Sep 2017 01:55:27 -0700 (PDT) X-Google-Smtp-Source: AOwi7QD84F2PNVWInbLYw5z/xC0BcSzD4lNjCTqXdIU+YFXUOAgOJGMx3AbOk0K9qVhRWEzmTW3c X-Received: by 10.28.54.30 with SMTP id d30mr38972wma.6.1505811327103; Tue, 19 Sep 2017 01:55:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1505811327; cv=none; d=google.com; s=arc-20160816; b=cR6l87183iflBORtkwlJbXQ3qDdjwokiyPUGvC6uKNQ5KGe3m3jxBKmRoOG0gMFbuB PvpNRLHizXKw1I4vN80WFiCWz/Qd9yFIlvoSSRu+oOS6AK0/9UMHtc5UayqO5QB46X9d OnvGlWappKfDCJXeWo1/nDi2AlPcFhTVw4umFQaWAAx0zH/JLNz9xSw6kEdfA4A7QasS s33YTVsW8u5evxAC+Fx86Ag/SgMXjo7j6UCpgG9mOor4PT2io+GdSJa0e/qTdCUcuyL/ LMo7j+1z+3mvQA7Wyl40mGUV1zMNcwqPUnr2XtsUEuWLbtTMOcE1GKEjAnNECwNUO8SH 0NCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=s6kQAK9MKqIGE3w28yjL8wrRbpHMS86GMvZyduMKoBQ=; b=0Fgj2QSZw2YrRUhOOGaK/+6XoUJ35LG1B6LLJ102L+hRZ2ewxIxUfYlQ5lQ38Elzgw TuCY6/WNtiyDL61rnKxKRh580/kjJg+SxFWrXMEWU6FPNNLdmG2egtTwPP1F0K1pOTfM DIVQ4W7WTyZiObn7oXee3I8PudbSIs4eeyK5xCnzodXwB1pKFtN+ybxQP/M2dW/OEhd6 AbMFP02BF5EckMx36wmzuzKTadiR3Cq/7MNTi2u59dg1A+0zf9NvN+v1cgODrF8baEpX 99cikJQXG3OscyLVqy+wmKPHGYKjRXFunYAi7lNVJDXvhcDN4x2K/HEWzZdUa+hbZG25 kkHg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 192.35.17.14 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) smtp.mailfrom=claudius.heine.ext@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id d82si86080wmd.1.2017.09.19.01.55.27 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Sep 2017 01:55:27 -0700 (PDT) Received-SPF: neutral (google.com: 192.35.17.14 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 192.35.17.14 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) smtp.mailfrom=claudius.heine.ext@siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id v8J8tQLQ011614 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 19 Sep 2017 10:55:26 +0200 Received: from [139.25.68.223] (linux-ses-ext02.ppmd.siemens.net [139.25.68.223]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id v8J8tPuv017207 for ; Tue, 19 Sep 2017 10:55:25 +0200 Subject: Re: Reproducibility of builds To: isar-users@googlegroups.com References: <20170918150558.GB3848@yssyq.radix50.net> From: Claudius Heine Message-ID: Date: Tue, 19 Sep 2017 10:55:25 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <20170918150558.GB3848@yssyq.radix50.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: EvSfKt5EqOIJ Hi Baurzhan, On 09/18/2017 05:05 PM, Baurzhan Ismagulov wrote: > What I don't like is the implementation via a http proxy. IMHO, it's too > indirect for the task (why bother with dynamic proxying if the list of packages > is defined statically in a given apt repo). Only if someone bothers to create a separate debian mirror repository for every product. It uses much more resources. It would be much easier to have a global package cache and a project local package index for it. IMO that would be only possible with a caching repo proxy. > It supports only one of apt's six > fetch methods (such as https, file, ssh, etc., see sources.list(5), more could > be defined in the future or in special environments). At first. I started with a http proxy it the easiest to implement. Its always possible to add additional functionality to the proxy to support other fetch methods if necessary. But IMO that not really that important. > The implementation is > going to be complex, since it needs to distinguish between different build > process chains in the same environment (two bitbakes running in a single > docker). Why? We have more then one port available, so we can run more then one proxy simultaneously for each build. My current implementation just chooses a free port and makes it available to the calling process. > It should be trivial to get a list of packages from multistrap. The same > functionality is available in debootstrap, when we move to it. The problem is we still need to use apt in the buildchroot to install additional build dependencies for each recipe. Those are not part of what multistrap/debootstrap lists out. But since it would go through the http proxy it would be part of the static package cache. > Mirroring could > be done by an existing or a new tool. The latter may be a step to identify > requirements and get experience with the workflow before integrating the > functionality into the former (possibly upon feedback from Debian community). As I said, I don't see the sense in creating a full debian mirror for every project. And partial mirrors are difficult to create because of multistrap/debootstrap (in case of the buildchroot) don't know about every package that is added to the image. > > Archiving of the apt repo is a CM issue outside of Isar. For reproducing older > versions, it should be managed in an SCM (e.g., git). That should be possible. Just archive the package index in a git repo and the packages in a git lfs repo. > Synchronization between > the right product and apt repo revisions is also outside Isar and could be > solved e.g. with kas. Never said that it is. But isar is responsible for providing ways to import/export some kind of package list into a build. > Or, one goes hard-core and commits apt stuff into the > product repo. That might depend on your 'product' definition, but for me product is not a image. So products can have varying package versions, while images obviously doesn't. So committing them together with products makes no sense to me. But committing them together with the final image with a reference of the used refspec of the product repository makes more sense. > In the future, we might come with a better solution for archiving > and version pinning; at this stage I'd like to utilize existing Debian means > first before going further. The details of the pinning concept would be > affected by bitbake debian/control backend implementation. I said nothing about pining, because IMO package updates etc. should still be possible on the target if wanted. But we should be able to recreate images at least from a package list. So apt package pinning is just a different solution for a different problem. If you mean pinning just in the bootstraping phase, then yes, that would be nice. But I don't know how that can solve the buildchroot problem. Also since the package index contains just one version of each package, I don't see how it would be possible to pin them to an older version at this stage, because those would no longer be available in the index and *bootstrap would not know where to fetch them. AFAIK Debian currently has no convenient means for solving these issues yet. I used apt-cacher-ng when I worked with elbe, but setting that up for every project separately is a big hassle. I want a easy solutions where this stuff is done inside of the normal bitbake process, where not every developer has to wire up her own process of building root file systems. Because if they have to build their own most developers don't care about it or it becomes impossible to recreate images because a couple of unknown software packages in some unknown version and with unknown configuration are necessary for it. So its important to use normal upstream package mirrors and have a process in place inside bitbake that cares about these issues transparently. IMO its important to not have to many options and unneeded complexity. So reproducibility should be the default and everyone is free to update/clear the package index manually via a single bitbake task. Cheers, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de