From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6449958519840964608 X-Received: by 10.28.156.86 with SMTP id f83mr372096wme.5.1504612251930; Tue, 05 Sep 2017 04:50:51 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.25.196.145 with SMTP id u139ls85011lff.14.gmail; Tue, 05 Sep 2017 04:50:51 -0700 (PDT) X-Google-Smtp-Source: ADKCNb4kbpJ26VHqKzVQOUEmizti2XCqcNMDMc7NPhZQl3t0f7jF21TU48s7OLBfohtNbNYuj39Q X-Received: by 10.46.77.204 with SMTP id c73mr404468ljd.9.1504612251233; Tue, 05 Sep 2017 04:50:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504612251; cv=none; d=google.com; s=arc-20160816; b=dQbFYAle9b2vlPJm9Q+fHnoPSqCP7zO5IHv6VIp7oW0QvsQo83DFibPSLAshTAbsQ5 83XlK0+fvXGBBAcnaGmJ4XarAPyM7AvrWG5Ir0uS4fn2qLxPPnPyvyiLcn6DClozn+Fd J8KXq8y/s/PcJymCh0R329x12kpoXK4zbv/backQM8UFAzaDy76poSr/UrZJ8MfwC2wy YCpBh01D2WltYScP0XJhO81vxKsTK6kPw8xri681SAIRLT4uLDloonM1KVrMOXF9l95O p8TCJyDm7tiIzPq+JyqlT9zKQEdfKozSfDcBa4TW8VCF5Dcw7UyhLmQSuDVn5NrMNt3q o7pg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=vjw1k+M8fwnBm5ik79SMsADztQgA5p5cWZ51w7nQjr8=; b=oAQaGau9J7lYiyoQPLZkMeHrCEpOoETCoxwPQLHmbqk/i4ymPQfN83107vh4OYCr4K uTYqjplie+IlDvCWmJUISPNBxkV/HMU4hiimoE7vCW2c6e4kNw57iSOQO7JZSHEgwXx2 t9GM7CkPbuY3hvCs1GFegGmKkwkDJT3z2Eu7Gmge+1j+GhcxNnj9ZrOo39ccda9W1mm+ Vm9xXu3Wku+hXjyDJcE42q9VBtNfyJWl74ND8upkvQWvkF7WkTeXgBOikom5yHzH8fxx XLCWayx/5RJVTuJkWWdWVk2EwPpEuhG1SJzfGjDjft/k421wxrISpVuY2R2nNVj91ef5 dyoA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id q129si37516wme.3.2017.09.05.04.50.50 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 05 Sep 2017 04:50:50 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id v85Bolt8013051 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 5 Sep 2017 13:50:48 +0200 Subject: Re: Reproducibility of builds To: Jan Kiszka , Claudius Heine , isar-users@googlegroups.com Cc: Alexander Smirnov , Baurzhan Ismagulov , Henning Schild References: <42b9bc93-5192-a62a-1e79-19cd572e7c03@siemens.com> <9b39df31-1549-0397-f52a-8643cbc9fcc4@siemens.com> <4b4649c1-399e-d690-786b-7099983c42b8@siemens.com> From: Alexander Smirnov Message-ID: <9c893c7a-a40b-d040-d5ec-36055702d11b@ilbers.de> Date: Tue, 5 Sep 2017 14:50:42 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <4b4649c1-399e-d690-786b-7099983c42b8@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: VmX535zwoho7 On 09/05/2017 01:38 PM, Jan Kiszka wrote: > On 2017-09-05 12:05, Alexander Smirnov wrote: >> >> >> On 08/28/2017 02:27 PM, Claudius Heine wrote: >>> Hi, >>> >>> On 08/21/2017 01:23 PM, [ext] Claudius Heine wrote: >>>> Hi, >>>> >>>> On 08/03/2017 10:13 AM, Claudius Heine wrote: >>>>> Hi, >>>>> >>>>> am I right that Isar supports or should support reproducible root >>>>> file system build? >>>>> >>>>> If I understand correctly, when multistrap is called, it fetches >>>>> always the latest version of all packages from the debian repository >>>>> mirrors. Am I mistaken or is this feature still on the roadmap? >>>>> >>>>> I that is on the roadmap, how are you thinking of solving this issue? >>>>> >>>>> The openembedded way would be to seperate the fetch and 'install' >>>>> step and first download all packages into the DL_DIR and then use >>>>> them from there. Maybe we could create this pipeline: >>>>> >>>>> dpkg-binary Recipe: >>>>> >>>>> fetch deb file into downloads -> insert into local repository >>>>> >>>>> dpkg-source Recipe: >>>>> >>>>> fetch sources into downloads -> build packages -> insert into local >>>>> repository >>>>> >>>>> image Recipe: >>>>> >>>>> fetch all required packages into downloads -> insert all of them >>>>> into the local repository -> create root fs using only the local >>>>> repository >>>>> >>>>> Multistrap provides a '--source-dir DIR' parameter, that stores all >>>>> installed packages into a directory. So if we would use that as a >>>>> fetcher, then we would create a temporary rootfs just to get all >>>>> required packages for the project. >>>>> >>>>> Are there other possible solutions for this? >>>> >>>> The problem with this solution is that its not possible to create >>>> multiple images with different sets of packages that share the >>>> version of the all the common packages. >>>> >>>> An alternative solution is to employ a repository cacher that caches >>>> the 'Packages.gz' of the first request. This way it would also be >>>> faster then running multistrap one additional time just to fetch all >>>> required packages. >>>> >>>> Maybe apt-cacher-ng or something similar can be used for this. >>>> However I am currently not sure how this can be integrated into the >>>> current build process. Some ideas? Maybe implementing a simple repo >>>> caching proxy that is integrated into isar? >>>> >>>> The repository cacher is likely a daemon running in parallel to >>>> multistrap and fetches everything to the DL_DIR that is requested by >>>> it. Maybe provide a 'clean_package_cache' task, that deletes the >>>> cached 'Packages.gz', causing the next root fs build to use new >>>> package versions. >>>> >>>> I would really like to hear some feedback on this. >>> >>> In our meeting today, it was discussed that we should collect all >>> requirements for this feature and discuss possible implementation >>> ideas based on those requirements. >>> >>> Here are some requirements from my side: >>> >>> 1 If multiple different images with some common set of packages are >>> build with one bitbake call, then all images should contain >>> exactly the same version of every package that it has in common >>> with any of the other images. >>> >>> 2 The resulting image should only depend on the build environment >>> and isar metadata, not on the point in time it is build. >>> This means if the environment, including the downloads directory, >>> is complete (for instance by an earlier build of the image), every >>> following build of this image recipe should result in exactly the >>> same packages installed on this image. >>> >>> 3 Binary and source packages should be part of the archival process. >>> Source packages are useful in case some package needs to be >>> patched at a later date. Binary packages are useful, because >>> building them from source packages is currently not 100% >>> reproducible in Debian upstream. [1] >>> >>> 4 For development, it should be possible to easily reset the >>> environment, triggering an upgrade of the packages on the next >>> image build. >>> >>> 5 Deployable in CI environments. What those are exactly should be >>> further discussed. Here are some: >>> >>> 5.1 Possibility to use a download cache, that is not bound to only >>> one product/image/environment >>> >>> 5.2 More than one build at the same time in one environment should >>> be possible >>> >>> 6 Efficiency: The reproducibility feature should be time and >>> resource efficient as possible. E.g. Process should only fetch and >>> store the required files. >>> >>> 7 Outputs a description file with the name and version of every >>> package deployed/used in the image/environment. >>> >>> To 5: Since I don't have much experience with CI systems, requirements >>> mentioned here might not be correct. >>> >>> Any comment or requirement additions are welcome. >> >> Thank you for the requirements, they quite good describe your usecase. >> Unfortunately, ATM I don't know all the capabilities of >> multistrap/debootstrap, so could not propose too much. > > Then I guess that needs to be explored further before we can decide > which patch to go. Who could contribute to this? I think the one who will investigate the opportunities, also should implement this. It's a complete feature, so I can't handle it right now due to load with another features. Alex > >> >> In general, I think there could be following solutions: >> >> - Create local apt cache with specified packages versions. >> - Patch multistrap to add capabilities to specify package versions. >> - Add hook to multistrap hooks (for example, in configscript.sh), that >> will re-install desired package versions via apt-get. >> >