From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6449958519840964608 X-Received: by 10.223.152.12 with SMTP id v12mr295560wrb.12.1504607910054; Tue, 05 Sep 2017 03:38:30 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.165.198 with SMTP id o189ls1704144wme.24.canary-gmail; Tue, 05 Sep 2017 03:38:29 -0700 (PDT) X-Google-Smtp-Source: ADKCNb5e9Gtj4gLH4Yrw37gP1XHX1L70XyWzpXJSeJl6aGp336q3skvI4RfWJhQ1nBcA4P89GuIa X-Received: by 10.28.17.129 with SMTP id 123mr363014wmr.23.1504607909626; Tue, 05 Sep 2017 03:38:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504607909; cv=none; d=google.com; s=arc-20160816; b=YNQj8XBZMDw7L4BtjSCKtXDx1SqhWLgK14+BSKUDU6C4ZgDFKvdBbKaoQHV3hAet+X Ez64wY2wVo8zoA7gkT7+jzsFq6PGQ+JVljhrpN6cUliAl8iaD1SAsS9lnfsJCzibMUv5 6xlsYKbPlwQu9U7cTVesCqungKc0FgYOkB43EuCeE6zK2JdAmIgSetz9lkI4KTFN6dbr SwTvph/abqazZov+eTchMX/JV7V1xTivc9yGue9ehfOy2n5l7dbQj6wIcQpRPoIC+jj5 LLi5XC3VCHJOALbqbYUfjV4xgvtYz9DPRAk/A2HVn4F5LCKl8JU41pZwzrYx7HCuieN3 UndQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=QtWlGKDW5ky8hzchk8jkS/ntXEnoqrKPgxLQ7/F36FM=; b=j+YdpydEbrkE5p5BaCk4I7u0zzBKed067NSDjWXMA4HxZr82RvTtXzcmQ9JOrgsjwg fOlBBy6AvO/8qnPIW9tXCm0atjYsO0G2/31QJ/ycmxOG9PF6sNC1iStAsrCzYsxIaMYz rjYC0gTtlqa6ecWt8IjYuRQdaPbcUrNE2K/ucVsEm+bihw9ebHfGNYR6pz5ICXlo3Whq QINyJAHtPXlGv0108NZ4nAVpBXOAFFSnGjs5niylAam9SwqsJ3hPvAeoQMKVDyZMnAkI XH5kWJsVyQtXI2MBses7wPP6yQr1O2QAcBV5ECPYkFOZnKXeOdubJ3zi1fGBOoiBKuRW sNlw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of jan.kiszka@siemens.com) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id b130si8488wme.0.2017.09.05.03.38.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 05 Sep 2017 03:38:29 -0700 (PDT) Received-SPF: neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of jan.kiszka@siemens.com) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of jan.kiszka@siemens.com) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail3.siemens.de (mail3.siemens.de [139.25.208.14]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id v85AcScG012240 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 5 Sep 2017 12:38:28 +0200 Received: from md1f2u6c.ww002.siemens.net ([139.25.68.37]) by mail3.siemens.de (8.15.2/8.15.2) with ESMTP id v85AcStI012416; Tue, 5 Sep 2017 12:38:28 +0200 Subject: Re: Reproducibility of builds To: Alexander Smirnov , Claudius Heine , isar-users@googlegroups.com Cc: Alexander Smirnov , Baurzhan Ismagulov , Henning Schild References: <42b9bc93-5192-a62a-1e79-19cd572e7c03@siemens.com> <9b39df31-1549-0397-f52a-8643cbc9fcc4@siemens.com> From: Jan Kiszka Message-ID: <4b4649c1-399e-d690-786b-7099983c42b8@siemens.com> Date: Tue, 5 Sep 2017 12:38:28 +0200 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: VxIWsZ8aTQPY On 2017-09-05 12:05, Alexander Smirnov wrote: > > > On 08/28/2017 02:27 PM, Claudius Heine wrote: >> Hi, >> >> On 08/21/2017 01:23 PM, [ext] Claudius Heine wrote: >>> Hi, >>> >>> On 08/03/2017 10:13 AM, Claudius Heine wrote: >>>> Hi, >>>> >>>> am I right that Isar supports or should support reproducible root >>>> file system build? >>>> >>>> If I understand correctly, when multistrap is called, it fetches >>>> always the latest version of all packages from the debian repository >>>> mirrors. Am I mistaken or is this feature still on the roadmap? >>>> >>>> I that is on the roadmap, how are you thinking of solving this issue? >>>> >>>> The openembedded way would be to seperate the fetch and 'install' >>>> step and first download all packages into the DL_DIR and then use >>>> them from there. Maybe we could create this pipeline: >>>> >>>> dpkg-binary Recipe: >>>> >>>> fetch deb file into downloads -> insert into local repository >>>> >>>> dpkg-source Recipe: >>>> >>>> fetch sources into downloads -> build packages -> insert into local >>>> repository >>>> >>>> image Recipe: >>>> >>>> fetch all required packages into downloads -> insert all of them >>>> into the local repository -> create root fs using only the local >>>> repository >>>> >>>> Multistrap provides a '--source-dir DIR' parameter, that stores all >>>> installed packages into a directory. So if we would use that as a >>>> fetcher, then we would create a temporary rootfs just to get all >>>> required packages for the project. >>>> >>>> Are there other possible solutions for this? >>> >>> The problem with this solution is that its not possible to create >>> multiple images with different sets of packages that share the >>> version of the all the common packages. >>> >>> An alternative solution is to employ a repository cacher that caches >>> the 'Packages.gz' of the first request. This way it would also be >>> faster then running multistrap one additional time just to fetch all >>> required packages. >>> >>> Maybe apt-cacher-ng or something similar can be used for this. >>> However I am currently not sure how this can be integrated into the >>> current build process. Some ideas? Maybe implementing a simple repo >>> caching proxy that is integrated into isar? >>> >>> The repository cacher is likely a daemon running in parallel to >>> multistrap and fetches everything to the DL_DIR that is requested by >>> it. Maybe provide a 'clean_package_cache' task, that deletes the >>> cached 'Packages.gz', causing the next root fs build to use new >>> package versions. >>> >>> I would really like to hear some feedback on this. >> >> In our meeting today, it was discussed that we should collect all >> requirements for this feature and discuss possible implementation >> ideas based on those requirements. >> >> Here are some requirements from my side: >> >>      1 If multiple different images with some common set of packages are >>        build with one bitbake call, then all images should contain >>        exactly the same version of every package that it has in common >>        with any of the other images. >> >>      2 The resulting image should only depend on the build environment >>        and isar metadata, not on the point in time it is build. >>        This means if the environment, including the downloads directory, >>        is complete (for instance by an earlier build of the image), every >>        following build of this image recipe should result in exactly the >>        same packages installed on this image. >> >>      3 Binary and source packages should be part of the archival process. >>        Source packages are useful in case some package needs to be >>        patched at a later date. Binary packages are useful, because >>        building them from source packages is currently not 100% >>        reproducible in Debian upstream. [1] >> >>      4 For development, it should be possible to easily reset the >>        environment, triggering an upgrade of the packages on the next >>        image build. >> >>      5 Deployable in CI environments. What those are exactly should be >>        further discussed. Here are some: >> >>        5.1 Possibility to use a download cache, that is not bound to only >>            one product/image/environment >> >>        5.2 More than one build at the same time in one environment should >>            be possible >> >>      6 Efficiency: The reproducibility feature should be time and >>        resource efficient as possible. E.g. Process should only fetch and >>        store the required files. >> >>      7 Outputs a description file with the name and version of every >>        package deployed/used in the image/environment. >> >> To 5: Since I don't have much experience with CI systems, requirements >> mentioned here might not be correct. >> >> Any comment or requirement additions are welcome. > > Thank you for the requirements, they quite good describe your usecase. > Unfortunately, ATM I don't know all the capabilities of > multistrap/debootstrap, so could not propose too much. Then I guess that needs to be explored further before we can decide which patch to go. Who could contribute to this? Jan > > In general, I think there could be following solutions: > >  - Create local apt cache with specified packages versions. >  - Patch multistrap to add capabilities to specify package versions. >  - Add hook to multistrap hooks (for example, in configscript.sh), that > will re-install desired package versions via apt-get. > -- Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux