From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6449958519840964608 X-Received: by 10.46.85.22 with SMTP id j22mr370942ljb.29.1504605962445; Tue, 05 Sep 2017 03:06:02 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.113.23 with SMTP id m23ls508481wmc.5.gmail; Tue, 05 Sep 2017 03:06:01 -0700 (PDT) X-Google-Smtp-Source: ADKCNb7VoDuk0YZsEt0Bxy11XSbCeboCdrvBTEI5/G7Xcru3U17q+kTtqP2vrWtvuyvo9imgYFKx X-Received: by 10.28.150.212 with SMTP id y203mr339543wmd.4.1504605961851; Tue, 05 Sep 2017 03:06:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504605961; cv=none; d=google.com; s=arc-20160816; b=CvTt/aebV/FxBz9spzW2zxQcOlDXGX7HAaClST/4efsZaOHHoUQxufYf5HqfmmKaob Y660DzOhLMXRZnSCL3nfY2jmKOG3Ssc0omKeWLniRDqveNjWQ9rVR0uR0GtYga38H5w0 hYOjHxlRggCKTxWVmr2LSSRmvN+taiqeSklLRrdA4a5fguAhodEQWw/vz4HQEITMtUCN nnFdiVAwNNQKQgGLFiN0vwQrdPdRuR4zWDRHqbfAcZIV+FcVAwqDUMYrvr3kok3vdOcW 7U7dwDpW/g2BVA6D6CVOpFVnnq3q+9THZqC4vig1jE74Ik0ctc0c4nufyWvOYB69Q1EP U4Aw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=bGdC8zpBxzv5KMX+FMTe9aDCjImQho5ayfJhKaJ615U=; b=GKopDRLU8tfVHQRgDDPMjK0xj58SdCPhOMCuDWRE2li17n1vIBHlER8u148c37LZ69 bwhL1xylKP4qPmCK48E+dVLtjuyld6n5D+q46o0rcnmDOBM1CC4uulrOZVT6+AH7YLZp RI+rqQzJinT/WMrw0RsQFn/O8u6ZE4HWvamI32E8okwiWapYyF48tNQQVbF3L8dccfvJ ctmv+2YZGLhbPF9Dw2+jnJXW2Q/92yN4/9HibPDli1kmUYxDrvsHgVZ4T5+PDh0f3fh/ QyNL3bwww5+jQkDDjjBhb00QwxZ0tiOXLMP4UmKbHSF87Cu6qnV1Sd7wXfwjxAnm5VXe /Chg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id p200si10751wmg.4.2017.09.05.03.06.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 05 Sep 2017 03:06:01 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id v85A5vLO012812 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 5 Sep 2017 12:05:58 +0200 Subject: Re: Reproducibility of builds To: Claudius Heine , isar-users@googlegroups.com Cc: Alexander Smirnov , Baurzhan Ismagulov , Henning Schild References: <42b9bc93-5192-a62a-1e79-19cd572e7c03@siemens.com> <9b39df31-1549-0397-f52a-8643cbc9fcc4@siemens.com> From: Alexander Smirnov Message-ID: Date: Tue, 5 Sep 2017 13:05:52 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <9b39df31-1549-0397-f52a-8643cbc9fcc4@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: OEB3seWNwHdw On 08/28/2017 02:27 PM, Claudius Heine wrote: > Hi, > > On 08/21/2017 01:23 PM, [ext] Claudius Heine wrote: >> Hi, >> >> On 08/03/2017 10:13 AM, Claudius Heine wrote: >>> Hi, >>> >>> am I right that Isar supports or should support reproducible root >>> file system build? >>> >>> If I understand correctly, when multistrap is called, it fetches >>> always the latest version of all packages from the debian repository >>> mirrors. Am I mistaken or is this feature still on the roadmap? >>> >>> I that is on the roadmap, how are you thinking of solving this issue? >>> >>> The openembedded way would be to seperate the fetch and 'install' >>> step and first download all packages into the DL_DIR and then use >>> them from there. Maybe we could create this pipeline: >>> >>> dpkg-binary Recipe: >>> >>> fetch deb file into downloads -> insert into local repository >>> >>> dpkg-source Recipe: >>> >>> fetch sources into downloads -> build packages -> insert into local >>> repository >>> >>> image Recipe: >>> >>> fetch all required packages into downloads -> insert all of them into >>> the local repository -> create root fs using only the local repository >>> >>> Multistrap provides a '--source-dir DIR' parameter, that stores all >>> installed packages into a directory. So if we would use that as a >>> fetcher, then we would create a temporary rootfs just to get all >>> required packages for the project. >>> >>> Are there other possible solutions for this? >> >> The problem with this solution is that its not possible to create >> multiple images with different sets of packages that share the version >> of the all the common packages. >> >> An alternative solution is to employ a repository cacher that caches >> the 'Packages.gz' of the first request. This way it would also be >> faster then running multistrap one additional time just to fetch all >> required packages. >> >> Maybe apt-cacher-ng or something similar can be used for this. >> However I am currently not sure how this can be integrated into the >> current build process. Some ideas? Maybe implementing a simple repo >> caching proxy that is integrated into isar? >> >> The repository cacher is likely a daemon running in parallel to >> multistrap and fetches everything to the DL_DIR that is requested by >> it. Maybe provide a 'clean_package_cache' task, that deletes the >> cached 'Packages.gz', causing the next root fs build to use new >> package versions. >> >> I would really like to hear some feedback on this. > > In our meeting today, it was discussed that we should collect all > requirements for this feature and discuss possible implementation ideas > based on those requirements. > > Here are some requirements from my side: > > 1 If multiple different images with some common set of packages are > build with one bitbake call, then all images should contain > exactly the same version of every package that it has in common > with any of the other images. > > 2 The resulting image should only depend on the build environment > and isar metadata, not on the point in time it is build. > This means if the environment, including the downloads directory, > is complete (for instance by an earlier build of the image), every > following build of this image recipe should result in exactly the > same packages installed on this image. > > 3 Binary and source packages should be part of the archival process. > Source packages are useful in case some package needs to be > patched at a later date. Binary packages are useful, because > building them from source packages is currently not 100% > reproducible in Debian upstream. [1] > > 4 For development, it should be possible to easily reset the > environment, triggering an upgrade of the packages on the next > image build. > > 5 Deployable in CI environments. What those are exactly should be > further discussed. Here are some: > > 5.1 Possibility to use a download cache, that is not bound to only > one product/image/environment > > 5.2 More than one build at the same time in one environment should > be possible > > 6 Efficiency: The reproducibility feature should be time and > resource efficient as possible. E.g. Process should only fetch and > store the required files. > > 7 Outputs a description file with the name and version of every > package deployed/used in the image/environment. > > To 5: Since I don't have much experience with CI systems, requirements > mentioned here might not be correct. > > Any comment or requirement additions are welcome. Thank you for the requirements, they quite good describe your usecase. Unfortunately, ATM I don't know all the capabilities of multistrap/debootstrap, so could not propose too much. In general, I think there could be following solutions: - Create local apt cache with specified packages versions. - Patch multistrap to add capabilities to specify package versions. - Add hook to multistrap hooks (for example, in configscript.sh), that will re-install desired package versions via apt-get. -- With best regards, Alexander Smirnov ilbers GmbH Baierbrunner Str. 28c D-81379 Munich +49 (89) 122 67 24-0 http://ilbers.de/ Commercial register Munich, HRB 214197 General manager: Baurzhan Ismagulov