From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6449958519840964608 X-Received: by 10.25.229.12 with SMTP id c12mr22547lfh.34.1503919679041; Mon, 28 Aug 2017 04:27:59 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.46.8.90 with SMTP id g26ls16594ljd.18.gmail; Mon, 28 Aug 2017 04:27:58 -0700 (PDT) X-Google-Smtp-Source: ADKCNb6qyYu//SvhnDXBXo4dJ65Hu2jxrB+U8ssvmG1MzgVEYkFRrnEbLwcEv5i7MnUzxn0WTw93 X-Received: by 10.25.141.81 with SMTP id p78mr24441lfd.18.1503919678556; Mon, 28 Aug 2017 04:27:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1503919678; cv=none; d=google.com; s=arc-20160816; b=vFvi8I6+sfJI3fa7xUVRqk0I4nBDzSdwSwVTdIyeOOUbfb3B9WZXD0P6GGjDktjwI+ 3b3n+ksk9oMTyK+xvhQ8/KPlDv4pwvJ0V1QNiuARA+VH5geenJbxUhGXqLThTEu9YiCd N1m12fmZxvTFtTOhH8JfxGabM4ebUEqSwFF0Eejae47uD7nWIpEJgBSrHAKuFqexsVIr vzWPxaXlYbKYDskQ4UeFnuimUE12jYCG/QDOayLGAoLyFUMPidZDfdsuE2GGQRtS8MSN BwxzQ+wtTXk/TALT0GbnPRaIRgabTbLofbeU69rWffzPeLWEvteAanXPaGMGPZeHRUQy oPjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:references:cc:to:from:subject :arc-authentication-results; bh=1xET/GharSaMlv2UQdumvVdzH/2d8dDaDsmFO16+c1o=; b=PD43lmcsWNvmaCaVcS01lNZ1TjP1QA4Djx7kUpxOm5WU2QXEA40UJeHjK+1jKKkRb/ Rs3E3aU3BRWCDiqE3pbxyRh9Pfbt5jMC/LOaNWx5EGt0FpAzaZ1+7anEGWAPbbNke/A2 I56TBwJN80Za2UUlgdRk8zXsNarV/J2EtWA0N0Dl8QWCPDdtEaTQNHcO189EACoaWhJq lLiaZ3rp+ucsrISiG3yVThd9BuXENvLshMQ7KrJql09d7tXBPFpdtbeSCjQe4uq76I59 xOYJ9LSEZomuCY9BMOt0R0bZqA5RXqit+l5D/nNUuZ7ecWmCx9lsLQ/Mkdov1IF4p+kW 11ng== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 194.138.37.40 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) smtp.mailfrom=claudius.heine.ext@siemens.com Return-Path: Received: from gecko.sbs.de (gecko.sbs.de. [194.138.37.40]) by gmr-mx.google.com with ESMTPS id y132si6193wme.2.2017.08.28.04.27.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 28 Aug 2017 04:27:58 -0700 (PDT) Received-SPF: neutral (google.com: 194.138.37.40 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) client-ip=194.138.37.40; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 194.138.37.40 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) smtp.mailfrom=claudius.heine.ext@siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by gecko.sbs.de (8.15.2/8.15.2) with ESMTPS id v7SBRkY0015893 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 28 Aug 2017 13:27:46 +0200 Received: from [139.25.68.223] (linux-ses-ext02.ppmd.siemens.net [139.25.68.223]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id v7SBRkB7028921; Mon, 28 Aug 2017 13:27:46 +0200 Subject: Re: Reproducibility of builds From: Claudius Heine To: isar-users@googlegroups.com Cc: Alexander Smirnov , Baurzhan Ismagulov , Henning Schild References: <42b9bc93-5192-a62a-1e79-19cd572e7c03@siemens.com> Message-ID: <9b39df31-1549-0397-f52a-8643cbc9fcc4@siemens.com> Date: Mon, 28 Aug 2017 13:27:46 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <42b9bc93-5192-a62a-1e79-19cd572e7c03@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: IaePFOIE70pq Hi, On 08/21/2017 01:23 PM, [ext] Claudius Heine wrote: > Hi, > > On 08/03/2017 10:13 AM, Claudius Heine wrote: >> Hi, >> >> am I right that Isar supports or should support reproducible root file >> system build? >> >> If I understand correctly, when multistrap is called, it fetches >> always the latest version of all packages from the debian repository >> mirrors. Am I mistaken or is this feature still on the roadmap? >> >> I that is on the roadmap, how are you thinking of solving this issue? >> >> The openembedded way would be to seperate the fetch and 'install' step >> and first download all packages into the DL_DIR and then use them from >> there. Maybe we could create this pipeline: >> >> dpkg-binary Recipe: >> >> fetch deb file into downloads -> insert into local repository >> >> dpkg-source Recipe: >> >> fetch sources into downloads -> build packages -> insert into local >> repository >> >> image Recipe: >> >> fetch all required packages into downloads -> insert all of them into >> the local repository -> create root fs using only the local repository >> >> Multistrap provides a '--source-dir DIR' parameter, that stores all >> installed packages into a directory. So if we would use that as a >> fetcher, then we would create a temporary rootfs just to get all >> required packages for the project. >> >> Are there other possible solutions for this? > > The problem with this solution is that its not possible to create > multiple images with different sets of packages that share the version > of the all the common packages. > > An alternative solution is to employ a repository cacher that caches the > 'Packages.gz' of the first request. This way it would also be faster > then running multistrap one additional time just to fetch all required > packages. > > Maybe apt-cacher-ng or something similar can be used for this. > However I am currently not sure how this can be integrated into the > current build process. Some ideas? Maybe implementing a simple repo > caching proxy that is integrated into isar? > > The repository cacher is likely a daemon running in parallel to > multistrap and fetches everything to the DL_DIR that is requested by it. > Maybe provide a 'clean_package_cache' task, that deletes the cached > 'Packages.gz', causing the next root fs build to use new package versions. > > I would really like to hear some feedback on this. In our meeting today, it was discussed that we should collect all requirements for this feature and discuss possible implementation ideas based on those requirements. Here are some requirements from my side: 1 If multiple different images with some common set of packages are build with one bitbake call, then all images should contain exactly the same version of every package that it has in common with any of the other images. 2 The resulting image should only depend on the build environment and isar metadata, not on the point in time it is build. This means if the environment, including the downloads directory, is complete (for instance by an earlier build of the image), every following build of this image recipe should result in exactly the same packages installed on this image. 3 Binary and source packages should be part of the archival process. Source packages are useful in case some package needs to be patched at a later date. Binary packages are useful, because building them from source packages is currently not 100% reproducible in Debian upstream. [1] 4 For development, it should be possible to easily reset the environment, triggering an upgrade of the packages on the next image build. 5 Deployable in CI environments. What those are exactly should be further discussed. Here are some: 5.1 Possibility to use a download cache, that is not bound to only one product/image/environment 5.2 More than one build at the same time in one environment should be possible 6 Efficiency: The reproducibility feature should be time and resource efficient as possible. E.g. Process should only fetch and store the required files. 7 Outputs a description file with the name and version of every package deployed/used in the image/environment. To 5: Since I don't have much experience with CI systems, requirements mentioned here might not be correct. Any comment or requirement additions are welcome. Cheers, Claudius [1] https://tests.reproducible-builds.org/debian/reproducible.html -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de