From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6509751388101148672 X-Received: by 10.25.196.67 with SMTP id u64mr660531lff.38.1516827218657; Wed, 24 Jan 2018 12:53:38 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.25.195.141 with SMTP id t135ls1178393lff.4.gmail; Wed, 24 Jan 2018 12:53:38 -0800 (PST) X-Google-Smtp-Source: AH8x227Zz4DlyPlZXTyYKC8ZYxGfcgt9uGER05mBVPjViv9FC0HU2a1RlWpS7efy3jg/EWcGgcC5 X-Received: by 10.46.9.71 with SMTP id 68mr596380ljj.12.1516827217990; Wed, 24 Jan 2018 12:53:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1516827217; cv=none; d=google.com; s=arc-20160816; b=nB+rHQ5YAl5r2nJF+sp4jryf0U8mO8rFIA4zzFtpMQlV8rkmegZAUG/oo/e8N2PUlM 97+mDy8D/jzoiuIwnkEWR28BFqi1A9xZs0RMAl4CQeLLWRDfe4JEDZu4T6u9IhfTD7EK eWNVHmZFvoAzaBNSfsGYP4hKKXE/bsb+xNizGdc052Ax5sdXTEqbXSfb6st6pIfvA1zy i21Tvbf2Z/p/YaYJHDN6LdnfWsn7R47at9NngSNUw+p8I1NHqUNf/S3eBEHZVqN/w4mX J4Mv94WQY+we2WbksBD9/3DarubxzazWpd1zFT59iW0GVaV49mtZunQr8bAiNmOBKai/ ztZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-language:content-transfer-encoding:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :dkim-signature:arc-authentication-results; bh=4SNVq2FF6NiEk9JzfDm/4QktfOnx9VQq8TjhH+k9+bA=; b=u5F02kJNDDzYSjre/bIOFDC4Aa1gBo5PCvx1hyje0zXQtfCeq6SXmh+RjInaia5zo2 vZ8ILUfiRqyR3fGXOYl3at2V66IioiYqiELKOhuktrsdkJlROK0nkswbbQuF+nhF+bum p0yoNl6LujNe3ccjn/sWtbN7svN8WreweknXV70ncMaeTAvBCF1bYioaP2vR+rCk5mED lpzQHI6/1TjPhPZiahA786uRSvcP+l4AVdyeiFv4aWh3t7QISNXGUe71fyj3AMfDHtLs WEpwncrDdhIBk1nw0d5VI80Vt8IlbuTdDA2YGG88NHZ18Qk6HUap8kSbKnf55+6VSG0g wzAg== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@googlemail.com header.s=20161025 header.b=nWrIDSMl; spf=pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c09::229 as permitted sender) smtp.mailfrom=benbrenson89@googlemail.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=googlemail.com Return-Path: Received: from mail-wm0-x229.google.com (mail-wm0-x229.google.com. [2a00:1450:400c:c09::229]) by gmr-mx.google.com with ESMTPS id w29si461748lfc.5.2018.01.24.12.53.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Jan 2018 12:53:37 -0800 (PST) Received-SPF: pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c09::229 as permitted sender) client-ip=2a00:1450:400c:c09::229; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@googlemail.com header.s=20161025 header.b=nWrIDSMl; spf=pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c09::229 as permitted sender) smtp.mailfrom=benbrenson89@googlemail.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=googlemail.com Received: by mail-wm0-x229.google.com with SMTP id v71so11146218wmv.2 for ; Wed, 24 Jan 2018 12:53:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=4SNVq2FF6NiEk9JzfDm/4QktfOnx9VQq8TjhH+k9+bA=; b=nWrIDSMlqnyL3OCAUXaBGyPUKZhMCAYbaq69ALgvjZnNK7Bv6pmnjT2tO/9SpQKZBm XTYEXFIn6muhEXdEto5WHzgP+Wsw7n0HV8atvsEtztS69bBOXMDjJNjPD5Cdr9oIvuWX c5+yvjafjvETnAPio2uD8NxsBYivXeRM/GyXqNfp9NkppFuSs5yFcyl1KmxGgC+JgQ+E IoW7cFi5AVaztLHKxkgd7f1f6jzXlnuz7SBZwg04ufDcrY/VZthOuhmMcVSiurZqbPX/ lahiLwd6VZgfU0t8lb1XI7cxXvouCGOAUiBzK1P1RbVkPha182qUyoxTTc5OxLrQG4dx DTyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=4SNVq2FF6NiEk9JzfDm/4QktfOnx9VQq8TjhH+k9+bA=; b=qrhoLZu6hOp4uTRdFHLEIlx0QhGrQ7hcmKzsYReiw0Xj46EqF3sfEsrGUEF5OLBJ7s i2Cgv1oy2XHZHZwqnyy5P3olx0y+rr7xpgTpLS6yk8J3lhaa1pkSQwQTJQaCSDn3q5pn DsbNbz0OpncaOyuxS/VqCpRVyelVE7J3CBkhq2waAs7X7zD87/zaEEK5ZTu/uT8AR17l Iyg9ssZUxIuUbxBMLuWQMHVeIMnU3aY2qkiB100sKgTUlCVTsW/+RyyCt+PJjC7uiPXe Kd4HZLZwPLtb/TnaQzywiNN6RAsPjX91S7OCyI+c4iOp1jOoR5Z2QCoH8Iks0ptDo3bE N1cQ== X-Gm-Message-State: AKwxytdZFJ/jq9bYHEB2nez+GtPRbUldmEFpM4Lk4D9L3tcLCh/0+kEN qwH7kC4eslMSvsudeoVzgD6SBkFg X-Received: by 10.28.100.215 with SMTP id y206mr6005341wmb.130.1516827217126; Wed, 24 Jan 2018 12:53:37 -0800 (PST) Return-Path: Received: from [192.168.0.11] (ipb21b4230.dynamic.kabel-deutschland.de. [178.27.66.48]) by smtp.gmail.com with ESMTPSA id k125sm1579661wmd.48.2018.01.24.12.53.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Jan 2018 12:53:36 -0800 (PST) Subject: Re: [RFC v2][PATCH 2/3] build-rep: Add helper class To: Jan Kiszka , isar-users References: <20180111111939.25667-1-asmirnov@ilbers.de> <20180111111939.25667-3-asmirnov@ilbers.de> <20180112133205.2d2d6615@mmd1pvb1c.ad001.siemens.net> <20180112172526.19bd23a2@mmd1pvb1c.ad001.siemens.net> From: Benedikt Niedermayr Message-ID: <4c5b731d-d337-8be2-cd2d-025768ee6497@googlemail.com> Date: Wed, 24 Jan 2018 21:53:35 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-TUID: uAat1NJqE02x Am 24.01.2018 um 19:48 schrieb Jan Kiszka: > On 2018-01-19 22:23, 'Benedikt Niedermayr' via isar-users wrote: >> Am 14.01.2018 um 17:53 schrieb Jan Kiszka: >>> On 2018-01-12 17:25, [ext] Henning Schild wrote: >>>> Am Fri, 12 Jan 2018 16:29:20 +0300 >>>> schrieb Alexander Smirnov : >>>> >>>>> On 01/12/2018 03:32 PM, Henning Schild wrote: >>>>>> Am Thu, 11 Jan 2018 14:19:38 +0300 >>>>>> schrieb Alexander Smirnov : >>>>>> >>>>>>> Add class that helps to implement build reproducibility. It >>>>>>> implements anonymous function that will get all the Debian >>>>>>> dependencies that are needed for current Isar tree. >>>>>>> >>>>>>> Until build reproducibility will be fully implemented, it's >>>>>>> disabled by default. To enable it just set ISAR_BUILD_REP to "1" >>>>>>> in local.conf. >>>>>>> >>>>>>> Signed-off-by: Alexander Smirnov >>>>>>> --- >>>>>>>    meta-isar/conf/local.conf.sample                 |  6 +++++ >>>>>>>    meta-isar/recipes-app/hello/hello.bb             |  2 ++ >>>>>>>    meta/classes/build-rep.bbclass                   | 32 >>>>>>> ++++++++++++++++++++++++ >>>>>>> meta/classes/dpkg-base.bbclass                   |  2 ++ >>>>>>> meta/classes/image.bbclass                       |  2 ++ >>>>>>> meta/recipes-devtools/buildchroot/buildchroot.bb |  2 ++ 6 files >>>>>>> changed, 46 insertions(+) create mode 100644 >>>>>>> meta/classes/build-rep.bbclass >>>>>>> >>>>>>> diff --git a/meta-isar/conf/local.conf.sample >>>>>>> b/meta-isar/conf/local.conf.sample index 660958f..45b8995 100644 >>>>>>> --- a/meta-isar/conf/local.conf.sample >>>>>>> +++ b/meta-isar/conf/local.conf.sample >>>>>>> @@ -162,3 +162,9 @@ BB_NUMBER_THREADS = "4" >>>>>>>    # >>>>>>>    # Number of attempts to try to get reprepro lock for access to >>>>>>> apt cache REPREPRO_LOCK_ATTEMPTS = "16" >>>>>>> + >>>>>>> +# Isar build reproducibility feature creates local repository >>>>>>> which contains +# copies for all the upstream Debian packages that >>>>>>> could be used to build +# your Isar tree. So fetching them once >>>>>>> will guarantee that all the next Isar +# builds will be >>>>>>> identically. +ISAR_BUILD_REP ?= "0" >>>>>>> diff --git a/meta-isar/recipes-app/hello/hello.bb >>>>>>> b/meta-isar/recipes-app/hello/hello.bb index 44b8bc3..fafda2e >>>>>>> 100644 --- a/meta-isar/recipes-app/hello/hello.bb >>>>>>> +++ b/meta-isar/recipes-app/hello/hello.bb >>>>>>> @@ -16,3 +16,5 @@ SRCREV = >>>>>>> "ad7065ecc4840cc436bfcdac427386dbba4ea719" SRC_DIR = "git" >>>>>>>       inherit dpkg >>>>>>> + >>>>>>> +DEBIAN_DEPENDS = "debhelper (>= 9), autotools-dev" >>>>>> If i understand it correctly, this approach is an absolute NoGo. >>>>>> >>>>>> DEBIAN_DEPENDS as it is used today are runtime deps of pre-built >>>>>> packages. Here you include build-deps of a package that Isar needs >>>>>> to build. And you do so by introducing a copy of a string that is >>>>>> already included in the sources /debian/-folder. In fact you would >>>>>> have to put build and runtime deps into that one variable. And what >>>>>> about packages where the content depends on DISTRO_ARCH and friends? >>>>>> >>>>>> I think we first need a working solution of two "inherit dpkg" >>>>>> packages actually depending on each-other. After that we will need >>>>>> a real build of the Image to fill the cache. That is the only >>>>>> reliable way to make sure that the repo will contain the runtime >>>>>> and the build deps of all packages. More "sed"-guesswork is not >>>>>> going to do that trick and is only getting us 95% of what we need. >>>>>> And maintaining copies of stuff that is in /debian/ should not be >>>>>> considered. >>>>> Let's look at this realistically: >>>>> >>>>> 1. The upstream Debian should be fetched via single operation. If you >>>>> want to populate the cache during image build - you can't guarantee >>>>> that all the packages were built in the same environment. As >>>>> Christoph wrote, at any time the upstream apt could be updated. So >>>>> the buildchroot and image filesystems should be generated using local >>>>> static apt, otherwise it makes no sense to claim reproducibility. >>>> If the versions change on the way we should end up with two versions of >>>> one package in the cache. When we build from that again we will not get >>>> what we got the first time because image and buildchroot will probably >>>> take the latest version, while in the first run they got different >>>> versions. I think that is fine and we could detect that and declare the >>>> first build as "failed because of upstream change". >>>> You will just have to do that over and over again until you find a >>>> 30min window where the mirror did not change, i suspect you will hardly >>>> ever hit the case where the mirror changed between the multistraps. >>>> >>>>> 2. Regarding 'debian/control' duplications. According to the idea, >>>>> that 'base-apt' should be generated first of all, in bitbake terms it >>>>> means that: >>>> INHO that idea is wrong, it should be generated as a side-product of an >>>> actual build. >>>> If you want to get this right you will end up reimplementing Debian >>>> internals and keep running after them. That does not sound like a >>>> promising way to go. >>>> >>>>>    *.bb:do_build() should depends on base-apt:do_build(). >>>>> >>>>> On the other hand, base-apt:do_build() should already know the full >>>>> list of all deps for *.bb. So for now I see two ways how to implement >>>>> this: >>>>> >>>>>    - Duplicate 'debian/control' in recipe and share this variable with >>>>> base-apt (like it's done in this series). >>>>> >>>>>    - Manually parse 'debian/control'. But in this case base-apt should >>>>> depend on *.bb:do_unpack. Due to bitbake is 'statically configured' >>>>> tool, base-apt should know the list of all the recipes, what is also >>>>> impossible. In previous series I've tried to implement this by >>>>> defining a list of images in some global variable. >>>> "Manuall parse" is a NoGo, just look at the multiple broken versions of >>>> perl code guessing the build-deps ... emulating something that debian >>>> itself just gets right. >>>> >>>>> 2. Question how to get the list of build and runtime dependencies is >>>>> still open, I've already asked about this in previous mail. So, the >>>>> goal is to have the full list of packages that should be fetched. >>>>> 'debian/control' contains very specific format, so I can't feed this >>>>> line to multistrap/apt-get as it is. I see the following solutions: >>>>>    - Implement custom tool to do this (apt/dpkg uses standard perl >>>>> libs which contains all this stuff). >>>>>    - Manually parse 'debian/control' >>>>>    - Drop the requirement about single upstream fetch. >>>> And again the infamous "manual parse" and even a "custom tool" ... No. >>>> >>>> The repo should come out of what the multiple "multistraps" and "apt-get >>>> installs" actually fetch when they are allowed to fetch from the >>>> outside. If you refer to that with "Drop the requirement about ..." i >>>> fully agree. >>> I agree with Henning's view. We should exploit Debian for building the >>> repo and only use bitbake recipes to control and direct Debian tools, >>> definitely not reimplement them. >>> >>> Having one build run that both generates outputs and fills that repos >>> which would then be used on succeeding runs would be ok from user >>> perspective. We do not necessarily need that strict ordering of >>> fetch-before-use in the first build. >>> >>> Jan >>> >> I do also agree with Henning's view. >> >> >> It would be better not struggling with those complexity within Isar. >> Maybe I'm a bit late. But is reprepro really the best solution for >> implementing the cache? >> Let's assume we implement it like Henning mentioned already. We let Isar >> generate its image and add all required packages to the local repository. >> >> >> Then I see a big problem when adding new recipes to Isar, which will extend >> the local apt repository with maybe new versions of a dependent package. >> You never know that before. >> In this case the new dependencies can trigger a bunch replacement of the >> old packages, and the old stable state of local repo is gone. >> The solution may be to create snapshots of the current local repository, >> but reprepro is IMHO therefore not the best solution: >> https://serverfault.com/questions/489504/using-snapshots-with-reprepro-to-enable-rollbacks >> >> I think "aptly" would be much better here, since aptly gives you the >> possibility to create partial snapshots of new package versions. >> It also comes along with snapshot merging and lot of things Isar >> currently doesn't need, but maybe will sometime in the future. >> >> >> I imagine the apt-cache as a real cache. Means this cache should be able >> to be the interface between Isar and downloading the packages. >> So the cache should download the package (and dependencies) into the >> cache and Isar installs those packages from the cache instead from the >> remote repository. >> I have not tested that with aptly, but the link looks like it can >> achieve that: >> https://www.aptly.info/tutorial/pull/ >> When aptly is capable of downloading single packages (and dependencies) >> from a remote repo, then i strongly recommend using this tool instead >> of reprepro. >> > I also started to look into aptly, which I had on my notes for a while. > One reason is that it could act as a manual workaround for the missing > version-freezing support in Isar (pull into aptly repo, take a snapshot, > and then build from it with Isar). > > Another is that we also need some tooling to manage mirrors and > snapshots on the long run, and reprepro, TBHO, smells strange. On first > glance, it looks to me like the multistrap of the repo management tools: > once a cool thing, but now unmaintained. > > Did you dig further into aptly? I'm currently wondering how it > internally manages snapshots, if it does proper deduplication and how > much of that can be preserved when publishing for consumption by, say, > Isar... > > Jan > Yes, Until now I have implemented two approaches. So I will offer some of my results tomorrow. What I first did, was to use aptly as a complete mirror. Aptly creates a mirror and each package, which is requested by Isar will be added to the mirror and then be downloaded from there. Unfortunately that was not as easy as expected, since aptly cannot download packages directly. Aptly was originally developed for creating complete debian mirrors. It has now the "-filter" option, which can be used for narrowing the packages (and also dependecies) to be mirrored. So for each additional package I extended the filter and updated the mirror afterwards, but that architecture appeared to suffer under less performance, since each extension of the filter and subsequent update caused a download of debians packages.gz file (~ 10MB). My current approach is to create only a local repo after the first Isar build suceeded. This avoids complexity for now, since Isar and all debian tools are resolving dependencies as usual, when running against upstream repos. After the first build I create a snapshot of the local repo (both is possible: creating snapshots of local repos and also mirrors). I'm just investigating the deduplication topic and will give you response. Benni