From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6518759238035046400 X-Received: by 10.25.223.65 with SMTP id q1mr3789562lfj.30.1517826702096; Mon, 05 Feb 2018 02:31:42 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.46.46.12 with SMTP id u12ls679103lju.4.gmail; Mon, 05 Feb 2018 02:31:41 -0800 (PST) X-Google-Smtp-Source: AH8x227EjQPvX4GKQ4eW7I+Qvp5DYIHTMnWOpgOi2I4+pYPAL4SqI4isOkY2s7ErEoubNOQboix4 X-Received: by 10.46.51.4 with SMTP id d4mr1798159ljc.36.1517826701434; Mon, 05 Feb 2018 02:31:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1517826701; cv=none; d=google.com; s=arc-20160816; b=bQiQkP0iITZFJhdmK+zL6kW1wep8/kRhGPKwEcDMF3QCkBqpphRRzu2EB2SgKLogFD 85l90/tO+vW7BsQY2Be/IGN9nM0dPqp0pYtjCubp2o4wbXjyeK88WiXvBUw8GfvSTrXT JtyZsT9fv7Z+8UFCaxu23pOz3FZaft21aMKzipnG3r8xys/P37bOo6+lWHCj2Oc6nHbH cPg316Qz5PhA4VRWAgUCtd/1tLmnHOav85wdx43p2VneHM87CuTgeghlcIJP0gfD+0eS NHR1EbPl/g1GHf8WGsq5gmlr78xLqaV+XjPAo4mqIM/nbtsExhdQcduYDZbEis7FfESL W4Dw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=B5cTm21q65Q4obPFW6gitLTMUSGxBnfgBmrXaooReGE=; b=QXVl9ikYz9qkKXJtJYRGG+yChbdZ02KXudsuYpq776gDm8tnJOPCdXxQGlZ+JM5CYE Jz1LgcmR6Ssgxye+PiYhfNNPUIcMrT5Iy3qqedEEWu3+9DTLpxLwOIf7PJDurhijrf5E 7NEnTm3KEW+yZXJJ3Xa85ZKRP6n/22k36GpfUkPE4qPM66IICec1hK09yoE9GrOIPbKE EQt8rN61cIZItq0DekXPi1cR0nwjjuc9+TkNOB84etAtE9R8IHLlRwSY/8OzSkmJO5Kc y/UTsoDJHSYDiJx8RtxnJVUVUDpS553TmsCHDjInRa1cCUTmSsKTYiJ3iuFJf0ZTuoJW ErZQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id h92si502791lfi.3.2018.02.05.02.31.41 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 05 Feb 2018 02:31:41 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail1.siemens.de (mail1.siemens.de [139.23.33.14]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id w15AVeZp002770 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 5 Feb 2018 11:31:40 +0100 Received: from [167.87.56.101] ([167.87.56.101]) by mail1.siemens.de (8.15.2/8.15.2) with ESMTP id w15AVd9r029677; Mon, 5 Feb 2018 11:31:39 +0100 Subject: Re: [PATCH 0/5] support creation of a full repo for offline/reproducible builds To: Alexander Smirnov , Cedric_Hombourger@mentor.com, isar-users@googlegroups.com Cc: Claudius Heine References: <20180204175454.220-1-Cedric_Hombourger@mentor.com> <79ae1eda-9b5b-a000-8740-b6753a894cd4@ilbers.de> From: Jan Kiszka Message-ID: Date: Mon, 5 Feb 2018 11:31:39 +0100 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <79ae1eda-9b5b-a000-8740-b6753a894cd4@ilbers.de> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: Y868UdbDxOaR On 2018-02-05 11:26, Alexander Smirnov wrote: > On 02/05/2018 10:18 AM, Jan Kiszka wrote: >> On 2018-02-04 18:54, Cedric_Hombourger@mentor.com wrote: >>> From: Cedric Hombourger >>> >>> The package repository created by isar using reprepro only includes >>> packages by isar. >>> To support offline/reproducible builds, this changeset adds a >>> do_populate task to >>> augment the repo with packages used during the build. The task may be >>> used against >>> the buildchroot and images recipes. It should be noted that isar >>> currently assumes >>> that the base distribution will provide both an -updates and security >>> feed. This is >>> certainly true for Debian but may not be the case for other >>> distributions or when >>> when using our own feed. >> >> The automatic addition of update and security feeds is more of a >> workaround until we have multi-repo support like Claudius is working on. >> I guess we can then drop this and just have repo lists for the different >> Debian versions with multiple entries. >> >>> >>> Some rework may be needed if the isar-apt changes get merged first. >>> Conceptually the >>> implementation may not change much (as far as I can tell!) >>> >>> Please review and let me know if any rework is required. >> >> There is indeed quite some overlap with what Alex and I were discussing >> yesterday at FOSDEM. However, also looking at these patches, we need to >> do some homework first. As you correctly stated in patch 1, there is the >> issue that we pull package list twice at different times: first for the >> buildchroot and then again for the image. That needs to go first so that >> we end up with a consistent build. Also, all those duplications in logic >> between the two chroot setup recipes are killing us. >> >> So I would propose the following roadmap: >> >> - consolidate chroot building into a common class that both buildchroot >>    and image recipes derive from, while doing that >>     - generate multistrap.conf (e.g. "cat <>       all over the place > > I'd propose here to start with buildchroot/image recipes clean up. I > mean there are lots of files in: > >  - meta/recipes-devtools/buildchroot/files >  - meta-isar/recipes-core/images/files > > Until these files will be unified, it makes no sense to implement common > class - you anyway have to keep several sets of these files per > image/buildchroot. That's why I proposed some time ago to drop > multistrap hooks and implement their actions inside bitbake recipe, but > this idea wasn't found support here. Did you start that effort back then? I don't remember if there was some code already or just the idea. In any case, the similarity is too high to keep on with the two instances. Let's start this now. > >> - build up a single apt cache that all chroot builders use > > In other words, there should be the only one component/recipe who > fetches the upstream packages. > > There is still open question, how to fetch all the packages: in a single > step or during the build. I think - as long as we fetch the package list only once - we can get away for now with fetching the actual deb files when they are needed. > > I want to have an option similar to Yocto's one: -c fetchall to have > possibility to fetch all the necessary artifacts and do not depend from > network anymore. Let's look into this as an add-on feature later, I would say. I you see that our implementation is making that needlessly harder, raise your hand, though. > >> - derive the mirror repo list after the first installation from that >>    cache which will then contain ALL required packages in the right >>    versions >> - describe the workflow (doc/) how to generate that mirror and how to >>    use it in succeeding builds >> >> Makes sense? >> >> We should than clarify who will work on what. >> >> BTW, the very first step is sorting out all the other patches and >> applying them to next. For that, Alex plans to first update bitbake and >> then go through what is pending (and still merges fine). > > My wish will be to keep an eye on this list and synchronize your > activities to avoid overlaps. > Right, mine as well. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux