From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7169469243442855936 X-Received: by 2002:a02:9469:0:b0:363:afc3:b405 with SMTP id a96-20020a029469000000b00363afc3b405mr11687000jai.243.1669440660211; Fri, 25 Nov 2022 21:31:00 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6e02:104b:b0:302:e57b:d8a9 with SMTP id p11-20020a056e02104b00b00302e57bd8a9ls901971ilj.1.-pod-prod-gmail; Fri, 25 Nov 2022 21:30:59 -0800 (PST) X-Google-Smtp-Source: AA0mqf6NvMnvZIzQicuLKmlkXyXbt4HxSWEXcxtQy61VeBDjru/4zotwyR/p4dic+r+ejuKHgnnd X-Received: by 2002:a92:d345:0:b0:2f9:b594:970 with SMTP id a5-20020a92d345000000b002f9b5940970mr17410675ilh.56.1669440659590; Fri, 25 Nov 2022 21:30:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669440659; cv=none; d=google.com; s=arc-20160816; b=kFicfFTHjlxtbYkHoJOOPUMKPpTCMI+Sp9NGVHBATYiADD24oJfl2w26MIZ9abvC4B HUogUHnKQF+/WPXPFtBzWbGuDR4gDcPlhLBEDzGUtNH8xWABIUzUkBZcNUnKJ3b/tj+l DTw1dqU8unqnYkuHMYP4VruFhd6Sy8m7mEFjEXCJ+ttfYN+csgH7rUlEgvBy84N9adYV fxRR+T8XqDc+rXWDiq69uQPZo7VYkLMuIfPkZYQYhdUHk0uqqkhTLXmcRTSc/cFaqEmO nNyqbKArOFWfLIofpvML4WjW6sq+ABdoA1lT7ZuKnfxBD40kcEVVnEk2lTxS6q8U5X9I Q1RQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=vlHEfz8ijpXi4n8xzTR9Mkr7n07/PQEs2csirV9Jj88=; b=FTRVFkFHFQQLQ67Ff2QNyoaa0igBdbT1Bi/9b0rLNNkTaCDk4l5BIuqomnhu0B6Z/V gSrLN3t/CDraqLksJX6cJCvxNs4pk7SmyaAdoN+C5PZcUrCMiMFW0wEulGp5EOFbj0Dq zSmo0pbK/8DudGlfPxrSbiFJFQ6OEMY2bKQWDhISEhcpEcg4s02kjZE4uSkFWn+Tp8EI yNXYvgDWSeID6FZC/Jrpmia69qxPRwUnJqWAX+MPVxadGhGcLbfUJckK6vynS/WK4kWG 9hdm44bQI51OIJho0UyJ7d7d8oEqh5G6hKeZFf7rUiTmYqavHg9DvAdl8NxEOZj017Tu T4Og== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id h1-20020a0566380f0100b003748fd49976si323312jas.0.2022.11.25.21.30.59 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 25 Nov 2022 21:30:59 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from home.localnet (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 2AQ5UuY4018859 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 26 Nov 2022 06:30:57 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com, Jan Kiszka Subject: Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs Date: Sat, 26 Nov 2022 08:30:51 +0300 Message-ID: <6502643.MDQidcC6GM@home> In-Reply-To: <9bd3bdad-ea34-bbe4-ee94-46e789073ccb@siemens.com> References: <20221125064253.21907-1-ubely@ilbers.de> <3531707.aCxCBeP46V@home> <9bd3bdad-ea34-bbe4-ee94-46e789073ccb@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: GnpncZ6ydDJE In the email from Friday, 25 November 2022 18:40:39 +03 user Jan Kiszka wrote: > On 25.11.22 15:48, Uladzimir Bely wrote: > > In the email from Friday, 25 November 2022 17:11:26 +03 user Jan Kiszka wrote: > >> On 25.11.22 08:56, Uladzimir Bely wrote: > >>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan Kiszka > > > > wrote: > >>>> On 25.11.22 07:42, Uladzimir Bely wrote: > >>>>> Downstreams may want to change DISTRO to some custom value. > >>>>> When cross-building, this leads to downloading packages for host > >>>>> and target distros to the different download subdirs. > >>>>> > >>>>> While base-apt is populated only from DISTRO download subdir, > >>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to > >>>>> missing packages in base-apt. > >>>>> > >>>>> The similar issue happens with cross-building raspberry targets > >>>>> since they use their own mirrors and DISTRO. > >>>>> > >>>>> Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO > >>>>> variable that is used instead of BASE_DISTRO for caching builds > >>>>> for host-related components (e.g., isar-bootstrap, buildchroot-host, > >>>>> sbuild-chroot-host). Target components still use BASE_DISTRO. > >>>>> > >>>>> Signed-off-by: Uladzimir Bely > >>>>> --- > >>>>> > >>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 + > >>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 + > >>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 + > >>>>> meta/conf/distro/debian-common.conf | 2 + > >>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++- > >>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47 > >>>>> ++++++++++--------- > >>>>> 6 files changed, 40 insertions(+), 24 deletions(-) > >>>>> > >>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf > >>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..66d965c9 > >>>>> 100644 > >>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf > >>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf > >>>>> @@ -6,6 +6,8 @@ > >>>>> > >>>>> BASE_DISTRO = "raspbian" > >>>>> BASE_DISTRO_CODENAME = "stretch" > >>>>> > >>>>> +BASE_HOST_DISTRO = "debian" > >>>> > >>>> Better prefix it with HOST, to be consistent: > >>>> > >>>> HOST_BASE_DISTRO > >>> > >>> Not a problem, if you think it would be better. > >>> > >>>>> + > >>>>> > >>>>> HOST_DISTRO ?= "debian-stretch" > >>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>>>> > >>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf > >>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..7bd81dd6 > >>>>> 100644 > >>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf > >>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf > >>>>> @@ -6,6 +6,8 @@ > >>>>> > >>>>> BASE_DISTRO = "raspios" > >>>>> BASE_DISTRO_arm64 = "debian" > >>>>> > >>>>> +BASE_HOST_DISTRO ?= "debian" > >>>>> + > >>>>> > >>>>> BASE_DISTRO_CODENAME = "bullseye" > >>>>> HOST_DISTRO ?= "debian-bullseye" > >>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>>>> > >>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf > >>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1 > >>>>> 100644 > >>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf > >>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf > >>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf > >>>>> > >>>>> BASE_DISTRO = "ubuntu" > >>>>> BASE_DISTRO_CODENAME = "focal" > >>>>> > >>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}" > >>>>> + > >>>>> > >>>>> DISTRO_APT_SOURCES_arm64 ?= > >>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" > >>>>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list > >>>>> conf/distro/${HOST_DISTRO}-ports.list"> > >>>>> > >>>>> diff --git a/meta/conf/distro/debian-common.conf > >>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b 100644 > >>>>> --- a/meta/conf/distro/debian-common.conf > >>>>> +++ b/meta/conf/distro/debian-common.conf > >>>>> @@ -5,6 +5,8 @@ > >>>>> > >>>>> BASE_DISTRO = "debian" > >>>>> > >>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}" > >>>>> + > >>>>> > >>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>>>> DISTRO_APT_SOURCES ?= > >>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> > >>>>> > >>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > >>>>> f32d192e..2ee0d4cc 100644 > >>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = > >>>>> ",gnupg" > >>>>> > >>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = > >>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if > >>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO = > >>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' > >>>>> else > >>>>> 'DISTRO')}"> > >>>>> > >>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if > >>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"> > >>>>> > >>>>> FILESEXTRAPATHS_append = ":${BBPATH}" > >>>>> > >>>>> inherit deb-dl-dir > >>>>> > >>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list): > >>>>> import re > >>>>> > >>>>> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): > >>>>> - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" > >>>>> + premirrors = "\S* > >>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> > >>>>> > >>>>> else: > >>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or "" > >>>>> > >>>>> mirror_list = [entry.split() > >>>>> > >>>>> @@ -319,11 +320,15 @@ do_bootstrap() { > >>>>> > >>>>> "${ROOTFSDIR}/etc/apt/preferences.d/bootstra > >>>>> p" > >>>>> > >>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" > >>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then > >>>>> > >>>>> - line="file:///base-apt/${BASE_DISTRO} > >>>>> ${BASE_DISTRO_CODENAME} > >>>>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} > >>>>> ${BASE_DISTRO_CODENAME} main"> > >>>>> > >>>>> if [ -z "${BASE_REPO_KEY}" ]; then > >>>>> > >>>>> line="[trusted=yes] ${line}" > >>>>> > >>>>> fi > >>>>> echo "deb ${line}" > > >>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" > >>>>> > >>>>> + line="file:///base-apt/${BASE_DISTRO} > >>>>> ${BASE_DISTRO_CODENAME} > >>>>> main" > >>>> > >>>> Won't this cause duplicate lines (with complaints of apt) in case of > >>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO? > >>> > >>> No, while I don't add new lines (no new "echo"s). I just modify > >>> "deb-src" > >>> line while it should point to BASE_DISTRO even for host case. Without > >>> it, > >>> there was a problem with "hello" (apt_fetch failed to download cached > >>> sources from "host" repo, while sources were cached in "target" one. > >>> > >>> That's how it looks now: > >>> | $ cat isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base- > >>> > >>> apt.list > >>> > >>> | deb [trusted=yes] file:///base-apt/raspios bullseye main > >>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main > >>> | $ cat > >>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.list > >>> | deb [trusted=yes] file:///base-apt/debian bullseye main > >>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main > >> > >> Indeed, I misintepreted the code. > >> > >> But I think we still have an issue on the deb-src population side: > >> > >> # ls build/downloads/deb-src/ > >> iot2050-debian > >> > >> And that was with this patch applied. > >> > >> Jan > > > > Actually, I checked the case with the following configuration: > > > > distro: my-debian > > > > local_conf_header: > > nodebian: | > > > > require conf/distro/debian-bullseye.conf > > DISTRO_NAME = "My Debian System" > > > > So, I simulated renaming distro that is done in meta-iot2050. > > > > Cached build was OK, "hello" package (that we build from sources) was > > > > downloaded at first build to "my-debian" directory: > > | $ ls build/downloads/deb-src/my-debian/hello/ > > | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz > > > > and second build picked the files with no problem. > > > > The same with "raspios-bullseye" target - "hello" sources are downloaded > > to > > > > the "target" dir: > > | $ ls build/downloads/deb-src/raspios-bullseye/hello/ > > | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz > > > > I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" and it > > was OK (honestly, I don't know who could require cached build with this > > option enabled :) ). > > > > Anyway, I'll also try to check my changes with meta-iot2050 layer also > > (despite some packages in it are currently not compatible with > > BB_NO_NETWORK option) > > Please then test without the coral packages, they have an own issue /wrt > offline builds. Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works > > Thanks, > Jan -- Uladzimir Bely