From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7169469243442855936 X-Received: by 2002:a17:90b:258:b0:219:396f:8a09 with SMTP id fz24-20020a17090b025800b00219396f8a09mr3093498pjb.20.1669617686855; Sun, 27 Nov 2022 22:41:26 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a63:84c2:0:b0:43c:2618:9c3f with SMTP id k185-20020a6384c2000000b0043c26189c3fls5138083pgd.9.-pod-prod-gmail; Sun, 27 Nov 2022 22:41:26 -0800 (PST) X-Google-Smtp-Source: AA0mqf5xWVLj1D+In8lXmLNpsbyyv7stgYLN3sXekOHB+DKnOO9hIHXdt37IWQb+znStRA8NaTYT X-Received: by 2002:a63:e341:0:b0:46f:13b0:8dab with SMTP id o1-20020a63e341000000b0046f13b08dabmr26247789pgj.497.1669617685943; Sun, 27 Nov 2022 22:41:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669617685; cv=none; d=google.com; s=arc-20160816; b=oYG8FuhCwyjP5f+V30gENGPFxl+tHfQfwrjOzIGpmBfXuwWP53t0Tc0stM6qSUJggc +rhzNnaRvxf4X1ZwPmLAwK31T8DpEAAhwr5OiGFoMogHjdswG4d08FakKBLOWNz+JKxZ mOrjCnYOKIsviYPZpvefqgbbkiMBcEMCtPQvi55nVdOh9rcPJqxR2RaSAFG+OXFYjFf7 9JGRJ1PGkiEsqRw39Gjs4i6NxI6GCQ2pEM5S1LFcOSsG4Gq6k+l6HjPbP2gkJpmbgGmp I6eMNk4pd4LMReIdWoqXB2tf2R+v83JlOhNu27Y01jNKA7sBwmXZfrqRgQNtkZbIi9sP M/GA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=o6r5p5yI9cocQjWsoqSFgrNSmYFsyTtu9mrruHg7JeU=; b=B5J6s242Jeev3QRd/cCWnXuQ9n/Lqd0Fb0G3eKzeUSWVFCEg8aBNAm5ZwhjWMnbNl7 O4UjpYPh3EmXDBwP3zis4bg73UHkrCA8CEi5ukDK5For/GK9rQstOvH/b3WmbSk1CAb6 1chrQdAjF48RuWseVTkxlQExT6HK8d+yvInhBoiCO/LYr2RAwWgnjbqjlGccgY1pS4Lj n8q9y+R77j4es7eKCZ/qj5FvFWkcYgWzpNb3SvY7gVecKMPLXcaqPCIPEKXRmqJo/FT4 xtyMH2Ei3FpHBlcEN4gd+eoBTbLEpaI5B4Pmk1CA4HfIVIShT5mg28PhpqHcV8i8Z11t WTxA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id f6-20020a170902ce8600b00188a88cc62fsi556846plg.12.2022.11.27.22.41.25 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Sun, 27 Nov 2022 22:41:25 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from home.localnet (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 2AS6fM2U027973 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 28 Nov 2022 07:41:23 +0100 From: Uladzimir Bely To: Jan Kiszka , isar-users@googlegroups.com Subject: Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs Date: Mon, 28 Nov 2022 09:41:17 +0300 Message-ID: <3464852.dWV9SEqChM@home> In-Reply-To: References: <20221125064253.21907-1-ubely@ilbers.de> <2082143.9o76ZdvQCi@home> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: lwOhlaBxEDLw In email from Monday, 28 November 2022 09:32:22 +03 You wrote: > On 28.11.22 07:05, Uladzimir Bely wrote: > > In the email from Sunday, 27 November 2022 22:05:53 +03 user Jan Kiszka= =20 wrote: > >> On 26.11.22 06:30, Uladzimir Bely wrote: > >>> In the email from Friday, 25 November 2022 18:40:39 +03 user Jan Kisz= ka > >=20 > > wrote: > >>>> On 25.11.22 15:48, Uladzimir Bely wrote: > >>>>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan > >>>>> Kiszka > >>>=20 > >>> wrote: > >>>>>> On 25.11.22 08:56, Uladzimir Bely wrote: > >>>>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan > >>>>>>> Kiszka > >>>>>=20 > >>>>> wrote: > >>>>>>>> On 25.11.22 07:42, Uladzimir Bely wrote: > >>>>>>>>> Downstreams may want to change DISTRO to some custom value. > >>>>>>>>> When cross-building, this leads to downloading packages for host > >>>>>>>>> and target distros to the different download subdirs. > >>>>>>>>>=20 > >>>>>>>>> While base-apt is populated only from DISTRO download subdir, > >>>>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO =3D "1") build fails d= ue to > >>>>>>>>> missing packages in base-apt. > >>>>>>>>>=20 > >>>>>>>>> The similar issue happens with cross-building raspberry targets > >>>>>>>>> since they use their own mirrors and DISTRO. > >>>>>>>>>=20 > >>>>>>>>> Fix the issue by splitting base-apt by introducing > >>>>>>>>> BASE_HOST_DISTRO > >>>>>>>>> variable that is used instead of BASE_DISTRO for caching builds > >>>>>>>>> for host-related components (e.g., isar-bootstrap, > >>>>>>>>> buildchroot-host, > >>>>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO. > >>>>>>>>>=20 > >>>>>>>>> Signed-off-by: Uladzimir Bely > >>>>>>>>> --- > >>>>>>>>>=20 > >>>>>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 + > >>>>>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 + > >>>>>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 + > >>>>>>>>> meta/conf/distro/debian-common.conf | 2 + > >>>>>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++- > >>>>>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47 > >>>>>>>>> ++++++++++--------- > >>>>>>>>> 6 files changed, 40 insertions(+), 24 deletions(-) > >>>>>>>>>=20 > >>>>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf > >>>>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index > >>>>>>>>> c8f523b3..66d965c9 > >>>>>>>>> 100644 > >>>>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf > >>>>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf > >>>>>>>>> @@ -6,6 +6,8 @@ > >>>>>>>>>=20 > >>>>>>>>> BASE_DISTRO =3D "raspbian" > >>>>>>>>> BASE_DISTRO_CODENAME =3D "stretch" > >>>>>>>>>=20 > >>>>>>>>> +BASE_HOST_DISTRO =3D "debian" > >>>>>>>>=20 > >>>>>>>> Better prefix it with HOST, to be consistent: > >>>>>>>>=20 > >>>>>>>> HOST_BASE_DISTRO > >>>>>>>=20 > >>>>>>> Not a problem, if you think it would be better. > >>>>>>>=20 > >>>>>>>>> + > >>>>>>>>>=20 > >>>>>>>>> HOST_DISTRO ?=3D "debian-stretch" > >>>>>>>>> HOST_DISTRO_APT_SOURCES ?=3D "conf/distro/${HOST_DISTRO}.list" > >>>>>>>>>=20 > >>>>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf > >>>>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index > >>>>>>>>> 5e4a09c9..7bd81dd6 > >>>>>>>>> 100644 > >>>>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf > >>>>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf > >>>>>>>>> @@ -6,6 +6,8 @@ > >>>>>>>>>=20 > >>>>>>>>> BASE_DISTRO =3D "raspios" > >>>>>>>>> BASE_DISTRO_arm64 =3D "debian" > >>>>>>>>>=20 > >>>>>>>>> +BASE_HOST_DISTRO ?=3D "debian" > >>>>>>>>> + > >>>>>>>>>=20 > >>>>>>>>> BASE_DISTRO_CODENAME =3D "bullseye" > >>>>>>>>> HOST_DISTRO ?=3D "debian-bullseye" > >>>>>>>>> HOST_DISTRO_APT_SOURCES ?=3D "conf/distro/${HOST_DISTRO}.list" > >>>>>>>>>=20 > >>>>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf > >>>>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da7= 9a1 > >>>>>>>>> 100644 > >>>>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf > >>>>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf > >>>>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf > >>>>>>>>>=20 > >>>>>>>>> BASE_DISTRO =3D "ubuntu" > >>>>>>>>> BASE_DISTRO_CODENAME =3D "focal" > >>>>>>>>>=20 > >>>>>>>>> +BASE_HOST_DISTRO =3D "${BASE_DISTRO}" > >>>>>>>>> + > >>>>>>>>>=20 > >>>>>>>>> DISTRO_APT_SOURCES_arm64 ?=3D > >>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" > >>>>>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?=3D "conf/distro/${HOST_DISTRO}= =2Elist > >>>>>>>>> conf/distro/${HOST_DISTRO}-ports.list"> > >>>>>>>>>=20 > >>>>>>>>> diff --git a/meta/conf/distro/debian-common.conf > >>>>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b > >>>>>>>>> 100644 > >>>>>>>>> --- a/meta/conf/distro/debian-common.conf > >>>>>>>>> +++ b/meta/conf/distro/debian-common.conf > >>>>>>>>> @@ -5,6 +5,8 @@ > >>>>>>>>>=20 > >>>>>>>>> BASE_DISTRO =3D "debian" > >>>>>>>>>=20 > >>>>>>>>> +BASE_HOST_DISTRO ?=3D "${BASE_DISTRO}" > >>>>>>>>> + > >>>>>>>>>=20 > >>>>>>>>> HOST_DISTRO_APT_SOURCES ?=3D "conf/distro/${HOST_DISTRO}.list" > >>>>>>>>> DISTRO_APT_SOURCES ?=3D > >>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> > >>>>>>>>>=20 > >>>>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > >>>>>>>>> f32d192e..2ee0d4cc 100644 > >>>>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = =3D > >>>>>>>>> ",gnupg" > >>>>>>>>>=20 > >>>>>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =3D > >>>>>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?=3D "${@'HOST_' if > >>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') =3D=3D '1' else ''}" BOOTSTRAP_= DISTRO > >>>>>>>>> =3D > >>>>>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') = =3D=3D > >>>>>>>>> '1' > >>>>>>>>> else > >>>>>>>>> 'DISTRO')}"> > >>>>>>>>>=20 > >>>>>>>>> +BOOTSTRAP_BASE_DISTRO =3D "${@d.getVar('BASE_HOST_DISTRO' if > >>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') =3D=3D '1' else 'BASE_DISTRO')}"> > >>>>>>>>>=20 > >>>>>>>>> FILESEXTRAPATHS_append =3D ":${BBPATH}" > >>>>>>>>> =20 > >>>>>>>>> inherit deb-dl-dir > >>>>>>>>>=20 > >>>>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, > >>>=20 > >>> aptsources_entry_list): > >>>>>>>>> import re > >>>>>>>>>=20 > >>>>>>>>> if=20 bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): > >>>>>>>>> - premirrors =3D "\S* > >>>>>>>>> file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" > >>>>>>>>> + premirrors =3D "\S* > >>>>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> > >>>>>>>>>=20 > >>>>>>>>> else: > >>>>>>>>> premirrors =3D d.getVar('DISTRO_APT_PREMIRRORS', True)= or > >>>>>>>>> "" > >>>>>>>>> =20 > >>>>>>>>> mirror_list =3D [entry.split() > >>>>>>>>>=20 > >>>>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() { > >>>>>>>>>=20 > >>>>>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/b= oot > >>>>>>>>> st > >>>>>>>>> ra > >>>>>>>>> p" > >>>>>>>>> =20 > >>>>>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" > >>>>>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" =3D "1" ]; then > >>>>>>>>>=20 > >>>>>>>>> - line=3D"file:///base-apt/${BASE_DISTRO} > >>>>>>>>> ${BASE_DISTRO_CODENAME} > >>>>>>>>> main" + line=3D"file:///base-apt/${BOOTSTRAP_BASE_DI= STRO} > >>>>>>>>> ${BASE_DISTRO_CODENAME} main"> > >>>>>>>>>=20 > >>>>>>>>> if [ -z "${BASE_REPO_KEY}" ]; then > >>>>>>>>> =20 > >>>>>>>>> line=3D"[trusted=3Dyes] ${line}" > >>>>>>>>> =20 > >>>>>>>>> fi > >>>>>>>>> echo "deb ${line}" > > >>>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" > >>>>>>>>>=20 > >>>>>>>>> + line=3D"file:///base-apt/${BASE_DISTRO} > >>>>>>>>> ${BASE_DISTRO_CODENAME} > >>>>>>>>> main" > >>>>>>>>=20 > >>>>>>>> Won't this cause duplicate lines (with complaints of apt) in case > >>>>>>>> of > >>>>>>>> BOOTSTRAP_BASE_DISTRO =3D=3D BASE_DISTRO? > >>>>>>>=20 > >>>>>>> No, while I don't add new lines (no new "echo"s). I just modify > >>>>>>> "deb-src" > >>>>>>> line while it should point to BASE_DISTRO even for host case. > >>>>>>> Without > >>>>>>> it, > >>>>>>> there was a problem with "hello" (apt_fetch failed to download > >>>>>>> cached > >>>>>>> sources from "host" repo, while sources were cached in "target" o= ne. > >>>>>>>=20 > >>>>>>> That's how it looks now: > >>>>>>> | $ cat > >>>>>>> | isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base- > >>>>>>>=20 > >>>>>>> apt.list > >>>>>>>=20 > >>>>>>> | deb [trusted=3Dyes] file:///base-apt/raspios bullseye main > >>>>>>> | deb-src [trusted=3Dyes] file:///base-apt/raspios bullseye main > >>>>>>> | $ cat > >>>>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-a= pt. > >>>>>>> | li > >>>>>>> | st > >>>>>>> | deb [trusted=3Dyes] file:///base-apt/debian bullseye main > >>>>>>> | deb-src [trusted=3Dyes] file:///base-apt/raspios bullseye main > >>>>>>=20 > >>>>>> Indeed, I misintepreted the code. > >>>>>>=20 > >>>>>> But I think we still have an issue on the deb-src population side: > >>>>>>=20 > >>>>>> # ls build/downloads/deb-src/ > >>>>>> iot2050-debian > >>>>>>=20 > >>>>>> And that was with this patch applied. > >>>>>>=20 > >>>>>> Jan > >>>>>=20 > >>>>> Actually, I checked the case with the following configuration: > >>>>>=20 > >>>>> distro: my-debian > >>>>>=20 > >>>>> local_conf_header: > >>>>> nodebian: | > >>>>> =20 > >>>>> require conf/distro/debian-bullseye.conf > >>>>> DISTRO_NAME =3D "My Debian System" > >>>>>=20 > >>>>> So, I simulated renaming distro that is done in meta-iot2050. > >>>>>=20 > >>>>> Cached build was OK, "hello" package (that we build from sources) w= as > >>>>>=20 > >>>>> downloaded at first build to "my-debian" directory: > >>>>> | $ ls build/downloads/deb-src/my-debian/hello/ > >>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar= =2Egz > >>>>>=20 > >>>>> and second build picked the files with no problem. > >>>>>=20 > >>>>> The same with "raspios-bullseye" target - "hello" sources are > >>>>> downloaded > >>>>> to > >>>>>=20 > >>>>> the "target" dir: > >>>>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/ > >>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar= =2Egz > >>>>>=20 > >>>>> I even tested the case with BASE_REPO_FEATURES +=3D "cache-deb-src= " and > >>>>> it > >>>>> was OK (honestly, I don't know who could require cached build with > >>>>> this > >>>>> option enabled :) ). > >>>>>=20 > >>>>> Anyway, I'll also try to check my changes with meta-iot2050 layer a= lso > >>>>> (despite some packages in it are currently not compatible with > >>>>> BB_NO_NETWORK option) > >>>>=20 > >>>> Please then test without the coral packages, they have an own issue > >>>> /wrt > >>>> offline builds. > >>>=20 > >>> Checked the patchset with `IOT2050_CORAL_SUPPORT =3D "0"` and it works > >>=20 > >> ...but it caches to the wrong downloads/deb-src path, no? > >>=20 > >> Jan > >=20 > > Why wrong? > >=20 > > | meta-iot2050/build $ tree -d downloads/deb* > > | downloads/deb > > | =E2=94=9C=E2=94=80=E2=94=80 debian-bullseye > > | =E2=94=94=E2=94=80=E2=94=80 iot2050-debian > > | downloads/deb-src > > | =E2=94=94=E2=94=80=E2=94=80 iot2050-debian > > |=20 > > | =E2=94=9C=E2=94=80=E2=94=80 openssl > > | =E2=94=94=E2=94=80=E2=94=80 swig > >=20 > > `do_apt_fetch` works with `${DISTRO}` and downloads sources to the > > corresponding directory. >=20 > deb-src fetching still lacks base-distro normalization, like deb also has: >=20 Saying "normalization", do you mean the following: if ${DISTRO} is just=20 renaming (e.g. "iot2050-debian") everything should be downloaded to and pic= ked=20 up from the directory named "debian-bullseye"? > # tree -d build/downloads/deb* > build/downloads/deb > =E2=94=94=E2=94=80=E2=94=80 debian-bullseye > build/downloads/deb-src > =E2=94=94=E2=94=80=E2=94=80 iot2050-debian > =E2=94=9C=E2=94=80=E2=94=80 openssl > =E2=94=94=E2=94=80=E2=94=80 swig >=20 > (your deb folder was likely outdated) >=20 > Jan Uladzimir Bely