From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7169469243442855936 X-Received: by 2002:a63:ff0b:0:b0:477:362d:85d3 with SMTP id k11-20020a63ff0b000000b00477362d85d3mr25884473pgi.395.1669615524147; Sun, 27 Nov 2022 22:05:24 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a63:84c2:0:b0:43c:2618:9c3f with SMTP id k185-20020a6384c2000000b0043c26189c3fls5094650pgd.9.-pod-prod-gmail; Sun, 27 Nov 2022 22:05:23 -0800 (PST) X-Google-Smtp-Source: AA0mqf5lGt1mgwMjToLd/EkY9dHPc7CiPl2utaL641Fyt6mQC1jk699umKyH3QfFZXTVx0F3Armu X-Received: by 2002:a62:79ce:0:b0:575:4e34:7ed9 with SMTP id u197-20020a6279ce000000b005754e347ed9mr695357pfc.52.1669615523083; Sun, 27 Nov 2022 22:05:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669615523; cv=none; d=google.com; s=arc-20160816; b=Ztg7ATDgxuSh2gtD2C6TfxAbBfFXl2ZaiQeywqrIpTYHDacqFCnKLTqADowki+2XSl 1aaC+fGXj/utzl+9uq2q3Wiez1Z+Th2uTkcIlVxRq0HVPEMDfcnl3XPTVW8faGVbLVVR MgeFxtYGdW0k8wAfxLD7sIXQi6YFTBWUOJIEtiSeTfNJolHvC/7eF5A02znbQkCjZQ00 sg7JQqhqQxhObKRrj1sbZ1fu1agCRn8Yoof/N7J6bp8D1fM0m04kykw0CE4JG+jZe26A wmctSi7NFL5THS1f+VJIgb9fOy99NM6ZMnFnFDzZdelGrmtaXvgG8EKRPkUOJDjtYpVW w0Cg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=Y8qhiJ96T9xJBjk7XaJSMPcZgX32zLeeKTFr2ytd31I=; b=IrHajP/bLYdsj21aPmhG2soi9ovfdI3Of9vzVfjxKFyUeTfGEKJ9JEtZ7uK2oQydE7 IoACNOuD/N5hc7LeBsi1PW/7nm6uYHhqKzNjq4oJbsgneJBHOtENYqV4TtTCRyENCALD XG/MD4yjxhLm19+nutsn6HmpGNzI8Tg1UMqCDIzYokY2oQk/QuiInjC/3aD3TfR6UhCo JOFEvWK/8d7O6xnEMSdmhP+iU1WFxgxQMjpRRvLxfiCXbwXJKJfiKaWVWi5wheF2jtBM poFIMSaEGbmHNyFI5VZYKFS+5HovW8lVeF0ffqP9PxGzLyJ0TSxEFCY0PkUJTUZIvXpN 1ooA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id k145-20020a628497000000b00573f624935dsi528331pfd.3.2022.11.27.22.05.22 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Sun, 27 Nov 2022 22:05:22 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from home.localnet (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 2AS65Jqf027825 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 28 Nov 2022 07:05:20 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com, Jan Kiszka Subject: Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs Date: Mon, 28 Nov 2022 09:05:14 +0300 Message-ID: <2082143.9o76ZdvQCi@home> In-Reply-To: References: <20221125064253.21907-1-ubely@ilbers.de> <6502643.MDQidcC6GM@home> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: OZzrc0pObqG8 In the email from Sunday, 27 November 2022 22:05:53 +03 user Jan Kiszka wro= te: > On 26.11.22 06:30, Uladzimir Bely wrote: > > In the email from Friday, 25 November 2022 18:40:39 +03 user Jan Kiszka= =20 wrote: > >> On 25.11.22 15:48, Uladzimir Bely wrote: > >>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan Kisz= ka > >=20 > > wrote: > >>>> On 25.11.22 08:56, Uladzimir Bely wrote: > >>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan > >>>>> Kiszka > >>>=20 > >>> wrote: > >>>>>> On 25.11.22 07:42, Uladzimir Bely wrote: > >>>>>>> Downstreams may want to change DISTRO to some custom value. > >>>>>>> When cross-building, this leads to downloading packages for host > >>>>>>> and target distros to the different download subdirs. > >>>>>>>=20 > >>>>>>> While base-apt is populated only from DISTRO download subdir, > >>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO =3D "1") build fails due= to > >>>>>>> missing packages in base-apt. > >>>>>>>=20 > >>>>>>> The similar issue happens with cross-building raspberry targets > >>>>>>> since they use their own mirrors and DISTRO. > >>>>>>>=20 > >>>>>>> Fix the issue by splitting base-apt by introducing BASE_HOST_DIST= RO > >>>>>>> variable that is used instead of BASE_DISTRO for caching builds > >>>>>>> for host-related components (e.g., isar-bootstrap, buildchroot-ho= st, > >>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO. > >>>>>>>=20 > >>>>>>> Signed-off-by: Uladzimir Bely > >>>>>>> --- > >>>>>>>=20 > >>>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 + > >>>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 + > >>>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 + > >>>>>>> meta/conf/distro/debian-common.conf | 2 + > >>>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++- > >>>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47 > >>>>>>> ++++++++++--------- > >>>>>>> 6 files changed, 40 insertions(+), 24 deletions(-) > >>>>>>>=20 > >>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf > >>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index > >>>>>>> c8f523b3..66d965c9 > >>>>>>> 100644 > >>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf > >>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf > >>>>>>> @@ -6,6 +6,8 @@ > >>>>>>>=20 > >>>>>>> BASE_DISTRO =3D "raspbian" > >>>>>>> BASE_DISTRO_CODENAME =3D "stretch" > >>>>>>>=20 > >>>>>>> +BASE_HOST_DISTRO =3D "debian" > >>>>>>=20 > >>>>>> Better prefix it with HOST, to be consistent: > >>>>>>=20 > >>>>>> HOST_BASE_DISTRO > >>>>>=20 > >>>>> Not a problem, if you think it would be better. > >>>>>=20 > >>>>>>> + > >>>>>>>=20 > >>>>>>> HOST_DISTRO ?=3D "debian-stretch" > >>>>>>> HOST_DISTRO_APT_SOURCES ?=3D "conf/distro/${HOST_DISTRO}.list" > >>>>>>>=20 > >>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf > >>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index > >>>>>>> 5e4a09c9..7bd81dd6 > >>>>>>> 100644 > >>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf > >>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf > >>>>>>> @@ -6,6 +6,8 @@ > >>>>>>>=20 > >>>>>>> BASE_DISTRO =3D "raspios" > >>>>>>> BASE_DISTRO_arm64 =3D "debian" > >>>>>>>=20 > >>>>>>> +BASE_HOST_DISTRO ?=3D "debian" > >>>>>>> + > >>>>>>>=20 > >>>>>>> BASE_DISTRO_CODENAME =3D "bullseye" > >>>>>>> HOST_DISTRO ?=3D "debian-bullseye" > >>>>>>> HOST_DISTRO_APT_SOURCES ?=3D "conf/distro/${HOST_DISTRO}.list" > >>>>>>>=20 > >>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf > >>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1 > >>>>>>> 100644 > >>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf > >>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf > >>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf > >>>>>>>=20 > >>>>>>> BASE_DISTRO =3D "ubuntu" > >>>>>>> BASE_DISTRO_CODENAME =3D "focal" > >>>>>>>=20 > >>>>>>> +BASE_HOST_DISTRO =3D "${BASE_DISTRO}" > >>>>>>> + > >>>>>>>=20 > >>>>>>> DISTRO_APT_SOURCES_arm64 ?=3D > >>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" > >>>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?=3D "conf/distro/${HOST_DISTRO}.l= ist > >>>>>>> conf/distro/${HOST_DISTRO}-ports.list"> > >>>>>>>=20 > >>>>>>> diff --git a/meta/conf/distro/debian-common.conf > >>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b > >>>>>>> 100644 > >>>>>>> --- a/meta/conf/distro/debian-common.conf > >>>>>>> +++ b/meta/conf/distro/debian-common.conf > >>>>>>> @@ -5,6 +5,8 @@ > >>>>>>>=20 > >>>>>>> BASE_DISTRO =3D "debian" > >>>>>>>=20 > >>>>>>> +BASE_HOST_DISTRO ?=3D "${BASE_DISTRO}" > >>>>>>> + > >>>>>>>=20 > >>>>>>> HOST_DISTRO_APT_SOURCES ?=3D "conf/distro/${HOST_DISTRO}.list" > >>>>>>> DISTRO_APT_SOURCES ?=3D > >>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> > >>>>>>>=20 > >>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > >>>>>>> f32d192e..2ee0d4cc 100644 > >>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg =3D > >>>>>>> ",gnupg" > >>>>>>>=20 > >>>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =3D > >>>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?=3D "${@'HOST_' if > >>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') =3D=3D '1' else ''}" BOOTSTRAP_DI= STRO =3D > >>>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') =3D= =3D '1' > >>>>>>> else > >>>>>>> 'DISTRO')}"> > >>>>>>>=20 > >>>>>>> +BOOTSTRAP_BASE_DISTRO =3D "${@d.getVar('BASE_HOST_DISTRO' if > >>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') =3D=3D '1' else 'BASE_DISTRO')}"> > >>>>>>>=20 > >>>>>>> FILESEXTRAPATHS_append =3D ":${BBPATH}" > >>>>>>> =20 > >>>>>>> inherit deb-dl-dir > >>>>>>>=20 > >>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, > >=20 > > aptsources_entry_list): > >>>>>>> import re > >>>>>>>=20 > >>>>>>> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')= ): > >>>>>>> - premirrors =3D "\S* file://${REPO_BASE_DIR}/${BASE_DISTR= O}\n" > >>>>>>> + premirrors =3D "\S* > >>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> > >>>>>>>=20 > >>>>>>> else: > >>>>>>> premirrors =3D d.getVar('DISTRO_APT_PREMIRRORS', True) o= r "" > >>>>>>> =20 > >>>>>>> mirror_list =3D [entry.split() > >>>>>>>=20 > >>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() { > >>>>>>>=20 > >>>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/boo= tst > >>>>>>> ra > >>>>>>> p" > >>>>>>> =20 > >>>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" > >>>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" =3D "1" ]; then > >>>>>>>=20 > >>>>>>> - line=3D"file:///base-apt/${BASE_DISTRO} > >>>>>>> ${BASE_DISTRO_CODENAME} > >>>>>>> main" + line=3D"file:///base-apt/${BOOTSTRAP_BASE_DIST= RO} > >>>>>>> ${BASE_DISTRO_CODENAME} main"> > >>>>>>>=20 > >>>>>>> if [ -z "${BASE_REPO_KEY}" ]; then > >>>>>>> =20 > >>>>>>> line=3D"[trusted=3Dyes] ${line}" > >>>>>>> =20 > >>>>>>> fi > >>>>>>> echo "deb ${line}" > > >>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" > >>>>>>>=20 > >>>>>>> + line=3D"file:///base-apt/${BASE_DISTRO} > >>>>>>> ${BASE_DISTRO_CODENAME} > >>>>>>> main" > >>>>>>=20 > >>>>>> Won't this cause duplicate lines (with complaints of apt) in case = of > >>>>>> BOOTSTRAP_BASE_DISTRO =3D=3D BASE_DISTRO? > >>>>>=20 > >>>>> No, while I don't add new lines (no new "echo"s). I just modify > >>>>> "deb-src" > >>>>> line while it should point to BASE_DISTRO even for host case. Witho= ut > >>>>> it, > >>>>> there was a problem with "hello" (apt_fetch failed to download cach= ed > >>>>> sources from "host" repo, while sources were cached in "target" one. > >>>>>=20 > >>>>> That's how it looks now: > >>>>> | $ cat > >>>>> | isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base- > >>>>>=20 > >>>>> apt.list > >>>>>=20 > >>>>> | deb [trusted=3Dyes] file:///base-apt/raspios bullseye main > >>>>> | deb-src [trusted=3Dyes] file:///base-apt/raspios bullseye main > >>>>> | $ cat > >>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt= =2Eli > >>>>> | st > >>>>> | deb [trusted=3Dyes] file:///base-apt/debian bullseye main > >>>>> | deb-src [trusted=3Dyes] file:///base-apt/raspios bullseye main > >>>>=20 > >>>> Indeed, I misintepreted the code. > >>>>=20 > >>>> But I think we still have an issue on the deb-src population side: > >>>>=20 > >>>> # ls build/downloads/deb-src/ > >>>> iot2050-debian > >>>>=20 > >>>> And that was with this patch applied. > >>>>=20 > >>>> Jan > >>>=20 > >>> Actually, I checked the case with the following configuration: > >>>=20 > >>> distro: my-debian > >>>=20 > >>> local_conf_header: > >>> nodebian: | > >>> =20 > >>> require conf/distro/debian-bullseye.conf > >>> DISTRO_NAME =3D "My Debian System" > >>>=20 > >>> So, I simulated renaming distro that is done in meta-iot2050. > >>>=20 > >>> Cached build was OK, "hello" package (that we build from sources) was > >>>=20 > >>> downloaded at first build to "my-debian" directory: > >>> | $ ls build/downloads/deb-src/my-debian/hello/ > >>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz > >>>=20 > >>> and second build picked the files with no problem. > >>>=20 > >>> The same with "raspios-bullseye" target - "hello" sources are downloa= ded > >>> to > >>>=20 > >>> the "target" dir: > >>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/ > >>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz > >>>=20 > >>> I even tested the case with BASE_REPO_FEATURES +=3D "cache-deb-src" = and > >>> it > >>> was OK (honestly, I don't know who could require cached build with th= is > >>> option enabled :) ). > >>>=20 > >>> Anyway, I'll also try to check my changes with meta-iot2050 layer also > >>> (despite some packages in it are currently not compatible with > >>> BB_NO_NETWORK option) > >>=20 > >> Please then test without the coral packages, they have an own issue /w= rt > >> offline builds. > >=20 > > Checked the patchset with `IOT2050_CORAL_SUPPORT =3D "0"` and it works >=20 > ...but it caches to the wrong downloads/deb-src path, no? >=20 > Jan Why wrong? | meta-iot2050/build $ tree -d downloads/deb* | downloads/deb | =E2=94=9C=E2=94=80=E2=94=80 debian-bullseye | =E2=94=94=E2=94=80=E2=94=80 iot2050-debian | downloads/deb-src | =E2=94=94=E2=94=80=E2=94=80 iot2050-debian | =E2=94=9C=E2=94=80=E2=94=80 openssl | =E2=94=94=E2=94=80=E2=94=80 swig `do_apt_fetch` works with `${DISTRO}` and downloads sources to the=20 corresponding directory. =2D-=20 Uladzimir Bely