From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7169469243442855936 X-Received: by 2002:a05:6638:4908:b0:389:a74a:4d63 with SMTP id cx8-20020a056638490800b00389a74a4d63mr9695165jab.42.1669721367331; Tue, 29 Nov 2022 03:29:27 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:984f:0:b0:6bc:c593:b6b3 with SMTP id p15-20020a5d984f000000b006bcc593b6b3ls1328063ios.6.-pod-prod-gmail; Tue, 29 Nov 2022 03:29:26 -0800 (PST) X-Google-Smtp-Source: AA0mqf5jbjRApDjN7A9NCAHuXiuLiJ+C2YqL+N1QTGPxJQjkPYRZ/Ug4WXIhUEFEtV4YBqLGExET X-Received: by 2002:a6b:8d87:0:b0:6dd:f2f4:b335 with SMTP id p129-20020a6b8d87000000b006ddf2f4b335mr16566817iod.99.1669721366666; Tue, 29 Nov 2022 03:29:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669721366; cv=none; d=google.com; s=arc-20160816; b=ZEsK5xYYUmNRXh2rCL3PuY17FW/OlaMC+cMt1v884pIkKYvHsRT40POpXMnFP9AR46 2ieMoFVMTuPrWsNe/Vskxd7xJ+5pLwhoL9Pcu/2tAC8Pn4qgkizNRncOCuljjVZr3XAc GgFEgY6aKvg6+UPKU52Zfqev5VcLkaY779MoUroPKIknX0VsYNcMtQjv0Xwp3JsJf68N lIYGYgQ4hMeDNyG8IzfRG4gjXxpUEYDjKJ4hk4cOgf83/IJYhxLIBoGriIigUrXkmCRj malmioDGl2crKOLcQX2eqoX7PR0xfYJgMBMDgefiM2SiJWp7rc2Ic81gQZxvc6+F7zeB uOuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=sEWee6hDrXyeRYbjsCCaYZ1je9TFmslu1vbq4ChaBE4=; b=gAIvfv6eBKlY5IO4r7rF0csZRjxxtEoYrOSaiLOKE251H5v4yUkV8CAn+5KCAcA49l QpKl9oJR9Vsa/rnTED4WdMLxHAsTrDUmypt1rGLDzubbfDCYb9Ke8rMTfYxWUuJm71s5 q4VOwNzpjeJwRiffAQgZvCxJUPdIP9nJtkTSpqZJXubI7CtNYP51Y4dUFPAhjckpe1OE JaNgEh1bX9aibtFRDviobzTnvxxEPCfkzBiy1V66xa9wbxl+92Zv3CUAMXimeixi7hJ2 NukCiB6Bpd9CFxYl0sKERbWyW+4kIsczh7XPcqmA4wY9fUPiW5U28LnJKkJ8CN6426Mb NO6g== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id r6-20020a6b5d06000000b006deecdea1dfsi574454iob.2.2022.11.29.03.29.26 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 29 Nov 2022 03:29:26 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from home.localnet (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 2ATBTNer000676 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 29 Nov 2022 12:29:24 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com, Jan Kiszka Subject: Re: [PATCH v4 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs Date: Tue, 29 Nov 2022 14:29:17 +0300 Message-ID: <4962749.Qq0lBPeGtt@home> In-Reply-To: <3638eaa3-18b8-f2f4-ac81-0dd90f930cb1@siemens.com> References: <20221128110534.31450-1-ubely@ilbers.de> <20221128110534.31450-2-ubely@ilbers.de> <3638eaa3-18b8-f2f4-ac81-0dd90f930cb1@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: ToX/OIbIqgWb In the email from Tuesday, 29 November 2022 13:49:11 +03 user Jan Kiszka wrote: > On 28.11.22 12:05, Uladzimir Bely wrote: > > Downstreams may want to change DISTRO to some custom value. > > When cross-building, this leads to downloading packages for host > > and target distros to the different download subdirs. > > > > While base-apt is populated only from DISTRO download subdir, > > second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to > > missing packages in base-apt. > > > > The similar issue happens with cross-building raspberry targets > > since they use their own mirrors and DISTRO. > > > > Fix the issue by splitting base-apt by introducing HOST_BASE_DISTRO > > variable that is used instead of BASE_DISTRO for caching builds > > for host-related components (e.g., isar-bootstrap, buildchroot-host, > > sbuild-chroot-host). Target components still use BASE_DISTRO. > > > > Signed-off-by: Uladzimir Bely > > --- > > > > meta-isar/conf/distro/raspbian-stretch.conf | 2 + > > meta-isar/conf/distro/raspios-bullseye.conf | 2 + > > meta-isar/conf/distro/ubuntu-focal.conf | 2 + > > meta/conf/distro/debian-common.conf | 2 + > > .../isar-bootstrap/isar-bootstrap.inc | 9 +++- > > meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++--------- > > 6 files changed, 40 insertions(+), 24 deletions(-) > > > > diff --git a/meta-isar/conf/distro/raspbian-stretch.conf > > b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..1ad0152f > > 100644 > > --- a/meta-isar/conf/distro/raspbian-stretch.conf > > +++ b/meta-isar/conf/distro/raspbian-stretch.conf > > @@ -6,6 +6,8 @@ > > > > BASE_DISTRO = "raspbian" > > BASE_DISTRO_CODENAME = "stretch" > > > > +HOST_BASE_DISTRO = "debian" > > + > > > > HOST_DISTRO ?= "debian-stretch" > > HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > > > > diff --git a/meta-isar/conf/distro/raspios-bullseye.conf > > b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..1f65c3cd > > 100644 > > --- a/meta-isar/conf/distro/raspios-bullseye.conf > > +++ b/meta-isar/conf/distro/raspios-bullseye.conf > > @@ -6,6 +6,8 @@ > > > > BASE_DISTRO = "raspios" > > BASE_DISTRO_arm64 = "debian" > > > > +HOST_BASE_DISTRO ?= "debian" > > + > > > > BASE_DISTRO_CODENAME = "bullseye" > > HOST_DISTRO ?= "debian-bullseye" > > HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > > > > diff --git a/meta-isar/conf/distro/ubuntu-focal.conf > > b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..084f8e3d 100644 > > --- a/meta-isar/conf/distro/ubuntu-focal.conf > > +++ b/meta-isar/conf/distro/ubuntu-focal.conf > > @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf > > > > BASE_DISTRO = "ubuntu" > > BASE_DISTRO_CODENAME = "focal" > > > > +HOST_BASE_DISTRO = "${BASE_DISTRO}" > > + > > > > DISTRO_APT_SOURCES_arm64 ?= > > "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" > > HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list > > conf/distro/${HOST_DISTRO}-ports.list"> > > diff --git a/meta/conf/distro/debian-common.conf > > b/meta/conf/distro/debian-common.conf index c18eebc2..ada7dc6e 100644 > > --- a/meta/conf/distro/debian-common.conf > > +++ b/meta/conf/distro/debian-common.conf > > @@ -5,6 +5,8 @@ > > > > BASE_DISTRO = "debian" > > > > +HOST_BASE_DISTRO ?= "${BASE_DISTRO}" > > + > > > > HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > > DISTRO_APT_SOURCES ?= > > "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > > f32d192e..14d876aa 100644 > > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg" > > > > DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = > > "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if > > d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO = > > "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else > > 'DISTRO')}"> > > +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if > > d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"> > > FILESEXTRAPATHS_append = ":${BBPATH}" > > > > inherit deb-dl-dir > > > > @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list): > > import re > > > > if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): > > - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" > > + premirrors = "\S* > > file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> > > else: > > premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or "" > > > > mirror_list = [entry.split() > > > > @@ -319,11 +320,15 @@ do_bootstrap() { > > > > "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap" > > > > mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" > > if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then > > > > - line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} > > main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} > > ${BASE_DISTRO_CODENAME} main" > There is still a conceptual glitch here: BASE_DISTRO_CODENAME may not be > the same as "HOST_BASE_DISTRO_CODENAME" - which does not exist. > > Jan > Currently there are no such cases (at least, I don't know any). Even if host/ target distros differ ("debian" and "raspbian"), they both use the same codename. Yes, technically we could add this variable "HOST_BASE_DISTRO_CODENAME" - only if we want, for instance, cross-build debian-based distros using ubuntu- based host or vice versa. But I'm not sure we really need it. By the way, I'm close to come with v5 of patchset that normalizes download directories. E.g., everything (both host and target packages) is in "download/ deb/debian-bullseye" even if DISTRO was changed. The only case when different directories are used if their BASE distros are different (raspberry case). Actually, there is still an issue with meta-iot2050 project. Deb files are duplicated (and, I guess, downloaded twice) in "downloads/deb", without "debian-bullseye". I didn't look at it deeply, but most probably it's related with incorrect usage of "deb_dl_dir_*" functions in "npm.bbclass". Since Isar commit "3afafbc" from Nov 26 2020 commit they should take 2 arguments, while in "meta-iot2050" layer they are called with only one (commit 69cef06a from Nov 23 2020). > > if [ -z "${BASE_REPO_KEY}" ]; then > > > > line="[trusted=yes] ${line}" > > > > fi > > echo "deb ${line}" > > > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" > > > > + line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} > > main" + if [ -z "${BASE_REPO_KEY}" ]; then > > + line="[trusted=yes] ${line}" > > + fi > > > > echo "deb-src ${line}" >> > > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" > > > > mkdir -p ${ROOTFSDIR}/base-apt > > > > diff --git a/meta/recipes-devtools/base-apt/base-apt.bb > > b/meta/recipes-devtools/base-apt/base-apt.bb index 506a28ff..05016d80 > > 100644 > > --- a/meta/recipes-devtools/base-apt/base-apt.bb > > +++ b/meta/recipes-devtools/base-apt/base-apt.bb > > @@ -13,7 +13,10 @@ KEYFILES ?= "" > > > > BASE_REPO_FEATURES ?= "" > > > > populate_base_apt() { > > > > - find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do > > + distro="${1}" > > + base_distro="${2}" > > + > > + find "${DEBDIR}"/"${distro}" -name '*\.deb' | while read package; do > > > > # NOTE: due to packages stored by reprepro are not modified, we > > can > > # use search by filename to check if package is already in repo. > > In > > # addition, md5sums are compared to ensure that the package is > > the > > > > @@ -22,25 +25,25 @@ populate_base_apt() { > > > > # Check if this package is already in base-apt > > ret=0 > > > > - repo_contains_package "${REPO_BASE_DIR}/${BASE_DISTRO}" > > "${package}" || + repo_contains_package > > "${REPO_BASE_DIR}/${base_distro}" "${package}" ||> > > ret=$? > > > > [ "${ret}" = "0" ] && continue > > if [ "${ret}" = "1" ]; then > > > > - repo_del_package "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > > - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > > + repo_del_package "${REPO_BASE_DIR}"/"${base_distro}" \ > > + "${REPO_BASE_DB_DIR}"/"${base_distro}" \ > > > > "${BASE_DISTRO_CODENAME}" \ > > > > - "${base_apt_p}" > > + "${package}" > > > > fi > > > > - repo_add_packages "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > > - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > > + repo_add_packages "${REPO_BASE_DIR}"/"${base_distro}" \ > > + "${REPO_BASE_DB_DIR}"/"${base_distro}" \ > > > > "${BASE_DISTRO_CODENAME}" \ > > "${package}" > > > > done > > > > - find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read package; > > do - repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ - > > "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > > + find "${DEBSRCDIR}"/"${distro}" -name '*\.dsc' | while read package; > > do + repo_add_srcpackage "${REPO_BASE_DIR}"/"${base_distro}" \ + > > "${REPO_BASE_DB_DIR}"/"${base_distro}" \ > > > > "${BASE_DISTRO_CODENAME}" \ > > "${package}" > > > > done > > > > @@ -55,20 +58,20 @@ repo() { > > > > "${BASE_DISTRO_CODENAME}" \ > > "${WORKDIR}/distributions.in" \ > > "${KEYFILES}" > > > > - > > - if [ -d '${BUILDCHROOT_HOST_DIR}/var/cache/apt' ] && > > - [ '${DISTRO}' != '${HOST_DISTRO}' ]; then > > - # We would need two separate repository paths for that. > > - # Otherwise packages (especially the 'all' arch ones) from one > > - # distribution can influence the package versions of the other > > - # distribution. > > - bbfatal "Different host and target distributions are currently > > not supported." \ - "Try it without cross-build." > > - fi > > - > > - populate_base_apt > > + populate_base_apt "${DISTRO}" "${BASE_DISTRO}" > > > > repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > > > > "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" > > > > + > > + if [ '${DISTRO}' != '${HOST_DISTRO}' ]; then > > + repo_create "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \ > > + "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}" \ > > + "${BASE_DISTRO_CODENAME}" \ > > + "${WORKDIR}/distributions.in" \ > > + "${KEYFILES}" > > + populate_base_apt "${HOST_DISTRO}" "${HOST_BASE_DISTRO}" > > + repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \ > > + "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}" > > + fi > > > > } > > > > python do_cache() { -- Uladzimir Bely