From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7169469243442855936 X-Received: by 2002:a05:600c:20b:b0:3cf:f2aa:4f24 with SMTP id 11-20020a05600c020b00b003cff2aa4f24mr25707479wmi.48.1669362993030; Thu, 24 Nov 2022 23:56:33 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:f211:0:b0:3cf:afd2:ab84 with SMTP id s17-20020a1cf211000000b003cfafd2ab84ls2180506wmc.2.-pod-control-gmail; Thu, 24 Nov 2022 23:56:31 -0800 (PST) X-Google-Smtp-Source: AA0mqf5VLLBsQmVB8j4HLgke6gshpKS5hkKIhr/QUqCXWmUGw296mGBzvAIPmEj+3AkSSCO2XSwA X-Received: by 2002:a7b:ca53:0:b0:3cf:74ef:3313 with SMTP id m19-20020a7bca53000000b003cf74ef3313mr14842404wml.41.1669362991649; Thu, 24 Nov 2022 23:56:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669362991; cv=none; d=google.com; s=arc-20160816; b=XPjBttpKwCXH6y7wu4I/esutgdfBLUcpmPs84Itc65h73RPOs3vZu1CN10LtEVhkV8 4cIaRjjPaLbT8TVoFuyQz0TVqrmoveF0nAFZPLEDr+UppOXDrUfNN9GRc6mTzxbBIf1W DHjnUgtyEqBp17fXJVYwW30YTS3qIf0PN8DDu+X2KmbBcnQVZ4J5Nj9QbGsfQvUGojKT 4eOF0Im6nQQBbOXsvWbMIdAcyKfV6OMnPyJri1EFBVdx/DANAHp5kUTVeVfwt4oLn4tT N7YTCmJQoqWOCfaSOL+Z/KFAan2o9+S1KRzT+Gx6m5DhMH5I86Bnfp1Aw+idnsegAVJn VLlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=HU2MKjcHcPBOsD2HBt5ODFZQ7iGNyAYpw4uubVjPvQ8=; b=xSgANQyhb/rs7FIA4mI8NGVSN5pHeCfOGaxE5nMU6HdXhCwYB4FjVvAf3sIb/UN4Rp jFPeU65U64phgIE8PFkIG/NhA5Rg5SSvrg8c6Q8WrDqezZ3iam2tjywnHnkdgYs7NYgV /+s5m8xMoy5TL/QRBhxT4NIUOabKfgGJvmmbh23hSI8NsyQZDIrqsok6MU5AWzYcUlFx HsFkgroDHiDmFRsbahKZEegX/iLnCVl/ewN/gYiCZFgh26Es5IP2CiqVk8DaCJufCXuH F3HHY90hlgtxCnLha8dBSpnjmwFsQRKPVSldE3N7y8cO1mVo+Mpk5IKClg+9kA9iyBzR ywzA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id by15-20020a056000098f00b00236e8baff63si142352wrb.0.2022.11.24.23.56.31 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 24 Nov 2022 23:56:31 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from home.localnet (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 2AP7uUKN014700 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 25 Nov 2022 08:56:30 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com, Jan Kiszka Subject: Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs Date: Fri, 25 Nov 2022 10:56:24 +0300 Message-ID: <2134415.yiUUSuA9gR@home> In-Reply-To: References: <20221125064253.21907-1-ubely@ilbers.de> <20221125064253.21907-2-ubely@ilbers.de> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: teENcXKH5DiF In the email from Friday, 25 November 2022 10:42:00 +03 user Jan Kiszka wrote: > On 25.11.22 07:42, Uladzimir Bely wrote: > > Downstreams may want to change DISTRO to some custom value. > > When cross-building, this leads to downloading packages for host > > and target distros to the different download subdirs. > > > > While base-apt is populated only from DISTRO download subdir, > > second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to > > missing packages in base-apt. > > > > The similar issue happens with cross-building raspberry targets > > since they use their own mirrors and DISTRO. > > > > Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO > > variable that is used instead of BASE_DISTRO for caching builds > > for host-related components (e.g., isar-bootstrap, buildchroot-host, > > sbuild-chroot-host). Target components still use BASE_DISTRO. > > > > Signed-off-by: Uladzimir Bely > > --- > > > > meta-isar/conf/distro/raspbian-stretch.conf | 2 + > > meta-isar/conf/distro/raspios-bullseye.conf | 2 + > > meta-isar/conf/distro/ubuntu-focal.conf | 2 + > > meta/conf/distro/debian-common.conf | 2 + > > .../isar-bootstrap/isar-bootstrap.inc | 9 +++- > > meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++--------- > > 6 files changed, 40 insertions(+), 24 deletions(-) > > > > diff --git a/meta-isar/conf/distro/raspbian-stretch.conf > > b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..66d965c9 > > 100644 > > --- a/meta-isar/conf/distro/raspbian-stretch.conf > > +++ b/meta-isar/conf/distro/raspbian-stretch.conf > > @@ -6,6 +6,8 @@ > > > > BASE_DISTRO = "raspbian" > > BASE_DISTRO_CODENAME = "stretch" > > > > +BASE_HOST_DISTRO = "debian" > > Better prefix it with HOST, to be consistent: > > HOST_BASE_DISTRO > Not a problem, if you think it would be better. > > + > > > > HOST_DISTRO ?= "debian-stretch" > > HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > > > > diff --git a/meta-isar/conf/distro/raspios-bullseye.conf > > b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..7bd81dd6 > > 100644 > > --- a/meta-isar/conf/distro/raspios-bullseye.conf > > +++ b/meta-isar/conf/distro/raspios-bullseye.conf > > @@ -6,6 +6,8 @@ > > > > BASE_DISTRO = "raspios" > > BASE_DISTRO_arm64 = "debian" > > > > +BASE_HOST_DISTRO ?= "debian" > > + > > > > BASE_DISTRO_CODENAME = "bullseye" > > HOST_DISTRO ?= "debian-bullseye" > > HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > > > > diff --git a/meta-isar/conf/distro/ubuntu-focal.conf > > b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1 100644 > > --- a/meta-isar/conf/distro/ubuntu-focal.conf > > +++ b/meta-isar/conf/distro/ubuntu-focal.conf > > @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf > > > > BASE_DISTRO = "ubuntu" > > BASE_DISTRO_CODENAME = "focal" > > > > +BASE_HOST_DISTRO = "${BASE_DISTRO}" > > + > > > > DISTRO_APT_SOURCES_arm64 ?= > > "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" > > HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list > > conf/distro/${HOST_DISTRO}-ports.list"> > > diff --git a/meta/conf/distro/debian-common.conf > > b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b 100644 > > --- a/meta/conf/distro/debian-common.conf > > +++ b/meta/conf/distro/debian-common.conf > > @@ -5,6 +5,8 @@ > > > > BASE_DISTRO = "debian" > > > > +BASE_HOST_DISTRO ?= "${BASE_DISTRO}" > > + > > > > HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > > DISTRO_APT_SOURCES ?= > > "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > > f32d192e..2ee0d4cc 100644 > > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg" > > > > DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = > > "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if > > d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO = > > "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else > > 'DISTRO')}"> > > +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if > > d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"> > > FILESEXTRAPATHS_append = ":${BBPATH}" > > > > inherit deb-dl-dir > > > > @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list): > > import re > > > > if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): > > - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" > > + premirrors = "\S* > > file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> > > else: > > premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or "" > > > > mirror_list = [entry.split() > > > > @@ -319,11 +320,15 @@ do_bootstrap() { > > > > "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap" > > > > mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" > > if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then > > > > - line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} > > main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} > > ${BASE_DISTRO_CODENAME} main"> > > if [ -z "${BASE_REPO_KEY}" ]; then > > > > line="[trusted=yes] ${line}" > > > > fi > > echo "deb ${line}" > > > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" > > > > + line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} > > main" > Won't this cause duplicate lines (with complaints of apt) in case of > BOOTSTRAP_BASE_DISTRO == BASE_DISTRO? No, while I don't add new lines (no new "echo"s). I just modify "deb-src" line while it should point to BASE_DISTRO even for host case. Without it, there was a problem with "hello" (apt_fetch failed to download cached sources from "host" repo, while sources were cached in "target" one. That's how it looks now: | $ cat isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base- apt.list | deb [trusted=yes] file:///base-apt/raspios bullseye main | deb-src [trusted=yes] file:///base-apt/raspios bullseye main | $ cat isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.list | deb [trusted=yes] file:///base-apt/debian bullseye main | deb-src [trusted=yes] file:///base-apt/raspios bullseye main > > > + if [ -z "${BASE_REPO_KEY}" ]; then > > + line="[trusted=yes] ${line}" > > + fi > > > > echo "deb-src ${line}" >> > > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" > > > > mkdir -p ${ROOTFSDIR}/base-apt > > > > diff --git a/meta/recipes-devtools/base-apt/base-apt.bb > > b/meta/recipes-devtools/base-apt/base-apt.bb index 506a28ff..e928cd52 > > 100644 > > --- a/meta/recipes-devtools/base-apt/base-apt.bb > > +++ b/meta/recipes-devtools/base-apt/base-apt.bb > > @@ -13,7 +13,10 @@ KEYFILES ?= "" > > > > BASE_REPO_FEATURES ?= "" > > > > populate_base_apt() { > > > > - find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do > > + distro="${1}" > > + base_distro="${2}" > > + > > + find "${DEBDIR}"/"${distro}" -name '*\.deb' | while read package; do > > > > # NOTE: due to packages stored by reprepro are not modified, we > > can > > # use search by filename to check if package is already in repo. > > In > > # addition, md5sums are compared to ensure that the package is > > the > > > > @@ -22,25 +25,25 @@ populate_base_apt() { > > > > # Check if this package is already in base-apt > > ret=0 > > > > - repo_contains_package "${REPO_BASE_DIR}/${BASE_DISTRO}" > > "${package}" || + repo_contains_package > > "${REPO_BASE_DIR}/${base_distro}" "${package}" ||> > > ret=$? > > > > [ "${ret}" = "0" ] && continue > > if [ "${ret}" = "1" ]; then > > > > - repo_del_package "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > > - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > > + repo_del_package "${REPO_BASE_DIR}"/"${base_distro}" \ > > + "${REPO_BASE_DB_DIR}"/"${base_distro}" \ > > > > "${BASE_DISTRO_CODENAME}" \ > > > > - "${base_apt_p}" > > + "${package}" > > > > fi > > > > - repo_add_packages "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > > - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > > + repo_add_packages "${REPO_BASE_DIR}"/"${base_distro}" \ > > + "${REPO_BASE_DB_DIR}"/"${base_distro}" \ > > > > "${BASE_DISTRO_CODENAME}" \ > > "${package}" > > > > done > > > > - find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read package; > > do - repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ - > > "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > > + find "${DEBSRCDIR}"/"${distro}" -name '*\.dsc' | while read package; > > do + repo_add_srcpackage "${REPO_BASE_DIR}"/"${base_distro}" \ + > > "${REPO_BASE_DB_DIR}"/"${base_distro}" \ > > > > "${BASE_DISTRO_CODENAME}" \ > > "${package}" > > > > done > > > > @@ -55,20 +58,20 @@ repo() { > > > > "${BASE_DISTRO_CODENAME}" \ > > "${WORKDIR}/distributions.in" \ > > "${KEYFILES}" > > > > - > > - if [ -d '${BUILDCHROOT_HOST_DIR}/var/cache/apt' ] && > > - [ '${DISTRO}' != '${HOST_DISTRO}' ]; then > > - # We would need two separate repository paths for that. > > - # Otherwise packages (especially the 'all' arch ones) from one > > - # distribution can influence the package versions of the other > > - # distribution. > > - bbfatal "Different host and target distributions are currently > > not supported." \ - "Try it without cross-build." > > - fi > > - > > - populate_base_apt > > + populate_base_apt "${DISTRO}" "${BASE_DISTRO}" > > > > repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > > > > "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" > > > > + > > + if [ '${DISTRO}' != '${HOST_DISTRO}' ]; then > > + repo_create "${REPO_BASE_DIR}"/"${BASE_HOST_DISTRO}" \ > > + "${REPO_BASE_DB_DIR}"/"${BASE_HOST_DISTRO}" \ > > + "${BASE_DISTRO_CODENAME}" \ > > + "${WORKDIR}/distributions.in" \ > > + "${KEYFILES}" > > + populate_base_apt "${HOST_DISTRO}" "${BASE_HOST_DISTRO}" > > + repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_HOST_DISTRO}" \ > > + "${REPO_BASE_DB_DIR}"/"${BASE_HOST_DISTRO}" > > + fi > > > > } > > > > python do_cache() { > > Looks promising! > > Jan -- Uladzimir Bely