From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7169469243442855936 X-Received: by 2002:a05:6512:3996:b0:4b4:e9f8:88ce with SMTP id j22-20020a056512399600b004b4e9f888cemr3741223lfu.323.1669358176279; Thu, 24 Nov 2022 22:36:16 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:5e38:0:b0:494:6c7d:cf65 with SMTP id o24-20020ac25e38000000b004946c7dcf65ls2404315lfg.2.-pod-prod-gmail; Thu, 24 Nov 2022 22:36:14 -0800 (PST) X-Google-Smtp-Source: AA0mqf5sDMuEyqI3bklMe1351RHK3yqKbKscT/mKrxEJp2tJPIsrEz9q7sNvIaH3HdB1JzoVcwes X-Received: by 2002:a19:8c4b:0:b0:494:b21e:426f with SMTP id i11-20020a198c4b000000b00494b21e426fmr11304327lfj.204.1669358174814; Thu, 24 Nov 2022 22:36:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669358174; cv=none; d=google.com; s=arc-20160816; b=gHpM7c4ujYphmKnsXit+Lm/ZVigjlQ9gqNqy7+KXZIZNm4XuRGQ1jrUayTkbbnx67e or4FBmlYTqu+qh/KMu5yIzFtM8sOFP6F9cnVy6cCEjbm5EJ+GnwP/19rfC/hBiDVS7Bf i8yRnqjGrx/5sbFWeJT4NL8KiYF4+u17D/ka5/VecLdPd2PXKm9UY8XZRGRXfgmN4GFL Qei6Tfn16Rf5NT70N4BOqQsAtQCDGlPe0VPtAB+zi7eOd1GMIJw7bmq5b3d939boQ6Oq 8hu7F2tz+AlAWozr/Q7aU6mjWTgEW6Dwhstw9fyFRfkGf76hAprR1o1IbgUWH/p6JQF/ D3vQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=xz3qJwDN2G+dAe287a2Rlvr/DTQxK6Yz4slBQVbpsVE=; b=BuX2s0JJauOq0Uern5HQregRg8oInulSMbXvqwlB9K6MmZiApvYdpmUoigCgA9K9VT DYnXSFztzMS+SxfOFGYvED9GN12BNtj2+BNdFkIVKrBqR3519UZVPJ4MbJwgy/MIbiF0 gUFOn+qEdaXoSI3E0n2kQhjxWMNG3cgUDxrKDEEFx1/W8bVwcbdHtF/d2dfv7mSd1CYJ 8E3ssrT2v2hHdpBvEt3ByBzYm0koaBXzCq9Ke4aj6RYqo3fHxZDkRYHwScJFAos0+xQB IWpn1jWyXHrx2tBCPHixdE5VYNRkCVDbLPhFiV1CC14MURJitPP0Rbf5jR6TldT0yvbk JsVw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id u15-20020a05651c130f00b0027976ad74c9si154111lja.5.2022.11.24.22.36.14 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 24 Nov 2022 22:36:14 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from baighyz.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 2AP6aCKf014230 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 25 Nov 2022 07:36:13 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v2 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs Date: Fri, 25 Nov 2022 07:36:10 +0100 Message-Id: <20221125063612.6932-2-ubely@ilbers.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20221125063612.6932-1-ubely@ilbers.de> References: <20221125063612.6932-1-ubely@ilbers.de> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: 43FzT/+tBx+p Downstreams may want to change DISTRO to some custom value. When cross-building, this leads to downloading packages for host and target distros to the different download subdirs. While base-apt is populated only from DISTRO download subdir, second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to missing packages in base-apt. The similar issue happens with cross-building raspberry targets since they use their own mirrors and DISTRO. Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO variable that is used instead of BASE_DISTRO for caching builds for host-related components (e.g., isar-bootstrap, buildchroot-host, sbuild-chroot-host). Target components still use BASE_DISTRO. Signed-off-by: Uladzimir Bely --- meta-isar/conf/distro/raspbian-stretch.conf | 2 + meta-isar/conf/distro/raspios-bullseye.conf | 2 + meta-isar/conf/distro/ubuntu-focal.conf | 2 + meta/conf/distro/debian-common.conf | 2 + .../isar-bootstrap/isar-bootstrap.inc | 9 +++- meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++--------- 6 files changed, 40 insertions(+), 24 deletions(-) diff --git a/meta-isar/conf/distro/raspbian-stretch.conf b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..66d965c9 100644 --- a/meta-isar/conf/distro/raspbian-stretch.conf +++ b/meta-isar/conf/distro/raspbian-stretch.conf @@ -6,6 +6,8 @@ BASE_DISTRO = "raspbian" BASE_DISTRO_CODENAME = "stretch" +BASE_HOST_DISTRO = "debian" + HOST_DISTRO ?= "debian-stretch" HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" diff --git a/meta-isar/conf/distro/raspios-bullseye.conf b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..7bd81dd6 100644 --- a/meta-isar/conf/distro/raspios-bullseye.conf +++ b/meta-isar/conf/distro/raspios-bullseye.conf @@ -6,6 +6,8 @@ BASE_DISTRO = "raspios" BASE_DISTRO_arm64 = "debian" +BASE_HOST_DISTRO ?= "debian" + BASE_DISTRO_CODENAME = "bullseye" HOST_DISTRO ?= "debian-bullseye" HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" diff --git a/meta-isar/conf/distro/ubuntu-focal.conf b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1 100644 --- a/meta-isar/conf/distro/ubuntu-focal.conf +++ b/meta-isar/conf/distro/ubuntu-focal.conf @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf BASE_DISTRO = "ubuntu" BASE_DISTRO_CODENAME = "focal" +BASE_HOST_DISTRO = "${BASE_DISTRO}" + DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list conf/distro/${HOST_DISTRO}-ports.list" diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b 100644 --- a/meta/conf/distro/debian-common.conf +++ b/meta/conf/distro/debian-common.conf @@ -5,6 +5,8 @@ BASE_DISTRO = "debian" +BASE_HOST_DISTRO ?= "${BASE_DISTRO}" + HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" DISTRO_APT_SOURCES ?= "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list" diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index f32d192e..2ee0d4cc 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg" DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}" +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}" FILESEXTRAPATHS_append = ":${BBPATH}" inherit deb-dl-dir @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list): import re if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" + premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" else: premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or "" mirror_list = [entry.split() @@ -319,11 +320,15 @@ do_bootstrap() { "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap" mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then - line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main" if [ -z "${BASE_REPO_KEY}" ]; then line="[trusted=yes] ${line}" fi echo "deb ${line}" > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" + line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main" + if [ -z "${BASE_REPO_KEY}" ]; then + line="[trusted=yes] ${line}" + fi echo "deb-src ${line}" >> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" mkdir -p ${ROOTFSDIR}/base-apt diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb index 506a28ff..e928cd52 100644 --- a/meta/recipes-devtools/base-apt/base-apt.bb +++ b/meta/recipes-devtools/base-apt/base-apt.bb @@ -13,7 +13,10 @@ KEYFILES ?= "" BASE_REPO_FEATURES ?= "" populate_base_apt() { - find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do + distro="${1}" + base_distro="${2}" + + find "${DEBDIR}"/"${distro}" -name '*\.deb' | while read package; do # NOTE: due to packages stored by reprepro are not modified, we can # use search by filename to check if package is already in repo. In # addition, md5sums are compared to ensure that the package is the @@ -22,25 +25,25 @@ populate_base_apt() { # Check if this package is already in base-apt ret=0 - repo_contains_package "${REPO_BASE_DIR}/${BASE_DISTRO}" "${package}" || + repo_contains_package "${REPO_BASE_DIR}/${base_distro}" "${package}" || ret=$? [ "${ret}" = "0" ] && continue if [ "${ret}" = "1" ]; then - repo_del_package "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ + repo_del_package "${REPO_BASE_DIR}"/"${base_distro}" \ + "${REPO_BASE_DB_DIR}"/"${base_distro}" \ "${BASE_DISTRO_CODENAME}" \ - "${base_apt_p}" + "${package}" fi - repo_add_packages "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ + repo_add_packages "${REPO_BASE_DIR}"/"${base_distro}" \ + "${REPO_BASE_DB_DIR}"/"${base_distro}" \ "${BASE_DISTRO_CODENAME}" \ "${package}" done - find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read package; do - repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ + find "${DEBSRCDIR}"/"${distro}" -name '*\.dsc' | while read package; do + repo_add_srcpackage "${REPO_BASE_DIR}"/"${base_distro}" \ + "${REPO_BASE_DB_DIR}"/"${base_distro}" \ "${BASE_DISTRO_CODENAME}" \ "${package}" done @@ -55,20 +58,20 @@ repo() { "${BASE_DISTRO_CODENAME}" \ "${WORKDIR}/distributions.in" \ "${KEYFILES}" - - if [ -d '${BUILDCHROOT_HOST_DIR}/var/cache/apt' ] && - [ '${DISTRO}' != '${HOST_DISTRO}' ]; then - # We would need two separate repository paths for that. - # Otherwise packages (especially the 'all' arch ones) from one - # distribution can influence the package versions of the other - # distribution. - bbfatal "Different host and target distributions are currently not supported." \ - "Try it without cross-build." - fi - - populate_base_apt + populate_base_apt "${DISTRO}" "${BASE_DISTRO}" repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" + + if [ '${DISTRO}' != '${HOST_DISTRO}' ]; then + repo_create "${REPO_BASE_DIR}"/"${BASE_HOST_DISTRO}" \ + "${REPO_BASE_DB_DIR}"/"${BASE_HOST_DISTRO}" \ + "${BASE_DISTRO_CODENAME}" \ + "${WORKDIR}/distributions.in" \ + "${KEYFILES}" + populate_base_apt "${HOST_DISTRO}" "${BASE_HOST_DISTRO}" + repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_HOST_DISTRO}" \ + "${REPO_BASE_DB_DIR}"/"${BASE_HOST_DISTRO}" + fi } python do_cache() { -- 2.20.1