* [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used
@ 2022-11-25 6:42 Uladzimir Bely
2022-11-25 6:42 ` [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs Uladzimir Bely
` (3 more replies)
0 siblings, 4 replies; 18+ messages in thread
From: Uladzimir Bely @ 2022-11-25 6:42 UTC (permalink / raw)
To: isar-users
This patchset is mostly intended for downstreams that prefer
to rename DISTRO variable (like `meat-iot2050`) and use
ISAR_CROSS_COMPILE = "1".
This also fixes the case with cross-build or raspberry targets
from cached base-apt repo. While it uses separate repositories
for host (Debian) and target (Raspbian) distros, base-apt is
splittend onto two parts corresponding to original repos.
Also, testsuite repro tests were updated to cover crosss-build case
and raspberry target is now used in it.
Changes since v2:
- Cleaned garbage in commit message for patch 2.
Changes since v1:
- Support cached cross-build from base-apt for different base distros
for host and target (e.g. raspberry).
Uladzimir Bely (3):
Populate base-apt from both DISTRO and HOST_DISTRO download dirs
testsuite: Run signed repro test in cross mode
testsuite: Replace bananapi with rpi-arm-v7 in signed repro tests
meta-isar/conf/distro/raspbian-stretch.conf | 2 +
meta-isar/conf/distro/raspios-bullseye.conf | 2 +
meta-isar/conf/distro/ubuntu-focal.conf | 2 +
meta/conf/distro/debian-common.conf | 2 +
.../isar-bootstrap/isar-bootstrap.inc | 9 +++-
meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++---------
testsuite/citest.py | 4 +-
7 files changed, 42 insertions(+), 26 deletions(-)
--
2.20.1
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs 2022-11-25 6:42 [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Uladzimir Bely @ 2022-11-25 6:42 ` Uladzimir Bely 2022-11-25 7:42 ` Jan Kiszka 2022-11-25 6:42 ` [PATCH v3 2/3] testsuite: Run signed repro test in cross mode Uladzimir Bely ` (2 subsequent siblings) 3 siblings, 1 reply; 18+ messages in thread From: Uladzimir Bely @ 2022-11-25 6:42 UTC (permalink / raw) To: isar-users Downstreams may want to change DISTRO to some custom value. When cross-building, this leads to downloading packages for host and target distros to the different download subdirs. While base-apt is populated only from DISTRO download subdir, second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to missing packages in base-apt. The similar issue happens with cross-building raspberry targets since they use their own mirrors and DISTRO. Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO variable that is used instead of BASE_DISTRO for caching builds for host-related components (e.g., isar-bootstrap, buildchroot-host, sbuild-chroot-host). Target components still use BASE_DISTRO. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- meta-isar/conf/distro/raspbian-stretch.conf | 2 + meta-isar/conf/distro/raspios-bullseye.conf | 2 + meta-isar/conf/distro/ubuntu-focal.conf | 2 + meta/conf/distro/debian-common.conf | 2 + .../isar-bootstrap/isar-bootstrap.inc | 9 +++- meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++--------- 6 files changed, 40 insertions(+), 24 deletions(-) diff --git a/meta-isar/conf/distro/raspbian-stretch.conf b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..66d965c9 100644 --- a/meta-isar/conf/distro/raspbian-stretch.conf +++ b/meta-isar/conf/distro/raspbian-stretch.conf @@ -6,6 +6,8 @@ BASE_DISTRO = "raspbian" BASE_DISTRO_CODENAME = "stretch" +BASE_HOST_DISTRO = "debian" + HOST_DISTRO ?= "debian-stretch" HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" diff --git a/meta-isar/conf/distro/raspios-bullseye.conf b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..7bd81dd6 100644 --- a/meta-isar/conf/distro/raspios-bullseye.conf +++ b/meta-isar/conf/distro/raspios-bullseye.conf @@ -6,6 +6,8 @@ BASE_DISTRO = "raspios" BASE_DISTRO_arm64 = "debian" +BASE_HOST_DISTRO ?= "debian" + BASE_DISTRO_CODENAME = "bullseye" HOST_DISTRO ?= "debian-bullseye" HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" diff --git a/meta-isar/conf/distro/ubuntu-focal.conf b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1 100644 --- a/meta-isar/conf/distro/ubuntu-focal.conf +++ b/meta-isar/conf/distro/ubuntu-focal.conf @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf BASE_DISTRO = "ubuntu" BASE_DISTRO_CODENAME = "focal" +BASE_HOST_DISTRO = "${BASE_DISTRO}" + DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list conf/distro/${HOST_DISTRO}-ports.list" diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b 100644 --- a/meta/conf/distro/debian-common.conf +++ b/meta/conf/distro/debian-common.conf @@ -5,6 +5,8 @@ BASE_DISTRO = "debian" +BASE_HOST_DISTRO ?= "${BASE_DISTRO}" + HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" DISTRO_APT_SOURCES ?= "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list" diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index f32d192e..2ee0d4cc 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg" DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}" +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}" FILESEXTRAPATHS_append = ":${BBPATH}" inherit deb-dl-dir @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list): import re if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" + premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" else: premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or "" mirror_list = [entry.split() @@ -319,11 +320,15 @@ do_bootstrap() { "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap" mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then - line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main" if [ -z "${BASE_REPO_KEY}" ]; then line="[trusted=yes] ${line}" fi echo "deb ${line}" > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" + line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main" + if [ -z "${BASE_REPO_KEY}" ]; then + line="[trusted=yes] ${line}" + fi echo "deb-src ${line}" >> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" mkdir -p ${ROOTFSDIR}/base-apt diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb index 506a28ff..e928cd52 100644 --- a/meta/recipes-devtools/base-apt/base-apt.bb +++ b/meta/recipes-devtools/base-apt/base-apt.bb @@ -13,7 +13,10 @@ KEYFILES ?= "" BASE_REPO_FEATURES ?= "" populate_base_apt() { - find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do + distro="${1}" + base_distro="${2}" + + find "${DEBDIR}"/"${distro}" -name '*\.deb' | while read package; do # NOTE: due to packages stored by reprepro are not modified, we can # use search by filename to check if package is already in repo. In # addition, md5sums are compared to ensure that the package is the @@ -22,25 +25,25 @@ populate_base_apt() { # Check if this package is already in base-apt ret=0 - repo_contains_package "${REPO_BASE_DIR}/${BASE_DISTRO}" "${package}" || + repo_contains_package "${REPO_BASE_DIR}/${base_distro}" "${package}" || ret=$? [ "${ret}" = "0" ] && continue if [ "${ret}" = "1" ]; then - repo_del_package "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ + repo_del_package "${REPO_BASE_DIR}"/"${base_distro}" \ + "${REPO_BASE_DB_DIR}"/"${base_distro}" \ "${BASE_DISTRO_CODENAME}" \ - "${base_apt_p}" + "${package}" fi - repo_add_packages "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ + repo_add_packages "${REPO_BASE_DIR}"/"${base_distro}" \ + "${REPO_BASE_DB_DIR}"/"${base_distro}" \ "${BASE_DISTRO_CODENAME}" \ "${package}" done - find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read package; do - repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ + find "${DEBSRCDIR}"/"${distro}" -name '*\.dsc' | while read package; do + repo_add_srcpackage "${REPO_BASE_DIR}"/"${base_distro}" \ + "${REPO_BASE_DB_DIR}"/"${base_distro}" \ "${BASE_DISTRO_CODENAME}" \ "${package}" done @@ -55,20 +58,20 @@ repo() { "${BASE_DISTRO_CODENAME}" \ "${WORKDIR}/distributions.in" \ "${KEYFILES}" - - if [ -d '${BUILDCHROOT_HOST_DIR}/var/cache/apt' ] && - [ '${DISTRO}' != '${HOST_DISTRO}' ]; then - # We would need two separate repository paths for that. - # Otherwise packages (especially the 'all' arch ones) from one - # distribution can influence the package versions of the other - # distribution. - bbfatal "Different host and target distributions are currently not supported." \ - "Try it without cross-build." - fi - - populate_base_apt + populate_base_apt "${DISTRO}" "${BASE_DISTRO}" repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" + + if [ '${DISTRO}' != '${HOST_DISTRO}' ]; then + repo_create "${REPO_BASE_DIR}"/"${BASE_HOST_DISTRO}" \ + "${REPO_BASE_DB_DIR}"/"${BASE_HOST_DISTRO}" \ + "${BASE_DISTRO_CODENAME}" \ + "${WORKDIR}/distributions.in" \ + "${KEYFILES}" + populate_base_apt "${HOST_DISTRO}" "${BASE_HOST_DISTRO}" + repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_HOST_DISTRO}" \ + "${REPO_BASE_DB_DIR}"/"${BASE_HOST_DISTRO}" + fi } python do_cache() { -- 2.20.1 ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs 2022-11-25 6:42 ` [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs Uladzimir Bely @ 2022-11-25 7:42 ` Jan Kiszka 2022-11-25 7:56 ` Uladzimir Bely 0 siblings, 1 reply; 18+ messages in thread From: Jan Kiszka @ 2022-11-25 7:42 UTC (permalink / raw) To: Uladzimir Bely, isar-users On 25.11.22 07:42, Uladzimir Bely wrote: > Downstreams may want to change DISTRO to some custom value. > When cross-building, this leads to downloading packages for host > and target distros to the different download subdirs. > > While base-apt is populated only from DISTRO download subdir, > second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to > missing packages in base-apt. > > The similar issue happens with cross-building raspberry targets > since they use their own mirrors and DISTRO. > > Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO > variable that is used instead of BASE_DISTRO for caching builds > for host-related components (e.g., isar-bootstrap, buildchroot-host, > sbuild-chroot-host). Target components still use BASE_DISTRO. > > Signed-off-by: Uladzimir Bely <ubely@ilbers.de> > --- > meta-isar/conf/distro/raspbian-stretch.conf | 2 + > meta-isar/conf/distro/raspios-bullseye.conf | 2 + > meta-isar/conf/distro/ubuntu-focal.conf | 2 + > meta/conf/distro/debian-common.conf | 2 + > .../isar-bootstrap/isar-bootstrap.inc | 9 +++- > meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++--------- > 6 files changed, 40 insertions(+), 24 deletions(-) > > diff --git a/meta-isar/conf/distro/raspbian-stretch.conf b/meta-isar/conf/distro/raspbian-stretch.conf > index c8f523b3..66d965c9 100644 > --- a/meta-isar/conf/distro/raspbian-stretch.conf > +++ b/meta-isar/conf/distro/raspbian-stretch.conf > @@ -6,6 +6,8 @@ > BASE_DISTRO = "raspbian" > BASE_DISTRO_CODENAME = "stretch" > > +BASE_HOST_DISTRO = "debian" Better prefix it with HOST, to be consistent: HOST_BASE_DISTRO > + > HOST_DISTRO ?= "debian-stretch" > HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > > diff --git a/meta-isar/conf/distro/raspios-bullseye.conf b/meta-isar/conf/distro/raspios-bullseye.conf > index 5e4a09c9..7bd81dd6 100644 > --- a/meta-isar/conf/distro/raspios-bullseye.conf > +++ b/meta-isar/conf/distro/raspios-bullseye.conf > @@ -6,6 +6,8 @@ > BASE_DISTRO = "raspios" > BASE_DISTRO_arm64 = "debian" > > +BASE_HOST_DISTRO ?= "debian" > + > BASE_DISTRO_CODENAME = "bullseye" > HOST_DISTRO ?= "debian-bullseye" > HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > diff --git a/meta-isar/conf/distro/ubuntu-focal.conf b/meta-isar/conf/distro/ubuntu-focal.conf > index 379b5b30..75da79a1 100644 > --- a/meta-isar/conf/distro/ubuntu-focal.conf > +++ b/meta-isar/conf/distro/ubuntu-focal.conf > @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf > BASE_DISTRO = "ubuntu" > BASE_DISTRO_CODENAME = "focal" > > +BASE_HOST_DISTRO = "${BASE_DISTRO}" > + > DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" > HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list conf/distro/${HOST_DISTRO}-ports.list" > > diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf > index c18eebc2..d7a13f4b 100644 > --- a/meta/conf/distro/debian-common.conf > +++ b/meta/conf/distro/debian-common.conf > @@ -5,6 +5,8 @@ > > BASE_DISTRO = "debian" > > +BASE_HOST_DISTRO ?= "${BASE_DISTRO}" > + > HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > DISTRO_APT_SOURCES ?= "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list" > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > index f32d192e..2ee0d4cc 100644 > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg" > DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "${@https_support(d)}" > DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" > BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}" > +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}" > FILESEXTRAPATHS_append = ":${BBPATH}" > > inherit deb-dl-dir > @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list): > import re > > if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): > - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" > + premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" > else: > premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or "" > mirror_list = [entry.split() > @@ -319,11 +320,15 @@ do_bootstrap() { > "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap" > mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" > if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then > - line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main" > + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main" > if [ -z "${BASE_REPO_KEY}" ]; then > line="[trusted=yes] ${line}" > fi > echo "deb ${line}" > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" > + line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main" Won't this cause duplicate lines (with complaints of apt) in case of BOOTSTRAP_BASE_DISTRO == BASE_DISTRO? > + if [ -z "${BASE_REPO_KEY}" ]; then > + line="[trusted=yes] ${line}" > + fi > echo "deb-src ${line}" >> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" > > mkdir -p ${ROOTFSDIR}/base-apt > diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb > index 506a28ff..e928cd52 100644 > --- a/meta/recipes-devtools/base-apt/base-apt.bb > +++ b/meta/recipes-devtools/base-apt/base-apt.bb > @@ -13,7 +13,10 @@ KEYFILES ?= "" > BASE_REPO_FEATURES ?= "" > > populate_base_apt() { > - find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do > + distro="${1}" > + base_distro="${2}" > + > + find "${DEBDIR}"/"${distro}" -name '*\.deb' | while read package; do > # NOTE: due to packages stored by reprepro are not modified, we can > # use search by filename to check if package is already in repo. In > # addition, md5sums are compared to ensure that the package is the > @@ -22,25 +25,25 @@ populate_base_apt() { > > # Check if this package is already in base-apt > ret=0 > - repo_contains_package "${REPO_BASE_DIR}/${BASE_DISTRO}" "${package}" || > + repo_contains_package "${REPO_BASE_DIR}/${base_distro}" "${package}" || > ret=$? > [ "${ret}" = "0" ] && continue > if [ "${ret}" = "1" ]; then > - repo_del_package "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > + repo_del_package "${REPO_BASE_DIR}"/"${base_distro}" \ > + "${REPO_BASE_DB_DIR}"/"${base_distro}" \ > "${BASE_DISTRO_CODENAME}" \ > - "${base_apt_p}" > + "${package}" > fi > > - repo_add_packages "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > + repo_add_packages "${REPO_BASE_DIR}"/"${base_distro}" \ > + "${REPO_BASE_DB_DIR}"/"${base_distro}" \ > "${BASE_DISTRO_CODENAME}" \ > "${package}" > done > > - find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read package; do > - repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > + find "${DEBSRCDIR}"/"${distro}" -name '*\.dsc' | while read package; do > + repo_add_srcpackage "${REPO_BASE_DIR}"/"${base_distro}" \ > + "${REPO_BASE_DB_DIR}"/"${base_distro}" \ > "${BASE_DISTRO_CODENAME}" \ > "${package}" > done > @@ -55,20 +58,20 @@ repo() { > "${BASE_DISTRO_CODENAME}" \ > "${WORKDIR}/distributions.in" \ > "${KEYFILES}" > - > - if [ -d '${BUILDCHROOT_HOST_DIR}/var/cache/apt' ] && > - [ '${DISTRO}' != '${HOST_DISTRO}' ]; then > - # We would need two separate repository paths for that. > - # Otherwise packages (especially the 'all' arch ones) from one > - # distribution can influence the package versions of the other > - # distribution. > - bbfatal "Different host and target distributions are currently not supported." \ > - "Try it without cross-build." > - fi > - > - populate_base_apt > + populate_base_apt "${DISTRO}" "${BASE_DISTRO}" > repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" > + > + if [ '${DISTRO}' != '${HOST_DISTRO}' ]; then > + repo_create "${REPO_BASE_DIR}"/"${BASE_HOST_DISTRO}" \ > + "${REPO_BASE_DB_DIR}"/"${BASE_HOST_DISTRO}" \ > + "${BASE_DISTRO_CODENAME}" \ > + "${WORKDIR}/distributions.in" \ > + "${KEYFILES}" > + populate_base_apt "${HOST_DISTRO}" "${BASE_HOST_DISTRO}" > + repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_HOST_DISTRO}" \ > + "${REPO_BASE_DB_DIR}"/"${BASE_HOST_DISTRO}" > + fi > } > > python do_cache() { Looks promising! Jan -- Siemens AG, Technology Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs 2022-11-25 7:42 ` Jan Kiszka @ 2022-11-25 7:56 ` Uladzimir Bely 2022-11-25 14:11 ` Jan Kiszka 0 siblings, 1 reply; 18+ messages in thread From: Uladzimir Bely @ 2022-11-25 7:56 UTC (permalink / raw) To: isar-users, Jan Kiszka In the email from Friday, 25 November 2022 10:42:00 +03 user Jan Kiszka wrote: > On 25.11.22 07:42, Uladzimir Bely wrote: > > Downstreams may want to change DISTRO to some custom value. > > When cross-building, this leads to downloading packages for host > > and target distros to the different download subdirs. > > > > While base-apt is populated only from DISTRO download subdir, > > second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to > > missing packages in base-apt. > > > > The similar issue happens with cross-building raspberry targets > > since they use their own mirrors and DISTRO. > > > > Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO > > variable that is used instead of BASE_DISTRO for caching builds > > for host-related components (e.g., isar-bootstrap, buildchroot-host, > > sbuild-chroot-host). Target components still use BASE_DISTRO. > > > > Signed-off-by: Uladzimir Bely <ubely@ilbers.de> > > --- > > > > meta-isar/conf/distro/raspbian-stretch.conf | 2 + > > meta-isar/conf/distro/raspios-bullseye.conf | 2 + > > meta-isar/conf/distro/ubuntu-focal.conf | 2 + > > meta/conf/distro/debian-common.conf | 2 + > > .../isar-bootstrap/isar-bootstrap.inc | 9 +++- > > meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++--------- > > 6 files changed, 40 insertions(+), 24 deletions(-) > > > > diff --git a/meta-isar/conf/distro/raspbian-stretch.conf > > b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..66d965c9 > > 100644 > > --- a/meta-isar/conf/distro/raspbian-stretch.conf > > +++ b/meta-isar/conf/distro/raspbian-stretch.conf > > @@ -6,6 +6,8 @@ > > > > BASE_DISTRO = "raspbian" > > BASE_DISTRO_CODENAME = "stretch" > > > > +BASE_HOST_DISTRO = "debian" > > Better prefix it with HOST, to be consistent: > > HOST_BASE_DISTRO > Not a problem, if you think it would be better. > > + > > > > HOST_DISTRO ?= "debian-stretch" > > HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > > > > diff --git a/meta-isar/conf/distro/raspios-bullseye.conf > > b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..7bd81dd6 > > 100644 > > --- a/meta-isar/conf/distro/raspios-bullseye.conf > > +++ b/meta-isar/conf/distro/raspios-bullseye.conf > > @@ -6,6 +6,8 @@ > > > > BASE_DISTRO = "raspios" > > BASE_DISTRO_arm64 = "debian" > > > > +BASE_HOST_DISTRO ?= "debian" > > + > > > > BASE_DISTRO_CODENAME = "bullseye" > > HOST_DISTRO ?= "debian-bullseye" > > HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > > > > diff --git a/meta-isar/conf/distro/ubuntu-focal.conf > > b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1 100644 > > --- a/meta-isar/conf/distro/ubuntu-focal.conf > > +++ b/meta-isar/conf/distro/ubuntu-focal.conf > > @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf > > > > BASE_DISTRO = "ubuntu" > > BASE_DISTRO_CODENAME = "focal" > > > > +BASE_HOST_DISTRO = "${BASE_DISTRO}" > > + > > > > DISTRO_APT_SOURCES_arm64 ?= > > "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" > > HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list > > conf/distro/${HOST_DISTRO}-ports.list"> > > diff --git a/meta/conf/distro/debian-common.conf > > b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b 100644 > > --- a/meta/conf/distro/debian-common.conf > > +++ b/meta/conf/distro/debian-common.conf > > @@ -5,6 +5,8 @@ > > > > BASE_DISTRO = "debian" > > > > +BASE_HOST_DISTRO ?= "${BASE_DISTRO}" > > + > > > > HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > > DISTRO_APT_SOURCES ?= > > "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > > f32d192e..2ee0d4cc 100644 > > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg" > > > > DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = > > "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if > > d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO = > > "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else > > 'DISTRO')}"> > > +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if > > d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"> > > FILESEXTRAPATHS_append = ":${BBPATH}" > > > > inherit deb-dl-dir > > > > @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list): > > import re > > > > if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): > > - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" > > + premirrors = "\S* > > file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> > > else: > > premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or "" > > > > mirror_list = [entry.split() > > > > @@ -319,11 +320,15 @@ do_bootstrap() { > > > > "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap" > > > > mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" > > if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then > > > > - line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} > > main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} > > ${BASE_DISTRO_CODENAME} main"> > > if [ -z "${BASE_REPO_KEY}" ]; then > > > > line="[trusted=yes] ${line}" > > > > fi > > echo "deb ${line}" > > > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" > > > > + line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} > > main" > Won't this cause duplicate lines (with complaints of apt) in case of > BOOTSTRAP_BASE_DISTRO == BASE_DISTRO? No, while I don't add new lines (no new "echo"s). I just modify "deb-src" line while it should point to BASE_DISTRO even for host case. Without it, there was a problem with "hello" (apt_fetch failed to download cached sources from "host" repo, while sources were cached in "target" one. That's how it looks now: | $ cat isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base- apt.list | deb [trusted=yes] file:///base-apt/raspios bullseye main | deb-src [trusted=yes] file:///base-apt/raspios bullseye main | $ cat isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.list | deb [trusted=yes] file:///base-apt/debian bullseye main | deb-src [trusted=yes] file:///base-apt/raspios bullseye main > > > + if [ -z "${BASE_REPO_KEY}" ]; then > > + line="[trusted=yes] ${line}" > > + fi > > > > echo "deb-src ${line}" >> > > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" > > > > mkdir -p ${ROOTFSDIR}/base-apt > > > > diff --git a/meta/recipes-devtools/base-apt/base-apt.bb > > b/meta/recipes-devtools/base-apt/base-apt.bb index 506a28ff..e928cd52 > > 100644 > > --- a/meta/recipes-devtools/base-apt/base-apt.bb > > +++ b/meta/recipes-devtools/base-apt/base-apt.bb > > @@ -13,7 +13,10 @@ KEYFILES ?= "" > > > > BASE_REPO_FEATURES ?= "" > > > > populate_base_apt() { > > > > - find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do > > + distro="${1}" > > + base_distro="${2}" > > + > > + find "${DEBDIR}"/"${distro}" -name '*\.deb' | while read package; do > > > > # NOTE: due to packages stored by reprepro are not modified, we > > can > > # use search by filename to check if package is already in repo. > > In > > # addition, md5sums are compared to ensure that the package is > > the > > > > @@ -22,25 +25,25 @@ populate_base_apt() { > > > > # Check if this package is already in base-apt > > ret=0 > > > > - repo_contains_package "${REPO_BASE_DIR}/${BASE_DISTRO}" > > "${package}" || + repo_contains_package > > "${REPO_BASE_DIR}/${base_distro}" "${package}" ||> > > ret=$? > > > > [ "${ret}" = "0" ] && continue > > if [ "${ret}" = "1" ]; then > > > > - repo_del_package "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > > - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > > + repo_del_package "${REPO_BASE_DIR}"/"${base_distro}" \ > > + "${REPO_BASE_DB_DIR}"/"${base_distro}" \ > > > > "${BASE_DISTRO_CODENAME}" \ > > > > - "${base_apt_p}" > > + "${package}" > > > > fi > > > > - repo_add_packages "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > > - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > > + repo_add_packages "${REPO_BASE_DIR}"/"${base_distro}" \ > > + "${REPO_BASE_DB_DIR}"/"${base_distro}" \ > > > > "${BASE_DISTRO_CODENAME}" \ > > "${package}" > > > > done > > > > - find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read package; > > do - repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ - > > "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > > + find "${DEBSRCDIR}"/"${distro}" -name '*\.dsc' | while read package; > > do + repo_add_srcpackage "${REPO_BASE_DIR}"/"${base_distro}" \ + > > "${REPO_BASE_DB_DIR}"/"${base_distro}" \ > > > > "${BASE_DISTRO_CODENAME}" \ > > "${package}" > > > > done > > > > @@ -55,20 +58,20 @@ repo() { > > > > "${BASE_DISTRO_CODENAME}" \ > > "${WORKDIR}/distributions.in" \ > > "${KEYFILES}" > > > > - > > - if [ -d '${BUILDCHROOT_HOST_DIR}/var/cache/apt' ] && > > - [ '${DISTRO}' != '${HOST_DISTRO}' ]; then > > - # We would need two separate repository paths for that. > > - # Otherwise packages (especially the 'all' arch ones) from one > > - # distribution can influence the package versions of the other > > - # distribution. > > - bbfatal "Different host and target distributions are currently > > not supported." \ - "Try it without cross-build." > > - fi > > - > > - populate_base_apt > > + populate_base_apt "${DISTRO}" "${BASE_DISTRO}" > > > > repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > > > > "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" > > > > + > > + if [ '${DISTRO}' != '${HOST_DISTRO}' ]; then > > + repo_create "${REPO_BASE_DIR}"/"${BASE_HOST_DISTRO}" \ > > + "${REPO_BASE_DB_DIR}"/"${BASE_HOST_DISTRO}" \ > > + "${BASE_DISTRO_CODENAME}" \ > > + "${WORKDIR}/distributions.in" \ > > + "${KEYFILES}" > > + populate_base_apt "${HOST_DISTRO}" "${BASE_HOST_DISTRO}" > > + repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_HOST_DISTRO}" \ > > + "${REPO_BASE_DB_DIR}"/"${BASE_HOST_DISTRO}" > > + fi > > > > } > > > > python do_cache() { > > Looks promising! > > Jan -- Uladzimir Bely ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs 2022-11-25 7:56 ` Uladzimir Bely @ 2022-11-25 14:11 ` Jan Kiszka 2022-11-25 14:48 ` Uladzimir Bely 0 siblings, 1 reply; 18+ messages in thread From: Jan Kiszka @ 2022-11-25 14:11 UTC (permalink / raw) To: Uladzimir Bely, isar-users On 25.11.22 08:56, Uladzimir Bely wrote: > In the email from Friday, 25 November 2022 10:42:00 +03 user Jan Kiszka wrote: >> On 25.11.22 07:42, Uladzimir Bely wrote: >>> Downstreams may want to change DISTRO to some custom value. >>> When cross-building, this leads to downloading packages for host >>> and target distros to the different download subdirs. >>> >>> While base-apt is populated only from DISTRO download subdir, >>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to >>> missing packages in base-apt. >>> >>> The similar issue happens with cross-building raspberry targets >>> since they use their own mirrors and DISTRO. >>> >>> Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO >>> variable that is used instead of BASE_DISTRO for caching builds >>> for host-related components (e.g., isar-bootstrap, buildchroot-host, >>> sbuild-chroot-host). Target components still use BASE_DISTRO. >>> >>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de> >>> --- >>> >>> meta-isar/conf/distro/raspbian-stretch.conf | 2 + >>> meta-isar/conf/distro/raspios-bullseye.conf | 2 + >>> meta-isar/conf/distro/ubuntu-focal.conf | 2 + >>> meta/conf/distro/debian-common.conf | 2 + >>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++- >>> meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++--------- >>> 6 files changed, 40 insertions(+), 24 deletions(-) >>> >>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf >>> b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..66d965c9 >>> 100644 >>> --- a/meta-isar/conf/distro/raspbian-stretch.conf >>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf >>> @@ -6,6 +6,8 @@ >>> >>> BASE_DISTRO = "raspbian" >>> BASE_DISTRO_CODENAME = "stretch" >>> >>> +BASE_HOST_DISTRO = "debian" >> >> Better prefix it with HOST, to be consistent: >> >> HOST_BASE_DISTRO >> > > Not a problem, if you think it would be better. > >>> + >>> >>> HOST_DISTRO ?= "debian-stretch" >>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>> >>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf >>> b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..7bd81dd6 >>> 100644 >>> --- a/meta-isar/conf/distro/raspios-bullseye.conf >>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf >>> @@ -6,6 +6,8 @@ >>> >>> BASE_DISTRO = "raspios" >>> BASE_DISTRO_arm64 = "debian" >>> >>> +BASE_HOST_DISTRO ?= "debian" >>> + >>> >>> BASE_DISTRO_CODENAME = "bullseye" >>> HOST_DISTRO ?= "debian-bullseye" >>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>> >>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf >>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1 100644 >>> --- a/meta-isar/conf/distro/ubuntu-focal.conf >>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf >>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf >>> >>> BASE_DISTRO = "ubuntu" >>> BASE_DISTRO_CODENAME = "focal" >>> >>> +BASE_HOST_DISTRO = "${BASE_DISTRO}" >>> + >>> >>> DISTRO_APT_SOURCES_arm64 ?= >>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" >>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list >>> conf/distro/${HOST_DISTRO}-ports.list"> >>> diff --git a/meta/conf/distro/debian-common.conf >>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b 100644 >>> --- a/meta/conf/distro/debian-common.conf >>> +++ b/meta/conf/distro/debian-common.conf >>> @@ -5,6 +5,8 @@ >>> >>> BASE_DISTRO = "debian" >>> >>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}" >>> + >>> >>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>> DISTRO_APT_SOURCES ?= >>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> >>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index >>> f32d192e..2ee0d4cc 100644 >>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg" >>> >>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = >>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if >>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO = >>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else >>> 'DISTRO')}"> >>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if >>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"> >>> FILESEXTRAPATHS_append = ":${BBPATH}" >>> >>> inherit deb-dl-dir >>> >>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list): >>> import re >>> >>> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): >>> - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" >>> + premirrors = "\S* >>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> >>> else: >>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or "" >>> >>> mirror_list = [entry.split() >>> >>> @@ -319,11 +320,15 @@ do_bootstrap() { >>> >>> "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap" >>> >>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" >>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then >>> >>> - line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} >>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} >>> ${BASE_DISTRO_CODENAME} main"> >>> if [ -z "${BASE_REPO_KEY}" ]; then >>> >>> line="[trusted=yes] ${line}" >>> >>> fi >>> echo "deb ${line}" > >>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" >>> >>> + line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} >>> main" >> Won't this cause duplicate lines (with complaints of apt) in case of >> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO? > > No, while I don't add new lines (no new "echo"s). I just modify "deb-src" line > while it should point to BASE_DISTRO even for host case. Without it, there was > a problem with "hello" (apt_fetch failed to download cached sources from > "host" repo, while sources were cached in "target" one. > > That's how it looks now: > > | $ cat isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base- > apt.list > | deb [trusted=yes] file:///base-apt/raspios bullseye main > | deb-src [trusted=yes] file:///base-apt/raspios bullseye main > | $ cat isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.list > | deb [trusted=yes] file:///base-apt/debian bullseye main > | deb-src [trusted=yes] file:///base-apt/raspios bullseye main > Indeed, I misintepreted the code. But I think we still have an issue on the deb-src population side: # ls build/downloads/deb-src/ iot2050-debian And that was with this patch applied. Jan -- Siemens AG, Technology Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs 2022-11-25 14:11 ` Jan Kiszka @ 2022-11-25 14:48 ` Uladzimir Bely 2022-11-25 15:40 ` Jan Kiszka 0 siblings, 1 reply; 18+ messages in thread From: Uladzimir Bely @ 2022-11-25 14:48 UTC (permalink / raw) To: isar-users, Jan Kiszka In the email from Friday, 25 November 2022 17:11:26 +03 user Jan Kiszka wrote: > On 25.11.22 08:56, Uladzimir Bely wrote: > > In the email from Friday, 25 November 2022 10:42:00 +03 user Jan Kiszka wrote: > >> On 25.11.22 07:42, Uladzimir Bely wrote: > >>> Downstreams may want to change DISTRO to some custom value. > >>> When cross-building, this leads to downloading packages for host > >>> and target distros to the different download subdirs. > >>> > >>> While base-apt is populated only from DISTRO download subdir, > >>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to > >>> missing packages in base-apt. > >>> > >>> The similar issue happens with cross-building raspberry targets > >>> since they use their own mirrors and DISTRO. > >>> > >>> Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO > >>> variable that is used instead of BASE_DISTRO for caching builds > >>> for host-related components (e.g., isar-bootstrap, buildchroot-host, > >>> sbuild-chroot-host). Target components still use BASE_DISTRO. > >>> > >>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de> > >>> --- > >>> > >>> meta-isar/conf/distro/raspbian-stretch.conf | 2 + > >>> meta-isar/conf/distro/raspios-bullseye.conf | 2 + > >>> meta-isar/conf/distro/ubuntu-focal.conf | 2 + > >>> meta/conf/distro/debian-common.conf | 2 + > >>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++- > >>> meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++--------- > >>> 6 files changed, 40 insertions(+), 24 deletions(-) > >>> > >>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf > >>> b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..66d965c9 > >>> 100644 > >>> --- a/meta-isar/conf/distro/raspbian-stretch.conf > >>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf > >>> @@ -6,6 +6,8 @@ > >>> > >>> BASE_DISTRO = "raspbian" > >>> BASE_DISTRO_CODENAME = "stretch" > >>> > >>> +BASE_HOST_DISTRO = "debian" > >> > >> Better prefix it with HOST, to be consistent: > >> > >> HOST_BASE_DISTRO > > > > Not a problem, if you think it would be better. > > > >>> + > >>> > >>> HOST_DISTRO ?= "debian-stretch" > >>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>> > >>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf > >>> b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..7bd81dd6 > >>> 100644 > >>> --- a/meta-isar/conf/distro/raspios-bullseye.conf > >>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf > >>> @@ -6,6 +6,8 @@ > >>> > >>> BASE_DISTRO = "raspios" > >>> BASE_DISTRO_arm64 = "debian" > >>> > >>> +BASE_HOST_DISTRO ?= "debian" > >>> + > >>> > >>> BASE_DISTRO_CODENAME = "bullseye" > >>> HOST_DISTRO ?= "debian-bullseye" > >>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>> > >>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf > >>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1 > >>> 100644 > >>> --- a/meta-isar/conf/distro/ubuntu-focal.conf > >>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf > >>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf > >>> > >>> BASE_DISTRO = "ubuntu" > >>> BASE_DISTRO_CODENAME = "focal" > >>> > >>> +BASE_HOST_DISTRO = "${BASE_DISTRO}" > >>> + > >>> > >>> DISTRO_APT_SOURCES_arm64 ?= > >>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" > >>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list > >>> conf/distro/${HOST_DISTRO}-ports.list"> > >>> > >>> diff --git a/meta/conf/distro/debian-common.conf > >>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b 100644 > >>> --- a/meta/conf/distro/debian-common.conf > >>> +++ b/meta/conf/distro/debian-common.conf > >>> @@ -5,6 +5,8 @@ > >>> > >>> BASE_DISTRO = "debian" > >>> > >>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}" > >>> + > >>> > >>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>> DISTRO_APT_SOURCES ?= > >>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> > >>> > >>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > >>> f32d192e..2ee0d4cc 100644 > >>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg" > >>> > >>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = > >>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if > >>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO = > >>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' > >>> else > >>> 'DISTRO')}"> > >>> > >>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if > >>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"> > >>> > >>> FILESEXTRAPATHS_append = ":${BBPATH}" > >>> > >>> inherit deb-dl-dir > >>> > >>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list): > >>> import re > >>> > >>> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): > >>> - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" > >>> + premirrors = "\S* > >>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> > >>> > >>> else: > >>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or "" > >>> > >>> mirror_list = [entry.split() > >>> > >>> @@ -319,11 +320,15 @@ do_bootstrap() { > >>> > >>> "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap" > >>> > >>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" > >>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then > >>> > >>> - line="file:///base-apt/${BASE_DISTRO} > >>> ${BASE_DISTRO_CODENAME} > >>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} > >>> ${BASE_DISTRO_CODENAME} main"> > >>> > >>> if [ -z "${BASE_REPO_KEY}" ]; then > >>> > >>> line="[trusted=yes] ${line}" > >>> > >>> fi > >>> echo "deb ${line}" > > >>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" > >>> > >>> + line="file:///base-apt/${BASE_DISTRO} > >>> ${BASE_DISTRO_CODENAME} > >>> main" > >> > >> Won't this cause duplicate lines (with complaints of apt) in case of > >> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO? > > > > No, while I don't add new lines (no new "echo"s). I just modify "deb-src" > > line while it should point to BASE_DISTRO even for host case. Without it, > > there was a problem with "hello" (apt_fetch failed to download cached > > sources from "host" repo, while sources were cached in "target" one. > > > > That's how it looks now: > > | $ cat isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base- > > > > apt.list > > > > | deb [trusted=yes] file:///base-apt/raspios bullseye main > > | deb-src [trusted=yes] file:///base-apt/raspios bullseye main > > | $ cat > > | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.list > > | deb [trusted=yes] file:///base-apt/debian bullseye main > > | deb-src [trusted=yes] file:///base-apt/raspios bullseye main > > Indeed, I misintepreted the code. > > But I think we still have an issue on the deb-src population side: > > # ls build/downloads/deb-src/ > iot2050-debian > > And that was with this patch applied. > > Jan Actually, I checked the case with the following configuration: distro: my-debian local_conf_header: nodebian: | require conf/distro/debian-bullseye.conf DISTRO_NAME = "My Debian System" So, I simulated renaming distro that is done in meta-iot2050. Cached build was OK, "hello" package (that we build from sources) was downloaded at first build to "my-debian" directory: | $ ls build/downloads/deb-src/my-debian/hello/ | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz and second build picked the files with no problem. The same with "raspios-bullseye" target - "hello" sources are downloaded to the "target" dir: | $ ls build/downloads/deb-src/raspios-bullseye/hello/ | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" and it was OK (honestly, I don't know who could require cached build with this option enabled :) ). Anyway, I'll also try to check my changes with meta-iot2050 layer also (despite some packages in it are currently not compatible with BB_NO_NETWORK option) - Uladzimir Bely ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs 2022-11-25 14:48 ` Uladzimir Bely @ 2022-11-25 15:40 ` Jan Kiszka 2022-11-26 5:30 ` Uladzimir Bely 0 siblings, 1 reply; 18+ messages in thread From: Jan Kiszka @ 2022-11-25 15:40 UTC (permalink / raw) To: Uladzimir Bely, isar-users On 25.11.22 15:48, Uladzimir Bely wrote: > In the email from Friday, 25 November 2022 17:11:26 +03 user Jan Kiszka wrote: >> On 25.11.22 08:56, Uladzimir Bely wrote: >>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan Kiszka > wrote: >>>> On 25.11.22 07:42, Uladzimir Bely wrote: >>>>> Downstreams may want to change DISTRO to some custom value. >>>>> When cross-building, this leads to downloading packages for host >>>>> and target distros to the different download subdirs. >>>>> >>>>> While base-apt is populated only from DISTRO download subdir, >>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to >>>>> missing packages in base-apt. >>>>> >>>>> The similar issue happens with cross-building raspberry targets >>>>> since they use their own mirrors and DISTRO. >>>>> >>>>> Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO >>>>> variable that is used instead of BASE_DISTRO for caching builds >>>>> for host-related components (e.g., isar-bootstrap, buildchroot-host, >>>>> sbuild-chroot-host). Target components still use BASE_DISTRO. >>>>> >>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de> >>>>> --- >>>>> >>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 + >>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 + >>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 + >>>>> meta/conf/distro/debian-common.conf | 2 + >>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++- >>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++--------- >>>>> 6 files changed, 40 insertions(+), 24 deletions(-) >>>>> >>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf >>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..66d965c9 >>>>> 100644 >>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf >>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf >>>>> @@ -6,6 +6,8 @@ >>>>> >>>>> BASE_DISTRO = "raspbian" >>>>> BASE_DISTRO_CODENAME = "stretch" >>>>> >>>>> +BASE_HOST_DISTRO = "debian" >>>> >>>> Better prefix it with HOST, to be consistent: >>>> >>>> HOST_BASE_DISTRO >>> >>> Not a problem, if you think it would be better. >>> >>>>> + >>>>> >>>>> HOST_DISTRO ?= "debian-stretch" >>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>>>> >>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf >>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..7bd81dd6 >>>>> 100644 >>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf >>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf >>>>> @@ -6,6 +6,8 @@ >>>>> >>>>> BASE_DISTRO = "raspios" >>>>> BASE_DISTRO_arm64 = "debian" >>>>> >>>>> +BASE_HOST_DISTRO ?= "debian" >>>>> + >>>>> >>>>> BASE_DISTRO_CODENAME = "bullseye" >>>>> HOST_DISTRO ?= "debian-bullseye" >>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>>>> >>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf >>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1 >>>>> 100644 >>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf >>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf >>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf >>>>> >>>>> BASE_DISTRO = "ubuntu" >>>>> BASE_DISTRO_CODENAME = "focal" >>>>> >>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}" >>>>> + >>>>> >>>>> DISTRO_APT_SOURCES_arm64 ?= >>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" >>>>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list >>>>> conf/distro/${HOST_DISTRO}-ports.list"> >>>>> >>>>> diff --git a/meta/conf/distro/debian-common.conf >>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b 100644 >>>>> --- a/meta/conf/distro/debian-common.conf >>>>> +++ b/meta/conf/distro/debian-common.conf >>>>> @@ -5,6 +5,8 @@ >>>>> >>>>> BASE_DISTRO = "debian" >>>>> >>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}" >>>>> + >>>>> >>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>>>> DISTRO_APT_SOURCES ?= >>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> >>>>> >>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index >>>>> f32d192e..2ee0d4cc 100644 >>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg" >>>>> >>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = >>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if >>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO = >>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' >>>>> else >>>>> 'DISTRO')}"> >>>>> >>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if >>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"> >>>>> >>>>> FILESEXTRAPATHS_append = ":${BBPATH}" >>>>> >>>>> inherit deb-dl-dir >>>>> >>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list): >>>>> import re >>>>> >>>>> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): >>>>> - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" >>>>> + premirrors = "\S* >>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> >>>>> >>>>> else: >>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or "" >>>>> >>>>> mirror_list = [entry.split() >>>>> >>>>> @@ -319,11 +320,15 @@ do_bootstrap() { >>>>> >>>>> "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap" >>>>> >>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" >>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then >>>>> >>>>> - line="file:///base-apt/${BASE_DISTRO} >>>>> ${BASE_DISTRO_CODENAME} >>>>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} >>>>> ${BASE_DISTRO_CODENAME} main"> >>>>> >>>>> if [ -z "${BASE_REPO_KEY}" ]; then >>>>> >>>>> line="[trusted=yes] ${line}" >>>>> >>>>> fi >>>>> echo "deb ${line}" > >>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" >>>>> >>>>> + line="file:///base-apt/${BASE_DISTRO} >>>>> ${BASE_DISTRO_CODENAME} >>>>> main" >>>> >>>> Won't this cause duplicate lines (with complaints of apt) in case of >>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO? >>> >>> No, while I don't add new lines (no new "echo"s). I just modify "deb-src" >>> line while it should point to BASE_DISTRO even for host case. Without it, >>> there was a problem with "hello" (apt_fetch failed to download cached >>> sources from "host" repo, while sources were cached in "target" one. >>> >>> That's how it looks now: >>> | $ cat isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base- >>> >>> apt.list >>> >>> | deb [trusted=yes] file:///base-apt/raspios bullseye main >>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main >>> | $ cat >>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.list >>> | deb [trusted=yes] file:///base-apt/debian bullseye main >>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main >> >> Indeed, I misintepreted the code. >> >> But I think we still have an issue on the deb-src population side: >> >> # ls build/downloads/deb-src/ >> iot2050-debian >> >> And that was with this patch applied. >> >> Jan > > Actually, I checked the case with the following configuration: > > distro: my-debian > local_conf_header: > nodebian: | > require conf/distro/debian-bullseye.conf > DISTRO_NAME = "My Debian System" > > So, I simulated renaming distro that is done in meta-iot2050. > > Cached build was OK, "hello" package (that we build from sources) was > downloaded at first build to "my-debian" directory: > > | $ ls build/downloads/deb-src/my-debian/hello/ > | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz > > and second build picked the files with no problem. > > The same with "raspios-bullseye" target - "hello" sources are downloaded to > the "target" dir: > > | $ ls build/downloads/deb-src/raspios-bullseye/hello/ > | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz > > I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" and it was > OK (honestly, I don't know who could require cached build with this option > enabled :) ). > > Anyway, I'll also try to check my changes with meta-iot2050 layer also > (despite some packages in it are currently not compatible with BB_NO_NETWORK > option) Please then test without the coral packages, they have an own issue /wrt offline builds. Thanks, Jan -- Siemens AG, Technology Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs 2022-11-25 15:40 ` Jan Kiszka @ 2022-11-26 5:30 ` Uladzimir Bely 2022-11-27 19:05 ` Jan Kiszka 0 siblings, 1 reply; 18+ messages in thread From: Uladzimir Bely @ 2022-11-26 5:30 UTC (permalink / raw) To: isar-users, Jan Kiszka In the email from Friday, 25 November 2022 18:40:39 +03 user Jan Kiszka wrote: > On 25.11.22 15:48, Uladzimir Bely wrote: > > In the email from Friday, 25 November 2022 17:11:26 +03 user Jan Kiszka wrote: > >> On 25.11.22 08:56, Uladzimir Bely wrote: > >>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan Kiszka > > > > wrote: > >>>> On 25.11.22 07:42, Uladzimir Bely wrote: > >>>>> Downstreams may want to change DISTRO to some custom value. > >>>>> When cross-building, this leads to downloading packages for host > >>>>> and target distros to the different download subdirs. > >>>>> > >>>>> While base-apt is populated only from DISTRO download subdir, > >>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to > >>>>> missing packages in base-apt. > >>>>> > >>>>> The similar issue happens with cross-building raspberry targets > >>>>> since they use their own mirrors and DISTRO. > >>>>> > >>>>> Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO > >>>>> variable that is used instead of BASE_DISTRO for caching builds > >>>>> for host-related components (e.g., isar-bootstrap, buildchroot-host, > >>>>> sbuild-chroot-host). Target components still use BASE_DISTRO. > >>>>> > >>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de> > >>>>> --- > >>>>> > >>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 + > >>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 + > >>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 + > >>>>> meta/conf/distro/debian-common.conf | 2 + > >>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++- > >>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47 > >>>>> ++++++++++--------- > >>>>> 6 files changed, 40 insertions(+), 24 deletions(-) > >>>>> > >>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf > >>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..66d965c9 > >>>>> 100644 > >>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf > >>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf > >>>>> @@ -6,6 +6,8 @@ > >>>>> > >>>>> BASE_DISTRO = "raspbian" > >>>>> BASE_DISTRO_CODENAME = "stretch" > >>>>> > >>>>> +BASE_HOST_DISTRO = "debian" > >>>> > >>>> Better prefix it with HOST, to be consistent: > >>>> > >>>> HOST_BASE_DISTRO > >>> > >>> Not a problem, if you think it would be better. > >>> > >>>>> + > >>>>> > >>>>> HOST_DISTRO ?= "debian-stretch" > >>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>>>> > >>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf > >>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..7bd81dd6 > >>>>> 100644 > >>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf > >>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf > >>>>> @@ -6,6 +6,8 @@ > >>>>> > >>>>> BASE_DISTRO = "raspios" > >>>>> BASE_DISTRO_arm64 = "debian" > >>>>> > >>>>> +BASE_HOST_DISTRO ?= "debian" > >>>>> + > >>>>> > >>>>> BASE_DISTRO_CODENAME = "bullseye" > >>>>> HOST_DISTRO ?= "debian-bullseye" > >>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>>>> > >>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf > >>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1 > >>>>> 100644 > >>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf > >>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf > >>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf > >>>>> > >>>>> BASE_DISTRO = "ubuntu" > >>>>> BASE_DISTRO_CODENAME = "focal" > >>>>> > >>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}" > >>>>> + > >>>>> > >>>>> DISTRO_APT_SOURCES_arm64 ?= > >>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" > >>>>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list > >>>>> conf/distro/${HOST_DISTRO}-ports.list"> > >>>>> > >>>>> diff --git a/meta/conf/distro/debian-common.conf > >>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b 100644 > >>>>> --- a/meta/conf/distro/debian-common.conf > >>>>> +++ b/meta/conf/distro/debian-common.conf > >>>>> @@ -5,6 +5,8 @@ > >>>>> > >>>>> BASE_DISTRO = "debian" > >>>>> > >>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}" > >>>>> + > >>>>> > >>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>>>> DISTRO_APT_SOURCES ?= > >>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> > >>>>> > >>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > >>>>> f32d192e..2ee0d4cc 100644 > >>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = > >>>>> ",gnupg" > >>>>> > >>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = > >>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if > >>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO = > >>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' > >>>>> else > >>>>> 'DISTRO')}"> > >>>>> > >>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if > >>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"> > >>>>> > >>>>> FILESEXTRAPATHS_append = ":${BBPATH}" > >>>>> > >>>>> inherit deb-dl-dir > >>>>> > >>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list): > >>>>> import re > >>>>> > >>>>> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): > >>>>> - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" > >>>>> + premirrors = "\S* > >>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> > >>>>> > >>>>> else: > >>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or "" > >>>>> > >>>>> mirror_list = [entry.split() > >>>>> > >>>>> @@ -319,11 +320,15 @@ do_bootstrap() { > >>>>> > >>>>> "${ROOTFSDIR}/etc/apt/preferences.d/bootstra > >>>>> p" > >>>>> > >>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" > >>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then > >>>>> > >>>>> - line="file:///base-apt/${BASE_DISTRO} > >>>>> ${BASE_DISTRO_CODENAME} > >>>>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} > >>>>> ${BASE_DISTRO_CODENAME} main"> > >>>>> > >>>>> if [ -z "${BASE_REPO_KEY}" ]; then > >>>>> > >>>>> line="[trusted=yes] ${line}" > >>>>> > >>>>> fi > >>>>> echo "deb ${line}" > > >>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" > >>>>> > >>>>> + line="file:///base-apt/${BASE_DISTRO} > >>>>> ${BASE_DISTRO_CODENAME} > >>>>> main" > >>>> > >>>> Won't this cause duplicate lines (with complaints of apt) in case of > >>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO? > >>> > >>> No, while I don't add new lines (no new "echo"s). I just modify > >>> "deb-src" > >>> line while it should point to BASE_DISTRO even for host case. Without > >>> it, > >>> there was a problem with "hello" (apt_fetch failed to download cached > >>> sources from "host" repo, while sources were cached in "target" one. > >>> > >>> That's how it looks now: > >>> | $ cat isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base- > >>> > >>> apt.list > >>> > >>> | deb [trusted=yes] file:///base-apt/raspios bullseye main > >>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main > >>> | $ cat > >>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.list > >>> | deb [trusted=yes] file:///base-apt/debian bullseye main > >>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main > >> > >> Indeed, I misintepreted the code. > >> > >> But I think we still have an issue on the deb-src population side: > >> > >> # ls build/downloads/deb-src/ > >> iot2050-debian > >> > >> And that was with this patch applied. > >> > >> Jan > > > > Actually, I checked the case with the following configuration: > > > > distro: my-debian > > > > local_conf_header: > > nodebian: | > > > > require conf/distro/debian-bullseye.conf > > DISTRO_NAME = "My Debian System" > > > > So, I simulated renaming distro that is done in meta-iot2050. > > > > Cached build was OK, "hello" package (that we build from sources) was > > > > downloaded at first build to "my-debian" directory: > > | $ ls build/downloads/deb-src/my-debian/hello/ > > | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz > > > > and second build picked the files with no problem. > > > > The same with "raspios-bullseye" target - "hello" sources are downloaded > > to > > > > the "target" dir: > > | $ ls build/downloads/deb-src/raspios-bullseye/hello/ > > | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz > > > > I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" and it > > was OK (honestly, I don't know who could require cached build with this > > option enabled :) ). > > > > Anyway, I'll also try to check my changes with meta-iot2050 layer also > > (despite some packages in it are currently not compatible with > > BB_NO_NETWORK option) > > Please then test without the coral packages, they have an own issue /wrt > offline builds. Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works > > Thanks, > Jan -- Uladzimir Bely ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs 2022-11-26 5:30 ` Uladzimir Bely @ 2022-11-27 19:05 ` Jan Kiszka 2022-11-28 6:05 ` Uladzimir Bely 0 siblings, 1 reply; 18+ messages in thread From: Jan Kiszka @ 2022-11-27 19:05 UTC (permalink / raw) To: Uladzimir Bely, isar-users On 26.11.22 06:30, Uladzimir Bely wrote: > In the email from Friday, 25 November 2022 18:40:39 +03 user Jan Kiszka wrote: >> On 25.11.22 15:48, Uladzimir Bely wrote: >>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan Kiszka > wrote: >>>> On 25.11.22 08:56, Uladzimir Bely wrote: >>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan Kiszka >>> >>> wrote: >>>>>> On 25.11.22 07:42, Uladzimir Bely wrote: >>>>>>> Downstreams may want to change DISTRO to some custom value. >>>>>>> When cross-building, this leads to downloading packages for host >>>>>>> and target distros to the different download subdirs. >>>>>>> >>>>>>> While base-apt is populated only from DISTRO download subdir, >>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to >>>>>>> missing packages in base-apt. >>>>>>> >>>>>>> The similar issue happens with cross-building raspberry targets >>>>>>> since they use their own mirrors and DISTRO. >>>>>>> >>>>>>> Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO >>>>>>> variable that is used instead of BASE_DISTRO for caching builds >>>>>>> for host-related components (e.g., isar-bootstrap, buildchroot-host, >>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO. >>>>>>> >>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de> >>>>>>> --- >>>>>>> >>>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 + >>>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 + >>>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 + >>>>>>> meta/conf/distro/debian-common.conf | 2 + >>>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++- >>>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47 >>>>>>> ++++++++++--------- >>>>>>> 6 files changed, 40 insertions(+), 24 deletions(-) >>>>>>> >>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf >>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..66d965c9 >>>>>>> 100644 >>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf >>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf >>>>>>> @@ -6,6 +6,8 @@ >>>>>>> >>>>>>> BASE_DISTRO = "raspbian" >>>>>>> BASE_DISTRO_CODENAME = "stretch" >>>>>>> >>>>>>> +BASE_HOST_DISTRO = "debian" >>>>>> >>>>>> Better prefix it with HOST, to be consistent: >>>>>> >>>>>> HOST_BASE_DISTRO >>>>> >>>>> Not a problem, if you think it would be better. >>>>> >>>>>>> + >>>>>>> >>>>>>> HOST_DISTRO ?= "debian-stretch" >>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>>>>>> >>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf >>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..7bd81dd6 >>>>>>> 100644 >>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf >>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf >>>>>>> @@ -6,6 +6,8 @@ >>>>>>> >>>>>>> BASE_DISTRO = "raspios" >>>>>>> BASE_DISTRO_arm64 = "debian" >>>>>>> >>>>>>> +BASE_HOST_DISTRO ?= "debian" >>>>>>> + >>>>>>> >>>>>>> BASE_DISTRO_CODENAME = "bullseye" >>>>>>> HOST_DISTRO ?= "debian-bullseye" >>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>>>>>> >>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf >>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1 >>>>>>> 100644 >>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf >>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf >>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf >>>>>>> >>>>>>> BASE_DISTRO = "ubuntu" >>>>>>> BASE_DISTRO_CODENAME = "focal" >>>>>>> >>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}" >>>>>>> + >>>>>>> >>>>>>> DISTRO_APT_SOURCES_arm64 ?= >>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" >>>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list >>>>>>> conf/distro/${HOST_DISTRO}-ports.list"> >>>>>>> >>>>>>> diff --git a/meta/conf/distro/debian-common.conf >>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b 100644 >>>>>>> --- a/meta/conf/distro/debian-common.conf >>>>>>> +++ b/meta/conf/distro/debian-common.conf >>>>>>> @@ -5,6 +5,8 @@ >>>>>>> >>>>>>> BASE_DISTRO = "debian" >>>>>>> >>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}" >>>>>>> + >>>>>>> >>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>>>>>> DISTRO_APT_SOURCES ?= >>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> >>>>>>> >>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index >>>>>>> f32d192e..2ee0d4cc 100644 >>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = >>>>>>> ",gnupg" >>>>>>> >>>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = >>>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if >>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO = >>>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' >>>>>>> else >>>>>>> 'DISTRO')}"> >>>>>>> >>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if >>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"> >>>>>>> >>>>>>> FILESEXTRAPATHS_append = ":${BBPATH}" >>>>>>> >>>>>>> inherit deb-dl-dir >>>>>>> >>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, > aptsources_entry_list): >>>>>>> import re >>>>>>> >>>>>>> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): >>>>>>> - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" >>>>>>> + premirrors = "\S* >>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> >>>>>>> >>>>>>> else: >>>>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or "" >>>>>>> >>>>>>> mirror_list = [entry.split() >>>>>>> >>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() { >>>>>>> >>>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/bootstra >>>>>>> p" >>>>>>> >>>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" >>>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then >>>>>>> >>>>>>> - line="file:///base-apt/${BASE_DISTRO} >>>>>>> ${BASE_DISTRO_CODENAME} >>>>>>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} >>>>>>> ${BASE_DISTRO_CODENAME} main"> >>>>>>> >>>>>>> if [ -z "${BASE_REPO_KEY}" ]; then >>>>>>> >>>>>>> line="[trusted=yes] ${line}" >>>>>>> >>>>>>> fi >>>>>>> echo "deb ${line}" > >>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" >>>>>>> >>>>>>> + line="file:///base-apt/${BASE_DISTRO} >>>>>>> ${BASE_DISTRO_CODENAME} >>>>>>> main" >>>>>> >>>>>> Won't this cause duplicate lines (with complaints of apt) in case of >>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO? >>>>> >>>>> No, while I don't add new lines (no new "echo"s). I just modify >>>>> "deb-src" >>>>> line while it should point to BASE_DISTRO even for host case. Without >>>>> it, >>>>> there was a problem with "hello" (apt_fetch failed to download cached >>>>> sources from "host" repo, while sources were cached in "target" one. >>>>> >>>>> That's how it looks now: >>>>> | $ cat isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base- >>>>> >>>>> apt.list >>>>> >>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main >>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main >>>>> | $ cat >>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.list >>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main >>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main >>>> >>>> Indeed, I misintepreted the code. >>>> >>>> But I think we still have an issue on the deb-src population side: >>>> >>>> # ls build/downloads/deb-src/ >>>> iot2050-debian >>>> >>>> And that was with this patch applied. >>>> >>>> Jan >>> >>> Actually, I checked the case with the following configuration: >>> >>> distro: my-debian >>> >>> local_conf_header: >>> nodebian: | >>> >>> require conf/distro/debian-bullseye.conf >>> DISTRO_NAME = "My Debian System" >>> >>> So, I simulated renaming distro that is done in meta-iot2050. >>> >>> Cached build was OK, "hello" package (that we build from sources) was >>> >>> downloaded at first build to "my-debian" directory: >>> | $ ls build/downloads/deb-src/my-debian/hello/ >>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz >>> >>> and second build picked the files with no problem. >>> >>> The same with "raspios-bullseye" target - "hello" sources are downloaded >>> to >>> >>> the "target" dir: >>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/ >>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz >>> >>> I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" and it >>> was OK (honestly, I don't know who could require cached build with this >>> option enabled :) ). >>> >>> Anyway, I'll also try to check my changes with meta-iot2050 layer also >>> (despite some packages in it are currently not compatible with >>> BB_NO_NETWORK option) >> >> Please then test without the coral packages, they have an own issue /wrt >> offline builds. > > Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works > ...but it caches to the wrong downloads/deb-src path, no? Jan -- Siemens AG, Technology Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs 2022-11-27 19:05 ` Jan Kiszka @ 2022-11-28 6:05 ` Uladzimir Bely 2022-11-28 6:32 ` Jan Kiszka 0 siblings, 1 reply; 18+ messages in thread From: Uladzimir Bely @ 2022-11-28 6:05 UTC (permalink / raw) To: isar-users, Jan Kiszka In the email from Sunday, 27 November 2022 22:05:53 +03 user Jan Kiszka wrote: > On 26.11.22 06:30, Uladzimir Bely wrote: > > In the email from Friday, 25 November 2022 18:40:39 +03 user Jan Kiszka wrote: > >> On 25.11.22 15:48, Uladzimir Bely wrote: > >>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan Kiszka > > > > wrote: > >>>> On 25.11.22 08:56, Uladzimir Bely wrote: > >>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan > >>>>> Kiszka > >>> > >>> wrote: > >>>>>> On 25.11.22 07:42, Uladzimir Bely wrote: > >>>>>>> Downstreams may want to change DISTRO to some custom value. > >>>>>>> When cross-building, this leads to downloading packages for host > >>>>>>> and target distros to the different download subdirs. > >>>>>>> > >>>>>>> While base-apt is populated only from DISTRO download subdir, > >>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to > >>>>>>> missing packages in base-apt. > >>>>>>> > >>>>>>> The similar issue happens with cross-building raspberry targets > >>>>>>> since they use their own mirrors and DISTRO. > >>>>>>> > >>>>>>> Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO > >>>>>>> variable that is used instead of BASE_DISTRO for caching builds > >>>>>>> for host-related components (e.g., isar-bootstrap, buildchroot-host, > >>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO. > >>>>>>> > >>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de> > >>>>>>> --- > >>>>>>> > >>>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 + > >>>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 + > >>>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 + > >>>>>>> meta/conf/distro/debian-common.conf | 2 + > >>>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++- > >>>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47 > >>>>>>> ++++++++++--------- > >>>>>>> 6 files changed, 40 insertions(+), 24 deletions(-) > >>>>>>> > >>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf > >>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index > >>>>>>> c8f523b3..66d965c9 > >>>>>>> 100644 > >>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf > >>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf > >>>>>>> @@ -6,6 +6,8 @@ > >>>>>>> > >>>>>>> BASE_DISTRO = "raspbian" > >>>>>>> BASE_DISTRO_CODENAME = "stretch" > >>>>>>> > >>>>>>> +BASE_HOST_DISTRO = "debian" > >>>>>> > >>>>>> Better prefix it with HOST, to be consistent: > >>>>>> > >>>>>> HOST_BASE_DISTRO > >>>>> > >>>>> Not a problem, if you think it would be better. > >>>>> > >>>>>>> + > >>>>>>> > >>>>>>> HOST_DISTRO ?= "debian-stretch" > >>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>>>>>> > >>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf > >>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index > >>>>>>> 5e4a09c9..7bd81dd6 > >>>>>>> 100644 > >>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf > >>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf > >>>>>>> @@ -6,6 +6,8 @@ > >>>>>>> > >>>>>>> BASE_DISTRO = "raspios" > >>>>>>> BASE_DISTRO_arm64 = "debian" > >>>>>>> > >>>>>>> +BASE_HOST_DISTRO ?= "debian" > >>>>>>> + > >>>>>>> > >>>>>>> BASE_DISTRO_CODENAME = "bullseye" > >>>>>>> HOST_DISTRO ?= "debian-bullseye" > >>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>>>>>> > >>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf > >>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1 > >>>>>>> 100644 > >>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf > >>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf > >>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf > >>>>>>> > >>>>>>> BASE_DISTRO = "ubuntu" > >>>>>>> BASE_DISTRO_CODENAME = "focal" > >>>>>>> > >>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}" > >>>>>>> + > >>>>>>> > >>>>>>> DISTRO_APT_SOURCES_arm64 ?= > >>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" > >>>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list > >>>>>>> conf/distro/${HOST_DISTRO}-ports.list"> > >>>>>>> > >>>>>>> diff --git a/meta/conf/distro/debian-common.conf > >>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b > >>>>>>> 100644 > >>>>>>> --- a/meta/conf/distro/debian-common.conf > >>>>>>> +++ b/meta/conf/distro/debian-common.conf > >>>>>>> @@ -5,6 +5,8 @@ > >>>>>>> > >>>>>>> BASE_DISTRO = "debian" > >>>>>>> > >>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}" > >>>>>>> + > >>>>>>> > >>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>>>>>> DISTRO_APT_SOURCES ?= > >>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> > >>>>>>> > >>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > >>>>>>> f32d192e..2ee0d4cc 100644 > >>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = > >>>>>>> ",gnupg" > >>>>>>> > >>>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = > >>>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if > >>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO = > >>>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' > >>>>>>> else > >>>>>>> 'DISTRO')}"> > >>>>>>> > >>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if > >>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"> > >>>>>>> > >>>>>>> FILESEXTRAPATHS_append = ":${BBPATH}" > >>>>>>> > >>>>>>> inherit deb-dl-dir > >>>>>>> > >>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, > > > > aptsources_entry_list): > >>>>>>> import re > >>>>>>> > >>>>>>> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): > >>>>>>> - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" > >>>>>>> + premirrors = "\S* > >>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> > >>>>>>> > >>>>>>> else: > >>>>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or "" > >>>>>>> > >>>>>>> mirror_list = [entry.split() > >>>>>>> > >>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() { > >>>>>>> > >>>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/bootst > >>>>>>> ra > >>>>>>> p" > >>>>>>> > >>>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" > >>>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then > >>>>>>> > >>>>>>> - line="file:///base-apt/${BASE_DISTRO} > >>>>>>> ${BASE_DISTRO_CODENAME} > >>>>>>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} > >>>>>>> ${BASE_DISTRO_CODENAME} main"> > >>>>>>> > >>>>>>> if [ -z "${BASE_REPO_KEY}" ]; then > >>>>>>> > >>>>>>> line="[trusted=yes] ${line}" > >>>>>>> > >>>>>>> fi > >>>>>>> echo "deb ${line}" > > >>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" > >>>>>>> > >>>>>>> + line="file:///base-apt/${BASE_DISTRO} > >>>>>>> ${BASE_DISTRO_CODENAME} > >>>>>>> main" > >>>>>> > >>>>>> Won't this cause duplicate lines (with complaints of apt) in case of > >>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO? > >>>>> > >>>>> No, while I don't add new lines (no new "echo"s). I just modify > >>>>> "deb-src" > >>>>> line while it should point to BASE_DISTRO even for host case. Without > >>>>> it, > >>>>> there was a problem with "hello" (apt_fetch failed to download cached > >>>>> sources from "host" repo, while sources were cached in "target" one. > >>>>> > >>>>> That's how it looks now: > >>>>> | $ cat > >>>>> | isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base- > >>>>> > >>>>> apt.list > >>>>> > >>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main > >>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main > >>>>> | $ cat > >>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.li > >>>>> | st > >>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main > >>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main > >>>> > >>>> Indeed, I misintepreted the code. > >>>> > >>>> But I think we still have an issue on the deb-src population side: > >>>> > >>>> # ls build/downloads/deb-src/ > >>>> iot2050-debian > >>>> > >>>> And that was with this patch applied. > >>>> > >>>> Jan > >>> > >>> Actually, I checked the case with the following configuration: > >>> > >>> distro: my-debian > >>> > >>> local_conf_header: > >>> nodebian: | > >>> > >>> require conf/distro/debian-bullseye.conf > >>> DISTRO_NAME = "My Debian System" > >>> > >>> So, I simulated renaming distro that is done in meta-iot2050. > >>> > >>> Cached build was OK, "hello" package (that we build from sources) was > >>> > >>> downloaded at first build to "my-debian" directory: > >>> | $ ls build/downloads/deb-src/my-debian/hello/ > >>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz > >>> > >>> and second build picked the files with no problem. > >>> > >>> The same with "raspios-bullseye" target - "hello" sources are downloaded > >>> to > >>> > >>> the "target" dir: > >>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/ > >>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz > >>> > >>> I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" and > >>> it > >>> was OK (honestly, I don't know who could require cached build with this > >>> option enabled :) ). > >>> > >>> Anyway, I'll also try to check my changes with meta-iot2050 layer also > >>> (despite some packages in it are currently not compatible with > >>> BB_NO_NETWORK option) > >> > >> Please then test without the coral packages, they have an own issue /wrt > >> offline builds. > > > > Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works > > ...but it caches to the wrong downloads/deb-src path, no? > > Jan Why wrong? | meta-iot2050/build $ tree -d downloads/deb* | downloads/deb | ├── debian-bullseye | └── iot2050-debian | downloads/deb-src | └── iot2050-debian | ├── openssl | └── swig `do_apt_fetch` works with `${DISTRO}` and downloads sources to the corresponding directory. -- Uladzimir Bely ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs 2022-11-28 6:05 ` Uladzimir Bely @ 2022-11-28 6:32 ` Jan Kiszka 2022-11-28 6:41 ` Uladzimir Bely 0 siblings, 1 reply; 18+ messages in thread From: Jan Kiszka @ 2022-11-28 6:32 UTC (permalink / raw) To: Uladzimir Bely, isar-users On 28.11.22 07:05, Uladzimir Bely wrote: > In the email from Sunday, 27 November 2022 22:05:53 +03 user Jan Kiszka wrote: >> On 26.11.22 06:30, Uladzimir Bely wrote: >>> In the email from Friday, 25 November 2022 18:40:39 +03 user Jan Kiszka > wrote: >>>> On 25.11.22 15:48, Uladzimir Bely wrote: >>>>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan Kiszka >>> >>> wrote: >>>>>> On 25.11.22 08:56, Uladzimir Bely wrote: >>>>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan >>>>>>> Kiszka >>>>> >>>>> wrote: >>>>>>>> On 25.11.22 07:42, Uladzimir Bely wrote: >>>>>>>>> Downstreams may want to change DISTRO to some custom value. >>>>>>>>> When cross-building, this leads to downloading packages for host >>>>>>>>> and target distros to the different download subdirs. >>>>>>>>> >>>>>>>>> While base-apt is populated only from DISTRO download subdir, >>>>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to >>>>>>>>> missing packages in base-apt. >>>>>>>>> >>>>>>>>> The similar issue happens with cross-building raspberry targets >>>>>>>>> since they use their own mirrors and DISTRO. >>>>>>>>> >>>>>>>>> Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO >>>>>>>>> variable that is used instead of BASE_DISTRO for caching builds >>>>>>>>> for host-related components (e.g., isar-bootstrap, buildchroot-host, >>>>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO. >>>>>>>>> >>>>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de> >>>>>>>>> --- >>>>>>>>> >>>>>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 + >>>>>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 + >>>>>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 + >>>>>>>>> meta/conf/distro/debian-common.conf | 2 + >>>>>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++- >>>>>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47 >>>>>>>>> ++++++++++--------- >>>>>>>>> 6 files changed, 40 insertions(+), 24 deletions(-) >>>>>>>>> >>>>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf >>>>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index >>>>>>>>> c8f523b3..66d965c9 >>>>>>>>> 100644 >>>>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf >>>>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf >>>>>>>>> @@ -6,6 +6,8 @@ >>>>>>>>> >>>>>>>>> BASE_DISTRO = "raspbian" >>>>>>>>> BASE_DISTRO_CODENAME = "stretch" >>>>>>>>> >>>>>>>>> +BASE_HOST_DISTRO = "debian" >>>>>>>> >>>>>>>> Better prefix it with HOST, to be consistent: >>>>>>>> >>>>>>>> HOST_BASE_DISTRO >>>>>>> >>>>>>> Not a problem, if you think it would be better. >>>>>>> >>>>>>>>> + >>>>>>>>> >>>>>>>>> HOST_DISTRO ?= "debian-stretch" >>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>>>>>>>> >>>>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf >>>>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index >>>>>>>>> 5e4a09c9..7bd81dd6 >>>>>>>>> 100644 >>>>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf >>>>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf >>>>>>>>> @@ -6,6 +6,8 @@ >>>>>>>>> >>>>>>>>> BASE_DISTRO = "raspios" >>>>>>>>> BASE_DISTRO_arm64 = "debian" >>>>>>>>> >>>>>>>>> +BASE_HOST_DISTRO ?= "debian" >>>>>>>>> + >>>>>>>>> >>>>>>>>> BASE_DISTRO_CODENAME = "bullseye" >>>>>>>>> HOST_DISTRO ?= "debian-bullseye" >>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>>>>>>>> >>>>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf >>>>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1 >>>>>>>>> 100644 >>>>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf >>>>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf >>>>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf >>>>>>>>> >>>>>>>>> BASE_DISTRO = "ubuntu" >>>>>>>>> BASE_DISTRO_CODENAME = "focal" >>>>>>>>> >>>>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}" >>>>>>>>> + >>>>>>>>> >>>>>>>>> DISTRO_APT_SOURCES_arm64 ?= >>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" >>>>>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list >>>>>>>>> conf/distro/${HOST_DISTRO}-ports.list"> >>>>>>>>> >>>>>>>>> diff --git a/meta/conf/distro/debian-common.conf >>>>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b >>>>>>>>> 100644 >>>>>>>>> --- a/meta/conf/distro/debian-common.conf >>>>>>>>> +++ b/meta/conf/distro/debian-common.conf >>>>>>>>> @@ -5,6 +5,8 @@ >>>>>>>>> >>>>>>>>> BASE_DISTRO = "debian" >>>>>>>>> >>>>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}" >>>>>>>>> + >>>>>>>>> >>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>>>>>>>> DISTRO_APT_SOURCES ?= >>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> >>>>>>>>> >>>>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index >>>>>>>>> f32d192e..2ee0d4cc 100644 >>>>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = >>>>>>>>> ",gnupg" >>>>>>>>> >>>>>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = >>>>>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if >>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO = >>>>>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' >>>>>>>>> else >>>>>>>>> 'DISTRO')}"> >>>>>>>>> >>>>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if >>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"> >>>>>>>>> >>>>>>>>> FILESEXTRAPATHS_append = ":${BBPATH}" >>>>>>>>> >>>>>>>>> inherit deb-dl-dir >>>>>>>>> >>>>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, >>> >>> aptsources_entry_list): >>>>>>>>> import re >>>>>>>>> >>>>>>>>> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): >>>>>>>>> - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" >>>>>>>>> + premirrors = "\S* >>>>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> >>>>>>>>> >>>>>>>>> else: >>>>>>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or "" >>>>>>>>> >>>>>>>>> mirror_list = [entry.split() >>>>>>>>> >>>>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() { >>>>>>>>> >>>>>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/bootst >>>>>>>>> ra >>>>>>>>> p" >>>>>>>>> >>>>>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" >>>>>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then >>>>>>>>> >>>>>>>>> - line="file:///base-apt/${BASE_DISTRO} >>>>>>>>> ${BASE_DISTRO_CODENAME} >>>>>>>>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} >>>>>>>>> ${BASE_DISTRO_CODENAME} main"> >>>>>>>>> >>>>>>>>> if [ -z "${BASE_REPO_KEY}" ]; then >>>>>>>>> >>>>>>>>> line="[trusted=yes] ${line}" >>>>>>>>> >>>>>>>>> fi >>>>>>>>> echo "deb ${line}" > >>>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" >>>>>>>>> >>>>>>>>> + line="file:///base-apt/${BASE_DISTRO} >>>>>>>>> ${BASE_DISTRO_CODENAME} >>>>>>>>> main" >>>>>>>> >>>>>>>> Won't this cause duplicate lines (with complaints of apt) in case of >>>>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO? >>>>>>> >>>>>>> No, while I don't add new lines (no new "echo"s). I just modify >>>>>>> "deb-src" >>>>>>> line while it should point to BASE_DISTRO even for host case. Without >>>>>>> it, >>>>>>> there was a problem with "hello" (apt_fetch failed to download cached >>>>>>> sources from "host" repo, while sources were cached in "target" one. >>>>>>> >>>>>>> That's how it looks now: >>>>>>> | $ cat >>>>>>> | isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base- >>>>>>> >>>>>>> apt.list >>>>>>> >>>>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main >>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main >>>>>>> | $ cat >>>>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.li >>>>>>> | st >>>>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main >>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main >>>>>> >>>>>> Indeed, I misintepreted the code. >>>>>> >>>>>> But I think we still have an issue on the deb-src population side: >>>>>> >>>>>> # ls build/downloads/deb-src/ >>>>>> iot2050-debian >>>>>> >>>>>> And that was with this patch applied. >>>>>> >>>>>> Jan >>>>> >>>>> Actually, I checked the case with the following configuration: >>>>> >>>>> distro: my-debian >>>>> >>>>> local_conf_header: >>>>> nodebian: | >>>>> >>>>> require conf/distro/debian-bullseye.conf >>>>> DISTRO_NAME = "My Debian System" >>>>> >>>>> So, I simulated renaming distro that is done in meta-iot2050. >>>>> >>>>> Cached build was OK, "hello" package (that we build from sources) was >>>>> >>>>> downloaded at first build to "my-debian" directory: >>>>> | $ ls build/downloads/deb-src/my-debian/hello/ >>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz >>>>> >>>>> and second build picked the files with no problem. >>>>> >>>>> The same with "raspios-bullseye" target - "hello" sources are downloaded >>>>> to >>>>> >>>>> the "target" dir: >>>>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/ >>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz >>>>> >>>>> I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" and >>>>> it >>>>> was OK (honestly, I don't know who could require cached build with this >>>>> option enabled :) ). >>>>> >>>>> Anyway, I'll also try to check my changes with meta-iot2050 layer also >>>>> (despite some packages in it are currently not compatible with >>>>> BB_NO_NETWORK option) >>>> >>>> Please then test without the coral packages, they have an own issue /wrt >>>> offline builds. >>> >>> Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works >> >> ...but it caches to the wrong downloads/deb-src path, no? >> >> Jan > > Why wrong? > > | meta-iot2050/build $ tree -d downloads/deb* > | downloads/deb > | ├── debian-bullseye > | └── iot2050-debian > | downloads/deb-src > | └── iot2050-debian > | ├── openssl > | └── swig > > `do_apt_fetch` works with `${DISTRO}` and downloads sources to the > corresponding directory. > deb-src fetching still lacks base-distro normalization, like deb also has: # tree -d build/downloads/deb* build/downloads/deb └── debian-bullseye build/downloads/deb-src └── iot2050-debian ├── openssl └── swig (your deb folder was likely outdated) Jan -- Siemens AG, Technology Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs 2022-11-28 6:32 ` Jan Kiszka @ 2022-11-28 6:41 ` Uladzimir Bely 2022-11-28 6:51 ` Jan Kiszka 0 siblings, 1 reply; 18+ messages in thread From: Uladzimir Bely @ 2022-11-28 6:41 UTC (permalink / raw) To: Jan Kiszka, isar-users In email from Monday, 28 November 2022 09:32:22 +03 You wrote: > On 28.11.22 07:05, Uladzimir Bely wrote: > > In the email from Sunday, 27 November 2022 22:05:53 +03 user Jan Kiszka wrote: > >> On 26.11.22 06:30, Uladzimir Bely wrote: > >>> In the email from Friday, 25 November 2022 18:40:39 +03 user Jan Kiszka > > > > wrote: > >>>> On 25.11.22 15:48, Uladzimir Bely wrote: > >>>>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan > >>>>> Kiszka > >>> > >>> wrote: > >>>>>> On 25.11.22 08:56, Uladzimir Bely wrote: > >>>>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan > >>>>>>> Kiszka > >>>>> > >>>>> wrote: > >>>>>>>> On 25.11.22 07:42, Uladzimir Bely wrote: > >>>>>>>>> Downstreams may want to change DISTRO to some custom value. > >>>>>>>>> When cross-building, this leads to downloading packages for host > >>>>>>>>> and target distros to the different download subdirs. > >>>>>>>>> > >>>>>>>>> While base-apt is populated only from DISTRO download subdir, > >>>>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to > >>>>>>>>> missing packages in base-apt. > >>>>>>>>> > >>>>>>>>> The similar issue happens with cross-building raspberry targets > >>>>>>>>> since they use their own mirrors and DISTRO. > >>>>>>>>> > >>>>>>>>> Fix the issue by splitting base-apt by introducing > >>>>>>>>> BASE_HOST_DISTRO > >>>>>>>>> variable that is used instead of BASE_DISTRO for caching builds > >>>>>>>>> for host-related components (e.g., isar-bootstrap, > >>>>>>>>> buildchroot-host, > >>>>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO. > >>>>>>>>> > >>>>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de> > >>>>>>>>> --- > >>>>>>>>> > >>>>>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 + > >>>>>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 + > >>>>>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 + > >>>>>>>>> meta/conf/distro/debian-common.conf | 2 + > >>>>>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++- > >>>>>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47 > >>>>>>>>> ++++++++++--------- > >>>>>>>>> 6 files changed, 40 insertions(+), 24 deletions(-) > >>>>>>>>> > >>>>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf > >>>>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index > >>>>>>>>> c8f523b3..66d965c9 > >>>>>>>>> 100644 > >>>>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf > >>>>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf > >>>>>>>>> @@ -6,6 +6,8 @@ > >>>>>>>>> > >>>>>>>>> BASE_DISTRO = "raspbian" > >>>>>>>>> BASE_DISTRO_CODENAME = "stretch" > >>>>>>>>> > >>>>>>>>> +BASE_HOST_DISTRO = "debian" > >>>>>>>> > >>>>>>>> Better prefix it with HOST, to be consistent: > >>>>>>>> > >>>>>>>> HOST_BASE_DISTRO > >>>>>>> > >>>>>>> Not a problem, if you think it would be better. > >>>>>>> > >>>>>>>>> + > >>>>>>>>> > >>>>>>>>> HOST_DISTRO ?= "debian-stretch" > >>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>>>>>>>> > >>>>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf > >>>>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index > >>>>>>>>> 5e4a09c9..7bd81dd6 > >>>>>>>>> 100644 > >>>>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf > >>>>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf > >>>>>>>>> @@ -6,6 +6,8 @@ > >>>>>>>>> > >>>>>>>>> BASE_DISTRO = "raspios" > >>>>>>>>> BASE_DISTRO_arm64 = "debian" > >>>>>>>>> > >>>>>>>>> +BASE_HOST_DISTRO ?= "debian" > >>>>>>>>> + > >>>>>>>>> > >>>>>>>>> BASE_DISTRO_CODENAME = "bullseye" > >>>>>>>>> HOST_DISTRO ?= "debian-bullseye" > >>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>>>>>>>> > >>>>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf > >>>>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1 > >>>>>>>>> 100644 > >>>>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf > >>>>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf > >>>>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf > >>>>>>>>> > >>>>>>>>> BASE_DISTRO = "ubuntu" > >>>>>>>>> BASE_DISTRO_CODENAME = "focal" > >>>>>>>>> > >>>>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}" > >>>>>>>>> + > >>>>>>>>> > >>>>>>>>> DISTRO_APT_SOURCES_arm64 ?= > >>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" > >>>>>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list > >>>>>>>>> conf/distro/${HOST_DISTRO}-ports.list"> > >>>>>>>>> > >>>>>>>>> diff --git a/meta/conf/distro/debian-common.conf > >>>>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b > >>>>>>>>> 100644 > >>>>>>>>> --- a/meta/conf/distro/debian-common.conf > >>>>>>>>> +++ b/meta/conf/distro/debian-common.conf > >>>>>>>>> @@ -5,6 +5,8 @@ > >>>>>>>>> > >>>>>>>>> BASE_DISTRO = "debian" > >>>>>>>>> > >>>>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}" > >>>>>>>>> + > >>>>>>>>> > >>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>>>>>>>> DISTRO_APT_SOURCES ?= > >>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> > >>>>>>>>> > >>>>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > >>>>>>>>> f32d192e..2ee0d4cc 100644 > >>>>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = > >>>>>>>>> ",gnupg" > >>>>>>>>> > >>>>>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = > >>>>>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if > >>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO > >>>>>>>>> = > >>>>>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == > >>>>>>>>> '1' > >>>>>>>>> else > >>>>>>>>> 'DISTRO')}"> > >>>>>>>>> > >>>>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if > >>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"> > >>>>>>>>> > >>>>>>>>> FILESEXTRAPATHS_append = ":${BBPATH}" > >>>>>>>>> > >>>>>>>>> inherit deb-dl-dir > >>>>>>>>> > >>>>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, > >>> > >>> aptsources_entry_list): > >>>>>>>>> import re > >>>>>>>>> > >>>>>>>>> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): > >>>>>>>>> - premirrors = "\S* > >>>>>>>>> file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" > >>>>>>>>> + premirrors = "\S* > >>>>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> > >>>>>>>>> > >>>>>>>>> else: > >>>>>>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or > >>>>>>>>> "" > >>>>>>>>> > >>>>>>>>> mirror_list = [entry.split() > >>>>>>>>> > >>>>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() { > >>>>>>>>> > >>>>>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/boot > >>>>>>>>> st > >>>>>>>>> ra > >>>>>>>>> p" > >>>>>>>>> > >>>>>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" > >>>>>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then > >>>>>>>>> > >>>>>>>>> - line="file:///base-apt/${BASE_DISTRO} > >>>>>>>>> ${BASE_DISTRO_CODENAME} > >>>>>>>>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} > >>>>>>>>> ${BASE_DISTRO_CODENAME} main"> > >>>>>>>>> > >>>>>>>>> if [ -z "${BASE_REPO_KEY}" ]; then > >>>>>>>>> > >>>>>>>>> line="[trusted=yes] ${line}" > >>>>>>>>> > >>>>>>>>> fi > >>>>>>>>> echo "deb ${line}" > > >>>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" > >>>>>>>>> > >>>>>>>>> + line="file:///base-apt/${BASE_DISTRO} > >>>>>>>>> ${BASE_DISTRO_CODENAME} > >>>>>>>>> main" > >>>>>>>> > >>>>>>>> Won't this cause duplicate lines (with complaints of apt) in case > >>>>>>>> of > >>>>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO? > >>>>>>> > >>>>>>> No, while I don't add new lines (no new "echo"s). I just modify > >>>>>>> "deb-src" > >>>>>>> line while it should point to BASE_DISTRO even for host case. > >>>>>>> Without > >>>>>>> it, > >>>>>>> there was a problem with "hello" (apt_fetch failed to download > >>>>>>> cached > >>>>>>> sources from "host" repo, while sources were cached in "target" one. > >>>>>>> > >>>>>>> That's how it looks now: > >>>>>>> | $ cat > >>>>>>> | isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base- > >>>>>>> > >>>>>>> apt.list > >>>>>>> > >>>>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main > >>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main > >>>>>>> | $ cat > >>>>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt. > >>>>>>> | li > >>>>>>> | st > >>>>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main > >>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main > >>>>>> > >>>>>> Indeed, I misintepreted the code. > >>>>>> > >>>>>> But I think we still have an issue on the deb-src population side: > >>>>>> > >>>>>> # ls build/downloads/deb-src/ > >>>>>> iot2050-debian > >>>>>> > >>>>>> And that was with this patch applied. > >>>>>> > >>>>>> Jan > >>>>> > >>>>> Actually, I checked the case with the following configuration: > >>>>> > >>>>> distro: my-debian > >>>>> > >>>>> local_conf_header: > >>>>> nodebian: | > >>>>> > >>>>> require conf/distro/debian-bullseye.conf > >>>>> DISTRO_NAME = "My Debian System" > >>>>> > >>>>> So, I simulated renaming distro that is done in meta-iot2050. > >>>>> > >>>>> Cached build was OK, "hello" package (that we build from sources) was > >>>>> > >>>>> downloaded at first build to "my-debian" directory: > >>>>> | $ ls build/downloads/deb-src/my-debian/hello/ > >>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz > >>>>> > >>>>> and second build picked the files with no problem. > >>>>> > >>>>> The same with "raspios-bullseye" target - "hello" sources are > >>>>> downloaded > >>>>> to > >>>>> > >>>>> the "target" dir: > >>>>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/ > >>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz > >>>>> > >>>>> I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" and > >>>>> it > >>>>> was OK (honestly, I don't know who could require cached build with > >>>>> this > >>>>> option enabled :) ). > >>>>> > >>>>> Anyway, I'll also try to check my changes with meta-iot2050 layer also > >>>>> (despite some packages in it are currently not compatible with > >>>>> BB_NO_NETWORK option) > >>>> > >>>> Please then test without the coral packages, they have an own issue > >>>> /wrt > >>>> offline builds. > >>> > >>> Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works > >> > >> ...but it caches to the wrong downloads/deb-src path, no? > >> > >> Jan > > > > Why wrong? > > > > | meta-iot2050/build $ tree -d downloads/deb* > > | downloads/deb > > | ├── debian-bullseye > > | └── iot2050-debian > > | downloads/deb-src > > | └── iot2050-debian > > | > > | ├── openssl > > | └── swig > > > > `do_apt_fetch` works with `${DISTRO}` and downloads sources to the > > corresponding directory. > > deb-src fetching still lacks base-distro normalization, like deb also has: > Saying "normalization", do you mean the following: if ${DISTRO} is just renaming (e.g. "iot2050-debian") everything should be downloaded to and picked up from the directory named "debian-bullseye"? > # tree -d build/downloads/deb* > build/downloads/deb > └── debian-bullseye > build/downloads/deb-src > └── iot2050-debian > ├── openssl > └── swig > > (your deb folder was likely outdated) > > Jan Uladzimir Bely ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs 2022-11-28 6:41 ` Uladzimir Bely @ 2022-11-28 6:51 ` Jan Kiszka 2022-11-28 11:12 ` Uladzimir Bely 0 siblings, 1 reply; 18+ messages in thread From: Jan Kiszka @ 2022-11-28 6:51 UTC (permalink / raw) To: Uladzimir Bely, isar-users On 28.11.22 07:41, Uladzimir Bely wrote: > In email from Monday, 28 November 2022 09:32:22 +03 You wrote: >> On 28.11.22 07:05, Uladzimir Bely wrote: >>> In the email from Sunday, 27 November 2022 22:05:53 +03 user Jan Kiszka > wrote: >>>> On 26.11.22 06:30, Uladzimir Bely wrote: >>>>> In the email from Friday, 25 November 2022 18:40:39 +03 user Jan Kiszka >>> >>> wrote: >>>>>> On 25.11.22 15:48, Uladzimir Bely wrote: >>>>>>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan >>>>>>> Kiszka >>>>> >>>>> wrote: >>>>>>>> On 25.11.22 08:56, Uladzimir Bely wrote: >>>>>>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan >>>>>>>>> Kiszka >>>>>>> >>>>>>> wrote: >>>>>>>>>> On 25.11.22 07:42, Uladzimir Bely wrote: >>>>>>>>>>> Downstreams may want to change DISTRO to some custom value. >>>>>>>>>>> When cross-building, this leads to downloading packages for host >>>>>>>>>>> and target distros to the different download subdirs. >>>>>>>>>>> >>>>>>>>>>> While base-apt is populated only from DISTRO download subdir, >>>>>>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to >>>>>>>>>>> missing packages in base-apt. >>>>>>>>>>> >>>>>>>>>>> The similar issue happens with cross-building raspberry targets >>>>>>>>>>> since they use their own mirrors and DISTRO. >>>>>>>>>>> >>>>>>>>>>> Fix the issue by splitting base-apt by introducing >>>>>>>>>>> BASE_HOST_DISTRO >>>>>>>>>>> variable that is used instead of BASE_DISTRO for caching builds >>>>>>>>>>> for host-related components (e.g., isar-bootstrap, >>>>>>>>>>> buildchroot-host, >>>>>>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO. >>>>>>>>>>> >>>>>>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de> >>>>>>>>>>> --- >>>>>>>>>>> >>>>>>>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 + >>>>>>>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 + >>>>>>>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 + >>>>>>>>>>> meta/conf/distro/debian-common.conf | 2 + >>>>>>>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++- >>>>>>>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47 >>>>>>>>>>> ++++++++++--------- >>>>>>>>>>> 6 files changed, 40 insertions(+), 24 deletions(-) >>>>>>>>>>> >>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf >>>>>>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index >>>>>>>>>>> c8f523b3..66d965c9 >>>>>>>>>>> 100644 >>>>>>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf >>>>>>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf >>>>>>>>>>> @@ -6,6 +6,8 @@ >>>>>>>>>>> >>>>>>>>>>> BASE_DISTRO = "raspbian" >>>>>>>>>>> BASE_DISTRO_CODENAME = "stretch" >>>>>>>>>>> >>>>>>>>>>> +BASE_HOST_DISTRO = "debian" >>>>>>>>>> >>>>>>>>>> Better prefix it with HOST, to be consistent: >>>>>>>>>> >>>>>>>>>> HOST_BASE_DISTRO >>>>>>>>> >>>>>>>>> Not a problem, if you think it would be better. >>>>>>>>> >>>>>>>>>>> + >>>>>>>>>>> >>>>>>>>>>> HOST_DISTRO ?= "debian-stretch" >>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>>>>>>>>>> >>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf >>>>>>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index >>>>>>>>>>> 5e4a09c9..7bd81dd6 >>>>>>>>>>> 100644 >>>>>>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf >>>>>>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf >>>>>>>>>>> @@ -6,6 +6,8 @@ >>>>>>>>>>> >>>>>>>>>>> BASE_DISTRO = "raspios" >>>>>>>>>>> BASE_DISTRO_arm64 = "debian" >>>>>>>>>>> >>>>>>>>>>> +BASE_HOST_DISTRO ?= "debian" >>>>>>>>>>> + >>>>>>>>>>> >>>>>>>>>>> BASE_DISTRO_CODENAME = "bullseye" >>>>>>>>>>> HOST_DISTRO ?= "debian-bullseye" >>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>>>>>>>>>> >>>>>>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf >>>>>>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1 >>>>>>>>>>> 100644 >>>>>>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf >>>>>>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf >>>>>>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf >>>>>>>>>>> >>>>>>>>>>> BASE_DISTRO = "ubuntu" >>>>>>>>>>> BASE_DISTRO_CODENAME = "focal" >>>>>>>>>>> >>>>>>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}" >>>>>>>>>>> + >>>>>>>>>>> >>>>>>>>>>> DISTRO_APT_SOURCES_arm64 ?= >>>>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" >>>>>>>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list >>>>>>>>>>> conf/distro/${HOST_DISTRO}-ports.list"> >>>>>>>>>>> >>>>>>>>>>> diff --git a/meta/conf/distro/debian-common.conf >>>>>>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b >>>>>>>>>>> 100644 >>>>>>>>>>> --- a/meta/conf/distro/debian-common.conf >>>>>>>>>>> +++ b/meta/conf/distro/debian-common.conf >>>>>>>>>>> @@ -5,6 +5,8 @@ >>>>>>>>>>> >>>>>>>>>>> BASE_DISTRO = "debian" >>>>>>>>>>> >>>>>>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}" >>>>>>>>>>> + >>>>>>>>>>> >>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>>>>>>>>>> DISTRO_APT_SOURCES ?= >>>>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> >>>>>>>>>>> >>>>>>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index >>>>>>>>>>> f32d192e..2ee0d4cc 100644 >>>>>>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>>>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>>>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = >>>>>>>>>>> ",gnupg" >>>>>>>>>>> >>>>>>>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = >>>>>>>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if >>>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO >>>>>>>>>>> = >>>>>>>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == >>>>>>>>>>> '1' >>>>>>>>>>> else >>>>>>>>>>> 'DISTRO')}"> >>>>>>>>>>> >>>>>>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if >>>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"> >>>>>>>>>>> >>>>>>>>>>> FILESEXTRAPATHS_append = ":${BBPATH}" >>>>>>>>>>> >>>>>>>>>>> inherit deb-dl-dir >>>>>>>>>>> >>>>>>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, >>>>> >>>>> aptsources_entry_list): >>>>>>>>>>> import re >>>>>>>>>>> >>>>>>>>>>> if > bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): >>>>>>>>>>> - premirrors = "\S* >>>>>>>>>>> file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" >>>>>>>>>>> + premirrors = "\S* >>>>>>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> >>>>>>>>>>> >>>>>>>>>>> else: >>>>>>>>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or >>>>>>>>>>> "" >>>>>>>>>>> >>>>>>>>>>> mirror_list = [entry.split() >>>>>>>>>>> >>>>>>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() { >>>>>>>>>>> >>>>>>>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/boot >>>>>>>>>>> st >>>>>>>>>>> ra >>>>>>>>>>> p" >>>>>>>>>>> >>>>>>>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" >>>>>>>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then >>>>>>>>>>> >>>>>>>>>>> - line="file:///base-apt/${BASE_DISTRO} >>>>>>>>>>> ${BASE_DISTRO_CODENAME} >>>>>>>>>>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} >>>>>>>>>>> ${BASE_DISTRO_CODENAME} main"> >>>>>>>>>>> >>>>>>>>>>> if [ -z "${BASE_REPO_KEY}" ]; then >>>>>>>>>>> >>>>>>>>>>> line="[trusted=yes] ${line}" >>>>>>>>>>> >>>>>>>>>>> fi >>>>>>>>>>> echo "deb ${line}" > >>>>>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" >>>>>>>>>>> >>>>>>>>>>> + line="file:///base-apt/${BASE_DISTRO} >>>>>>>>>>> ${BASE_DISTRO_CODENAME} >>>>>>>>>>> main" >>>>>>>>>> >>>>>>>>>> Won't this cause duplicate lines (with complaints of apt) in case >>>>>>>>>> of >>>>>>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO? >>>>>>>>> >>>>>>>>> No, while I don't add new lines (no new "echo"s). I just modify >>>>>>>>> "deb-src" >>>>>>>>> line while it should point to BASE_DISTRO even for host case. >>>>>>>>> Without >>>>>>>>> it, >>>>>>>>> there was a problem with "hello" (apt_fetch failed to download >>>>>>>>> cached >>>>>>>>> sources from "host" repo, while sources were cached in "target" one. >>>>>>>>> >>>>>>>>> That's how it looks now: >>>>>>>>> | $ cat >>>>>>>>> | isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base- >>>>>>>>> >>>>>>>>> apt.list >>>>>>>>> >>>>>>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main >>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main >>>>>>>>> | $ cat >>>>>>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt. >>>>>>>>> | li >>>>>>>>> | st >>>>>>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main >>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main >>>>>>>> >>>>>>>> Indeed, I misintepreted the code. >>>>>>>> >>>>>>>> But I think we still have an issue on the deb-src population side: >>>>>>>> >>>>>>>> # ls build/downloads/deb-src/ >>>>>>>> iot2050-debian >>>>>>>> >>>>>>>> And that was with this patch applied. >>>>>>>> >>>>>>>> Jan >>>>>>> >>>>>>> Actually, I checked the case with the following configuration: >>>>>>> >>>>>>> distro: my-debian >>>>>>> >>>>>>> local_conf_header: >>>>>>> nodebian: | >>>>>>> >>>>>>> require conf/distro/debian-bullseye.conf >>>>>>> DISTRO_NAME = "My Debian System" >>>>>>> >>>>>>> So, I simulated renaming distro that is done in meta-iot2050. >>>>>>> >>>>>>> Cached build was OK, "hello" package (that we build from sources) was >>>>>>> >>>>>>> downloaded at first build to "my-debian" directory: >>>>>>> | $ ls build/downloads/deb-src/my-debian/hello/ >>>>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz >>>>>>> >>>>>>> and second build picked the files with no problem. >>>>>>> >>>>>>> The same with "raspios-bullseye" target - "hello" sources are >>>>>>> downloaded >>>>>>> to >>>>>>> >>>>>>> the "target" dir: >>>>>>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/ >>>>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz >>>>>>> >>>>>>> I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" and >>>>>>> it >>>>>>> was OK (honestly, I don't know who could require cached build with >>>>>>> this >>>>>>> option enabled :) ). >>>>>>> >>>>>>> Anyway, I'll also try to check my changes with meta-iot2050 layer also >>>>>>> (despite some packages in it are currently not compatible with >>>>>>> BB_NO_NETWORK option) >>>>>> >>>>>> Please then test without the coral packages, they have an own issue >>>>>> /wrt >>>>>> offline builds. >>>>> >>>>> Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works >>>> >>>> ...but it caches to the wrong downloads/deb-src path, no? >>>> >>>> Jan >>> >>> Why wrong? >>> >>> | meta-iot2050/build $ tree -d downloads/deb* >>> | downloads/deb >>> | ├── debian-bullseye >>> | └── iot2050-debian >>> | downloads/deb-src >>> | └── iot2050-debian >>> | >>> | ├── openssl >>> | └── swig >>> >>> `do_apt_fetch` works with `${DISTRO}` and downloads sources to the >>> corresponding directory. >> >> deb-src fetching still lacks base-distro normalization, like deb also has: >> > > Saying "normalization", do you mean the following: if ${DISTRO} is just > renaming (e.g. "iot2050-debian") everything should be downloaded to and picked > up from the directory named "debian-bullseye"? > Yes, just like you did for binary debs: Reduce the DISTRO to its BASE_DISTRO, then use that as key for the cache. The current situation is inconsistent and non-optimal, even if working. Jan -- Siemens AG, Technology Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs 2022-11-28 6:51 ` Jan Kiszka @ 2022-11-28 11:12 ` Uladzimir Bely 2022-11-28 11:45 ` Jan Kiszka 0 siblings, 1 reply; 18+ messages in thread From: Uladzimir Bely @ 2022-11-28 11:12 UTC (permalink / raw) To: isar-users, Jan Kiszka In the email from Monday, 28 November 2022 09:51:12 +03 user Jan Kiszka wrote: > On 28.11.22 07:41, Uladzimir Bely wrote: > > In email from Monday, 28 November 2022 09:32:22 +03 You wrote: > >> On 28.11.22 07:05, Uladzimir Bely wrote: > >>> In the email from Sunday, 27 November 2022 22:05:53 +03 user Jan Kiszka > > > > wrote: > >>>> On 26.11.22 06:30, Uladzimir Bely wrote: > >>>>> In the email from Friday, 25 November 2022 18:40:39 +03 user Jan > >>>>> Kiszka > >>> > >>> wrote: > >>>>>> On 25.11.22 15:48, Uladzimir Bely wrote: > >>>>>>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan > >>>>>>> Kiszka > >>>>> > >>>>> wrote: > >>>>>>>> On 25.11.22 08:56, Uladzimir Bely wrote: > >>>>>>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan > >>>>>>>>> Kiszka > >>>>>>> > >>>>>>> wrote: > >>>>>>>>>> On 25.11.22 07:42, Uladzimir Bely wrote: > >>>>>>>>>>> Downstreams may want to change DISTRO to some custom value. > >>>>>>>>>>> When cross-building, this leads to downloading packages for host > >>>>>>>>>>> and target distros to the different download subdirs. > >>>>>>>>>>> > >>>>>>>>>>> While base-apt is populated only from DISTRO download subdir, > >>>>>>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due > >>>>>>>>>>> to > >>>>>>>>>>> missing packages in base-apt. > >>>>>>>>>>> > >>>>>>>>>>> The similar issue happens with cross-building raspberry targets > >>>>>>>>>>> since they use their own mirrors and DISTRO. > >>>>>>>>>>> > >>>>>>>>>>> Fix the issue by splitting base-apt by introducing > >>>>>>>>>>> BASE_HOST_DISTRO > >>>>>>>>>>> variable that is used instead of BASE_DISTRO for caching builds > >>>>>>>>>>> for host-related components (e.g., isar-bootstrap, > >>>>>>>>>>> buildchroot-host, > >>>>>>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO. > >>>>>>>>>>> > >>>>>>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de> > >>>>>>>>>>> --- > >>>>>>>>>>> > >>>>>>>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 + > >>>>>>>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 + > >>>>>>>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 + > >>>>>>>>>>> meta/conf/distro/debian-common.conf | 2 + > >>>>>>>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++- > >>>>>>>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47 > >>>>>>>>>>> ++++++++++--------- > >>>>>>>>>>> 6 files changed, 40 insertions(+), 24 deletions(-) > >>>>>>>>>>> > >>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf > >>>>>>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index > >>>>>>>>>>> c8f523b3..66d965c9 > >>>>>>>>>>> 100644 > >>>>>>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf > >>>>>>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf > >>>>>>>>>>> @@ -6,6 +6,8 @@ > >>>>>>>>>>> > >>>>>>>>>>> BASE_DISTRO = "raspbian" > >>>>>>>>>>> BASE_DISTRO_CODENAME = "stretch" > >>>>>>>>>>> > >>>>>>>>>>> +BASE_HOST_DISTRO = "debian" > >>>>>>>>>> > >>>>>>>>>> Better prefix it with HOST, to be consistent: > >>>>>>>>>> > >>>>>>>>>> HOST_BASE_DISTRO > >>>>>>>>> > >>>>>>>>> Not a problem, if you think it would be better. > >>>>>>>>> > >>>>>>>>>>> + > >>>>>>>>>>> > >>>>>>>>>>> HOST_DISTRO ?= "debian-stretch" > >>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>>>>>>>>>> > >>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf > >>>>>>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index > >>>>>>>>>>> 5e4a09c9..7bd81dd6 > >>>>>>>>>>> 100644 > >>>>>>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf > >>>>>>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf > >>>>>>>>>>> @@ -6,6 +6,8 @@ > >>>>>>>>>>> > >>>>>>>>>>> BASE_DISTRO = "raspios" > >>>>>>>>>>> BASE_DISTRO_arm64 = "debian" > >>>>>>>>>>> > >>>>>>>>>>> +BASE_HOST_DISTRO ?= "debian" > >>>>>>>>>>> + > >>>>>>>>>>> > >>>>>>>>>>> BASE_DISTRO_CODENAME = "bullseye" > >>>>>>>>>>> HOST_DISTRO ?= "debian-bullseye" > >>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>>>>>>>>>> > >>>>>>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf > >>>>>>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index > >>>>>>>>>>> 379b5b30..75da79a1 > >>>>>>>>>>> 100644 > >>>>>>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf > >>>>>>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf > >>>>>>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf > >>>>>>>>>>> > >>>>>>>>>>> BASE_DISTRO = "ubuntu" > >>>>>>>>>>> BASE_DISTRO_CODENAME = "focal" > >>>>>>>>>>> > >>>>>>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}" > >>>>>>>>>>> + > >>>>>>>>>>> > >>>>>>>>>>> DISTRO_APT_SOURCES_arm64 ?= > >>>>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" > >>>>>>>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?= > >>>>>>>>>>> "conf/distro/${HOST_DISTRO}.list > >>>>>>>>>>> conf/distro/${HOST_DISTRO}-ports.list"> > >>>>>>>>>>> > >>>>>>>>>>> diff --git a/meta/conf/distro/debian-common.conf > >>>>>>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b > >>>>>>>>>>> 100644 > >>>>>>>>>>> --- a/meta/conf/distro/debian-common.conf > >>>>>>>>>>> +++ b/meta/conf/distro/debian-common.conf > >>>>>>>>>>> @@ -5,6 +5,8 @@ > >>>>>>>>>>> > >>>>>>>>>>> BASE_DISTRO = "debian" > >>>>>>>>>>> > >>>>>>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}" > >>>>>>>>>>> + > >>>>>>>>>>> > >>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" > >>>>>>>>>>> DISTRO_APT_SOURCES ?= > >>>>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> > >>>>>>>>>>> > >>>>>>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > >>>>>>>>>>> f32d192e..2ee0d4cc 100644 > >>>>>>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>>>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > >>>>>>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = > >>>>>>>>>>> ",gnupg" > >>>>>>>>>>> > >>>>>>>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = > >>>>>>>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if > >>>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" > >>>>>>>>>>> BOOTSTRAP_DISTRO > >>>>>>>>>>> = > >>>>>>>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == > >>>>>>>>>>> '1' > >>>>>>>>>>> else > >>>>>>>>>>> 'DISTRO')}"> > >>>>>>>>>>> > >>>>>>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if > >>>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"> > >>>>>>>>>>> > >>>>>>>>>>> FILESEXTRAPATHS_append = ":${BBPATH}" > >>>>>>>>>>> > >>>>>>>>>>> inherit deb-dl-dir > >>>>>>>>>>> > >>>>>>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, > >>>>> > >>>>> aptsources_entry_list): > >>>>>>>>>>> import re > >>>>>>>>>>> > >>>>>>>>>>> if > > > > bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): > >>>>>>>>>>> - premirrors = "\S* > >>>>>>>>>>> file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" > >>>>>>>>>>> + premirrors = "\S* > >>>>>>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> > >>>>>>>>>>> > >>>>>>>>>>> else: > >>>>>>>>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or > >>>>>>>>>>> "" > >>>>>>>>>>> > >>>>>>>>>>> mirror_list = [entry.split() > >>>>>>>>>>> > >>>>>>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() { > >>>>>>>>>>> > >>>>>>>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/bo > >>>>>>>>>>> ot > >>>>>>>>>>> st > >>>>>>>>>>> ra > >>>>>>>>>>> p" > >>>>>>>>>>> > >>>>>>>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" > >>>>>>>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then > >>>>>>>>>>> > >>>>>>>>>>> - line="file:///base-apt/${BASE_DISTRO} > >>>>>>>>>>> ${BASE_DISTRO_CODENAME} > >>>>>>>>>>> main" + > >>>>>>>>>>> line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} > >>>>>>>>>>> ${BASE_DISTRO_CODENAME} main"> > >>>>>>>>>>> > >>>>>>>>>>> if [ -z "${BASE_REPO_KEY}" ]; then > >>>>>>>>>>> > >>>>>>>>>>> line="[trusted=yes] ${line}" > >>>>>>>>>>> > >>>>>>>>>>> fi > >>>>>>>>>>> echo "deb ${line}" > > >>>>>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" > >>>>>>>>>>> > >>>>>>>>>>> + line="file:///base-apt/${BASE_DISTRO} > >>>>>>>>>>> ${BASE_DISTRO_CODENAME} > >>>>>>>>>>> main" > >>>>>>>>>> > >>>>>>>>>> Won't this cause duplicate lines (with complaints of apt) in case > >>>>>>>>>> of > >>>>>>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO? > >>>>>>>>> > >>>>>>>>> No, while I don't add new lines (no new "echo"s). I just modify > >>>>>>>>> "deb-src" > >>>>>>>>> line while it should point to BASE_DISTRO even for host case. > >>>>>>>>> Without > >>>>>>>>> it, > >>>>>>>>> there was a problem with "hello" (apt_fetch failed to download > >>>>>>>>> cached > >>>>>>>>> sources from "host" repo, while sources were cached in "target" > >>>>>>>>> one. > >>>>>>>>> > >>>>>>>>> That's how it looks now: > >>>>>>>>> | $ cat > >>>>>>>>> | isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base- > >>>>>>>>> > >>>>>>>>> apt.list > >>>>>>>>> > >>>>>>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main > >>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main > >>>>>>>>> | $ cat > >>>>>>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-ap > >>>>>>>>> | t. > >>>>>>>>> | li > >>>>>>>>> | st > >>>>>>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main > >>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main > >>>>>>>> > >>>>>>>> Indeed, I misintepreted the code. > >>>>>>>> > >>>>>>>> But I think we still have an issue on the deb-src population side: > >>>>>>>> > >>>>>>>> # ls build/downloads/deb-src/ > >>>>>>>> iot2050-debian > >>>>>>>> > >>>>>>>> And that was with this patch applied. > >>>>>>>> > >>>>>>>> Jan > >>>>>>> > >>>>>>> Actually, I checked the case with the following configuration: > >>>>>>> > >>>>>>> distro: my-debian > >>>>>>> > >>>>>>> local_conf_header: > >>>>>>> nodebian: | > >>>>>>> > >>>>>>> require conf/distro/debian-bullseye.conf > >>>>>>> DISTRO_NAME = "My Debian System" > >>>>>>> > >>>>>>> So, I simulated renaming distro that is done in meta-iot2050. > >>>>>>> > >>>>>>> Cached build was OK, "hello" package (that we build from sources) > >>>>>>> was > >>>>>>> > >>>>>>> downloaded at first build to "my-debian" directory: > >>>>>>> | $ ls build/downloads/deb-src/my-debian/hello/ > >>>>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc > >>>>>>> | hello_2.10.orig.tar.gz > >>>>>>> > >>>>>>> and second build picked the files with no problem. > >>>>>>> > >>>>>>> The same with "raspios-bullseye" target - "hello" sources are > >>>>>>> downloaded > >>>>>>> to > >>>>>>> > >>>>>>> the "target" dir: > >>>>>>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/ > >>>>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc > >>>>>>> | hello_2.10.orig.tar.gz > >>>>>>> > >>>>>>> I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" > >>>>>>> and > >>>>>>> it > >>>>>>> was OK (honestly, I don't know who could require cached build with > >>>>>>> this > >>>>>>> option enabled :) ). > >>>>>>> > >>>>>>> Anyway, I'll also try to check my changes with meta-iot2050 layer > >>>>>>> also > >>>>>>> (despite some packages in it are currently not compatible with > >>>>>>> BB_NO_NETWORK option) > >>>>>> > >>>>>> Please then test without the coral packages, they have an own issue > >>>>>> /wrt > >>>>>> offline builds. > >>>>> > >>>>> Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works > >>>> > >>>> ...but it caches to the wrong downloads/deb-src path, no? > >>>> > >>>> Jan > >>> > >>> Why wrong? > >>> > >>> | meta-iot2050/build $ tree -d downloads/deb* > >>> | downloads/deb > >>> | ├── debian-bullseye > >>> | └── iot2050-debian > >>> | downloads/deb-src > >>> | └── iot2050-debian > >>> | > >>> | ├── openssl > >>> | └── swig > >>> > >>> `do_apt_fetch` works with `${DISTRO}` and downloads sources to the > >>> corresponding directory. > >> > >> deb-src fetching still lacks base-distro normalization, like deb also has: > > Saying "normalization", do you mean the following: if ${DISTRO} is just > > renaming (e.g. "iot2050-debian") everything should be downloaded to and > > picked up from the directory named "debian-bullseye"? > > Yes, just like you did for binary debs: Reduce the DISTRO to its > BASE_DISTRO, then use that as key for the cache. The current situation > is inconsistent and non-optimal, even if working. > > Jan I've just pushed v4 of patchset that still doesn't include this wish. For now, I would apply it to the isar/next to have cached cross build working. We have some plans to improve base-apt in general (there were some PoC patches on list), so improvements you mention could be done a bit later - for now I need to switch to some other tasks. -- Uladzimir Bely ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs 2022-11-28 11:12 ` Uladzimir Bely @ 2022-11-28 11:45 ` Jan Kiszka 0 siblings, 0 replies; 18+ messages in thread From: Jan Kiszka @ 2022-11-28 11:45 UTC (permalink / raw) To: Uladzimir Bely, isar-users On 28.11.22 12:12, Uladzimir Bely wrote: > In the email from Monday, 28 November 2022 09:51:12 +03 user Jan Kiszka wrote: >> On 28.11.22 07:41, Uladzimir Bely wrote: >>> In email from Monday, 28 November 2022 09:32:22 +03 You wrote: >>>> On 28.11.22 07:05, Uladzimir Bely wrote: >>>>> In the email from Sunday, 27 November 2022 22:05:53 +03 user Jan Kiszka >>> >>> wrote: >>>>>> On 26.11.22 06:30, Uladzimir Bely wrote: >>>>>>> In the email from Friday, 25 November 2022 18:40:39 +03 user Jan >>>>>>> Kiszka >>>>> >>>>> wrote: >>>>>>>> On 25.11.22 15:48, Uladzimir Bely wrote: >>>>>>>>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan >>>>>>>>> Kiszka >>>>>>> >>>>>>> wrote: >>>>>>>>>> On 25.11.22 08:56, Uladzimir Bely wrote: >>>>>>>>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan >>>>>>>>>>> Kiszka >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>>>>> On 25.11.22 07:42, Uladzimir Bely wrote: >>>>>>>>>>>>> Downstreams may want to change DISTRO to some custom value. >>>>>>>>>>>>> When cross-building, this leads to downloading packages for host >>>>>>>>>>>>> and target distros to the different download subdirs. >>>>>>>>>>>>> >>>>>>>>>>>>> While base-apt is populated only from DISTRO download subdir, >>>>>>>>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due >>>>>>>>>>>>> to >>>>>>>>>>>>> missing packages in base-apt. >>>>>>>>>>>>> >>>>>>>>>>>>> The similar issue happens with cross-building raspberry targets >>>>>>>>>>>>> since they use their own mirrors and DISTRO. >>>>>>>>>>>>> >>>>>>>>>>>>> Fix the issue by splitting base-apt by introducing >>>>>>>>>>>>> BASE_HOST_DISTRO >>>>>>>>>>>>> variable that is used instead of BASE_DISTRO for caching builds >>>>>>>>>>>>> for host-related components (e.g., isar-bootstrap, >>>>>>>>>>>>> buildchroot-host, >>>>>>>>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO. >>>>>>>>>>>>> >>>>>>>>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de> >>>>>>>>>>>>> --- >>>>>>>>>>>>> >>>>>>>>>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 + >>>>>>>>>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 + >>>>>>>>>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 + >>>>>>>>>>>>> meta/conf/distro/debian-common.conf | 2 + >>>>>>>>>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++- >>>>>>>>>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47 >>>>>>>>>>>>> ++++++++++--------- >>>>>>>>>>>>> 6 files changed, 40 insertions(+), 24 deletions(-) >>>>>>>>>>>>> >>>>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf >>>>>>>>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index >>>>>>>>>>>>> c8f523b3..66d965c9 >>>>>>>>>>>>> 100644 >>>>>>>>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf >>>>>>>>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf >>>>>>>>>>>>> @@ -6,6 +6,8 @@ >>>>>>>>>>>>> >>>>>>>>>>>>> BASE_DISTRO = "raspbian" >>>>>>>>>>>>> BASE_DISTRO_CODENAME = "stretch" >>>>>>>>>>>>> >>>>>>>>>>>>> +BASE_HOST_DISTRO = "debian" >>>>>>>>>>>> >>>>>>>>>>>> Better prefix it with HOST, to be consistent: >>>>>>>>>>>> >>>>>>>>>>>> HOST_BASE_DISTRO >>>>>>>>>>> >>>>>>>>>>> Not a problem, if you think it would be better. >>>>>>>>>>> >>>>>>>>>>>>> + >>>>>>>>>>>>> >>>>>>>>>>>>> HOST_DISTRO ?= "debian-stretch" >>>>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>>>>>>>>>>>> >>>>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf >>>>>>>>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index >>>>>>>>>>>>> 5e4a09c9..7bd81dd6 >>>>>>>>>>>>> 100644 >>>>>>>>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf >>>>>>>>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf >>>>>>>>>>>>> @@ -6,6 +6,8 @@ >>>>>>>>>>>>> >>>>>>>>>>>>> BASE_DISTRO = "raspios" >>>>>>>>>>>>> BASE_DISTRO_arm64 = "debian" >>>>>>>>>>>>> >>>>>>>>>>>>> +BASE_HOST_DISTRO ?= "debian" >>>>>>>>>>>>> + >>>>>>>>>>>>> >>>>>>>>>>>>> BASE_DISTRO_CODENAME = "bullseye" >>>>>>>>>>>>> HOST_DISTRO ?= "debian-bullseye" >>>>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>>>>>>>>>>>> >>>>>>>>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf >>>>>>>>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index >>>>>>>>>>>>> 379b5b30..75da79a1 >>>>>>>>>>>>> 100644 >>>>>>>>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf >>>>>>>>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf >>>>>>>>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf >>>>>>>>>>>>> >>>>>>>>>>>>> BASE_DISTRO = "ubuntu" >>>>>>>>>>>>> BASE_DISTRO_CODENAME = "focal" >>>>>>>>>>>>> >>>>>>>>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}" >>>>>>>>>>>>> + >>>>>>>>>>>>> >>>>>>>>>>>>> DISTRO_APT_SOURCES_arm64 ?= >>>>>>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list" >>>>>>>>>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?= >>>>>>>>>>>>> "conf/distro/${HOST_DISTRO}.list >>>>>>>>>>>>> conf/distro/${HOST_DISTRO}-ports.list"> >>>>>>>>>>>>> >>>>>>>>>>>>> diff --git a/meta/conf/distro/debian-common.conf >>>>>>>>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b >>>>>>>>>>>>> 100644 >>>>>>>>>>>>> --- a/meta/conf/distro/debian-common.conf >>>>>>>>>>>>> +++ b/meta/conf/distro/debian-common.conf >>>>>>>>>>>>> @@ -5,6 +5,8 @@ >>>>>>>>>>>>> >>>>>>>>>>>>> BASE_DISTRO = "debian" >>>>>>>>>>>>> >>>>>>>>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}" >>>>>>>>>>>>> + >>>>>>>>>>>>> >>>>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list" >>>>>>>>>>>>> DISTRO_APT_SOURCES ?= >>>>>>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> >>>>>>>>>>>>> >>>>>>>>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>>>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index >>>>>>>>>>>>> f32d192e..2ee0d4cc 100644 >>>>>>>>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>>>>>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>>>>>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = >>>>>>>>>>>>> ",gnupg" >>>>>>>>>>>>> >>>>>>>>>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = >>>>>>>>>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if >>>>>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" >>>>>>>>>>>>> BOOTSTRAP_DISTRO >>>>>>>>>>>>> = >>>>>>>>>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == >>>>>>>>>>>>> '1' >>>>>>>>>>>>> else >>>>>>>>>>>>> 'DISTRO')}"> >>>>>>>>>>>>> >>>>>>>>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if >>>>>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"> >>>>>>>>>>>>> >>>>>>>>>>>>> FILESEXTRAPATHS_append = ":${BBPATH}" >>>>>>>>>>>>> >>>>>>>>>>>>> inherit deb-dl-dir >>>>>>>>>>>>> >>>>>>>>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, >>>>>>> >>>>>>> aptsources_entry_list): >>>>>>>>>>>>> import re >>>>>>>>>>>>> >>>>>>>>>>>>> if >>> >>> bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): >>>>>>>>>>>>> - premirrors = "\S* >>>>>>>>>>>>> file://${REPO_BASE_DIR}/${BASE_DISTRO}\n" >>>>>>>>>>>>> + premirrors = "\S* >>>>>>>>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> >>>>>>>>>>>>> >>>>>>>>>>>>> else: >>>>>>>>>>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or >>>>>>>>>>>>> "" >>>>>>>>>>>>> >>>>>>>>>>>>> mirror_list = [entry.split() >>>>>>>>>>>>> >>>>>>>>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() { >>>>>>>>>>>>> >>>>>>>>>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/bo >>>>>>>>>>>>> ot >>>>>>>>>>>>> st >>>>>>>>>>>>> ra >>>>>>>>>>>>> p" >>>>>>>>>>>>> >>>>>>>>>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" >>>>>>>>>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then >>>>>>>>>>>>> >>>>>>>>>>>>> - line="file:///base-apt/${BASE_DISTRO} >>>>>>>>>>>>> ${BASE_DISTRO_CODENAME} >>>>>>>>>>>>> main" + >>>>>>>>>>>>> line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} >>>>>>>>>>>>> ${BASE_DISTRO_CODENAME} main"> >>>>>>>>>>>>> >>>>>>>>>>>>> if [ -z "${BASE_REPO_KEY}" ]; then >>>>>>>>>>>>> >>>>>>>>>>>>> line="[trusted=yes] ${line}" >>>>>>>>>>>>> >>>>>>>>>>>>> fi >>>>>>>>>>>>> echo "deb ${line}" > >>>>>>>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" >>>>>>>>>>>>> >>>>>>>>>>>>> + line="file:///base-apt/${BASE_DISTRO} >>>>>>>>>>>>> ${BASE_DISTRO_CODENAME} >>>>>>>>>>>>> main" >>>>>>>>>>>> >>>>>>>>>>>> Won't this cause duplicate lines (with complaints of apt) in case >>>>>>>>>>>> of >>>>>>>>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO? >>>>>>>>>>> >>>>>>>>>>> No, while I don't add new lines (no new "echo"s). I just modify >>>>>>>>>>> "deb-src" >>>>>>>>>>> line while it should point to BASE_DISTRO even for host case. >>>>>>>>>>> Without >>>>>>>>>>> it, >>>>>>>>>>> there was a problem with "hello" (apt_fetch failed to download >>>>>>>>>>> cached >>>>>>>>>>> sources from "host" repo, while sources were cached in "target" >>>>>>>>>>> one. >>>>>>>>>>> >>>>>>>>>>> That's how it looks now: >>>>>>>>>>> | $ cat >>>>>>>>>>> | isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base- >>>>>>>>>>> >>>>>>>>>>> apt.list >>>>>>>>>>> >>>>>>>>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main >>>>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main >>>>>>>>>>> | $ cat >>>>>>>>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-ap >>>>>>>>>>> | t. >>>>>>>>>>> | li >>>>>>>>>>> | st >>>>>>>>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main >>>>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main >>>>>>>>>> >>>>>>>>>> Indeed, I misintepreted the code. >>>>>>>>>> >>>>>>>>>> But I think we still have an issue on the deb-src population side: >>>>>>>>>> >>>>>>>>>> # ls build/downloads/deb-src/ >>>>>>>>>> iot2050-debian >>>>>>>>>> >>>>>>>>>> And that was with this patch applied. >>>>>>>>>> >>>>>>>>>> Jan >>>>>>>>> >>>>>>>>> Actually, I checked the case with the following configuration: >>>>>>>>> >>>>>>>>> distro: my-debian >>>>>>>>> >>>>>>>>> local_conf_header: >>>>>>>>> nodebian: | >>>>>>>>> >>>>>>>>> require conf/distro/debian-bullseye.conf >>>>>>>>> DISTRO_NAME = "My Debian System" >>>>>>>>> >>>>>>>>> So, I simulated renaming distro that is done in meta-iot2050. >>>>>>>>> >>>>>>>>> Cached build was OK, "hello" package (that we build from sources) >>>>>>>>> was >>>>>>>>> >>>>>>>>> downloaded at first build to "my-debian" directory: >>>>>>>>> | $ ls build/downloads/deb-src/my-debian/hello/ >>>>>>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc >>>>>>>>> | hello_2.10.orig.tar.gz >>>>>>>>> >>>>>>>>> and second build picked the files with no problem. >>>>>>>>> >>>>>>>>> The same with "raspios-bullseye" target - "hello" sources are >>>>>>>>> downloaded >>>>>>>>> to >>>>>>>>> >>>>>>>>> the "target" dir: >>>>>>>>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/ >>>>>>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc >>>>>>>>> | hello_2.10.orig.tar.gz >>>>>>>>> >>>>>>>>> I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" >>>>>>>>> and >>>>>>>>> it >>>>>>>>> was OK (honestly, I don't know who could require cached build with >>>>>>>>> this >>>>>>>>> option enabled :) ). >>>>>>>>> >>>>>>>>> Anyway, I'll also try to check my changes with meta-iot2050 layer >>>>>>>>> also >>>>>>>>> (despite some packages in it are currently not compatible with >>>>>>>>> BB_NO_NETWORK option) >>>>>>>> >>>>>>>> Please then test without the coral packages, they have an own issue >>>>>>>> /wrt >>>>>>>> offline builds. >>>>>>> >>>>>>> Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works >>>>>> >>>>>> ...but it caches to the wrong downloads/deb-src path, no? >>>>>> >>>>>> Jan >>>>> >>>>> Why wrong? >>>>> >>>>> | meta-iot2050/build $ tree -d downloads/deb* >>>>> | downloads/deb >>>>> | ├── debian-bullseye >>>>> | └── iot2050-debian >>>>> | downloads/deb-src >>>>> | └── iot2050-debian >>>>> | >>>>> | ├── openssl >>>>> | └── swig >>>>> >>>>> `do_apt_fetch` works with `${DISTRO}` and downloads sources to the >>>>> corresponding directory. >>>> >>>> deb-src fetching still lacks base-distro normalization, like deb also > has: >>> Saying "normalization", do you mean the following: if ${DISTRO} is just >>> renaming (e.g. "iot2050-debian") everything should be downloaded to and >>> picked up from the directory named "debian-bullseye"? >> >> Yes, just like you did for binary debs: Reduce the DISTRO to its >> BASE_DISTRO, then use that as key for the cache. The current situation >> is inconsistent and non-optimal, even if working. >> >> Jan > > I've just pushed v4 of patchset that still doesn't include this wish. For now, > I would apply it to the isar/next to have cached cross build working. > > We have some plans to improve base-apt in general (there were some PoC patches > on list), so improvements you mention could be done a bit later - for now I > need to switch to some other tasks. > The problem is in meta/classes/rootfs.bbclass. It lacks normalization of ROOTFS_DISTRO to some "ROOTFS_BASE_DISTRO". That also explains why you still saw same packages under deb/iot2050-debian - you ran the final rootfs step which I didn't (due to earlier errors). Jan -- Siemens AG, Technology Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH v3 2/3] testsuite: Run signed repro test in cross mode 2022-11-25 6:42 [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Uladzimir Bely 2022-11-25 6:42 ` [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs Uladzimir Bely @ 2022-11-25 6:42 ` Uladzimir Bely 2022-11-25 6:42 ` [PATCH v3 3/3] testsuite: Replace bananapi with rpi-arm-v7 in signed repro tests Uladzimir Bely 2022-11-25 7:37 ` [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Jan Kiszka 3 siblings, 0 replies; 18+ messages in thread From: Uladzimir Bely @ 2022-11-25 6:42 UTC (permalink / raw) To: isar-users Both signed and unsigned tests are done in non-cross mode. Changing CROSS_COMPILE to "1" for one signed variant in order to cover cross-build from local `base-apt` testing Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- testsuite/citest.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/testsuite/citest.py b/testsuite/citest.py index 2dc78015..cf9139f8 100755 --- a/testsuite/citest.py +++ b/testsuite/citest.py @@ -32,7 +32,7 @@ class ReproTest(CIBaseTest): self.init() try: - self.perform_repro_test(targets, signed=True) + self.perform_repro_test(targets, signed=True, cross=True) finally: self.move_in_build_dir('tmp', 'tmp_repro_signed') -- 2.20.1 ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH v3 3/3] testsuite: Replace bananapi with rpi-arm-v7 in signed repro tests 2022-11-25 6:42 [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Uladzimir Bely 2022-11-25 6:42 ` [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs Uladzimir Bely 2022-11-25 6:42 ` [PATCH v3 2/3] testsuite: Run signed repro test in cross mode Uladzimir Bely @ 2022-11-25 6:42 ` Uladzimir Bely 2022-11-25 7:37 ` [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Jan Kiszka 3 siblings, 0 replies; 18+ messages in thread From: Uladzimir Bely @ 2022-11-25 6:42 UTC (permalink / raw) To: isar-users While rpi-arm-v7 target uses own repositories and DISTRO variable, it has more chances to fail in cross-build repro test, so it looks like a better choice for this test than debian-only bananapi target. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- testsuite/citest.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/testsuite/citest.py b/testsuite/citest.py index cf9139f8..d4fb177f 100755 --- a/testsuite/citest.py +++ b/testsuite/citest.py @@ -26,7 +26,7 @@ class ReproTest(CIBaseTest): """ def test_repro_signed(self): targets = [ - 'mc:bananapi-bullseye:isar-image-base', + 'mc:rpi-arm-v7-bullseye:isar-image-base', 'mc:qemuarm64-bullseye:isar-image-base' ] -- 2.20.1 ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used 2022-11-25 6:42 [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Uladzimir Bely ` (2 preceding siblings ...) 2022-11-25 6:42 ` [PATCH v3 3/3] testsuite: Replace bananapi with rpi-arm-v7 in signed repro tests Uladzimir Bely @ 2022-11-25 7:37 ` Jan Kiszka 3 siblings, 0 replies; 18+ messages in thread From: Jan Kiszka @ 2022-11-25 7:37 UTC (permalink / raw) To: Uladzimir Bely, isar-users On 25.11.22 07:42, Uladzimir Bely wrote: > This patchset is mostly intended for downstreams that prefer > to rename DISTRO variable (like `meat-iot2050`) and use ^^^^ Nice typo :) > ISAR_CROSS_COMPILE = "1". > > This also fixes the case with cross-build or raspberry targets > from cached base-apt repo. While it uses separate repositories > for host (Debian) and target (Raspbian) distros, base-apt is > splittend onto two parts corresponding to original repos. split > > Also, testsuite repro tests were updated to cover crosss-build case cross > and raspberry target is now used in it. > > Changes since v2: > - Cleaned garbage in commit message for patch 2. > > Changes since v1: > - Support cached cross-build from base-apt for different base distros > for host and target (e.g. raspberry). > > > Uladzimir Bely (3): > Populate base-apt from both DISTRO and HOST_DISTRO download dirs > testsuite: Run signed repro test in cross mode > testsuite: Replace bananapi with rpi-arm-v7 in signed repro tests > > meta-isar/conf/distro/raspbian-stretch.conf | 2 + > meta-isar/conf/distro/raspios-bullseye.conf | 2 + > meta-isar/conf/distro/ubuntu-focal.conf | 2 + > meta/conf/distro/debian-common.conf | 2 + > .../isar-bootstrap/isar-bootstrap.inc | 9 +++- > meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++--------- > testsuite/citest.py | 4 +- > 7 files changed, 42 insertions(+), 26 deletions(-) > Jan -- Siemens AG, Technology Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2022-11-28 11:45 UTC | newest] Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-11-25 6:42 [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Uladzimir Bely 2022-11-25 6:42 ` [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs Uladzimir Bely 2022-11-25 7:42 ` Jan Kiszka 2022-11-25 7:56 ` Uladzimir Bely 2022-11-25 14:11 ` Jan Kiszka 2022-11-25 14:48 ` Uladzimir Bely 2022-11-25 15:40 ` Jan Kiszka 2022-11-26 5:30 ` Uladzimir Bely 2022-11-27 19:05 ` Jan Kiszka 2022-11-28 6:05 ` Uladzimir Bely 2022-11-28 6:32 ` Jan Kiszka 2022-11-28 6:41 ` Uladzimir Bely 2022-11-28 6:51 ` Jan Kiszka 2022-11-28 11:12 ` Uladzimir Bely 2022-11-28 11:45 ` Jan Kiszka 2022-11-25 6:42 ` [PATCH v3 2/3] testsuite: Run signed repro test in cross mode Uladzimir Bely 2022-11-25 6:42 ` [PATCH v3 3/3] testsuite: Replace bananapi with rpi-arm-v7 in signed repro tests Uladzimir Bely 2022-11-25 7:37 ` [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Jan Kiszka
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox