* [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used
@ 2022-11-25 6:42 Uladzimir Bely
2022-11-25 6:42 ` [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs Uladzimir Bely
` (3 more replies)
0 siblings, 4 replies; 18+ messages in thread
From: Uladzimir Bely @ 2022-11-25 6:42 UTC (permalink / raw)
To: isar-users
This patchset is mostly intended for downstreams that prefer
to rename DISTRO variable (like `meat-iot2050`) and use
ISAR_CROSS_COMPILE = "1".
This also fixes the case with cross-build or raspberry targets
from cached base-apt repo. While it uses separate repositories
for host (Debian) and target (Raspbian) distros, base-apt is
splittend onto two parts corresponding to original repos.
Also, testsuite repro tests were updated to cover crosss-build case
and raspberry target is now used in it.
Changes since v2:
- Cleaned garbage in commit message for patch 2.
Changes since v1:
- Support cached cross-build from base-apt for different base distros
for host and target (e.g. raspberry).
Uladzimir Bely (3):
Populate base-apt from both DISTRO and HOST_DISTRO download dirs
testsuite: Run signed repro test in cross mode
testsuite: Replace bananapi with rpi-arm-v7 in signed repro tests
meta-isar/conf/distro/raspbian-stretch.conf | 2 +
meta-isar/conf/distro/raspios-bullseye.conf | 2 +
meta-isar/conf/distro/ubuntu-focal.conf | 2 +
meta/conf/distro/debian-common.conf | 2 +
.../isar-bootstrap/isar-bootstrap.inc | 9 +++-
meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++---------
testsuite/citest.py | 4 +-
7 files changed, 42 insertions(+), 26 deletions(-)
--
2.20.1
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs
2022-11-25 6:42 [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Uladzimir Bely
@ 2022-11-25 6:42 ` Uladzimir Bely
2022-11-25 7:42 ` Jan Kiszka
2022-11-25 6:42 ` [PATCH v3 2/3] testsuite: Run signed repro test in cross mode Uladzimir Bely
` (2 subsequent siblings)
3 siblings, 1 reply; 18+ messages in thread
From: Uladzimir Bely @ 2022-11-25 6:42 UTC (permalink / raw)
To: isar-users
Downstreams may want to change DISTRO to some custom value.
When cross-building, this leads to downloading packages for host
and target distros to the different download subdirs.
While base-apt is populated only from DISTRO download subdir,
second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to
missing packages in base-apt.
The similar issue happens with cross-building raspberry targets
since they use their own mirrors and DISTRO.
Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO
variable that is used instead of BASE_DISTRO for caching builds
for host-related components (e.g., isar-bootstrap, buildchroot-host,
sbuild-chroot-host). Target components still use BASE_DISTRO.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta-isar/conf/distro/raspbian-stretch.conf | 2 +
meta-isar/conf/distro/raspios-bullseye.conf | 2 +
meta-isar/conf/distro/ubuntu-focal.conf | 2 +
meta/conf/distro/debian-common.conf | 2 +
.../isar-bootstrap/isar-bootstrap.inc | 9 +++-
meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++---------
6 files changed, 40 insertions(+), 24 deletions(-)
diff --git a/meta-isar/conf/distro/raspbian-stretch.conf b/meta-isar/conf/distro/raspbian-stretch.conf
index c8f523b3..66d965c9 100644
--- a/meta-isar/conf/distro/raspbian-stretch.conf
+++ b/meta-isar/conf/distro/raspbian-stretch.conf
@@ -6,6 +6,8 @@
BASE_DISTRO = "raspbian"
BASE_DISTRO_CODENAME = "stretch"
+BASE_HOST_DISTRO = "debian"
+
HOST_DISTRO ?= "debian-stretch"
HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
diff --git a/meta-isar/conf/distro/raspios-bullseye.conf b/meta-isar/conf/distro/raspios-bullseye.conf
index 5e4a09c9..7bd81dd6 100644
--- a/meta-isar/conf/distro/raspios-bullseye.conf
+++ b/meta-isar/conf/distro/raspios-bullseye.conf
@@ -6,6 +6,8 @@
BASE_DISTRO = "raspios"
BASE_DISTRO_arm64 = "debian"
+BASE_HOST_DISTRO ?= "debian"
+
BASE_DISTRO_CODENAME = "bullseye"
HOST_DISTRO ?= "debian-bullseye"
HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
diff --git a/meta-isar/conf/distro/ubuntu-focal.conf b/meta-isar/conf/distro/ubuntu-focal.conf
index 379b5b30..75da79a1 100644
--- a/meta-isar/conf/distro/ubuntu-focal.conf
+++ b/meta-isar/conf/distro/ubuntu-focal.conf
@@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
BASE_DISTRO = "ubuntu"
BASE_DISTRO_CODENAME = "focal"
+BASE_HOST_DISTRO = "${BASE_DISTRO}"
+
DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list conf/distro/${HOST_DISTRO}-ports.list"
diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf
index c18eebc2..d7a13f4b 100644
--- a/meta/conf/distro/debian-common.conf
+++ b/meta/conf/distro/debian-common.conf
@@ -5,6 +5,8 @@
BASE_DISTRO = "debian"
+BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
+
HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
DISTRO_APT_SOURCES ?= "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index f32d192e..2ee0d4cc 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg"
DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "${@https_support(d)}"
DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}"
+BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"
FILESEXTRAPATHS_append = ":${BBPATH}"
inherit deb-dl-dir
@@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list):
import re
if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
- premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
+ premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"
else:
premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or ""
mirror_list = [entry.split()
@@ -319,11 +320,15 @@ do_bootstrap() {
"${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
- line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
+ line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
if [ -z "${BASE_REPO_KEY}" ]; then
line="[trusted=yes] ${line}"
fi
echo "deb ${line}" > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
+ line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
+ if [ -z "${BASE_REPO_KEY}" ]; then
+ line="[trusted=yes] ${line}"
+ fi
echo "deb-src ${line}" >> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
mkdir -p ${ROOTFSDIR}/base-apt
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index 506a28ff..e928cd52 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -13,7 +13,10 @@ KEYFILES ?= ""
BASE_REPO_FEATURES ?= ""
populate_base_apt() {
- find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do
+ distro="${1}"
+ base_distro="${2}"
+
+ find "${DEBDIR}"/"${distro}" -name '*\.deb' | while read package; do
# NOTE: due to packages stored by reprepro are not modified, we can
# use search by filename to check if package is already in repo. In
# addition, md5sums are compared to ensure that the package is the
@@ -22,25 +25,25 @@ populate_base_apt() {
# Check if this package is already in base-apt
ret=0
- repo_contains_package "${REPO_BASE_DIR}/${BASE_DISTRO}" "${package}" ||
+ repo_contains_package "${REPO_BASE_DIR}/${base_distro}" "${package}" ||
ret=$?
[ "${ret}" = "0" ] && continue
if [ "${ret}" = "1" ]; then
- repo_del_package "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
- "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
+ repo_del_package "${REPO_BASE_DIR}"/"${base_distro}" \
+ "${REPO_BASE_DB_DIR}"/"${base_distro}" \
"${BASE_DISTRO_CODENAME}" \
- "${base_apt_p}"
+ "${package}"
fi
- repo_add_packages "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
- "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
+ repo_add_packages "${REPO_BASE_DIR}"/"${base_distro}" \
+ "${REPO_BASE_DB_DIR}"/"${base_distro}" \
"${BASE_DISTRO_CODENAME}" \
"${package}"
done
- find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read package; do
- repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
- "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
+ find "${DEBSRCDIR}"/"${distro}" -name '*\.dsc' | while read package; do
+ repo_add_srcpackage "${REPO_BASE_DIR}"/"${base_distro}" \
+ "${REPO_BASE_DB_DIR}"/"${base_distro}" \
"${BASE_DISTRO_CODENAME}" \
"${package}"
done
@@ -55,20 +58,20 @@ repo() {
"${BASE_DISTRO_CODENAME}" \
"${WORKDIR}/distributions.in" \
"${KEYFILES}"
-
- if [ -d '${BUILDCHROOT_HOST_DIR}/var/cache/apt' ] &&
- [ '${DISTRO}' != '${HOST_DISTRO}' ]; then
- # We would need two separate repository paths for that.
- # Otherwise packages (especially the 'all' arch ones) from one
- # distribution can influence the package versions of the other
- # distribution.
- bbfatal "Different host and target distributions are currently not supported." \
- "Try it without cross-build."
- fi
-
- populate_base_apt
+ populate_base_apt "${DISTRO}" "${BASE_DISTRO}"
repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
"${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
+
+ if [ '${DISTRO}' != '${HOST_DISTRO}' ]; then
+ repo_create "${REPO_BASE_DIR}"/"${BASE_HOST_DISTRO}" \
+ "${REPO_BASE_DB_DIR}"/"${BASE_HOST_DISTRO}" \
+ "${BASE_DISTRO_CODENAME}" \
+ "${WORKDIR}/distributions.in" \
+ "${KEYFILES}"
+ populate_base_apt "${HOST_DISTRO}" "${BASE_HOST_DISTRO}"
+ repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_HOST_DISTRO}" \
+ "${REPO_BASE_DB_DIR}"/"${BASE_HOST_DISTRO}"
+ fi
}
python do_cache() {
--
2.20.1
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH v3 2/3] testsuite: Run signed repro test in cross mode
2022-11-25 6:42 [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Uladzimir Bely
2022-11-25 6:42 ` [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs Uladzimir Bely
@ 2022-11-25 6:42 ` Uladzimir Bely
2022-11-25 6:42 ` [PATCH v3 3/3] testsuite: Replace bananapi with rpi-arm-v7 in signed repro tests Uladzimir Bely
2022-11-25 7:37 ` [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Jan Kiszka
3 siblings, 0 replies; 18+ messages in thread
From: Uladzimir Bely @ 2022-11-25 6:42 UTC (permalink / raw)
To: isar-users
Both signed and unsigned tests are done in non-cross mode.
Changing CROSS_COMPILE to "1" for one signed variant in order
to cover cross-build from local `base-apt` testing
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
testsuite/citest.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 2dc78015..cf9139f8 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -32,7 +32,7 @@ class ReproTest(CIBaseTest):
self.init()
try:
- self.perform_repro_test(targets, signed=True)
+ self.perform_repro_test(targets, signed=True, cross=True)
finally:
self.move_in_build_dir('tmp', 'tmp_repro_signed')
--
2.20.1
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH v3 3/3] testsuite: Replace bananapi with rpi-arm-v7 in signed repro tests
2022-11-25 6:42 [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Uladzimir Bely
2022-11-25 6:42 ` [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs Uladzimir Bely
2022-11-25 6:42 ` [PATCH v3 2/3] testsuite: Run signed repro test in cross mode Uladzimir Bely
@ 2022-11-25 6:42 ` Uladzimir Bely
2022-11-25 7:37 ` [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Jan Kiszka
3 siblings, 0 replies; 18+ messages in thread
From: Uladzimir Bely @ 2022-11-25 6:42 UTC (permalink / raw)
To: isar-users
While rpi-arm-v7 target uses own repositories and DISTRO variable,
it has more chances to fail in cross-build repro test, so it looks
like a better choice for this test than debian-only bananapi target.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
testsuite/citest.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/testsuite/citest.py b/testsuite/citest.py
index cf9139f8..d4fb177f 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -26,7 +26,7 @@ class ReproTest(CIBaseTest):
"""
def test_repro_signed(self):
targets = [
- 'mc:bananapi-bullseye:isar-image-base',
+ 'mc:rpi-arm-v7-bullseye:isar-image-base',
'mc:qemuarm64-bullseye:isar-image-base'
]
--
2.20.1
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used
2022-11-25 6:42 [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Uladzimir Bely
` (2 preceding siblings ...)
2022-11-25 6:42 ` [PATCH v3 3/3] testsuite: Replace bananapi with rpi-arm-v7 in signed repro tests Uladzimir Bely
@ 2022-11-25 7:37 ` Jan Kiszka
3 siblings, 0 replies; 18+ messages in thread
From: Jan Kiszka @ 2022-11-25 7:37 UTC (permalink / raw)
To: Uladzimir Bely, isar-users
On 25.11.22 07:42, Uladzimir Bely wrote:
> This patchset is mostly intended for downstreams that prefer
> to rename DISTRO variable (like `meat-iot2050`) and use
^^^^
Nice typo :)
> ISAR_CROSS_COMPILE = "1".
>
> This also fixes the case with cross-build or raspberry targets
> from cached base-apt repo. While it uses separate repositories
> for host (Debian) and target (Raspbian) distros, base-apt is
> splittend onto two parts corresponding to original repos.
split
>
> Also, testsuite repro tests were updated to cover crosss-build case
cross
> and raspberry target is now used in it.
>
> Changes since v2:
> - Cleaned garbage in commit message for patch 2.
>
> Changes since v1:
> - Support cached cross-build from base-apt for different base distros
> for host and target (e.g. raspberry).
>
>
> Uladzimir Bely (3):
> Populate base-apt from both DISTRO and HOST_DISTRO download dirs
> testsuite: Run signed repro test in cross mode
> testsuite: Replace bananapi with rpi-arm-v7 in signed repro tests
>
> meta-isar/conf/distro/raspbian-stretch.conf | 2 +
> meta-isar/conf/distro/raspios-bullseye.conf | 2 +
> meta-isar/conf/distro/ubuntu-focal.conf | 2 +
> meta/conf/distro/debian-common.conf | 2 +
> .../isar-bootstrap/isar-bootstrap.inc | 9 +++-
> meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++---------
> testsuite/citest.py | 4 +-
> 7 files changed, 42 insertions(+), 26 deletions(-)
>
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs
2022-11-25 6:42 ` [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs Uladzimir Bely
@ 2022-11-25 7:42 ` Jan Kiszka
2022-11-25 7:56 ` Uladzimir Bely
0 siblings, 1 reply; 18+ messages in thread
From: Jan Kiszka @ 2022-11-25 7:42 UTC (permalink / raw)
To: Uladzimir Bely, isar-users
On 25.11.22 07:42, Uladzimir Bely wrote:
> Downstreams may want to change DISTRO to some custom value.
> When cross-building, this leads to downloading packages for host
> and target distros to the different download subdirs.
>
> While base-apt is populated only from DISTRO download subdir,
> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to
> missing packages in base-apt.
>
> The similar issue happens with cross-building raspberry targets
> since they use their own mirrors and DISTRO.
>
> Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO
> variable that is used instead of BASE_DISTRO for caching builds
> for host-related components (e.g., isar-bootstrap, buildchroot-host,
> sbuild-chroot-host). Target components still use BASE_DISTRO.
>
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
> meta-isar/conf/distro/raspbian-stretch.conf | 2 +
> meta-isar/conf/distro/raspios-bullseye.conf | 2 +
> meta-isar/conf/distro/ubuntu-focal.conf | 2 +
> meta/conf/distro/debian-common.conf | 2 +
> .../isar-bootstrap/isar-bootstrap.inc | 9 +++-
> meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++---------
> 6 files changed, 40 insertions(+), 24 deletions(-)
>
> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf b/meta-isar/conf/distro/raspbian-stretch.conf
> index c8f523b3..66d965c9 100644
> --- a/meta-isar/conf/distro/raspbian-stretch.conf
> +++ b/meta-isar/conf/distro/raspbian-stretch.conf
> @@ -6,6 +6,8 @@
> BASE_DISTRO = "raspbian"
> BASE_DISTRO_CODENAME = "stretch"
>
> +BASE_HOST_DISTRO = "debian"
Better prefix it with HOST, to be consistent:
HOST_BASE_DISTRO
> +
> HOST_DISTRO ?= "debian-stretch"
> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>
> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf b/meta-isar/conf/distro/raspios-bullseye.conf
> index 5e4a09c9..7bd81dd6 100644
> --- a/meta-isar/conf/distro/raspios-bullseye.conf
> +++ b/meta-isar/conf/distro/raspios-bullseye.conf
> @@ -6,6 +6,8 @@
> BASE_DISTRO = "raspios"
> BASE_DISTRO_arm64 = "debian"
>
> +BASE_HOST_DISTRO ?= "debian"
> +
> BASE_DISTRO_CODENAME = "bullseye"
> HOST_DISTRO ?= "debian-bullseye"
> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf b/meta-isar/conf/distro/ubuntu-focal.conf
> index 379b5b30..75da79a1 100644
> --- a/meta-isar/conf/distro/ubuntu-focal.conf
> +++ b/meta-isar/conf/distro/ubuntu-focal.conf
> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
> BASE_DISTRO = "ubuntu"
> BASE_DISTRO_CODENAME = "focal"
>
> +BASE_HOST_DISTRO = "${BASE_DISTRO}"
> +
> DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list conf/distro/${HOST_DISTRO}-ports.list"
>
> diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf
> index c18eebc2..d7a13f4b 100644
> --- a/meta/conf/distro/debian-common.conf
> +++ b/meta/conf/distro/debian-common.conf
> @@ -5,6 +5,8 @@
>
> BASE_DISTRO = "debian"
>
> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
> +
> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> DISTRO_APT_SOURCES ?= "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"
>
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index f32d192e..2ee0d4cc 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg"
> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "${@https_support(d)}"
> DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
> BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}"
> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"
> FILESEXTRAPATHS_append = ":${BBPATH}"
>
> inherit deb-dl-dir
> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list):
> import re
>
> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
> + premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"
> else:
> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or ""
> mirror_list = [entry.split()
> @@ -319,11 +320,15 @@ do_bootstrap() {
> "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> - line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
> + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
> if [ -z "${BASE_REPO_KEY}" ]; then
> line="[trusted=yes] ${line}"
> fi
> echo "deb ${line}" > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
> + line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
Won't this cause duplicate lines (with complaints of apt) in case of
BOOTSTRAP_BASE_DISTRO == BASE_DISTRO?
> + if [ -z "${BASE_REPO_KEY}" ]; then
> + line="[trusted=yes] ${line}"
> + fi
> echo "deb-src ${line}" >> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
>
> mkdir -p ${ROOTFSDIR}/base-apt
> diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
> index 506a28ff..e928cd52 100644
> --- a/meta/recipes-devtools/base-apt/base-apt.bb
> +++ b/meta/recipes-devtools/base-apt/base-apt.bb
> @@ -13,7 +13,10 @@ KEYFILES ?= ""
> BASE_REPO_FEATURES ?= ""
>
> populate_base_apt() {
> - find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do
> + distro="${1}"
> + base_distro="${2}"
> +
> + find "${DEBDIR}"/"${distro}" -name '*\.deb' | while read package; do
> # NOTE: due to packages stored by reprepro are not modified, we can
> # use search by filename to check if package is already in repo. In
> # addition, md5sums are compared to ensure that the package is the
> @@ -22,25 +25,25 @@ populate_base_apt() {
>
> # Check if this package is already in base-apt
> ret=0
> - repo_contains_package "${REPO_BASE_DIR}/${BASE_DISTRO}" "${package}" ||
> + repo_contains_package "${REPO_BASE_DIR}/${base_distro}" "${package}" ||
> ret=$?
> [ "${ret}" = "0" ] && continue
> if [ "${ret}" = "1" ]; then
> - repo_del_package "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
> - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
> + repo_del_package "${REPO_BASE_DIR}"/"${base_distro}" \
> + "${REPO_BASE_DB_DIR}"/"${base_distro}" \
> "${BASE_DISTRO_CODENAME}" \
> - "${base_apt_p}"
> + "${package}"
> fi
>
> - repo_add_packages "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
> - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
> + repo_add_packages "${REPO_BASE_DIR}"/"${base_distro}" \
> + "${REPO_BASE_DB_DIR}"/"${base_distro}" \
> "${BASE_DISTRO_CODENAME}" \
> "${package}"
> done
>
> - find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read package; do
> - repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
> - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
> + find "${DEBSRCDIR}"/"${distro}" -name '*\.dsc' | while read package; do
> + repo_add_srcpackage "${REPO_BASE_DIR}"/"${base_distro}" \
> + "${REPO_BASE_DB_DIR}"/"${base_distro}" \
> "${BASE_DISTRO_CODENAME}" \
> "${package}"
> done
> @@ -55,20 +58,20 @@ repo() {
> "${BASE_DISTRO_CODENAME}" \
> "${WORKDIR}/distributions.in" \
> "${KEYFILES}"
> -
> - if [ -d '${BUILDCHROOT_HOST_DIR}/var/cache/apt' ] &&
> - [ '${DISTRO}' != '${HOST_DISTRO}' ]; then
> - # We would need two separate repository paths for that.
> - # Otherwise packages (especially the 'all' arch ones) from one
> - # distribution can influence the package versions of the other
> - # distribution.
> - bbfatal "Different host and target distributions are currently not supported." \
> - "Try it without cross-build."
> - fi
> -
> - populate_base_apt
> + populate_base_apt "${DISTRO}" "${BASE_DISTRO}"
> repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
> "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
> +
> + if [ '${DISTRO}' != '${HOST_DISTRO}' ]; then
> + repo_create "${REPO_BASE_DIR}"/"${BASE_HOST_DISTRO}" \
> + "${REPO_BASE_DB_DIR}"/"${BASE_HOST_DISTRO}" \
> + "${BASE_DISTRO_CODENAME}" \
> + "${WORKDIR}/distributions.in" \
> + "${KEYFILES}"
> + populate_base_apt "${HOST_DISTRO}" "${BASE_HOST_DISTRO}"
> + repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_HOST_DISTRO}" \
> + "${REPO_BASE_DB_DIR}"/"${BASE_HOST_DISTRO}"
> + fi
> }
>
> python do_cache() {
Looks promising!
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs
2022-11-25 7:42 ` Jan Kiszka
@ 2022-11-25 7:56 ` Uladzimir Bely
2022-11-25 14:11 ` Jan Kiszka
0 siblings, 1 reply; 18+ messages in thread
From: Uladzimir Bely @ 2022-11-25 7:56 UTC (permalink / raw)
To: isar-users, Jan Kiszka
In the email from Friday, 25 November 2022 10:42:00 +03 user Jan Kiszka wrote:
> On 25.11.22 07:42, Uladzimir Bely wrote:
> > Downstreams may want to change DISTRO to some custom value.
> > When cross-building, this leads to downloading packages for host
> > and target distros to the different download subdirs.
> >
> > While base-apt is populated only from DISTRO download subdir,
> > second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to
> > missing packages in base-apt.
> >
> > The similar issue happens with cross-building raspberry targets
> > since they use their own mirrors and DISTRO.
> >
> > Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO
> > variable that is used instead of BASE_DISTRO for caching builds
> > for host-related components (e.g., isar-bootstrap, buildchroot-host,
> > sbuild-chroot-host). Target components still use BASE_DISTRO.
> >
> > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > ---
> >
> > meta-isar/conf/distro/raspbian-stretch.conf | 2 +
> > meta-isar/conf/distro/raspios-bullseye.conf | 2 +
> > meta-isar/conf/distro/ubuntu-focal.conf | 2 +
> > meta/conf/distro/debian-common.conf | 2 +
> > .../isar-bootstrap/isar-bootstrap.inc | 9 +++-
> > meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++---------
> > 6 files changed, 40 insertions(+), 24 deletions(-)
> >
> > diff --git a/meta-isar/conf/distro/raspbian-stretch.conf
> > b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..66d965c9
> > 100644
> > --- a/meta-isar/conf/distro/raspbian-stretch.conf
> > +++ b/meta-isar/conf/distro/raspbian-stretch.conf
> > @@ -6,6 +6,8 @@
> >
> > BASE_DISTRO = "raspbian"
> > BASE_DISTRO_CODENAME = "stretch"
> >
> > +BASE_HOST_DISTRO = "debian"
>
> Better prefix it with HOST, to be consistent:
>
> HOST_BASE_DISTRO
>
Not a problem, if you think it would be better.
> > +
> >
> > HOST_DISTRO ?= "debian-stretch"
> > HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >
> > diff --git a/meta-isar/conf/distro/raspios-bullseye.conf
> > b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..7bd81dd6
> > 100644
> > --- a/meta-isar/conf/distro/raspios-bullseye.conf
> > +++ b/meta-isar/conf/distro/raspios-bullseye.conf
> > @@ -6,6 +6,8 @@
> >
> > BASE_DISTRO = "raspios"
> > BASE_DISTRO_arm64 = "debian"
> >
> > +BASE_HOST_DISTRO ?= "debian"
> > +
> >
> > BASE_DISTRO_CODENAME = "bullseye"
> > HOST_DISTRO ?= "debian-bullseye"
> > HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >
> > diff --git a/meta-isar/conf/distro/ubuntu-focal.conf
> > b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1 100644
> > --- a/meta-isar/conf/distro/ubuntu-focal.conf
> > +++ b/meta-isar/conf/distro/ubuntu-focal.conf
> > @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
> >
> > BASE_DISTRO = "ubuntu"
> > BASE_DISTRO_CODENAME = "focal"
> >
> > +BASE_HOST_DISTRO = "${BASE_DISTRO}"
> > +
> >
> > DISTRO_APT_SOURCES_arm64 ?=
> > "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
> > HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list
> > conf/distro/${HOST_DISTRO}-ports.list">
> > diff --git a/meta/conf/distro/debian-common.conf
> > b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b 100644
> > --- a/meta/conf/distro/debian-common.conf
> > +++ b/meta/conf/distro/debian-common.conf
> > @@ -5,6 +5,8 @@
> >
> > BASE_DISTRO = "debian"
> >
> > +BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
> > +
> >
> > HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> > DISTRO_APT_SOURCES ?=
> > "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list">
> > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > f32d192e..2ee0d4cc 100644
> > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg"
> >
> > DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =
> > "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if
> > d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO =
> > "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else
> > 'DISTRO')}">
> > +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if
> > d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}">
> > FILESEXTRAPATHS_append = ":${BBPATH}"
> >
> > inherit deb-dl-dir
> >
> > @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list):
> > import re
> >
> > if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> > - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
> > + premirrors = "\S*
> > file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n">
> > else:
> > premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or ""
> >
> > mirror_list = [entry.split()
> >
> > @@ -319,11 +320,15 @@ do_bootstrap() {
> >
> > "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
> >
> > mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
> > if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> >
> > - line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME}
> > main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO}
> > ${BASE_DISTRO_CODENAME} main">
> > if [ -z "${BASE_REPO_KEY}" ]; then
> >
> > line="[trusted=yes] ${line}"
> >
> > fi
> > echo "deb ${line}" >
> > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
> >
> > + line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME}
> > main"
> Won't this cause duplicate lines (with complaints of apt) in case of
> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO?
No, while I don't add new lines (no new "echo"s). I just modify "deb-src" line
while it should point to BASE_DISTRO even for host case. Without it, there was
a problem with "hello" (apt_fetch failed to download cached sources from
"host" repo, while sources were cached in "target" one.
That's how it looks now:
| $ cat isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base-
apt.list
| deb [trusted=yes] file:///base-apt/raspios bullseye main
| deb-src [trusted=yes] file:///base-apt/raspios bullseye main
| $ cat isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.list
| deb [trusted=yes] file:///base-apt/debian bullseye main
| deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>
> > + if [ -z "${BASE_REPO_KEY}" ]; then
> > + line="[trusted=yes] ${line}"
> > + fi
> >
> > echo "deb-src ${line}" >>
> > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
> >
> > mkdir -p ${ROOTFSDIR}/base-apt
> >
> > diff --git a/meta/recipes-devtools/base-apt/base-apt.bb
> > b/meta/recipes-devtools/base-apt/base-apt.bb index 506a28ff..e928cd52
> > 100644
> > --- a/meta/recipes-devtools/base-apt/base-apt.bb
> > +++ b/meta/recipes-devtools/base-apt/base-apt.bb
> > @@ -13,7 +13,10 @@ KEYFILES ?= ""
> >
> > BASE_REPO_FEATURES ?= ""
> >
> > populate_base_apt() {
> >
> > - find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do
> > + distro="${1}"
> > + base_distro="${2}"
> > +
> > + find "${DEBDIR}"/"${distro}" -name '*\.deb' | while read package; do
> >
> > # NOTE: due to packages stored by reprepro are not modified, we
> > can
> > # use search by filename to check if package is already in repo.
> > In
> > # addition, md5sums are compared to ensure that the package is
> > the
> >
> > @@ -22,25 +25,25 @@ populate_base_apt() {
> >
> > # Check if this package is already in base-apt
> > ret=0
> >
> > - repo_contains_package "${REPO_BASE_DIR}/${BASE_DISTRO}"
> > "${package}" || + repo_contains_package
> > "${REPO_BASE_DIR}/${base_distro}" "${package}" ||>
> > ret=$?
> >
> > [ "${ret}" = "0" ] && continue
> > if [ "${ret}" = "1" ]; then
> >
> > - repo_del_package "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
> > - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
> > + repo_del_package "${REPO_BASE_DIR}"/"${base_distro}" \
> > + "${REPO_BASE_DB_DIR}"/"${base_distro}" \
> >
> > "${BASE_DISTRO_CODENAME}" \
> >
> > - "${base_apt_p}"
> > + "${package}"
> >
> > fi
> >
> > - repo_add_packages "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
> > - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
> > + repo_add_packages "${REPO_BASE_DIR}"/"${base_distro}" \
> > + "${REPO_BASE_DB_DIR}"/"${base_distro}" \
> >
> > "${BASE_DISTRO_CODENAME}" \
> > "${package}"
> >
> > done
> >
> > - find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read package;
> > do - repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ -
> > "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
> > + find "${DEBSRCDIR}"/"${distro}" -name '*\.dsc' | while read package;
> > do + repo_add_srcpackage "${REPO_BASE_DIR}"/"${base_distro}" \ +
> > "${REPO_BASE_DB_DIR}"/"${base_distro}" \
> >
> > "${BASE_DISTRO_CODENAME}" \
> > "${package}"
> >
> > done
> >
> > @@ -55,20 +58,20 @@ repo() {
> >
> > "${BASE_DISTRO_CODENAME}" \
> > "${WORKDIR}/distributions.in" \
> > "${KEYFILES}"
> >
> > -
> > - if [ -d '${BUILDCHROOT_HOST_DIR}/var/cache/apt' ] &&
> > - [ '${DISTRO}' != '${HOST_DISTRO}' ]; then
> > - # We would need two separate repository paths for that.
> > - # Otherwise packages (especially the 'all' arch ones) from one
> > - # distribution can influence the package versions of the other
> > - # distribution.
> > - bbfatal "Different host and target distributions are currently
> > not supported." \ - "Try it without cross-build."
> > - fi
> > -
> > - populate_base_apt
> > + populate_base_apt "${DISTRO}" "${BASE_DISTRO}"
> >
> > repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
> >
> > "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
> >
> > +
> > + if [ '${DISTRO}' != '${HOST_DISTRO}' ]; then
> > + repo_create "${REPO_BASE_DIR}"/"${BASE_HOST_DISTRO}" \
> > + "${REPO_BASE_DB_DIR}"/"${BASE_HOST_DISTRO}" \
> > + "${BASE_DISTRO_CODENAME}" \
> > + "${WORKDIR}/distributions.in" \
> > + "${KEYFILES}"
> > + populate_base_apt "${HOST_DISTRO}" "${BASE_HOST_DISTRO}"
> > + repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_HOST_DISTRO}" \
> > + "${REPO_BASE_DB_DIR}"/"${BASE_HOST_DISTRO}"
> > + fi
> >
> > }
> >
> > python do_cache() {
>
> Looks promising!
>
> Jan
--
Uladzimir Bely
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs
2022-11-25 7:56 ` Uladzimir Bely
@ 2022-11-25 14:11 ` Jan Kiszka
2022-11-25 14:48 ` Uladzimir Bely
0 siblings, 1 reply; 18+ messages in thread
From: Jan Kiszka @ 2022-11-25 14:11 UTC (permalink / raw)
To: Uladzimir Bely, isar-users
On 25.11.22 08:56, Uladzimir Bely wrote:
> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan Kiszka wrote:
>> On 25.11.22 07:42, Uladzimir Bely wrote:
>>> Downstreams may want to change DISTRO to some custom value.
>>> When cross-building, this leads to downloading packages for host
>>> and target distros to the different download subdirs.
>>>
>>> While base-apt is populated only from DISTRO download subdir,
>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to
>>> missing packages in base-apt.
>>>
>>> The similar issue happens with cross-building raspberry targets
>>> since they use their own mirrors and DISTRO.
>>>
>>> Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO
>>> variable that is used instead of BASE_DISTRO for caching builds
>>> for host-related components (e.g., isar-bootstrap, buildchroot-host,
>>> sbuild-chroot-host). Target components still use BASE_DISTRO.
>>>
>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
>>> ---
>>>
>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 +
>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 +
>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 +
>>> meta/conf/distro/debian-common.conf | 2 +
>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++-
>>> meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++---------
>>> 6 files changed, 40 insertions(+), 24 deletions(-)
>>>
>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf
>>> b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..66d965c9
>>> 100644
>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf
>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf
>>> @@ -6,6 +6,8 @@
>>>
>>> BASE_DISTRO = "raspbian"
>>> BASE_DISTRO_CODENAME = "stretch"
>>>
>>> +BASE_HOST_DISTRO = "debian"
>>
>> Better prefix it with HOST, to be consistent:
>>
>> HOST_BASE_DISTRO
>>
>
> Not a problem, if you think it would be better.
>
>>> +
>>>
>>> HOST_DISTRO ?= "debian-stretch"
>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>
>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf
>>> b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..7bd81dd6
>>> 100644
>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf
>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf
>>> @@ -6,6 +6,8 @@
>>>
>>> BASE_DISTRO = "raspios"
>>> BASE_DISTRO_arm64 = "debian"
>>>
>>> +BASE_HOST_DISTRO ?= "debian"
>>> +
>>>
>>> BASE_DISTRO_CODENAME = "bullseye"
>>> HOST_DISTRO ?= "debian-bullseye"
>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>
>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf
>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1 100644
>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf
>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf
>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
>>>
>>> BASE_DISTRO = "ubuntu"
>>> BASE_DISTRO_CODENAME = "focal"
>>>
>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}"
>>> +
>>>
>>> DISTRO_APT_SOURCES_arm64 ?=
>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
>>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list
>>> conf/distro/${HOST_DISTRO}-ports.list">
>>> diff --git a/meta/conf/distro/debian-common.conf
>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b 100644
>>> --- a/meta/conf/distro/debian-common.conf
>>> +++ b/meta/conf/distro/debian-common.conf
>>> @@ -5,6 +5,8 @@
>>>
>>> BASE_DISTRO = "debian"
>>>
>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
>>> +
>>>
>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>> DISTRO_APT_SOURCES ?=
>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list">
>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
>>> f32d192e..2ee0d4cc 100644
>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg"
>>>
>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =
>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if
>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO =
>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else
>>> 'DISTRO')}">
>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if
>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}">
>>> FILESEXTRAPATHS_append = ":${BBPATH}"
>>>
>>> inherit deb-dl-dir
>>>
>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list):
>>> import re
>>>
>>> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>>> - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
>>> + premirrors = "\S*
>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n">
>>> else:
>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or ""
>>>
>>> mirror_list = [entry.split()
>>>
>>> @@ -319,11 +320,15 @@ do_bootstrap() {
>>>
>>> "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
>>>
>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
>>>
>>> - line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME}
>>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO}
>>> ${BASE_DISTRO_CODENAME} main">
>>> if [ -z "${BASE_REPO_KEY}" ]; then
>>>
>>> line="[trusted=yes] ${line}"
>>>
>>> fi
>>> echo "deb ${line}" >
>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
>>>
>>> + line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME}
>>> main"
>> Won't this cause duplicate lines (with complaints of apt) in case of
>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO?
>
> No, while I don't add new lines (no new "echo"s). I just modify "deb-src" line
> while it should point to BASE_DISTRO even for host case. Without it, there was
> a problem with "hello" (apt_fetch failed to download cached sources from
> "host" repo, while sources were cached in "target" one.
>
> That's how it looks now:
>
> | $ cat isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base-
> apt.list
> | deb [trusted=yes] file:///base-apt/raspios bullseye main
> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
> | $ cat isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.list
> | deb [trusted=yes] file:///base-apt/debian bullseye main
> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>
Indeed, I misintepreted the code.
But I think we still have an issue on the deb-src population side:
# ls build/downloads/deb-src/
iot2050-debian
And that was with this patch applied.
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs
2022-11-25 14:11 ` Jan Kiszka
@ 2022-11-25 14:48 ` Uladzimir Bely
2022-11-25 15:40 ` Jan Kiszka
0 siblings, 1 reply; 18+ messages in thread
From: Uladzimir Bely @ 2022-11-25 14:48 UTC (permalink / raw)
To: isar-users, Jan Kiszka
In the email from Friday, 25 November 2022 17:11:26 +03 user Jan Kiszka wrote:
> On 25.11.22 08:56, Uladzimir Bely wrote:
> > In the email from Friday, 25 November 2022 10:42:00 +03 user Jan Kiszka
wrote:
> >> On 25.11.22 07:42, Uladzimir Bely wrote:
> >>> Downstreams may want to change DISTRO to some custom value.
> >>> When cross-building, this leads to downloading packages for host
> >>> and target distros to the different download subdirs.
> >>>
> >>> While base-apt is populated only from DISTRO download subdir,
> >>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to
> >>> missing packages in base-apt.
> >>>
> >>> The similar issue happens with cross-building raspberry targets
> >>> since they use their own mirrors and DISTRO.
> >>>
> >>> Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO
> >>> variable that is used instead of BASE_DISTRO for caching builds
> >>> for host-related components (e.g., isar-bootstrap, buildchroot-host,
> >>> sbuild-chroot-host). Target components still use BASE_DISTRO.
> >>>
> >>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> >>> ---
> >>>
> >>> meta-isar/conf/distro/raspbian-stretch.conf | 2 +
> >>> meta-isar/conf/distro/raspios-bullseye.conf | 2 +
> >>> meta-isar/conf/distro/ubuntu-focal.conf | 2 +
> >>> meta/conf/distro/debian-common.conf | 2 +
> >>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++-
> >>> meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++---------
> >>> 6 files changed, 40 insertions(+), 24 deletions(-)
> >>>
> >>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf
> >>> b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..66d965c9
> >>> 100644
> >>> --- a/meta-isar/conf/distro/raspbian-stretch.conf
> >>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf
> >>> @@ -6,6 +6,8 @@
> >>>
> >>> BASE_DISTRO = "raspbian"
> >>> BASE_DISTRO_CODENAME = "stretch"
> >>>
> >>> +BASE_HOST_DISTRO = "debian"
> >>
> >> Better prefix it with HOST, to be consistent:
> >>
> >> HOST_BASE_DISTRO
> >
> > Not a problem, if you think it would be better.
> >
> >>> +
> >>>
> >>> HOST_DISTRO ?= "debian-stretch"
> >>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>>
> >>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf
> >>> b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..7bd81dd6
> >>> 100644
> >>> --- a/meta-isar/conf/distro/raspios-bullseye.conf
> >>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf
> >>> @@ -6,6 +6,8 @@
> >>>
> >>> BASE_DISTRO = "raspios"
> >>> BASE_DISTRO_arm64 = "debian"
> >>>
> >>> +BASE_HOST_DISTRO ?= "debian"
> >>> +
> >>>
> >>> BASE_DISTRO_CODENAME = "bullseye"
> >>> HOST_DISTRO ?= "debian-bullseye"
> >>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>>
> >>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf
> >>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1
> >>> 100644
> >>> --- a/meta-isar/conf/distro/ubuntu-focal.conf
> >>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf
> >>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
> >>>
> >>> BASE_DISTRO = "ubuntu"
> >>> BASE_DISTRO_CODENAME = "focal"
> >>>
> >>> +BASE_HOST_DISTRO = "${BASE_DISTRO}"
> >>> +
> >>>
> >>> DISTRO_APT_SOURCES_arm64 ?=
> >>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
> >>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list
> >>> conf/distro/${HOST_DISTRO}-ports.list">
> >>>
> >>> diff --git a/meta/conf/distro/debian-common.conf
> >>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b 100644
> >>> --- a/meta/conf/distro/debian-common.conf
> >>> +++ b/meta/conf/distro/debian-common.conf
> >>> @@ -5,6 +5,8 @@
> >>>
> >>> BASE_DISTRO = "debian"
> >>>
> >>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
> >>> +
> >>>
> >>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>> DISTRO_APT_SOURCES ?=
> >>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list">
> >>>
> >>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> >>> f32d192e..2ee0d4cc 100644
> >>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg"
> >>>
> >>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =
> >>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if
> >>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO =
> >>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1'
> >>> else
> >>> 'DISTRO')}">
> >>>
> >>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if
> >>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}">
> >>>
> >>> FILESEXTRAPATHS_append = ":${BBPATH}"
> >>>
> >>> inherit deb-dl-dir
> >>>
> >>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list):
> >>> import re
> >>>
> >>> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> >>> - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
> >>> + premirrors = "\S*
> >>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n">
> >>>
> >>> else:
> >>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or ""
> >>>
> >>> mirror_list = [entry.split()
> >>>
> >>> @@ -319,11 +320,15 @@ do_bootstrap() {
> >>>
> >>> "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
> >>>
> >>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
> >>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> >>>
> >>> - line="file:///base-apt/${BASE_DISTRO}
> >>> ${BASE_DISTRO_CODENAME}
> >>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO}
> >>> ${BASE_DISTRO_CODENAME} main">
> >>>
> >>> if [ -z "${BASE_REPO_KEY}" ]; then
> >>>
> >>> line="[trusted=yes] ${line}"
> >>>
> >>> fi
> >>> echo "deb ${line}" >
> >>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
> >>>
> >>> + line="file:///base-apt/${BASE_DISTRO}
> >>> ${BASE_DISTRO_CODENAME}
> >>> main"
> >>
> >> Won't this cause duplicate lines (with complaints of apt) in case of
> >> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO?
> >
> > No, while I don't add new lines (no new "echo"s). I just modify "deb-src"
> > line while it should point to BASE_DISTRO even for host case. Without it,
> > there was a problem with "hello" (apt_fetch failed to download cached
> > sources from "host" repo, while sources were cached in "target" one.
> >
> > That's how it looks now:
> > | $ cat isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base-
> >
> > apt.list
> >
> > | deb [trusted=yes] file:///base-apt/raspios bullseye main
> > | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
> > | $ cat
> > | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.list
> > | deb [trusted=yes] file:///base-apt/debian bullseye main
> > | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>
> Indeed, I misintepreted the code.
>
> But I think we still have an issue on the deb-src population side:
>
> # ls build/downloads/deb-src/
> iot2050-debian
>
> And that was with this patch applied.
>
> Jan
Actually, I checked the case with the following configuration:
distro: my-debian
local_conf_header:
nodebian: |
require conf/distro/debian-bullseye.conf
DISTRO_NAME = "My Debian System"
So, I simulated renaming distro that is done in meta-iot2050.
Cached build was OK, "hello" package (that we build from sources) was
downloaded at first build to "my-debian" directory:
| $ ls build/downloads/deb-src/my-debian/hello/
| hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz
and second build picked the files with no problem.
The same with "raspios-bullseye" target - "hello" sources are downloaded to
the "target" dir:
| $ ls build/downloads/deb-src/raspios-bullseye/hello/
| hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz
I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" and it was
OK (honestly, I don't know who could require cached build with this option
enabled :) ).
Anyway, I'll also try to check my changes with meta-iot2050 layer also
(despite some packages in it are currently not compatible with BB_NO_NETWORK
option)
-
Uladzimir Bely
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs
2022-11-25 14:48 ` Uladzimir Bely
@ 2022-11-25 15:40 ` Jan Kiszka
2022-11-26 5:30 ` Uladzimir Bely
0 siblings, 1 reply; 18+ messages in thread
From: Jan Kiszka @ 2022-11-25 15:40 UTC (permalink / raw)
To: Uladzimir Bely, isar-users
On 25.11.22 15:48, Uladzimir Bely wrote:
> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan Kiszka wrote:
>> On 25.11.22 08:56, Uladzimir Bely wrote:
>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan Kiszka
> wrote:
>>>> On 25.11.22 07:42, Uladzimir Bely wrote:
>>>>> Downstreams may want to change DISTRO to some custom value.
>>>>> When cross-building, this leads to downloading packages for host
>>>>> and target distros to the different download subdirs.
>>>>>
>>>>> While base-apt is populated only from DISTRO download subdir,
>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to
>>>>> missing packages in base-apt.
>>>>>
>>>>> The similar issue happens with cross-building raspberry targets
>>>>> since they use their own mirrors and DISTRO.
>>>>>
>>>>> Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO
>>>>> variable that is used instead of BASE_DISTRO for caching builds
>>>>> for host-related components (e.g., isar-bootstrap, buildchroot-host,
>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO.
>>>>>
>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
>>>>> ---
>>>>>
>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 +
>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 +
>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 +
>>>>> meta/conf/distro/debian-common.conf | 2 +
>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++-
>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47 ++++++++++---------
>>>>> 6 files changed, 40 insertions(+), 24 deletions(-)
>>>>>
>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf
>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..66d965c9
>>>>> 100644
>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf
>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf
>>>>> @@ -6,6 +6,8 @@
>>>>>
>>>>> BASE_DISTRO = "raspbian"
>>>>> BASE_DISTRO_CODENAME = "stretch"
>>>>>
>>>>> +BASE_HOST_DISTRO = "debian"
>>>>
>>>> Better prefix it with HOST, to be consistent:
>>>>
>>>> HOST_BASE_DISTRO
>>>
>>> Not a problem, if you think it would be better.
>>>
>>>>> +
>>>>>
>>>>> HOST_DISTRO ?= "debian-stretch"
>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>
>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf
>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..7bd81dd6
>>>>> 100644
>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf
>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf
>>>>> @@ -6,6 +6,8 @@
>>>>>
>>>>> BASE_DISTRO = "raspios"
>>>>> BASE_DISTRO_arm64 = "debian"
>>>>>
>>>>> +BASE_HOST_DISTRO ?= "debian"
>>>>> +
>>>>>
>>>>> BASE_DISTRO_CODENAME = "bullseye"
>>>>> HOST_DISTRO ?= "debian-bullseye"
>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>
>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf
>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1
>>>>> 100644
>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf
>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf
>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
>>>>>
>>>>> BASE_DISTRO = "ubuntu"
>>>>> BASE_DISTRO_CODENAME = "focal"
>>>>>
>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}"
>>>>> +
>>>>>
>>>>> DISTRO_APT_SOURCES_arm64 ?=
>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list
>>>>> conf/distro/${HOST_DISTRO}-ports.list">
>>>>>
>>>>> diff --git a/meta/conf/distro/debian-common.conf
>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b 100644
>>>>> --- a/meta/conf/distro/debian-common.conf
>>>>> +++ b/meta/conf/distro/debian-common.conf
>>>>> @@ -5,6 +5,8 @@
>>>>>
>>>>> BASE_DISTRO = "debian"
>>>>>
>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
>>>>> +
>>>>>
>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>> DISTRO_APT_SOURCES ?=
>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list">
>>>>>
>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
>>>>> f32d192e..2ee0d4cc 100644
>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg"
>>>>>
>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =
>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if
>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO =
>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1'
>>>>> else
>>>>> 'DISTRO')}">
>>>>>
>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if
>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}">
>>>>>
>>>>> FILESEXTRAPATHS_append = ":${BBPATH}"
>>>>>
>>>>> inherit deb-dl-dir
>>>>>
>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list):
>>>>> import re
>>>>>
>>>>> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>>>>> - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
>>>>> + premirrors = "\S*
>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n">
>>>>>
>>>>> else:
>>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or ""
>>>>>
>>>>> mirror_list = [entry.split()
>>>>>
>>>>> @@ -319,11 +320,15 @@ do_bootstrap() {
>>>>>
>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
>>>>>
>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
>>>>>
>>>>> - line="file:///base-apt/${BASE_DISTRO}
>>>>> ${BASE_DISTRO_CODENAME}
>>>>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO}
>>>>> ${BASE_DISTRO_CODENAME} main">
>>>>>
>>>>> if [ -z "${BASE_REPO_KEY}" ]; then
>>>>>
>>>>> line="[trusted=yes] ${line}"
>>>>>
>>>>> fi
>>>>> echo "deb ${line}" >
>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
>>>>>
>>>>> + line="file:///base-apt/${BASE_DISTRO}
>>>>> ${BASE_DISTRO_CODENAME}
>>>>> main"
>>>>
>>>> Won't this cause duplicate lines (with complaints of apt) in case of
>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO?
>>>
>>> No, while I don't add new lines (no new "echo"s). I just modify "deb-src"
>>> line while it should point to BASE_DISTRO even for host case. Without it,
>>> there was a problem with "hello" (apt_fetch failed to download cached
>>> sources from "host" repo, while sources were cached in "target" one.
>>>
>>> That's how it looks now:
>>> | $ cat isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base-
>>>
>>> apt.list
>>>
>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main
>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>>> | $ cat
>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.list
>>> | deb [trusted=yes] file:///base-apt/debian bullseye main
>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>>
>> Indeed, I misintepreted the code.
>>
>> But I think we still have an issue on the deb-src population side:
>>
>> # ls build/downloads/deb-src/
>> iot2050-debian
>>
>> And that was with this patch applied.
>>
>> Jan
>
> Actually, I checked the case with the following configuration:
>
> distro: my-debian
> local_conf_header:
> nodebian: |
> require conf/distro/debian-bullseye.conf
> DISTRO_NAME = "My Debian System"
>
> So, I simulated renaming distro that is done in meta-iot2050.
>
> Cached build was OK, "hello" package (that we build from sources) was
> downloaded at first build to "my-debian" directory:
>
> | $ ls build/downloads/deb-src/my-debian/hello/
> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz
>
> and second build picked the files with no problem.
>
> The same with "raspios-bullseye" target - "hello" sources are downloaded to
> the "target" dir:
>
> | $ ls build/downloads/deb-src/raspios-bullseye/hello/
> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz
>
> I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" and it was
> OK (honestly, I don't know who could require cached build with this option
> enabled :) ).
>
> Anyway, I'll also try to check my changes with meta-iot2050 layer also
> (despite some packages in it are currently not compatible with BB_NO_NETWORK
> option)
Please then test without the coral packages, they have an own issue /wrt
offline builds.
Thanks,
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs
2022-11-25 15:40 ` Jan Kiszka
@ 2022-11-26 5:30 ` Uladzimir Bely
2022-11-27 19:05 ` Jan Kiszka
0 siblings, 1 reply; 18+ messages in thread
From: Uladzimir Bely @ 2022-11-26 5:30 UTC (permalink / raw)
To: isar-users, Jan Kiszka
In the email from Friday, 25 November 2022 18:40:39 +03 user Jan Kiszka wrote:
> On 25.11.22 15:48, Uladzimir Bely wrote:
> > In the email from Friday, 25 November 2022 17:11:26 +03 user Jan Kiszka
wrote:
> >> On 25.11.22 08:56, Uladzimir Bely wrote:
> >>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan Kiszka
> >
> > wrote:
> >>>> On 25.11.22 07:42, Uladzimir Bely wrote:
> >>>>> Downstreams may want to change DISTRO to some custom value.
> >>>>> When cross-building, this leads to downloading packages for host
> >>>>> and target distros to the different download subdirs.
> >>>>>
> >>>>> While base-apt is populated only from DISTRO download subdir,
> >>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to
> >>>>> missing packages in base-apt.
> >>>>>
> >>>>> The similar issue happens with cross-building raspberry targets
> >>>>> since they use their own mirrors and DISTRO.
> >>>>>
> >>>>> Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO
> >>>>> variable that is used instead of BASE_DISTRO for caching builds
> >>>>> for host-related components (e.g., isar-bootstrap, buildchroot-host,
> >>>>> sbuild-chroot-host). Target components still use BASE_DISTRO.
> >>>>>
> >>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> >>>>> ---
> >>>>>
> >>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 +
> >>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 +
> >>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 +
> >>>>> meta/conf/distro/debian-common.conf | 2 +
> >>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++-
> >>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47
> >>>>> ++++++++++---------
> >>>>> 6 files changed, 40 insertions(+), 24 deletions(-)
> >>>>>
> >>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf
> >>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..66d965c9
> >>>>> 100644
> >>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf
> >>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf
> >>>>> @@ -6,6 +6,8 @@
> >>>>>
> >>>>> BASE_DISTRO = "raspbian"
> >>>>> BASE_DISTRO_CODENAME = "stretch"
> >>>>>
> >>>>> +BASE_HOST_DISTRO = "debian"
> >>>>
> >>>> Better prefix it with HOST, to be consistent:
> >>>>
> >>>> HOST_BASE_DISTRO
> >>>
> >>> Not a problem, if you think it would be better.
> >>>
> >>>>> +
> >>>>>
> >>>>> HOST_DISTRO ?= "debian-stretch"
> >>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>>>>
> >>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf
> >>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..7bd81dd6
> >>>>> 100644
> >>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf
> >>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf
> >>>>> @@ -6,6 +6,8 @@
> >>>>>
> >>>>> BASE_DISTRO = "raspios"
> >>>>> BASE_DISTRO_arm64 = "debian"
> >>>>>
> >>>>> +BASE_HOST_DISTRO ?= "debian"
> >>>>> +
> >>>>>
> >>>>> BASE_DISTRO_CODENAME = "bullseye"
> >>>>> HOST_DISTRO ?= "debian-bullseye"
> >>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>>>>
> >>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf
> >>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1
> >>>>> 100644
> >>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf
> >>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf
> >>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
> >>>>>
> >>>>> BASE_DISTRO = "ubuntu"
> >>>>> BASE_DISTRO_CODENAME = "focal"
> >>>>>
> >>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}"
> >>>>> +
> >>>>>
> >>>>> DISTRO_APT_SOURCES_arm64 ?=
> >>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
> >>>>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list
> >>>>> conf/distro/${HOST_DISTRO}-ports.list">
> >>>>>
> >>>>> diff --git a/meta/conf/distro/debian-common.conf
> >>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b 100644
> >>>>> --- a/meta/conf/distro/debian-common.conf
> >>>>> +++ b/meta/conf/distro/debian-common.conf
> >>>>> @@ -5,6 +5,8 @@
> >>>>>
> >>>>> BASE_DISTRO = "debian"
> >>>>>
> >>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
> >>>>> +
> >>>>>
> >>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>>>> DISTRO_APT_SOURCES ?=
> >>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list">
> >>>>>
> >>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> >>>>> f32d192e..2ee0d4cc 100644
> >>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg =
> >>>>> ",gnupg"
> >>>>>
> >>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =
> >>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if
> >>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO =
> >>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1'
> >>>>> else
> >>>>> 'DISTRO')}">
> >>>>>
> >>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if
> >>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}">
> >>>>>
> >>>>> FILESEXTRAPATHS_append = ":${BBPATH}"
> >>>>>
> >>>>> inherit deb-dl-dir
> >>>>>
> >>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d,
aptsources_entry_list):
> >>>>> import re
> >>>>>
> >>>>> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> >>>>> - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
> >>>>> + premirrors = "\S*
> >>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n">
> >>>>>
> >>>>> else:
> >>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or ""
> >>>>>
> >>>>> mirror_list = [entry.split()
> >>>>>
> >>>>> @@ -319,11 +320,15 @@ do_bootstrap() {
> >>>>>
> >>>>> "${ROOTFSDIR}/etc/apt/preferences.d/bootstra
> >>>>> p"
> >>>>>
> >>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
> >>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> >>>>>
> >>>>> - line="file:///base-apt/${BASE_DISTRO}
> >>>>> ${BASE_DISTRO_CODENAME}
> >>>>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO}
> >>>>> ${BASE_DISTRO_CODENAME} main">
> >>>>>
> >>>>> if [ -z "${BASE_REPO_KEY}" ]; then
> >>>>>
> >>>>> line="[trusted=yes] ${line}"
> >>>>>
> >>>>> fi
> >>>>> echo "deb ${line}" >
> >>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
> >>>>>
> >>>>> + line="file:///base-apt/${BASE_DISTRO}
> >>>>> ${BASE_DISTRO_CODENAME}
> >>>>> main"
> >>>>
> >>>> Won't this cause duplicate lines (with complaints of apt) in case of
> >>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO?
> >>>
> >>> No, while I don't add new lines (no new "echo"s). I just modify
> >>> "deb-src"
> >>> line while it should point to BASE_DISTRO even for host case. Without
> >>> it,
> >>> there was a problem with "hello" (apt_fetch failed to download cached
> >>> sources from "host" repo, while sources were cached in "target" one.
> >>>
> >>> That's how it looks now:
> >>> | $ cat isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base-
> >>>
> >>> apt.list
> >>>
> >>> | deb [trusted=yes] file:///base-apt/raspios bullseye main
> >>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
> >>> | $ cat
> >>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.list
> >>> | deb [trusted=yes] file:///base-apt/debian bullseye main
> >>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
> >>
> >> Indeed, I misintepreted the code.
> >>
> >> But I think we still have an issue on the deb-src population side:
> >>
> >> # ls build/downloads/deb-src/
> >> iot2050-debian
> >>
> >> And that was with this patch applied.
> >>
> >> Jan
> >
> > Actually, I checked the case with the following configuration:
> >
> > distro: my-debian
> >
> > local_conf_header:
> > nodebian: |
> >
> > require conf/distro/debian-bullseye.conf
> > DISTRO_NAME = "My Debian System"
> >
> > So, I simulated renaming distro that is done in meta-iot2050.
> >
> > Cached build was OK, "hello" package (that we build from sources) was
> >
> > downloaded at first build to "my-debian" directory:
> > | $ ls build/downloads/deb-src/my-debian/hello/
> > | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz
> >
> > and second build picked the files with no problem.
> >
> > The same with "raspios-bullseye" target - "hello" sources are downloaded
> > to
> >
> > the "target" dir:
> > | $ ls build/downloads/deb-src/raspios-bullseye/hello/
> > | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz
> >
> > I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" and it
> > was OK (honestly, I don't know who could require cached build with this
> > option enabled :) ).
> >
> > Anyway, I'll also try to check my changes with meta-iot2050 layer also
> > (despite some packages in it are currently not compatible with
> > BB_NO_NETWORK option)
>
> Please then test without the coral packages, they have an own issue /wrt
> offline builds.
Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works
>
> Thanks,
> Jan
--
Uladzimir Bely
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs
2022-11-26 5:30 ` Uladzimir Bely
@ 2022-11-27 19:05 ` Jan Kiszka
2022-11-28 6:05 ` Uladzimir Bely
0 siblings, 1 reply; 18+ messages in thread
From: Jan Kiszka @ 2022-11-27 19:05 UTC (permalink / raw)
To: Uladzimir Bely, isar-users
On 26.11.22 06:30, Uladzimir Bely wrote:
> In the email from Friday, 25 November 2022 18:40:39 +03 user Jan Kiszka wrote:
>> On 25.11.22 15:48, Uladzimir Bely wrote:
>>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan Kiszka
> wrote:
>>>> On 25.11.22 08:56, Uladzimir Bely wrote:
>>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan Kiszka
>>>
>>> wrote:
>>>>>> On 25.11.22 07:42, Uladzimir Bely wrote:
>>>>>>> Downstreams may want to change DISTRO to some custom value.
>>>>>>> When cross-building, this leads to downloading packages for host
>>>>>>> and target distros to the different download subdirs.
>>>>>>>
>>>>>>> While base-apt is populated only from DISTRO download subdir,
>>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to
>>>>>>> missing packages in base-apt.
>>>>>>>
>>>>>>> The similar issue happens with cross-building raspberry targets
>>>>>>> since they use their own mirrors and DISTRO.
>>>>>>>
>>>>>>> Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO
>>>>>>> variable that is used instead of BASE_DISTRO for caching builds
>>>>>>> for host-related components (e.g., isar-bootstrap, buildchroot-host,
>>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO.
>>>>>>>
>>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
>>>>>>> ---
>>>>>>>
>>>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 +
>>>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 +
>>>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 +
>>>>>>> meta/conf/distro/debian-common.conf | 2 +
>>>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++-
>>>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47
>>>>>>> ++++++++++---------
>>>>>>> 6 files changed, 40 insertions(+), 24 deletions(-)
>>>>>>>
>>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..66d965c9
>>>>>>> 100644
>>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>> @@ -6,6 +6,8 @@
>>>>>>>
>>>>>>> BASE_DISTRO = "raspbian"
>>>>>>> BASE_DISTRO_CODENAME = "stretch"
>>>>>>>
>>>>>>> +BASE_HOST_DISTRO = "debian"
>>>>>>
>>>>>> Better prefix it with HOST, to be consistent:
>>>>>>
>>>>>> HOST_BASE_DISTRO
>>>>>
>>>>> Not a problem, if you think it would be better.
>>>>>
>>>>>>> +
>>>>>>>
>>>>>>> HOST_DISTRO ?= "debian-stretch"
>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>
>>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..7bd81dd6
>>>>>>> 100644
>>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>> @@ -6,6 +6,8 @@
>>>>>>>
>>>>>>> BASE_DISTRO = "raspios"
>>>>>>> BASE_DISTRO_arm64 = "debian"
>>>>>>>
>>>>>>> +BASE_HOST_DISTRO ?= "debian"
>>>>>>> +
>>>>>>>
>>>>>>> BASE_DISTRO_CODENAME = "bullseye"
>>>>>>> HOST_DISTRO ?= "debian-bullseye"
>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>
>>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1
>>>>>>> 100644
>>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
>>>>>>>
>>>>>>> BASE_DISTRO = "ubuntu"
>>>>>>> BASE_DISTRO_CODENAME = "focal"
>>>>>>>
>>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}"
>>>>>>> +
>>>>>>>
>>>>>>> DISTRO_APT_SOURCES_arm64 ?=
>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
>>>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list
>>>>>>> conf/distro/${HOST_DISTRO}-ports.list">
>>>>>>>
>>>>>>> diff --git a/meta/conf/distro/debian-common.conf
>>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b 100644
>>>>>>> --- a/meta/conf/distro/debian-common.conf
>>>>>>> +++ b/meta/conf/distro/debian-common.conf
>>>>>>> @@ -5,6 +5,8 @@
>>>>>>>
>>>>>>> BASE_DISTRO = "debian"
>>>>>>>
>>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
>>>>>>> +
>>>>>>>
>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>> DISTRO_APT_SOURCES ?=
>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list">
>>>>>>>
>>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
>>>>>>> f32d192e..2ee0d4cc 100644
>>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg =
>>>>>>> ",gnupg"
>>>>>>>
>>>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =
>>>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if
>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO =
>>>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1'
>>>>>>> else
>>>>>>> 'DISTRO')}">
>>>>>>>
>>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if
>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}">
>>>>>>>
>>>>>>> FILESEXTRAPATHS_append = ":${BBPATH}"
>>>>>>>
>>>>>>> inherit deb-dl-dir
>>>>>>>
>>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d,
> aptsources_entry_list):
>>>>>>> import re
>>>>>>>
>>>>>>> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>>>>>>> - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
>>>>>>> + premirrors = "\S*
>>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n">
>>>>>>>
>>>>>>> else:
>>>>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or ""
>>>>>>>
>>>>>>> mirror_list = [entry.split()
>>>>>>>
>>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() {
>>>>>>>
>>>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/bootstra
>>>>>>> p"
>>>>>>>
>>>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
>>>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
>>>>>>>
>>>>>>> - line="file:///base-apt/${BASE_DISTRO}
>>>>>>> ${BASE_DISTRO_CODENAME}
>>>>>>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO}
>>>>>>> ${BASE_DISTRO_CODENAME} main">
>>>>>>>
>>>>>>> if [ -z "${BASE_REPO_KEY}" ]; then
>>>>>>>
>>>>>>> line="[trusted=yes] ${line}"
>>>>>>>
>>>>>>> fi
>>>>>>> echo "deb ${line}" >
>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
>>>>>>>
>>>>>>> + line="file:///base-apt/${BASE_DISTRO}
>>>>>>> ${BASE_DISTRO_CODENAME}
>>>>>>> main"
>>>>>>
>>>>>> Won't this cause duplicate lines (with complaints of apt) in case of
>>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO?
>>>>>
>>>>> No, while I don't add new lines (no new "echo"s). I just modify
>>>>> "deb-src"
>>>>> line while it should point to BASE_DISTRO even for host case. Without
>>>>> it,
>>>>> there was a problem with "hello" (apt_fetch failed to download cached
>>>>> sources from "host" repo, while sources were cached in "target" one.
>>>>>
>>>>> That's how it looks now:
>>>>> | $ cat isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base-
>>>>>
>>>>> apt.list
>>>>>
>>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main
>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>>>>> | $ cat
>>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.list
>>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main
>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>>>>
>>>> Indeed, I misintepreted the code.
>>>>
>>>> But I think we still have an issue on the deb-src population side:
>>>>
>>>> # ls build/downloads/deb-src/
>>>> iot2050-debian
>>>>
>>>> And that was with this patch applied.
>>>>
>>>> Jan
>>>
>>> Actually, I checked the case with the following configuration:
>>>
>>> distro: my-debian
>>>
>>> local_conf_header:
>>> nodebian: |
>>>
>>> require conf/distro/debian-bullseye.conf
>>> DISTRO_NAME = "My Debian System"
>>>
>>> So, I simulated renaming distro that is done in meta-iot2050.
>>>
>>> Cached build was OK, "hello" package (that we build from sources) was
>>>
>>> downloaded at first build to "my-debian" directory:
>>> | $ ls build/downloads/deb-src/my-debian/hello/
>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz
>>>
>>> and second build picked the files with no problem.
>>>
>>> The same with "raspios-bullseye" target - "hello" sources are downloaded
>>> to
>>>
>>> the "target" dir:
>>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/
>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz
>>>
>>> I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" and it
>>> was OK (honestly, I don't know who could require cached build with this
>>> option enabled :) ).
>>>
>>> Anyway, I'll also try to check my changes with meta-iot2050 layer also
>>> (despite some packages in it are currently not compatible with
>>> BB_NO_NETWORK option)
>>
>> Please then test without the coral packages, they have an own issue /wrt
>> offline builds.
>
> Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works
>
...but it caches to the wrong downloads/deb-src path, no?
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs
2022-11-27 19:05 ` Jan Kiszka
@ 2022-11-28 6:05 ` Uladzimir Bely
2022-11-28 6:32 ` Jan Kiszka
0 siblings, 1 reply; 18+ messages in thread
From: Uladzimir Bely @ 2022-11-28 6:05 UTC (permalink / raw)
To: isar-users, Jan Kiszka
In the email from Sunday, 27 November 2022 22:05:53 +03 user Jan Kiszka wrote:
> On 26.11.22 06:30, Uladzimir Bely wrote:
> > In the email from Friday, 25 November 2022 18:40:39 +03 user Jan Kiszka
wrote:
> >> On 25.11.22 15:48, Uladzimir Bely wrote:
> >>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan Kiszka
> >
> > wrote:
> >>>> On 25.11.22 08:56, Uladzimir Bely wrote:
> >>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan
> >>>>> Kiszka
> >>>
> >>> wrote:
> >>>>>> On 25.11.22 07:42, Uladzimir Bely wrote:
> >>>>>>> Downstreams may want to change DISTRO to some custom value.
> >>>>>>> When cross-building, this leads to downloading packages for host
> >>>>>>> and target distros to the different download subdirs.
> >>>>>>>
> >>>>>>> While base-apt is populated only from DISTRO download subdir,
> >>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to
> >>>>>>> missing packages in base-apt.
> >>>>>>>
> >>>>>>> The similar issue happens with cross-building raspberry targets
> >>>>>>> since they use their own mirrors and DISTRO.
> >>>>>>>
> >>>>>>> Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO
> >>>>>>> variable that is used instead of BASE_DISTRO for caching builds
> >>>>>>> for host-related components (e.g., isar-bootstrap, buildchroot-host,
> >>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO.
> >>>>>>>
> >>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> >>>>>>> ---
> >>>>>>>
> >>>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 +
> >>>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 +
> >>>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 +
> >>>>>>> meta/conf/distro/debian-common.conf | 2 +
> >>>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++-
> >>>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47
> >>>>>>> ++++++++++---------
> >>>>>>> 6 files changed, 40 insertions(+), 24 deletions(-)
> >>>>>>>
> >>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf
> >>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index
> >>>>>>> c8f523b3..66d965c9
> >>>>>>> 100644
> >>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf
> >>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf
> >>>>>>> @@ -6,6 +6,8 @@
> >>>>>>>
> >>>>>>> BASE_DISTRO = "raspbian"
> >>>>>>> BASE_DISTRO_CODENAME = "stretch"
> >>>>>>>
> >>>>>>> +BASE_HOST_DISTRO = "debian"
> >>>>>>
> >>>>>> Better prefix it with HOST, to be consistent:
> >>>>>>
> >>>>>> HOST_BASE_DISTRO
> >>>>>
> >>>>> Not a problem, if you think it would be better.
> >>>>>
> >>>>>>> +
> >>>>>>>
> >>>>>>> HOST_DISTRO ?= "debian-stretch"
> >>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>>>>>>
> >>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf
> >>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index
> >>>>>>> 5e4a09c9..7bd81dd6
> >>>>>>> 100644
> >>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf
> >>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf
> >>>>>>> @@ -6,6 +6,8 @@
> >>>>>>>
> >>>>>>> BASE_DISTRO = "raspios"
> >>>>>>> BASE_DISTRO_arm64 = "debian"
> >>>>>>>
> >>>>>>> +BASE_HOST_DISTRO ?= "debian"
> >>>>>>> +
> >>>>>>>
> >>>>>>> BASE_DISTRO_CODENAME = "bullseye"
> >>>>>>> HOST_DISTRO ?= "debian-bullseye"
> >>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>>>>>>
> >>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf
> >>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1
> >>>>>>> 100644
> >>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf
> >>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf
> >>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
> >>>>>>>
> >>>>>>> BASE_DISTRO = "ubuntu"
> >>>>>>> BASE_DISTRO_CODENAME = "focal"
> >>>>>>>
> >>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}"
> >>>>>>> +
> >>>>>>>
> >>>>>>> DISTRO_APT_SOURCES_arm64 ?=
> >>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
> >>>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list
> >>>>>>> conf/distro/${HOST_DISTRO}-ports.list">
> >>>>>>>
> >>>>>>> diff --git a/meta/conf/distro/debian-common.conf
> >>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b
> >>>>>>> 100644
> >>>>>>> --- a/meta/conf/distro/debian-common.conf
> >>>>>>> +++ b/meta/conf/distro/debian-common.conf
> >>>>>>> @@ -5,6 +5,8 @@
> >>>>>>>
> >>>>>>> BASE_DISTRO = "debian"
> >>>>>>>
> >>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
> >>>>>>> +
> >>>>>>>
> >>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>>>>>> DISTRO_APT_SOURCES ?=
> >>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list">
> >>>>>>>
> >>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> >>>>>>> f32d192e..2ee0d4cc 100644
> >>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg =
> >>>>>>> ",gnupg"
> >>>>>>>
> >>>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =
> >>>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if
> >>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO =
> >>>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1'
> >>>>>>> else
> >>>>>>> 'DISTRO')}">
> >>>>>>>
> >>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if
> >>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}">
> >>>>>>>
> >>>>>>> FILESEXTRAPATHS_append = ":${BBPATH}"
> >>>>>>>
> >>>>>>> inherit deb-dl-dir
> >>>>>>>
> >>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d,
> >
> > aptsources_entry_list):
> >>>>>>> import re
> >>>>>>>
> >>>>>>> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> >>>>>>> - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
> >>>>>>> + premirrors = "\S*
> >>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n">
> >>>>>>>
> >>>>>>> else:
> >>>>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or ""
> >>>>>>>
> >>>>>>> mirror_list = [entry.split()
> >>>>>>>
> >>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() {
> >>>>>>>
> >>>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/bootst
> >>>>>>> ra
> >>>>>>> p"
> >>>>>>>
> >>>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
> >>>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> >>>>>>>
> >>>>>>> - line="file:///base-apt/${BASE_DISTRO}
> >>>>>>> ${BASE_DISTRO_CODENAME}
> >>>>>>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO}
> >>>>>>> ${BASE_DISTRO_CODENAME} main">
> >>>>>>>
> >>>>>>> if [ -z "${BASE_REPO_KEY}" ]; then
> >>>>>>>
> >>>>>>> line="[trusted=yes] ${line}"
> >>>>>>>
> >>>>>>> fi
> >>>>>>> echo "deb ${line}" >
> >>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
> >>>>>>>
> >>>>>>> + line="file:///base-apt/${BASE_DISTRO}
> >>>>>>> ${BASE_DISTRO_CODENAME}
> >>>>>>> main"
> >>>>>>
> >>>>>> Won't this cause duplicate lines (with complaints of apt) in case of
> >>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO?
> >>>>>
> >>>>> No, while I don't add new lines (no new "echo"s). I just modify
> >>>>> "deb-src"
> >>>>> line while it should point to BASE_DISTRO even for host case. Without
> >>>>> it,
> >>>>> there was a problem with "hello" (apt_fetch failed to download cached
> >>>>> sources from "host" repo, while sources were cached in "target" one.
> >>>>>
> >>>>> That's how it looks now:
> >>>>> | $ cat
> >>>>> | isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base-
> >>>>>
> >>>>> apt.list
> >>>>>
> >>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main
> >>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
> >>>>> | $ cat
> >>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.li
> >>>>> | st
> >>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main
> >>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
> >>>>
> >>>> Indeed, I misintepreted the code.
> >>>>
> >>>> But I think we still have an issue on the deb-src population side:
> >>>>
> >>>> # ls build/downloads/deb-src/
> >>>> iot2050-debian
> >>>>
> >>>> And that was with this patch applied.
> >>>>
> >>>> Jan
> >>>
> >>> Actually, I checked the case with the following configuration:
> >>>
> >>> distro: my-debian
> >>>
> >>> local_conf_header:
> >>> nodebian: |
> >>>
> >>> require conf/distro/debian-bullseye.conf
> >>> DISTRO_NAME = "My Debian System"
> >>>
> >>> So, I simulated renaming distro that is done in meta-iot2050.
> >>>
> >>> Cached build was OK, "hello" package (that we build from sources) was
> >>>
> >>> downloaded at first build to "my-debian" directory:
> >>> | $ ls build/downloads/deb-src/my-debian/hello/
> >>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz
> >>>
> >>> and second build picked the files with no problem.
> >>>
> >>> The same with "raspios-bullseye" target - "hello" sources are downloaded
> >>> to
> >>>
> >>> the "target" dir:
> >>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/
> >>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz
> >>>
> >>> I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" and
> >>> it
> >>> was OK (honestly, I don't know who could require cached build with this
> >>> option enabled :) ).
> >>>
> >>> Anyway, I'll also try to check my changes with meta-iot2050 layer also
> >>> (despite some packages in it are currently not compatible with
> >>> BB_NO_NETWORK option)
> >>
> >> Please then test without the coral packages, they have an own issue /wrt
> >> offline builds.
> >
> > Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works
>
> ...but it caches to the wrong downloads/deb-src path, no?
>
> Jan
Why wrong?
| meta-iot2050/build $ tree -d downloads/deb*
| downloads/deb
| ├── debian-bullseye
| └── iot2050-debian
| downloads/deb-src
| └── iot2050-debian
| ├── openssl
| └── swig
`do_apt_fetch` works with `${DISTRO}` and downloads sources to the
corresponding directory.
--
Uladzimir Bely
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs
2022-11-28 6:05 ` Uladzimir Bely
@ 2022-11-28 6:32 ` Jan Kiszka
2022-11-28 6:41 ` Uladzimir Bely
0 siblings, 1 reply; 18+ messages in thread
From: Jan Kiszka @ 2022-11-28 6:32 UTC (permalink / raw)
To: Uladzimir Bely, isar-users
On 28.11.22 07:05, Uladzimir Bely wrote:
> In the email from Sunday, 27 November 2022 22:05:53 +03 user Jan Kiszka wrote:
>> On 26.11.22 06:30, Uladzimir Bely wrote:
>>> In the email from Friday, 25 November 2022 18:40:39 +03 user Jan Kiszka
> wrote:
>>>> On 25.11.22 15:48, Uladzimir Bely wrote:
>>>>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan Kiszka
>>>
>>> wrote:
>>>>>> On 25.11.22 08:56, Uladzimir Bely wrote:
>>>>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan
>>>>>>> Kiszka
>>>>>
>>>>> wrote:
>>>>>>>> On 25.11.22 07:42, Uladzimir Bely wrote:
>>>>>>>>> Downstreams may want to change DISTRO to some custom value.
>>>>>>>>> When cross-building, this leads to downloading packages for host
>>>>>>>>> and target distros to the different download subdirs.
>>>>>>>>>
>>>>>>>>> While base-apt is populated only from DISTRO download subdir,
>>>>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to
>>>>>>>>> missing packages in base-apt.
>>>>>>>>>
>>>>>>>>> The similar issue happens with cross-building raspberry targets
>>>>>>>>> since they use their own mirrors and DISTRO.
>>>>>>>>>
>>>>>>>>> Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO
>>>>>>>>> variable that is used instead of BASE_DISTRO for caching builds
>>>>>>>>> for host-related components (e.g., isar-bootstrap, buildchroot-host,
>>>>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
>>>>>>>>> ---
>>>>>>>>>
>>>>>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 +
>>>>>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 +
>>>>>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 +
>>>>>>>>> meta/conf/distro/debian-common.conf | 2 +
>>>>>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++-
>>>>>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47
>>>>>>>>> ++++++++++---------
>>>>>>>>> 6 files changed, 40 insertions(+), 24 deletions(-)
>>>>>>>>>
>>>>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index
>>>>>>>>> c8f523b3..66d965c9
>>>>>>>>> 100644
>>>>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>>>> @@ -6,6 +6,8 @@
>>>>>>>>>
>>>>>>>>> BASE_DISTRO = "raspbian"
>>>>>>>>> BASE_DISTRO_CODENAME = "stretch"
>>>>>>>>>
>>>>>>>>> +BASE_HOST_DISTRO = "debian"
>>>>>>>>
>>>>>>>> Better prefix it with HOST, to be consistent:
>>>>>>>>
>>>>>>>> HOST_BASE_DISTRO
>>>>>>>
>>>>>>> Not a problem, if you think it would be better.
>>>>>>>
>>>>>>>>> +
>>>>>>>>>
>>>>>>>>> HOST_DISTRO ?= "debian-stretch"
>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>>>
>>>>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index
>>>>>>>>> 5e4a09c9..7bd81dd6
>>>>>>>>> 100644
>>>>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>>>> @@ -6,6 +6,8 @@
>>>>>>>>>
>>>>>>>>> BASE_DISTRO = "raspios"
>>>>>>>>> BASE_DISTRO_arm64 = "debian"
>>>>>>>>>
>>>>>>>>> +BASE_HOST_DISTRO ?= "debian"
>>>>>>>>> +
>>>>>>>>>
>>>>>>>>> BASE_DISTRO_CODENAME = "bullseye"
>>>>>>>>> HOST_DISTRO ?= "debian-bullseye"
>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>>>
>>>>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1
>>>>>>>>> 100644
>>>>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
>>>>>>>>>
>>>>>>>>> BASE_DISTRO = "ubuntu"
>>>>>>>>> BASE_DISTRO_CODENAME = "focal"
>>>>>>>>>
>>>>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}"
>>>>>>>>> +
>>>>>>>>>
>>>>>>>>> DISTRO_APT_SOURCES_arm64 ?=
>>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
>>>>>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list
>>>>>>>>> conf/distro/${HOST_DISTRO}-ports.list">
>>>>>>>>>
>>>>>>>>> diff --git a/meta/conf/distro/debian-common.conf
>>>>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b
>>>>>>>>> 100644
>>>>>>>>> --- a/meta/conf/distro/debian-common.conf
>>>>>>>>> +++ b/meta/conf/distro/debian-common.conf
>>>>>>>>> @@ -5,6 +5,8 @@
>>>>>>>>>
>>>>>>>>> BASE_DISTRO = "debian"
>>>>>>>>>
>>>>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
>>>>>>>>> +
>>>>>>>>>
>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>>> DISTRO_APT_SOURCES ?=
>>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list">
>>>>>>>>>
>>>>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
>>>>>>>>> f32d192e..2ee0d4cc 100644
>>>>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg =
>>>>>>>>> ",gnupg"
>>>>>>>>>
>>>>>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =
>>>>>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if
>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO =
>>>>>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1'
>>>>>>>>> else
>>>>>>>>> 'DISTRO')}">
>>>>>>>>>
>>>>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if
>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}">
>>>>>>>>>
>>>>>>>>> FILESEXTRAPATHS_append = ":${BBPATH}"
>>>>>>>>>
>>>>>>>>> inherit deb-dl-dir
>>>>>>>>>
>>>>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d,
>>>
>>> aptsources_entry_list):
>>>>>>>>> import re
>>>>>>>>>
>>>>>>>>> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>>>>>>>>> - premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
>>>>>>>>> + premirrors = "\S*
>>>>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n">
>>>>>>>>>
>>>>>>>>> else:
>>>>>>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or ""
>>>>>>>>>
>>>>>>>>> mirror_list = [entry.split()
>>>>>>>>>
>>>>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() {
>>>>>>>>>
>>>>>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/bootst
>>>>>>>>> ra
>>>>>>>>> p"
>>>>>>>>>
>>>>>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
>>>>>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
>>>>>>>>>
>>>>>>>>> - line="file:///base-apt/${BASE_DISTRO}
>>>>>>>>> ${BASE_DISTRO_CODENAME}
>>>>>>>>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO}
>>>>>>>>> ${BASE_DISTRO_CODENAME} main">
>>>>>>>>>
>>>>>>>>> if [ -z "${BASE_REPO_KEY}" ]; then
>>>>>>>>>
>>>>>>>>> line="[trusted=yes] ${line}"
>>>>>>>>>
>>>>>>>>> fi
>>>>>>>>> echo "deb ${line}" >
>>>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
>>>>>>>>>
>>>>>>>>> + line="file:///base-apt/${BASE_DISTRO}
>>>>>>>>> ${BASE_DISTRO_CODENAME}
>>>>>>>>> main"
>>>>>>>>
>>>>>>>> Won't this cause duplicate lines (with complaints of apt) in case of
>>>>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO?
>>>>>>>
>>>>>>> No, while I don't add new lines (no new "echo"s). I just modify
>>>>>>> "deb-src"
>>>>>>> line while it should point to BASE_DISTRO even for host case. Without
>>>>>>> it,
>>>>>>> there was a problem with "hello" (apt_fetch failed to download cached
>>>>>>> sources from "host" repo, while sources were cached in "target" one.
>>>>>>>
>>>>>>> That's how it looks now:
>>>>>>> | $ cat
>>>>>>> | isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base-
>>>>>>>
>>>>>>> apt.list
>>>>>>>
>>>>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main
>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>>>>>>> | $ cat
>>>>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.li
>>>>>>> | st
>>>>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main
>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>>>>>>
>>>>>> Indeed, I misintepreted the code.
>>>>>>
>>>>>> But I think we still have an issue on the deb-src population side:
>>>>>>
>>>>>> # ls build/downloads/deb-src/
>>>>>> iot2050-debian
>>>>>>
>>>>>> And that was with this patch applied.
>>>>>>
>>>>>> Jan
>>>>>
>>>>> Actually, I checked the case with the following configuration:
>>>>>
>>>>> distro: my-debian
>>>>>
>>>>> local_conf_header:
>>>>> nodebian: |
>>>>>
>>>>> require conf/distro/debian-bullseye.conf
>>>>> DISTRO_NAME = "My Debian System"
>>>>>
>>>>> So, I simulated renaming distro that is done in meta-iot2050.
>>>>>
>>>>> Cached build was OK, "hello" package (that we build from sources) was
>>>>>
>>>>> downloaded at first build to "my-debian" directory:
>>>>> | $ ls build/downloads/deb-src/my-debian/hello/
>>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz
>>>>>
>>>>> and second build picked the files with no problem.
>>>>>
>>>>> The same with "raspios-bullseye" target - "hello" sources are downloaded
>>>>> to
>>>>>
>>>>> the "target" dir:
>>>>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/
>>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz
>>>>>
>>>>> I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" and
>>>>> it
>>>>> was OK (honestly, I don't know who could require cached build with this
>>>>> option enabled :) ).
>>>>>
>>>>> Anyway, I'll also try to check my changes with meta-iot2050 layer also
>>>>> (despite some packages in it are currently not compatible with
>>>>> BB_NO_NETWORK option)
>>>>
>>>> Please then test without the coral packages, they have an own issue /wrt
>>>> offline builds.
>>>
>>> Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works
>>
>> ...but it caches to the wrong downloads/deb-src path, no?
>>
>> Jan
>
> Why wrong?
>
> | meta-iot2050/build $ tree -d downloads/deb*
> | downloads/deb
> | ├── debian-bullseye
> | └── iot2050-debian
> | downloads/deb-src
> | └── iot2050-debian
> | ├── openssl
> | └── swig
>
> `do_apt_fetch` works with `${DISTRO}` and downloads sources to the
> corresponding directory.
>
deb-src fetching still lacks base-distro normalization, like deb also has:
# tree -d build/downloads/deb*
build/downloads/deb
└── debian-bullseye
build/downloads/deb-src
└── iot2050-debian
├── openssl
└── swig
(your deb folder was likely outdated)
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs
2022-11-28 6:32 ` Jan Kiszka
@ 2022-11-28 6:41 ` Uladzimir Bely
2022-11-28 6:51 ` Jan Kiszka
0 siblings, 1 reply; 18+ messages in thread
From: Uladzimir Bely @ 2022-11-28 6:41 UTC (permalink / raw)
To: Jan Kiszka, isar-users
In email from Monday, 28 November 2022 09:32:22 +03 You wrote:
> On 28.11.22 07:05, Uladzimir Bely wrote:
> > In the email from Sunday, 27 November 2022 22:05:53 +03 user Jan Kiszka
wrote:
> >> On 26.11.22 06:30, Uladzimir Bely wrote:
> >>> In the email from Friday, 25 November 2022 18:40:39 +03 user Jan Kiszka
> >
> > wrote:
> >>>> On 25.11.22 15:48, Uladzimir Bely wrote:
> >>>>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan
> >>>>> Kiszka
> >>>
> >>> wrote:
> >>>>>> On 25.11.22 08:56, Uladzimir Bely wrote:
> >>>>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan
> >>>>>>> Kiszka
> >>>>>
> >>>>> wrote:
> >>>>>>>> On 25.11.22 07:42, Uladzimir Bely wrote:
> >>>>>>>>> Downstreams may want to change DISTRO to some custom value.
> >>>>>>>>> When cross-building, this leads to downloading packages for host
> >>>>>>>>> and target distros to the different download subdirs.
> >>>>>>>>>
> >>>>>>>>> While base-apt is populated only from DISTRO download subdir,
> >>>>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to
> >>>>>>>>> missing packages in base-apt.
> >>>>>>>>>
> >>>>>>>>> The similar issue happens with cross-building raspberry targets
> >>>>>>>>> since they use their own mirrors and DISTRO.
> >>>>>>>>>
> >>>>>>>>> Fix the issue by splitting base-apt by introducing
> >>>>>>>>> BASE_HOST_DISTRO
> >>>>>>>>> variable that is used instead of BASE_DISTRO for caching builds
> >>>>>>>>> for host-related components (e.g., isar-bootstrap,
> >>>>>>>>> buildchroot-host,
> >>>>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO.
> >>>>>>>>>
> >>>>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> >>>>>>>>> ---
> >>>>>>>>>
> >>>>>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 +
> >>>>>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 +
> >>>>>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 +
> >>>>>>>>> meta/conf/distro/debian-common.conf | 2 +
> >>>>>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++-
> >>>>>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47
> >>>>>>>>> ++++++++++---------
> >>>>>>>>> 6 files changed, 40 insertions(+), 24 deletions(-)
> >>>>>>>>>
> >>>>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf
> >>>>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index
> >>>>>>>>> c8f523b3..66d965c9
> >>>>>>>>> 100644
> >>>>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf
> >>>>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf
> >>>>>>>>> @@ -6,6 +6,8 @@
> >>>>>>>>>
> >>>>>>>>> BASE_DISTRO = "raspbian"
> >>>>>>>>> BASE_DISTRO_CODENAME = "stretch"
> >>>>>>>>>
> >>>>>>>>> +BASE_HOST_DISTRO = "debian"
> >>>>>>>>
> >>>>>>>> Better prefix it with HOST, to be consistent:
> >>>>>>>>
> >>>>>>>> HOST_BASE_DISTRO
> >>>>>>>
> >>>>>>> Not a problem, if you think it would be better.
> >>>>>>>
> >>>>>>>>> +
> >>>>>>>>>
> >>>>>>>>> HOST_DISTRO ?= "debian-stretch"
> >>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>>>>>>>>
> >>>>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf
> >>>>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index
> >>>>>>>>> 5e4a09c9..7bd81dd6
> >>>>>>>>> 100644
> >>>>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf
> >>>>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf
> >>>>>>>>> @@ -6,6 +6,8 @@
> >>>>>>>>>
> >>>>>>>>> BASE_DISTRO = "raspios"
> >>>>>>>>> BASE_DISTRO_arm64 = "debian"
> >>>>>>>>>
> >>>>>>>>> +BASE_HOST_DISTRO ?= "debian"
> >>>>>>>>> +
> >>>>>>>>>
> >>>>>>>>> BASE_DISTRO_CODENAME = "bullseye"
> >>>>>>>>> HOST_DISTRO ?= "debian-bullseye"
> >>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>>>>>>>>
> >>>>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf
> >>>>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1
> >>>>>>>>> 100644
> >>>>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf
> >>>>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf
> >>>>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
> >>>>>>>>>
> >>>>>>>>> BASE_DISTRO = "ubuntu"
> >>>>>>>>> BASE_DISTRO_CODENAME = "focal"
> >>>>>>>>>
> >>>>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}"
> >>>>>>>>> +
> >>>>>>>>>
> >>>>>>>>> DISTRO_APT_SOURCES_arm64 ?=
> >>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
> >>>>>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list
> >>>>>>>>> conf/distro/${HOST_DISTRO}-ports.list">
> >>>>>>>>>
> >>>>>>>>> diff --git a/meta/conf/distro/debian-common.conf
> >>>>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b
> >>>>>>>>> 100644
> >>>>>>>>> --- a/meta/conf/distro/debian-common.conf
> >>>>>>>>> +++ b/meta/conf/distro/debian-common.conf
> >>>>>>>>> @@ -5,6 +5,8 @@
> >>>>>>>>>
> >>>>>>>>> BASE_DISTRO = "debian"
> >>>>>>>>>
> >>>>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
> >>>>>>>>> +
> >>>>>>>>>
> >>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>>>>>>>> DISTRO_APT_SOURCES ?=
> >>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list">
> >>>>>>>>>
> >>>>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> >>>>>>>>> f32d192e..2ee0d4cc 100644
> >>>>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>>>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>>>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg =
> >>>>>>>>> ",gnupg"
> >>>>>>>>>
> >>>>>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =
> >>>>>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if
> >>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO
> >>>>>>>>> =
> >>>>>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') ==
> >>>>>>>>> '1'
> >>>>>>>>> else
> >>>>>>>>> 'DISTRO')}">
> >>>>>>>>>
> >>>>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if
> >>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}">
> >>>>>>>>>
> >>>>>>>>> FILESEXTRAPATHS_append = ":${BBPATH}"
> >>>>>>>>>
> >>>>>>>>> inherit deb-dl-dir
> >>>>>>>>>
> >>>>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d,
> >>>
> >>> aptsources_entry_list):
> >>>>>>>>> import re
> >>>>>>>>>
> >>>>>>>>> if
bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> >>>>>>>>> - premirrors = "\S*
> >>>>>>>>> file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
> >>>>>>>>> + premirrors = "\S*
> >>>>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n">
> >>>>>>>>>
> >>>>>>>>> else:
> >>>>>>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or
> >>>>>>>>> ""
> >>>>>>>>>
> >>>>>>>>> mirror_list = [entry.split()
> >>>>>>>>>
> >>>>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() {
> >>>>>>>>>
> >>>>>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/boot
> >>>>>>>>> st
> >>>>>>>>> ra
> >>>>>>>>> p"
> >>>>>>>>>
> >>>>>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
> >>>>>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> >>>>>>>>>
> >>>>>>>>> - line="file:///base-apt/${BASE_DISTRO}
> >>>>>>>>> ${BASE_DISTRO_CODENAME}
> >>>>>>>>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO}
> >>>>>>>>> ${BASE_DISTRO_CODENAME} main">
> >>>>>>>>>
> >>>>>>>>> if [ -z "${BASE_REPO_KEY}" ]; then
> >>>>>>>>>
> >>>>>>>>> line="[trusted=yes] ${line}"
> >>>>>>>>>
> >>>>>>>>> fi
> >>>>>>>>> echo "deb ${line}" >
> >>>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
> >>>>>>>>>
> >>>>>>>>> + line="file:///base-apt/${BASE_DISTRO}
> >>>>>>>>> ${BASE_DISTRO_CODENAME}
> >>>>>>>>> main"
> >>>>>>>>
> >>>>>>>> Won't this cause duplicate lines (with complaints of apt) in case
> >>>>>>>> of
> >>>>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO?
> >>>>>>>
> >>>>>>> No, while I don't add new lines (no new "echo"s). I just modify
> >>>>>>> "deb-src"
> >>>>>>> line while it should point to BASE_DISTRO even for host case.
> >>>>>>> Without
> >>>>>>> it,
> >>>>>>> there was a problem with "hello" (apt_fetch failed to download
> >>>>>>> cached
> >>>>>>> sources from "host" repo, while sources were cached in "target" one.
> >>>>>>>
> >>>>>>> That's how it looks now:
> >>>>>>> | $ cat
> >>>>>>> | isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base-
> >>>>>>>
> >>>>>>> apt.list
> >>>>>>>
> >>>>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main
> >>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
> >>>>>>> | $ cat
> >>>>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.
> >>>>>>> | li
> >>>>>>> | st
> >>>>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main
> >>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
> >>>>>>
> >>>>>> Indeed, I misintepreted the code.
> >>>>>>
> >>>>>> But I think we still have an issue on the deb-src population side:
> >>>>>>
> >>>>>> # ls build/downloads/deb-src/
> >>>>>> iot2050-debian
> >>>>>>
> >>>>>> And that was with this patch applied.
> >>>>>>
> >>>>>> Jan
> >>>>>
> >>>>> Actually, I checked the case with the following configuration:
> >>>>>
> >>>>> distro: my-debian
> >>>>>
> >>>>> local_conf_header:
> >>>>> nodebian: |
> >>>>>
> >>>>> require conf/distro/debian-bullseye.conf
> >>>>> DISTRO_NAME = "My Debian System"
> >>>>>
> >>>>> So, I simulated renaming distro that is done in meta-iot2050.
> >>>>>
> >>>>> Cached build was OK, "hello" package (that we build from sources) was
> >>>>>
> >>>>> downloaded at first build to "my-debian" directory:
> >>>>> | $ ls build/downloads/deb-src/my-debian/hello/
> >>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz
> >>>>>
> >>>>> and second build picked the files with no problem.
> >>>>>
> >>>>> The same with "raspios-bullseye" target - "hello" sources are
> >>>>> downloaded
> >>>>> to
> >>>>>
> >>>>> the "target" dir:
> >>>>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/
> >>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz
> >>>>>
> >>>>> I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" and
> >>>>> it
> >>>>> was OK (honestly, I don't know who could require cached build with
> >>>>> this
> >>>>> option enabled :) ).
> >>>>>
> >>>>> Anyway, I'll also try to check my changes with meta-iot2050 layer also
> >>>>> (despite some packages in it are currently not compatible with
> >>>>> BB_NO_NETWORK option)
> >>>>
> >>>> Please then test without the coral packages, they have an own issue
> >>>> /wrt
> >>>> offline builds.
> >>>
> >>> Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works
> >>
> >> ...but it caches to the wrong downloads/deb-src path, no?
> >>
> >> Jan
> >
> > Why wrong?
> >
> > | meta-iot2050/build $ tree -d downloads/deb*
> > | downloads/deb
> > | ├── debian-bullseye
> > | └── iot2050-debian
> > | downloads/deb-src
> > | └── iot2050-debian
> > |
> > | ├── openssl
> > | └── swig
> >
> > `do_apt_fetch` works with `${DISTRO}` and downloads sources to the
> > corresponding directory.
>
> deb-src fetching still lacks base-distro normalization, like deb also has:
>
Saying "normalization", do you mean the following: if ${DISTRO} is just
renaming (e.g. "iot2050-debian") everything should be downloaded to and picked
up from the directory named "debian-bullseye"?
> # tree -d build/downloads/deb*
> build/downloads/deb
> └── debian-bullseye
> build/downloads/deb-src
> └── iot2050-debian
> ├── openssl
> └── swig
>
> (your deb folder was likely outdated)
>
> Jan
Uladzimir Bely
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs
2022-11-28 6:41 ` Uladzimir Bely
@ 2022-11-28 6:51 ` Jan Kiszka
2022-11-28 11:12 ` Uladzimir Bely
0 siblings, 1 reply; 18+ messages in thread
From: Jan Kiszka @ 2022-11-28 6:51 UTC (permalink / raw)
To: Uladzimir Bely, isar-users
On 28.11.22 07:41, Uladzimir Bely wrote:
> In email from Monday, 28 November 2022 09:32:22 +03 You wrote:
>> On 28.11.22 07:05, Uladzimir Bely wrote:
>>> In the email from Sunday, 27 November 2022 22:05:53 +03 user Jan Kiszka
> wrote:
>>>> On 26.11.22 06:30, Uladzimir Bely wrote:
>>>>> In the email from Friday, 25 November 2022 18:40:39 +03 user Jan Kiszka
>>>
>>> wrote:
>>>>>> On 25.11.22 15:48, Uladzimir Bely wrote:
>>>>>>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan
>>>>>>> Kiszka
>>>>>
>>>>> wrote:
>>>>>>>> On 25.11.22 08:56, Uladzimir Bely wrote:
>>>>>>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan
>>>>>>>>> Kiszka
>>>>>>>
>>>>>>> wrote:
>>>>>>>>>> On 25.11.22 07:42, Uladzimir Bely wrote:
>>>>>>>>>>> Downstreams may want to change DISTRO to some custom value.
>>>>>>>>>>> When cross-building, this leads to downloading packages for host
>>>>>>>>>>> and target distros to the different download subdirs.
>>>>>>>>>>>
>>>>>>>>>>> While base-apt is populated only from DISTRO download subdir,
>>>>>>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to
>>>>>>>>>>> missing packages in base-apt.
>>>>>>>>>>>
>>>>>>>>>>> The similar issue happens with cross-building raspberry targets
>>>>>>>>>>> since they use their own mirrors and DISTRO.
>>>>>>>>>>>
>>>>>>>>>>> Fix the issue by splitting base-apt by introducing
>>>>>>>>>>> BASE_HOST_DISTRO
>>>>>>>>>>> variable that is used instead of BASE_DISTRO for caching builds
>>>>>>>>>>> for host-related components (e.g., isar-bootstrap,
>>>>>>>>>>> buildchroot-host,
>>>>>>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO.
>>>>>>>>>>>
>>>>>>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
>>>>>>>>>>> ---
>>>>>>>>>>>
>>>>>>>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 +
>>>>>>>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 +
>>>>>>>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 +
>>>>>>>>>>> meta/conf/distro/debian-common.conf | 2 +
>>>>>>>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++-
>>>>>>>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47
>>>>>>>>>>> ++++++++++---------
>>>>>>>>>>> 6 files changed, 40 insertions(+), 24 deletions(-)
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index
>>>>>>>>>>> c8f523b3..66d965c9
>>>>>>>>>>> 100644
>>>>>>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>>>>>> @@ -6,6 +6,8 @@
>>>>>>>>>>>
>>>>>>>>>>> BASE_DISTRO = "raspbian"
>>>>>>>>>>> BASE_DISTRO_CODENAME = "stretch"
>>>>>>>>>>>
>>>>>>>>>>> +BASE_HOST_DISTRO = "debian"
>>>>>>>>>>
>>>>>>>>>> Better prefix it with HOST, to be consistent:
>>>>>>>>>>
>>>>>>>>>> HOST_BASE_DISTRO
>>>>>>>>>
>>>>>>>>> Not a problem, if you think it would be better.
>>>>>>>>>
>>>>>>>>>>> +
>>>>>>>>>>>
>>>>>>>>>>> HOST_DISTRO ?= "debian-stretch"
>>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index
>>>>>>>>>>> 5e4a09c9..7bd81dd6
>>>>>>>>>>> 100644
>>>>>>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>>>>>> @@ -6,6 +6,8 @@
>>>>>>>>>>>
>>>>>>>>>>> BASE_DISTRO = "raspios"
>>>>>>>>>>> BASE_DISTRO_arm64 = "debian"
>>>>>>>>>>>
>>>>>>>>>>> +BASE_HOST_DISTRO ?= "debian"
>>>>>>>>>>> +
>>>>>>>>>>>
>>>>>>>>>>> BASE_DISTRO_CODENAME = "bullseye"
>>>>>>>>>>> HOST_DISTRO ?= "debian-bullseye"
>>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1
>>>>>>>>>>> 100644
>>>>>>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
>>>>>>>>>>>
>>>>>>>>>>> BASE_DISTRO = "ubuntu"
>>>>>>>>>>> BASE_DISTRO_CODENAME = "focal"
>>>>>>>>>>>
>>>>>>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}"
>>>>>>>>>>> +
>>>>>>>>>>>
>>>>>>>>>>> DISTRO_APT_SOURCES_arm64 ?=
>>>>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
>>>>>>>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list
>>>>>>>>>>> conf/distro/${HOST_DISTRO}-ports.list">
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/meta/conf/distro/debian-common.conf
>>>>>>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b
>>>>>>>>>>> 100644
>>>>>>>>>>> --- a/meta/conf/distro/debian-common.conf
>>>>>>>>>>> +++ b/meta/conf/distro/debian-common.conf
>>>>>>>>>>> @@ -5,6 +5,8 @@
>>>>>>>>>>>
>>>>>>>>>>> BASE_DISTRO = "debian"
>>>>>>>>>>>
>>>>>>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
>>>>>>>>>>> +
>>>>>>>>>>>
>>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>>>>> DISTRO_APT_SOURCES ?=
>>>>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list">
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
>>>>>>>>>>> f32d192e..2ee0d4cc 100644
>>>>>>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg =
>>>>>>>>>>> ",gnupg"
>>>>>>>>>>>
>>>>>>>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =
>>>>>>>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if
>>>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO
>>>>>>>>>>> =
>>>>>>>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') ==
>>>>>>>>>>> '1'
>>>>>>>>>>> else
>>>>>>>>>>> 'DISTRO')}">
>>>>>>>>>>>
>>>>>>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if
>>>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}">
>>>>>>>>>>>
>>>>>>>>>>> FILESEXTRAPATHS_append = ":${BBPATH}"
>>>>>>>>>>>
>>>>>>>>>>> inherit deb-dl-dir
>>>>>>>>>>>
>>>>>>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d,
>>>>>
>>>>> aptsources_entry_list):
>>>>>>>>>>> import re
>>>>>>>>>>>
>>>>>>>>>>> if
> bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>>>>>>>>>>> - premirrors = "\S*
>>>>>>>>>>> file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
>>>>>>>>>>> + premirrors = "\S*
>>>>>>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n">
>>>>>>>>>>>
>>>>>>>>>>> else:
>>>>>>>>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or
>>>>>>>>>>> ""
>>>>>>>>>>>
>>>>>>>>>>> mirror_list = [entry.split()
>>>>>>>>>>>
>>>>>>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() {
>>>>>>>>>>>
>>>>>>>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/boot
>>>>>>>>>>> st
>>>>>>>>>>> ra
>>>>>>>>>>> p"
>>>>>>>>>>>
>>>>>>>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
>>>>>>>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
>>>>>>>>>>>
>>>>>>>>>>> - line="file:///base-apt/${BASE_DISTRO}
>>>>>>>>>>> ${BASE_DISTRO_CODENAME}
>>>>>>>>>>> main" + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO}
>>>>>>>>>>> ${BASE_DISTRO_CODENAME} main">
>>>>>>>>>>>
>>>>>>>>>>> if [ -z "${BASE_REPO_KEY}" ]; then
>>>>>>>>>>>
>>>>>>>>>>> line="[trusted=yes] ${line}"
>>>>>>>>>>>
>>>>>>>>>>> fi
>>>>>>>>>>> echo "deb ${line}" >
>>>>>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
>>>>>>>>>>>
>>>>>>>>>>> + line="file:///base-apt/${BASE_DISTRO}
>>>>>>>>>>> ${BASE_DISTRO_CODENAME}
>>>>>>>>>>> main"
>>>>>>>>>>
>>>>>>>>>> Won't this cause duplicate lines (with complaints of apt) in case
>>>>>>>>>> of
>>>>>>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO?
>>>>>>>>>
>>>>>>>>> No, while I don't add new lines (no new "echo"s). I just modify
>>>>>>>>> "deb-src"
>>>>>>>>> line while it should point to BASE_DISTRO even for host case.
>>>>>>>>> Without
>>>>>>>>> it,
>>>>>>>>> there was a problem with "hello" (apt_fetch failed to download
>>>>>>>>> cached
>>>>>>>>> sources from "host" repo, while sources were cached in "target" one.
>>>>>>>>>
>>>>>>>>> That's how it looks now:
>>>>>>>>> | $ cat
>>>>>>>>> | isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base-
>>>>>>>>>
>>>>>>>>> apt.list
>>>>>>>>>
>>>>>>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main
>>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>>>>>>>>> | $ cat
>>>>>>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.
>>>>>>>>> | li
>>>>>>>>> | st
>>>>>>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main
>>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>>>>>>>>
>>>>>>>> Indeed, I misintepreted the code.
>>>>>>>>
>>>>>>>> But I think we still have an issue on the deb-src population side:
>>>>>>>>
>>>>>>>> # ls build/downloads/deb-src/
>>>>>>>> iot2050-debian
>>>>>>>>
>>>>>>>> And that was with this patch applied.
>>>>>>>>
>>>>>>>> Jan
>>>>>>>
>>>>>>> Actually, I checked the case with the following configuration:
>>>>>>>
>>>>>>> distro: my-debian
>>>>>>>
>>>>>>> local_conf_header:
>>>>>>> nodebian: |
>>>>>>>
>>>>>>> require conf/distro/debian-bullseye.conf
>>>>>>> DISTRO_NAME = "My Debian System"
>>>>>>>
>>>>>>> So, I simulated renaming distro that is done in meta-iot2050.
>>>>>>>
>>>>>>> Cached build was OK, "hello" package (that we build from sources) was
>>>>>>>
>>>>>>> downloaded at first build to "my-debian" directory:
>>>>>>> | $ ls build/downloads/deb-src/my-debian/hello/
>>>>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz
>>>>>>>
>>>>>>> and second build picked the files with no problem.
>>>>>>>
>>>>>>> The same with "raspios-bullseye" target - "hello" sources are
>>>>>>> downloaded
>>>>>>> to
>>>>>>>
>>>>>>> the "target" dir:
>>>>>>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/
>>>>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc hello_2.10.orig.tar.gz
>>>>>>>
>>>>>>> I even tested the case with BASE_REPO_FEATURES += "cache-deb-src" and
>>>>>>> it
>>>>>>> was OK (honestly, I don't know who could require cached build with
>>>>>>> this
>>>>>>> option enabled :) ).
>>>>>>>
>>>>>>> Anyway, I'll also try to check my changes with meta-iot2050 layer also
>>>>>>> (despite some packages in it are currently not compatible with
>>>>>>> BB_NO_NETWORK option)
>>>>>>
>>>>>> Please then test without the coral packages, they have an own issue
>>>>>> /wrt
>>>>>> offline builds.
>>>>>
>>>>> Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works
>>>>
>>>> ...but it caches to the wrong downloads/deb-src path, no?
>>>>
>>>> Jan
>>>
>>> Why wrong?
>>>
>>> | meta-iot2050/build $ tree -d downloads/deb*
>>> | downloads/deb
>>> | ├── debian-bullseye
>>> | └── iot2050-debian
>>> | downloads/deb-src
>>> | └── iot2050-debian
>>> |
>>> | ├── openssl
>>> | └── swig
>>>
>>> `do_apt_fetch` works with `${DISTRO}` and downloads sources to the
>>> corresponding directory.
>>
>> deb-src fetching still lacks base-distro normalization, like deb also has:
>>
>
> Saying "normalization", do you mean the following: if ${DISTRO} is just
> renaming (e.g. "iot2050-debian") everything should be downloaded to and picked
> up from the directory named "debian-bullseye"?
>
Yes, just like you did for binary debs: Reduce the DISTRO to its
BASE_DISTRO, then use that as key for the cache. The current situation
is inconsistent and non-optimal, even if working.
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs
2022-11-28 6:51 ` Jan Kiszka
@ 2022-11-28 11:12 ` Uladzimir Bely
2022-11-28 11:45 ` Jan Kiszka
0 siblings, 1 reply; 18+ messages in thread
From: Uladzimir Bely @ 2022-11-28 11:12 UTC (permalink / raw)
To: isar-users, Jan Kiszka
In the email from Monday, 28 November 2022 09:51:12 +03 user Jan Kiszka wrote:
> On 28.11.22 07:41, Uladzimir Bely wrote:
> > In email from Monday, 28 November 2022 09:32:22 +03 You wrote:
> >> On 28.11.22 07:05, Uladzimir Bely wrote:
> >>> In the email from Sunday, 27 November 2022 22:05:53 +03 user Jan Kiszka
> >
> > wrote:
> >>>> On 26.11.22 06:30, Uladzimir Bely wrote:
> >>>>> In the email from Friday, 25 November 2022 18:40:39 +03 user Jan
> >>>>> Kiszka
> >>>
> >>> wrote:
> >>>>>> On 25.11.22 15:48, Uladzimir Bely wrote:
> >>>>>>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan
> >>>>>>> Kiszka
> >>>>>
> >>>>> wrote:
> >>>>>>>> On 25.11.22 08:56, Uladzimir Bely wrote:
> >>>>>>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan
> >>>>>>>>> Kiszka
> >>>>>>>
> >>>>>>> wrote:
> >>>>>>>>>> On 25.11.22 07:42, Uladzimir Bely wrote:
> >>>>>>>>>>> Downstreams may want to change DISTRO to some custom value.
> >>>>>>>>>>> When cross-building, this leads to downloading packages for host
> >>>>>>>>>>> and target distros to the different download subdirs.
> >>>>>>>>>>>
> >>>>>>>>>>> While base-apt is populated only from DISTRO download subdir,
> >>>>>>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due
> >>>>>>>>>>> to
> >>>>>>>>>>> missing packages in base-apt.
> >>>>>>>>>>>
> >>>>>>>>>>> The similar issue happens with cross-building raspberry targets
> >>>>>>>>>>> since they use their own mirrors and DISTRO.
> >>>>>>>>>>>
> >>>>>>>>>>> Fix the issue by splitting base-apt by introducing
> >>>>>>>>>>> BASE_HOST_DISTRO
> >>>>>>>>>>> variable that is used instead of BASE_DISTRO for caching builds
> >>>>>>>>>>> for host-related components (e.g., isar-bootstrap,
> >>>>>>>>>>> buildchroot-host,
> >>>>>>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO.
> >>>>>>>>>>>
> >>>>>>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> >>>>>>>>>>> ---
> >>>>>>>>>>>
> >>>>>>>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 +
> >>>>>>>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 +
> >>>>>>>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 +
> >>>>>>>>>>> meta/conf/distro/debian-common.conf | 2 +
> >>>>>>>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++-
> >>>>>>>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47
> >>>>>>>>>>> ++++++++++---------
> >>>>>>>>>>> 6 files changed, 40 insertions(+), 24 deletions(-)
> >>>>>>>>>>>
> >>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf
> >>>>>>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index
> >>>>>>>>>>> c8f523b3..66d965c9
> >>>>>>>>>>> 100644
> >>>>>>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf
> >>>>>>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf
> >>>>>>>>>>> @@ -6,6 +6,8 @@
> >>>>>>>>>>>
> >>>>>>>>>>> BASE_DISTRO = "raspbian"
> >>>>>>>>>>> BASE_DISTRO_CODENAME = "stretch"
> >>>>>>>>>>>
> >>>>>>>>>>> +BASE_HOST_DISTRO = "debian"
> >>>>>>>>>>
> >>>>>>>>>> Better prefix it with HOST, to be consistent:
> >>>>>>>>>>
> >>>>>>>>>> HOST_BASE_DISTRO
> >>>>>>>>>
> >>>>>>>>> Not a problem, if you think it would be better.
> >>>>>>>>>
> >>>>>>>>>>> +
> >>>>>>>>>>>
> >>>>>>>>>>> HOST_DISTRO ?= "debian-stretch"
> >>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>>>>>>>>>>
> >>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf
> >>>>>>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index
> >>>>>>>>>>> 5e4a09c9..7bd81dd6
> >>>>>>>>>>> 100644
> >>>>>>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf
> >>>>>>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf
> >>>>>>>>>>> @@ -6,6 +6,8 @@
> >>>>>>>>>>>
> >>>>>>>>>>> BASE_DISTRO = "raspios"
> >>>>>>>>>>> BASE_DISTRO_arm64 = "debian"
> >>>>>>>>>>>
> >>>>>>>>>>> +BASE_HOST_DISTRO ?= "debian"
> >>>>>>>>>>> +
> >>>>>>>>>>>
> >>>>>>>>>>> BASE_DISTRO_CODENAME = "bullseye"
> >>>>>>>>>>> HOST_DISTRO ?= "debian-bullseye"
> >>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>>>>>>>>>>
> >>>>>>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf
> >>>>>>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index
> >>>>>>>>>>> 379b5b30..75da79a1
> >>>>>>>>>>> 100644
> >>>>>>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf
> >>>>>>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf
> >>>>>>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
> >>>>>>>>>>>
> >>>>>>>>>>> BASE_DISTRO = "ubuntu"
> >>>>>>>>>>> BASE_DISTRO_CODENAME = "focal"
> >>>>>>>>>>>
> >>>>>>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}"
> >>>>>>>>>>> +
> >>>>>>>>>>>
> >>>>>>>>>>> DISTRO_APT_SOURCES_arm64 ?=
> >>>>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
> >>>>>>>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?=
> >>>>>>>>>>> "conf/distro/${HOST_DISTRO}.list
> >>>>>>>>>>> conf/distro/${HOST_DISTRO}-ports.list">
> >>>>>>>>>>>
> >>>>>>>>>>> diff --git a/meta/conf/distro/debian-common.conf
> >>>>>>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b
> >>>>>>>>>>> 100644
> >>>>>>>>>>> --- a/meta/conf/distro/debian-common.conf
> >>>>>>>>>>> +++ b/meta/conf/distro/debian-common.conf
> >>>>>>>>>>> @@ -5,6 +5,8 @@
> >>>>>>>>>>>
> >>>>>>>>>>> BASE_DISTRO = "debian"
> >>>>>>>>>>>
> >>>>>>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
> >>>>>>>>>>> +
> >>>>>>>>>>>
> >>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>>>>>>>>>> DISTRO_APT_SOURCES ?=
> >>>>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list">
> >>>>>>>>>>>
> >>>>>>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>>>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> >>>>>>>>>>> f32d192e..2ee0d4cc 100644
> >>>>>>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>>>>>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>>>>>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg =
> >>>>>>>>>>> ",gnupg"
> >>>>>>>>>>>
> >>>>>>>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =
> >>>>>>>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if
> >>>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
> >>>>>>>>>>> BOOTSTRAP_DISTRO
> >>>>>>>>>>> =
> >>>>>>>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') ==
> >>>>>>>>>>> '1'
> >>>>>>>>>>> else
> >>>>>>>>>>> 'DISTRO')}">
> >>>>>>>>>>>
> >>>>>>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if
> >>>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}">
> >>>>>>>>>>>
> >>>>>>>>>>> FILESEXTRAPATHS_append = ":${BBPATH}"
> >>>>>>>>>>>
> >>>>>>>>>>> inherit deb-dl-dir
> >>>>>>>>>>>
> >>>>>>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d,
> >>>>>
> >>>>> aptsources_entry_list):
> >>>>>>>>>>> import re
> >>>>>>>>>>>
> >>>>>>>>>>> if
> >
> > bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> >>>>>>>>>>> - premirrors = "\S*
> >>>>>>>>>>> file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
> >>>>>>>>>>> + premirrors = "\S*
> >>>>>>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n">
> >>>>>>>>>>>
> >>>>>>>>>>> else:
> >>>>>>>>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or
> >>>>>>>>>>> ""
> >>>>>>>>>>>
> >>>>>>>>>>> mirror_list = [entry.split()
> >>>>>>>>>>>
> >>>>>>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() {
> >>>>>>>>>>>
> >>>>>>>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/bo
> >>>>>>>>>>> ot
> >>>>>>>>>>> st
> >>>>>>>>>>> ra
> >>>>>>>>>>> p"
> >>>>>>>>>>>
> >>>>>>>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
> >>>>>>>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> >>>>>>>>>>>
> >>>>>>>>>>> - line="file:///base-apt/${BASE_DISTRO}
> >>>>>>>>>>> ${BASE_DISTRO_CODENAME}
> >>>>>>>>>>> main" +
> >>>>>>>>>>> line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO}
> >>>>>>>>>>> ${BASE_DISTRO_CODENAME} main">
> >>>>>>>>>>>
> >>>>>>>>>>> if [ -z "${BASE_REPO_KEY}" ]; then
> >>>>>>>>>>>
> >>>>>>>>>>> line="[trusted=yes] ${line}"
> >>>>>>>>>>>
> >>>>>>>>>>> fi
> >>>>>>>>>>> echo "deb ${line}" >
> >>>>>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
> >>>>>>>>>>>
> >>>>>>>>>>> + line="file:///base-apt/${BASE_DISTRO}
> >>>>>>>>>>> ${BASE_DISTRO_CODENAME}
> >>>>>>>>>>> main"
> >>>>>>>>>>
> >>>>>>>>>> Won't this cause duplicate lines (with complaints of apt) in case
> >>>>>>>>>> of
> >>>>>>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO?
> >>>>>>>>>
> >>>>>>>>> No, while I don't add new lines (no new "echo"s). I just modify
> >>>>>>>>> "deb-src"
> >>>>>>>>> line while it should point to BASE_DISTRO even for host case.
> >>>>>>>>> Without
> >>>>>>>>> it,
> >>>>>>>>> there was a problem with "hello" (apt_fetch failed to download
> >>>>>>>>> cached
> >>>>>>>>> sources from "host" repo, while sources were cached in "target"
> >>>>>>>>> one.
> >>>>>>>>>
> >>>>>>>>> That's how it looks now:
> >>>>>>>>> | $ cat
> >>>>>>>>> | isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base-
> >>>>>>>>>
> >>>>>>>>> apt.list
> >>>>>>>>>
> >>>>>>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main
> >>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
> >>>>>>>>> | $ cat
> >>>>>>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-ap
> >>>>>>>>> | t.
> >>>>>>>>> | li
> >>>>>>>>> | st
> >>>>>>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main
> >>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
> >>>>>>>>
> >>>>>>>> Indeed, I misintepreted the code.
> >>>>>>>>
> >>>>>>>> But I think we still have an issue on the deb-src population side:
> >>>>>>>>
> >>>>>>>> # ls build/downloads/deb-src/
> >>>>>>>> iot2050-debian
> >>>>>>>>
> >>>>>>>> And that was with this patch applied.
> >>>>>>>>
> >>>>>>>> Jan
> >>>>>>>
> >>>>>>> Actually, I checked the case with the following configuration:
> >>>>>>>
> >>>>>>> distro: my-debian
> >>>>>>>
> >>>>>>> local_conf_header:
> >>>>>>> nodebian: |
> >>>>>>>
> >>>>>>> require conf/distro/debian-bullseye.conf
> >>>>>>> DISTRO_NAME = "My Debian System"
> >>>>>>>
> >>>>>>> So, I simulated renaming distro that is done in meta-iot2050.
> >>>>>>>
> >>>>>>> Cached build was OK, "hello" package (that we build from sources)
> >>>>>>> was
> >>>>>>>
> >>>>>>> downloaded at first build to "my-debian" directory:
> >>>>>>> | $ ls build/downloads/deb-src/my-debian/hello/
> >>>>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc
> >>>>>>> | hello_2.10.orig.tar.gz
> >>>>>>>
> >>>>>>> and second build picked the files with no problem.
> >>>>>>>
> >>>>>>> The same with "raspios-bullseye" target - "hello" sources are
> >>>>>>> downloaded
> >>>>>>> to
> >>>>>>>
> >>>>>>> the "target" dir:
> >>>>>>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/
> >>>>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc
> >>>>>>> | hello_2.10.orig.tar.gz
> >>>>>>>
> >>>>>>> I even tested the case with BASE_REPO_FEATURES += "cache-deb-src"
> >>>>>>> and
> >>>>>>> it
> >>>>>>> was OK (honestly, I don't know who could require cached build with
> >>>>>>> this
> >>>>>>> option enabled :) ).
> >>>>>>>
> >>>>>>> Anyway, I'll also try to check my changes with meta-iot2050 layer
> >>>>>>> also
> >>>>>>> (despite some packages in it are currently not compatible with
> >>>>>>> BB_NO_NETWORK option)
> >>>>>>
> >>>>>> Please then test without the coral packages, they have an own issue
> >>>>>> /wrt
> >>>>>> offline builds.
> >>>>>
> >>>>> Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works
> >>>>
> >>>> ...but it caches to the wrong downloads/deb-src path, no?
> >>>>
> >>>> Jan
> >>>
> >>> Why wrong?
> >>>
> >>> | meta-iot2050/build $ tree -d downloads/deb*
> >>> | downloads/deb
> >>> | ├── debian-bullseye
> >>> | └── iot2050-debian
> >>> | downloads/deb-src
> >>> | └── iot2050-debian
> >>> |
> >>> | ├── openssl
> >>> | └── swig
> >>>
> >>> `do_apt_fetch` works with `${DISTRO}` and downloads sources to the
> >>> corresponding directory.
> >>
> >> deb-src fetching still lacks base-distro normalization, like deb also
has:
> > Saying "normalization", do you mean the following: if ${DISTRO} is just
> > renaming (e.g. "iot2050-debian") everything should be downloaded to and
> > picked up from the directory named "debian-bullseye"?
>
> Yes, just like you did for binary debs: Reduce the DISTRO to its
> BASE_DISTRO, then use that as key for the cache. The current situation
> is inconsistent and non-optimal, even if working.
>
> Jan
I've just pushed v4 of patchset that still doesn't include this wish. For now,
I would apply it to the isar/next to have cached cross build working.
We have some plans to improve base-apt in general (there were some PoC patches
on list), so improvements you mention could be done a bit later - for now I
need to switch to some other tasks.
--
Uladzimir Bely
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs
2022-11-28 11:12 ` Uladzimir Bely
@ 2022-11-28 11:45 ` Jan Kiszka
0 siblings, 0 replies; 18+ messages in thread
From: Jan Kiszka @ 2022-11-28 11:45 UTC (permalink / raw)
To: Uladzimir Bely, isar-users
On 28.11.22 12:12, Uladzimir Bely wrote:
> In the email from Monday, 28 November 2022 09:51:12 +03 user Jan Kiszka wrote:
>> On 28.11.22 07:41, Uladzimir Bely wrote:
>>> In email from Monday, 28 November 2022 09:32:22 +03 You wrote:
>>>> On 28.11.22 07:05, Uladzimir Bely wrote:
>>>>> In the email from Sunday, 27 November 2022 22:05:53 +03 user Jan Kiszka
>>>
>>> wrote:
>>>>>> On 26.11.22 06:30, Uladzimir Bely wrote:
>>>>>>> In the email from Friday, 25 November 2022 18:40:39 +03 user Jan
>>>>>>> Kiszka
>>>>>
>>>>> wrote:
>>>>>>>> On 25.11.22 15:48, Uladzimir Bely wrote:
>>>>>>>>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan
>>>>>>>>> Kiszka
>>>>>>>
>>>>>>> wrote:
>>>>>>>>>> On 25.11.22 08:56, Uladzimir Bely wrote:
>>>>>>>>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan
>>>>>>>>>>> Kiszka
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>> On 25.11.22 07:42, Uladzimir Bely wrote:
>>>>>>>>>>>>> Downstreams may want to change DISTRO to some custom value.
>>>>>>>>>>>>> When cross-building, this leads to downloading packages for host
>>>>>>>>>>>>> and target distros to the different download subdirs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> While base-apt is populated only from DISTRO download subdir,
>>>>>>>>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due
>>>>>>>>>>>>> to
>>>>>>>>>>>>> missing packages in base-apt.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The similar issue happens with cross-building raspberry targets
>>>>>>>>>>>>> since they use their own mirrors and DISTRO.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Fix the issue by splitting base-apt by introducing
>>>>>>>>>>>>> BASE_HOST_DISTRO
>>>>>>>>>>>>> variable that is used instead of BASE_DISTRO for caching builds
>>>>>>>>>>>>> for host-related components (e.g., isar-bootstrap,
>>>>>>>>>>>>> buildchroot-host,
>>>>>>>>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
>>>>>>>>>>>>> ---
>>>>>>>>>>>>>
>>>>>>>>>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 +
>>>>>>>>>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 +
>>>>>>>>>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 +
>>>>>>>>>>>>> meta/conf/distro/debian-common.conf | 2 +
>>>>>>>>>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++-
>>>>>>>>>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47
>>>>>>>>>>>>> ++++++++++---------
>>>>>>>>>>>>> 6 files changed, 40 insertions(+), 24 deletions(-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>>>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index
>>>>>>>>>>>>> c8f523b3..66d965c9
>>>>>>>>>>>>> 100644
>>>>>>>>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>>>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>>>>>>>> @@ -6,6 +6,8 @@
>>>>>>>>>>>>>
>>>>>>>>>>>>> BASE_DISTRO = "raspbian"
>>>>>>>>>>>>> BASE_DISTRO_CODENAME = "stretch"
>>>>>>>>>>>>>
>>>>>>>>>>>>> +BASE_HOST_DISTRO = "debian"
>>>>>>>>>>>>
>>>>>>>>>>>> Better prefix it with HOST, to be consistent:
>>>>>>>>>>>>
>>>>>>>>>>>> HOST_BASE_DISTRO
>>>>>>>>>>>
>>>>>>>>>>> Not a problem, if you think it would be better.
>>>>>>>>>>>
>>>>>>>>>>>>> +
>>>>>>>>>>>>>
>>>>>>>>>>>>> HOST_DISTRO ?= "debian-stretch"
>>>>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>>>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index
>>>>>>>>>>>>> 5e4a09c9..7bd81dd6
>>>>>>>>>>>>> 100644
>>>>>>>>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>>>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>>>>>>>> @@ -6,6 +6,8 @@
>>>>>>>>>>>>>
>>>>>>>>>>>>> BASE_DISTRO = "raspios"
>>>>>>>>>>>>> BASE_DISTRO_arm64 = "debian"
>>>>>>>>>>>>>
>>>>>>>>>>>>> +BASE_HOST_DISTRO ?= "debian"
>>>>>>>>>>>>> +
>>>>>>>>>>>>>
>>>>>>>>>>>>> BASE_DISTRO_CODENAME = "bullseye"
>>>>>>>>>>>>> HOST_DISTRO ?= "debian-bullseye"
>>>>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>>>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index
>>>>>>>>>>>>> 379b5b30..75da79a1
>>>>>>>>>>>>> 100644
>>>>>>>>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>>>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>>>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
>>>>>>>>>>>>>
>>>>>>>>>>>>> BASE_DISTRO = "ubuntu"
>>>>>>>>>>>>> BASE_DISTRO_CODENAME = "focal"
>>>>>>>>>>>>>
>>>>>>>>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}"
>>>>>>>>>>>>> +
>>>>>>>>>>>>>
>>>>>>>>>>>>> DISTRO_APT_SOURCES_arm64 ?=
>>>>>>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
>>>>>>>>>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?=
>>>>>>>>>>>>> "conf/distro/${HOST_DISTRO}.list
>>>>>>>>>>>>> conf/distro/${HOST_DISTRO}-ports.list">
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/meta/conf/distro/debian-common.conf
>>>>>>>>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b
>>>>>>>>>>>>> 100644
>>>>>>>>>>>>> --- a/meta/conf/distro/debian-common.conf
>>>>>>>>>>>>> +++ b/meta/conf/distro/debian-common.conf
>>>>>>>>>>>>> @@ -5,6 +5,8 @@
>>>>>>>>>>>>>
>>>>>>>>>>>>> BASE_DISTRO = "debian"
>>>>>>>>>>>>>
>>>>>>>>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
>>>>>>>>>>>>> +
>>>>>>>>>>>>>
>>>>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>>>>>>> DISTRO_APT_SOURCES ?=
>>>>>>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list">
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
>>>>>>>>>>>>> f32d192e..2ee0d4cc 100644
>>>>>>>>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>>>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>>>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg =
>>>>>>>>>>>>> ",gnupg"
>>>>>>>>>>>>>
>>>>>>>>>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =
>>>>>>>>>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if
>>>>>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
>>>>>>>>>>>>> BOOTSTRAP_DISTRO
>>>>>>>>>>>>> =
>>>>>>>>>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') ==
>>>>>>>>>>>>> '1'
>>>>>>>>>>>>> else
>>>>>>>>>>>>> 'DISTRO')}">
>>>>>>>>>>>>>
>>>>>>>>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if
>>>>>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}">
>>>>>>>>>>>>>
>>>>>>>>>>>>> FILESEXTRAPATHS_append = ":${BBPATH}"
>>>>>>>>>>>>>
>>>>>>>>>>>>> inherit deb-dl-dir
>>>>>>>>>>>>>
>>>>>>>>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d,
>>>>>>>
>>>>>>> aptsources_entry_list):
>>>>>>>>>>>>> import re
>>>>>>>>>>>>>
>>>>>>>>>>>>> if
>>>
>>> bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>>>>>>>>>>>>> - premirrors = "\S*
>>>>>>>>>>>>> file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
>>>>>>>>>>>>> + premirrors = "\S*
>>>>>>>>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n">
>>>>>>>>>>>>>
>>>>>>>>>>>>> else:
>>>>>>>>>>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or
>>>>>>>>>>>>> ""
>>>>>>>>>>>>>
>>>>>>>>>>>>> mirror_list = [entry.split()
>>>>>>>>>>>>>
>>>>>>>>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() {
>>>>>>>>>>>>>
>>>>>>>>>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/bo
>>>>>>>>>>>>> ot
>>>>>>>>>>>>> st
>>>>>>>>>>>>> ra
>>>>>>>>>>>>> p"
>>>>>>>>>>>>>
>>>>>>>>>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
>>>>>>>>>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
>>>>>>>>>>>>>
>>>>>>>>>>>>> - line="file:///base-apt/${BASE_DISTRO}
>>>>>>>>>>>>> ${BASE_DISTRO_CODENAME}
>>>>>>>>>>>>> main" +
>>>>>>>>>>>>> line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO}
>>>>>>>>>>>>> ${BASE_DISTRO_CODENAME} main">
>>>>>>>>>>>>>
>>>>>>>>>>>>> if [ -z "${BASE_REPO_KEY}" ]; then
>>>>>>>>>>>>>
>>>>>>>>>>>>> line="[trusted=yes] ${line}"
>>>>>>>>>>>>>
>>>>>>>>>>>>> fi
>>>>>>>>>>>>> echo "deb ${line}" >
>>>>>>>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
>>>>>>>>>>>>>
>>>>>>>>>>>>> + line="file:///base-apt/${BASE_DISTRO}
>>>>>>>>>>>>> ${BASE_DISTRO_CODENAME}
>>>>>>>>>>>>> main"
>>>>>>>>>>>>
>>>>>>>>>>>> Won't this cause duplicate lines (with complaints of apt) in case
>>>>>>>>>>>> of
>>>>>>>>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO?
>>>>>>>>>>>
>>>>>>>>>>> No, while I don't add new lines (no new "echo"s). I just modify
>>>>>>>>>>> "deb-src"
>>>>>>>>>>> line while it should point to BASE_DISTRO even for host case.
>>>>>>>>>>> Without
>>>>>>>>>>> it,
>>>>>>>>>>> there was a problem with "hello" (apt_fetch failed to download
>>>>>>>>>>> cached
>>>>>>>>>>> sources from "host" repo, while sources were cached in "target"
>>>>>>>>>>> one.
>>>>>>>>>>>
>>>>>>>>>>> That's how it looks now:
>>>>>>>>>>> | $ cat
>>>>>>>>>>> | isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base-
>>>>>>>>>>>
>>>>>>>>>>> apt.list
>>>>>>>>>>>
>>>>>>>>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main
>>>>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>>>>>>>>>>> | $ cat
>>>>>>>>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-ap
>>>>>>>>>>> | t.
>>>>>>>>>>> | li
>>>>>>>>>>> | st
>>>>>>>>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main
>>>>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>>>>>>>>>>
>>>>>>>>>> Indeed, I misintepreted the code.
>>>>>>>>>>
>>>>>>>>>> But I think we still have an issue on the deb-src population side:
>>>>>>>>>>
>>>>>>>>>> # ls build/downloads/deb-src/
>>>>>>>>>> iot2050-debian
>>>>>>>>>>
>>>>>>>>>> And that was with this patch applied.
>>>>>>>>>>
>>>>>>>>>> Jan
>>>>>>>>>
>>>>>>>>> Actually, I checked the case with the following configuration:
>>>>>>>>>
>>>>>>>>> distro: my-debian
>>>>>>>>>
>>>>>>>>> local_conf_header:
>>>>>>>>> nodebian: |
>>>>>>>>>
>>>>>>>>> require conf/distro/debian-bullseye.conf
>>>>>>>>> DISTRO_NAME = "My Debian System"
>>>>>>>>>
>>>>>>>>> So, I simulated renaming distro that is done in meta-iot2050.
>>>>>>>>>
>>>>>>>>> Cached build was OK, "hello" package (that we build from sources)
>>>>>>>>> was
>>>>>>>>>
>>>>>>>>> downloaded at first build to "my-debian" directory:
>>>>>>>>> | $ ls build/downloads/deb-src/my-debian/hello/
>>>>>>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc
>>>>>>>>> | hello_2.10.orig.tar.gz
>>>>>>>>>
>>>>>>>>> and second build picked the files with no problem.
>>>>>>>>>
>>>>>>>>> The same with "raspios-bullseye" target - "hello" sources are
>>>>>>>>> downloaded
>>>>>>>>> to
>>>>>>>>>
>>>>>>>>> the "target" dir:
>>>>>>>>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/
>>>>>>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc
>>>>>>>>> | hello_2.10.orig.tar.gz
>>>>>>>>>
>>>>>>>>> I even tested the case with BASE_REPO_FEATURES += "cache-deb-src"
>>>>>>>>> and
>>>>>>>>> it
>>>>>>>>> was OK (honestly, I don't know who could require cached build with
>>>>>>>>> this
>>>>>>>>> option enabled :) ).
>>>>>>>>>
>>>>>>>>> Anyway, I'll also try to check my changes with meta-iot2050 layer
>>>>>>>>> also
>>>>>>>>> (despite some packages in it are currently not compatible with
>>>>>>>>> BB_NO_NETWORK option)
>>>>>>>>
>>>>>>>> Please then test without the coral packages, they have an own issue
>>>>>>>> /wrt
>>>>>>>> offline builds.
>>>>>>>
>>>>>>> Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works
>>>>>>
>>>>>> ...but it caches to the wrong downloads/deb-src path, no?
>>>>>>
>>>>>> Jan
>>>>>
>>>>> Why wrong?
>>>>>
>>>>> | meta-iot2050/build $ tree -d downloads/deb*
>>>>> | downloads/deb
>>>>> | ├── debian-bullseye
>>>>> | └── iot2050-debian
>>>>> | downloads/deb-src
>>>>> | └── iot2050-debian
>>>>> |
>>>>> | ├── openssl
>>>>> | └── swig
>>>>>
>>>>> `do_apt_fetch` works with `${DISTRO}` and downloads sources to the
>>>>> corresponding directory.
>>>>
>>>> deb-src fetching still lacks base-distro normalization, like deb also
> has:
>>> Saying "normalization", do you mean the following: if ${DISTRO} is just
>>> renaming (e.g. "iot2050-debian") everything should be downloaded to and
>>> picked up from the directory named "debian-bullseye"?
>>
>> Yes, just like you did for binary debs: Reduce the DISTRO to its
>> BASE_DISTRO, then use that as key for the cache. The current situation
>> is inconsistent and non-optimal, even if working.
>>
>> Jan
>
> I've just pushed v4 of patchset that still doesn't include this wish. For now,
> I would apply it to the isar/next to have cached cross build working.
>
> We have some plans to improve base-apt in general (there were some PoC patches
> on list), so improvements you mention could be done a bit later - for now I
> need to switch to some other tasks.
>
The problem is in meta/classes/rootfs.bbclass. It lacks normalization of
ROOTFS_DISTRO to some "ROOTFS_BASE_DISTRO". That also explains why you
still saw same packages under deb/iot2050-debian - you ran the final
rootfs step which I didn't (due to earlier errors).
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2022-11-28 11:45 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-25 6:42 [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Uladzimir Bely
2022-11-25 6:42 ` [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs Uladzimir Bely
2022-11-25 7:42 ` Jan Kiszka
2022-11-25 7:56 ` Uladzimir Bely
2022-11-25 14:11 ` Jan Kiszka
2022-11-25 14:48 ` Uladzimir Bely
2022-11-25 15:40 ` Jan Kiszka
2022-11-26 5:30 ` Uladzimir Bely
2022-11-27 19:05 ` Jan Kiszka
2022-11-28 6:05 ` Uladzimir Bely
2022-11-28 6:32 ` Jan Kiszka
2022-11-28 6:41 ` Uladzimir Bely
2022-11-28 6:51 ` Jan Kiszka
2022-11-28 11:12 ` Uladzimir Bely
2022-11-28 11:45 ` Jan Kiszka
2022-11-25 6:42 ` [PATCH v3 2/3] testsuite: Run signed repro test in cross mode Uladzimir Bely
2022-11-25 6:42 ` [PATCH v3 3/3] testsuite: Replace bananapi with rpi-arm-v7 in signed repro tests Uladzimir Bely
2022-11-25 7:37 ` [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Jan Kiszka
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox