[PATCH 0/2] Fix cross-build from base-apt when custom DISTRO name used

[PATCH 0/2] Fix cross-build from base-apt when custom DISTRO name used

[Uladzimir Bely]

This patchset is mostly intended for downstreams that prefer
to rename DISTRO variable (like `meat-iot2050`) and use
ISAR_CROSS_COMPILE = "1".

This fixes only the case when renamed DISTRO still uses the same
repositories as HOST_DISTRO. E.g, "raspberry pi" still can't be
rebuilt from local `base-apt` while host distro (e.g., Debian)
and target one (Raspbian / RaspiOS) use different repositories
and it happens that packages with the same name and version in
both repos have different size/md5sum and can't be placed to
`base-apt` repo simultaneously.

Also, testsute updated to cover cross-build case.

Uladzimir Bely (2):
  Populate base-apt from both DISTRO and HOST_DISTRO download dirs
  testsuite: Run signed repro test in cross mode

 meta/recipes-devtools/base-apt/base-apt.bb | 11 ++++++++---
 testsuite/citest.py                        |  2 +-
 2 files changed, 9 insertions(+), 4 deletions(-)

-- 
2.20.1

[PATCH 1/2] Populate base-apt from both DISTRO and HOST_DISTRO download dirs

[Uladzimir Bely]

Downstreams may want to change DISTRO to some custom value.
When cross-building, this leads to downloading packages for host
and target distros to the different download subdirs.

While base-apt is populated only from DISTRO download subdir,
second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to
missing packages in base-apt.

Fix it here by populating base-apt from both download subdirs.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/recipes-devtools/base-apt/base-apt.bb | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index 506a28ff..e0d47625 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -13,7 +13,9 @@ KEYFILES ?= ""
 BASE_REPO_FEATURES ?= ""
 
 populate_base_apt() {
-    find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do
+    distro="${1}"
+
+    find "${DEBDIR}"/"${distro}" -name '*\.deb' | while read package; do
         # NOTE: due to packages stored by reprepro are not modified, we can
         # use search by filename to check if package is already in repo. In
         # addition, md5sums are compared to ensure that the package is the
@@ -38,7 +40,7 @@ populate_base_apt() {
             "${package}"
     done
 
-    find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read package; do
+    find "${DEBSRCDIR}"/"${distro}" -name '*\.dsc' | while read package; do
         repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
             "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
             "${BASE_DISTRO_CODENAME}" \
@@ -66,7 +68,10 @@ repo() {
                 "Try it without cross-build."
     fi
 
-    populate_base_apt
+    populate_base_apt "${DISTRO}"
+    if [ '${DISTRO}' != '${HOST_DISTRO}' ]; then
+        populate_base_apt "${HOST_DISTRO}"
+    fi
     repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
         "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
 }
-- 
2.20.1

Re: [PATCH 1/2] Populate base-apt from both DISTRO and HOST_DISTRO download dirs

[Jan Kiszka]

On 24.11.22 07:42, Uladzimir Bely wrote:
> Downstreams may want to change DISTRO to some custom value.
> When cross-building, this leads to downloading packages for host
> and target distros to the different download subdirs.
> 
> While base-apt is populated only from DISTRO download subdir,
> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to
> missing packages in base-apt.
> 
> Fix it here by populating base-apt from both download subdirs.
> 
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  meta/recipes-devtools/base-apt/base-apt.bb | 11 ++++++++---
>  1 file changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
> index 506a28ff..e0d47625 100644
> --- a/meta/recipes-devtools/base-apt/base-apt.bb
> +++ b/meta/recipes-devtools/base-apt/base-apt.bb
> @@ -13,7 +13,9 @@ KEYFILES ?= ""
>  BASE_REPO_FEATURES ?= ""
>  
>  populate_base_apt() {
> -    find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do
> +    distro="${1}"
> +
> +    find "${DEBDIR}"/"${distro}" -name '*\.deb' | while read package; do
>          # NOTE: due to packages stored by reprepro are not modified, we can
>          # use search by filename to check if package is already in repo. In
>          # addition, md5sums are compared to ensure that the package is the
> @@ -38,7 +40,7 @@ populate_base_apt() {
>              "${package}"
>      done
>  
> -    find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read package; do
> +    find "${DEBSRCDIR}"/"${distro}" -name '*\.dsc' | while read package; do
>          repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
>              "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
>              "${BASE_DISTRO_CODENAME}" \
> @@ -66,7 +68,10 @@ repo() {
>                  "Try it without cross-build."
>      fi
>  
> -    populate_base_apt
> +    populate_base_apt "${DISTRO}"
> +    if [ '${DISTRO}' != '${HOST_DISTRO}' ]; then
> +        populate_base_apt "${HOST_DISTRO}"
> +    fi

This assumes BASE_DISTRO(DISTRO) == HOST_DISTRO - well...

Let's do this properly, split base-apt into base-apt-{target,host}, just
like isar-bootstrap. On top, we can then switch from DISTRO/HOST_DISTRO
to their corresponding BASE_DISTROs, though that will be more
complicated as there is no simply algorithm for BASE_DISTRO(HOST_DISTRO)
- unless we assume downstream is not overloading HOST_DISTRO with
anything custom.

Jan

>      repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
>          "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
>  }

-- 
Siemens AG, Technology
Competence Center Embedded Linux

[PATCH 2/2] testsuite: Run signed repro test in cross mode

[Uladzimir Bely]

Both signed and unsigned tests are done in non-cross mode.
Changing CROSS_COMPILE to "1" for one of them allows to cover
cross-build from local `base-apt` repo without increasing
build time.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/citest.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 2dc78015..cf9139f8 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -32,7 +32,7 @@ class ReproTest(CIBaseTest):
 
         self.init()
         try:
-            self.perform_repro_test(targets, signed=True)
+            self.perform_repro_test(targets, signed=True, cross=True)
         finally:
             self.move_in_build_dir('tmp', 'tmp_repro_signed')
 
-- 
2.20.1