[PATCH v7 0/6] use xz and gzip on host (outside chroot)

[PATCH v7 0/6] use xz and gzip on host (outside chroot)

[Felix Moessbauer]

Changes since v6:

- reorder tests in testsuite to not break vm_boot tests
- rebase against current next

Changes since v5:

- before changing WIC deploy logic, revert edc10d93 (p3)
- only reploy requested files (p4)

Changes since v4:

- add option WIC_DEPLOY_PARTITIONS = "0" to control if individual wic partitions are deployed (p3)
  Note, that the default set to disabled is important.
  This fixes the huge disk consumption when running the CI with WIC tests
- add test for WIC_DEPLOY_PARTITIONS logic (p4)
- add logic to handle cross-imaging when distro_arch == host_arch (similar to buildchroot-(target|host)) (p1)

Please note, that the additions around WIC_DEPLOY_PARTITIONS are standalone patches which are only sent as part of this series to fix the disk space blow up in CI.

Changes since v3:

- add test for CROSS=1 wic image including compression
- integrate patch 2 into this series instead of standalone.

Please note, this series supersedes the "add wic.xz image to default qemuarm64 target" series but does only requires little more disk space for the CI tests (compared to ISAR-next).
By that, the CI can still be executed on standard machines.

Changes since v1/v2:

instead of switching the compression logic, this patch adds support to run the whole imager in the buildchroot-host.

Best regards,
Felix

Felix Moessbauer (6):
  do not crossbuild SDK for host-arch eq dist-arch
  run imager in buildchroot-host on cross
  Revert "wic: move out all files ending on "direct*""
  add option to control deploy of raw wic partitions
  add test for wic partition deploy logic
  add wic.xz image to qemuarm64-bookworm

 RECIPE-API-CHANGELOG.md                       |  9 +++++++
 doc/user_manual.md                            | 11 +++++++++
 meta-isar/conf/local.conf.sample              |  3 +++
 .../conf/multiconfig/qemuarm64-bookworm.conf  |  3 +++
 meta/classes/image-tools-extension.bbclass    |  3 ---
 meta/classes/imagetypes_wic.bbclass           | 16 +++++++++----
 meta/classes/sdk.bbclass                      | 14 +++++++++--
 testsuite/cibase.py                           | 10 ++++++++
 testsuite/cibuilder.py                        |  5 +++-
 testsuite/citest.py                           | 24 +++++++++++++++++++
 10 files changed, 88 insertions(+), 10 deletions(-)

-- 
2.30.2

[PATCH v7 1/6] do not crossbuild SDK for host-arch eq dist-arch

[Felix Moessbauer]

This patch adds logic to detect when ISAR_CROSS_COMPILE
is set, but host-arch equals dist-arch.
In this case, we do not add the crossbuild-essential package,
but instead add the build-essential one.

To implement this, we transfer the existing logic into a
anonymous python function to make it better readable.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/sdk.bbclass | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sdk.bbclass b/meta/classes/sdk.bbclass
index 32ee2de7..f42a2ba1 100644
--- a/meta/classes/sdk.bbclass
+++ b/meta/classes/sdk.bbclass
@@ -41,8 +41,18 @@ SDK_PREINSTALL += " \
     devscripts \
     equivs"
 
-TOOLCHAIN = "${@'crossbuild-essential-${DISTRO_ARCH}' if d.getVar('ISAR_CROSS_COMPILE', True) == '1' else 'build-essential'}"
-TOOLCHAIN_append_compat-arch = " crossbuild-essential-${COMPAT_DISTRO_ARCH}"
+# Choose the correct toolchain: cross or native
+python __anonymous() {
+    mode = d.getVar('ISAR_CROSS_COMPILE', True)
+    distro_arch = d.getVar('DISTRO_ARCH')
+    if mode == "0" or d.getVar('HOST_ARCH') ==  distro_arch:
+        toolchain = "build-essential"
+    else:
+        toolchain = "crossbuild-essential-" + distro_arch
+    if d.getVar('ISAR_ENABLE_COMPAT_ARCH', True) == "1":
+        toolchain += " crossbuild-essential-" + d.getVar('COMPAT_DISTRO_ARCH')
+    d.setVar('TOOLCHAIN', toolchain)
+}
 
 # rootfs/image overrides for the SDK
 ROOTFS_ARCH_class-sdk = "${HOST_ARCH}"
-- 
2.30.2

[PATCH v7 2/6] run imager in buildchroot-host on cross

[Felix Moessbauer]

When globally enabling ISAR_CROSS_COMPILE, the imager and compression
tasks run in the host buildchroot as well (instead of the target
buildchroot).

Similar to the cross support for DPKG recipes, this can be toggled
on a per-image basis by setting ISAR_CROSS_COMPILE in the image recipe.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 doc/user_manual.md                         | 11 +++++++++++
 meta/classes/image-tools-extension.bbclass |  3 ---
 2 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 5b2387d8..f38a133e 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -986,6 +986,17 @@ built for the compat arch need to be tagged individually by setting
 `PACKAGE_ARCH = "${COMPAT_DISTRO_ARCH}"` in the package recipe. Non-tagged
 packages will continue to be built for the primary target architecture.
 
+### Cross Support for Imagers
+
+If `ISAR_CROSS_COMPILE = "1"`, the imager and optional compression tasks
+run in the host buildchroot instead of the target buildchroot.
+This gives a significant speedup when compressing the generated image,
+as the compression is not emulated.
+
+In case your setup does not support cross-imaging, you can disable this
+just for the particular image by adding `ISAR_CROSS_COMPILE = "0"` to your
+image recipe.
+
 
 ## Examining and debugging package generation inside their buildchroot
 
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index b9968139..e13d4a3f 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -5,9 +5,6 @@
 #
 # This file extends the image.bbclass to supply tools for futher imager functions
 
-# Imager are expected to run natively, thus will use the target buildchroot.
-ISAR_CROSS_COMPILE = "0"
-
 inherit buildchroot
 
 IMAGER_INSTALL ??= ""
-- 
2.30.2

[PATCH v7 3/6] Revert "wic: move out all files ending on "direct*""

[Felix Moessbauer]

This reverts commit edc10d9361e79f36c8ea5488b1dab4c468213a8d.
---
 meta/classes/imagetypes_wic.bbclass | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/meta/classes/imagetypes_wic.bbclass b/meta/classes/imagetypes_wic.bbclass
index 61a74d4a..81788faf 100644
--- a/meta/classes/imagetypes_wic.bbclass
+++ b/meta/classes/imagetypes_wic.bbclass
@@ -204,11 +204,10 @@ generate_wic_image() {
               ${WIC_CREATE_EXTRA_ARGS}
 
     sudo chown -R $(stat -c "%U" ${LAYERDIR_core}) ${LAYERDIR_core} ${LAYERDIR_isar} ${SCRIPTSDIR} || true
+    WIC_DIRECT=$(ls -t -1 ${BUILDCHROOT_DIR}/$WICTMP/${IMAGE_FULLNAME}.wic/*.direct | head -1)
     sudo chown -R $(id -u):$(id -g) ${BUILDCHROOT_DIR}/${WICTMP}
-    find ${BUILDCHROOT_DIR}/${WICTMP} -type f -name "*.direct*" | while read f; do
-        suffix=$(basename $f | sed 's/\(.*\)\(\.direct\)\(.*\)/\3/')
-        mv -f ${f} "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic${suffix}"
-    done
+    mv -f ${WIC_DIRECT} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic
+    mv -f ${WIC_DIRECT}.bmap ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.bmap
     rm -rf ${BUILDCHROOT_DIR}/${WICTMP}
     rm -rf ${IMAGE_ROOTFS}/../pseudo
 }
-- 
2.30.2

[PATCH v7 4/6] add option to control deploy of raw wic partitions

[Felix Moessbauer]

This patch adds the WIC_DEPLOY_PARTITIONS option to control
if the raw partition files are deployed by wic.

We also set the default to not deploy to be closer to OE and to
optimize for the average case.
In addition, this significantly reduces the disk size required to
run the CI.

An API changelog entry is added.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 RECIPE-API-CHANGELOG.md             | 9 +++++++++
 meta-isar/conf/local.conf.sample    | 3 +++
 meta/classes/imagetypes_wic.bbclass | 9 +++++++++
 3 files changed, 21 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 621d110e..a5f5abee 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -419,3 +419,12 @@ packages and will be lost after a given build session finishes.
 
 Any package build requirements for the rootfs should be satisfied in the
 Debian way via package dependencies.
+
+### Individual WIC partitions are no longer automatically deployed
+
+We used to copy all temporary WIC files, like the partitions, into the deploy directory.
+That was intended actually only for compressed wic images where wic itself would do the compression.
+It was never intended to also deploy those partitions, so that will also not be done (automatically) anymore.
+To explicitly deploy the individual partition files (e.g. for swupdate), set `WIC_DEPLOY_PARTITIONS = "1"`.
+
+For compressed wic images `IMAGE_FSTYPES` should simply be extended with a compressed wic format, like "wic.xz".
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index ce7b58ee..57d06202 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -246,6 +246,9 @@ USER_isar[flags] += "clear-text-password"
 # Uncomment the below line to debug WIC.
 # WIC_CREATE_EXTRA_ARGS += "-D"
 
+# Uncomment this to also deploy each wic partition as separate file (e.g. for swupdate)
+#WIC_DEPLOY_PARTITIONS = "1"
+
 # Uncomment this to use ccache for custom packages
 #USE_CCACHE = "1"
 # Uncomment and set own top level ccache directory to share between builds
diff --git a/meta/classes/imagetypes_wic.bbclass b/meta/classes/imagetypes_wic.bbclass
index 81788faf..c0f3429a 100644
--- a/meta/classes/imagetypes_wic.bbclass
+++ b/meta/classes/imagetypes_wic.bbclass
@@ -98,6 +98,7 @@ RECIPE_SYSROOT_NATIVE ?= "/"
 BUILDCHROOT_DIR = "${BUILDCHROOT_TARGET_DIR}"
 
 WIC_CREATE_EXTRA_ARGS ?= ""
+WIC_DEPLOY_PARTITIONS ?= "0"
 
 # taken from OE, do not touch directly
 WICVARS += "\
@@ -208,6 +209,14 @@ generate_wic_image() {
     sudo chown -R $(id -u):$(id -g) ${BUILDCHROOT_DIR}/${WICTMP}
     mv -f ${WIC_DIRECT} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic
     mv -f ${WIC_DIRECT}.bmap ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.bmap
+    # deploy partition files if requested (ending with .p<x>)
+    if [ "${WIC_DEPLOY_PARTITIONS}" -eq "1" ]; then
+        # locate *.direct.p<x> partition files
+        find ${BUILDCHROOT_DIR}/${WICTMP} -type f -regextype sed -regex ".*\.direct.*\.p[0-9]\{1,\}" | while read f; do
+            suffix=$(basename $f | sed 's/.*\.direct\(.*\)/\1/')
+            mv -f ${f} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic${suffix}
+        done
+    fi
     rm -rf ${BUILDCHROOT_DIR}/${WICTMP}
     rm -rf ${IMAGE_ROOTFS}/../pseudo
 }
-- 
2.30.2

[PATCH v7 5/6] add test for wic partition deploy logic

[Felix Moessbauer]

This patch adds tests to check if the wic partition files
are copied to the deploydir if requested.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 testsuite/cibase.py    | 10 ++++++++++
 testsuite/cibuilder.py |  5 ++++-
 testsuite/citest.py    | 23 +++++++++++++++++++++++
 3 files changed, 37 insertions(+), 1 deletion(-)

diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index 722d7bd2..6239b4de 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -17,6 +17,16 @@ class CIBaseTest(CIBuilder):
 
         self.bitbake(targets, **kwargs)
 
+    def perform_wic_partition_test(self, targets, wic_deploy_parts, **kwargs):
+        self.configure(wic_deploy_parts=wic_deploy_parts, **kwargs)
+        self.bitbake(targets, **kwargs)
+
+        partition_files = set(glob.glob(f'{self.build_dir}/tmp/deploy/images/*/*.wic.p1'))
+        if wic_deploy_parts and len(partition_files) == 0:
+            self.fail('Found raw wic partitions in DEPLOY_DIR')
+        if not wic_deploy_parts and len(partition_files) != 0:
+            self.fail('Did not find raw wic partitions in DEPLOY_DIR')
+
     def perform_repro_test(self, targets, signed=False, **kwargs):
         gpg_pub_key = os.path.dirname(__file__) + '/keys/base-apt/test_pub.key'
         gpg_priv_key = os.path.dirname(__file__) + '/keys/base-apt/test_priv.key'
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index dfb0a376..d5c83b22 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -54,7 +54,7 @@ class CIBuilder(Test):
 
     def configure(self, compat_arch=True, cross=None, debsrc_cache=False,
                   container=False, ccache=False, sstate=False, offline=False,
-                  gpg_pub_key=None, **kwargs):
+                  gpg_pub_key=None, wic_deploy_parts=False, **kwargs):
         # write configuration file and set bitbake_args
         # can run multiple times per test case
         self.check_init()
@@ -77,6 +77,7 @@ class CIBuilder(Test):
                       f'  ccache = {ccache}\n'
                       f'  sstate = {sstate}\n'
                       f'  gpg_pub_key = {gpg_pub_key}\n'
+                      f'  wic_deploy_parts = {wic_deploy_parts}\n'
                       f'===================================================')
 
         # determine bitbake_args
@@ -105,6 +106,8 @@ class CIBuilder(Test):
                 f.write('IMAGE_INSTALL_remove = "example-module-${KERNEL_NAME} enable-fsck"\n')
             if gpg_pub_key:
                 f.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
+            if wic_deploy_parts:
+                f.write('WIC_DEPLOY_PARTITIONS = "1"\n')
             if distro_apt_premir:
                 f.write('DISTRO_APT_PREMIRRORS = "%s"\n' % distro_apt_premir)
             if ccache:
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 6c8eb26e..e8cc299e 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -318,3 +318,26 @@ class VmBootTestFull(CIBaseTest):
     def test_amd64_focal(self):
         self.init()
         self.vm_start('amd64','focal')
+
+class WicTest(CIBaseTest):
+
+    """
+    Test creation of wic images
+
+    :avocado: tags=wic,full
+    """
+    def test_wic_nodeploy_partitions(self):
+        targets = ['mc:qemuarm64-bookworm:isar-image-base']
+
+        self.init()
+        self.delete_from_build_dir('tmp')
+        self.perform_wic_partition_test(targets,
+            wic_deploy_parts=False, debsrc_cache=True, compat_arch=False)
+
+    def test_wic_deploy_partitions(self):
+        targets = ['mc:qemuarm64-bookworm:isar-image-base']
+
+        self.init()
+        # reuse artifacts
+        self.perform_wic_partition_test(targets,
+            wic_deploy_parts=True, debsrc_cache=True, compat_arch=False)
-- 
2.30.2

[PATCH v7 6/6] add wic.xz image to qemuarm64-bookworm

[Felix Moessbauer]

This patch adds generation of a wic.xz image for the mc:qemuarm64-bookworm target.
By that, the cross and non-cross imaging is automatically tested in CI.
Further, we add a qemuarm64-bookworm target to the cross build test.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta-isar/conf/multiconfig/qemuarm64-bookworm.conf | 3 +++
 testsuite/citest.py                                | 1 +
 2 files changed, 4 insertions(+)

diff --git a/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf b/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
index c6b92d2e..a04c9d1f 100644
--- a/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
+++ b/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
@@ -4,3 +4,6 @@
 
 MACHINE ?= "qemuarm64"
 DISTRO ?= "debian-bookworm"
+
+IMAGE_FSTYPES += "wic.xz"
+IMAGER_INSTALL += "${GRUB_BOOTLOADER_INSTALL}"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index e8cc299e..a0e7e8b9 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -72,6 +72,7 @@ class CrossTest(CIBaseTest):
             'mc:qemuarm-buster:isar-image-base',
             'mc:qemuarm-bullseye:isar-image-base',
             'mc:qemuarm64-bullseye:isar-image-base',
+            'mc:qemuarm64-bookworm:isar-image-base',
             'mc:de0-nano-soc-bullseye:isar-image-base',
             'mc:stm32mp15x-buster:isar-image-base'
                   ]
-- 
2.30.2

Re: [PATCH v7 0/6] use xz and gzip on host (outside chroot)

[Anton Mikanovich]

27.09.2022 12:51, Felix Moessbauer wrote:
> Changes since v6:
>
> - reorder tests in testsuite to not break vm_boot tests
> - rebase against current next
>
> Changes since v5:
>
> - before changing WIC deploy logic, revert edc10d93 (p3)
> - only reploy requested files (p4)
>
> Changes since v4:
>
> - add option WIC_DEPLOY_PARTITIONS = "0" to control if individual wic partitions are deployed (p3)
>    Note, that the default set to disabled is important.
>    This fixes the huge disk consumption when running the CI with WIC tests
> - add test for WIC_DEPLOY_PARTITIONS logic (p4)
> - add logic to handle cross-imaging when distro_arch == host_arch (similar to buildchroot-(target|host)) (p1)
>
> Please note, that the additions around WIC_DEPLOY_PARTITIONS are standalone patches which are only sent as part of this series to fix the disk space blow up in CI.
>
> Changes since v3:
>
> - add test for CROSS=1 wic image including compression
> - integrate patch 2 into this series instead of standalone.
>
> Please note, this series supersedes the "add wic.xz image to default qemuarm64 target" series but does only requires little more disk space for the CI tests (compared to ISAR-next).
> By that, the CI can still be executed on standard machines.
>
> Changes since v1/v2:
>
> instead of switching the compression logic, this patch adds support to run the whole imager in the buildchroot-host.
>
> Best regards,
> Felix
>
> Felix Moessbauer (6):
>    do not crossbuild SDK for host-arch eq dist-arch
>    run imager in buildchroot-host on cross
>    Revert "wic: move out all files ending on "direct*""
>    add option to control deploy of raw wic partitions
>    add test for wic partition deploy logic
>    add wic.xz image to qemuarm64-bookworm
>
>   RECIPE-API-CHANGELOG.md                       |  9 +++++++
>   doc/user_manual.md                            | 11 +++++++++
>   meta-isar/conf/local.conf.sample              |  3 +++
>   .../conf/multiconfig/qemuarm64-bookworm.conf  |  3 +++
>   meta/classes/image-tools-extension.bbclass    |  3 ---
>   meta/classes/imagetypes_wic.bbclass           | 16 +++++++++----
>   meta/classes/sdk.bbclass                      | 14 +++++++++--
>   testsuite/cibase.py                           | 10 ++++++++
>   testsuite/cibuilder.py                        |  5 +++-
>   testsuite/citest.py                           | 24 +++++++++++++++++++
>   10 files changed, 88 insertions(+), 10 deletions(-)
>
Applied to next, thanks.

Re: [PATCH v7 0/6] use xz and gzip on host (outside chroot)

[Uladzimir Bely]

In the email from Friday, 7 October 2022 21:53:39 +03 user Anton Mikanovich 
wrote:
> 27.09.2022 12:51, Felix Moessbauer wrote:
> > Changes since v6:
> > 
> > - reorder tests in testsuite to not break vm_boot tests
> > - rebase against current next
> > 
> > Changes since v5:
> > 
> > - before changing WIC deploy logic, revert edc10d93 (p3)
> > - only reploy requested files (p4)
> > 
> > Changes since v4:
> > 
> > - add option WIC_DEPLOY_PARTITIONS = "0" to control if individual wic
> > partitions are deployed (p3)> 
> >    Note, that the default set to disabled is important.
> >    This fixes the huge disk consumption when running the CI with WIC tests
> > 
> > - add test for WIC_DEPLOY_PARTITIONS logic (p4)
> > - add logic to handle cross-imaging when distro_arch == host_arch (similar
> > to buildchroot-(target|host)) (p1)
> > 
> > Please note, that the additions around WIC_DEPLOY_PARTITIONS are
> > standalone patches which are only sent as part of this series to fix the
> > disk space blow up in CI.
> > 
> > Changes since v3:
> > 
> > - add test for CROSS=1 wic image including compression
> > - integrate patch 2 into this series instead of standalone.
> > 
> > Please note, this series supersedes the "add wic.xz image to default
> > qemuarm64 target" series but does only requires little more disk space
> > for the CI tests (compared to ISAR-next). By that, the CI can still be
> > executed on standard machines.
> > 
> > Changes since v1/v2:
> > 
> > instead of switching the compression logic, this patch adds support to run
> > the whole imager in the buildchroot-host.
> > 
> > Best regards,
> > Felix
> > 
> > Felix Moessbauer (6):
> >    do not crossbuild SDK for host-arch eq dist-arch
> >    run imager in buildchroot-host on cross
> >    Revert "wic: move out all files ending on "direct*""
> >    add option to control deploy of raw wic partitions
> >    add test for wic partition deploy logic
> >    add wic.xz image to qemuarm64-bookworm
> >   
> >   RECIPE-API-CHANGELOG.md                       |  9 +++++++
> >   doc/user_manual.md                            | 11 +++++++++
> >   meta-isar/conf/local.conf.sample              |  3 +++
> >   .../conf/multiconfig/qemuarm64-bookworm.conf  |  3 +++
> >   meta/classes/image-tools-extension.bbclass    |  3 ---
> >   meta/classes/imagetypes_wic.bbclass           | 16 +++++++++----
> >   meta/classes/sdk.bbclass                      | 14 +++++++++--
> >   testsuite/cibase.py                           | 10 ++++++++
> >   testsuite/cibuilder.py                        |  5 +++-
> >   testsuite/citest.py                           | 24 +++++++++++++++++++
> >   10 files changed, 88 insertions(+), 10 deletions(-)
> 
> Applied to next, thanks.

Now, with "[PATCH v7 5/6] add test for wic partition deploy logic" we lost all 
"nocross" CI logs from "tmp/work" (the test deletes "tmp" directory).

While our tests are still sometimes depend on each other, the easiest fix 
would be placing "wic" tests before "nocross" ones.

-- 
Uladzimir Bely

RE: [PATCH v7 0/6] use xz and gzip on host (outside chroot)

[MOESSBAUER, FELIX JONATHAN]

> -----Original Message-----
> From: Uladzimir Bely <ubely@ilbers.de>
> Sent: Wednesday, October 12, 2022 3:41 PM
> To: Moessbauer, Felix Jonathan (T CED INW-CN)
> <felix.moessbauer@siemens.com>; isar-users@googlegroups.com
> Cc: Schild, Henning (T CED SES-DE) <henning.schild@siemens.com>; Kiszka, Jan
> (T CED) <jan.kiszka@siemens.com>; ibr@ilbers.de; Schmidt, Adriaan (T CED SES-
> DE) <adriaan.schmidt@siemens.com>; Anton Mikanovich <amikan@ilbers.de>
> Subject: Re: [PATCH v7 0/6] use xz and gzip on host (outside chroot)
> 
> In the email from Friday, 7 October 2022 21:53:39 +03 user Anton Mikanovich
> wrote:
> > 27.09.2022 12:51, Felix Moessbauer wrote:
> > > Changes since v6:
> > >
> > > - reorder tests in testsuite to not break vm_boot tests
> > > - rebase against current next
> > >
> > > Changes since v5:
> > >
> > > - before changing WIC deploy logic, revert edc10d93 (p3)
> > > - only reploy requested files (p4)
> > >
> > > Changes since v4:
> > >
> > > - add option WIC_DEPLOY_PARTITIONS = "0" to control if individual
> > > wic partitions are deployed (p3)>
> > >    Note, that the default set to disabled is important.
> > >    This fixes the huge disk consumption when running the CI with WIC
> > > tests
> > >
> > > - add test for WIC_DEPLOY_PARTITIONS logic (p4)
> > > - add logic to handle cross-imaging when distro_arch == host_arch
> > > (similar to buildchroot-(target|host)) (p1)
> > >
> > > Please note, that the additions around WIC_DEPLOY_PARTITIONS are
> > > standalone patches which are only sent as part of this series to fix
> > > the disk space blow up in CI.
> > >
> > > Changes since v3:
> > >
> > > - add test for CROSS=1 wic image including compression
> > > - integrate patch 2 into this series instead of standalone.
> > >
> > > Please note, this series supersedes the "add wic.xz image to default
> > > qemuarm64 target" series but does only requires little more disk
> > > space for the CI tests (compared to ISAR-next). By that, the CI can
> > > still be executed on standard machines.
> > >
> > > Changes since v1/v2:
> > >
> > > instead of switching the compression logic, this patch adds support
> > > to run the whole imager in the buildchroot-host.
> > >
> > > Best regards,
> > > Felix
> > >
> > > Felix Moessbauer (6):
> > >    do not crossbuild SDK for host-arch eq dist-arch
> > >    run imager in buildchroot-host on cross
> > >    Revert "wic: move out all files ending on "direct*""
> > >    add option to control deploy of raw wic partitions
> > >    add test for wic partition deploy logic
> > >    add wic.xz image to qemuarm64-bookworm
> > >
> > >   RECIPE-API-CHANGELOG.md                       |  9 +++++++
> > >   doc/user_manual.md                            | 11 +++++++++
> > >   meta-isar/conf/local.conf.sample              |  3 +++
> > >   .../conf/multiconfig/qemuarm64-bookworm.conf  |  3 +++
> > >   meta/classes/image-tools-extension.bbclass    |  3 ---
> > >   meta/classes/imagetypes_wic.bbclass           | 16 +++++++++----
> > >   meta/classes/sdk.bbclass                      | 14 +++++++++--
> > >   testsuite/cibase.py                           | 10 ++++++++
> > >   testsuite/cibuilder.py                        |  5 +++-
> > >   testsuite/citest.py                           | 24 +++++++++++++++++++
> > >   10 files changed, 88 insertions(+), 10 deletions(-)
> >
> > Applied to next, thanks.
> 
> Now, with "[PATCH v7 5/6] add test for wic partition deploy logic" we lost all
> "nocross" CI logs from "tmp/work" (the test deletes "tmp" directory).
> 
> While our tests are still sometimes depend on each other, the easiest fix would
> be placing "wic" tests before "nocross" ones.

Hi Uladzimir,

Could you please try to implement this minimal patch.
I have a really hard time to run the full testsuite behind our proxies.

The whole CI currently has a lot of inter-test dependencies which make it really hard to add tests for new features.
Ideally, the test order should not matter at all.
In case the build-time is an issue, we should better rely on the sstate cache than on the artifacts of previous tests.

Felix


> 
> --
> Uladzimir Bely
> 
>