[PATCH v2 00/17] Various improvements to the testsuite

[PATCH v2 00/17] Various improvements to the testsuite

['Felix Moessbauer' via isar-users]

This series aims at simplifying and speeding up the testsuite execution.
This should enable contributors to actually run it and test their patches.

As pointed out in [1], the testsuite currently tests various features over
and over again, while others are not tested at all. We try to improve this
by focusing on "feature" tests, whereby each test case checks a set of
features instead of a whole image. We further identified long running tasks
(like the kernel builds) and move them to a dedicated test.

The execution time reduction varies greatly depending on the hardware,
network and sstate usage. When applying these changes on recently added
patches, the time was often decreased by more than 50% on "fast".

Note, that more refactorings of the testsuite are needed - and planned,
but this series should already address the biggest pain points.

[1] https://groups.google.com/g/isar-users/c/uZBTIHzLp8Q/m/8k0L1vfMDAAJ

Changes since v1:

- continued work on testsuite refactoring, this supersedes:
  - testsuite: fix SignatureTest by avoiding absolute path in bblayers
  - testsuite: add rootfs target for rootfs only tests
  - Enable SState cache in GitLab CI
  - testsuite: fix typo in log message in perform_signature_lint
- completed exclusion / refactoring of db2m sbuild chroot
- GitLab CI: increased after script timeout to give sstate upload more time
- GitLab CI: do not upload sstate artifacts of images (as almost no reuse)
- do not add SignatureTest to fast tag (as otherwise VM tests break...)
- rebuild container test images in VM test to not depend on test order
- rebased onto next

As always, the patches are written in a linear way, so only parts of them
can already be integrated.

That's a wrap for this year, isar developers!
See you next year!

Best regards,
Felix Moessbauer
Siemens AG

Felix Moessbauer (17):
  testsuite: move targets with custom kernel to separate test
  testsuite: enable ccache on kernel tests
  testsuite: make prebuilt container a feature test
  testsuite: make compat test standalone test
  testsuite: handle IMAGE_INSTALL solely in cibuilder.py
  testsuite: limit cross_debsrc test to subset of packages
  testsuite: forward SSTATE_MIRRORS into CI env on sstate
  ci: add support for sstate cache
  testsuite: make test_cross_deps more specific
  testsuite: fix typo in log message in perform_signature_lint
  testsuite: fix SignatureTest by avoiding absolute path in bblayers
  testsuite: use more recent distros in SignatureTest
  testsuite: make SignatureTest idempotent
  testsuite: add rootfs target for rootfs only tests
  testsuite: refactor sbom tests to avoid overhead
  testsuite: make sbuild-flavor test standalone
  testsuite: skip VM tests if images are not available

 .gitlab-ci.yml                                |  36 ++++-
 meta-test/conf/bblayers.conf.sample           |   8 +-
 meta-test/conf/local.conf.sample              |  10 --
 .../hello-isar/hello-isar.bbappend            |   3 +
 .../recipes-app/libhello/libhello.bbappend    |   5 +
 .../recipes-core/images/isar-image-ci.bb      |   2 -
 .../recipes-core/images/isar-rootfs-ci.bb     |  17 +++
 testsuite/cibase.py                           |  34 ++++-
 testsuite/cibuilder.py                        |  49 ++++++-
 testsuite/citest.py                           | 131 ++++++++++++++++--
 10 files changed, 255 insertions(+), 40 deletions(-)
 create mode 100644 meta-test/recipes-app/libhello/libhello.bbappend
 create mode 100644 meta-test/recipes-core/images/isar-rootfs-ci.bb

-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-1-felix.moessbauer%40siemens.com.

[PATCH v2 01/17] testsuite: move targets with custom kernel to separate test

['Felix Moessbauer' via isar-users]

By moving these tests out of the cross test (which is a fast test), we
significantly reduce the test time on fast.

The tests are now placed in the KernelTests class in a dedicated test,
suitable for applying further optimizations (separate commit).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 testsuite/citest.py | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 76a3df69..cb63a8fe 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -190,7 +190,8 @@ class InstallerTest(CIBaseTest):
 class CrossTest(CIBaseTest):
 
     """
-    Start cross build for the defined set of configurations
+    Start cross build for the defined set of configurations.
+    As this is a fast test, ensure to not depend on custom kernels.
 
     :avocado: tags=cross,fast
     """
@@ -199,12 +200,9 @@ class CrossTest(CIBaseTest):
         targets = [
             'mc:qemuarm-buster:isar-image-ci',
             'mc:qemuarm-bullseye:isar-image-ci',
-            'mc:de0-nano-soc-bullseye:isar-image-base',
-            'mc:stm32mp15x-bullseye:isar-image-base',
             'mc:qemuarm-bookworm:isar-image-ci',
             'mc:qemuarm64-focal:isar-image-base',
             'mc:nanopi-neo-efi-bookworm:isar-image-base',
-            'mc:phyboard-mira-bookworm:isar-image-base',
         ]
 
         self.init()
@@ -273,6 +271,17 @@ class KernelTests(CIBaseTest):
     :avocado: tags=kernel,full
     """
 
+    def test_kernel_cross(self):
+        """Targets that build a custom kernel"""
+        targets = [
+            'mc:de0-nano-soc-bullseye:isar-image-base',
+            'mc:stm32mp15x-bullseye:isar-image-base',
+            'mc:phyboard-mira-bookworm:isar-image-base',
+        ]
+
+        self.init()
+        self.perform_build_test(targets)
+
     def test_per_kernel(self):
         """Test per-kernel recipe variants for external kernel modules."""
 
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-2-felix.moessbauer%40siemens.com.

[PATCH v2 02/17] testsuite: enable ccache on kernel tests

['Felix Moessbauer' via isar-users]

The kernels are anyways usually built with ccache enabled to reduce the
buildtime. By enabling it in the test as well, we can significantly
bring down test time on repeated builds.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 testsuite/citest.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index cb63a8fe..cba3c008 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -280,7 +280,7 @@ class KernelTests(CIBaseTest):
         ]
 
         self.init()
-        self.perform_build_test(targets)
+        self.perform_build_test(targets, ccache=True)
 
     def test_per_kernel(self):
         """Test per-kernel recipe variants for external kernel modules."""
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-3-felix.moessbauer%40siemens.com.

[PATCH v2 03/17] testsuite: make prebuilt container a feature test

['Felix Moessbauer' via isar-users]

This reworks how we handle feature tests: Instead of binding a feature
to a target via appends in isar-image-ci, we implement the feature by
using the test setup function. By that, we have fine grained control over
where to test a feature and can avoid testing the same feature over and
over again. This leads to a much cleaner architecture and faster test
execution.

We start implementing this approach with the prebuilt container test.
As the images with containers are currently re-used in the
VmBootTestFull, these tests might break depending on the test execution
order. To fix this, we rebuild the needed images in the VM test itself,
which takes less than a minute when running with sstate cache.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 .../recipes-core/images/isar-image-ci.bb      |  2 --
 testsuite/citest.py                           | 25 +++++++++++++++++++
 2 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb
index 7f2b404d..58aa5738 100644
--- a/meta-test/recipes-core/images/isar-image-ci.bb
+++ b/meta-test/recipes-core/images/isar-image-ci.bb
@@ -16,7 +16,6 @@ IMAGE_INSTALL += "sshd-regen-keys"
 
 # qemuamd64-bookworm
 WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
-IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
 
 # qemuamd64-bullseye
 IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.zst tar.zst"
@@ -52,4 +51,3 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= " ${SYSTEMD_BOOTLOADER_INSTALL}
 # qemuarm64-bookworm
 IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
 IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= " ${GRUB_BOOTLOADER_INSTALL}"
-IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index cba3c008..eec90c6e 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -265,6 +265,25 @@ class CrossTest(CIBaseTest):
         self.init()
         self.perform_build_test(targets, lines=lines)
 
+class PrebuiltTest(CIBaseTest):
+    """
+    Tests associated with prebuilt artifacts (containers, debs).
+    :avocado: tags=prebuilt,fast
+    """
+
+    def test_prebuilt_containers(self):
+        targets = [
+            'mc:qemuamd64-bookworm:isar-image-ci',
+            'mc:qemuarm64-bookworm:isar-image-ci',
+        ]
+
+        self.init()
+        self.perform_build_test(
+            targets,
+            bitbake_cmd='do_rootfs_install',
+            image_install="prebuilt-docker-img prebuilt-podman-img")
+
+
 class KernelTests(CIBaseTest):
     """
     Tests associated with kernel builds and development.
@@ -984,11 +1003,17 @@ class VmBootTestFull(CIBaseTest):
 
     def test_amd64_bookworm_prebuilt_containers(self):
         self.init()
+        self.perform_build_test(
+            ['mc:qemuamd64-bookworm:isar-image-ci'],
+            image_install="prebuilt-docker-img prebuilt-podman-img")
         self.vm_start('amd64', 'bookworm', image='isar-image-ci',
                       script='test_prebuilt_containers.sh')
 
     def test_arm64_bookworm_prebuilt_containers(self):
         self.init()
+        self.perform_build_test(
+            ['mc:qemuarm64-bookworm:isar-image-ci'],
+            image_install="prebuilt-docker-img prebuilt-podman-img")
         self.vm_start('arm64', 'bookworm', image='isar-image-ci',
                       script='test_prebuilt_containers.sh')
 
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-4-felix.moessbauer%40siemens.com.

[PATCH v2 04/17] testsuite: make compat test standalone test

['Felix Moessbauer' via isar-users]

Currently, enabling compat mode during configuration also changes the
set of packages we build. As this is done with appends, some packages
are unconditionally added, despite not being part of the original test
case. We change this by only enabling compat mode when selecting compat.
To actually test the compat feature, we add a simple and fast test that
builds the hello-isar-compat package.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 testsuite/cibuilder.py |  4 ----
 testsuite/citest.py    | 19 +++++++++++++++++++
 2 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 46bfc546..25e3b6cb 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -180,11 +180,7 @@ class CIBuilder(Test):
             if compat_arch:
                 f.write(
                     'ISAR_ENABLE_COMPAT_ARCH:amd64 = "1"\n'
-                    'IMAGE_INSTALL:remove:amd64 = "hello-isar"\n'
-                    'IMAGE_INSTALL:append:amd64 = " hello-isar-compat"\n'
                     'ISAR_ENABLE_COMPAT_ARCH:arm64 = "1"\n'
-                    'IMAGE_INSTALL:remove:arm64 = "hello-isar"\n'
-                    'IMAGE_INSTALL:append:arm64 = " hello-isar-compat"\n'
                 )
             if not cross:
                 f.write('ISAR_CROSS_COMPILE = "0"\n')
diff --git a/testsuite/citest.py b/testsuite/citest.py
index eec90c6e..5ff4a4ed 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -112,6 +112,25 @@ class DevTest(CIBaseTest):
         self.vm_start('arm', 'bookworm', skip_modulecheck=True)
 
 
+class CompatTest(CIBaseTest):
+    """
+    Test compilation of recipes for compat architecture.
+    This also tests a custom sbuild chroot for compat.
+    :avocado: tags=compat,fast
+    """
+
+    def test_compat_recipe(self):
+        targets = [
+            'mc:qemuamd64-bookworm:hello-isar-compat',
+            'mc:qemuarm64-bookworm:hello-isar-compat',
+            'mc:qemuamd64-trixie:hello-isar-compat',
+            'mc:qemuarm64-trixie:hello-isar-compat',
+        ]
+
+        self.init()
+        self.perform_build_test(targets, compat_arch=True)
+
+
 class ReproTest(CIBaseTest):
 
     """
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-5-felix.moessbauer%40siemens.com.

[PATCH v2 05/17] testsuite: handle IMAGE_INSTALL solely in cibuilder.py

['Felix Moessbauer' via isar-users]

By moving the IMAGE_INSTALL parts out of the ci local conf sample, we
make this part reusable across tests. By that, tests can easily
customize this list without either taking it as-is or completely
replacing it. We further get rid of overrides that are not used in the
CI anyways.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta-test/conf/local.conf.sample | 10 ----------
 testsuite/cibuilder.py           | 20 ++++++++++++++++++++
 2 files changed, 20 insertions(+), 10 deletions(-)

diff --git a/meta-test/conf/local.conf.sample b/meta-test/conf/local.conf.sample
index 092d6ba6..862bea47 100644
--- a/meta-test/conf/local.conf.sample
+++ b/meta-test/conf/local.conf.sample
@@ -27,16 +27,6 @@ BB_DISKMON_DIRS = "\
 MIRRORS ?= "git?://salsa\.debian\.org/debian/.* git://github.com/ilbers/BASENAME"
 MIRRORS += "https?://cdn\.kernel\.org/.* https://mirrors.edge.kernel.org/PATH"
 
-# The default list of extra packages
-IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs samefile hello isar-disable-apt-cache cowsay example-prebuilt"
-
-# Container machines don't need example module and enable-fsck.
-IMAGE_INSTALL:remove:container-amd64 = "example-module-${KERNEL_NAME} enable-fsck"
-
-# Machines with secure boot should use signed modules
-IMAGE_INSTALL:remove:qemuamd64-sb = "example-module-${KERNEL_NAME}"
-IMAGE_INSTALL:append:qemuamd64-sb = " example-module-signed-${KERNEL_NAME}"
-
 # Users and groups
 USERS += "root"
 USER_root[password] ??= "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 25e3b6cb..26002ade 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -30,6 +30,18 @@ sys.path.append(os.path.join(os.path.dirname(__file__), '../bitbake/lib'))
 import bb
 
 DEF_VM_TO_SEC = 600
+IMAGE_INSTALL_DEFAULT = [
+    'hello-isar',
+    'example-raw',
+    'example-module-${KERNEL_NAME}',
+    'enable-fsck',
+    'isar-exclude-docs',
+    'samefile',
+    'hello',
+    'isar-disable-apt-cache',
+    'cowsay',
+    'example-prebuilt'
+]
 
 isar_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
 backup_prefix = '.ci-backup'
@@ -222,6 +234,14 @@ class CIBuilder(Test):
                 f.write('SSTATE_DIR = "%s"\n' % sstate_dir)
             if image_install is not None:
                 f.write('IMAGE_INSTALL = "%s"\n' % image_install)
+            else:
+                if container:
+                    # strip kernel modules from default package install list
+                    _image_install = [p for p in IMAGE_INSTALL_DEFAULT if "-module-" not in p]
+                else:
+                    _image_install = IMAGE_INSTALL_DEFAULT
+                f.write('IMAGE_INSTALL = "%s"\n' % ' '.join(_image_install))
+
             if fail_on_cleanup == '1':
                 f.write('ISAR_FAIL_ON_CLEANUP = "1"\n')
             if installer_image:
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-6-felix.moessbauer%40siemens.com.

[PATCH v2 06/17] testsuite: limit cross_debsrc test to subset of packages

['Felix Moessbauer' via isar-users]

To test if the debsrc infrastructure works it is sufficient to build a
single package. This significantly speeds up the CI and reduce the space
needed.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 testsuite/citest.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 5ff4a4ed..3c92f788 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -233,7 +233,8 @@ class CrossTest(CIBaseTest):
         ]
 
         self.init()
-        self.perform_build_test(targets, debsrc_cache=True)
+        # only build a single custom package to speedup test
+        self.perform_build_test(targets, debsrc_cache=True, image_install='cowsay')
 
     def test_cross_trixie(self):
         targets = [
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-7-felix.moessbauer%40siemens.com.

[PATCH v2 07/17] testsuite: forward SSTATE_MIRRORS into CI env on sstate

['Felix Moessbauer' via isar-users]

When running the testsuite with sstate caching enabled and also the
environment variable SSTATE_MIRRORS is set, add this to the local conf,
so the CI can also use remote sstate caches (important for CI).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 testsuite/cibuilder.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 26002ade..310a3836 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -232,6 +232,8 @@ class CIBuilder(Test):
                 f.write('DL_DIR = "%s"\n' % dl_dir)
             if sstate_dir:
                 f.write('SSTATE_DIR = "%s"\n' % sstate_dir)
+            if sstate and 'SSTATE_MIRRORS' in os.environ:
+                f.write('SSTATE_MIRRORS = "%s"\n' % os.environ['SSTATE_MIRRORS'])
             if image_install is not None:
                 f.write('IMAGE_INSTALL = "%s"\n' % image_install)
             else:
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-8-felix.moessbauer%40siemens.com.

[PATCH v2 08/17] ci: add support for sstate cache

['Felix Moessbauer' via isar-users]

The gitlab CI jobs should finish in a timely manner to give quick
feedback to the developer. By running the tests with sstate cache,
simple changes require far less CI time.

We prepare the gitlab ci job description to pick up the sstate cache
configuration from the environment and enable the caching for all jobs
(we always enable it, even if running without sstate cache, as then the
cache is simply thrown away later on). We further introduce a (manual)
info task to show the state of the cache, as well as a cleanup task to
drain the cache.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 .gitlab-ci.yml | 36 ++++++++++++++++++++++++++++++------
 1 file changed, 30 insertions(+), 6 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 1a7abcb8..caaa56a8 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -6,12 +6,22 @@ variables:
 
 .common-build: &common-build
   stage: build
+  variables:
+    RUNNER_AFTER_SCRIPT_TIMEOUT: 15m
   before_script:
     - export http_proxy=$HTTP_PROXY
     - export https_proxy=$HTTPS_PROXY
     - export ftp_proxy=$FTP_PROXY
     - export no_proxy=$NO_PROXY
     - export DISTRO_APT_PREMIRRORS=$DISTRO_APT_PREMIRRORS
+  after_script:
+    - |
+      if [[ -n "${SSTATE_LOCATION}" ]] && [[ -d "sstate-cache" ]]; then
+        echo "=== Upload sstate artifacts to ${SSTATE_LOCATION} ==="
+        ./scripts/isar-sstate --filter '^(?!isar-image-)' upload "sstate-cache" "${SSTATE_LOCATION}"
+        ./scripts/isar-sstate info -v "${SSTATE_LOCATION}"
+      fi
+
   artifacts:
     name: "logs-$CI_JOB_ID"
     paths:
@@ -48,7 +58,7 @@ dev-ci:
     - *use-default-image
     - if: $TESTSUITE == 'dev' || $CI_PIPELINE_SOURCE != 'schedule'
   script:
-    - scripts/ci_build.sh -T dev
+    - scripts/ci_build.sh --sstate 1 -T dev
 
 fast-ci:
   <<: *common-build
@@ -56,7 +66,7 @@ fast-ci:
     - *use-default-image
     - if: $TESTSUITE == 'fast'
   script:
-    - scripts/ci_build.sh -T fast
+    - scripts/ci_build.sh --sstate 1 -T fast
 
 full-ci:
   <<: *common-build
@@ -68,7 +78,7 @@ full-ci:
     - PREVIOUS_SHA="$(cat .CI_COMMIT_SHA || true)"
     - if [ "$CI_COMMIT_SHA" != "$PREVIOUS_SHA" ]; then
           echo "$CI_COMMIT_SHA" > .CI_COMMIT_SHA;
-          scripts/ci_build.sh -T full;
+          scripts/ci_build.sh --sstate 1 -T full;
       fi
   cache:
     key: "$CI_COMMIT_REF_SLUG"
@@ -83,7 +93,7 @@ dev-ci-isar:
     - *use-docker-isar-image
     - if: $TESTSUITE == 'dev'
   script:
-    - scripts/ci_build.sh -T dev
+    - scripts/ci_build.sh --sstate 1 -T dev
 
 fast-ci-isar:
   <<: *docker-isar
@@ -92,7 +102,7 @@ fast-ci-isar:
     - *use-docker-isar-image
     - if: $TESTSUITE == 'fast'
   script:
-    - scripts/ci_build.sh -T fast
+    - scripts/ci_build.sh --sstate 1 -T fast
 
 full-ci-isar:
   <<: *docker-isar
@@ -101,4 +111,18 @@ full-ci-isar:
     - *use-docker-isar-image
     - if: $TESTSUITE == 'full'
   script:
-    - scripts/ci_build.sh -T full
+    - scripts/ci_build.sh --sstate 1 -T full
+
+sstate-cache-info:
+  stage: build
+  when: manual
+  script:
+    - ./scripts/isar-sstate info -v "${SSTATE_LOCATION}"
+
+sstate-cache-clean:
+  stage: build
+  when: manual
+  variables:
+    SSTATE_MAX_AGE: "0d"
+  script:
+    - ./scripts/isar-sstate clean "${SSTATE_LOCATION}" --max-age "${SSTATE_MAX_AGE}"
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-9-felix.moessbauer%40siemens.com.

[PATCH v2 09/17] testsuite: make test_cross_deps more specific

['Felix Moessbauer' via isar-users]

We don't need to build all packages to just check the cross dependency
propagation. Just install the single package we are interested in.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 testsuite/citest.py | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 3c92f788..52f86486 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -278,12 +278,11 @@ class CrossTest(CIBaseTest):
             'mc:qemuarm64-bookworm:isar-image-ci',
         ]
 
-        lines = [f"IMAGER_BUILD_DEPS:append = ' test-all-depnocross-native'",
-                 f"IMAGE_INSTALL:append = ' test-all-deponlycross'",
-        ]
+        lines = [f"IMAGER_BUILD_DEPS:append = ' test-all-depnocross-native'"]
 
         self.init()
-        self.perform_build_test(targets, lines=lines)
+        self.perform_build_test(targets, lines=lines,
+                                image_install='test-all-deponlycross')
 
 class PrebuiltTest(CIBaseTest):
     """
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-10-felix.moessbauer%40siemens.com.

[PATCH v2 10/17] testsuite: fix typo in log message in perform_signature_lint

['Felix Moessbauer' via isar-users]

No functional change.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 testsuite/cibase.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index e59653a2..5b0139de 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -168,7 +168,7 @@ class CIBaseTest(CIBuilder):
         **kwargs,
     ):
         """
-        Generate signature data for target(s) and check for cachability issues
+        Generate signature data for target(s) and check for cacheability issues
         """
         self.configure(**kwargs)
         self.move_in_build_dir('tmp', 'tmp_before_sstate')
@@ -189,7 +189,7 @@ class CIBaseTest(CIBuilder):
             ansi_escape = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])')
             for line in output.splitlines():
                 self.log.error(ansi_escape.sub('', line))
-            self.fail("Detected cachability issues")
+            self.fail("Detected cacheability issues")
 
     def perform_sstate_test(self, image_target, package_target, **kwargs):
         def check_executed_tasks(target, expected):
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-11-felix.moessbauer%40siemens.com.

[PATCH v2 11/17] testsuite: fix SignatureTest by avoiding absolute path in bblayers

['Felix Moessbauer' via isar-users]

The SignatureTest currently fails, as it correctly detects absolute path
that influence the sstate signatures. These path come from the ISARROOT
substitution done in the isar-init-build-env script. Depending on how
the bblayers is setup (which again depends on who does it), we might end
up with absolute paths.

To fix this, we apply the same strategy as yocto and kas uses: Making
the layer imports relative to the TOPDIR. As we can only guarantee the
correctness of this path within the testsuite, we also only apply the
fix there.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta-test/conf/bblayers.conf.sample | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta-test/conf/bblayers.conf.sample b/meta-test/conf/bblayers.conf.sample
index dcec6cf6..f37625d2 100644
--- a/meta-test/conf/bblayers.conf.sample
+++ b/meta-test/conf/bblayers.conf.sample
@@ -9,10 +9,10 @@ BBPATH = "${TOPDIR}"
 BBFILES ?= ""
 
 BBLAYERS ?= " \
-  ##ISARROOT##/meta \
-  ##ISARROOT##/meta-isar \
-  ##ISARROOT##/meta-test \
+  ${TOPDIR}/../meta \
+  ${TOPDIR}/../meta-isar \
+  ${TOPDIR}/../meta-test \
   "
 BBLAYERS_NON_REMOVABLE ?= " \
-  ##ISARROOT##/meta \
+  ${TOPDIR}/../meta \
   "
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-12-felix.moessbauer%40siemens.com.

[PATCH v2 12/17] testsuite: use more recent distros in SignatureTest

['Felix Moessbauer' via isar-users]

In general, that should not make a difference. However, we better test
things that users actually use than old-old-stable distros. As this test
is just a parser test, it is fast. Hence add the fast tag.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 testsuite/citest.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 52f86486..eaa4c440 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -708,10 +708,10 @@ class SignatureTest(CIBaseTest):
     def test_signature_lint(self):
         verbose = bool(int(self.params.get('verbose', default=0)))
         targets = [
-            'mc:qemuamd64-bullseye:isar-image-ci',
-            'mc:qemuarm-bullseye:isar-image-base',
-            'mc:qemuarm-bullseye:isar-image-base:do_populate_sdk',
-            'mc:qemuamd64-focal:isar-image-base',
+            'mc:qemuamd64-trixie:isar-image-ci',
+            'mc:qemuarm-trixie:isar-image-base',
+            'mc:qemuarm-trixie:isar-image-base:do_populate_sdk',
+            'mc:qemuamd64-noble:isar-image-base',
         ]
 
         self.init()
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-13-felix.moessbauer%40siemens.com.

[PATCH v2 13/17] testsuite: make SignatureTest idempotent

['Felix Moessbauer' via isar-users]

The SignatureTest copies the tmp dir for later comparison. However, this
currently fails in case the test was run before as the target dir then
already exists. We fix it by clearing the target dir upfront. We further
reduce the disk consumption by removing the tmp dir copy in case the
test was successfull. On error, we keep the copy for manual
analysis.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 testsuite/cibase.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index 5b0139de..5ef1a5b5 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -171,6 +171,7 @@ class CIBaseTest(CIBuilder):
         Generate signature data for target(s) and check for cacheability issues
         """
         self.configure(**kwargs)
+        self.delete_from_build_dir('tmp_before_sstate')
         self.move_in_build_dir('tmp', 'tmp_before_sstate')
         self.bitbake(targets, sig_handler='none')
 
@@ -190,6 +191,9 @@ class CIBaseTest(CIBuilder):
             for line in output.splitlines():
                 self.log.error(ansi_escape.sub('', line))
             self.fail("Detected cacheability issues")
+        else:
+            # on success, cleanup temporary copy (keep on failure to inspect)
+            self.delete_from_build_dir('tmp_before_sstate')
 
     def perform_sstate_test(self, image_target, package_target, **kwargs):
         def check_executed_tasks(target, expected):
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-14-felix.moessbauer%40siemens.com.

[PATCH v2 14/17] testsuite: add rootfs target for rootfs only tests

['Felix Moessbauer' via isar-users]

As a preparation to speedup the testsuite, we add a rootfs recipe that
behaves similar to the image recipe, however does not carry a initrd or
a kernel which avoids the huge emulation overhead on non native
architectures. This recipe also can be used to check rootfs features
independently.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta-test/recipes-core/images/isar-rootfs-ci.bb | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
 create mode 100644 meta-test/recipes-core/images/isar-rootfs-ci.bb

diff --git a/meta-test/recipes-core/images/isar-rootfs-ci.bb b/meta-test/recipes-core/images/isar-rootfs-ci.bb
new file mode 100644
index 00000000..a87fd1a8
--- /dev/null
+++ b/meta-test/recipes-core/images/isar-rootfs-ci.bb
@@ -0,0 +1,17 @@
+# CI root filesystem for target installation (without kernel, initrd, ...)
+#
+# This software is a part of ISAR.
+# Copyright (C) 2025 Siemens
+
+# Bill-of-material
+ROOTFS_MANIFEST_DEPLOY_DIR = "${DEPLOY_DIR_IMAGE}"
+
+ROOTFSDIR = "${WORKDIR}/rootfs"
+ROOTFS_FEATURES = "generate-sbom"
+
+inherit multiarch
+inherit rootfs
+
+# behave similar to image class, so we can reuse the testing infrastructure
+DEPENDS += "${IMAGE_INSTALL}"
+ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${@isar_multiarch_packages('IMAGE_INSTALL', d)}"
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-15-felix.moessbauer%40siemens.com.

[PATCH v2 15/17] testsuite: refactor sbom tests to avoid overhead

['Felix Moessbauer' via isar-users]

We currently test the SBOM infrastructure in all image builds, which
adds a significant overhead. We now change this to not generate SBOMs in
general (and by that avoid building the dependencies). To not have a
testing gap, we add a dedicated SBOM test that checks the SBOM creation
for various targets. In addition, we now also check the content of the
SBOM for plausibility.

In the future, the SBOM test can be extended without slowing down the
overall test execution.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 testsuite/cibase.py    | 26 ++++++++++++++++++++++++++
 testsuite/cibuilder.py |  4 ++++
 testsuite/citest.py    | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 63 insertions(+)

diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index 5ef1a5b5..fd6a3df9 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -140,6 +140,32 @@ class CIBaseTest(CIBuilder):
         self.delete_from_build_dir('ccache')
         self.unconfigure()
 
+    def perform_sbom_test(self, targets, **kwargs):
+        """
+        Build a rootfs containing a needle package and check if that package
+        is added to the sbom.
+        """
+        import json
+
+        needle_pkg = 'cowsay'
+        self.perform_build_test(
+            targets, image_install=needle_pkg,
+            generate_sbom=True
+        )
+
+        for t in targets:
+            ds, pn, distro, machine = \
+                CIUtils.getVars('DEPLOY_DIR_SBOM', 'PN', 'DISTRO', 'MACHINE',
+                                target=t)
+            for t in ["cdx", "spdx"]:
+                sbom_path = os.path.join(ds, f'{pn}-{distro}-{machine}.{t}.json')
+                self.log.info(f"Check {t} SBOM in {sbom_path}")
+                with open(sbom_path) as f:
+                    sbom = json.load(f)
+                pkg_key = 'components' if t == 'cdx' else 'packages'
+                if not any(c for c in sbom[pkg_key] if c['name'] == needle_pkg):
+                    self.fail(f'{needle_pkg} package not found in SBOM {sbom_path}')
+
     def perform_sstate_populate(self, image_target, **kwargs):
         # Use a different isar root for populating sstate cache
         isar_sstate = f"{isar_root}/isar-sstate"
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 310a3836..614a3397 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -126,6 +126,7 @@ class CIBuilder(Test):
         installer_distro=None,
         installer_device=None,
         customizations=None,
+        generate_sbom=False,
         lines=None,
         **kwargs,
     ):
@@ -176,6 +177,7 @@ class CIBuilder(Test):
             f"  image_install = {image_install}\n"
             f"  installer_image = {installer_image}\n"
             f"  customizations = {customizations}\n"
+            f"  generate_sbom = {generate_sbom}\n"
             f"  lines = {strlines}\n"
             f"==================================================="
         )
@@ -275,6 +277,8 @@ class CIBuilder(Test):
                     'CUSTOMIZATION_FOR_IMAGES:append = " isar-image-ci"\n'
                     'HOSTNAME:isar-image-ci = "isar-ci"\n'
                 )
+            if generate_sbom is False:
+                f.write('ROOTFS_FEATURES:remove = "generate-sbom"\n')
             if lines is not None:
                 f.writelines((line + '\n' if not line.endswith('\n') else line) for line in lines)
 
diff --git a/testsuite/citest.py b/testsuite/citest.py
index eaa4c440..d908f9bc 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -694,6 +694,39 @@ class CustomizationsTest(CIBaseTest):
         )
 
 
+class SbomTest(CIBaseTest):
+    """
+    Test to check if sbom is generated and contains expected packages.
+    Most tests are rootfs tests to avoid costly initrd build and imaging.
+
+    :avocado: tags=sbom,fast
+    """
+
+    def test_sbom_rootfs_generate(self):
+        targets = [
+            'mc:qemuamd64-bookworm:isar-rootfs-ci',
+            'mc:qemuarm64-bookworm:isar-rootfs-ci',
+            'mc:qemuamd64-trixie:isar-rootfs-ci',
+            'mc:qemuarm64-trixie:isar-rootfs-ci',
+            'mc:qemuamd64-noble:isar-rootfs-ci',
+        ]
+
+        self.init()
+        self.perform_sbom_test(targets)
+
+    def test_sbom_unsupported(self):
+        targets = [
+            'mc:qemuamd64-bullseye:isar-rootfs-ci',
+            'mc:qemuamd64-focal:isar-rootfs-ci',
+        ]
+
+        self.init()
+        self.perform_build_test(
+            targets, bitbake_cmd='do_rootfs', image_install='cowsay',
+            generate_sbom=True
+        )
+
+
 class SignatureTest(CIBaseTest):
 
     """
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-16-felix.moessbauer%40siemens.com.

[PATCH v2 16/17] testsuite: make sbuild-flavor test standalone

['Felix Moessbauer' via isar-users]

As the hello-isar recipe acts as an example for the SBUILD_FLAVOR
feature, we also pull in a dedicated chroot into almost all tests. This
is very costly and does not add much value. We change this by setting
the SBUILD_FLAVOR of hello-isar to none in the CI layer and add a
dedicated test that just tests the SBUILD_FLAVOR feature.

This only slightly reduces the test coverage, but it significantly
speeds up the test execution.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 .../hello-isar/hello-isar.bbappend            |  3 +++
 .../recipes-app/libhello/libhello.bbappend    |  5 +++++
 testsuite/citest.py                           | 21 +++++++++++++++++++
 3 files changed, 29 insertions(+)
 create mode 100644 meta-test/recipes-app/libhello/libhello.bbappend

diff --git a/meta-test/recipes-app/hello-isar/hello-isar.bbappend b/meta-test/recipes-app/hello-isar/hello-isar.bbappend
index 44686458..27212262 100644
--- a/meta-test/recipes-app/hello-isar/hello-isar.bbappend
+++ b/meta-test/recipes-app/hello-isar/hello-isar.bbappend
@@ -15,3 +15,6 @@ SRC_URI:append = " \
 "
 SRC_URI:remove = "file://nonexist-file"
 SRC_URI:remove = "git://nonexist-git"
+
+# avoid creating a dedicated sbuild chroot
+SBUILD_FLAVOR = ""
diff --git a/meta-test/recipes-app/libhello/libhello.bbappend b/meta-test/recipes-app/libhello/libhello.bbappend
new file mode 100644
index 00000000..3c88a741
--- /dev/null
+++ b/meta-test/recipes-app/libhello/libhello.bbappend
@@ -0,0 +1,5 @@
+# This software is a part of ISAR.
+# Copyright (C) Siemens
+
+# avoid creating a dedicated sbuild chroot
+SBUILD_FLAVOR = ""
diff --git a/testsuite/citest.py b/testsuite/citest.py
index d908f9bc..49fcdec0 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -131,6 +131,27 @@ class CompatTest(CIBaseTest):
         self.perform_build_test(targets, compat_arch=True)
 
 
+class SbuildFlavor(CIBaseTest):
+    """
+    Test package build with a custom sbuild chroot.
+    :avocado: tags=sbuildflavor,fast
+    """
+
+    def test_sbuild_flavor(self):
+        targets = [
+            'mc:qemuamd64-trixie:hello-isar',
+            'mc:qemuarm64-trixie:hello-isar',
+        ]
+
+        lines = [
+            'SBUILD_FLAVOR:hello-isar = "db2m"',
+            'SBUILD_FLAVOR:libhello = "db2m"'
+        ]
+
+        self.init()
+        self.perform_build_test(targets, lines=lines)
+
+
 class ReproTest(CIBaseTest):
 
     """
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-17-felix.moessbauer%40siemens.com.

[PATCH v2 17/17] testsuite: skip VM tests if images are not available

['Felix Moessbauer' via isar-users]

We currently fail the test if the image is not available. Instead, we
now skip it and report what is missing. By that, we stay semantically
correct by not failing tests when pre-conditions are not fulfilled.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 testsuite/cibuilder.py | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 614a3397..b64d808c 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -24,6 +24,7 @@ from utils import CIUtils
 from avocado import Test
 from avocado.utils import path
 from avocado.utils import process
+from avocado.core import exceptions
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '../bitbake/lib'))
 
@@ -703,6 +704,7 @@ class CIBuilder(Test):
         keep=False,
     ):
         time_to_wait = self.params.get('time_to_wait', default=DEF_VM_TO_SEC)
+        self.skip_if_vm_image_missing(arch, distro, image)
 
         self.log.info("===================================================")
         self.log.info(f"Running Isar VM boot test for ({distro}-{arch})")
@@ -800,3 +802,20 @@ class CIBuilder(Test):
             self.vm_turn_off(vm)
 
         return stdout, stderr
+
+    def skip_if_vm_image_missing(self, arch, distro, image):
+        (
+            image_fstypes,
+            image_fullname,
+            deploy_dir_image,
+        ) = CIUtils.getVars(
+            'IMAGE_FSTYPES',
+            'IMAGE_FULLNAME',
+            'DEPLOY_DIR_IMAGE',
+            target=f"mc:qemu{arch}-{distro}:{image}",
+        )
+        image_type = image_fstypes.split()[0]
+        rootfs_image = f"{image_fullname}.{image_type}"
+        rootfs_image_path = os.path.join(deploy_dir_image, rootfs_image)
+        if not os.path.exists(rootfs_image_path):
+            raise exceptions.TestSkipError(f'VM image missing: {rootfs_image}')
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251219131923.1504046-18-felix.moessbauer%40siemens.com.