* [PATCH v3 01/20] testsuite: move targets with custom kernel to separate test
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 02/20] testsuite: enable ccache on kernel tests Zhihang Wei
` (20 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
By moving these tests out of the cross test (which is a fast test), we
significantly reduce the test time on fast.
The tests are now placed in the KernelTests class in a dedicated test,
suitable for applying further optimizations (separate commit).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
testsuite/citest.py | 17 +++++++++++++----
1 file changed, 13 insertions(+), 4 deletions(-)
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 76a3df69..cb63a8fe 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -190,7 +190,8 @@ class InstallerTest(CIBaseTest):
class CrossTest(CIBaseTest):
"""
- Start cross build for the defined set of configurations
+ Start cross build for the defined set of configurations.
+ As this is a fast test, ensure to not depend on custom kernels.
:avocado: tags=cross,fast
"""
@@ -199,12 +200,9 @@ class CrossTest(CIBaseTest):
targets = [
'mc:qemuarm-buster:isar-image-ci',
'mc:qemuarm-bullseye:isar-image-ci',
- 'mc:de0-nano-soc-bullseye:isar-image-base',
- 'mc:stm32mp15x-bullseye:isar-image-base',
'mc:qemuarm-bookworm:isar-image-ci',
'mc:qemuarm64-focal:isar-image-base',
'mc:nanopi-neo-efi-bookworm:isar-image-base',
- 'mc:phyboard-mira-bookworm:isar-image-base',
]
self.init()
@@ -273,6 +271,17 @@ class KernelTests(CIBaseTest):
:avocado: tags=kernel,full
"""
+ def test_kernel_cross(self):
+ """Targets that build a custom kernel"""
+ targets = [
+ 'mc:de0-nano-soc-bullseye:isar-image-base',
+ 'mc:stm32mp15x-bullseye:isar-image-base',
+ 'mc:phyboard-mira-bookworm:isar-image-base',
+ ]
+
+ self.init()
+ self.perform_build_test(targets)
+
def test_per_kernel(self):
"""Test per-kernel recipe variants for external kernel modules."""
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-2-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 02/20] testsuite: enable ccache on kernel tests
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 01/20] testsuite: move targets with custom kernel to separate test Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 03/20] testsuite: make prebuilt container a feature test Zhihang Wei
` (19 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
The kernels are anyways usually built with ccache enabled to reduce the
buildtime. By enabling it in the test as well, we can significantly
bring down test time on repeated builds.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
testsuite/citest.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/testsuite/citest.py b/testsuite/citest.py
index cb63a8fe..cba3c008 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -280,7 +280,7 @@ class KernelTests(CIBaseTest):
]
self.init()
- self.perform_build_test(targets)
+ self.perform_build_test(targets, ccache=True)
def test_per_kernel(self):
"""Test per-kernel recipe variants for external kernel modules."""
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-3-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 03/20] testsuite: make prebuilt container a feature test
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 01/20] testsuite: move targets with custom kernel to separate test Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 02/20] testsuite: enable ccache on kernel tests Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 04/20] testsuite: make compat test standalone test Zhihang Wei
` (18 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
This reworks how we handle feature tests: Instead of binding a feature
to a target via appends in isar-image-ci, we implement the feature by
using the test setup function. By that, we have fine grained control over
where to test a feature and can avoid testing the same feature over and
over again. This leads to a much cleaner architecture and faster test
execution.
We start implementing this approach with the prebuilt container test.
As the images with containers are currently re-used in the
VmBootTestFull, these tests might break depending on the test execution
order. To fix this, we rebuild the needed images in the VM test itself,
which takes less than a minute when running with sstate cache.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
.../recipes-core/images/isar-image-ci.bb | 2 --
testsuite/citest.py | 25 +++++++++++++++++++
2 files changed, 25 insertions(+), 2 deletions(-)
diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb
index 7f2b404d..58aa5738 100644
--- a/meta-test/recipes-core/images/isar-image-ci.bb
+++ b/meta-test/recipes-core/images/isar-image-ci.bb
@@ -16,7 +16,6 @@ IMAGE_INSTALL += "sshd-regen-keys"
# qemuamd64-bookworm
WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
-IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
# qemuamd64-bullseye
IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.zst tar.zst"
@@ -52,4 +51,3 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= " ${SYSTEMD_BOOTLOADER_INSTALL}
# qemuarm64-bookworm
IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= " ${GRUB_BOOTLOADER_INSTALL}"
-IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index cba3c008..eec90c6e 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -265,6 +265,25 @@ class CrossTest(CIBaseTest):
self.init()
self.perform_build_test(targets, lines=lines)
+class PrebuiltTest(CIBaseTest):
+ """
+ Tests associated with prebuilt artifacts (containers, debs).
+ :avocado: tags=prebuilt,fast
+ """
+
+ def test_prebuilt_containers(self):
+ targets = [
+ 'mc:qemuamd64-bookworm:isar-image-ci',
+ 'mc:qemuarm64-bookworm:isar-image-ci',
+ ]
+
+ self.init()
+ self.perform_build_test(
+ targets,
+ bitbake_cmd='do_rootfs_install',
+ image_install="prebuilt-docker-img prebuilt-podman-img")
+
+
class KernelTests(CIBaseTest):
"""
Tests associated with kernel builds and development.
@@ -984,11 +1003,17 @@ class VmBootTestFull(CIBaseTest):
def test_amd64_bookworm_prebuilt_containers(self):
self.init()
+ self.perform_build_test(
+ ['mc:qemuamd64-bookworm:isar-image-ci'],
+ image_install="prebuilt-docker-img prebuilt-podman-img")
self.vm_start('amd64', 'bookworm', image='isar-image-ci',
script='test_prebuilt_containers.sh')
def test_arm64_bookworm_prebuilt_containers(self):
self.init()
+ self.perform_build_test(
+ ['mc:qemuarm64-bookworm:isar-image-ci'],
+ image_install="prebuilt-docker-img prebuilt-podman-img")
self.vm_start('arm64', 'bookworm', image='isar-image-ci',
script='test_prebuilt_containers.sh')
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-4-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 04/20] testsuite: make compat test standalone test
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (2 preceding siblings ...)
2026-01-23 8:24 ` [PATCH v3 03/20] testsuite: make prebuilt container a feature test Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 05/20] testsuite: handle IMAGE_INSTALL solely in cibuilder.py Zhihang Wei
` (17 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
Currently, enabling compat mode during configuration also changes the
set of packages we build. As this is done with appends, some packages
are unconditionally added, despite not being part of the original test
case. We change this by only enabling compat mode when selecting compat.
To actually test the compat feature, we add a simple and fast test that
builds the hello-isar-compat package.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
testsuite/cibuilder.py | 4 ----
testsuite/citest.py | 19 +++++++++++++++++++
2 files changed, 19 insertions(+), 4 deletions(-)
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index dea57405..d23ba84b 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -180,11 +180,7 @@ class CIBuilder(Test):
if compat_arch:
f.write(
'ISAR_ENABLE_COMPAT_ARCH:amd64 = "1"\n'
- 'IMAGE_INSTALL:remove:amd64 = "hello-isar"\n'
- 'IMAGE_INSTALL:append:amd64 = " hello-isar-compat"\n'
'ISAR_ENABLE_COMPAT_ARCH:arm64 = "1"\n'
- 'IMAGE_INSTALL:remove:arm64 = "hello-isar"\n'
- 'IMAGE_INSTALL:append:arm64 = " hello-isar-compat"\n'
)
if not cross:
f.write('ISAR_CROSS_COMPILE = "0"\n')
diff --git a/testsuite/citest.py b/testsuite/citest.py
index eec90c6e..5ff4a4ed 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -112,6 +112,25 @@ class DevTest(CIBaseTest):
self.vm_start('arm', 'bookworm', skip_modulecheck=True)
+class CompatTest(CIBaseTest):
+ """
+ Test compilation of recipes for compat architecture.
+ This also tests a custom sbuild chroot for compat.
+ :avocado: tags=compat,fast
+ """
+
+ def test_compat_recipe(self):
+ targets = [
+ 'mc:qemuamd64-bookworm:hello-isar-compat',
+ 'mc:qemuarm64-bookworm:hello-isar-compat',
+ 'mc:qemuamd64-trixie:hello-isar-compat',
+ 'mc:qemuarm64-trixie:hello-isar-compat',
+ ]
+
+ self.init()
+ self.perform_build_test(targets, compat_arch=True)
+
+
class ReproTest(CIBaseTest):
"""
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-5-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 05/20] testsuite: handle IMAGE_INSTALL solely in cibuilder.py
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (3 preceding siblings ...)
2026-01-23 8:24 ` [PATCH v3 04/20] testsuite: make compat test standalone test Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 06/20] testsuite: limit cross_debsrc test to subset of packages Zhihang Wei
` (16 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
By moving the IMAGE_INSTALL parts out of the ci local conf sample, we
make this part reusable across tests. By that, tests can easily
customize this list without either taking it as-is or completely
replacing it. We further get rid of overrides that are not used in the
CI anyways.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
meta-test/conf/local.conf.sample | 10 ----------
testsuite/cibuilder.py | 20 ++++++++++++++++++++
2 files changed, 20 insertions(+), 10 deletions(-)
diff --git a/meta-test/conf/local.conf.sample b/meta-test/conf/local.conf.sample
index 092d6ba6..862bea47 100644
--- a/meta-test/conf/local.conf.sample
+++ b/meta-test/conf/local.conf.sample
@@ -27,16 +27,6 @@ BB_DISKMON_DIRS = "\
MIRRORS ?= "git?://salsa\.debian\.org/debian/.* git://github.com/ilbers/BASENAME"
MIRRORS += "https?://cdn\.kernel\.org/.* https://mirrors.edge.kernel.org/PATH"
-# The default list of extra packages
-IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs samefile hello isar-disable-apt-cache cowsay example-prebuilt"
-
-# Container machines don't need example module and enable-fsck.
-IMAGE_INSTALL:remove:container-amd64 = "example-module-${KERNEL_NAME} enable-fsck"
-
-# Machines with secure boot should use signed modules
-IMAGE_INSTALL:remove:qemuamd64-sb = "example-module-${KERNEL_NAME}"
-IMAGE_INSTALL:append:qemuamd64-sb = " example-module-signed-${KERNEL_NAME}"
-
# Users and groups
USERS += "root"
USER_root[password] ??= "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index d23ba84b..4a181357 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -30,6 +30,18 @@ sys.path.append(os.path.join(os.path.dirname(__file__), '../bitbake/lib'))
import bb
DEF_VM_TO_SEC = 600
+IMAGE_INSTALL_DEFAULT = [
+ 'hello-isar',
+ 'example-raw',
+ 'example-module-${KERNEL_NAME}',
+ 'enable-fsck',
+ 'isar-exclude-docs',
+ 'samefile',
+ 'hello',
+ 'isar-disable-apt-cache',
+ 'cowsay',
+ 'example-prebuilt'
+]
isar_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
backup_prefix = '.ci-backup'
@@ -222,6 +234,14 @@ class CIBuilder(Test):
f.write('SSTATE_DIR = "%s"\n' % sstate_dir)
if image_install is not None:
f.write('IMAGE_INSTALL = "%s"\n' % image_install)
+ else:
+ if container:
+ # strip kernel modules from default package install list
+ _image_install = [p for p in IMAGE_INSTALL_DEFAULT if "-module-" not in p]
+ else:
+ _image_install = IMAGE_INSTALL_DEFAULT
+ f.write('IMAGE_INSTALL = "%s"\n' % ' '.join(_image_install))
+
if fail_on_cleanup == '1':
f.write('ISAR_FAIL_ON_CLEANUP = "1"\n')
if installer_image:
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-6-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 06/20] testsuite: limit cross_debsrc test to subset of packages
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (4 preceding siblings ...)
2026-01-23 8:24 ` [PATCH v3 05/20] testsuite: handle IMAGE_INSTALL solely in cibuilder.py Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 07/20] testsuite: forward SSTATE_MIRRORS into CI env on sstate Zhihang Wei
` (15 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
To test if the debsrc infrastructure works it is sufficient to build a
single package. This significantly speeds up the CI and reduce the space
needed.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
testsuite/citest.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 5ff4a4ed..3c92f788 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -233,7 +233,8 @@ class CrossTest(CIBaseTest):
]
self.init()
- self.perform_build_test(targets, debsrc_cache=True)
+ # only build a single custom package to speedup test
+ self.perform_build_test(targets, debsrc_cache=True, image_install='cowsay')
def test_cross_trixie(self):
targets = [
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-7-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 07/20] testsuite: forward SSTATE_MIRRORS into CI env on sstate
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (5 preceding siblings ...)
2026-01-23 8:24 ` [PATCH v3 06/20] testsuite: limit cross_debsrc test to subset of packages Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 08/20] ci: add support for sstate cache Zhihang Wei
` (14 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
When running the testsuite with sstate caching enabled and also the
environment variable SSTATE_MIRRORS is set, add this to the local conf,
so the CI can also use remote sstate caches (important for CI).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
testsuite/cibuilder.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 4a181357..9c97115b 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -232,6 +232,8 @@ class CIBuilder(Test):
f.write('DL_DIR = "%s"\n' % dl_dir)
if sstate_dir:
f.write('SSTATE_DIR = "%s"\n' % sstate_dir)
+ if sstate and 'SSTATE_MIRRORS' in os.environ:
+ f.write('SSTATE_MIRRORS = "%s"\n' % os.environ['SSTATE_MIRRORS'])
if image_install is not None:
f.write('IMAGE_INSTALL = "%s"\n' % image_install)
else:
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-8-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 08/20] ci: add support for sstate cache
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (6 preceding siblings ...)
2026-01-23 8:24 ` [PATCH v3 07/20] testsuite: forward SSTATE_MIRRORS into CI env on sstate Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 09/20] testsuite: make test_cross_deps more specific Zhihang Wei
` (13 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
The gitlab CI jobs should finish in a timely manner to give quick
feedback to the developer. By running the tests with sstate cache,
simple changes require far less CI time.
We prepare the gitlab ci job description to pick up the sstate cache
configuration from the environment and enable the caching for all jobs
(we always enable it, even if running without sstate cache, as then the
cache is simply thrown away later on). We further introduce a (manual)
info task to show the state of the cache, as well as a cleanup task to
drain the cache.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
.gitlab-ci.yml | 36 ++++++++++++++++++++++++++++++------
1 file changed, 30 insertions(+), 6 deletions(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index f82ddf11..ce12e26c 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -6,12 +6,22 @@ variables:
.common-build: &common-build
stage: build
+ variables:
+ RUNNER_AFTER_SCRIPT_TIMEOUT: 15m
before_script:
- export http_proxy=$HTTP_PROXY
- export https_proxy=$HTTPS_PROXY
- export ftp_proxy=$FTP_PROXY
- export no_proxy=$NO_PROXY
- export DISTRO_APT_PREMIRRORS=$DISTRO_APT_PREMIRRORS
+ after_script:
+ - |
+ if [[ -n "${SSTATE_LOCATION}" ]] && [[ -d "sstate-cache" ]]; then
+ echo "=== Upload sstate artifacts to ${SSTATE_LOCATION} ==="
+ ./scripts/isar-sstate --filter '^(?!isar-image-)' upload "sstate-cache" "${SSTATE_LOCATION}"
+ ./scripts/isar-sstate info -v "${SSTATE_LOCATION}"
+ fi
+
artifacts:
name: "logs-$CI_JOB_ID"
paths:
@@ -48,7 +58,7 @@ dev-ci:
- *use-default-image
- if: $TESTSUITE == 'dev' || $CI_PIPELINE_SOURCE != 'schedule'
script:
- - scripts/ci_build.sh -T dev
+ - scripts/ci_build.sh --sstate 1 -T dev
fast-ci:
<<: *common-build
@@ -56,7 +66,7 @@ fast-ci:
- *use-default-image
- if: $TESTSUITE == 'fast'
script:
- - scripts/ci_build.sh -T fast
+ - scripts/ci_build.sh --sstate 1 -T fast
full-ci:
<<: *common-build
@@ -68,7 +78,7 @@ full-ci:
- PREVIOUS_SHA="$(cat .CI_COMMIT_SHA || true)"
- if [ "$CI_COMMIT_SHA" != "$PREVIOUS_SHA" ]; then
echo "$CI_COMMIT_SHA" > .CI_COMMIT_SHA;
- scripts/ci_build.sh -T full;
+ scripts/ci_build.sh --sstate 1 -T full;
fi
cache:
key: "$CI_COMMIT_REF_SLUG"
@@ -83,7 +93,7 @@ dev-ci-isar:
- *use-docker-isar-image
- if: $TESTSUITE == 'dev'
script:
- - scripts/ci_build.sh -T dev
+ - scripts/ci_build.sh --sstate 1 -T dev
fast-ci-isar:
<<: *docker-isar
@@ -92,7 +102,7 @@ fast-ci-isar:
- *use-docker-isar-image
- if: $TESTSUITE == 'fast'
script:
- - scripts/ci_build.sh -T fast
+ - scripts/ci_build.sh --sstate 1 -T fast
full-ci-isar:
<<: *docker-isar
@@ -101,4 +111,18 @@ full-ci-isar:
- *use-docker-isar-image
- if: $TESTSUITE == 'full'
script:
- - scripts/ci_build.sh -T full
+ - scripts/ci_build.sh --sstate 1 -T full
+
+sstate-cache-info:
+ stage: build
+ when: manual
+ script:
+ - ./scripts/isar-sstate info -v "${SSTATE_LOCATION}"
+
+sstate-cache-clean:
+ stage: build
+ when: manual
+ variables:
+ SSTATE_MAX_AGE: "0d"
+ script:
+ - ./scripts/isar-sstate clean "${SSTATE_LOCATION}" --max-age "${SSTATE_MAX_AGE}"
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-9-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 09/20] testsuite: make test_cross_deps more specific
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (7 preceding siblings ...)
2026-01-23 8:24 ` [PATCH v3 08/20] ci: add support for sstate cache Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 10/20] testsuite: fix typo in log message in perform_signature_lint Zhihang Wei
` (12 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
We don't need to build all packages to just check the cross dependency
propagation. Just install the single package we are interested in.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
testsuite/citest.py | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 3c92f788..52f86486 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -278,12 +278,11 @@ class CrossTest(CIBaseTest):
'mc:qemuarm64-bookworm:isar-image-ci',
]
- lines = [f"IMAGER_BUILD_DEPS:append = ' test-all-depnocross-native'",
- f"IMAGE_INSTALL:append = ' test-all-deponlycross'",
- ]
+ lines = [f"IMAGER_BUILD_DEPS:append = ' test-all-depnocross-native'"]
self.init()
- self.perform_build_test(targets, lines=lines)
+ self.perform_build_test(targets, lines=lines,
+ image_install='test-all-deponlycross')
class PrebuiltTest(CIBaseTest):
"""
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-10-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 10/20] testsuite: fix typo in log message in perform_signature_lint
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (8 preceding siblings ...)
2026-01-23 8:24 ` [PATCH v3 09/20] testsuite: make test_cross_deps more specific Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 11/20] testsuite: fix SignatureTest by avoiding absolute path in bblayers Zhihang Wei
` (11 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
No functional change.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
testsuite/cibase.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index e59653a2..5b0139de 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -168,7 +168,7 @@ class CIBaseTest(CIBuilder):
**kwargs,
):
"""
- Generate signature data for target(s) and check for cachability issues
+ Generate signature data for target(s) and check for cacheability issues
"""
self.configure(**kwargs)
self.move_in_build_dir('tmp', 'tmp_before_sstate')
@@ -189,7 +189,7 @@ class CIBaseTest(CIBuilder):
ansi_escape = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])')
for line in output.splitlines():
self.log.error(ansi_escape.sub('', line))
- self.fail("Detected cachability issues")
+ self.fail("Detected cacheability issues")
def perform_sstate_test(self, image_target, package_target, **kwargs):
def check_executed_tasks(target, expected):
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-11-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 11/20] testsuite: fix SignatureTest by avoiding absolute path in bblayers
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (9 preceding siblings ...)
2026-01-23 8:24 ` [PATCH v3 10/20] testsuite: fix typo in log message in perform_signature_lint Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 12/20] testsuite: use more recent distros in SignatureTest Zhihang Wei
` (10 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
The SignatureTest currently fails, as it correctly detects absolute path
that influence the sstate signatures. These path come from the ISARROOT
substitution done in the isar-init-build-env script. Depending on how
the bblayers is setup (which again depends on who does it), we might end
up with absolute paths.
To fix this, we apply the same strategy as yocto and kas uses: Making
the layer imports relative to the TOPDIR. As we can only guarantee the
correctness of this path within the testsuite, we also only apply the
fix there.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
meta-test/conf/bblayers.conf.sample | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/meta-test/conf/bblayers.conf.sample b/meta-test/conf/bblayers.conf.sample
index dcec6cf6..f37625d2 100644
--- a/meta-test/conf/bblayers.conf.sample
+++ b/meta-test/conf/bblayers.conf.sample
@@ -9,10 +9,10 @@ BBPATH = "${TOPDIR}"
BBFILES ?= ""
BBLAYERS ?= " \
- ##ISARROOT##/meta \
- ##ISARROOT##/meta-isar \
- ##ISARROOT##/meta-test \
+ ${TOPDIR}/../meta \
+ ${TOPDIR}/../meta-isar \
+ ${TOPDIR}/../meta-test \
"
BBLAYERS_NON_REMOVABLE ?= " \
- ##ISARROOT##/meta \
+ ${TOPDIR}/../meta \
"
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-12-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 12/20] testsuite: use more recent distros in SignatureTest
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (10 preceding siblings ...)
2026-01-23 8:24 ` [PATCH v3 11/20] testsuite: fix SignatureTest by avoiding absolute path in bblayers Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 13/20] testsuite: make SignatureTest idempotent Zhihang Wei
` (9 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
In general, that should not make a difference. However, we better test
things that users actually use than old-old-stable distros. As this test
is just a parser test, it is fast. Hence add the fast tag.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
testsuite/citest.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 52f86486..eaa4c440 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -708,10 +708,10 @@ class SignatureTest(CIBaseTest):
def test_signature_lint(self):
verbose = bool(int(self.params.get('verbose', default=0)))
targets = [
- 'mc:qemuamd64-bullseye:isar-image-ci',
- 'mc:qemuarm-bullseye:isar-image-base',
- 'mc:qemuarm-bullseye:isar-image-base:do_populate_sdk',
- 'mc:qemuamd64-focal:isar-image-base',
+ 'mc:qemuamd64-trixie:isar-image-ci',
+ 'mc:qemuarm-trixie:isar-image-base',
+ 'mc:qemuarm-trixie:isar-image-base:do_populate_sdk',
+ 'mc:qemuamd64-noble:isar-image-base',
]
self.init()
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-13-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 13/20] testsuite: make SignatureTest idempotent
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (11 preceding siblings ...)
2026-01-23 8:24 ` [PATCH v3 12/20] testsuite: use more recent distros in SignatureTest Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 14/20] testsuite: add rootfs target for rootfs only tests Zhihang Wei
` (8 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
The SignatureTest copies the tmp dir for later comparison. However, this
currently fails in case the test was run before as the target dir then
already exists. We fix it by clearing the target dir upfront. We further
reduce the disk consumption by removing the tmp dir copy in case the
test was successfull. On error, we keep the copy for manual
analysis.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
testsuite/cibase.py | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index 5b0139de..5ef1a5b5 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -171,6 +171,7 @@ class CIBaseTest(CIBuilder):
Generate signature data for target(s) and check for cacheability issues
"""
self.configure(**kwargs)
+ self.delete_from_build_dir('tmp_before_sstate')
self.move_in_build_dir('tmp', 'tmp_before_sstate')
self.bitbake(targets, sig_handler='none')
@@ -190,6 +191,9 @@ class CIBaseTest(CIBuilder):
for line in output.splitlines():
self.log.error(ansi_escape.sub('', line))
self.fail("Detected cacheability issues")
+ else:
+ # on success, cleanup temporary copy (keep on failure to inspect)
+ self.delete_from_build_dir('tmp_before_sstate')
def perform_sstate_test(self, image_target, package_target, **kwargs):
def check_executed_tasks(target, expected):
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-14-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 14/20] testsuite: add rootfs target for rootfs only tests
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (12 preceding siblings ...)
2026-01-23 8:24 ` [PATCH v3 13/20] testsuite: make SignatureTest idempotent Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 15/20] testsuite: refactor sbom tests to avoid overhead Zhihang Wei
` (7 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
As a preparation to speedup the testsuite, we add a rootfs recipe that
behaves similar to the image recipe, however does not carry a initrd or
a kernel which avoids the huge emulation overhead on non native
architectures. This recipe also can be used to check rootfs features
independently.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
meta-test/recipes-core/images/isar-rootfs-ci.bb | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
create mode 100644 meta-test/recipes-core/images/isar-rootfs-ci.bb
diff --git a/meta-test/recipes-core/images/isar-rootfs-ci.bb b/meta-test/recipes-core/images/isar-rootfs-ci.bb
new file mode 100644
index 00000000..a87fd1a8
--- /dev/null
+++ b/meta-test/recipes-core/images/isar-rootfs-ci.bb
@@ -0,0 +1,17 @@
+# CI root filesystem for target installation (without kernel, initrd, ...)
+#
+# This software is a part of ISAR.
+# Copyright (C) 2025 Siemens
+
+# Bill-of-material
+ROOTFS_MANIFEST_DEPLOY_DIR = "${DEPLOY_DIR_IMAGE}"
+
+ROOTFSDIR = "${WORKDIR}/rootfs"
+ROOTFS_FEATURES = "generate-sbom"
+
+inherit multiarch
+inherit rootfs
+
+# behave similar to image class, so we can reuse the testing infrastructure
+DEPENDS += "${IMAGE_INSTALL}"
+ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${@isar_multiarch_packages('IMAGE_INSTALL', d)}"
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-15-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 15/20] testsuite: refactor sbom tests to avoid overhead
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (13 preceding siblings ...)
2026-01-23 8:24 ` [PATCH v3 14/20] testsuite: add rootfs target for rootfs only tests Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 16/20] testsuite: make sbuild-flavor test standalone Zhihang Wei
` (6 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
We currently test the SBOM infrastructure in all image builds, which
adds a significant overhead. We now change this to not generate SBOMs in
general (and by that avoid building the dependencies). To not have a
testing gap, we add a dedicated SBOM test that checks the SBOM creation
for various targets. In addition, we now also check the content of the
SBOM for plausibility.
In the future, the SBOM test can be extended without slowing down the
overall test execution.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
testsuite/cibase.py | 26 ++++++++++++++++++++++++++
testsuite/cibuilder.py | 4 ++++
testsuite/citest.py | 33 +++++++++++++++++++++++++++++++++
3 files changed, 63 insertions(+)
diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index 5ef1a5b5..fd6a3df9 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -140,6 +140,32 @@ class CIBaseTest(CIBuilder):
self.delete_from_build_dir('ccache')
self.unconfigure()
+ def perform_sbom_test(self, targets, **kwargs):
+ """
+ Build a rootfs containing a needle package and check if that package
+ is added to the sbom.
+ """
+ import json
+
+ needle_pkg = 'cowsay'
+ self.perform_build_test(
+ targets, image_install=needle_pkg,
+ generate_sbom=True
+ )
+
+ for t in targets:
+ ds, pn, distro, machine = \
+ CIUtils.getVars('DEPLOY_DIR_SBOM', 'PN', 'DISTRO', 'MACHINE',
+ target=t)
+ for t in ["cdx", "spdx"]:
+ sbom_path = os.path.join(ds, f'{pn}-{distro}-{machine}.{t}.json')
+ self.log.info(f"Check {t} SBOM in {sbom_path}")
+ with open(sbom_path) as f:
+ sbom = json.load(f)
+ pkg_key = 'components' if t == 'cdx' else 'packages'
+ if not any(c for c in sbom[pkg_key] if c['name'] == needle_pkg):
+ self.fail(f'{needle_pkg} package not found in SBOM {sbom_path}')
+
def perform_sstate_populate(self, image_target, **kwargs):
# Use a different isar root for populating sstate cache
isar_sstate = f"{isar_root}/isar-sstate"
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 9c97115b..7538ade2 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -126,6 +126,7 @@ class CIBuilder(Test):
installer_distro=None,
installer_device=None,
customizations=None,
+ generate_sbom=False,
lines=None,
**kwargs,
):
@@ -176,6 +177,7 @@ class CIBuilder(Test):
f" image_install = {image_install}\n"
f" installer_image = {installer_image}\n"
f" customizations = {customizations}\n"
+ f" generate_sbom = {generate_sbom}\n"
f" lines = {strlines}\n"
f"==================================================="
)
@@ -275,6 +277,8 @@ class CIBuilder(Test):
'CUSTOMIZATION_FOR_IMAGES:append = " isar-image-ci"\n'
'HOSTNAME:isar-image-ci = "isar-ci"\n'
)
+ if generate_sbom is False:
+ f.write('ROOTFS_FEATURES:remove = "generate-sbom"\n')
if lines is not None:
f.writelines((line + '\n' if not line.endswith('\n') else line) for line in lines)
diff --git a/testsuite/citest.py b/testsuite/citest.py
index eaa4c440..d908f9bc 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -694,6 +694,39 @@ class CustomizationsTest(CIBaseTest):
)
+class SbomTest(CIBaseTest):
+ """
+ Test to check if sbom is generated and contains expected packages.
+ Most tests are rootfs tests to avoid costly initrd build and imaging.
+
+ :avocado: tags=sbom,fast
+ """
+
+ def test_sbom_rootfs_generate(self):
+ targets = [
+ 'mc:qemuamd64-bookworm:isar-rootfs-ci',
+ 'mc:qemuarm64-bookworm:isar-rootfs-ci',
+ 'mc:qemuamd64-trixie:isar-rootfs-ci',
+ 'mc:qemuarm64-trixie:isar-rootfs-ci',
+ 'mc:qemuamd64-noble:isar-rootfs-ci',
+ ]
+
+ self.init()
+ self.perform_sbom_test(targets)
+
+ def test_sbom_unsupported(self):
+ targets = [
+ 'mc:qemuamd64-bullseye:isar-rootfs-ci',
+ 'mc:qemuamd64-focal:isar-rootfs-ci',
+ ]
+
+ self.init()
+ self.perform_build_test(
+ targets, bitbake_cmd='do_rootfs', image_install='cowsay',
+ generate_sbom=True
+ )
+
+
class SignatureTest(CIBaseTest):
"""
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-16-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 16/20] testsuite: make sbuild-flavor test standalone
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (14 preceding siblings ...)
2026-01-23 8:24 ` [PATCH v3 15/20] testsuite: refactor sbom tests to avoid overhead Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 17/20] testsuite: skip VM tests if images are not available Zhihang Wei
` (5 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
As the hello-isar recipe acts as an example for the SBUILD_FLAVOR
feature, we also pull in a dedicated chroot into almost all tests. This
is very costly and does not add much value. We change this by setting
the SBUILD_FLAVOR of hello-isar to none in the CI layer and add a
dedicated test that just tests the SBUILD_FLAVOR feature.
This only slightly reduces the test coverage, but it significantly
speeds up the test execution.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
.../hello-isar/hello-isar.bbappend | 3 +++
.../recipes-app/libhello/libhello.bbappend | 5 +++++
testsuite/citest.py | 21 +++++++++++++++++++
3 files changed, 29 insertions(+)
create mode 100644 meta-test/recipes-app/libhello/libhello.bbappend
diff --git a/meta-test/recipes-app/hello-isar/hello-isar.bbappend b/meta-test/recipes-app/hello-isar/hello-isar.bbappend
index 44686458..27212262 100644
--- a/meta-test/recipes-app/hello-isar/hello-isar.bbappend
+++ b/meta-test/recipes-app/hello-isar/hello-isar.bbappend
@@ -15,3 +15,6 @@ SRC_URI:append = " \
"
SRC_URI:remove = "file://nonexist-file"
SRC_URI:remove = "git://nonexist-git"
+
+# avoid creating a dedicated sbuild chroot
+SBUILD_FLAVOR = ""
diff --git a/meta-test/recipes-app/libhello/libhello.bbappend b/meta-test/recipes-app/libhello/libhello.bbappend
new file mode 100644
index 00000000..3c88a741
--- /dev/null
+++ b/meta-test/recipes-app/libhello/libhello.bbappend
@@ -0,0 +1,5 @@
+# This software is a part of ISAR.
+# Copyright (C) Siemens
+
+# avoid creating a dedicated sbuild chroot
+SBUILD_FLAVOR = ""
diff --git a/testsuite/citest.py b/testsuite/citest.py
index d908f9bc..49fcdec0 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -131,6 +131,27 @@ class CompatTest(CIBaseTest):
self.perform_build_test(targets, compat_arch=True)
+class SbuildFlavor(CIBaseTest):
+ """
+ Test package build with a custom sbuild chroot.
+ :avocado: tags=sbuildflavor,fast
+ """
+
+ def test_sbuild_flavor(self):
+ targets = [
+ 'mc:qemuamd64-trixie:hello-isar',
+ 'mc:qemuarm64-trixie:hello-isar',
+ ]
+
+ lines = [
+ 'SBUILD_FLAVOR:hello-isar = "db2m"',
+ 'SBUILD_FLAVOR:libhello = "db2m"'
+ ]
+
+ self.init()
+ self.perform_build_test(targets, lines=lines)
+
+
class ReproTest(CIBaseTest):
"""
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-17-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 17/20] testsuite: skip VM tests if images are not available
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (15 preceding siblings ...)
2026-01-23 8:24 ` [PATCH v3 16/20] testsuite: make sbuild-flavor test standalone Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-28 11:54 ` Zhihang Wei
2026-01-23 8:24 ` [PATCH v3 18/20] testsuite: Group prebuilt_containers related test cases together Zhihang Wei
` (4 subsequent siblings)
21 siblings, 1 reply; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
We currently fail the test if the image is not available. Instead, we
now skip it and report what is missing. By that, we stay semantically
correct by not failing tests when pre-conditions are not fulfilled.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
testsuite/cibuilder.py | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 7538ade2..6faa9038 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -24,6 +24,7 @@ from utils import CIUtils
from avocado import Test
from avocado.utils import path
from avocado.utils import process
+from avocado.core import exceptions
sys.path.append(os.path.join(os.path.dirname(__file__), '../bitbake/lib'))
@@ -703,6 +704,7 @@ class CIBuilder(Test):
keep=False,
):
time_to_wait = self.params.get('time_to_wait', default=DEF_VM_TO_SEC)
+ self.skip_if_vm_image_missing(arch, distro, image)
self.log.info("===================================================")
self.log.info(f"Running Isar VM boot test for ({distro}-{arch})")
@@ -800,3 +802,20 @@ class CIBuilder(Test):
self.vm_turn_off(vm)
return stdout, stderr
+
+ def skip_if_vm_image_missing(self, arch, distro, image):
+ (
+ image_fstypes,
+ image_fullname,
+ deploy_dir_image,
+ ) = CIUtils.getVars(
+ 'IMAGE_FSTYPES',
+ 'IMAGE_FULLNAME',
+ 'DEPLOY_DIR_IMAGE',
+ target=f"mc:qemu{arch}-{distro}:{image}",
+ )
+ image_type = image_fstypes.split()[0]
+ rootfs_image = f"{image_fullname}.{image_type}"
+ rootfs_image_path = os.path.join(deploy_dir_image, rootfs_image)
+ if not os.path.exists(rootfs_image_path):
+ raise exceptions.TestSkipError(f'VM image missing: {rootfs_image}')
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-18-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v3 17/20] testsuite: skip VM tests if images are not available
2026-01-23 8:24 ` [PATCH v3 17/20] testsuite: skip VM tests if images are not available Zhihang Wei
@ 2026-01-28 11:54 ` Zhihang Wei
2026-01-28 12:46 ` 'MOESSBAUER, Felix' via isar-users
0 siblings, 1 reply; 25+ messages in thread
From: Zhihang Wei @ 2026-01-28 11:54 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: cedric.hombourger
On 1/23/26 09:24, Zhihang Wei wrote:
> From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
>
> We currently fail the test if the image is not available. Instead, we
> now skip it and report what is missing. By that, we stay semantically
> correct by not failing tests when pre-conditions are not fulfilled.
>
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
> testsuite/cibuilder.py | 19 +++++++++++++++++++
> 1 file changed, 19 insertions(+)
>
> diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
> index 7538ade2..6faa9038 100755
> --- a/testsuite/cibuilder.py
> +++ b/testsuite/cibuilder.py
> @@ -24,6 +24,7 @@ from utils import CIUtils
> from avocado import Test
> from avocado.utils import path
> from avocado.utils import process
> +from avocado.core import exceptions
>
> sys.path.append(os.path.join(os.path.dirname(__file__), '../bitbake/lib'))
>
> @@ -703,6 +704,7 @@ class CIBuilder(Test):
> keep=False,
> ):
> time_to_wait = self.params.get('time_to_wait', default=DEF_VM_TO_SEC)
> + self.skip_if_vm_image_missing(arch, distro, image)
>
> self.log.info("===================================================")
> self.log.info(f"Running Isar VM boot test for ({distro}-{arch})")
> @@ -800,3 +802,20 @@ class CIBuilder(Test):
> self.vm_turn_off(vm)
>
> return stdout, stderr
> +
> + def skip_if_vm_image_missing(self, arch, distro, image):
> + (
> + image_fstypes,
> + image_fullname,
> + deploy_dir_image,
> + ) = CIUtils.getVars(
> + 'IMAGE_FSTYPES',
> + 'IMAGE_FULLNAME',
> + 'DEPLOY_DIR_IMAGE',
> + target=f"mc:qemu{arch}-{distro}:{image}",
> + )
> + image_type = image_fstypes.split()[0]
> + rootfs_image = f"{image_fullname}.{image_type}"
> + rootfs_image_path = os.path.join(deploy_dir_image, rootfs_image)
> + if not os.path.exists(rootfs_image_path):
> + raise exceptions.TestSkipError(f'VM image missing: {rootfs_image}')
Hi,
this particular patch (p17) causes Avocado to crash because it encounters an
unrecognized log level. This is a known issue in Avocado that has been fixed
in more recent versions.
While we plan to update Avocado on our CI server, we would prefer to
partially
apply the patch set without p17 for now, as that patch is quite independent
of the others.
Zhihang
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/22e10b87-7601-47aa-a4be-b0ec6eb785bf%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v3 17/20] testsuite: skip VM tests if images are not available
2026-01-28 11:54 ` Zhihang Wei
@ 2026-01-28 12:46 ` 'MOESSBAUER, Felix' via isar-users
0 siblings, 0 replies; 25+ messages in thread
From: 'MOESSBAUER, Felix' via isar-users @ 2026-01-28 12:46 UTC (permalink / raw)
To: amikan, isar-users, wzh; +Cc: Hombourger, Cedric
On Wed, 2026-01-28 at 12:54 +0100, Zhihang Wei wrote:
> On 1/23/26 09:24, Zhihang Wei wrote:
> > From: "MOESSBAUER, Felix" <felix.moessbauer@siemens.com>
> >
> > We currently fail the test if the image is not available. Instead, we
> > now skip it and report what is missing. By that, we stay semantically
> > correct by not failing tests when pre-conditions are not fulfilled.
> >
> > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> > ---
> > testsuite/cibuilder.py | 19 +++++++++++++++++++
> > 1 file changed, 19 insertions(+)
> >
> > diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
> > index 7538ade2..6faa9038 100755
> > --- a/testsuite/cibuilder.py
> > +++ b/testsuite/cibuilder.py
> > @@ -24,6 +24,7 @@ from utils import CIUtils
> > from avocado import Test
> > from avocado.utils import path
> > from avocado.utils import process
> > +from avocado.core import exceptions
> >
> > sys.path.append(os.path.join(os.path.dirname(__file__), '../bitbake/lib'))
> >
> > @@ -703,6 +704,7 @@ class CIBuilder(Test):
> > keep=False,
> > ):
> > time_to_wait = self.params.get('time_to_wait', default=DEF_VM_TO_SEC)
> > + self.skip_if_vm_image_missing(arch, distro, image)
> >
> > self.log.info("===================================================")
> > self.log.info(f"Running Isar VM boot test for ({distro}-{arch})")
> > @@ -800,3 +802,20 @@ class CIBuilder(Test):
> > self.vm_turn_off(vm)
> >
> > return stdout, stderr
> > +
> > + def skip_if_vm_image_missing(self, arch, distro, image):
> > + (
> > + image_fstypes,
> > + image_fullname,
> > + deploy_dir_image,
> > + ) = CIUtils.getVars(
> > + 'IMAGE_FSTYPES',
> > + 'IMAGE_FULLNAME',
> > + 'DEPLOY_DIR_IMAGE',
> > + target=f"mc:qemu{arch}-{distro}:{image}",
> > + )
> > + image_type = image_fstypes.split()[0]
> > + rootfs_image = f"{image_fullname}.{image_type}"
> > + rootfs_image_path = os.path.join(deploy_dir_image, rootfs_image)
> > + if not os.path.exists(rootfs_image_path):
> > + raise exceptions.TestSkipError(f'VM image missing: {rootfs_image}')
> Hi,
>
> this particular patch (p17) causes Avocado to crash because it encounters an
> unrecognized log level. This is a known issue in Avocado that has been fixed
> in more recent versions.
>
> While we plan to update Avocado on our CI server, we would prefer to
> partially
> apply the patch set without p17 for now, as that patch is quite independent
> of the others.
That's fine for me. Once we update avocado, we still can re-apply the
patch.
Felix
>
> Zhihang
--
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/e8093f91afe1168aec9b79b53d0c49fd03dd0d79.camel%40siemens.com.
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v3 18/20] testsuite: Group prebuilt_containers related test cases together
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (16 preceding siblings ...)
2026-01-23 8:24 ` [PATCH v3 17/20] testsuite: skip VM tests if images are not available Zhihang Wei
@ 2026-01-23 8:24 ` Zhihang Wei
2026-01-23 8:25 ` [PATCH v3 19/20] testsuite: Execute startvm testcases after building Zhihang Wei
` (3 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:24 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
From: Anton Mikanovich <amikan@ilbers.de>
The prebuilt_containers related startvm test cases are now moved right
after where the artificates were built. Also, the prebuilt_containers
test case now build full images anyway, to avoid dependency on other
test suits. As the startvm tests were in full, this test suite now
belong to full.
Signed-off-by: Zhihang Wei <wzh@ilbers.de>
---
testsuite/citest.py | 35 +++++++++++++++++------------------
1 file changed, 17 insertions(+), 18 deletions(-)
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 49fcdec0..14230062 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -308,7 +308,7 @@ class CrossTest(CIBaseTest):
class PrebuiltTest(CIBaseTest):
"""
Tests associated with prebuilt artifacts (containers, debs).
- :avocado: tags=prebuilt,fast
+ :avocado: tags=prebuilt,full
"""
def test_prebuilt_containers(self):
@@ -320,9 +320,24 @@ class PrebuiltTest(CIBaseTest):
self.init()
self.perform_build_test(
targets,
- bitbake_cmd='do_rootfs_install',
image_install="prebuilt-docker-img prebuilt-podman-img")
+ def test_run_amd64_bookworm_prebuilt_containers(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('amd64', 'bookworm', image='isar-image-ci',
+ script='test_prebuilt_containers.sh')
+
+ def test_run_arm64_bookworm_prebuilt_containers(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('arm64', 'bookworm', image='isar-image-ci',
+ script='test_prebuilt_containers.sh')
+
class KernelTests(CIBaseTest):
"""
@@ -1074,22 +1089,6 @@ class VmBootTestFull(CIBaseTest):
except exceptions.TestFail:
self.cancel('KFAIL')
- def test_amd64_bookworm_prebuilt_containers(self):
- self.init()
- self.perform_build_test(
- ['mc:qemuamd64-bookworm:isar-image-ci'],
- image_install="prebuilt-docker-img prebuilt-podman-img")
- self.vm_start('amd64', 'bookworm', image='isar-image-ci',
- script='test_prebuilt_containers.sh')
-
- def test_arm64_bookworm_prebuilt_containers(self):
- self.init()
- self.perform_build_test(
- ['mc:qemuarm64-bookworm:isar-image-ci'],
- image_install="prebuilt-docker-img prebuilt-podman-img")
- self.vm_start('arm64', 'bookworm', image='isar-image-ci',
- script='test_prebuilt_containers.sh')
-
def test_amd64_bookworm_iso(self):
self.init()
self.vm_start('amd64-iso', 'bookworm', image='isar-image-ci',
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-19-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 19/20] testsuite: Execute startvm testcases after building
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (17 preceding siblings ...)
2026-01-23 8:24 ` [PATCH v3 18/20] testsuite: Group prebuilt_containers related test cases together Zhihang Wei
@ 2026-01-23 8:25 ` Zhihang Wei
2026-01-23 8:25 ` [PATCH v3 20/20] testsuite: Remove SstateTest from full testsuite Zhihang Wei
` (2 subsequent siblings)
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:25 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
Move startvm testcases up to be executed right after following targets
got built. Tag them as startvm to allow dsabling with '-startvm'.
Signed-off-by: Anton Mikanovich amikan@ilbers.de
Signed-off-by: Zhihang Wei <wzh@ilbers.de>
---
testsuite/citest.py | 588 +++++++++++++++++++++++++-------------------
1 file changed, 337 insertions(+), 251 deletions(-)
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 14230062..7912566f 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -248,6 +248,102 @@ class CrossTest(CIBaseTest):
self.init()
self.perform_build_test(targets)
+ def test_run_arm_bullseye_base(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('arm', 'bullseye', image='isar-image-ci', keep=True)
+
+ def test_run_arm_bullseye_example_module(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start(
+ 'arm',
+ 'bullseye',
+ image='isar-image-ci',
+ cmd='lsmod | grep example_module',
+ keep=True,
+ )
+
+ def test_run_arm_bullseye_getty_target(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start(
+ 'arm',
+ 'bullseye',
+ image='isar-image-ci',
+ script='test_systemd_unit.sh getty.target 10',
+ )
+
+ def test_run_arm_buster_base(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('arm', 'buster', image='isar-image-ci', keep=True)
+
+ def test_run_arm_buster_getty_target(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start(
+ 'arm',
+ 'buster',
+ image='isar-image-ci',
+ cmd='systemctl is-active getty.target',
+ keep=True,
+ )
+
+ def test_run_arm_buster_example_module(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start(
+ 'arm',
+ 'buster',
+ image='isar-image-ci',
+ script='test_kernel_module.sh example_module',
+ )
+
+ def test_run_arm_bookworm_base(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('arm', 'bookworm', image='isar-image-ci', keep=True)
+
+ def test_run_arm_bookworm_example_module(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start(
+ 'arm',
+ 'bookworm',
+ image='isar-image-ci',
+ cmd='lsmod | grep example_module',
+ keep=True,
+ )
+
+ def test_run_arm_bookworm_getty_target(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start(
+ 'arm',
+ 'bookworm',
+ image='isar-image-ci',
+ script='test_systemd_unit.sh getty.target 10',
+ )
+
def test_cross_debsrc(self):
targets = [
'mc:qemuarm64-bookworm:isar-image-ci',
@@ -269,6 +365,20 @@ class CrossTest(CIBaseTest):
except exceptions.TestFail:
self.cancel('KFAIL')
+ def test_run_amd64_trixie(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('amd64', 'trixie')
+
+ def test_run_arm64_trixie(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('arm64', 'trixie')
+
def test_cross_kselftest(self):
targets = [
'mc:qemuarm-buster:kselftest',
@@ -612,6 +722,195 @@ class NoCrossTest(CIBaseTest):
self.init()
self.perform_build_test(targets, cross=False)
+ def test_run_arm_bullseye(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('arm', 'bullseye')
+
+ def test_run_arm_buster_base(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('arm', 'buster', image='isar-image-ci', keep=True)
+
+ def test_run_arm_buster_example_module(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start(
+ 'arm',
+ 'buster',
+ image='isar-image-ci',
+ cmd='lsmod | grep example_module',
+ keep=True,
+ )
+
+ def test_run_arm_buster_getty_target(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start(
+ 'arm',
+ 'buster',
+ image='isar-image-ci',
+ script='test_systemd_unit.sh getty.target 10',
+ )
+
+ def test_run_arm64_bookworm_base(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('arm64', 'bookworm', image='isar-image-ci', keep=True)
+
+ def test_run_arm64_bookworm_getty_target(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start(
+ 'arm64',
+ 'bookworm',
+ image='isar-image-ci',
+ cmd='systemctl is-active getty.target',
+ keep=True,
+ )
+
+ def test_run_arm64_bookworm_example_module(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start(
+ 'arm64',
+ 'bookworm',
+ image='isar-image-ci',
+ script='test_kernel_module.sh example_module',
+ )
+
+ def test_run_i386_buster(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('i386', 'buster')
+
+ def test_run_amd64_buster_base(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ # test efi boot
+ self.vm_start('amd64', 'buster', image='isar-image-ci')
+
+ def test_run_amd64_buster_pcbios(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ # test pcbios boot
+ self.vm_start('amd64', 'buster', True, image='isar-image-ci')
+
+ def test_run_amd64_focal_base(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('amd64', 'focal', image='isar-image-ci', keep=True)
+
+ def test_run_amd64_focal_example_module(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start(
+ 'amd64',
+ 'focal',
+ image='isar-image-ci',
+ cmd='lsmod | grep example_module',
+ keep=True,
+ )
+
+ def test_run_amd64_focal_getty_target(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start(
+ 'amd64',
+ 'focal',
+ image='isar-image-ci',
+ script='test_systemd_unit.sh getty.target 10',
+ )
+
+ def test_run_amd64_bookworm(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('amd64', 'bookworm', image='isar-image-ci')
+
+ def test_run_i386_bookworm(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('i386', 'bookworm')
+
+ def test_run_mipsel_bookworm_base(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('mipsel', 'bookworm', image='isar-image-ci', keep=True)
+
+ def test_run_mipsel_bookworm_getty_target(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start(
+ 'mipsel',
+ 'bookworm',
+ image='isar-image-ci',
+ cmd='systemctl is-active getty.target',
+ keep=True,
+ )
+
+ def test_run_mipsel_bookworm_example_module(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start(
+ 'mipsel',
+ 'bookworm',
+ image='isar-image-ci',
+ script='test_kernel_module.sh example_module',
+ )
+
+ def test_run_amd64_bookworm_iso_base(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('amd64-iso', 'bookworm', image='isar-image-ci',
+ keep=True
+ )
+
+ def test_run_amd64_bookworm_iso_system_check(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('amd64-iso', 'bookworm', image='isar-image-ci',
+ script='test_system_running.sh 30')
+
def test_nocross_debsrc(self):
targets = [
'mc:qemuarm-bookworm:isar-image-ci',
@@ -622,6 +921,13 @@ class NoCrossTest(CIBaseTest):
self.init()
self.perform_build_test(targets, cross=False, debsrc_cache=True)
+ def test_run_arm_bookworm(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('arm', 'bookworm', image='isar-image-ci')
+
def test_nocross_rpi(self):
targets = [
'mc:rpi-arm-bullseye:isar-image-base',
@@ -658,6 +964,37 @@ class NoCrossTest(CIBaseTest):
except exceptions.TestFail:
self.cancel('KFAIL')
+ def test_run_arm_trixie(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ try:
+ self.vm_start('arm', 'trixie')
+ except exceptions.TestFail:
+ self.cancel('KFAIL')
+
+ def test_run_arm64_trixie(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('arm64', 'trixie')
+
+ def test_run_amd64_trixie(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('amd64', 'trixie')
+
+ def test_run_riscv64_trixie(self):
+ """
+ :avocado: tags=startvm
+ """
+ self.init()
+ self.vm_start('riscv64', 'trixie')
+
def test_nocross_sid(self):
targets = [
'mc:qemuamd64-sid:isar-image-base',
@@ -850,257 +1187,6 @@ class SourceTest(CIBaseTest):
self.perform_source_test(targets)
-class VmBootTestFast(CIBaseTest):
-
- """
- Test QEMU image start (fast)
-
- :avocado: tags=startvm,fast
- """
-
- def test_arm_bullseye(self):
- self.init()
- self.vm_start('arm', 'bullseye', image='isar-image-ci', keep=True)
-
- def test_arm_bullseye_example_module(self):
- self.init()
- self.vm_start(
- 'arm',
- 'bullseye',
- image='isar-image-ci',
- cmd='lsmod | grep example_module',
- keep=True,
- )
-
- def test_arm_bullseye_getty_target(self):
- self.init()
- self.vm_start(
- 'arm',
- 'bullseye',
- image='isar-image-ci',
- script='test_systemd_unit.sh getty.target 10',
- )
-
- def test_arm_buster(self):
- self.init()
- self.vm_start('arm', 'buster', image='isar-image-ci', keep=True)
-
- def test_arm_buster_getty_target(self):
- self.init()
- self.vm_start(
- 'arm',
- 'buster',
- image='isar-image-ci',
- cmd='systemctl is-active getty.target',
- keep=True,
- )
-
- def test_arm_buster_example_module(self):
- self.init()
- self.vm_start(
- 'arm',
- 'buster',
- image='isar-image-ci',
- script='test_kernel_module.sh example_module',
- )
-
- def test_arm_bookworm(self):
- self.init()
- self.vm_start('arm', 'bookworm', image='isar-image-ci', keep=True)
-
- def test_arm_bookworm_example_module(self):
- self.init()
- self.vm_start(
- 'arm',
- 'bookworm',
- image='isar-image-ci',
- cmd='lsmod | grep example_module',
- keep=True,
- )
-
- def test_arm_bookworm_getty_target(self):
- self.init()
- self.vm_start(
- 'arm',
- 'bookworm',
- image='isar-image-ci',
- script='test_systemd_unit.sh getty.target 10',
- )
-
- def test_amd64_trixie(self):
- self.init()
- self.vm_start('amd64', 'trixie')
-
- def test_arm64_trixie(self):
- self.init()
- self.vm_start('arm64', 'trixie')
-
-
-class VmBootTestFull(CIBaseTest):
-
- """
- Test QEMU image start (full)
-
- :avocado: tags=startvm,full
- """
-
- def test_arm_bullseye(self):
- self.init()
- self.vm_start('arm', 'bullseye')
-
- def test_arm_buster(self):
- self.init()
- self.vm_start('arm', 'buster', image='isar-image-ci', keep=True)
-
- def test_arm_buster_example_module(self):
- self.init()
- self.vm_start(
- 'arm',
- 'buster',
- image='isar-image-ci',
- cmd='lsmod | grep example_module',
- keep=True,
- )
-
- def test_arm_buster_getty_target(self):
- self.init()
- self.vm_start(
- 'arm',
- 'buster',
- image='isar-image-ci',
- script='test_systemd_unit.sh getty.target 10',
- )
-
- def test_arm_trixie(self):
- self.init()
- try:
- self.vm_start('arm', 'trixie')
- except exceptions.TestFail:
- self.cancel('KFAIL')
-
- def test_arm64_bookworm(self):
- self.init()
- self.vm_start('arm64', 'bookworm', image='isar-image-ci', keep=True)
-
- def test_arm64_bookworm_getty_target(self):
- self.init()
- self.vm_start(
- 'arm64',
- 'bookworm',
- image='isar-image-ci',
- cmd='systemctl is-active getty.target',
- keep=True,
- )
-
- def test_arm64_bookworm_example_module(self):
- self.init()
- self.vm_start(
- 'arm64',
- 'bookworm',
- image='isar-image-ci',
- script='test_kernel_module.sh example_module',
- keep=True,
- )
-
- def test_arm64_trixie(self):
- self.init()
- self.vm_start('arm64', 'trixie')
-
- def test_i386_buster(self):
- self.init()
- self.vm_start('i386', 'buster')
-
- def test_amd64_buster(self):
- self.init()
- # test efi boot
- self.vm_start('amd64', 'buster', image='isar-image-ci')
-
- def test_amd64_buster_pcbios(self):
- self.init()
- # test pcbios boot
- self.vm_start('amd64', 'buster', True, image='isar-image-ci')
-
- def test_amd64_focal(self):
- self.init()
- self.vm_start('amd64', 'focal', image='isar-image-ci', keep=True)
-
- def test_amd64_focal_example_module(self):
- self.init()
- self.vm_start(
- 'amd64',
- 'focal',
- image='isar-image-ci',
- cmd='lsmod | grep example_module',
- keep=True,
- )
-
- def test_amd64_focal_getty_target(self):
- self.init()
- self.vm_start(
- 'amd64',
- 'focal',
- image='isar-image-ci',
- script='test_systemd_unit.sh getty.target 10',
- )
-
- def test_amd64_bookworm(self):
- self.init()
- self.vm_start('amd64', 'bookworm', image='isar-image-ci', keep=True)
-
- def test_arm_bookworm(self):
- self.init()
- self.vm_start('arm', 'bookworm', image='isar-image-ci')
-
- def test_i386_bookworm(self):
- self.init()
- self.vm_start('i386', 'bookworm')
-
- def test_mipsel_bookworm(self):
- self.init()
- self.vm_start('mipsel', 'bookworm', image='isar-image-ci', keep=True)
-
- def test_amd64_trixie(self):
- self.init()
- self.vm_start('amd64', 'trixie')
-
- def test_mipsel_bookworm_getty_target(self):
- self.init()
- self.vm_start(
- 'mipsel',
- 'bookworm',
- image='isar-image-ci',
- cmd='systemctl is-active getty.target',
- keep=True,
- )
-
- def test_mipsel_bookworm_example_module(self):
- self.init()
- self.vm_start(
- 'mipsel',
- 'bookworm',
- image='isar-image-ci',
- script='test_kernel_module.sh example_module',
- )
-
- def test_riscv64_trixie(self):
- self.init()
- try:
- self.vm_start('riscv64', 'trixie')
- except exceptions.TestFail:
- self.cancel('KFAIL')
-
- def test_amd64_bookworm_iso(self):
- self.init()
- self.vm_start('amd64-iso', 'bookworm', image='isar-image-ci',
- keep = True
- )
-
- def test_amd64_bookworm_iso_system_check(self):
- self.init()
- self.vm_start('amd64-iso', 'bookworm', image='isar-image-ci',
- script='test_system_running.sh 30')
-
-
class World(CIBaseTest):
"""
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-20-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v3 20/20] testsuite: Remove SstateTest from full testsuite
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (18 preceding siblings ...)
2026-01-23 8:25 ` [PATCH v3 19/20] testsuite: Execute startvm testcases after building Zhihang Wei
@ 2026-01-23 8:25 ` Zhihang Wei
2026-01-23 8:37 ` [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
2026-01-28 13:01 ` Zhihang Wei
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:25 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: wzh, cedric.hombourger
The SstateTest uses a different directory than other test cases and
fails after "testsuite: fix SignatureTest by avoiding absolute path in
bblayers". Since sstate is used heavily and has lower priotity on
testing, we remove SstateTest from full right now.
Signed-off-by: Zhihang Wei <wzh@ilbers.de>
---
testsuite/citest.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 7912566f..f11cd5af 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -1129,7 +1129,7 @@ class SstateTest(CIBaseTest):
"""
Test builds with artifacts taken from sstate cache
- :avocado: tags=sstate,full
+ :avocado: tags=sstate
"""
def test_sstate_populate(self):
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-21-wzh%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v3 00/20] Various improvements to the testsuite
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (19 preceding siblings ...)
2026-01-23 8:25 ` [PATCH v3 20/20] testsuite: Remove SstateTest from full testsuite Zhihang Wei
@ 2026-01-23 8:37 ` Zhihang Wei
2026-01-28 13:01 ` Zhihang Wei
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-23 8:37 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: cedric.hombourger
On 1/23/26 09:24, Zhihang Wei wrote:
> This series aims at simplifying and speeding up the testsuite execution.
> This should enable contributors to actually run it and test their patches.
>
> As pointed out in [1], the testsuite currently tests various features over
> and over again, while others are not tested at all. We try to improve this
> by focusing on "feature" tests, whereby each test case checks a set of
> features instead of a whole image. We further identified long running tasks
> (like the kernel builds) and move them to a dedicated test.
>
> The execution time reduction varies greatly depending on the hardware,
> network and sstate usage. When applying these changes on recently added
> patches, the time was often decreased by more than 50% on "fast".
>
> Note, that more refactorings of the testsuite are needed - and planned,
> but this series should already address the biggest pain points.
>
> [1] https://groups.google.com/g/isar-users/c/uZBTIHzLp8Q/m/8k0L1vfMDAAJ
>
> Changes since v2:
>
> - move startvm tests after each build to avoid inter-testsuite dependency
> - remove SstateTest from full
v3 has passed fast and full CI.
Zhihang
> Changes since v1:
>
> - continued work on testsuite refactoring, this supersedes:
> - testsuite: fix SignatureTest by avoiding absolute path in bblayers
> - testsuite: add rootfs target for rootfs only tests
> - Enable SState cache in GitLab CI
> - testsuite: fix typo in log message in perform_signature_lint
> - completed exclusion / refactoring of db2m sbuild chroot
> - GitLab CI: increased after script timeout to give sstate upload more time
> - GitLab CI: do not upload sstate artifacts of images (as almost no reuse)
> - do not add SignatureTest to fast tag (as otherwise VM tests break...)
> - rebuild container test images in VM test to not depend on test order
> - rebased onto next
>
> Anton Mikanovich (1):
> testsuite: Group prebuilt_containers related test cases together
>
> MOESSBAUER, Felix (17):
> testsuite: move targets with custom kernel to separate test
> testsuite: enable ccache on kernel tests
> testsuite: make prebuilt container a feature test
> testsuite: make compat test standalone test
> testsuite: handle IMAGE_INSTALL solely in cibuilder.py
> testsuite: limit cross_debsrc test to subset of packages
> testsuite: forward SSTATE_MIRRORS into CI env on sstate
> ci: add support for sstate cache
> testsuite: make test_cross_deps more specific
> testsuite: fix typo in log message in perform_signature_lint
> testsuite: fix SignatureTest by avoiding absolute path in bblayers
> testsuite: use more recent distros in SignatureTest
> testsuite: make SignatureTest idempotent
> testsuite: add rootfs target for rootfs only tests
> testsuite: refactor sbom tests to avoid overhead
> testsuite: make sbuild-flavor test standalone
> testsuite: skip VM tests if images are not available
>
> Zhihang Wei (2):
> testsuite: Execute startvm testcases after building
> testsuite: Remove SstateTest from full testsuite
>
> .gitlab-ci.yml | 36 +-
> meta-test/conf/bblayers.conf.sample | 8 +-
> meta-test/conf/local.conf.sample | 10 -
> .../hello-isar/hello-isar.bbappend | 3 +
> .../recipes-app/libhello/libhello.bbappend | 5 +
> .../recipes-core/images/isar-image-ci.bb | 2 -
> .../recipes-core/images/isar-rootfs-ci.bb | 17 +
> testsuite/cibase.py | 34 +-
> testsuite/cibuilder.py | 49 +-
> testsuite/citest.py | 740 +++++++++++-------
> 10 files changed, 602 insertions(+), 302 deletions(-)
> create mode 100644 meta-test/recipes-app/libhello/libhello.bbappend
> create mode 100644 meta-test/recipes-core/images/isar-rootfs-ci.bb
>
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/27f0e44f-5f2f-4cd2-990d-c232bda4ff38%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v3 00/20] Various improvements to the testsuite
2026-01-23 8:24 [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
` (20 preceding siblings ...)
2026-01-23 8:37 ` [PATCH v3 00/20] Various improvements to the testsuite Zhihang Wei
@ 2026-01-28 13:01 ` Zhihang Wei
21 siblings, 0 replies; 25+ messages in thread
From: Zhihang Wei @ 2026-01-28 13:01 UTC (permalink / raw)
To: isar-users, felix.moessbauer, amikan; +Cc: cedric.hombourger
p1-16 and p18-20 were applied to next. Thanks.
Zhihang
On 1/23/26 09:24, Zhihang Wei wrote:
> This series aims at simplifying and speeding up the testsuite execution.
> This should enable contributors to actually run it and test their patches.
>
> As pointed out in [1], the testsuite currently tests various features over
> and over again, while others are not tested at all. We try to improve this
> by focusing on "feature" tests, whereby each test case checks a set of
> features instead of a whole image. We further identified long running tasks
> (like the kernel builds) and move them to a dedicated test.
>
> The execution time reduction varies greatly depending on the hardware,
> network and sstate usage. When applying these changes on recently added
> patches, the time was often decreased by more than 50% on "fast".
>
> Note, that more refactorings of the testsuite are needed - and planned,
> but this series should already address the biggest pain points.
>
> [1] https://groups.google.com/g/isar-users/c/uZBTIHzLp8Q/m/8k0L1vfMDAAJ
>
> Changes since v2:
>
> - move startvm tests after each build to avoid inter-testsuite dependency
> - remove SstateTest from full
>
> Changes since v1:
>
> - continued work on testsuite refactoring, this supersedes:
> - testsuite: fix SignatureTest by avoiding absolute path in bblayers
> - testsuite: add rootfs target for rootfs only tests
> - Enable SState cache in GitLab CI
> - testsuite: fix typo in log message in perform_signature_lint
> - completed exclusion / refactoring of db2m sbuild chroot
> - GitLab CI: increased after script timeout to give sstate upload more time
> - GitLab CI: do not upload sstate artifacts of images (as almost no reuse)
> - do not add SignatureTest to fast tag (as otherwise VM tests break...)
> - rebuild container test images in VM test to not depend on test order
> - rebased onto next
>
> Anton Mikanovich (1):
> testsuite: Group prebuilt_containers related test cases together
>
> MOESSBAUER, Felix (17):
> testsuite: move targets with custom kernel to separate test
> testsuite: enable ccache on kernel tests
> testsuite: make prebuilt container a feature test
> testsuite: make compat test standalone test
> testsuite: handle IMAGE_INSTALL solely in cibuilder.py
> testsuite: limit cross_debsrc test to subset of packages
> testsuite: forward SSTATE_MIRRORS into CI env on sstate
> ci: add support for sstate cache
> testsuite: make test_cross_deps more specific
> testsuite: fix typo in log message in perform_signature_lint
> testsuite: fix SignatureTest by avoiding absolute path in bblayers
> testsuite: use more recent distros in SignatureTest
> testsuite: make SignatureTest idempotent
> testsuite: add rootfs target for rootfs only tests
> testsuite: refactor sbom tests to avoid overhead
> testsuite: make sbuild-flavor test standalone
> testsuite: skip VM tests if images are not available
>
> Zhihang Wei (2):
> testsuite: Execute startvm testcases after building
> testsuite: Remove SstateTest from full testsuite
>
> .gitlab-ci.yml | 36 +-
> meta-test/conf/bblayers.conf.sample | 8 +-
> meta-test/conf/local.conf.sample | 10 -
> .../hello-isar/hello-isar.bbappend | 3 +
> .../recipes-app/libhello/libhello.bbappend | 5 +
> .../recipes-core/images/isar-image-ci.bb | 2 -
> .../recipes-core/images/isar-rootfs-ci.bb | 17 +
> testsuite/cibase.py | 34 +-
> testsuite/cibuilder.py | 49 +-
> testsuite/citest.py | 740 +++++++++++-------
> 10 files changed, 602 insertions(+), 302 deletions(-)
> create mode 100644 meta-test/recipes-app/libhello/libhello.bbappend
> create mode 100644 meta-test/recipes-core/images/isar-rootfs-ci.bb
>
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/ced9d412-3c6a-46c4-ba1d-088baf8cebf1%40ilbers.de.
^ permalink raw reply [flat|nested] 25+ messages in thread