* [PATCH 2/3] testsuite: refactor sbom tests to avoid overhead
2025-12-18 15:53 [PATCH 1/3] testsuite: add rootfs target for rootfs only tests 'Felix Moessbauer' via isar-users
@ 2025-12-18 15:53 ` 'Felix Moessbauer' via isar-users
2025-12-18 15:53 ` [PATCH 3/3] testsuite: make sbuild-flavor test standalone 'Felix Moessbauer' via isar-users
1 sibling, 0 replies; 3+ messages in thread
From: 'Felix Moessbauer' via isar-users @ 2025-12-18 15:53 UTC (permalink / raw)
To: isar-users; +Cc: Felix Moessbauer
We currently test the SBOM infrastructure in all image builds, which
adds a significant overhead. We now change this to not generate SBOMs in
general (and by that avoid building the dependencies). To not have a
testing gap, we add a dedicated SBOM test that checks the SBOM creation
for various targets. In addition, we now also check the content of the
SBOM for plausibility.
In the future, the SBOM test can be extended without slowing down the
overall test execution.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
testsuite/cibase.py | 26 ++++++++++++++++++++++++++
testsuite/cibuilder.py | 4 ++++
testsuite/citest.py | 33 +++++++++++++++++++++++++++++++++
3 files changed, 63 insertions(+)
diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index 5ef1a5b5..fd6a3df9 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -140,6 +140,32 @@ class CIBaseTest(CIBuilder):
self.delete_from_build_dir('ccache')
self.unconfigure()
+ def perform_sbom_test(self, targets, **kwargs):
+ """
+ Build a rootfs containing a needle package and check if that package
+ is added to the sbom.
+ """
+ import json
+
+ needle_pkg = 'cowsay'
+ self.perform_build_test(
+ targets, image_install=needle_pkg,
+ generate_sbom=True
+ )
+
+ for t in targets:
+ ds, pn, distro, machine = \
+ CIUtils.getVars('DEPLOY_DIR_SBOM', 'PN', 'DISTRO', 'MACHINE',
+ target=t)
+ for t in ["cdx", "spdx"]:
+ sbom_path = os.path.join(ds, f'{pn}-{distro}-{machine}.{t}.json')
+ self.log.info(f"Check {t} SBOM in {sbom_path}")
+ with open(sbom_path) as f:
+ sbom = json.load(f)
+ pkg_key = 'components' if t == 'cdx' else 'packages'
+ if not any(c for c in sbom[pkg_key] if c['name'] == needle_pkg):
+ self.fail(f'{needle_pkg} package not found in SBOM {sbom_path}')
+
def perform_sstate_populate(self, image_target, **kwargs):
# Use a different isar root for populating sstate cache
isar_sstate = f"{isar_root}/isar-sstate"
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 310a3836..614a3397 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -126,6 +126,7 @@ class CIBuilder(Test):
installer_distro=None,
installer_device=None,
customizations=None,
+ generate_sbom=False,
lines=None,
**kwargs,
):
@@ -176,6 +177,7 @@ class CIBuilder(Test):
f" image_install = {image_install}\n"
f" installer_image = {installer_image}\n"
f" customizations = {customizations}\n"
+ f" generate_sbom = {generate_sbom}\n"
f" lines = {strlines}\n"
f"==================================================="
)
@@ -275,6 +277,8 @@ class CIBuilder(Test):
'CUSTOMIZATION_FOR_IMAGES:append = " isar-image-ci"\n'
'HOSTNAME:isar-image-ci = "isar-ci"\n'
)
+ if generate_sbom is False:
+ f.write('ROOTFS_FEATURES:remove = "generate-sbom"\n')
if lines is not None:
f.writelines((line + '\n' if not line.endswith('\n') else line) for line in lines)
diff --git a/testsuite/citest.py b/testsuite/citest.py
index cc2bdf41..2b3efa4a 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -692,6 +692,39 @@ class CustomizationsTest(CIBaseTest):
)
+class SbomTest(CIBaseTest):
+ """
+ Test to check if sbom is generated and contains expected packages.
+ Most tests are rootfs tests to avoid costly initrd build and imaging.
+
+ :avocado: tags=sbom,fast
+ """
+
+ def test_sbom_rootfs_generate(self):
+ targets = [
+ 'mc:qemuamd64-bookworm:isar-rootfs-ci',
+ 'mc:qemuarm64-bookworm:isar-rootfs-ci',
+ 'mc:qemuamd64-trixie:isar-rootfs-ci',
+ 'mc:qemuarm64-trixie:isar-rootfs-ci',
+ 'mc:qemuamd64-noble:isar-rootfs-ci',
+ ]
+
+ self.init()
+ self.perform_sbom_test(targets)
+
+ def test_sbom_unsupported(self):
+ targets = [
+ 'mc:qemuamd64-bullseye:isar-rootfs-ci',
+ 'mc:qemuamd64-focal:isar-rootfs-ci',
+ ]
+
+ self.init()
+ self.perform_build_test(
+ targets, bitbake_cmd='do_rootfs', image_install='cowsay',
+ generate_sbom=True
+ )
+
+
class SignatureTest(CIBaseTest):
"""
--
2.51.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251218155347.9929-2-felix.moessbauer%40siemens.com.
^ permalink raw reply [flat|nested] 3+ messages in thread* [PATCH 3/3] testsuite: make sbuild-flavor test standalone
2025-12-18 15:53 [PATCH 1/3] testsuite: add rootfs target for rootfs only tests 'Felix Moessbauer' via isar-users
2025-12-18 15:53 ` [PATCH 2/3] testsuite: refactor sbom tests to avoid overhead 'Felix Moessbauer' via isar-users
@ 2025-12-18 15:53 ` 'Felix Moessbauer' via isar-users
1 sibling, 0 replies; 3+ messages in thread
From: 'Felix Moessbauer' via isar-users @ 2025-12-18 15:53 UTC (permalink / raw)
To: isar-users; +Cc: Felix Moessbauer
As the hello-isar recipe acts as an example for the SBUILD_FLAVOR
feature, we also pull in a dedicated chroot into almost all tests. This
is very costly and does not add much value. We change this by setting
the SBUILD_FLAVOR of hello-isar to none in the CI layer and add a
dedicated test that just tests the SBUILD_FLAVOR feature.
This only slightly reduces the test coverage, but it significantly
speeds up the test execution.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
.../hello-isar/hello-isar.bbappend | 3 +++
testsuite/citest.py | 20 +++++++++++++++++++
2 files changed, 23 insertions(+)
diff --git a/meta-test/recipes-app/hello-isar/hello-isar.bbappend b/meta-test/recipes-app/hello-isar/hello-isar.bbappend
index 44686458..27212262 100644
--- a/meta-test/recipes-app/hello-isar/hello-isar.bbappend
+++ b/meta-test/recipes-app/hello-isar/hello-isar.bbappend
@@ -15,3 +15,6 @@ SRC_URI:append = " \
"
SRC_URI:remove = "file://nonexist-file"
SRC_URI:remove = "git://nonexist-git"
+
+# avoid creating a dedicated sbuild chroot
+SBUILD_FLAVOR = ""
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 2b3efa4a..3ec5e824 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -131,6 +131,26 @@ class CompatTest(CIBaseTest):
self.perform_build_test(targets, compat_arch=True)
+class SbuildFlavor(CIBaseTest):
+ """
+ Test package build with a custom sbuild chroot.
+ :avocado: tags=sbuildflavor,fast
+ """
+
+ def test_sbuild_flavor(self):
+ targets = [
+ 'mc:qemuamd64-trixie:hello-isar',
+ 'mc:qemuarm64-trixie:hello-isar',
+ ]
+
+ lines = [
+ 'SBUILD_FLAVOR:hello-isar = "db2m"'
+ ]
+
+ self.init()
+ self.perform_build_test(targets, lines=lines)
+
+
class ReproTest(CIBaseTest):
"""
--
2.51.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251218155347.9929-3-felix.moessbauer%40siemens.com.
^ permalink raw reply [flat|nested] 3+ messages in thread