[PATCH 0/3] Add 'make world' test case

[PATCH 0/3] Add 'make world' test case

[Anton Mikanovich]

To be able to test all the targets currently supported by Isar we need
to add separate test case which can build (and run in qemu if possible)
everything with single command.

There are 2 running options possible:

1. Run all the targets with single bitbake command.
This is the fastest way with sharing build dirs between targets.

2. Run every target in separate build dir, one target per testcase.
This allows to check every target separately, but requires external
yaml with target list and varianter plugin for avocado.

Also fix container example-module usage by default for targets, so this
machine can be built without excluding example-module manually.

Anton Mikanovich (3):
  CI: Remove container package list hack from testsuite
  CI: Add make world test case
  CI: Run startvm in world test case

 .gitignore                       |   2 +-
 meta-isar/conf/local.conf.sample |   4 +
 meta-test/conf/local.conf.sample |   3 +
 scripts/ci_build.sh              |   8 +-
 testsuite/cibuilder.py           |   4 -
 testsuite/citest.py              |  55 +++++++++++++
 testsuite/data/targets.yml       | 132 +++++++++++++++++++++++++++++++
 testsuite/targets_gen.py         |  13 +++
 testsuite/utils.py               |  35 +++++++-
 9 files changed, 249 insertions(+), 7 deletions(-)
 create mode 100644 testsuite/data/targets.yml
 create mode 100755 testsuite/targets_gen.py

-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250512083238.2089774-1-amikan%40ilbers.de.

[PATCH 1/3] CI: Remove container package list hack from testsuite

[Anton Mikanovich]

As IMAGE_INSTALL list is set in local.conf removing items should be
also done in local.conf.
This will allow to build container targets just like any other.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta-isar/conf/local.conf.sample | 4 ++++
 meta-test/conf/local.conf.sample | 3 +++
 testsuite/cibuilder.py           | 4 ----
 3 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 52ea2fb5..9bc95dfd 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -154,6 +154,10 @@ CONF_VERSION = "1"
 # The default list of extra packages to be installed.
 IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs samefile hello isar-disable-apt-cache cowsay example-prebuilt"
 
+#
+# Container machines don't need example module and enable-fsck.
+IMAGE_INSTALL:remove:container-amd64 = "example-module-${KERNEL_NAME} enable-fsck"
+
 #
 # Machines with secure boot should use signed modules.
 IMAGE_INSTALL:remove:qemuamd64-sb = "example-module-${KERNEL_NAME}"
diff --git a/meta-test/conf/local.conf.sample b/meta-test/conf/local.conf.sample
index f692f533..092d6ba6 100644
--- a/meta-test/conf/local.conf.sample
+++ b/meta-test/conf/local.conf.sample
@@ -30,6 +30,9 @@ MIRRORS += "https?://cdn\.kernel\.org/.* https://mirrors.edge.kernel.org/PATH"
 # The default list of extra packages
 IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs samefile hello isar-disable-apt-cache cowsay example-prebuilt"
 
+# Container machines don't need example module and enable-fsck.
+IMAGE_INSTALL:remove:container-amd64 = "example-module-${KERNEL_NAME} enable-fsck"
+
 # Machines with secure boot should use signed modules
 IMAGE_INSTALL:remove:qemuamd64-sb = "example-module-${KERNEL_NAME}"
 IMAGE_INSTALL:append:qemuamd64-sb = " example-module-signed-${KERNEL_NAME}"
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index c2520f7b..7d702b35 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -199,10 +199,6 @@ class CIBuilder(Test):
                 f.write('BB_NO_NETWORK = "1"\n')
             if container:
                 f.write('SDK_FORMATS = "docker-archive"\n')
-                f.write(
-                    'IMAGE_INSTALL:remove = '
-                    '"example-module-${KERNEL_NAME} enable-fsck"\n'
-                )
             if gpg_pub_key:
                 f.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
             if wic_deploy_parts:
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250512083238.2089774-2-amikan%40ilbers.de.

[PATCH 2/3] CI: Add make world test case

[Anton Mikanovich]

Add test case which builds all available targets.

Single bitbake usage (directly from mc.conf):
$ avocado run testsuite/citest.py -t world

Multiple bitbakes usage (from generated yaml file):
$ avocado run testsuite/citest.py -t world -m testsuite/data/targets.yml

Or the same with ci_build.sh:

There is also a script to regenerate yaml from mc.conf:
$ ./testsuite/utils/targets_gen.py

This test case requires avocado-framework-plugin-varianter-yaml-to-mux.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .gitignore                 |   2 +-
 scripts/ci_build.sh        |   8 ++-
 testsuite/citest.py        |  33 ++++++++++
 testsuite/data/targets.yml | 132 +++++++++++++++++++++++++++++++++++++
 testsuite/targets_gen.py   |  13 ++++
 testsuite/utils.py         |  35 +++++++++-
 6 files changed, 220 insertions(+), 3 deletions(-)
 create mode 100644 testsuite/data/targets.yml
 create mode 100755 testsuite/targets_gen.py

diff --git a/.gitignore b/.gitignore
index 81c09fc1..fff155f1 100644
--- a/.gitignore
+++ b/.gitignore
@@ -3,4 +3,4 @@
 __pycache__
 .config.yaml*
 .kas_shell_history
-build/
+build*
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 338557ad..690dadd4 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -23,6 +23,7 @@ if ! command -v avocado > /dev/null; then
     # shellcheck source=/dev/null
     source /tmp/avocado_venv/bin/activate
     pip install avocado-framework==103.0
+    pip install avocado-framework-plugin-varianter-yaml-to-mux==103.0
 fi
 
 # Get Avocado build tests path
@@ -49,6 +50,7 @@ show_help() {
     echo "    -b, --base BASE_DIR      set path to base directory. If not set,"
     echo "                             the tests will be started in current path."
     echo "    -d, --debug              enable debug bitbake output."
+    echo "    -m, --muxfile            input file for yaml-to-mux plugin."
     echo "    -T, --tags               specify basic avocado tags."
     echo "    --help                   display this message and exit."
     echo
@@ -94,6 +96,10 @@ do
         NORUN="1"
         echo "warning: deprecated parameter '$key', consider using '-T <TAG>,-startvm' instead"
         ;;
+    -m|--muxfile)
+        MUXFILE="-m $2"
+        shift
+        ;;
     -t|--timeout)
         TIMEOUT="-p time_to_wait=$2"
         shift
@@ -149,7 +155,7 @@ set -x
 
 avocado ${VERBOSE} run "${TESTSUITE_DIR}/citest.py" \
     -t "${TAGS}" --max-parallel-tasks=1 --disable-sysinfo \
-    ${SSTATE} ${TIMEOUT} \
+    ${SSTATE} ${MUXFILE} ${TIMEOUT} \
     || ret=$?
 
 python3 ${TESTSUITE_DIR}/cleanup.py
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 9f18a51f..501dbf37 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -1,4 +1,10 @@
 #!/usr/bin/env python3
+#
+# This software is a part of ISAR.
+# Copyright (C) 2022-2025 ilbers GmbH
+# Copyright (C) 2022-2025 Siemens AG
+#
+# SPDX-License-Identifier: MIT
 
 from avocado import skipUnless
 from avocado.core import exceptions
@@ -734,3 +740,30 @@ class VmBootTestFull(CIBaseTest):
         self.init()
         self.vm_start('amd64-iso', 'bookworm', image='isar-image-ci',
                       script='test_system_running.sh 30')
+
+
+class World(CIBaseTest):
+
+    """
+    All targets build test
+
+    :avocado: tags=world
+    """
+
+    def test_world(self):
+        name = self.params.get('name')
+        image = self.params.get('image', default='isar-image-ci')
+        targets = []
+
+        if name is None:
+            self.init()
+            for target in CIUtils.get_targets():
+                for image in CIUtils.get_test_images():
+                    targets.append(f'mc:{target}:{image}')
+        else:
+            targets.append(f'mc:{name}:{image}')
+            self.init(f'build-{name}')
+
+        self.perform_build_test(
+            targets, container=name and name.startswith('container')
+        )
diff --git a/testsuite/data/targets.yml b/testsuite/data/targets.yml
new file mode 100644
index 00000000..5c9b5196
--- /dev/null
+++ b/testsuite/data/targets.yml
@@ -0,0 +1,132 @@
+a: !mux
+  qemuarm-buster:
+    name: qemuarm-buster
+  qemuarm-bullseye:
+    name: qemuarm-bullseye
+  qemuarm-bookworm:
+    name: qemuarm-bookworm
+  qemuarm-trixie:
+    name: qemuarm-trixie
+  qemuarm64-buster:
+    name: qemuarm64-buster
+  qemuarm64-bullseye:
+    name: qemuarm64-bullseye
+  qemuarm64-bookworm:
+    name: qemuarm64-bookworm
+  qemuarm64-trixie:
+    name: qemuarm64-trixie
+  qemui386-buster:
+    name: qemui386-buster
+  qemui386-bullseye:
+    name: qemui386-bullseye
+  qemui386-bookworm:
+    name: qemui386-bookworm
+  qemuamd64-buster:
+    name: qemuamd64-buster
+  qemuamd64-bullseye:
+    name: qemuamd64-bullseye
+  qemuamd64-sb-bullseye:
+    name: qemuamd64-sb-bullseye
+  qemuamd64-bookworm:
+    name: qemuamd64-bookworm
+  qemuamd64-cip-bookworm:
+    name: qemuamd64-cip-bookworm
+  qemuamd64-iso-bookworm:
+    name: qemuamd64-iso-bookworm
+  qemuamd64-trixie:
+    name: qemuamd64-trixie
+  container-amd64-buster:
+    name: container-amd64-buster
+  container-amd64-bullseye:
+    name: container-amd64-bullseye
+  container-amd64-bookworm:
+    name: container-amd64-bookworm
+  qemumipsel-bullseye:
+    name: qemumipsel-bullseye
+  qemumipsel-bookworm:
+    name: qemumipsel-bookworm
+  qemuriscv64-trixie:
+    name: qemuriscv64-trixie
+  bananapi-buster:
+    name: bananapi-buster
+  bananapi-bullseye:
+    name: bananapi-bullseye
+  bananapi-bookworm:
+    name: bananapi-bookworm
+  beagleplay-bookworm:
+    name: beagleplay-bookworm
+  de0-nano-soc-buster:
+    name: de0-nano-soc-buster
+  de0-nano-soc-bullseye:
+    name: de0-nano-soc-bullseye
+  de0-nano-soc-bookworm:
+    name: de0-nano-soc-bookworm
+  hikey-bullseye:
+    name: hikey-bullseye
+  hikey-bookworm:
+    name: hikey-bookworm
+  imx6-sabrelite-buster:
+    name: imx6-sabrelite-buster
+  imx6-sabrelite-bullseye:
+    name: imx6-sabrelite-bullseye
+  phyboard-mira-bullseye:
+    name: phyboard-mira-bullseye
+  phyboard-mira-bookworm:
+    name: phyboard-mira-bookworm
+  nanopi-neo-buster:
+    name: nanopi-neo-buster
+  nanopi-neo-bullseye:
+    name: nanopi-neo-bullseye
+  nanopi-neo-bookworm:
+    name: nanopi-neo-bookworm
+  nanopi-neo-efi-bookworm:
+    name: nanopi-neo-efi-bookworm
+  stm32mp15x-bullseye:
+    name: stm32mp15x-bullseye
+  virtualbox-bullseye:
+    name: virtualbox-bullseye
+  virtualbox-bookworm:
+    name: virtualbox-bookworm
+  x86-pc-bookworm:
+    name: x86-pc-bookworm
+  rpi-arm-bullseye:
+    name: rpi-arm-bullseye
+  rpi-arm-bookworm:
+    name: rpi-arm-bookworm
+  rpi-arm-v7-bullseye:
+    name: rpi-arm-v7-bullseye
+  rpi-arm-v7-bookworm:
+    name: rpi-arm-v7-bookworm
+  rpi-arm-v7l-bullseye:
+    name: rpi-arm-v7l-bullseye
+  rpi-arm-v7l-bookworm:
+    name: rpi-arm-v7l-bookworm
+  rpi-arm64-v8-bullseye:
+    name: rpi-arm64-v8-bullseye
+  rpi-arm64-v8-bookworm:
+    name: rpi-arm64-v8-bookworm
+  sifive-fu540-trixie:
+    name: sifive-fu540-trixie
+  starfive-visionfive2-trixie:
+    name: starfive-visionfive2-trixie
+  qemuarm64-focal:
+    name: qemuarm64-focal
+  qemuarm64-jammy:
+    name: qemuarm64-jammy
+  qemuarm64-noble:
+    name: qemuarm64-noble
+  qemuamd64-focal:
+    name: qemuamd64-focal
+  qemuamd64-jammy:
+    name: qemuamd64-jammy
+  qemuamd64-noble:
+    name: qemuamd64-noble
+  qemuamd64-sid:
+    name: qemuamd64-sid
+  qemuarm64-sid:
+    name: qemuarm64-sid
+b: !mux
+  base:
+    image: isar-image-base
+  ci:
+    image: isar-image-ci
diff --git a/testsuite/targets_gen.py b/testsuite/targets_gen.py
new file mode 100755
index 00000000..41b4d644
--- /dev/null
+++ b/testsuite/targets_gen.py
@@ -0,0 +1,13 @@
+#!/usr/bin/env python3
+
+"""
+# This software is a part of Isar.
+# Copyright (C) 2025 ilbers GmbH
+
+# targets_gen.py: Generates yaml for yaml-to-mux Avocado varianter plugin.
+"""
+
+from utils import CIUtils
+
+if __name__ == "__main__":
+    CIUtils.gen_targets_yaml()
diff --git a/testsuite/utils.py b/testsuite/utils.py
index ffcb5cc7..9b18999b 100755
--- a/testsuite/utils.py
+++ b/testsuite/utils.py
@@ -1,4 +1,9 @@
 #!/usr/bin/env python3
+#
+# This software is a part of ISAR.
+# Copyright (C) 2024-2025 ilbers GmbH
+#
+# SPDX-License-Identifier: MIT
 
 import os
 import sys
@@ -10,8 +15,10 @@ sys.path.append(os.path.join(os.path.dirname(__file__), '../bitbake/lib'))
 import bb
 import bb.tinfoil
 
+isar_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
 
-class CIUtils():
+
+class CIUtils:
     @staticmethod
     def getVars(*vars, target=None):
         def fixStream(stream):
@@ -53,3 +60,29 @@ class CIUtils():
             return tar.getnames()
         except Exception:
             return []
+
+    @staticmethod
+    def get_test_images():
+        return ['isar-image-base', 'isar-image-ci']
+
+    @staticmethod
+    def get_targets():
+        d = bb.data.init()
+        d.setVar('BBPATH', os.path.join(isar_root, 'meta-isar'))
+        d = bb.cookerdata.parse_config_file('conf/mc.conf', d, False)
+        return d.getVar('BBMULTICONFIG').split()
+
+    @staticmethod
+    def gen_targets_yaml(fn='targets.yml'):
+        targetsfile = os.path.join(os.path.dirname(__file__), 'data', fn)
+        with open(targetsfile, 'w') as f:
+            f.write('a: !mux\n')
+            for target in CIUtils.get_targets():
+                f.write(f'  {target}:\n    name: {target}\n')
+            f.write('b: !mux\n')
+            prefix = 'isar-image-'
+            for image in CIUtils.get_test_images():
+                nodename = image
+                if image.startswith(prefix):
+                    nodename = image[len(prefix):]
+                f.write(f'  {nodename}:\n    image: {image}\n')
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250512083238.2089774-3-amikan%40ilbers.de.

[PATCH 3/3] CI: Run startvm in world test case

[Anton Mikanovich]

To cover also QEMU running for qemu- targets.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/citest.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 501dbf37..e7ba1720 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -767,3 +767,25 @@ class World(CIBaseTest):
         self.perform_build_test(
             targets, container=name and name.startswith('container')
         )
+
+    def test_runworld(self):
+        name = self.params.get('name')
+        image = self.params.get('image', default='isar-image-ci')
+        targets = []
+
+        if name is None:
+            self.init()
+            for target in CIUtils.get_targets():
+                for image in CIUtils.get_test_images():
+                    targets.append(f'mc:{target}:{image}')
+        else:
+            targets.append(f'mc:{name}:{image}')
+            self.init(f'build-{name}')
+
+        for target in targets:
+            t_list = target.split(':', 2)
+            if not t_list[1].startswith('qemu'):
+                self.cancel(f"{t_list[1]} skipped as non qemu")
+            distro = t_list[1].split('-')[-1]
+            arch = t_list[1].removeprefix('qemu').removesuffix(f"-{distro}")
+            self.vm_start(arch, distro, image=t_list[2])
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250512083238.2089774-4-amikan%40ilbers.de.