[PATCH 0/4] Add test-cases for dracut-based initrds

[PATCH 0/4] Add test-cases for dracut-based initrds

['Cedric Hombourger' via isar-users]

[1] is adding support for dracut as the initrd generator. That
patch series missed to add tests. Add several smoke tests.
Note that a small change (add "set -e" to the sudo script used
to generate the initrd) is needed for all initrd tests to pass.

Build results with pristine "Add support for dracut" v5:

    builder@ed1e0b7e8d4e:/work/testsuite$ trun --max-parallel-tasks=1 -p sstate=1 -t initrd citest.py
    JOB ID     : f1601d733e2a811ad04738eef7a1bdba28dbcd52
    JOB LOG    : /tmp/tmp72bozbdi/avocado/job-results/job-2025-11-01T10.19-f1601d7/job.log
     (1/5) citest.py:InitRdTest.test_dracut_in_image: STARTED
     (1/5) citest.py:InitRdTest.test_dracut_in_image: PASS (206.52 s)
     (2/5) citest.py:InitRdTest.test_dracut_build_initrd: STARTED
     (2/5) citest.py:InitRdTest.test_dracut_build_initrd: PASS (57.86 s)
     (3/5) citest.py:InitRdTest.test_dracut_build_failure: STARTED
     (3/5) citest.py:InitRdTest.test_dracut_build_failure: FAIL: Bitbake suceeded but was expected to fail! (159.91 s)
     (4/5) citest.py:InitRdCrossTests.test_dracut_in_image: STARTED
     (4/5) citest.py:InitRdCrossTests.test_dracut_in_image: PASS (1000.05 s)
     (5/5) citest.py:InitRdCrossTests.test_dracut_build_initrd: STARTED
     (5/5) citest.py:InitRdCrossTests.test_dracut_build_initrd: PASS (233.79 s)
    RESULTS    : PASS 4 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
    JOB TIME   : 1663.01 s

With the suggested change:

    builder@ed1e0b7e8d4e:/work/testsuite$ trun --max-parallel-tasks=1 -p sstate=1 -t initrd citest.py
    JOB ID     : dc55f13f05f5997669aaccce9760bca4bed7a400
    JOB LOG    : /tmp/tmp72bozbdi/avocado/job-results/job-2025-11-01T11.54-dc55f13/job.log
     (1/5) citest.py:InitRdTest.test_dracut_in_image: STARTED
     (1/5) citest.py:InitRdTest.test_dracut_in_image: PASS (117.49 s)
     (2/5) citest.py:InitRdTest.test_dracut_build_initrd: STARTED
     (2/5) citest.py:InitRdTest.test_dracut_build_initrd: PASS (30.33 s)
     (3/5) citest.py:InitRdTest.test_dracut_build_failure: STARTED
     (3/5) citest.py:InitRdTest.test_dracut_build_failure: PASS (73.31 s)
     (4/5) citest.py:InitRdCrossTests.test_dracut_in_image: STARTED
    
     (4/5) citest.py:InitRdCrossTests.test_dracut_in_image: PASS (430.85 s)
     (5/5) citest.py:InitRdCrossTests.test_dracut_build_initrd: STARTED
     (5/5) citest.py:InitRdCrossTests.test_dracut_build_initrd: PASS (55.30 s)
    RESULTS    : PASS 5 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
    JOB TIME   : 711.89 s

[1] https://lists.isar-build.org/isar-users/20251030094451.1303871-1-Quirin.Gylstorff@siemens.com/T/#t

Cedric Hombourger (4):
  CI: support negative build tests
  CI: add smoke-tests for switch to dracut in isar-image-ci
  CI: add build tests for isar-dracut initrd images
  CI: check if the build fails when dracut fails

 testsuite/cibase.py    |  4 +--
 testsuite/cibuilder.py | 12 ++++---
 testsuite/citest.py    | 81 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 91 insertions(+), 6 deletions(-)

-- 
2.47.3

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251101111326.600581-1-cedric.hombourger%40siemens.com.

[PATCH 1/4] CI: support negative build tests

['Cedric Hombourger' via isar-users]

It appears that our build tests are only check for successful
builds. We also want to make sure builds fail when they are
expected to. This adds a should_fail boolean to bitbake() and
perform_build_test()

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 testsuite/cibase.py    |  4 ++--
 testsuite/cibuilder.py | 12 ++++++++----
 2 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index 6ffedcea..0a006a21 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -14,12 +14,12 @@ from avocado.utils import process
 
 
 class CIBaseTest(CIBuilder):
-    def perform_build_test(self, targets, **kwargs):
+    def perform_build_test(self, targets, should_fail=False, **kwargs):
         self.configure(**kwargs)
 
         self.log.info("Starting build...")
 
-        self.bitbake(targets, **kwargs)
+        self.bitbake(targets, should_fail=should_fail, **kwargs)
 
     def perform_wic_partition_test(self, targets, wic_deploy_parts, **kwargs):
         self.configure(wic_deploy_parts=wic_deploy_parts, **kwargs)
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index cd5e70ba..366f6a1f 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -281,7 +281,8 @@ class CIBuilder(Test):
         if os.path.exists(self.build_dir + '/' + src):
             shutil.move(self.build_dir + '/' + src, self.build_dir + '/' + dst)
 
-    def bitbake(self, target, bitbake_cmd=None, sig_handler=None, **kwargs):
+    def bitbake(self, target, bitbake_cmd=None, should_fail=False,
+                sig_handler=None, **kwargs):
         self.check_init()
         self.log.info("===================================================")
         self.log.info(f"Building {str(target)}")
@@ -318,13 +319,16 @@ class CIBuilder(Test):
                         continue
                     if fd == p1.stdout.fileno():
                         self.log.info(p1.stdout.readline().rstrip())
-                    if fd == p1.stderr.fileno():
+                    if fd == p1.stderr.fileno() and should_fail is False:
                         app_log.error(p1.stderr.readline().rstrip())
                 if p1.poll() is not None:
                     break
             p1.wait()
-            if p1.returncode:
-                self.fail("Bitbake failed")
+            if should_fail is False:
+                if p1.returncode:
+                    self.fail("Bitbake failed")
+            elif p1.returncode == 0:
+                self.fail("Bitbake suceeded but was expected to fail!")
 
     def backupfile(self, path):
         self.check_init()
-- 
2.47.3

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251101111326.600581-2-cedric.hombourger%40siemens.com.

[PATCH 2/4] CI: add smoke-tests for switch to dracut in isar-image-ci

['Cedric Hombourger' via isar-users]

Use IMAGE_PREINSTALL to install dracut in the isar-image-ci
image and verify that it builds and boots. We also check that
a dracut-shutdown service is active (as an indication that
the system was booted using a dracut initrd). A single test
is provided as well as cross-{machine,distro} tests.

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 testsuite/citest.py | 58 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index a1214e9c..23900a6c 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -278,6 +278,64 @@ class KernelTests(CIBaseTest):
         self.perform_build_test(targets, image_install=' '.join(modules), lines=lines)
 
 
+class InitRdBaseTest(CIBaseTest):
+    """
+    Helper class for InitRd build tests
+    """
+
+    DRACUT_CONF = [
+        "IMAGE_PREINSTALL:append = ' dracut'",
+        "IMAGE_INSTALL:remove = 'enable-fsck'",
+    ]
+
+    def dracut_in_image(self, targets):
+        machine = 'qemuamd64'
+        distro = 'bookworm'
+        image = 'isar-image-ci'
+        self.init()
+        self.perform_build_test(targets, lines=InitRdBaseTest.DRACUT_CONF)
+
+        for target in targets:
+            config = target.split(':')
+            machine = config[1].split('-')[0]
+            distro = config[1].split('-')[1]
+            image = config[2]
+
+            # systems booted via a dracut initrd have a dracut-shutdown service
+            # check its presence
+            self.vm_start(
+                machine.removeprefix('qemu'), distro, image=image,
+                cmd="systemctl is-active dracut-shutdown"
+            )
+
+
+class InitRdTest(InitRdBaseTest):
+    """
+    Tests for initrd generation
+    :avocado: tags=initrd,single,full
+    """
+
+    def test_dracut_in_image(self):
+        """Test switch to dracut in an image recipe."""
+        self.dracut_in_image(['mc:qemuamd64-bookworm:isar-image-ci'])
+
+
+class InitRdCrossTests(InitRdBaseTest):
+    """
+    Tests for initrd generation
+    :avocado: tags=initrd,full
+    """
+
+    def test_dracut_in_image(self):
+        """Test switch to dracut in an image recipe."""
+        self.dracut_in_image([
+            'mc:qemuamd64-bookworm:isar-image-ci',
+            'mc:qemuamd64-trixie:isar-image-ci',
+            'mc:qemuarm64-bookworm:isar-image-ci',
+            'mc:qemuarm64-trixie:isar-image-ci'
+        ])
+
+
 class WicTest(CIBaseTest):
 
     """
-- 
2.47.3

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251101111326.600581-3-cedric.hombourger%40siemens.com.

[PATCH 3/4] CI: add build tests for isar-dracut initrd images

['Cedric Hombourger' via isar-users]

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 testsuite/citest.py | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 23900a6c..7f046a7f 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -319,6 +319,11 @@ class InitRdTest(InitRdBaseTest):
         """Test switch to dracut in an image recipe."""
         self.dracut_in_image(['mc:qemuamd64-bookworm:isar-image-ci'])
 
+    def test_dracut_build_initrd(self):
+        """ Test build of an initrd image that uses dracut."""
+        self.init()
+        self.perform_build_test(['mc:qemuamd64-bookworm:isar-dracut'])
+
 
 class InitRdCrossTests(InitRdBaseTest):
     """
@@ -335,6 +340,16 @@ class InitRdCrossTests(InitRdBaseTest):
             'mc:qemuarm64-trixie:isar-image-ci'
         ])
 
+    def test_dracut_build_initrd(self):
+        """ Test build of initrd images that use dracut."""
+        self.init()
+        self.perform_build_test([
+            'mc:qemuamd64-bookworm:isar-dracut',
+            'mc:qemuamd64-trixie:isar-dracut',
+            'mc:qemuarm64-bookworm:isar-dracut',
+            'mc:qemuarm64-trixie:isar-dracut'
+        ])
+
 
 class WicTest(CIBaseTest):
 
-- 
2.47.3

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251101111326.600581-4-cedric.hombourger%40siemens.com.

[PATCH 4/4] CI: check if the build fails when dracut fails

['Cedric Hombourger' via isar-users]

The build should not succeed if dracut fails to generate an
initrd. Amend the dracut command line to add an invalid option
(--unknown-option) to force a build failure and check that
bitbake does fail (it did not in the "Add support for dracut"
patch series v5).

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 testsuite/citest.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 7f046a7f..e53dfa9b 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -324,6 +324,14 @@ class InitRdTest(InitRdBaseTest):
         self.init()
         self.perform_build_test(['mc:qemuamd64-bookworm:isar-dracut'])
 
+    def test_dracut_build_failure(self):
+        """ Check if the build fails if dracut fails to generate an initrd."""
+        lines = InitRdBaseTest.DRACUT_CONF
+        lines.append("ROOTFS_INITRAMFS_GENERATOR_CMDLINE:append = ' --unknown-option'")
+        self.init()
+        self.perform_build_test('mc:qemuamd64-bookworm:isar-image-ci',
+                                should_fail=True, lines=lines)
+
 
 class InitRdCrossTests(InitRdBaseTest):
     """
-- 
2.47.3

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251101111326.600581-5-cedric.hombourger%40siemens.com.

Re: [PATCH 2/4] CI: add smoke-tests for switch to dracut in isar-image-ci

[Zhihang Wei]

On 11/1/25 12:13, 'Cedric Hombourger' via isar-users wrote:
> Use IMAGE_PREINSTALL to install dracut in the isar-image-ci
> image and verify that it builds and boots. We also check that
> a dracut-shutdown service is active (as an indication that
> the system was booted using a dracut initrd). A single test
> is provided as well as cross-{machine,distro} tests.
>
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
>   testsuite/citest.py | 58 +++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 58 insertions(+)
>
> diff --git a/testsuite/citest.py b/testsuite/citest.py
> index a1214e9c..23900a6c 100755
> --- a/testsuite/citest.py
> +++ b/testsuite/citest.py
> @@ -278,6 +278,64 @@ class KernelTests(CIBaseTest):
>           self.perform_build_test(targets, image_install=' '.join(modules), lines=lines)
>   
>   
> +class InitRdBaseTest(CIBaseTest):
> +    """
> +    Helper class for InitRd build tests
> +    """
> +
> +    DRACUT_CONF = [
> +        "IMAGE_PREINSTALL:append = ' dracut'",
> +        "IMAGE_INSTALL:remove = 'enable-fsck'",
> +    ]
> +
> +    def dracut_in_image(self, targets):
> +        machine = 'qemuamd64'
> +        distro = 'bookworm'
> +        image = 'isar-image-ci'
> +        self.init()
> +        self.perform_build_test(targets, lines=InitRdBaseTest.DRACUT_CONF)
> +
> +        for target in targets:
> +            config = target.split(':')
> +            machine = config[1].split('-')[0]
> +            distro = config[1].split('-')[1]
> +            image = config[2]
> +
> +            # systems booted via a dracut initrd have a dracut-shutdown service
> +            # check its presence
> +            self.vm_start(
> +                machine.removeprefix('qemu'), distro, image=image,
> +                cmd="systemctl is-active dracut-shutdown"
> +            )
> +
> +
I would suggest to move the definition of class InitRdBaseTest(CIBaseTest)
to cibase.py. In citest.py there should only be definition of tests that we
can directly run.
> +class InitRdTest(InitRdBaseTest):
> +    """
> +    Tests for initrd generation
> +    :avocado: tags=initrd,single,full
> +    """
> +
> +    def test_dracut_in_image(self):
> +        """Test switch to dracut in an image recipe."""
> +        self.dracut_in_image(['mc:qemuamd64-bookworm:isar-image-ci'])
> +
> +
> +class InitRdCrossTests(InitRdBaseTest):
> +    """
> +    Tests for initrd generation
> +    :avocado: tags=initrd,full
> +    """
> +
> +    def test_dracut_in_image(self):
> +        """Test switch to dracut in an image recipe."""
> +        self.dracut_in_image([
> +            'mc:qemuamd64-bookworm:isar-image-ci',
> +            'mc:qemuamd64-trixie:isar-image-ci',
> +            'mc:qemuarm64-bookworm:isar-image-ci',
> +            'mc:qemuarm64-trixie:isar-image-ci'
> +        ])
> +
> +
>   class WicTest(CIBaseTest):
>   
>       """
Thanks for adding new test cases. I have tested the v7 dracut patch set
against these test cases and others in current CI, didn't show any error.
We'll continue with this patch set after dracut can be applied.

Best regards,
Zhihang

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/bda24ac5-21de-4881-bc64-704a58147f6b%40ilbers.de.

Re: [PATCH 2/4] CI: add smoke-tests for switch to dracut in isar-image-ci

['cedric.hombourger@siemens.com' via isar-users]

On Fri, 2025-11-07 at 14:09 +0100, Zhihang Wei wrote:
> 
> 
> On 11/1/25 12:13, 'Cedric Hombourger' via isar-users wrote:
> > Use IMAGE_PREINSTALL to install dracut in the isar-image-ci
> > image and verify that it builds and boots. We also check that
> > a dracut-shutdown service is active (as an indication that
> > the system was booted using a dracut initrd). A single test
> > is provided as well as cross-{machine,distro} tests.
> > 
> > Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> > ---
> >   testsuite/citest.py | 58
> > +++++++++++++++++++++++++++++++++++++++++++++
> >   1 file changed, 58 insertions(+)
> > 
> > diff --git a/testsuite/citest.py b/testsuite/citest.py
> > index a1214e9c..23900a6c 100755
> > --- a/testsuite/citest.py
> > +++ b/testsuite/citest.py
> > @@ -278,6 +278,64 @@ class KernelTests(CIBaseTest):
> >           self.perform_build_test(targets, image_install='
> > '.join(modules), lines=lines)
> >   
> >   
> > +class InitRdBaseTest(CIBaseTest):
> > +    """
> > +    Helper class for InitRd build tests
> > +    """
> > +
> > +    DRACUT_CONF = [
> > +        "IMAGE_PREINSTALL:append = ' dracut'",
> > +        "IMAGE_INSTALL:remove = 'enable-fsck'",
> > +    ]
> > +
> > +    def dracut_in_image(self, targets):
> > +        machine = 'qemuamd64'
> > +        distro = 'bookworm'
> > +        image = 'isar-image-ci'
> > +        self.init()
> > +        self.perform_build_test(targets,
> > lines=InitRdBaseTest.DRACUT_CONF)
> > +
> > +        for target in targets:
> > +            config = target.split(':')
> > +            machine = config[1].split('-')[0]
> > +            distro = config[1].split('-')[1]
> > +            image = config[2]
> > +
> > +            # systems booted via a dracut initrd have a dracut-
> > shutdown service
> > +            # check its presence
> > +            self.vm_start(
> > +                machine.removeprefix('qemu'), distro, image=image,
> > +                cmd="systemctl is-active dracut-shutdown"
> > +            )
> > +
> > +
> I would suggest to move the definition of class
> InitRdBaseTest(CIBaseTest)
> to cibase.py. In citest.py there should only be definition of tests
> that we
> can directly run.

Are we sure we want to do this? While that's an easy change, I fear
that it would negatively impact readability of the tests that are
defined below. cibase is really at the framework level. The above class
is getting into the specifics.

We will anyhow need to revisit the structure of our test. If we are
successful in getting more test-case contributions (as I am really
hoping and now pushing for), then we would end-up with an enormous
citest.py file. We will need to break it down.

I would therefore like to give us some time to re-think the overall
structure of our testsuite/.

Is that agreeable?

> > +class InitRdTest(InitRdBaseTest):
> > +    """
> > +    Tests for initrd generation
> > +    :avocado: tags=initrd,single,full
> > +    """
> > +
> > +    def test_dracut_in_image(self):
> > +        """Test switch to dracut in an image recipe."""
> > +        self.dracut_in_image(['mc:qemuamd64-bookworm:isar-image-
> > ci'])
> > +
> > +
> > +class InitRdCrossTests(InitRdBaseTest):
> > +    """
> > +    Tests for initrd generation
> > +    :avocado: tags=initrd,full
> > +    """
> > +
> > +    def test_dracut_in_image(self):
> > +        """Test switch to dracut in an image recipe."""
> > +        self.dracut_in_image([
> > +            'mc:qemuamd64-bookworm:isar-image-ci',
> > +            'mc:qemuamd64-trixie:isar-image-ci',
> > +            'mc:qemuarm64-bookworm:isar-image-ci',
> > +            'mc:qemuarm64-trixie:isar-image-ci'
> > +        ])
> > +
> > +
> >   class WicTest(CIBaseTest):
> >   
> >       """
> Thanks for adding new test cases. I have tested the v7 dracut patch
> set
> against these test cases and others in current CI, didn't show any
> error.
> We'll continue with this patch set after dracut can be applied.
> 
> Best regards,
> Zhihang

-- 
Cedric Hombourger
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/a25db0cef1c44cf29a1da221316ae251be8a034d.camel%40siemens.com.