[PATCH 0/4] Speedup testsuites

[PATCH 0/4] Speedup testsuites

[Anton Mikanovich]

Isar CI running takes too much time already, so speed up optimizations
need to be done. Remove some test case duplications and minimize time
consuming parts keeping the same test coverage.

On the test machine improvement is ~50%:
dev: 30min -> 22min
fast: 168min -> 113min
full: TBD

Based on `Additional CI improvements` patchset.

Anton Mikanovich (4):
  CI: Remove cleanup before nocross build
  CI: Minimize debsrc_cache checking
  CI: Minimize kselftest checking
  CI: Minimize testsuites duplication

 testsuite/cibuilder.py |  1 -
 testsuite/citest.py    | 64 +++++++++++++++++++++++++++++++++---------
 2 files changed, 50 insertions(+), 15 deletions(-)

-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240814090005.1381865-1-amikan%40ilbers.de.

[PATCH 1/4] CI: Remove cleanup before nocross build

[Anton Mikanovich]

Long time ago Isar was failing if running QEMU building right after
cross build test case. To mask that failure cleanup between nocross
and cross test cases was introduced. There is no more such an issue,
so we can safely remove this hack from the testsuite.

If nocross building will not work after switching from cross mode it
should be considered as a bug.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/citest.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 18d3af97..3f98fb3b 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -230,8 +230,6 @@ class NoCrossTest(CIBaseTest):
         ]
 
         self.init()
-        # Cleanup after cross build
-        self.move_in_build_dir('tmp', 'tmp_before_nocross')
         self.perform_build_test(targets, cross=False, debsrc_cache=True)
 
     def test_nocross_rpi(self):
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240814090005.1381865-2-amikan%40ilbers.de.

[PATCH 2/4] CI: Minimize debsrc_cache checking

[Anton Mikanovich]

Enabling debsrc_cache for such a big set of targets is not needed for
checking this functionality. Speedup testsuites by enabling
debsrc_cache partially.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/citest.py | 38 ++++++++++++++++++++++++++++++--------
 1 file changed, 30 insertions(+), 8 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 3f98fb3b..336eea4e 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -137,12 +137,19 @@ class CrossTest(CIBaseTest):
             'mc:de0-nano-soc-bullseye:isar-image-base',
             'mc:stm32mp15x-bullseye:isar-image-base',
             'mc:qemuarm-bookworm:isar-image-ci',
-            'mc:qemuarm64-bookworm:isar-image-ci',
             'mc:qemuarm64-focal:isar-image-base',
             'mc:nanopi-neo-efi-bookworm:isar-image-base',
             'mc:phyboard-mira-bookworm:isar-image-base',
         ]
 
+        self.init()
+        self.perform_build_test(targets)
+
+    def test_cross_debsrc(self):
+        targets = [
+            'mc:qemuarm64-bookworm:isar-image-ci',
+        ]
+
         self.init()
         self.perform_build_test(targets, debsrc_cache=True)
 
@@ -153,7 +160,7 @@ class CrossTest(CIBaseTest):
 
         self.init()
         try:
-            self.perform_build_test(targets, debsrc_cache=True)
+            self.perform_build_test(targets)
         except exceptions.TestFail:
             self.cancel('KFAIL')
 
@@ -174,7 +181,6 @@ class WicTest(CIBaseTest):
         self.perform_wic_partition_test(
             targets,
             wic_deploy_parts=False,
-            debsrc_cache=True,
             compat_arch=False,
         )
 
@@ -186,7 +192,6 @@ class WicTest(CIBaseTest):
         self.perform_wic_partition_test(
             targets,
             wic_deploy_parts=True,
-            debsrc_cache=True,
             compat_arch=False,
         )
 
@@ -218,17 +223,24 @@ class NoCrossTest(CIBaseTest):
             'mc:bananapi-bookworm:isar-image-base',
             'mc:nanopi-neo-bullseye:isar-image-base',
             'mc:nanopi-neo-bookworm:isar-image-base',
-            'mc:stm32mp15x-bullseye:isar-image-base',
             'mc:qemuamd64-focal:isar-image-ci',
             'mc:qemuamd64-bookworm:isar-image-ci',
-            'mc:qemuarm-bookworm:isar-image-ci',
             'mc:qemui386-bookworm:isar-image-base',
             'mc:qemumipsel-bookworm:isar-image-ci',
             'mc:hikey-bookworm:isar-image-base',
-            'mc:de0-nano-soc-bookworm:isar-image-base',
             'mc:beagleplay-bookworm:isar-image-base',
         ]
 
+        self.init()
+        self.perform_build_test(targets, cross=False)
+
+    def test_nocross_debsrc(self):
+        targets = [
+            'mc:qemuarm-bookworm:isar-image-ci',
+            'mc:stm32mp15x-bullseye:isar-image-base',
+            'mc:de0-nano-soc-bookworm:isar-image-base',
+        ]
+
         self.init()
         self.perform_build_test(targets, cross=False, debsrc_cache=True)
 
@@ -238,10 +250,20 @@ class NoCrossTest(CIBaseTest):
             'mc:rpi-arm-v7-bullseye:isar-image-base',
             'mc:rpi-arm-v7l-bullseye:isar-image-base',
             'mc:rpi-arm64-v8-bullseye:isar-image-base',
+            'mc:rpi-arm64-v8-bookworm:isar-image-base',
+        ]
+
+        self.init()
+        try:
+            self.perform_build_test(targets, cross=False)
+        except exceptions.TestFail:
+            self.cancel('KFAIL')
+
+    def test_nocross_rpi_debsrc(self):
+        targets = [
             'mc:rpi-arm-bookworm:isar-image-base',
             'mc:rpi-arm-v7-bookworm:isar-image-base',
             'mc:rpi-arm-v7l-bookworm:isar-image-base',
-            'mc:rpi-arm64-v8-bookworm:isar-image-base',
         ]
 
         self.init()
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240814090005.1381865-3-amikan%40ilbers.de.

[PATCH 3/4] CI: Minimize kselftest checking

[Anton Mikanovich]

There is no need in building kselftest for all the targets.
Move it to the separate test case.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/cibuilder.py |  1 -
 testsuite/citest.py    | 16 ++++++++++++++++
 2 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 3f77cd42..2311631d 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -167,7 +167,6 @@ class CIBuilder(Test):
                 f.write('ISAR_ENABLE_COMPAT_ARCH:arm64 = "1"\n')
                 f.write('IMAGE_INSTALL:remove:arm64 = "hello-isar"\n')
                 f.write('IMAGE_INSTALL:append:arm64 = " hello-isar-compat"\n')
-                f.write('IMAGE_INSTALL += "kselftest"\n')
             if cross:
                 f.write('ISAR_CROSS_COMPILE = "1"\n')
                 f.write(
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 336eea4e..1f7e854d 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -153,6 +153,22 @@ class CrossTest(CIBaseTest):
         self.init()
         self.perform_build_test(targets, debsrc_cache=True)
 
+    def test_cross_kselftest(self):
+        targets = [
+            'mc:qemuarm-buster:isar-image-ci',
+            'mc:qemuarm-bullseye:isar-image-ci',
+            'mc:de0-nano-soc-bullseye:isar-image-base',
+            'mc:stm32mp15x-bullseye:isar-image-base',
+            'mc:qemuarm-bookworm:isar-image-ci',
+            'mc:qemuarm64-bookworm:isar-image-ci',
+            'mc:qemuarm64-focal:isar-image-base',
+            'mc:nanopi-neo-efi-bookworm:isar-image-base',
+            'mc:phyboard-mira-bookworm:isar-image-base',
+        ]
+
+        self.init()
+        self.perform_build_test(targets, image_install="kselftest")
+
     def test_cross_rpi(self):
         targets = [
             'mc:rpi-arm-v7-bullseye:isar-image-base',
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240814090005.1381865-4-amikan%40ilbers.de.

[PATCH 4/4] CI: Minimize testsuites duplication

[Anton Mikanovich]

Separate test cases between fast and full testsuites to avoid
duplication.
From now on full testsuite will not also execute dev and fast
test cases. Container test cases will be executed only in full.
The same coverage should be done by running several testsuites.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/citest.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 1f7e854d..278e0bd3 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -23,7 +23,7 @@ class DevTest(CIBaseTest):
     """
     Developer's test
 
-    :avocado: tags=dev,fast,full
+    :avocado: tags=dev,fast
     """
 
     def test_dev(self):
@@ -127,7 +127,7 @@ class CrossTest(CIBaseTest):
     """
     Start cross build for the defined set of configurations
 
-    :avocado: tags=cross,fast,full
+    :avocado: tags=cross,fast
     """
 
     def test_cross(self):
@@ -320,7 +320,7 @@ class ContainerImageTest(CIBaseTest):
     """
     Test containerized images creation
 
-    :avocado: tags=containerbuild,fast,full,container
+    :avocado: tags=containerbuild,full,container
     """
 
     @skipUnless(UMOCI_AVAILABLE and SKOPEO_AVAILABLE, 'umoci/skopeo not found')
@@ -340,7 +340,7 @@ class ContainerSdkTest(CIBaseTest):
     """
     Test SDK container image creation
 
-    :avocado: tags=containersdk,fast,full,container
+    :avocado: tags=containersdk,full,container
     """
 
     @skipUnless(UMOCI_AVAILABLE and SKOPEO_AVAILABLE, 'umoci/skopeo not found')
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240814090005.1381865-5-amikan%40ilbers.de.