[PATCH v2 0/4] Speedup testsuites

[PATCH v2 0/4] Speedup testsuites

[Anton Mikanovich]

Isar CI running takes too much time already, so speed up optimizations
need to be done. Remove some test case duplications and minimize time
consuming parts keeping the same test coverage.

On the test machine improvement is ~30-50%:
dev: 30min -> 22min
fast: 168min -> 113min
full: 636min -> 483min

Based on `Additional CI improvements` patchset.

Changes since v1:
- Fixed kselftest test case.
- Add full testsuite time improve statistics.

Anton Mikanovich (4):
  CI: Remove cleanup before nocross build
  CI: Minimize debsrc_cache checking
  CI: Minimize kselftest checking
  CI: Minimize testsuites duplication

 testsuite/cibuilder.py |  1 -
 testsuite/citest.py    | 64 +++++++++++++++++++++++++++++++++---------
 2 files changed, 50 insertions(+), 15 deletions(-)

-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240816072526.65382-1-amikan%40ilbers.de.

[PATCH v2 1/4] CI: Remove cleanup before nocross build

[Anton Mikanovich]

Long time ago Isar was failing if running QEMU building right after
cross build test case. To mask that failure cleanup between nocross
and cross test cases was introduced. There is no more such an issue,
so we can safely remove this hack from the testsuite.

If nocross building will not work after switching from cross mode it
should be considered as a bug.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/citest.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 18d3af97..3f98fb3b 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -230,8 +230,6 @@ class NoCrossTest(CIBaseTest):
         ]
 
         self.init()
-        # Cleanup after cross build
-        self.move_in_build_dir('tmp', 'tmp_before_nocross')
         self.perform_build_test(targets, cross=False, debsrc_cache=True)
 
     def test_nocross_rpi(self):
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240816072526.65382-2-amikan%40ilbers.de.

[PATCH v2 2/4] CI: Minimize debsrc_cache checking

[Anton Mikanovich]

Enabling debsrc_cache for such a big set of targets is not needed for
checking this functionality. Speedup testsuites by enabling
debsrc_cache partially.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/citest.py | 38 ++++++++++++++++++++++++++++++--------
 1 file changed, 30 insertions(+), 8 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 3f98fb3b..336eea4e 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -137,12 +137,19 @@ class CrossTest(CIBaseTest):
             'mc:de0-nano-soc-bullseye:isar-image-base',
             'mc:stm32mp15x-bullseye:isar-image-base',
             'mc:qemuarm-bookworm:isar-image-ci',
-            'mc:qemuarm64-bookworm:isar-image-ci',
             'mc:qemuarm64-focal:isar-image-base',
             'mc:nanopi-neo-efi-bookworm:isar-image-base',
             'mc:phyboard-mira-bookworm:isar-image-base',
         ]
 
+        self.init()
+        self.perform_build_test(targets)
+
+    def test_cross_debsrc(self):
+        targets = [
+            'mc:qemuarm64-bookworm:isar-image-ci',
+        ]
+
         self.init()
         self.perform_build_test(targets, debsrc_cache=True)
 
@@ -153,7 +160,7 @@ class CrossTest(CIBaseTest):
 
         self.init()
         try:
-            self.perform_build_test(targets, debsrc_cache=True)
+            self.perform_build_test(targets)
         except exceptions.TestFail:
             self.cancel('KFAIL')
 
@@ -174,7 +181,6 @@ class WicTest(CIBaseTest):
         self.perform_wic_partition_test(
             targets,
             wic_deploy_parts=False,
-            debsrc_cache=True,
             compat_arch=False,
         )
 
@@ -186,7 +192,6 @@ class WicTest(CIBaseTest):
         self.perform_wic_partition_test(
             targets,
             wic_deploy_parts=True,
-            debsrc_cache=True,
             compat_arch=False,
         )
 
@@ -218,17 +223,24 @@ class NoCrossTest(CIBaseTest):
             'mc:bananapi-bookworm:isar-image-base',
             'mc:nanopi-neo-bullseye:isar-image-base',
             'mc:nanopi-neo-bookworm:isar-image-base',
-            'mc:stm32mp15x-bullseye:isar-image-base',
             'mc:qemuamd64-focal:isar-image-ci',
             'mc:qemuamd64-bookworm:isar-image-ci',
-            'mc:qemuarm-bookworm:isar-image-ci',
             'mc:qemui386-bookworm:isar-image-base',
             'mc:qemumipsel-bookworm:isar-image-ci',
             'mc:hikey-bookworm:isar-image-base',
-            'mc:de0-nano-soc-bookworm:isar-image-base',
             'mc:beagleplay-bookworm:isar-image-base',
         ]
 
+        self.init()
+        self.perform_build_test(targets, cross=False)
+
+    def test_nocross_debsrc(self):
+        targets = [
+            'mc:qemuarm-bookworm:isar-image-ci',
+            'mc:stm32mp15x-bullseye:isar-image-base',
+            'mc:de0-nano-soc-bookworm:isar-image-base',
+        ]
+
         self.init()
         self.perform_build_test(targets, cross=False, debsrc_cache=True)
 
@@ -238,10 +250,20 @@ class NoCrossTest(CIBaseTest):
             'mc:rpi-arm-v7-bullseye:isar-image-base',
             'mc:rpi-arm-v7l-bullseye:isar-image-base',
             'mc:rpi-arm64-v8-bullseye:isar-image-base',
+            'mc:rpi-arm64-v8-bookworm:isar-image-base',
+        ]
+
+        self.init()
+        try:
+            self.perform_build_test(targets, cross=False)
+        except exceptions.TestFail:
+            self.cancel('KFAIL')
+
+    def test_nocross_rpi_debsrc(self):
+        targets = [
             'mc:rpi-arm-bookworm:isar-image-base',
             'mc:rpi-arm-v7-bookworm:isar-image-base',
             'mc:rpi-arm-v7l-bookworm:isar-image-base',
-            'mc:rpi-arm64-v8-bookworm:isar-image-base',
         ]
 
         self.init()
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240816072526.65382-3-amikan%40ilbers.de.

[PATCH v2 3/4] CI: Minimize kselftest checking

[Anton Mikanovich]

There is no need in building kselftest for all the targets.
Move it to the separate test case.
Use the same targets set as cross build test case.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/cibuilder.py |  1 -
 testsuite/citest.py    | 16 ++++++++++++++++
 2 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 3f77cd42..2311631d 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -167,7 +167,6 @@ class CIBuilder(Test):
                 f.write('ISAR_ENABLE_COMPAT_ARCH:arm64 = "1"\n')
                 f.write('IMAGE_INSTALL:remove:arm64 = "hello-isar"\n')
                 f.write('IMAGE_INSTALL:append:arm64 = " hello-isar-compat"\n')
-                f.write('IMAGE_INSTALL += "kselftest"\n')
             if cross:
                 f.write('ISAR_CROSS_COMPILE = "1"\n')
                 f.write(
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 336eea4e..29641d38 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -153,6 +153,22 @@ class CrossTest(CIBaseTest):
         self.init()
         self.perform_build_test(targets, debsrc_cache=True)
 
+    def test_cross_kselftest(self):
+        targets = [
+            'mc:qemuarm-buster:kselftest',
+            'mc:qemuarm-bullseye:kselftest',
+            'mc:de0-nano-soc-bullseye:kselftest',
+            'mc:stm32mp15x-bullseye:kselftest',
+            'mc:qemuarm-bookworm:kselftest',
+            'mc:qemuarm64-bookworm:kselftest',
+            'mc:qemuarm64-focal:kselftest',
+            'mc:nanopi-neo-efi-bookworm:kselftest',
+            'mc:phyboard-mira-bookworm:kselftest',
+        ]
+
+        self.init()
+        self.perform_build_test(targets)
+
     def test_cross_rpi(self):
         targets = [
             'mc:rpi-arm-v7-bullseye:isar-image-base',
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240816072526.65382-4-amikan%40ilbers.de.

[PATCH v2 4/4] CI: Minimize testsuites duplication

[Anton Mikanovich]

Separate test cases between fast and full testsuites to avoid
duplication.
From now on full testsuite will not also execute dev and fast
test cases. Container test cases will be executed only in full.
The same coverage should be done by running several testsuites.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/citest.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 29641d38..82affb1a 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -23,7 +23,7 @@ class DevTest(CIBaseTest):
     """
     Developer's test
 
-    :avocado: tags=dev,fast,full
+    :avocado: tags=dev,fast
     """
 
     def test_dev(self):
@@ -127,7 +127,7 @@ class CrossTest(CIBaseTest):
     """
     Start cross build for the defined set of configurations
 
-    :avocado: tags=cross,fast,full
+    :avocado: tags=cross,fast
     """
 
     def test_cross(self):
@@ -320,7 +320,7 @@ class ContainerImageTest(CIBaseTest):
     """
     Test containerized images creation
 
-    :avocado: tags=containerbuild,fast,full,container
+    :avocado: tags=containerbuild,full,container
     """
 
     @skipUnless(UMOCI_AVAILABLE and SKOPEO_AVAILABLE, 'umoci/skopeo not found')
@@ -340,7 +340,7 @@ class ContainerSdkTest(CIBaseTest):
     """
     Test SDK container image creation
 
-    :avocado: tags=containersdk,fast,full,container
+    :avocado: tags=containersdk,full,container
     """
 
     @skipUnless(UMOCI_AVAILABLE and SKOPEO_AVAILABLE, 'umoci/skopeo not found')
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240816072526.65382-5-amikan%40ilbers.de.

Re: [PATCH v2 0/4] Speedup testsuites

[Uladzimir Bely]

On Fri, 2024-08-16 at 10:25 +0300, Anton Mikanovich wrote:
> Isar CI running takes too much time already, so speed up
> optimizations
> need to be done. Remove some test case duplications and minimize time
> consuming parts keeping the same test coverage.
> 
> On the test machine improvement is ~30-50%:
> dev: 30min -> 22min
> fast: 168min -> 113min
> full: 636min -> 483min
> 
> Based on `Additional CI improvements` patchset.
> 
> Changes since v1:
> - Fixed kselftest test case.
> - Add full testsuite time improve statistics.
> 
> Anton Mikanovich (4):
>   CI: Remove cleanup before nocross build
>   CI: Minimize debsrc_cache checking
>   CI: Minimize kselftest checking
>   CI: Minimize testsuites duplication
> 
>  testsuite/cibuilder.py |  1 -
>  testsuite/citest.py    | 64 +++++++++++++++++++++++++++++++++-------
> --
>  2 files changed, 50 insertions(+), 15 deletions(-)
> 
> -- 
> 2.34.1
> 

Applied to next.

-- 
Best regards,
Uladzimir.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/8d48f27bc7df16bcdfe8d7417d11bf1934977298.camel%40ilbers.de.