[PATCH v3 00/20] Various improvements to the testsuite

[Zhihang Wei <wzh@ilbers.de>]

This series aims at simplifying and speeding up the testsuite execution.
This should enable contributors to actually run it and test their patches.

As pointed out in [1], the testsuite currently tests various features over
and over again, while others are not tested at all. We try to improve this
by focusing on "feature" tests, whereby each test case checks a set of
features instead of a whole image. We further identified long running tasks
(like the kernel builds) and move them to a dedicated test.

The execution time reduction varies greatly depending on the hardware,
network and sstate usage. When applying these changes on recently added
patches, the time was often decreased by more than 50% on "fast".

Note, that more refactorings of the testsuite are needed - and planned,
but this series should already address the biggest pain points.

[1] https://groups.google.com/g/isar-users/c/uZBTIHzLp8Q/m/8k0L1vfMDAAJ

Changes since v2:

- move startvm tests after each build to avoid inter-testsuite dependency
- remove SstateTest from full

Changes since v1:

- continued work on testsuite refactoring, this supersedes:
  - testsuite: fix SignatureTest by avoiding absolute path in bblayers
  - testsuite: add rootfs target for rootfs only tests
  - Enable SState cache in GitLab CI
  - testsuite: fix typo in log message in perform_signature_lint
- completed exclusion / refactoring of db2m sbuild chroot
- GitLab CI: increased after script timeout to give sstate upload more time
- GitLab CI: do not upload sstate artifacts of images (as almost no reuse)
- do not add SignatureTest to fast tag (as otherwise VM tests break...)
- rebuild container test images in VM test to not depend on test order
- rebased onto next

Anton Mikanovich (1):
  testsuite: Group prebuilt_containers related test cases together

MOESSBAUER, Felix (17):
  testsuite: move targets with custom kernel to separate test
  testsuite: enable ccache on kernel tests
  testsuite: make prebuilt container a feature test
  testsuite: make compat test standalone test
  testsuite: handle IMAGE_INSTALL solely in cibuilder.py
  testsuite: limit cross_debsrc test to subset of packages
  testsuite: forward SSTATE_MIRRORS into CI env on sstate
  ci: add support for sstate cache
  testsuite: make test_cross_deps more specific
  testsuite: fix typo in log message in perform_signature_lint
  testsuite: fix SignatureTest by avoiding absolute path in bblayers
  testsuite: use more recent distros in SignatureTest
  testsuite: make SignatureTest idempotent
  testsuite: add rootfs target for rootfs only tests
  testsuite: refactor sbom tests to avoid overhead
  testsuite: make sbuild-flavor test standalone
  testsuite: skip VM tests if images are not available

Zhihang Wei (2):
  testsuite: Execute startvm testcases after building
  testsuite: Remove SstateTest from full testsuite

 .gitlab-ci.yml                                |  36 +-
 meta-test/conf/bblayers.conf.sample           |   8 +-
 meta-test/conf/local.conf.sample              |  10 -
 .../hello-isar/hello-isar.bbappend            |   3 +
 .../recipes-app/libhello/libhello.bbappend    |   5 +
 .../recipes-core/images/isar-image-ci.bb      |   2 -
 .../recipes-core/images/isar-rootfs-ci.bb     |  17 +
 testsuite/cibase.py                           |  34 +-
 testsuite/cibuilder.py                        |  49 +-
 testsuite/citest.py                           | 740 +++++++++++-------
 10 files changed, 602 insertions(+), 302 deletions(-)
 create mode 100644 meta-test/recipes-app/libhello/libhello.bbappend
 create mode 100644 meta-test/recipes-core/images/isar-rootfs-ci.bb

-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260123082501.240751-1-wzh%40ilbers.de.