[RFC] Test cases regrouping concept

[RFC] Test cases regrouping concept

[Anton Mikanovich]

We are about to migrate on Avocado framework for CI test and now we have 
a proposal about new test case structure.
There are 3 main goals we need to have after test case rearrange:

1. Minimize 'fast' stage of the CI.
After adding a lot of new test cases our fast mode is no more really fast.
There are no need in testing some special stuff like containerized 
rootfs or cached base repository in the build supposed to be fast.

2. Cover all machines and distros.
There are a lot of configurations not covered by current CI.

3. Create some middle stage in between roughly similar to the current 
fast CI coverage.

So we can split all the test cases into 3 groups:

1) Fast (Smoke) testing - extra-fast test running to check that the new 
code does not introduce regressions into the core Isar logic
2) Regular testing - fast testing for daily incoming patchsets checking, 
will also include Fast
3) Full testing - full testing that cover all the machines and distros, 
will also include Fast and Regular

We should move linux kernel from Fast to Regular because it takes too 
much time.

Because Avocado supports tags we also can split any group more precisely 
to be able to run any part of the group (like fast-cross, 
regular-native, etc).

Here is how it can look like:

1. Fast test
  a) Cross Fast test
   mc:qemuarm64-bullseye:isar-image-base
   mc:qemuamd64-focal:isar-image-base
  b) Native Fast test
   mc:qemuarm-buster:isar-image-base
   mc:rpi-stretch:isar-image-base

2. Regular test
  a) Cross Regular test
   mc:qemuamd64-stretch:isar-image-base
   mc:qemuarm64-bookworm:isar-image-base
   mc:qemuarm-focal:isar-image-base
   mc:de0-nano-soc-buster:isar-image-base
   mc:stm32mp15x-buster:isar-image-base
   mc:rpi-stretch:isar-image-base
  b) Native Regular test
   mc:qemuamd64-bullseye:isar-image-base
   mc:qemuarm64-stretch:isar-image-base
   mc:qemumipsel-bullseye:isar-image-base
   mc:qemui386-buster:isar-image-base
   mc:nand-ubi-demo-buster:isar-image-ubi
   mc:qemuamd64-buster-tgz:isar-image-base
   mc:qemuamd64-buster:isar-initramfs
  c) SDK test
   mc:qemuarm-bullseye:isar-image-base:do_populate_sdk
  d) Image rebuild test
   mc:qemuamd64-bullseye:isar-image-base
  e) Test wic --exclude-path
   mc:qemuamd64-bullseye:isar-image-base
  f) Containerized rootfs test
   mc:container-amd64-bullseye:isar-image-base
  g) Containerized SDK test
   mc:container-amd64-bullseye:isar-image-base:do_populate_sdk
  h) Repro signed test
   mc:qemuamd64-bookworm:isar-image-base
   mc:qemuarm64-stretch:isar-image-base
  i) Repro unsigned
   mc:qemuamd64-bullseye:isar-image-base
   mc:qemuarm-buster:isar-image-base

3. Full test
  a) Cross Full test
   mc:qemuamd64-bullseye:isar-image-base
   mc:qemuamd64-buster:isar-image-base
   mc:qemuarm64-buster:isar-image-base
   mc:qemuarm64-focal:isar-image-base
   mc:qemuarm-stretch:isar-image-base
   mc:qemuarm-bullseye:isar-image-base
   mc:qemuarm-bookworm:isar-image-base
   mc:qemumipsel-stretch:isar-image-base
   mc:qemumipsel-buster:isar-image-base
   mc:qemumipsel-bullseye:isar-image-base
   mc:qemumipsel-bookworm:isar-image-base
   mc:qemumipsel-focal:isar-image-base
   mc:qemui386-buster:isar-image-base
   mc:qemui386-bullseye:isar-image-base
   mc:qemui386-bookworm:isar-image-base
   mc:qemui386-focal:isar-image-base
   mc:cipamd64-bullseye:isar-image-base
   mc:sifive-fu540-sid-ports:isar-image-base
   mc:hikey-stretch:isar-image-base
   mc:bananapi-buster:isar-image-base
   mc:nanopi-neo-buster:isar-image-base
  b) Native Full test
   mc:qemuarm-stretch:isar-image-base
   mc:qemuarm-bullseye:isar-image-base
   mc:qemuarm-bookworm:isar-image-base
   mc:qemuarm-focal:isar-image-base
   mc:qemuarm64-buster:isar-image-base
   mc:qemuarm64-bullseye:isar-image-base
   mc:qemuarm64-bookworm:isar-image-base
   mc:qemuarm64-focal:isar-image-base
   mc:qemumipsel-stretch:isar-image-base
   mc:qemumipsel-buster:isar-image-base
   mc:qemumipsel-bookworm:isar-image-base
   mc:qemumipsel-focal:isar-image-base
   mc:qemui386-stretch:isar-image-base
   mc:qemui386-bullseye:isar-image-base
   mc:qemui386-bookworm:isar-image-base
   mc:qemui386-focal:isar-image-base
   mc:qemuriscv64-sid-ports:isar-image-base
  c) Containerized rootfs test
   mc:container-amd64-stretch:isar-image-base
   mc:container-amd64-buster:isar-image-base
   mc:container-amd64-bookworm:isar-image-base

Multiconfigs of one subtest will be executed in one bitbake call.

We would like to have a testsuite definition for building and running 
images.

If anyone have more ideas how to improve this concept any comments are 
welcome.

-- 
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov