Re: [PATCH 0/8] CI rework of gitlab fast job

[Baurzhan Ismagulov <ibr@radix50.net>]

On Fri, Jan 13, 2023 at 06:14:01PM +0100, Henning Schild wrote:
> think we should not add a third pipeline. Too many switches and
> variations just mean that we will need testing for the tests and nobody
> will know which to run when.

This is a very valid point. Our problem is, we need two (or maybe more :( )
testsuites for efficient handling of maintenance tasks, and they are very
unsuitable for an average contributor. So this has appeared out of necessity.

I fully agree regarding the switches, I also hate them for the same reason.
We've considered moving to the raw avocado command line, but it is neither
short nor obvious, so we left ci_build.sh wrapper at this stage.

So, my proposal is to make "dev" the default and everyone should normally run
ci_build.sh without any switches (unless anyone explicitly wants to experience
some diversity in life). Of course, the maintainers should then regularly run
it.


> I did not try but i bet if "fast" would simply use Sstate caches it
> would run in less than 30min for small changes. And we would naturally
> test Sstate and actually become much better on "partial rebuild".
> 
> So in my book "dev" is "fast" or even "full" with warm caches.

Caching is also a good point. We can run "dev" with sstate by default; we just
need a way to specify local site configuration out of git (e.g., in an
environment variable).

I wouldn't like to have "dev" = "full" + caching because we're evaluating
adding rebuild testcases with and without specific options (like sstate, etc.)
and running both in "full".

"Fast" is rather a misnomer; its goal is to cover some 95+% of functionality in
somewhat reasonable time. The mentioned testcases have been very consciously
included there (e.g., sdk was broken several times); we can discuss what to
include but in general we'd like to keep and increase the current coverage
(execution time permitting) without affecting the users who don't need it.

"Dev" would further diverge from "fast" due to e.g.:

* Omitted "pedantic / infrastructure" bits as you mentioned

* Chosen rebuild testcases as this has often been broken in the past

* To be useful, "dev" should be kept green as far as possible, with all the
  implications.

  E.g., testcases can fail due to Debian issues, so "dev" shouldn't include
  Debian-ports (riscv). But the maintainers need to "quickly" assess the state
  of all supported arches.

* Testcase granularity (one testcase covering one arch + distro) to have a
  clear indication in the test results which specific combos have failed and to
  be able to re-run that failed combo only.

  "Fast" needs to keep the multiconfig building to catch races.


With kind regards,
Baurzhan