Re: [PATCH 0/8] CI rework of gitlab fast job

[Henning Schild <henning.schild@siemens.com>]

Am Sat, 14 Jan 2023 15:40:47 +0100
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> On Fri, Jan 13, 2023 at 06:14:01PM +0100, Henning Schild wrote:
> > think we should not add a third pipeline. Too many switches and
> > variations just mean that we will need testing for the tests and
> > nobody will know which to run when.  
> 
> This is a very valid point. Our problem is, we need two (or maybe
> more :( ) testsuites for efficient handling of maintenance tasks, and
> they are very unsuitable for an average contributor. So this has
> appeared out of necessity.
> 
> I fully agree regarding the switches, I also hate them for the same
> reason. We've considered moving to the raw avocado command line, but
> it is neither short nor obvious, so we left ci_build.sh wrapper at
> this stage.

I would suggest a wrapper for every pipeline, rather than a switch to
one of the wrappers. We can leave switches for advanced users but maybe
only $@ to be passed to avocado.

So a simple ls would tell one which tests are available, where the
files hopefully have good names and the README or CONTRIBUTING files
say which tests to best run after changing anything.

Henning

> 
> So, my proposal is to make "dev" the default and everyone should
> normally run ci_build.sh without any switches (unless anyone
> explicitly wants to experience some diversity in life). Of course,
> the maintainers should then regularly run it.
> 
> 
> > I did not try but i bet if "fast" would simply use Sstate caches it
> > would run in less than 30min for small changes. And we would
> > naturally test Sstate and actually become much better on "partial
> > rebuild".
> > 
> > So in my book "dev" is "fast" or even "full" with warm caches.  
> 
> Caching is also a good point. We can run "dev" with sstate by
> default; we just need a way to specify local site configuration out
> of git (e.g., in an environment variable).
> 
> I wouldn't like to have "dev" = "full" + caching because we're
> evaluating adding rebuild testcases with and without specific options
> (like sstate, etc.) and running both in "full".
> 
> "Fast" is rather a misnomer; its goal is to cover some 95+% of
> functionality in somewhat reasonable time. The mentioned testcases
> have been very consciously included there (e.g., sdk was broken
> several times); we can discuss what to include but in general we'd
> like to keep and increase the current coverage (execution time
> permitting) without affecting the users who don't need it.
> 
> "Dev" would further diverge from "fast" due to e.g.:
> 
> * Omitted "pedantic / infrastructure" bits as you mentioned
> 
> * Chosen rebuild testcases as this has often been broken in the past
> 
> * To be useful, "dev" should be kept green as far as possible, with
> all the implications.
> 
>   E.g., testcases can fail due to Debian issues, so "dev" shouldn't
> include Debian-ports (riscv). But the maintainers need to "quickly"
> assess the state of all supported arches.
> 
> * Testcase granularity (one testcase covering one arch + distro) to
> have a clear indication in the test results which specific combos
> have failed and to be able to re-run that failed combo only.
> 
>   "Fast" needs to keep the multiconfig building to catch races.
> 
> 
> With kind regards,
> Baurzhan
>