Re: [PATCH 0/8] CI rework of gitlab fast job

[Henning Schild <henning.schild@siemens.com>]

Am Fri, 13 Jan 2023 14:14:58 +0100
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> On Fri, Jan 13, 2023 at 12:43:15AM +0100, Henning Schild wrote:
> > At the end of the day i want any variant of the testsuite to run on
> > any setup just like it does on ilbers jenkins.
> > 
> > That should be based on the kas containers for anyone not using your
> > jenkins.
> > 
> > With that any contributor would be able to run it manually or in
> > their CI.
> > 
> > We currently can not run "full" in gitlab CI because it takes too
> > much time and our gitlab runners would take such jobs for at most
> > 12h. But that is a value i could adjust if we find that it really
> > needs to take that long. I hope we can get much faster and still
> > have the same coverage.  
> 
> Thanks for the summary, that's fully in line with our last off-list
> discussion:
> 
> * The preferred downstream testsuite is "dev" (runs for ~ 2 h IIRC).
> 
> * The testsuite runs manually on the host or in kas.
> 
> * The testsuite runs automatically in GitLab, Jenkins or Azure using
> kas.
> 
> * Separate testcases for every arch / distro combo. One can run the
> complete testsuite or individual testcases.
> 
> * The maintainer CI provides test status and build control on a
> publicly accessible, easily scalable infrastructure.
> 
> * The CI setup is easily clonable for the downstreams.
> 
> * Downstreams can run Isar testsuites and / or their own testsuites.
> 
> 
> So, I propose that we update and apply the "dev" testsuite series,
> then apply yours with the following changes:
> 
> 1. Use "dev" in "[PATCH 3/8] CI: install qemu-system when qemu
> testing is requested".
> 
> 2. Patches 4-8 "testsuite: Remove X test from "fast" set" will be
> unnecessary (we'd take this into account during the "dev" series
> update -- those testcases are not part of the "dev" v1 anyway).
> 
> For the final review of the use cases I'll wait for Anton.

Let us see how we merge all that. I never looked at that dev series and
think we should not add a third pipeline. Too many switches and
variations just mean that we will need testing for the tests and nobody
will know which to run when.

I did not try but i bet if "fast" would simply use Sstate caches it
would run in less than 30min for small changes. And we would naturally
test Sstate and actually become much better on "partial rebuild".

So in my book "dev" is "fast" or even "full" with warm caches.

Henning

> 
> With kind regards,
> Baurzhan