Re: FYI: Feasibility of CI on github

[Henning Schild <henning.schild@siemens.com>]

Am Fri, 12 Feb 2021 11:53:06 +0100
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> On Fri, Feb 12, 2021 at 11:28:40AM +0100, Henning Schild wrote:
> > Last time i checked avocado was not packaged properly  
> 
> When we made our initial PoC with Avocado, we used their Debian
> packaging. Unfortunately, it got broken with buster. I'll talk with
> upstream regarding this.

That would be good. I think having debian would be good enough but not
having it is really bad.

> > Plus the patches have been merged without a proper review  
> 
> Your feedback happened to come after the merge; it has been addressed
> in our patches to come. If you have further feedback, please share
> with us.

I do not remember the details. It is quite possible. But a feature
merge without a user is something that should never have happened.
Like an RFC merge should never happen, but not the point of the thread.
We are already too far away from github workflows.

> 
> > There is something. Run many tests in parallel instead of all mc in
> > one. The all mc has its place because we have mc, but maybe not the
> > best choice for the default.  
> 
> Ok, that is a viable option. We'd continue to test mc anyway, but
> maybe in a separate, non-default job. I think we would still need to
> look at the problems that we encountered with mc and define some
> minimal mc test case for the default scenario.
>
> 
> > And one really important first step would be decoupling of CI and
> > CT.  
> 
> If CT = continuous testing, I've indeed handled that as one step in
> our process; what could we decouple with which benefit?

Yes i mean continous testing.

Starting those qemus and parsing their output is clearly testing and
should be another job. All the qemu-related variables need to leave the
machine configs.
These configs serve as a very bad example where people might think they
can configure the kernel cmdline in the machine cfg, even though they
might be on wic.

> > > What I don't understand is, after we have individual test cases,
> > > how does that help us with the long run time? Would you schedule
> > > every test case in different steps?  
> > 
> > You get more control if you want to disable some aspects, you get
> > better reporting, you can fail individual tasks without failing "in
> > total". You do not need to "keep going" when something failed, or
> > you can keep going if that is what you want ... more control.  
> 
> Those are arguments for a test framework in general -- I agree, this
> is why we've introduced Avocado in the first place. If I understood
> Jan correctly, his point was to make the run time shorter -- this
> could rather be solved by a shorter default testsuite. We would then
> run the full testsuite on better hardware before merging. That said,
> we already have the shorter variant; maybe we should switch it to be
> the default?

Ideally the test CI and CT framwork of Isar would allow layering. While
main Isar would be mc and slow on its thorough CI/CT layers could reuse
parts.
I think a proper modularization will save time even given the current
CI.

A green build might still take the full time. But a red build will be
faster and tell you which modules failed. Which will also allow you to
write patches for multiple problems for one red run ... better
reporting will result in better understanding and less builds. Which
saves build count and a lot of time on that end, even if the time of
such a build remains "high".

And pipelines with multiple jobs allow to hit retry buttons for just
the failed job.

I think you can see this
https://code.siemens.com/linux/las/-/pipelines/8264154

To get there we would need a CI system where the pipelines are part of
the code, so probably not jenkins.

Henning
 
> 
> With kind regards,
> Baurzhan.
>