Re: FYI: Feasibility of CI on github

[Baurzhan Ismagulov <ibr@radix50.net>]

On Fri, Feb 12, 2021 at 11:28:40AM +0100, Henning Schild wrote:
> Last time i checked avocado was not packaged properly

When we made our initial PoC with Avocado, we used their Debian packaging.
Unfortunately, it got broken with buster. I'll talk with upstream regarding
this.


> Plus the patches have been merged without a proper review

Your feedback happened to come after the merge; it has been addressed in our
patches to come. If you have further feedback, please share with us.


> There is something. Run many tests in parallel instead of all mc in
> one. The all mc has its place because we have mc, but maybe not the
> best choice for the default.

Ok, that is a viable option. We'd continue to test mc anyway, but maybe in a
separate, non-default job. I think we would still need to look at the problems
that we encountered with mc and define some minimal mc test case for the
default scenario.


> And one really important first step would be decoupling of CI and CT.

If CT = continuous testing, I've indeed handled that as one step in our
process; what could we decouple with which benefit?


> > What I don't understand is, after we have individual test cases, how
> > does that help us with the long run time? Would you schedule every
> > test case in different steps?
> 
> You get more control if you want to disable some aspects, you get
> better reporting, you can fail individual tasks without failing "in
> total". You do not need to "keep going" when something failed, or you
> can keep going if that is what you want ... more control.

Those are arguments for a test framework in general -- I agree, this is why
we've introduced Avocado in the first place. If I understood Jan correctly, his
point was to make the run time shorter -- this could rather be solved by a
shorter default testsuite. We would then run the full testsuite on better
hardware before merging. That said, we already have the shorter variant; maybe
we should switch it to be the default?


With kind regards,
Baurzhan.