Re: FYI: Feasibility of CI on github

[Jan Kiszka <jan.kiszka@siemens.com>]

On 12.02.21 11:53, Baurzhan Ismagulov wrote:
> On Fri, Feb 12, 2021 at 11:28:40AM +0100, Henning Schild wrote:
>> Last time i checked avocado was not packaged properly
> 
> When we made our initial PoC with Avocado, we used their Debian packaging.
> Unfortunately, it got broken with buster. I'll talk with upstream regarding
> this.
> 
> 
>> Plus the patches have been merged without a proper review
> 
> Your feedback happened to come after the merge; it has been addressed in our
> patches to come. If you have further feedback, please share with us.
> 
> 
>> There is something. Run many tests in parallel instead of all mc in
>> one. The all mc has its place because we have mc, but maybe not the
>> best choice for the default.
> 
> Ok, that is a viable option. We'd continue to test mc anyway, but maybe in a
> separate, non-default job. I think we would still need to look at the problems
> that we encountered with mc and define some minimal mc test case for the
> default scenario.
> 
> 
>> And one really important first step would be decoupling of CI and CT.
> 
> If CT = continuous testing, I've indeed handled that as one step in our
> process; what could we decouple with which benefit?
> 
> 
>>> What I don't understand is, after we have individual test cases, how
>>> does that help us with the long run time? Would you schedule every
>>> test case in different steps?
>>
>> You get more control if you want to disable some aspects, you get
>> better reporting, you can fail individual tasks without failing "in
>> total". You do not need to "keep going" when something failed, or you
>> can keep going if that is what you want ... more control.
> 
> Those are arguments for a test framework in general -- I agree, this is why
> we've introduced Avocado in the first place. If I understood Jan correctly, his
> point was to make the run time shorter -- this could rather be solved by a
> shorter default testsuite. We would then run the full testsuite on better
> hardware before merging. That said, we already have the shorter variant; maybe
> we should switch it to be the default?
> 

Time is not our primary issue (you can easily run the sequential parts
of the CI script in separate jobs and gain quite a bit). It's the disk
size. But when we want to stress the full mc: pool, there is likely no
way around dedicated runners for the time being.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux