Re: FYI: Feasibility of CI on github

[Baurzhan Ismagulov <ibr@radix50.net>]

On Fri, Feb 12, 2021 at 08:20:51PM +0100, Henning Schild wrote:
> > When we made our initial PoC with Avocado, we used their Debian
> > packaging. Unfortunately, it got broken with buster. I'll talk with
> > upstream regarding this.
> 
> That would be good. I think having debian would be good enough but not
> having it is really bad.

I've looked at the code, they've meanwhile updated it. I've submitted small
fixes, but we have the package and will test it.


> I do not remember the details. It is quite possible. But a feature
> merge without a user is something that should never have happened.

The code that is not used by default doesn't necessarily mean it is not used.
It's exactly because we wanted to experiment with different test case splits
that we didn't enable it by default too early. In the first step, we'd like to
mimic the existing test suite, then maybe add test cases for individual machine
builds, then switch the default.

That said, this topic is something we are often confronted with: We get an
obviously useful patch (e.g., a fix for a build failure in a certain scenario),
but don't have a test case for it. By this definition, it is code without a
user. As much as I'd like to have test cases for everything, I doubt we can
live up to that 100%.


> > > And one really important first step would be decoupling of CI and
> > > CT.  
> > 
> > If CT = continuous testing, I've indeed handled that as one step in
> > our process; what could we decouple with which benefit?
> 
> Yes i mean continous testing.
> 
> Starting those qemus and parsing their output is clearly testing and
> should be another job.

Ok, roughly, CI is building and CT is running. Yes, starting the images will go
to their own test cases. That said, at Isar CI we'd still like to include them
in the process. But downstreams can decide that themselves -- IIUC, it isn't
used at the Siemens CI today.


> All the qemu-related variables need to leave the machine configs.
> 
> These configs serve as a very bad example where people might think they
> can configure the kernel cmdline in the machine cfg, even though they
> might be on wic.

If the docs can be improved to avoid misunderstandings, I'd start with that
first. The qemu vars are already prefixed. Scattered settings is one of the
main complaints about bitbake, so I'm not yet convinced moving is justified in
this case.


> Ideally the test CI and CT framwork of Isar would allow layering. While
> main Isar would be mc and slow on its thorough CI/CT layers could reuse
> parts.

Yes, that requires dependency support, we'll continue looking how to address
that. An alternative would be bitbake, which we didn't do :) .


> I think a proper modularization will save time even given the current
> CI.

As mentioned, we already agree that a framework would be useful.


> To get there we would need a CI system where the pipelines are part of
> the code, so probably not jenkins.

Just as a remark, Jenkins does support pipelines, we just don't use them; maybe
I should prioritize that. We did resurrect our GitLab, and I don't see how it
is better. Ideally, I want a different tool :) . In practice, many people use
one of the two, so I'd be interested in documenting both workflows.


With kind regards,
Baurzhan.