FYI: Feasibility of CI on github

FYI: Feasibility of CI on github

[Jan Kiszka]

Hi all,

just a note: github workflows can theoratically host our CI as well [1].
In practice, though, we are constrained by its disk limit [2], likely
also the job runtime limit (6h) for a full CI run. So, if we wanted to
make use of this, splitting up the CI script into smaller steps would be
needed.

That you can build full isar images on github has been proven in [3]
already.

Jan

[1] https://github.com/siemens/isar/commit/2340460507ebfffcbded835c9dff7c179cc9defc
[2] https://github.com/siemens/isar/runs/1862634380?check_suite_focus=true#step:5:1712
[3] https://github.com/siemens/meta-iot2050/actions

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: FYI: Feasibility of CI on github

[Henning Schild]

Hi,

that is very good news. And the current CI is a total mess anyways. All
in one 6h job. Maybe someone will find the time to improve this with
something like bats or pytest. And rip out that Avocado stuff ... does
anyone even use this?

Henning

Am Tue, 9 Feb 2021 13:53:29 +0100
schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:

> Hi all,
> 
> just a note: github workflows can theoratically host our CI as well
> [1]. In practice, though, we are constrained by its disk limit [2],
> likely also the job runtime limit (6h) for a full CI run. So, if we
> wanted to make use of this, splitting up the CI script into smaller
> steps would be needed.
> 
> That you can build full isar images on github has been proven in [3]
> already.
> 
> Jan
> 
> [1]
> https://github.com/siemens/isar/commit/2340460507ebfffcbded835c9dff7c179cc9defc
> [2]
> https://github.com/siemens/isar/runs/1862634380?check_suite_focus=true#step:5:1712
> [3] https://github.com/siemens/meta-iot2050/actions
> 

Re: FYI: Feasibility of CI on github

[Baurzhan Ismagulov]

On Fri, Feb 12, 2021 at 10:16:47AM +0100, Henning Schild wrote:
> that is very good news. And the current CI is a total mess anyways. All
> in one 6h job. Maybe someone will find the time to improve this with
> something like bats or pytest. And rip out that Avocado stuff ... does
> anyone even use this?

We have the current testsuite in Avocado, will send patches.

The long run time is the reason why we use our own servers for CI; currently,
we get it built in 4.5 h. This could be improved to some extent through
parallelizing the test cases, but CPU and storage utilization is about 100%, so
that would require several nodes. Also, looking at the task timing diagram, it
seems that dpkg-buildpackage is completely serialized; maybe we could improve
here, too.

We did look around before introducing Avocado. Pytest is fine for module tests,
but Avocado has more infrastructure for black box and target tests that we
need. We haven't looked at bats, but I'd like stay away from shell in favor of
Python.

What I don't understand is, after we have individual test cases, how does that
help us with the long run time? Would you schedule every test case in different
steps?

With kind regards,
Baurzhan.

Re: FYI: Feasibility of CI on github

[Henning Schild]

Am Fri, 12 Feb 2021 11:01:21 +0100
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> On Fri, Feb 12, 2021 at 10:16:47AM +0100, Henning Schild wrote:
> > that is very good news. And the current CI is a total mess anyways.
> > All in one 6h job. Maybe someone will find the time to improve this
> > with something like bats or pytest. And rip out that Avocado stuff
> > ... does anyone even use this?  
> 
> We have the current testsuite in Avocado, will send patches.

Last time i checked avocado was not packaged properly and pip is not an
option. Plus the patches have been merged without a proper review and
we are still waiting for a user ... dead code without review where
people already expressed concerns on the choice of the tool

> The long run time is the reason why we use our own servers for CI;
> currently, we get it built in 4.5 h. This could be improved to some
> extent through parallelizing the test cases, but CPU and storage
> utilization is about 100%, so that would require several nodes. Also,
> looking at the task timing diagram, it seems that dpkg-buildpackage
> is completely serialized; maybe we could improve here, too.

There is something. Run many tests in parallel instead of all mc in
one. The all mc has its place because we have mc, but maybe not the
best choice for the default.

> We did look around before introducing Avocado. Pytest is fine for
> module tests, but Avocado has more infrastructure for black box and
> target tests that we need. We haven't looked at bats, but I'd like
> stay away from shell in favor of Python.

We never talked about it. I just came up with two examples. We should
probably find something that offers all we want but is easy to install
and maintain.
And one really important first step would be decoupling of CI and CT.

> What I don't understand is, after we have individual test cases, how
> does that help us with the long run time? Would you schedule every
> test case in different steps?

You get more control if you want to disable some aspects, you get
better reporting, you can fail individual tasks without failing "in
total". You do not need to "keep going" when something failed, or you
can keep going if that is what you want ... more control.

I do not even want to imagine how much time we all waste because of
that. Show me one person that can read the output and tell a temporary
network glitch from a real problem. One "retries" ... maybe a few times,
later compare the errors and eventually merge even it the test was red
all the time. 12+hours to test a single patch

I sometimes trigger your CI on "next" it did not go green in ages.

Henning

> With kind regards,
> Baurzhan.
> 

Re: FYI: Feasibility of CI on github

[Baurzhan Ismagulov]

On Fri, Feb 12, 2021 at 11:28:40AM +0100, Henning Schild wrote:
> Last time i checked avocado was not packaged properly

When we made our initial PoC with Avocado, we used their Debian packaging.
Unfortunately, it got broken with buster. I'll talk with upstream regarding
this.


> Plus the patches have been merged without a proper review

Your feedback happened to come after the merge; it has been addressed in our
patches to come. If you have further feedback, please share with us.


> There is something. Run many tests in parallel instead of all mc in
> one. The all mc has its place because we have mc, but maybe not the
> best choice for the default.

Ok, that is a viable option. We'd continue to test mc anyway, but maybe in a
separate, non-default job. I think we would still need to look at the problems
that we encountered with mc and define some minimal mc test case for the
default scenario.


> And one really important first step would be decoupling of CI and CT.

If CT = continuous testing, I've indeed handled that as one step in our
process; what could we decouple with which benefit?


> > What I don't understand is, after we have individual test cases, how
> > does that help us with the long run time? Would you schedule every
> > test case in different steps?
> 
> You get more control if you want to disable some aspects, you get
> better reporting, you can fail individual tasks without failing "in
> total". You do not need to "keep going" when something failed, or you
> can keep going if that is what you want ... more control.

Those are arguments for a test framework in general -- I agree, this is why
we've introduced Avocado in the first place. If I understood Jan correctly, his
point was to make the run time shorter -- this could rather be solved by a
shorter default testsuite. We would then run the full testsuite on better
hardware before merging. That said, we already have the shorter variant; maybe
we should switch it to be the default?


With kind regards,
Baurzhan.

Re: FYI: Feasibility of CI on github

[Jan Kiszka]

On 12.02.21 11:53, Baurzhan Ismagulov wrote:
> On Fri, Feb 12, 2021 at 11:28:40AM +0100, Henning Schild wrote:
>> Last time i checked avocado was not packaged properly
> 
> When we made our initial PoC with Avocado, we used their Debian packaging.
> Unfortunately, it got broken with buster. I'll talk with upstream regarding
> this.
> 
> 
>> Plus the patches have been merged without a proper review
> 
> Your feedback happened to come after the merge; it has been addressed in our
> patches to come. If you have further feedback, please share with us.
> 
> 
>> There is something. Run many tests in parallel instead of all mc in
>> one. The all mc has its place because we have mc, but maybe not the
>> best choice for the default.
> 
> Ok, that is a viable option. We'd continue to test mc anyway, but maybe in a
> separate, non-default job. I think we would still need to look at the problems
> that we encountered with mc and define some minimal mc test case for the
> default scenario.
> 
> 
>> And one really important first step would be decoupling of CI and CT.
> 
> If CT = continuous testing, I've indeed handled that as one step in our
> process; what could we decouple with which benefit?
> 
> 
>>> What I don't understand is, after we have individual test cases, how
>>> does that help us with the long run time? Would you schedule every
>>> test case in different steps?
>>
>> You get more control if you want to disable some aspects, you get
>> better reporting, you can fail individual tasks without failing "in
>> total". You do not need to "keep going" when something failed, or you
>> can keep going if that is what you want ... more control.
> 
> Those are arguments for a test framework in general -- I agree, this is why
> we've introduced Avocado in the first place. If I understood Jan correctly, his
> point was to make the run time shorter -- this could rather be solved by a
> shorter default testsuite. We would then run the full testsuite on better
> hardware before merging. That said, we already have the shorter variant; maybe
> we should switch it to be the default?
> 

Time is not our primary issue (you can easily run the sequential parts
of the CI script in separate jobs and gain quite a bit). It's the disk
size. But when we want to stress the full mc: pool, there is likely no
way around dedicated runners for the time being.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: FYI: Feasibility of CI on github

[Baurzhan Ismagulov]

On Fri, Feb 12, 2021 at 05:55:23PM +0100, Jan Kiszka wrote:
> Time is not our primary issue (you can easily run the sequential parts
> of the CI script in separate jobs and gain quite a bit). It's the disk
> size. But when we want to stress the full mc: pool, there is likely no
> way around dedicated runners for the time being.

Yes, I think we might be able to improve on the parts (beyond just splitting
into steps), but the full mc seems like a challenge for the time being.

With kind regards,
Baurzhan.

Re: FYI: Feasibility of CI on github

[Henning Schild]

Am Fri, 12 Feb 2021 11:53:06 +0100
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> On Fri, Feb 12, 2021 at 11:28:40AM +0100, Henning Schild wrote:
> > Last time i checked avocado was not packaged properly  
> 
> When we made our initial PoC with Avocado, we used their Debian
> packaging. Unfortunately, it got broken with buster. I'll talk with
> upstream regarding this.

That would be good. I think having debian would be good enough but not
having it is really bad.

> > Plus the patches have been merged without a proper review  
> 
> Your feedback happened to come after the merge; it has been addressed
> in our patches to come. If you have further feedback, please share
> with us.

I do not remember the details. It is quite possible. But a feature
merge without a user is something that should never have happened.
Like an RFC merge should never happen, but not the point of the thread.
We are already too far away from github workflows.

> 
> > There is something. Run many tests in parallel instead of all mc in
> > one. The all mc has its place because we have mc, but maybe not the
> > best choice for the default.  
> 
> Ok, that is a viable option. We'd continue to test mc anyway, but
> maybe in a separate, non-default job. I think we would still need to
> look at the problems that we encountered with mc and define some
> minimal mc test case for the default scenario.
>
> 
> > And one really important first step would be decoupling of CI and
> > CT.  
> 
> If CT = continuous testing, I've indeed handled that as one step in
> our process; what could we decouple with which benefit?

Yes i mean continous testing.

Starting those qemus and parsing their output is clearly testing and
should be another job. All the qemu-related variables need to leave the
machine configs.
These configs serve as a very bad example where people might think they
can configure the kernel cmdline in the machine cfg, even though they
might be on wic.

> > > What I don't understand is, after we have individual test cases,
> > > how does that help us with the long run time? Would you schedule
> > > every test case in different steps?  
> > 
> > You get more control if you want to disable some aspects, you get
> > better reporting, you can fail individual tasks without failing "in
> > total". You do not need to "keep going" when something failed, or
> > you can keep going if that is what you want ... more control.  
> 
> Those are arguments for a test framework in general -- I agree, this
> is why we've introduced Avocado in the first place. If I understood
> Jan correctly, his point was to make the run time shorter -- this
> could rather be solved by a shorter default testsuite. We would then
> run the full testsuite on better hardware before merging. That said,
> we already have the shorter variant; maybe we should switch it to be
> the default?

Ideally the test CI and CT framwork of Isar would allow layering. While
main Isar would be mc and slow on its thorough CI/CT layers could reuse
parts.
I think a proper modularization will save time even given the current
CI.

A green build might still take the full time. But a red build will be
faster and tell you which modules failed. Which will also allow you to
write patches for multiple problems for one red run ... better
reporting will result in better understanding and less builds. Which
saves build count and a lot of time on that end, even if the time of
such a build remains "high".

And pipelines with multiple jobs allow to hit retry buttons for just
the failed job.

I think you can see this
https://code.siemens.com/linux/las/-/pipelines/8264154

To get there we would need a CI system where the pipelines are part of
the code, so probably not jenkins.

Henning
 
> 
> With kind regards,
> Baurzhan.
> 

Re: FYI: Feasibility of CI on github

[Baurzhan Ismagulov]

On Fri, Feb 12, 2021 at 08:20:51PM +0100, Henning Schild wrote:
> > When we made our initial PoC with Avocado, we used their Debian
> > packaging. Unfortunately, it got broken with buster. I'll talk with
> > upstream regarding this.
> 
> That would be good. I think having debian would be good enough but not
> having it is really bad.

I've looked at the code, they've meanwhile updated it. I've submitted small
fixes, but we have the package and will test it.


> I do not remember the details. It is quite possible. But a feature
> merge without a user is something that should never have happened.

The code that is not used by default doesn't necessarily mean it is not used.
It's exactly because we wanted to experiment with different test case splits
that we didn't enable it by default too early. In the first step, we'd like to
mimic the existing test suite, then maybe add test cases for individual machine
builds, then switch the default.

That said, this topic is something we are often confronted with: We get an
obviously useful patch (e.g., a fix for a build failure in a certain scenario),
but don't have a test case for it. By this definition, it is code without a
user. As much as I'd like to have test cases for everything, I doubt we can
live up to that 100%.


> > > And one really important first step would be decoupling of CI and
> > > CT.  
> > 
> > If CT = continuous testing, I've indeed handled that as one step in
> > our process; what could we decouple with which benefit?
> 
> Yes i mean continous testing.
> 
> Starting those qemus and parsing their output is clearly testing and
> should be another job.

Ok, roughly, CI is building and CT is running. Yes, starting the images will go
to their own test cases. That said, at Isar CI we'd still like to include them
in the process. But downstreams can decide that themselves -- IIUC, it isn't
used at the Siemens CI today.


> All the qemu-related variables need to leave the machine configs.
> 
> These configs serve as a very bad example where people might think they
> can configure the kernel cmdline in the machine cfg, even though they
> might be on wic.

If the docs can be improved to avoid misunderstandings, I'd start with that
first. The qemu vars are already prefixed. Scattered settings is one of the
main complaints about bitbake, so I'm not yet convinced moving is justified in
this case.


> Ideally the test CI and CT framwork of Isar would allow layering. While
> main Isar would be mc and slow on its thorough CI/CT layers could reuse
> parts.

Yes, that requires dependency support, we'll continue looking how to address
that. An alternative would be bitbake, which we didn't do :) .


> I think a proper modularization will save time even given the current
> CI.

As mentioned, we already agree that a framework would be useful.


> To get there we would need a CI system where the pipelines are part of
> the code, so probably not jenkins.

Just as a remark, Jenkins does support pipelines, we just don't use them; maybe
I should prioritize that. We did resurrect our GitLab, and I don't see how it
is better. Ideally, I want a different tool :) . In practice, many people use
one of the two, so I'd be interested in documenting both workflows.


With kind regards,
Baurzhan.

Re: FYI: Feasibility of CI on github

[Jan Kiszka]

On 15.02.21 12:27, Baurzhan Ismagulov wrote:
> On Fri, Feb 12, 2021 at 08:20:51PM +0100, Henning Schild wrote:
>> To get there we would need a CI system where the pipelines are part of
>> the code, so probably not jenkins.
> 
> Just as a remark, Jenkins does support pipelines, we just don't use them; maybe
> I should prioritize that. We did resurrect our GitLab, and I don't see how it
> is better. Ideally, I want a different tool :) . In practice, many people use
> one of the two, so I'd be interested in documenting both workflows.
> 

Either use github workflows, registering your runner as a private one
against the repo (and control who can trigger it, likely not PRs which
we don't do anyway). That way, CI would become publicly readable.

Gitlab is fine as well as long as the instance is on the Internet
(private one or gitlab.com). That's how we do it in Xenomai right now
(with private AWS runners sponsored by us).

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux