Re: [PATCH v10 17/17] ci_build: Install Avocado if needed

[Baurzhan Ismagulov <ibr@radix50.net>]

On Thu, Aug 12, 2021 at 05:58:58PM +0200, Henning Schild wrote:
> > diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
...
> > +# Install Avocado if needed
...
> I do not like this.
...
> apt must not be used in scripts ... and it tells you about it.

That was only intended to help testing the series, dropped in v11. The
execution environment should provide the tools, with Debian packages or pip.


> The package should be mainlined into debian

We've filed an ITP: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993862


> And i want more than "buster".

Provided stretch, buster, bullseye, bookworm at
http://deb.isar-build.org/debian-isar. The bookworm one should work for sid,
but we haven't tested it.


> I started looking into the avocado tests we currently have. Seems that they
> are still pretty different from what we used to have. Will this new version
> implement all the tests we used to have?
...
> I was only playing with what we currently have in next. But that seems to
> serialize the builds listed in that yaml. While before bitbake did run them
> all at once.

The initial Avocado implementation (in next today) provided only a matrix for
building all of individual multiconfigs, one per testcase. Yes, v11 intends to
provide exactly the same testcases that we have in ci_build.sh today. We could
add the individual single multiconfigs later if needed.


> My solution was to install that very unlucky choice of software with
> virtualenv

I assume this is about Avocado -- perhaps you could share some specific issues
you'd like to have addressed?

We did spend some time to evaluate our alternatives before choosing a specific
tool. Avocado is suitable for running Python and shell testcases, is not
limited to module testing, and is written in Python. It is a part of an
ecosystem supporting use cases like virtual machines, parallel execution,
easy testsuite composing via testcase tags, etc. It is the successor of
Autotest and has wide and very active community. We've used it in other
projects before Isar and got good impression. Having a test framework is waaay
better than maintaining ci_build.sh -- in the same way Isar is better than
build.sh ;) .

Of course, we are open to suggestions and improvements. One thing we have on
our radar is artifact reusing and testcase dependencies. E.g., with parallel
testcase execution, a start_vm testcase could depend on a bitbake one.


> I think the tests from that yaml should run in parallel to cause the same
> kind of stress ... just parallel in the testsuite instead of bitbake. And to
> hopefully speed things up a bit.

For the migration step, we aim at providing exactly the same testsuite as in
ci_build.sh and rework in later steps -- we'll share an RFC for that. Regarding
parallel execution, I think we need both parallel and bitbake with several
multiconfigs. This tradeoff isn't only about performance and stress; the latter
use case is necessary for per-build stuff (build numbers, release management,
etc.) and could potentially catch issues like missing dependencies, locking,
etc. that would be missed with the parallel execution of single multiconfig
builds.

A note on yaml -- that approach was very rigid, it supported only combinations
of single multiconfigs per testcase (we need more as described above) and
executed all of them (we need less to minimize the execution time).


With kind regards,
Baurzhan.