Re: Reminder: hotfix fast path needed

[Zhihang Wei <wzh@ilbers.de>]

On 10/24/25 13:23, 'MOESSBAUER, Felix' via isar-users wrote:
> On Thu, 2025-10-23 at 11:05 +0000, 'MOESSBAUER, Felix' via isar-users
> wrote:
>> On Thu, 2025-10-23 at 12:42 +0200, Baurzhan Ismagulov wrote:
>>> Hello Jan,
>>>
>>> On 2025-10-23 07:52, Jan Kiszka wrote:
>>>> Patches that carry "Fixes:" tag and address build breakages should be on
>>>> a fast path for QA and merge. I'm pointing colleagues to [1] and [2]
>>>> almost on a daily basis, but those are just two recent examples.
>>>>
>>>> [1]https://patchwork.isar-build.org/project/isar/patch/20251017085344.2647058-1-felix.moessbauer@siemens.com/
>>>> [2]https://patchwork.isar-build.org/project/isar/patch/0f16240d-aca7-4f03-b0f7-1567c5b4c26f@siemens.com/
>>> I acknowledge the desire to have fast path for "Fixes:" and build breakages.
>>> We've merged #2. The full results for #1 will be available tomorrow. There is a
>>> number of failures due to e.g. external networking issues; we'll be sharing
>>> patches for those.
>> That's one of the core problems: The test suite itself is too unstable
>> and also way too complicated to be executed on each addition (actually
>> tests should be executed on a per-commit level, not per patch-series
>> level).
>>
>>> I think introducing fast path for certain changes is the easiest part of the
>>> equation.
>>>
>> That's at least something which can be implemented process-wise without
>> a major refactoring of the testsuite.
> I again tried really hard to work with the testsuite, but this beast
> has a will of its own...
>
> I tried executing the tests with sstate cache enabled (via the
> undocumented -p sstate=1 option), only to find out that my
> tests sporadically fail due to bitbake server errors (do I have to run
> with --max-parallel-task=1?).
Yes, "-max-parallel-task=1" is needed in this case. 
"citest.py:CrossTest" is a
test suite that contains multiple test cases. Running multiple test cases at
the same time could cause failures when dependencies exist.

An example to run a single test case could be:
$ avocado run citest.py:CrossTest.test_cross_rpi

Another issue is that running a command like:
$ avocado run citest.py:CrossTest.test_cross
will make avocado run all test cases whose names start with "test_cross"
simultaneously. This is actually a feature of avocado. To avoid this, a 
$ sign
is needed at the end of the command:
$ avocado run citest.py:CrossTest.test_cross$
I'll specify the exact command that avoids these issues when reporting CI
failures.
> Once done that, the CI failed because of
> missing disk space (in which mountpoint?) I still have plenty of space
> in the build dir.
Docker does create files under /var/lib/docker, I notice it could take 10+GB
when a lot of targets are being built at the same time.
> ./kas/kas-container shell kas/isar.yaml --command \
>       "rm -rf /work/build/conf && /work/scripts/ci_setup.sh"
> cd /work/testsuite
> avocado run citest.py:CrossTest -p sstate=1
>
> For me as a developer, the testsuite is currently not usable. I'm
> willing to add tests again, once the infrastructure is ready for that.
> But this requires upfront redesign.
>
> Felix
I’m aware that some tests take a long time to run. For example,
"citest.py:NoCrossTest.test_nocross$" includes 30 targets and can take 3 
hours
on our CI server. When a failure occurs in these tests, I'll identify the
exact target that caused the failure and provide the steps to build only the
necessary target — for example, by sharing a diff file for citest.py that
temporarily disable unrelated targets.

I hope this approach will make it easier for developers to reproduce CI
failures with avocado.

Best,
Zhihang


-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/c3db73d0-5753-41cd-a300-21bec09a0c96%40ilbers.de.