Re: Reminder: hotfix fast path needed

["'MOESSBAUER, Felix' via isar-users" <isar-users@googlegroups.com>]

On Thu, 2025-10-23 at 12:42 +0200, Baurzhan Ismagulov wrote:
> Hello Jan,
> 
> On 2025-10-23 07:52, Jan Kiszka wrote:
> > Patches that carry "Fixes:" tag and address build breakages should be on
> > a fast path for QA and merge. I'm pointing colleagues to [1] and [2]
> > almost on a daily basis, but those are just two recent examples.
> > 
> > [1] https://patchwork.isar-build.org/project/isar/patch/20251017085344.2647058-1-felix.moessbauer@siemens.com/
> > [2] https://patchwork.isar-build.org/project/isar/patch/0f16240d-aca7-4f03-b0f7-1567c5b4c26f@siemens.com/
> 
> I acknowledge the desire to have fast path for "Fixes:" and build breakages.
> We've merged #2. The full results for #1 will be available tomorrow. There is a
> number of failures due to e.g. external networking issues; we'll be sharing
> patches for those.

That's one of the core problems: The test suite itself is too unstable
and also way too complicated to be executed on each addition (actually
tests should be executed on a per-commit level, not per patch-series
level).

> 
> I think introducing fast path for certain changes is the easiest part of the
> equation.
> 

That's at least something which can be implemented process-wise without
a major refactoring of the testsuite.

> In #2, you also express regret that there is no testcase for the
> feature in question. In general, we should think about adding testcases on use
> case introduction and bug fixes -- here I agree with Cedric.
> 

I fully agree, but guidance is needed regarding how to write the tests
(e.g. to which tag to add them). Also, the test execution takes WAY too
long. We should consider allowing the use of the sstate cache and
download caching in the tests to significantly speedup the test
execution. That's also what Yocto is doing.

Regarding the test-execution itself: What we need is a simply CLI to
execute the tests in a container (this CLI should offer options to list
the testcases without executing them and executing just some tests).

> I think this has
> larger contribution to the quality, benefits all downstreams and saves the
> overall fixing effort, as every applied change could break the feature. Once a
> fix goes in, a testcase never follows. Currently, we are developing testcases
> for certain changes from the list; longer-term, this will not work.

Also agree, but we need to start somewhere. In the long term, I
envision test automation based on the ML, so whenever you send a patch
to the ML a set of possibly related tests is executed and the result is
sent back to the ML. For that, the cost of the test execution needs to
be reduced (e.g. executing a per-patch test run must not take longer
than 10 mins) and hardware needs to be setup.

> 
> The Linux "tested by many" model helps only to some extent. We also see e.g.
> conflicting changes being applied back and forth, even in Linux. Having the use
> case view backed by tests at the Isar level would avoid this.

With syzcaller and alike being added to Linux, this model is anyways no
longer the only used one.

Best regards,
Felix

> 
> To summarize, we'll evaluate this proposal and also invite for more
> collaboration on testcases.
> 
> With kind regards,
> Baurzhan

-- 
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/e862e11d340776b8913e932fb9111e129a7122e2.camel%40siemens.com.