Re: Reminder: hotfix fast path needed

["'cedric.hombourger@siemens.com' via isar-users" <isar-users@googlegroups.com>]

On Thu, 2025-10-23 at 11:05 +0000, 'MOESSBAUER, Felix' via isar-users
wrote:
> On Thu, 2025-10-23 at 12:42 +0200, Baurzhan Ismagulov wrote:
> > Hello Jan,
> > 
> > On 2025-10-23 07:52, Jan Kiszka wrote:
> > > Patches that carry "Fixes:" tag and address build breakages
> > > should be on
> > > a fast path for QA and merge. I'm pointing colleagues to [1] and
> > > [2]
> > > almost on a daily basis, but those are just two recent examples.
> > > 
> > > [1]
> > > https://patchwork.isar-build.org/project/isar/patch/20251017085344.2647058-1-felix.moessbauer@siemens.com/
> > > [2]
> > > https://patchwork.isar-build.org/project/isar/patch/0f16240d-aca7-4f03-b0f7-1567c5b4c26f@siemens.com/
> > 
> > I acknowledge the desire to have fast path for "Fixes:" and build
> > breakages.
> > We've merged #2. The full results for #1 will be available
> > tomorrow. There is a
> > number of failures due to e.g. external networking issues; we'll be
> > sharing
> > patches for those.
> 
> That's one of the core problems: The test suite itself is too
> unstable
> and also way too complicated to be executed on each addition
> (actually
> tests should be executed on a per-commit level, not per patch-series
> level).
> 
> > 
> > I think introducing fast path for certain changes is the easiest
> > part of the
> > equation.
> > 
> 
> That's at least something which can be implemented process-wise
> without
> a major refactoring of the testsuite.
> 
> > In #2, you also express regret that there is no testcase for the
> > feature in question. In general, we should think about adding
> > testcases on use
> > case introduction and bug fixes -- here I agree with Cedric.
> > 
> 
> I fully agree, but guidance is needed regarding how to write the
> tests
> (e.g. to which tag to add them). Also, the test execution takes WAY
> too
> long. We should consider allowing the use of the sstate cache and
> download caching in the tests to significantly speedup the test
> execution. That's also what Yocto is doing.
> 
> Regarding the test-execution itself: What we need is a simply CLI to
> execute the tests in a container (this CLI should offer options to
> list
> the testcases without executing them and executing just some tests).
> 

Some weeks back, we have added some hints on how to start a container
where you may run the test-suite with all of its dependencies
installed:

kas/kas-container shell \
    --command /work/isar/scripts/ci_setup.sh kas.yml

I think we can add a simple wrapper to make it even easier to invoke (I
always have to open the contributing guide to look it up).

I could modify the entry script to give a few hints such as:
- run "avocado list citest.py" to list existing tests
- use "avocado -p sstate=1 run citest.py" to use sstate (thanks Felix
for that hint!)
- use "avocado run citest.py:TestClass.test_method" to run a specific
test

As I am modifying some test cases to catch some of the recent
breakages, I am volunteering to enhance the documentation to get the
ball rolling

> > I think this has
> > larger contribution to the quality, benefits all downstreams and
> > saves the
> > overall fixing effort, as every applied change could break the
> > feature. Once a
> > fix goes in, a testcase never follows. Currently, we are developing
> > testcases
> > for certain changes from the list; longer-term, this will not work.
> 
> Also agree, but we need to start somewhere. In the long term, I
> envision test automation based on the ML, so whenever you send a
> patch
> to the ML a set of possibly related tests is executed and the result
> is
> sent back to the ML. For that, the cost of the test execution needs
> to
> be reduced (e.g. executing a per-patch test run must not take longer
> than 10 mins) and hardware needs to be setup.
> 
> > 
> > The Linux "tested by many" model helps only to some extent. We also
> > see e.g.
> > conflicting changes being applied back and forth, even in Linux.
> > Having the use
> > case view backed by tests at the Isar level would avoid this.
> 
> With syzcaller and alike being added to Linux, this model is anyways
> no
> longer the only used one.
> 
> Best regards,
> Felix
> 
> > 
> > To summarize, we'll evaluate this proposal and also invite for more
> > collaboration on testcases.
> > 
> > With kind regards,
> > Baurzhan
> 
> -- 
> Siemens AG
> Linux Expert Center
> Friedrich-Ludwig-Bauer-Str. 3
> 85748 Garching, Germany
> 

-- 
Cedric Hombourger
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/44608f2ccd231c60eed3e8d446c2e89023d5e303.camel%40siemens.com.