Re: [PATCH 0/3] Make CI targets be configured

[Henning Schild <henning.schild@siemens.com>]

Am Wed, 27 Apr 2022 20:50:32 +0300
schrieb Anton Mikanovich <amikan@ilbers.de>:

> 27.04.2022 18:59, Henning Schild wrote:
> > The test suite already has too many knobs, now including env
> > variables would be the number one cause of "works for me" ... does
> > not work in CI-A and even different in CI-B.
> >
> > I think if i pushed this series to our CI it would simply fail,
> > because i would not have any of the variables set there ... and no
> > clue how you did set them on your jenkins.
> >
> > Tests that are known to not work should probably be skipped in
> > general, not kfailed because that would just waste time.
> >
> > If we ever see certain tests not working in certain setups we can
> > see how we make skipping configurable, but it is imho a very bad
> > idea to introduce that without a need.
> >
> > So i would say let us get rid of some of the KFAILs and turn the
> > still valid ones into SKIPs ... and see about the rest laster.
> >
> > Henning
> >
> > Am Wed, 27 Apr 2022 15:32:02 +0300
> > schrieb Anton Mikanovich <amikan@ilbers.de>:
> >  
> >> This patchset removes hardcoded KFAIL protection from targets and
> >> makes any target be KFAILed or SKIPed based on execution
> >> environment. ISAR_CI_KFAIL and ISAR_CI_SKIP variables are used to
> >> store those lists.
> >>
> >> Anton Mikanovich (3):
> >>    ci: Implement dynamic KFAIL checking
> >>    ci: Implement dynamic tests skipping
> >>    ci: Correct container test case name
> >>
> >>   testsuite/cibuilder.py | 56
> >> ++++++++++++++++++++++++++++-------------- testsuite/citest.py    |
> >> 32 ++++++------------------ 2 files changed, 45 insertions(+), 43
> >> deletions(-)  
> 
> Using env variables is the only suitable way for Gitlab, that's why
> it was done
> that way. Env is quite hard to use for Jenkins CI, so any better 
> solutions are
> welcome.

Simply not making that configurable, use code as we used to.

> Applying those patches without setting any env variables will just
> make CI to
> run without any KFAILs, so it's not a problem.

The pipeline will turn red on known issues. And always red would render
the pipeline less meaningful, the overall result would even be
meaningless.

> We need to have KFAIL mostly for upstream related issues as temporary 
> hotfix,
> so we can't maintain this via commits. Removing KFAILs will make us

Yes i suggest to use code, that will magically spread to all local and
CI builds. And all the known upstream issues will be dealt with,
without everybody really knowing why.

> to freeze
> all merging until any single upstream issue on unstable target will
> be fixed.

I guess the merge policy is up to the maintainer. In certain cases it
might sure be fine to merge a red pipeline.

regards,
Henning

> Otherwise if we keep static KFAILs we will always have 'hidden'
> issues just like we already had with python update on bookworm.
> 
> SKIPs are implemented for testing purposes to make CI run faster then
> all targets check during single user-specific testing cases.
>