Re: [PATCH 0/3] Make CI targets be configured

[Anton Mikanovich <amikan@ilbers.de>]

27.04.2022 18:59, Henning Schild wrote:
> The test suite already has too many knobs, now including env variables
> would be the number one cause of "works for me" ... does not work in
> CI-A and even different in CI-B.
>
> I think if i pushed this series to our CI it would simply fail, because
> i would not have any of the variables set there ... and no clue how you
> did set them on your jenkins.
>
> Tests that are known to not work should probably be skipped in general,
> not kfailed because that would just waste time.
>
> If we ever see certain tests not working in certain setups we can see
> how we make skipping configurable, but it is imho a very bad idea to
> introduce that without a need.
>
> So i would say let us get rid of some of the KFAILs and turn the still
> valid ones into SKIPs ... and see about the rest laster.
>
> Henning
>
> Am Wed, 27 Apr 2022 15:32:02 +0300
> schrieb Anton Mikanovich <amikan@ilbers.de>:
>
>> This patchset removes hardcoded KFAIL protection from targets and
>> makes any target be KFAILed or SKIPed based on execution environment.
>> ISAR_CI_KFAIL and ISAR_CI_SKIP variables are used to store those
>> lists.
>>
>> Anton Mikanovich (3):
>>    ci: Implement dynamic KFAIL checking
>>    ci: Implement dynamic tests skipping
>>    ci: Correct container test case name
>>
>>   testsuite/cibuilder.py | 56
>> ++++++++++++++++++++++++++++-------------- testsuite/citest.py    |
>> 32 ++++++------------------ 2 files changed, 45 insertions(+), 43
>> deletions(-)

Using env variables is the only suitable way for Gitlab, that's why it 
was done
that way. Env is quite hard to use for Jenkins CI, so any better 
solutions are
welcome.

Applying those patches without setting any env variables will just make 
CI to
run without any KFAILs, so it's not a problem.

We need to have KFAIL mostly for upstream related issues as temporary 
hotfix,
so we can't maintain this via commits. Removing KFAILs will make us to 
freeze
all merging until any single upstream issue on unstable target will be 
fixed.
Otherwise if we keep static KFAILs we will always have 'hidden' issues just
like we already had with python update on bookworm.

SKIPs are implemented for testing purposes to make CI run faster then all
targets check during single user-specific testing cases.