From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7091260455110311936 X-Received: by 2002:a19:501b:0:b0:471:f9dd:775e with SMTP id e27-20020a19501b000000b00471f9dd775emr16293532lfb.582.1651081842715; Wed, 27 Apr 2022 10:50:42 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:3f16:b0:449:f5bf:6f6a with SMTP id y22-20020a0565123f1600b00449f5bf6f6als2564400lfa.2.gmail; Wed, 27 Apr 2022 10:50:41 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxM/98cWSmNxyuIsThcGWPuDXj7XK9RKV7SZ8U+pfex5qVpkCO/OmxFHnDs+EG8e0RbUH4A X-Received: by 2002:a05:6512:3e0c:b0:472:34bc:621b with SMTP id i12-20020a0565123e0c00b0047234bc621bmr466125lfv.244.1651081841600; Wed, 27 Apr 2022 10:50:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1651081841; cv=none; d=google.com; s=arc-20160816; b=J0mJGqtnO/TwFQCD/56/wSD13kYl+r2MGzS4JXMZxoKZ0sjhzY9NeXg+hvHOuNLkiN ZIENSD5S/DyiwJ2Y86vQGWd749ieT2fKa/lC4AkAtt5E25JbLp95X65ZgzCrQIu9ksn0 b81BtbgmL2zp85Ysg6gzSma9kWtrAA87XfVtJWSocdWklvBTt9eRbVQlvDS3sM4MfFbZ CFsX4FG2Ud2dKn3nXO2QqfjDZ4BswFzlRXDmpKkMGDknDVks5QKbEfgHUYuDOJf88fuC PTh/4N9ZdOYClLFp/y7DjULRGFeuMt9HfgjbP+HOIkmfybn1hljSaeIgaLlalZqvRR9x ytKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id; bh=7nVI3GpjYqdmR63WCsOPrhOrwkDBXB9OWdqIf+Uz3+M=; b=cCK0C/2YftlYXCDW6PoBznsSCgrn6p18W5SY54iLE97eQHitZcU1yl2cv2TBzAAUs6 Z0lB4nphfjyG2NulwMlm/yNh9wcMI0tOyOf9p+UCBg4WQAtL9NTlg7nj2GZhUNyotYnT TZGXb/zOYeZm5hEIqkDrjyrQx9CKYFVPNmS8PhTSVwgZfsXm6aE4W9IlLWvBN5kjyuJ3 vM10YkhS8qyGgRtkXf520JoxNpea5RaWg23cxYL5nEqXjJximU7vqa5TAiHqapL1pwRA q9XbUGhvbb/s+dKk+paJ0Iyw8WIdfUgXNWFcnGpTatYdiXLEX4qAmV9uqERdqQiWgzQT TjjA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id i42-20020a0565123e2a00b004721a3e27cfsi117363lfv.12.2022.04.27.10.50.41 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 27 Apr 2022 10:50:41 -0700 (PDT) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from [127.0.0.1] (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 23RHobrC003485 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 27 Apr 2022 19:50:38 +0200 Message-ID: <4a35965d-cef5-8e1c-cd59-51da369ecf59@ilbers.de> Date: Wed, 27 Apr 2022 20:50:32 +0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.5.0 Subject: Re: [PATCH 0/3] Make CI targets be configured Content-Language: en-US To: Henning Schild Cc: isar-users@googlegroups.com, Baurzhan Ismagulov References: <20220427123205.30111-1-amikan@ilbers.de> <20220427175942.73ea0c28@md1za8fc.ad001.siemens.net> From: Anton Mikanovich In-Reply-To: <20220427175942.73ea0c28@md1za8fc.ad001.siemens.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: MB87VOXPbTH/ 27.04.2022 18:59, Henning Schild wrote: > The test suite already has too many knobs, now including env variables > would be the number one cause of "works for me" ... does not work in > CI-A and even different in CI-B. > > I think if i pushed this series to our CI it would simply fail, because > i would not have any of the variables set there ... and no clue how you > did set them on your jenkins. > > Tests that are known to not work should probably be skipped in general, > not kfailed because that would just waste time. > > If we ever see certain tests not working in certain setups we can see > how we make skipping configurable, but it is imho a very bad idea to > introduce that without a need. > > So i would say let us get rid of some of the KFAILs and turn the still > valid ones into SKIPs ... and see about the rest laster. > > Henning > > Am Wed, 27 Apr 2022 15:32:02 +0300 > schrieb Anton Mikanovich : > >> This patchset removes hardcoded KFAIL protection from targets and >> makes any target be KFAILed or SKIPed based on execution environment. >> ISAR_CI_KFAIL and ISAR_CI_SKIP variables are used to store those >> lists. >> >> Anton Mikanovich (3): >> ci: Implement dynamic KFAIL checking >> ci: Implement dynamic tests skipping >> ci: Correct container test case name >> >> testsuite/cibuilder.py | 56 >> ++++++++++++++++++++++++++++-------------- testsuite/citest.py | >> 32 ++++++------------------ 2 files changed, 45 insertions(+), 43 >> deletions(-) Using env variables is the only suitable way for Gitlab, that's why it was done that way. Env is quite hard to use for Jenkins CI, so any better solutions are welcome. Applying those patches without setting any env variables will just make CI to run without any KFAILs, so it's not a problem. We need to have KFAIL mostly for upstream related issues as temporary hotfix, so we can't maintain this via commits. Removing KFAILs will make us to freeze all merging until any single upstream issue on unstable target will be fixed. Otherwise if we keep static KFAILs we will always have 'hidden' issues just like we already had with python update on bookworm. SKIPs are implemented for testing purposes to make CI run faster then all targets check during single user-specific testing cases.