From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7091260455110311936 X-Received: by 2002:a5d:5309:0:b0:20a:cee3:9e3f with SMTP id e9-20020a5d5309000000b0020acee39e3fmr21844120wrv.570.1651132089646; Thu, 28 Apr 2022 00:48:09 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:adf:f950:0:b0:205:dbf5:72d8 with SMTP id q16-20020adff950000000b00205dbf572d8ls3293865wrr.0.gmail; Thu, 28 Apr 2022 00:48:08 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxPv3gCo86N/WJpXzDEugEvjXAMMTbNGfKz1l2ckBU9/4lkZyfQ/bJUrtDgFFQMaIQs/iXl X-Received: by 2002:a05:6000:188d:b0:20a:a014:7ff6 with SMTP id a13-20020a056000188d00b0020aa0147ff6mr25269407wri.104.1651132088732; Thu, 28 Apr 2022 00:48:08 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1651132088; cv=pass; d=google.com; s=arc-20160816; b=KlY6zlVaMa508ex0uaGjQSkQpVejTUfhy0myT4y0zKbwJRAbUE40GneWbbEiJUkE+3 RMMUYG+SPVIvw/rhhZ4SXn9R1BtfjSKzi7zUF5v+dLcSgnQM9d+9wZAKc7HeGbSrVcYf G/4bnHk8vhBs/ymvUGs+N9Wps5AFJ2R5ERjWT0hMJDVllswzDpYlpdFve8Qv3jXVyLrL K6DVMH9O+W9gCBe9F5csto4rGgiNGYIcx1cNaocNnhIVbY3QuNbZdLifBOSt1bdNZAqA Zzku9JBvItHe+sl+VGRBQ72TM4gHWgGQKvMyVzPMSR3Eam3Yow3hcDN3uCcRwss6LUbG h0ow== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:references:in-reply-to :message-id:subject:cc:to:from:date:dkim-signature; bh=jQlOXhn4XMqSfx8OD8oWS4qC84QBIAjHyWlyxXJxNHo=; b=g8cLe9W5ZDnKX4aPM0BkpsVONKoFtwXQ7YofbjgdF4teiUqR0jzvvEv3MPxa2pFzft bfDTJ6r0hFPrNh5i5/3VU9NEwtOLejZdxH/HdcTIFmenint4IVoPkpObJbB2BuwCZzRf wp6Cm3exnPVUQOB3+XQFCB/VCVHH+h8Y+WS6+BiKrq0dg3dstI2nwfuLTDiOBAeVkrBW ShMFh7pNbaiBX8pWgXpae+NyAQqNWE6edqar+pSvwFMNVVDTB6fw/4zMgqATjnpd4Cuu mdIHjl/sZ0cEs9Vl4zgVXhLGbEDX2hqyHt8nIi/4EN4qZ+Zdiu1oIXr1bSrMS1COD3Vr S3XA== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=BDCnc3wr; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe0d::602 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-he1eur04on0602.outbound.protection.outlook.com. [2a01:111:f400:fe0d::602]) by gmr-mx.google.com with ESMTPS id ay12-20020a5d6f0c000000b0020ae674a22asi142738wrb.2.2022.04.28.00.48.08 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 28 Apr 2022 00:48:08 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe0d::602 as permitted sender) client-ip=2a01:111:f400:fe0d::602; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=BDCnc3wr; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe0d::602 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=E0ljCNA/Z8hl4RCya2k9e82aoDbdxwr+Tbk51XgMYnRM5dIF/2hl4XRX1bJQD0bo+E3DCfhPfGwqP86ZEoF4+DBGseqW9x/0isJCKB9Fk0pYZvTUMsHZFm2OByOdbz/8sz2xMFWPaDpgQ6SFQxc/aW1iF8/zuomeqqzHPd4mMlkj6ubAnNuq6j6EiOAMxUvkaag52VD9fUKUFElJZ7CQQSlvOnbakEIyH1jJmhM/bIqAbcUZk5RNQID+07gD3mI7AHBhBaz1eXJjesHSMoUjP8VgEDu1mO71CQQggc2remAv+pRoKYHBmXmYpO7MhVr1XuKDP0Iqqn6XDbz3mL8z1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jQlOXhn4XMqSfx8OD8oWS4qC84QBIAjHyWlyxXJxNHo=; b=hJdraMenM1/7kB3jnlltp8C2nCokvW80BgjIKsBORlzt36MlF3HPXU0y7zI5m+589qsFOw90jW3Si1jw9IodpF9wlg53zQ9YfCfHqk5zehpWVDt2uE13yWduo57z/JVp50po9dZBiL37j8EN/usfwM9nRjSRjPJYMKPGXZAGX0XR5qGeXU6oyNvljF7x3Z7znMDC9Avqy5dNAbxTamCkWJz9LKzpcwKsCIW8TE4yzXicfjq/i6jAiwSTyB7C7x6AOJqlpPtX1faM3WFTTuHJuebAfq0oV67X835+IRUAvYFiY966mZXe2Ce5VYC5dzrdklUXjShcipOqIEsOhGbHNQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jQlOXhn4XMqSfx8OD8oWS4qC84QBIAjHyWlyxXJxNHo=; b=BDCnc3wrU6pz7ZxQjp6bKKNOzV3b8q3ZdlpdmOKr82ACe2zxCmB633WLvrB0G3eq86vPCSKeFc2CYQMKStuk3lRTesUpxJWidD3zWSgnBvT5nYsODc33OHGP6HuYLcf6gZOG11dYD33fLeHyQUmhAR6bpnM5Nn8Nh4i0f7Q7T2Mu3yV2no7W4V1BD7iAfx53prulPA88R0nA2/+mgeHc5F4sIm1KNwAvbhGK+h+KCAduSGMi4BVhH3HnPU1z3Z1fNwK/uTuq3qofpe64v8Z4bT1779DiDC3PH92jNv3CfIYrFVBE6A8v3aTLzGI6NM4PJ+gHjAXdCYk9Gd/PheTneg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:269::8) by VI1PR10MB2288.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:803:7c::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.13; Thu, 28 Apr 2022 07:48:07 +0000 Received: from PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM ([fe80::f4be:8b5d:4314:c2d8]) by PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM ([fe80::f4be:8b5d:4314:c2d8%4]) with mapi id 15.20.5186.021; Thu, 28 Apr 2022 07:48:07 +0000 Date: Thu, 28 Apr 2022 09:48:03 +0200 From: Henning Schild To: Anton Mikanovich Cc: isar-users@googlegroups.com, Baurzhan Ismagulov Subject: Re: [PATCH 0/3] Make CI targets be configured Message-ID: <20220428094803.1d9977dc@md1za8fc.ad001.siemens.net> In-Reply-To: <4a35965d-cef5-8e1c-cd59-51da369ecf59@ilbers.de> References: <20220427123205.30111-1-amikan@ilbers.de> <20220427175942.73ea0c28@md1za8fc.ad001.siemens.net> <4a35965d-cef5-8e1c-cd59-51da369ecf59@ilbers.de> X-Mailer: Claws Mail 4.1.0 (GTK 3.24.31; x86_64-pc-linux-gnu) Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-ClientProxiedBy: AS9PR06CA0392.eurprd06.prod.outlook.com (2603:10a6:20b:461::32) To PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:269::8) Return-Path: henning.schild@siemens.com MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3ec369f0-7ffe-42ff-0bbe-08da28eb6ec7 X-MS-TrafficTypeDiagnostic: VI1PR10MB2288:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: cVr6CpQGCAf8zAVP4hRdSbsLOS3iPfoQBCUeZuX9MrllBXaxaefhWPklCtExvRxXIOvT+cnC0GghURo0zaSrglrBzRLa1Vlpp5CfjdgPnbe+1fmqmKHnBinZJBrcA6k+/DcUwa/8DEgNS2vosyYrB3mM3LHBSfkik6JXcmGfdCyoPSSAh8XwV27a8MOHi7IoLrdroizTy2FJu+ZUxkKYJdSmFZjw0c800MXDH+uH2rQ1M+tFWBF0tuTH0lAdDs0u4DNIPdeWOs4bhzLA2ouXdeFQa6bp3vH8HIkUKDcCk+pxwYYSsE5BBp4M8InhtXNBRpUwT8K2qtnjbLL41rzbbDesUJY/s/hlITPkXAVH6rIRqzYiwvdGLLsrep1MczK3ttZc/TDA10zQ1ah7VhpKu4f/oI3c2rqMfpYZ/DniQb/n5ucCxslZe3AotCBppeMn+afSU8mBaqwF35TgMwJ3hvBYBxfcwn8d8X0n6nG40djCIxEL6GdPk9edNHEgjAKlEmkikKZQJJ2F+H8SpT5UlVl6VwVujMqcg4UaYpaLCdkze5U+aouzscjlaKnHpg1phBaqmtu/LX9IaOFHT9b5up3n5zdnWYjwCx8HR/cKkXF0GeN7bgTS3UysLJlgOo2jqmKEAQF3xQE6/gOogH3ROQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(2906002)(44832011)(66946007)(5660300002)(186003)(66476007)(316002)(66556008)(83380400001)(8676002)(4326008)(6916009)(8936002)(86362001)(6486002)(508600001)(38100700002)(82960400001)(6666004)(1076003)(9686003)(6512007)(6506007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 2 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?syzl6xYJzIGJKeArsDMEK4Jl3uE0PFNl/lM3eDqatTkNPxFlAm0uNa2lSacB?= =?us-ascii?Q?KOypAJN2XHK8MH5Pn+FSbUp7YHbyg74Q2kkAfaHcWVfIRVZd8BKiQARcVT6Y?= =?us-ascii?Q?fzIPqc4ThkB/VmIBh8ygKSkMDskhuzjIZssFQejplBQvFfvuepc55riJxL69?= =?us-ascii?Q?nF2GmN+rUNb+bKDWhviDfwBrHp1MHv4qNMljAhtW9rQCLR7h89UuTrWxgSNa?= =?us-ascii?Q?cdToTvO8+j48Ne5Av4KHpiaAHBPLC+u+gSnmw1hxByljNng/Wzd/qjAEcspb?= =?us-ascii?Q?3i93fylAameTRP2ayMyvZI5jhR6gqDatEQWGwb+8mUOn4XGFTtpAzcvht3X6?= =?us-ascii?Q?QoMZK/noBv9pdgAG4kOgvsaxibqBHpna+qLIgXXZ5AEus4dUQbi/m3k/JJTG?= =?us-ascii?Q?AQ0DRDbl/H/UD9uBj3hZFpghcPhJ4LPYS2p1MLFAET1m8mH2+OC61Z61ZYfN?= =?us-ascii?Q?DSTSKGmpn6Miejt787Sl+kb22B+02XWgKPtJRbPzsqWThKJn1eXd83obHHeR?= =?us-ascii?Q?N+Dj4CM7DdJDIxMJjS4vB9r98QiQMnedMb4KGA9M3rm4YJoderfxeRuqx08y?= =?us-ascii?Q?GcZLbZnLSFbFKTIadZKUycL34HfTTsdgHGh/A1kn3mT5YQYgM/Xjkadfpp1C?= =?us-ascii?Q?MX7Z+YcGQXlDbCEjM16Lo/whbnmCZDgdeN5r5eIiYYKsVRp7JRFTO7M1W9Ex?= =?us-ascii?Q?GtbUqYdDwWfSsEMUaEJyhPnWfBBmngyzsOONuUcs00R89xA8GMqYYjGj8CiX?= =?us-ascii?Q?xyBJpmbJkaPUMr8Lwk4HqMAHsyW7bvDTrfkP45z48sfGXnIxkcdrD5J+1bR9?= =?us-ascii?Q?ZGSBMkLC2Pf0hfi9gdIpPc3FzLu7rp3HzgFYVLUo81dZ6cCrkPMDiq9r22F5?= =?us-ascii?Q?g4lRPNT0OQnxNCWZjHmQE350RAU5NHxvceqmwnR8+zx2cbdjrCsN6aN7B02+?= =?us-ascii?Q?Yj3w1VtXwU467I6k8qMPnxFju5zsUVQ/X6QLAXoKH/QcVdI1D90DId3A/Qg8?= =?us-ascii?Q?jElgBpfr47gEr0D6YWwvpX3yhwPW53PHsaEJz5xWjd7HK9I1zJxCYljC3t42?= =?us-ascii?Q?dJR2Y0izciVamO2kBFnbF8OlVhwO3SIvtJGzmfPVg7oqar5a344NRSZzvOTM?= =?us-ascii?Q?pz64gxFmJVIdWmGM0Nu6Cs22a6du83NIBDN0LYZhyRyXeJCgU8ifXta4C15U?= =?us-ascii?Q?4J8XOC1Nn4oX1tUvn3Lk++dUyRaqkNOfd+mWoXU2gfaXPNenr+2B0SZ1TWLE?= =?us-ascii?Q?lEt4iOQ0i+gokMTlfRxD9DNPAIe2s98njLw7uZVoPfMJbnCRvyXqBoX+mgPO?= =?us-ascii?Q?kLS48CYSnLDiRU7X+r+zNhrS9NooVYz3OJM+gSdDD+QClPcPB+5fP2fBZyE2?= =?us-ascii?Q?fBFPm4ts7k2mbso8ReCw8g5Xo6wWtgjYdIqMTETGoPGFnFChfQ5wmMKtdJQ9?= =?us-ascii?Q?odIpKIqs0V5/cF1Y4TGy+Pae5vICznpuHMz2mOXbAe9cSvGb4lfBYeK6oqT/?= =?us-ascii?Q?dBv1PbIxp4B7UmG6jOvGb1rYxIpEKF/rndjrG71a1R5Vlwy597VQo5Wzb7Lh?= =?us-ascii?Q?gdPHlQjJOxxl5U3ChnYBBoGWe5yYPK79QOMfxPj1Lnk77/hXVI4W41eQdRXc?= =?us-ascii?Q?lw/K4QESfYtKVslbR26pZwlwXdYkIzKVrejF7KwPRhsDQxypO2tz7khRw+QH?= =?us-ascii?Q?77B6Pgu7DRruMAqB5e/EItCFv8pwULaoVWIth9kgN5tt8F+igXNPc7QWkz4s?= =?us-ascii?Q?tqHVSdwtIUre4+ec2Moa+rhYjv5cIlibccX8tSCrAAax/CBuAIKJaKrd7EkI?= X-MS-Exchange-AntiSpam-MessageData-1: JUk0XvxE0GYg+CZt5iYPNdVkYEPWzVuM5mg= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3ec369f0-7ffe-42ff-0bbe-08da28eb6ec7 X-MS-Exchange-CrossTenant-AuthSource: PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Apr 2022 07:48:06.9824 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: FJTBif9qK6nlKg7JQh6S6GNjve6mviw4k2XcimFhHLLe8bvuJZgUsgHFIsaDJ84Xku8c2mnZB4E/vpTUvgAK5bh/d9/L4Ay22gcqLZNTo40= X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR10MB2288 X-TUID: z4G4vLzjggl3 Am Wed, 27 Apr 2022 20:50:32 +0300 schrieb Anton Mikanovich : > 27.04.2022 18:59, Henning Schild wrote: > > The test suite already has too many knobs, now including env > > variables would be the number one cause of "works for me" ... does > > not work in CI-A and even different in CI-B. > > > > I think if i pushed this series to our CI it would simply fail, > > because i would not have any of the variables set there ... and no > > clue how you did set them on your jenkins. > > > > Tests that are known to not work should probably be skipped in > > general, not kfailed because that would just waste time. > > > > If we ever see certain tests not working in certain setups we can > > see how we make skipping configurable, but it is imho a very bad > > idea to introduce that without a need. > > > > So i would say let us get rid of some of the KFAILs and turn the > > still valid ones into SKIPs ... and see about the rest laster. > > > > Henning > > > > Am Wed, 27 Apr 2022 15:32:02 +0300 > > schrieb Anton Mikanovich : > > > >> This patchset removes hardcoded KFAIL protection from targets and > >> makes any target be KFAILed or SKIPed based on execution > >> environment. ISAR_CI_KFAIL and ISAR_CI_SKIP variables are used to > >> store those lists. > >> > >> Anton Mikanovich (3): > >> ci: Implement dynamic KFAIL checking > >> ci: Implement dynamic tests skipping > >> ci: Correct container test case name > >> > >> testsuite/cibuilder.py | 56 > >> ++++++++++++++++++++++++++++-------------- testsuite/citest.py | > >> 32 ++++++------------------ 2 files changed, 45 insertions(+), 43 > >> deletions(-) > > Using env variables is the only suitable way for Gitlab, that's why > it was done > that way. Env is quite hard to use for Jenkins CI, so any better > solutions are > welcome. Simply not making that configurable, use code as we used to. > Applying those patches without setting any env variables will just > make CI to > run without any KFAILs, so it's not a problem. The pipeline will turn red on known issues. And always red would render the pipeline less meaningful, the overall result would even be meaningless. > We need to have KFAIL mostly for upstream related issues as temporary > hotfix, > so we can't maintain this via commits. Removing KFAILs will make us Yes i suggest to use code, that will magically spread to all local and CI builds. And all the known upstream issues will be dealt with, without everybody really knowing why. > to freeze > all merging until any single upstream issue on unstable target will > be fixed. I guess the merge policy is up to the maintainer. In certain cases it might sure be fine to merge a red pipeline. regards, Henning > Otherwise if we keep static KFAILs we will always have 'hidden' > issues just like we already had with python update on bookworm. > > SKIPs are implemented for testing purposes to make CI run faster then > all targets check during single user-specific testing cases. >