From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6927246286546534400 X-Received: by 2002:a05:6402:31a3:: with SMTP id dj3mr2545433edb.172.1613125722505; Fri, 12 Feb 2021 02:28:42 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:906:364e:: with SMTP id r14ls4423020ejb.3.gmail; Fri, 12 Feb 2021 02:28:41 -0800 (PST) X-Google-Smtp-Source: ABdhPJxZTXKRqx+K0i/rol5KFm7uxBiV5mb02SMZaJLCZKLVc+8+/pOXqys+B4CBToz9Zn/hTaXL X-Received: by 2002:a17:907:948d:: with SMTP id dm13mr2209837ejc.545.1613125721619; Fri, 12 Feb 2021 02:28:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1613125721; cv=none; d=google.com; s=arc-20160816; b=gkB79AXvtpG2iSQIdjr3FeDCOnfo2QH7SPzHc4ZOzQ+xHIwMzN00Jy0s/9GQdYhk5+ Bb5LS3Ta5vDyVnjltdSjZBJn8wIiddkmt2IrSlVSx93n30naf+lLMGKHKO+/g/9iSVp+ vgJztxHOKFmQS8ta9Sonb98ApYJT75lMhcRcIcNBzxeXIxOj7Y/I2rM1FMR3iAE5uPmI VbE2N8fIpM+IzVLWoHcFLdDOLJfpZbXmOG51KJP2YV2UUFPoquA+JlWf/dd8LdW81eVU qKeDgN7F8KjDzSDH8+GLIvybaNt6kFw5zcIZXj/BjSGxXpkBhgxOZ2NpW5cvbzy6vYtr 19tw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date; bh=+U4qhJJJOh4J6Af+zir9g9FtrH1dOTZlL493ajDvG8o=; b=GPP7qGFkwsXpaC9YcggMbqovgmA/0ftaSNz1TWrjrCt9Tm/QTZgYGy8D7JvO1iXJvx Ym3NxMKOiJkabcOrtwHRe8QpxTHLVvjBmQq+NtrKZ+By5dga82uuxKxjJSuAJOw0GftG KohMFzWOR7I2+4hTFSu7PMauu8HZiRuE8KVPHgJR91b9LNOTWeIa2VO7/VlROpRN8gQ0 l8tpWGFk85XzMwivM7YUKjEKfroWMRYHY52atpe9rligJdn6sJrv+Pp2cFr33MqYrF+W in8Vh50Cubz0jSNwXCct8NgUsxLQGhFgEsL2r5YuXBOlfCSRzkubiwuHkbl7pv1I963p /DpQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from gecko.sbs.de (gecko.sbs.de. [194.138.37.40]) by gmr-mx.google.com with ESMTPS id c14si454026edr.4.2021.02.12.02.28.41 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 12 Feb 2021 02:28:41 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.40 as permitted sender) client-ip=194.138.37.40; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by gecko.sbs.de (8.15.2/8.15.2) with ESMTPS id 11CASfqN010406 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Feb 2021 11:28:41 +0100 Received: from md1za8fc.ad001.siemens.net ([167.87.17.8]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id 11CASebW001089; Fri, 12 Feb 2021 11:28:40 +0100 Date: Fri, 12 Feb 2021 11:28:40 +0100 From: Henning Schild To: Baurzhan Ismagulov Cc: isar-users Subject: Re: FYI: Feasibility of CI on github Message-ID: <20210212112840.70690286@md1za8fc.ad001.siemens.net> In-Reply-To: <20210212100121.GE20742@yssyq.m.ilbers.de> References: <20f63dff-b266-24b9-4356-341d99782420@siemens.com> <20210212101647.69e6b7cd@md1za8fc.ad001.siemens.net> <20210212100121.GE20742@yssyq.m.ilbers.de> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: npw9c1BHxxYk Am Fri, 12 Feb 2021 11:01:21 +0100 schrieb Baurzhan Ismagulov : > On Fri, Feb 12, 2021 at 10:16:47AM +0100, Henning Schild wrote: > > that is very good news. And the current CI is a total mess anyways. > > All in one 6h job. Maybe someone will find the time to improve this > > with something like bats or pytest. And rip out that Avocado stuff > > ... does anyone even use this? > > We have the current testsuite in Avocado, will send patches. Last time i checked avocado was not packaged properly and pip is not an option. Plus the patches have been merged without a proper review and we are still waiting for a user ... dead code without review where people already expressed concerns on the choice of the tool > The long run time is the reason why we use our own servers for CI; > currently, we get it built in 4.5 h. This could be improved to some > extent through parallelizing the test cases, but CPU and storage > utilization is about 100%, so that would require several nodes. Also, > looking at the task timing diagram, it seems that dpkg-buildpackage > is completely serialized; maybe we could improve here, too. There is something. Run many tests in parallel instead of all mc in one. The all mc has its place because we have mc, but maybe not the best choice for the default. > We did look around before introducing Avocado. Pytest is fine for > module tests, but Avocado has more infrastructure for black box and > target tests that we need. We haven't looked at bats, but I'd like > stay away from shell in favor of Python. We never talked about it. I just came up with two examples. We should probably find something that offers all we want but is easy to install and maintain. And one really important first step would be decoupling of CI and CT. > What I don't understand is, after we have individual test cases, how > does that help us with the long run time? Would you schedule every > test case in different steps? You get more control if you want to disable some aspects, you get better reporting, you can fail individual tasks without failing "in total". You do not need to "keep going" when something failed, or you can keep going if that is what you want ... more control. I do not even want to imagine how much time we all waste because of that. Show me one person that can read the output and tell a temporary network glitch from a real problem. One "retries" ... maybe a few times, later compare the errors and eventually merge even it the test was red all the time. 12+hours to test a single patch I sometimes trigger your CI on "next" it did not go green in ages. Henning > With kind regards, > Baurzhan. >