From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6927246286546534400 X-Received: by 2002:a05:651c:10d1:: with SMTP id l17mr1313613ljn.205.1613127190565; Fri, 12 Feb 2021 02:53:10 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:5748:: with SMTP id r8ls1649808ljd.8.gmail; Fri, 12 Feb 2021 02:53:09 -0800 (PST) X-Google-Smtp-Source: ABdhPJz6NR5qHWHBj/AN6d2voNOJ3v9pBLRiVs1f6yZyX/KLwpWiFcp+lL+sQDkE5in3mal/hpET X-Received: by 2002:a2e:b81b:: with SMTP id u27mr1362015ljo.365.1613127189465; Fri, 12 Feb 2021 02:53:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1613127189; cv=none; d=google.com; s=arc-20160816; b=m5tnDhCwfve/vtVmYsbBG4u5cXakGR8b70G00j/i+ukKcx1PxGSggAgqVyP7FGFP99 KzJn+2g6bckfME9yfj6O//i3Vfu2Z/uukh9qxRRkvxN7Yn4dImkh5kL616+CL8g57SBT MDZvBHZjAEN+oEdfzkw/5WAdxhj2lyjDvAqVH+1fEw5TGaVpE4TlnsdFQHjwJtuLZ9Z1 XLP34t/qu4RmT7JIerIt+d0kLLbaDXbCDoRIaH7RMHk7yhaoR3yFcg+tSU+M3azzIZvM AOWYfBFdXg2Jkb1WbohwMfSMCa8gJ1T+KEpeQoIkRAnOY9emTcJIarxC9x/JVw3Yj062 NIyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:to:from:date; bh=TSKRmqWnDHKwm8tQ27H9BEPL6igDx8YqYP57TO28ous=; b=zzoAzudrcZaf2kCft036/ewCcnLppowDEe66DWlXraFxEbl0+Po8Yj0eRHjfgYDXij j3Xevv3AETyNutb4F2KoTA5YPZZ694pdrE2Y0WpcXLXlIvOJp/2WRPDisjUN/uGViRk/ KklIaIT+IVYxCa6a+mpn4BgSI+Ams1GZJ/ZNx4ikVSAo33queg/1eoH04CGofCPp0NJ9 MeWBmBM5/KEZwLJKxtAhjv948s0Chq7k0nIn6HOEl2KowwvLoLM91Atc2TiGRVWfnBTH 2l7P4wf6dNm3Km9NxfSwL6EARKWxXPkbTEVom1XdAzasoV6YpGJhpJrDEaWNgXIa0x3M Fr6A== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id j2si359820lfe.5.2021.02.12.02.53.09 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 12 Feb 2021 02:53:09 -0800 (PST) Received-SPF: neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Received: from yssyq.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 11CAr6SI021696 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 12 Feb 2021 11:53:08 +0100 Date: Fri, 12 Feb 2021 11:53:06 +0100 From: Baurzhan Ismagulov To: isar-users Subject: Re: FYI: Feasibility of CI on github Message-ID: <20210212105306.GG20742@yssyq.m.ilbers.de> Mail-Followup-To: isar-users References: <20f63dff-b266-24b9-4356-341d99782420@siemens.com> <20210212101647.69e6b7cd@md1za8fc.ad001.siemens.net> <20210212100121.GE20742@yssyq.m.ilbers.de> <20210212112840.70690286@md1za8fc.ad001.siemens.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210212112840.70690286@md1za8fc.ad001.siemens.net> User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: LkO/ybw4NGsJ On Fri, Feb 12, 2021 at 11:28:40AM +0100, Henning Schild wrote: > Last time i checked avocado was not packaged properly When we made our initial PoC with Avocado, we used their Debian packaging. Unfortunately, it got broken with buster. I'll talk with upstream regarding this. > Plus the patches have been merged without a proper review Your feedback happened to come after the merge; it has been addressed in our patches to come. If you have further feedback, please share with us. > There is something. Run many tests in parallel instead of all mc in > one. The all mc has its place because we have mc, but maybe not the > best choice for the default. Ok, that is a viable option. We'd continue to test mc anyway, but maybe in a separate, non-default job. I think we would still need to look at the problems that we encountered with mc and define some minimal mc test case for the default scenario. > And one really important first step would be decoupling of CI and CT. If CT = continuous testing, I've indeed handled that as one step in our process; what could we decouple with which benefit? > > What I don't understand is, after we have individual test cases, how > > does that help us with the long run time? Would you schedule every > > test case in different steps? > > You get more control if you want to disable some aspects, you get > better reporting, you can fail individual tasks without failing "in > total". You do not need to "keep going" when something failed, or you > can keep going if that is what you want ... more control. Those are arguments for a test framework in general -- I agree, this is why we've introduced Avocado in the first place. If I understood Jan correctly, his point was to make the run time shorter -- this could rather be solved by a shorter default testsuite. We would then run the full testsuite on better hardware before merging. That said, we already have the shorter variant; maybe we should switch it to be the default? With kind regards, Baurzhan.