From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6990633379674718208 X-Received: by 2002:a17:907:98a5:: with SMTP id ju5mr6122670ejc.378.1631801497588; Thu, 16 Sep 2021 07:11:37 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6402:254d:: with SMTP id l13ls3797907edb.3.gmail; Thu, 16 Sep 2021 07:11:36 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxKBiR7lAAs0+pLNpFjhqfOdbeRcNms4OYKn+u/z45SBbN27F6exHda+f0eFdh2rHfMGOdM X-Received: by 2002:a05:6402:3186:: with SMTP id di6mr6543434edb.225.1631801496624; Thu, 16 Sep 2021 07:11:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1631801496; cv=none; d=google.com; s=arc-20160816; b=L6SKFvVXDDbr1xxMhm+7/uZoZOq0/jRZNwBa6RYUE0Sw30fg8G0gYdxZ0PIdNZxb8L SR6sPF5iJm4j3uaTUCefqXLA2Q9gjpyUa2KaJPxuLqIWIG/P5obKCw0xrqWzOeuzieVU KvnjfutL3JwhaNB2MStA5zBnTqDXNQedAWhqgjw5TOfAOYd3FboxcwNwtuAMz8Td0DS6 WOOVSmey4Onb1TldFvyFnQvDep6WYkpSe1DRQotfkNIhGn/pWVdFH8WIw6PuSs7WEwBz IemVESkPC73y/0hTqdDkMDzQGO/vXrbVbDl45PwjKZL/qpzSjRKrjnF/oGPlPFRF9aYN p7vg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:to:from:date; bh=BfRtgwSjZtbU6JYwVGrvKsZamrSkXZZrwryAPFbmU5Y=; b=YIvbgOkoa8Asb6i7PK/d+iKhtqwoWB/NQgG4PjuRUhrScexLaaPYva+R8ey6lcWjrj SpyYrumNAKHktCulAcl2fJRv/8ExaW5ojoAhVsFNpah8htw4XjtqbEBFyoLwJM7kiuvf +ybJ5EegjFGQ508nKVN00tNd5Zc3IEntbTA6BAwDfmg4zJ3Bv22MMkIpiN+lx+10KXvK no+VmS1d1VL4wByKnKFruFX1LnKFtT9WBqc8ansgZjPDA1OzFK09eKD380sHNygakaUR Pg5Nlp0rZXBOEiRR4J43i7OxfO93vuhPwNdNCGqOjFR2BbbsJkR9eMvRSNXAxukVjAsm SS0w== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id n10si296930eje.0.2021.09.16.07.11.36 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 16 Sep 2021 07:11:36 -0700 (PDT) Received-SPF: neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Received: from yssyq.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 18GEBWwI022874 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 16 Sep 2021 16:11:34 +0200 Date: Thu, 16 Sep 2021 16:11:32 +0200 From: Baurzhan Ismagulov To: isar-users@googlegroups.com Subject: Re: [PATCH v10 17/17] ci_build: Install Avocado if needed Message-ID: <20210916141132.GO8902@yssyq.m.ilbers.de> Mail-Followup-To: isar-users@googlegroups.com References: <20210730082739.15042-1-amikan@ilbers.de> <20210730082739.15042-18-amikan@ilbers.de> <20210812175858.4264d095@md1za8fc.ad001.siemens.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210812175858.4264d095@md1za8fc.ad001.siemens.net> User-Agent: Mutt/1.10.1 (2018-07-13) X-TUID: cdUDFY519J+8 On Thu, Aug 12, 2021 at 05:58:58PM +0200, Henning Schild wrote: > > diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh ... > > +# Install Avocado if needed ... > I do not like this. ... > apt must not be used in scripts ... and it tells you about it. That was only intended to help testing the series, dropped in v11. The execution environment should provide the tools, with Debian packages or pip. > The package should be mainlined into debian We've filed an ITP: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993862 > And i want more than "buster". Provided stretch, buster, bullseye, bookworm at http://deb.isar-build.org/debian-isar. The bookworm one should work for sid, but we haven't tested it. > I started looking into the avocado tests we currently have. Seems that they > are still pretty different from what we used to have. Will this new version > implement all the tests we used to have? ... > I was only playing with what we currently have in next. But that seems to > serialize the builds listed in that yaml. While before bitbake did run them > all at once. The initial Avocado implementation (in next today) provided only a matrix for building all of individual multiconfigs, one per testcase. Yes, v11 intends to provide exactly the same testcases that we have in ci_build.sh today. We could add the individual single multiconfigs later if needed. > My solution was to install that very unlucky choice of software with > virtualenv I assume this is about Avocado -- perhaps you could share some specific issues you'd like to have addressed? We did spend some time to evaluate our alternatives before choosing a specific tool. Avocado is suitable for running Python and shell testcases, is not limited to module testing, and is written in Python. It is a part of an ecosystem supporting use cases like virtual machines, parallel execution, easy testsuite composing via testcase tags, etc. It is the successor of Autotest and has wide and very active community. We've used it in other projects before Isar and got good impression. Having a test framework is waaay better than maintaining ci_build.sh -- in the same way Isar is better than build.sh ;) . Of course, we are open to suggestions and improvements. One thing we have on our radar is artifact reusing and testcase dependencies. E.g., with parallel testcase execution, a start_vm testcase could depend on a bitbake one. > I think the tests from that yaml should run in parallel to cause the same > kind of stress ... just parallel in the testsuite instead of bitbake. And to > hopefully speed things up a bit. For the migration step, we aim at providing exactly the same testsuite as in ci_build.sh and rework in later steps -- we'll share an RFC for that. Regarding parallel execution, I think we need both parallel and bitbake with several multiconfigs. This tradeoff isn't only about performance and stress; the latter use case is necessary for per-build stuff (build numbers, release management, etc.) and could potentially catch issues like missing dependencies, locking, etc. that would be missed with the parallel execution of single multiconfig builds. A note on yaml -- that approach was very rigid, it supported only combinations of single multiconfigs per testcase (we need more as described above) and executed all of them (we need less to minimize the execution time). With kind regards, Baurzhan.