From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6927246286546534400 X-Received: by 2002:a05:651c:1127:: with SMTP id e7mr2586073ljo.5.1613158555469; Fri, 12 Feb 2021 11:35:55 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:90d2:: with SMTP id o18ls1975843ljg.1.gmail; Fri, 12 Feb 2021 11:35:54 -0800 (PST) X-Google-Smtp-Source: ABdhPJypqF9OMzEfgcC/9uazy2RM5KvjYXzbYMxZhSKwSEabYzD3SlmnAACSdBrWV84iydXODd8z X-Received: by 2002:a2e:85c3:: with SMTP id h3mr2473467ljj.431.1613158554438; Fri, 12 Feb 2021 11:35:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1613158554; cv=none; d=google.com; s=arc-20160816; b=WmlbcPHJdZKHPKnUT1UT6EJKsdHPYNc/FuJFgBGzailEsEUHxPlaoGB5f8v6+e2nNa Tq2xVnrR9/f24vu08LYg41waQYB9LxMWs7chdjLdreKwxSJdeXJ3Ed3uAMFsyP5s2SrS 9EXZ1Ak8a3UKapFLKTw+OJLkjAFLbSw5k1n7tNDDfYb0R+FL81WB0i1HRAit0z/1qH+8 QCq/ZDnVirG81VeiZF7DLqvpuJ4e6vKm84YC7/Z92GpbS6apwc7vW7uL71SbF/JVVKlG lI72daiaBJW2f30P22SKgQ8rYNCcXX97ViaomApr9q+ImB63PdJXcrVLvWoLzS3QEjKD O+5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date; bh=w4URoAaBfLhGSb58DcGan1bqBJBb1auMjyBPOWFn3l4=; b=glOkVyTStRecMOpEpr6MYo7v0PEXaZU/1fkH9nZJfCsWl/z8uj+LfajdIG6QArSkc4 xFg8a3Br5m779Rv9XjpwynRLQub8dnlDNy45YhXXe/hqmj8PRo3rjGdewFeOXKdI/jpu bk04fC4UzA3Kiczpqt1Nl2zoGVKO6tWUeJ9Iv8Wee5J5aw2OcD7UdFDHVEzZC6dTW3VG sZ3BB6eL8pLHrFWx4i3xq9AG15ZAIxTqY4C2GxtJxxZhMvN1PZXFlM5IMxJskXb25awJ 7u3KY0S9rBJpoAi8UYKwck/WtSnI1gSsl4mkY52cNImcGJmHbO5vkxvbwUjFXgaWR3cp 0hFQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id m17si310477lfg.0.2021.02.12.11.35.54 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 12 Feb 2021 11:35:54 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id 11CJZr2g020723 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Feb 2021 20:35:53 +0100 Received: from md1za8fc.ad001.siemens.net ([167.87.17.8]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id 11CJKqXZ013589; Fri, 12 Feb 2021 20:20:53 +0100 Date: Fri, 12 Feb 2021 20:20:51 +0100 From: Henning Schild To: Baurzhan Ismagulov Cc: isar-users Subject: Re: FYI: Feasibility of CI on github Message-ID: <20210212202051.3446d863@md1za8fc.ad001.siemens.net> In-Reply-To: <20210212105306.GG20742@yssyq.m.ilbers.de> References: <20f63dff-b266-24b9-4356-341d99782420@siemens.com> <20210212101647.69e6b7cd@md1za8fc.ad001.siemens.net> <20210212100121.GE20742@yssyq.m.ilbers.de> <20210212112840.70690286@md1za8fc.ad001.siemens.net> <20210212105306.GG20742@yssyq.m.ilbers.de> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: tmMSpEFdyBrh Am Fri, 12 Feb 2021 11:53:06 +0100 schrieb Baurzhan Ismagulov : > On Fri, Feb 12, 2021 at 11:28:40AM +0100, Henning Schild wrote: > > Last time i checked avocado was not packaged properly > > When we made our initial PoC with Avocado, we used their Debian > packaging. Unfortunately, it got broken with buster. I'll talk with > upstream regarding this. That would be good. I think having debian would be good enough but not having it is really bad. > > Plus the patches have been merged without a proper review > > Your feedback happened to come after the merge; it has been addressed > in our patches to come. If you have further feedback, please share > with us. I do not remember the details. It is quite possible. But a feature merge without a user is something that should never have happened. Like an RFC merge should never happen, but not the point of the thread. We are already too far away from github workflows. > > > There is something. Run many tests in parallel instead of all mc in > > one. The all mc has its place because we have mc, but maybe not the > > best choice for the default. > > Ok, that is a viable option. We'd continue to test mc anyway, but > maybe in a separate, non-default job. I think we would still need to > look at the problems that we encountered with mc and define some > minimal mc test case for the default scenario. > > > > And one really important first step would be decoupling of CI and > > CT. > > If CT = continuous testing, I've indeed handled that as one step in > our process; what could we decouple with which benefit? Yes i mean continous testing. Starting those qemus and parsing their output is clearly testing and should be another job. All the qemu-related variables need to leave the machine configs. These configs serve as a very bad example where people might think they can configure the kernel cmdline in the machine cfg, even though they might be on wic. > > > What I don't understand is, after we have individual test cases, > > > how does that help us with the long run time? Would you schedule > > > every test case in different steps? > > > > You get more control if you want to disable some aspects, you get > > better reporting, you can fail individual tasks without failing "in > > total". You do not need to "keep going" when something failed, or > > you can keep going if that is what you want ... more control. > > Those are arguments for a test framework in general -- I agree, this > is why we've introduced Avocado in the first place. If I understood > Jan correctly, his point was to make the run time shorter -- this > could rather be solved by a shorter default testsuite. We would then > run the full testsuite on better hardware before merging. That said, > we already have the shorter variant; maybe we should switch it to be > the default? Ideally the test CI and CT framwork of Isar would allow layering. While main Isar would be mc and slow on its thorough CI/CT layers could reuse parts. I think a proper modularization will save time even given the current CI. A green build might still take the full time. But a red build will be faster and tell you which modules failed. Which will also allow you to write patches for multiple problems for one red run ... better reporting will result in better understanding and less builds. Which saves build count and a lot of time on that end, even if the time of such a build remains "high". And pipelines with multiple jobs allow to hit retry buttons for just the failed job. I think you can see this https://code.siemens.com/linux/las/-/pipelines/8264154 To get there we would need a CI system where the pipelines are part of the code, so probably not jenkins. Henning > > With kind regards, > Baurzhan. >