From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6927246286546534400 X-Received: by 2002:a5d:6b47:: with SMTP id x7mr18570978wrw.170.1613388452308; Mon, 15 Feb 2021 03:27:32 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:640a:: with SMTP id z10ls573460wru.0.gmail; Mon, 15 Feb 2021 03:27:31 -0800 (PST) X-Google-Smtp-Source: ABdhPJx3SLdFzy4SnKhgjtExFvqv99px8SAKU70o6leIcbfhzaPDU5dGF5gFHcSQzZ4BNuJIaAKz X-Received: by 2002:adf:f3c4:: with SMTP id g4mr18408285wrp.61.1613388451512; Mon, 15 Feb 2021 03:27:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1613388451; cv=none; d=google.com; s=arc-20160816; b=p0+LxELsihooqU7VLH+lUd8tP34LcxzaL0mw7MlqW1QgdTscB/KCKSso+1nOTCdvkm v/aZjNXLQ3wMF7U/l+lvlDeNo1YoMgR094ThJao6bBSEjiBJb8Xvm1iSv5zocnbuiv1j Lj6ZeMvxX946dNi92lwwfYImZlZ3ucdOED+1kv45iQ77pyke+2at8VwofaKWGfMZ39KS Feb2nm0sxxhJKcarWUyAywVwfp5nvhv43Ynn/UMIr+LjJldv41OTl4Fhc0iv9VLsqkiq LNlzhBFJAmoRPVYLKX+dCfwkn8HcxigsD7sIPd5B0SypyrTul3MtwIPoF61886bne0qy C9pw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:to:from:date; bh=xoaoA4/3wEvL/X8A60R5ZPWAmXt+Ynx56tTjbBLocMg=; b=rhn0Z5PVxg8xHjKwZDWbw5MW58QQuaEgttCjDAEVM+xpAVale5WOSY1jGAjQMZp0+n XC1tA8+APoXDBEIUqAL37Agb4Qvy8WTvOJaOG1WhdHe8eDzdykGvYZBGpmev12sQtAzA ccnT8eetw3Mf9Hkc/P0f/s/IrUdaOz1exDQeKtjneei7+dtSU93VYoKeqcvxW1duPAfx qKsgpEY3Qez26Gju9gUJX9zRU7OTs17LSbPL6XPwzXe9wJPfbmgURLkgJe8B4TvUZKPu qGU+0WPvKHlA+NT66kmxd6i1lcqpeWGOrvTtJ9fzTKjnWatYbbM2Mqe32q0o00HuRj7V 3egQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id z16si749136wml.1.2021.02.15.03.27.31 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 15 Feb 2021 03:27:31 -0800 (PST) Received-SPF: neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Received: from yssyq.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 11FBRSDc004169 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 15 Feb 2021 12:27:30 +0100 Date: Mon, 15 Feb 2021 12:27:28 +0100 From: Baurzhan Ismagulov To: isar-users Subject: Re: FYI: Feasibility of CI on github Message-ID: <20210215112728.GO20742@yssyq.m.ilbers.de> Mail-Followup-To: isar-users References: <20f63dff-b266-24b9-4356-341d99782420@siemens.com> <20210212101647.69e6b7cd@md1za8fc.ad001.siemens.net> <20210212100121.GE20742@yssyq.m.ilbers.de> <20210212112840.70690286@md1za8fc.ad001.siemens.net> <20210212105306.GG20742@yssyq.m.ilbers.de> <20210212202051.3446d863@md1za8fc.ad001.siemens.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210212202051.3446d863@md1za8fc.ad001.siemens.net> User-Agent: Mutt/1.10.1 (2018-07-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: MJerUIoOF8mV On Fri, Feb 12, 2021 at 08:20:51PM +0100, Henning Schild wrote: > > When we made our initial PoC with Avocado, we used their Debian > > packaging. Unfortunately, it got broken with buster. I'll talk with > > upstream regarding this. > > That would be good. I think having debian would be good enough but not > having it is really bad. I've looked at the code, they've meanwhile updated it. I've submitted small fixes, but we have the package and will test it. > I do not remember the details. It is quite possible. But a feature > merge without a user is something that should never have happened. The code that is not used by default doesn't necessarily mean it is not used. It's exactly because we wanted to experiment with different test case splits that we didn't enable it by default too early. In the first step, we'd like to mimic the existing test suite, then maybe add test cases for individual machine builds, then switch the default. That said, this topic is something we are often confronted with: We get an obviously useful patch (e.g., a fix for a build failure in a certain scenario), but don't have a test case for it. By this definition, it is code without a user. As much as I'd like to have test cases for everything, I doubt we can live up to that 100%. > > > And one really important first step would be decoupling of CI and > > > CT. > > > > If CT = continuous testing, I've indeed handled that as one step in > > our process; what could we decouple with which benefit? > > Yes i mean continous testing. > > Starting those qemus and parsing their output is clearly testing and > should be another job. Ok, roughly, CI is building and CT is running. Yes, starting the images will go to their own test cases. That said, at Isar CI we'd still like to include them in the process. But downstreams can decide that themselves -- IIUC, it isn't used at the Siemens CI today. > All the qemu-related variables need to leave the machine configs. > > These configs serve as a very bad example where people might think they > can configure the kernel cmdline in the machine cfg, even though they > might be on wic. If the docs can be improved to avoid misunderstandings, I'd start with that first. The qemu vars are already prefixed. Scattered settings is one of the main complaints about bitbake, so I'm not yet convinced moving is justified in this case. > Ideally the test CI and CT framwork of Isar would allow layering. While > main Isar would be mc and slow on its thorough CI/CT layers could reuse > parts. Yes, that requires dependency support, we'll continue looking how to address that. An alternative would be bitbake, which we didn't do :) . > I think a proper modularization will save time even given the current > CI. As mentioned, we already agree that a framework would be useful. > To get there we would need a CI system where the pipelines are part of > the code, so probably not jenkins. Just as a remark, Jenkins does support pipelines, we just don't use them; maybe I should prioritize that. We did resurrect our GitLab, and I don't see how it is better. Ideally, I want a different tool :) . In practice, many people use one of the two, so I'd be interested in documenting both workflows. With kind regards, Baurzhan.