From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6927246286546534400 X-Received: by 2002:a1c:4ca:: with SMTP id 193mr3550785wme.178.1613149224983; Fri, 12 Feb 2021 09:00:24 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:640a:: with SMTP id z10ls805113wru.0.gmail; Fri, 12 Feb 2021 09:00:24 -0800 (PST) X-Google-Smtp-Source: ABdhPJxVTkIkKt+b+UcTLPb6dsrpf0XRyTrlA3fBqQXMG1m6BKBBiP8mE3FPHrtmVA+MErNEd5vc X-Received: by 2002:a05:6000:1141:: with SMTP id d1mr4497039wrx.47.1613149223988; Fri, 12 Feb 2021 09:00:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1613149223; cv=none; d=google.com; s=arc-20160816; b=hBxNnKyF0Spfd3gDB6+3LUI1TGWabg8cQBm/c8bD0wG3VihJ+dVbf/KLG9SKu4GXjQ HeMAvUdn/KE0BodORDr90/bqhA3uAgYV6IJQjmHLGxCSHEBxM7WYRXDjFDfWxTEFgY4q +uMocZ5450IObxrKwpPIimqgjXbQ9DgZxUug+I42dufc9/34NlIqBuPbTBSCAOte69Cr XWH2prRXOLcF7xP3W7Fm2cpIvQqpzAjxsTk+9otI9TlhlCVBsDbCbIXRSOliGmFGN3wF eBd5/h8LocB40RS2HKdhAqtvtunU5R75ww/LH4RpkWidIp366xg1U9aP9z6pXCJZCUY2 xR5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject; bh=rApqTTNdkVy6Ux85TkymUGi6fxghZ6OKtXD28PDRGgk=; b=Bd8DfsmxkEo+NaTw65n1Gq1Ul93ojhvzi/hLnLAwAedCE2IZSS6jng2qZ1DTPEqW+P IYS9jyt8GlZzNoewuT/saH60D+AUST39qXpoqWz6ps7pCmBRpUZVjeZCdBvXep5EnHBT oscpGNPfHRSHleF5ggGFo+49K7s+RScDGW+Dw/l8qSL9znz7Iun0lsskcSHwqxRe/nBF maZ+8NKOzJhLWd/vPiR88LPEp6jKv50HzcjyOI8LxenrYCmFs2ZDZVs2AmdjhAEPqvNf GB4eFthmqRjIzoFOtlxg9n7dy8LmCCkYpvtgI8g0RFxXI9PVOXGazP7Cy7IbaAT+vEfX rYng== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from gecko.sbs.de (gecko.sbs.de. [194.138.37.40]) by gmr-mx.google.com with ESMTPS id s3si354438wrt.5.2021.02.12.09.00.23 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 12 Feb 2021 09:00:23 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) client-ip=194.138.37.40; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by gecko.sbs.de (8.15.2/8.15.2) with ESMTPS id 11CH0N9u012455 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Fri, 12 Feb 2021 18:00:23 +0100 Received: from [167.87.240.24] ([167.87.240.24]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 11CGtNj5017592 for ; Fri, 12 Feb 2021 17:55:23 +0100 Subject: Re: FYI: Feasibility of CI on github To: isar-users References: <20f63dff-b266-24b9-4356-341d99782420@siemens.com> <20210212101647.69e6b7cd@md1za8fc.ad001.siemens.net> <20210212100121.GE20742@yssyq.m.ilbers.de> <20210212112840.70690286@md1za8fc.ad001.siemens.net> <20210212105306.GG20742@yssyq.m.ilbers.de> From: Jan Kiszka Message-ID: Date: Fri, 12 Feb 2021 17:55:23 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.0 MIME-Version: 1.0 In-Reply-To: <20210212105306.GG20742@yssyq.m.ilbers.de> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: 7ZwZYcUO5PEf On 12.02.21 11:53, Baurzhan Ismagulov wrote: > On Fri, Feb 12, 2021 at 11:28:40AM +0100, Henning Schild wrote: >> Last time i checked avocado was not packaged properly > > When we made our initial PoC with Avocado, we used their Debian packaging. > Unfortunately, it got broken with buster. I'll talk with upstream regarding > this. > > >> Plus the patches have been merged without a proper review > > Your feedback happened to come after the merge; it has been addressed in our > patches to come. If you have further feedback, please share with us. > > >> There is something. Run many tests in parallel instead of all mc in >> one. The all mc has its place because we have mc, but maybe not the >> best choice for the default. > > Ok, that is a viable option. We'd continue to test mc anyway, but maybe in a > separate, non-default job. I think we would still need to look at the problems > that we encountered with mc and define some minimal mc test case for the > default scenario. > > >> And one really important first step would be decoupling of CI and CT. > > If CT = continuous testing, I've indeed handled that as one step in our > process; what could we decouple with which benefit? > > >>> What I don't understand is, after we have individual test cases, how >>> does that help us with the long run time? Would you schedule every >>> test case in different steps? >> >> You get more control if you want to disable some aspects, you get >> better reporting, you can fail individual tasks without failing "in >> total". You do not need to "keep going" when something failed, or you >> can keep going if that is what you want ... more control. > > Those are arguments for a test framework in general -- I agree, this is why > we've introduced Avocado in the first place. If I understood Jan correctly, his > point was to make the run time shorter -- this could rather be solved by a > shorter default testsuite. We would then run the full testsuite on better > hardware before merging. That said, we already have the shorter variant; maybe > we should switch it to be the default? > Time is not our primary issue (you can easily run the sequential parts of the CI script in separate jobs and gain quite a bit). It's the disk size. But when we want to stress the full mc: pool, there is likely no way around dedicated runners for the time being. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux