From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7024137565936549888 X-Received: by 2002:a17:906:1706:: with SMTP id c6mr44106899eje.343.1635839720751; Tue, 02 Nov 2021 00:55:20 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:907:75f1:: with SMTP id jz17ls967848ejc.2.gmail; Tue, 02 Nov 2021 00:55:19 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzVLOPjxXr1/+IojIDfDcp5nZCfQni8SNnp6D5njGEjl8O8Nx14Q5nn1za5IiUwUf98z15b X-Received: by 2002:a17:907:6297:: with SMTP id nd23mr44458163ejc.62.1635839719732; Tue, 02 Nov 2021 00:55:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1635839719; cv=none; d=google.com; s=arc-20160816; b=gKbop9IGzu7oEThkDDhDbNFWWOBDn24apBjtU4MX9FtM/Ta0cF81d0yNZ5G4uZTX4C 1LkE6EW7/gxdX34DZjmDuAArkmjjtNJwaqeanvNtiRQld5FTOf/9S4WSj9298OKFaPow ExC5MtkxZWOgSnasy7RpWb5w6Dv/Lp/eMLzl8PQ/vQ41G4zcefUlGRB05ZU88M37BM+f GkH6W4scfRlBYlZCojpLDp8qN9Q0cwUUptWM6dNGlLWwRGCUme3+5zyBREoLm9AY018y sFyu8nSRUPdaDTmPzNrJeQ9KxjWrihKEijTC76QVdzgiGEZHggXIzc0TA7LsGKDnIBll zegg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject; bh=kXmgrBtE9i/3Jix7jbqUjEI7Xizltgmr0JqBSqnN9+4=; b=kmWijrybZbHF7WoIkF0tHgcpl9Qak3DlaOqshCnStntyd2P9S5kq//d+qGrhgzijMA bzTbJKZYMZSNiuoI3CWuOG0amwNG69lqvzyk9ArSy6uNihgmGt17GfmjJ80laQpLXoUP gF3NKfjiP0nms05woNexoY0t5ZQSb2tLjMSTcBDYqVGyZjadlr9lWyQBZ1GfGg29lH8E HPstD3wiOL/hFdwNu7zLl8bNNHKo6/hF66mLrVmf01oEerFmCQBjjvZBf5uE/OQ8K8R5 k9fGRwmqE0tD+KYXWVcG9IOQzENazSVxHWKuQ+WZtrQbAn0zZuD9hoG5WOdi+7EWqwWj mLcA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from gecko.sbs.de (gecko.sbs.de. [194.138.37.40]) by gmr-mx.google.com with ESMTPS id 14si768541edv.4.2021.11.02.00.55.19 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Nov 2021 00:55:19 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) client-ip=194.138.37.40; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by gecko.sbs.de (8.15.2/8.15.2) with ESMTPS id 1A27tJek002223 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Nov 2021 08:55:19 +0100 Received: from [167.87.72.48] ([167.87.72.48]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 1A27tI6m031555; Tue, 2 Nov 2021 08:55:18 +0100 Subject: Re: [PATCH] CI: Add sstate-cache testcase To: "Schmidt, Adriaan (T RDA IOT SES-DE)" , "Schild, Henning (T RDA IOT SES-DE)" , Anton Mikanovich Cc: "isar-users@googlegroups.com" References: <20211028152056.244471-1-amikan@ilbers.de> <1822c3ac-88d1-32cc-8886-c83d4746fe3a@ilbers.de> <20211028202157.1ef109e5@md1za8fc.ad001.siemens.net> From: Jan Kiszka Message-ID: <075effef-5258-346a-51cf-ad6ddcb68f91@siemens.com> Date: Tue, 2 Nov 2021 08:55:18 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: Lzmsxk0MBu20 On 02.11.21 06:57, Schmidt, Adriaan (T RDA IOT SES-DE) wrote: > Henning Schild, Donnerstag, 28. Oktober 2021 20:22: >> Am Thu, 28 Oct 2021 20:22:58 +0300 >> schrieb Anton Mikanovich : >> >>> 28.10.2021 18:34, Jan Kiszka wrote: >>>> On 28.10.21 17:20, Anton Mikanovich wrote: >>>>> Test rebuild time improve after cleanup to be sure sstate-cache >>>>> works. >>>>> >>>>> Signed-off-by: Anton Mikanovich >>>>> --- >>>>> testsuite/build_test/build_test.py | 40 >>>>> ++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2 >>>>> deletions(-) >>>>> >>>>> diff --git a/testsuite/build_test/build_test.py >>>>> b/testsuite/build_test/build_test.py index d39c10c0..244f6fc0 >>>>> 100644 --- a/testsuite/build_test/build_test.py >>>>> +++ b/testsuite/build_test/build_test.py >>>>> @@ -1,9 +1,9 @@ >>>>> #!/usr/bin/env python3 >>>>> >>>>> -import os >>>>> +import os, time >>>>> >>>>> from avocado import skipUnless >>>>> -from avocado.utils import path >>>>> +from avocado.utils import path, process >>>>> from cibase import CIBaseTest >>>>> >>>>> UMOCI_AVAILABLE = True >>>>> @@ -206,3 +206,39 @@ class ContainerSdkTest(CIBaseTest): >>>>> targets = ['mc:container-amd64-stretch:isar-image-base'] >>>>> >>>>> self.perform_container_test(targets, 'do_populate_sdk') >>>>> + >>>>> +class SstateTest(CIBaseTest): >>>>> + >>>>> + """ >>>>> + Test rebuild speed improve with sstate-cache >>>>> + >>>>> + :avocado: tags=sstate >>>>> + """ >>>>> + def test_sstate_rebuild(self): >>>>> + speedup_k = 2.0 >>>>> + >>>>> + targets = [ >>>>> + 'mc:qemuamd64-bullseye:isar-image-base' >>>>> + ] >>>>> + >>>>> + # Cleanup everything before build >>>>> + build_dir = self.params.get('build_dir', >>>>> + default=os.path.dirname(__file__) + >>>>> '/../../build') >>>>> + process.run('rm -rf ' + build_dir + '/sstate-cache', >>>>> sudo=True) >>>>> + self.deletetmp(build_dir) >>>>> + >>>>> + start = time.time() >>>>> + self.perform_build_test(targets, 1, None) >>>>> + first_time = time.time() - start >>>>> + self.log.info('Non-cached build:' + str(round(first_time, >>>>> 2))) + >>>>> + # Cleanup everything but cache files >>>>> + self.deletetmp(build_dir) >>>>> + >>>>> + start = time.time() >>>>> + self.perform_build_test(targets, 1, None) >>>>> + second_time = time.time() - start >>>>> + self.log.info('Cached build:' + str(round(second_time, >>>>> 2))) + >>>>> + if first_time / second_time < speedup_k: >>>>> + self.fail('No speedup after rebuild with >>>>> sstate-cache') >>>> Is there no better way than measuring time to test if sstate was in >>>> place? Maybe some hit/miss statistics? >>>> >>>> Jan >>>> >>> I've tried to find some Avocado API to access final testcase >>> statistics from within other testcase but didn't find so far. >>> Any advice is welcome. >> >> sstate hit rates are in the bitbake logs, i hope avocado can open >> artifacts > > You want to look for the line "Sstate summary", which is shown at the beginning of the bitbake run, e.g.: > > Sstate summary: Wanted 121 Local 121 Network 0 Missed 0 Current 0 (100% match, 0% complete) > > But in a test it's probably better to examine log.task_order and check which tasks were executed. > >> But this is infra ... we will see when it fails and do not have to test >> that. > > It's worth running some tests to see that builds succeed with artifacts taken from cache. I can imagine something like: > > - Build image to populate cache > - Remove TMPDIR > - Rebuild the image. Should have 100% cache hits. (This would take the target rootfs from cache, and the buildchroot required for image-building) > - Remove TMPDIR > - Build a single package (that was already contained in the image). Should have a cache hit, and only dpkg_build_setscene and deploy_deb should run (plus minimal steps to initialize isar-apt). > > And optionally: > - Remove TMPDIR > - Force rebuild of one package (e.g. by deleting its sstate artifact) > - Rebuild the image (should take the buildchroot and all but the one package from cache, and rebuild the one package and the image) > BTW, we likely now also need a cleansstate target, like OE has. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux