From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7095678487044816896 X-Received: by 2002:a19:4303:0:b0:473:f5fb:27b2 with SMTP id q3-20020a194303000000b00473f5fb27b2mr19948761lfa.626.1652871738195; Wed, 18 May 2022 04:02:18 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:bc0e:0:b0:24f:4ed5:39e4 with SMTP id b14-20020a2ebc0e000000b0024f4ed539e4ls3542115ljf.8.gmail; Wed, 18 May 2022 04:02:17 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwZjH8dBCPjQaKP6A2vNHOeOuIeuYFJupA8de/yeva8auHenjvZjy/RDRAqa6QyZo1Sn24b X-Received: by 2002:a2e:bd83:0:b0:24e:fe7b:7235 with SMTP id o3-20020a2ebd83000000b0024efe7b7235mr16997105ljq.409.1652871737043; Wed, 18 May 2022 04:02:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1652871737; cv=none; d=google.com; s=arc-20160816; b=CS8mUsYXpqkwp3Qlpvfti8EIZrXwaPPTGS1BTnKqlGBnF6un50QWNrH8YutFHf6awL 9xgL4viEJo1m6XdGrnBrlz4hnMGFyyptAxENZcBJUWVxDCrsFzsrL1tR/Rymp/kJ3CUx ddlbaUpdB6tyFo+jJkZ2+vbCQdwQ4FRXiglwglN2Ien/B9slJlF9Dsiy0crHcgoiiGGj 4241e7iCKhf4IFzh1Mh5FsQ80ck9LwiuSvVVhWHaKpJqvlr0rnXqog+9D/g9uBIH2CNK LOEhaVGZkPBuSP0u/m5dHP1LsQjpiZxCSqSY29pBTx7nTBGHCUjK1PJMB09CFiFw+gZ+ eU1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id; bh=rfZwZfUHaVfDbk5DxbyibHR4bhLYEZ9BKmJIde9HIm8=; b=j80wwU8hjo1Qc4Meym2xJLRJKSSmwIxAvgOATF3H9J+wonbYabQyoJWue53DHkO4pM Fw1rH8+3OM0b+oy2YPkjBbvKKHjcHdHmBAlMncix3gmzN8dCDIj2vM7Rpoa938r9TBac 8hRj32PzQZr0FZQf/mISqeqZcz9bBLu4smWIVnDAACUcKqxB5GRZeASPyVKwAFGKwpb4 8ZZ3yg4MqtnQ1ywW373GstGRHsmVSYH5iD/jvZeP8puYK/w+wOd+h/KTSaWM0BQY0JW4 vqNvQMozg4xI4LYqbXrOPrk8VJ7IOdO9GHGhScMRXbje7uSUe27dXun7U4TOSXrwycgt knRg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id be31-20020a05651c171f00b0024eee872899si101564ljb.0.2022.05.18.04.02.16 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 18 May 2022 04:02:16 -0700 (PDT) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from [127.0.0.1] (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 24IB29HM001484 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 18 May 2022 13:02:15 +0200 Message-ID: <382ebe88-4215-3fda-feca-68832d1f52dd@ilbers.de> Date: Wed, 18 May 2022 14:02:09 +0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.5.0 Subject: Re: [PATCH v2 0/7] Sstate maintenance script Content-Language: en-US To: Adriaan Schmidt , isar-users@googlegroups.com References: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> From: Anton Mikanovich In-Reply-To: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: 6d0hD4Ibr5tZ 09.05.2022 13:15, Adriaan Schmidt wrote: > We have been running CI with shared sstate caches for some months now, in > several downstream projects. This is the cache maintenance script that has > evolved during that time. Detailed documentation is in the script itself. > Main features: > - upload cache artifacts to shared caches on filesystem, http, or s3 > - clean old artifacts from shared caches > - analyze in detail why cache misses happen (what has changed in the signatures) > - check the sstate signatures for absolute paths pointing to the build host > > The last two are especially interesting, and have already yielded some > improvements to the cacheability of Isar, some already merged, and some > more in "[PATCH 0/7] Further improve cachability of ISAR". > > p1 handles another absolute path in a variable (LAYERDIR_isar). > p2..3 are minor patches to bitbake (both already upstream) that greatly > improve accuracy and performance of the sstate analysis. > p4 refactors handling of the apt_* tasks. This was motivated by the sstate > analysis, but I think it also makes the code cleaner. > p5..6 add the sstate maintenance script (2 authors, hence 2 patches). > p7 includes a signature check into the sstate test case. This requires the > changes from "[PATCH 0/7] Further improve cachability of ISAR" for the > test to pass. > > One issue: testing! > This is not easy, because it involves infrastructure, and artificial tests > that provide decent coverage would be quite complex to design. > > If we declare that we sufficiently trust the sstate code, we could add a > shared/persistent cache to the Isar CI infrastructure. This would further test > the sstate feature and all steps involved in maintaining such a setup. > In addition, it would significantly speed up CI builds. > > changes since v1: > - generally improved script > - analysis and cachability improvements in bitbake, dpkg-base, and meta-isar > - added "sstate linting" to the testsuite > > Adriaan Schmidt (6): > meta-isar: improve cachability > bitbake-diffsigs: make finding of changed signatures more robust > bitbake-diffsigs: break on first dependent task difference > dpkg-base: refactor dependencies of apt_* tasks > scripts: add isar-sstate > testsuite: add cachability analysis to sstate test > > Felix Moessbauer (1): > isar-sstate: add tool to check for caching issues > > bitbake/lib/bb/siggen.py | 11 +- > meta-isar/conf/layer.conf | 1 + > meta/classes/dpkg-base.bbclass | 14 +- > meta/classes/dpkg.bbclass | 2 +- > scripts/isar-sstate | 863 +++++++++++++++++++++++++++++++++ > testsuite/cibase.py | 8 +- > 6 files changed, 884 insertions(+), 15 deletions(-) > create mode 100755 scripts/isar-sstate > Applied to next, thanks.