From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7085973421677346816 X-Received: by 2002:a17:90b:4f8f:b0:1cb:55da:6bdc with SMTP id qe15-20020a17090b4f8f00b001cb55da6bdcmr9016866pjb.114.1649831752547; Tue, 12 Apr 2022 23:35:52 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6a00:179d:b0:505:9501:adc5 with SMTP id s29-20020a056a00179d00b005059501adc5ls952131pfg.2.gmail; Tue, 12 Apr 2022 23:35:51 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwhK1fFAjvmFyPlxT/jxbIJAXxAtR9OQIANUCrS5DtjWq0yjHAS5fT2l7uMfnbPIWQUmXYq X-Received: by 2002:aa7:8888:0:b0:505:a3e1:d246 with SMTP id z8-20020aa78888000000b00505a3e1d246mr19053592pfe.76.1649831751781; Tue, 12 Apr 2022 23:35:51 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1649831751; cv=pass; d=google.com; s=arc-20160816; b=nvupHwVNhScG2NimfC+M5H1T433SevWRpbHE1stCMhhyzPE0IMEcC1AYQOkXLfpjJ9 spMd687yj/5uVY4ev/QODGJSHGU9c/8Js7WlBNL4sKFHgAwna0xAA86nUoy4EcdKpq/C dw291aZ8ejt/DP78Dvo6HTVnhxSXiGzVOAJp6L/ZK92aI/CXIIi/8frlflDZ4ZsKBS3W sdrMUiH+i3tVehTpm5lzuM+/2dOgJ8RkQBi5V8uCvX9hKU0xvf8HT9u7g5K31VY5v5G7 3TTvuSpdfK/uu1+IycG65rAnINun44svqInQ3xgplMIgBWWwJHidB7uE3MC975K1iprY LxrA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dkim-signature; bh=E0CaKH8+c99DFRZd6PHg2ah+/UymOSYz4rDwXKqgaXc=; b=JaX9wVrkPeNjXxtZWhECv31XW/fQTb7aFL/fcCW9SMIqbJOkVM6runJIdhJSxF4GML j8j+mnKptwYW2hokrItMLXX0EXUYKVkbxPaTpozQadz91jS2tdXwwJTn/pVajUN0/jgm Z6lbQD70kVWZsd0xVrXGpMW3xPohhgTuCR6Ga9kDZI62J6zzPo5awzCK8IE1faAL0Wmt E8c6t5fkSW513LWgsPRBcCyhZWDcmCfkZpw5dLqZMPBbjJrw1FqirsRewypwBcamU/Da XCOSA/KD3rQ/iYNp4D4hhhQQOtmUuX4PP3zZ45T3RgSfXbrHpDcLaUx4BIcNJ1SYWNRJ ivlg== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=KkjQoOw6; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0e::60a as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-vi1eur04on060a.outbound.protection.outlook.com. [2a01:111:f400:fe0e::60a]) by gmr-mx.google.com with ESMTPS id m8-20020a17090a34c800b001caa0d88ff2si803526pjf.2.2022.04.12.23.35.51 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Apr 2022 23:35:51 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0e::60a as permitted sender) client-ip=2a01:111:f400:fe0e::60a; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=KkjQoOw6; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0e::60a as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZyrNIkF+D6m+rNsvAE5kT1cPkkFJWFo7m+Ww8aSuu7k6mHC4Dqciz7ZGV5zQ6ojxf2g4OdgrDTtIxBZaRMLGWo9ZUweKRJWUzp3TKGreFyFs1OeT2uk0lALBFZKE8fMN5JP6BEj9arw6nR5rp6ZcMa5mddAVtBTZr4RopuEv0KmdwdqNneQpzVMpkkJqbrIUIV+GsPyihxiGHO5SOQ0NXBPR5fRoOHK6eym+lCayjUoqvoS5OomkmyLi/sSv3GD/fWt0tjOUnWYS2ajP5ic1i6Y5kRg7a4JMCv8WzngHncrrjd/JeQHc02ukOdDvTqoW1/6qnCts70IF3EMXgE2EKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=E0CaKH8+c99DFRZd6PHg2ah+/UymOSYz4rDwXKqgaXc=; b=HoMr7NpX8ORxweTumQsvgpI8Q5q7YS1LH+kzFVquZFC9pxdoPQOqklC1luwKShm0BOEgvzI9PT/4Lm8C85JKB6PYkGAjOSlRTvcc1DOxAjzP539bmg4F2tSPgJloh+t8wg/gAdqQZDTn/MHeozcVUafUbzr1JW+q5oXWCGceCnJixRBLOIGCKdsVoVg4+yJD5pK4ZJV1Oh2fUl6D6qlsAeLpQOmm4FHWixUTGYvcPPASnmozH05h2oJgK/n6e1n73wAKulUcKSPmdbfEhARbO9cNFgE8FPRFIyzfMVu5Ghzk8yYproFNB9s51IGec/oS2jAQUPIU9Gxl/K6OuPGIGA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 194.138.21.73) smtp.rcpttodomain=googlegroups.com smtp.mailfrom=siemens.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=siemens.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=E0CaKH8+c99DFRZd6PHg2ah+/UymOSYz4rDwXKqgaXc=; b=KkjQoOw6PS9PwPGiq7fJovVlemMWu0Kmo8tstG17AUwMWriQdfQ9i2tWR8hRslPASyd2SkkhPkHOTg37RXdVk+0oljF3i19IxsEy2gJ/cRPDVZNpJI6AKWwPC5CzTW7vqiLMzYe3IEITQgT80v50YxSkI3albgbgp0j9NjlZxe/32H/CPZW6343Ke4+jEFgk4lTpBJ39vBHWgudbMUIjr2C8yNgHVo7sDsPL3Y5ChQ+mKzCANi0eMrJNkUjFtBPz7x9rF4WwqYnrQWJf/nSZq/mf/+z55D53ihOTtQkL3Bd7fJxEau/MrwBjQ8QmBj93ixUG7lHPuyvmhAGlscY+mA== Received: from SV0P279CA0043.NORP279.PROD.OUTLOOK.COM (2603:10a6:f10:13::12) by DB9PR10MB4716.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:10:255::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5144.28; Wed, 13 Apr 2022 06:35:49 +0000 Received: from HE1EUR01FT107.eop-EUR01.prod.protection.outlook.com (2603:10a6:f10:13:cafe::81) by SV0P279CA0043.outlook.office365.com (2603:10a6:f10:13::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5164.20 via Frontend Transport; Wed, 13 Apr 2022 06:35:48 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 194.138.21.73) smtp.mailfrom=siemens.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=siemens.com; Received-SPF: Pass (protection.outlook.com: domain of siemens.com designates 194.138.21.73 as permitted sender) receiver=protection.outlook.com; client-ip=194.138.21.73; helo=hybrid.siemens.com; Received: from hybrid.siemens.com (194.138.21.73) by HE1EUR01FT107.mail.protection.outlook.com (10.152.0.244) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5164.19 via Frontend Transport; Wed, 13 Apr 2022 06:35:48 +0000 Received: from DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) by DEMCHDC9SNA.ad011.siemens.net (194.138.21.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Wed, 13 Apr 2022 08:35:48 +0200 Received: from random.ppmd.siemens.net (139.25.68.25) by DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Wed, 13 Apr 2022 08:35:48 +0200 From: Adriaan Schmidt To: CC: Adriaan Schmidt Subject: [PATCH 0/2] Sstate maintenance script Date: Wed, 13 Apr 2022 08:35:32 +0200 Message-ID: <20220413063534.799526-1-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain Return-Path: adriaan.schmidt@siemens.com X-Originating-IP: [139.25.68.25] X-ClientProxiedBy: DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) To DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: bf9d30c4-6762-4017-cef1-08da1d17d8f9 X-MS-TrafficTypeDiagnostic: DB9PR10MB4716:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: FGINQqzOBAQFmVuuvGRzRVgSDkCfJ16yWej9zLvbYB4Lb3HAds8toYPJxCqz5jPzC5Og90ziRJNkimds6NjLx3kJ7oz9w2DqrvtDAZarT5nsUZu7IDPpbpTfodfJ2Pg/HlxonkwE2z1pmBoj0Xd+DFm0YPZ/+6CQ59GtLyWjH2J1MrN6LYNpb8uUbfxaGX0ixakumslTom0uJsSkuO6Y89ew2vJ5FhRTDs0JWQirvATdcRoyHIGCWuTjK+2/gLtplAfVPwPOd34L2IGfgIbSaGCFENreE5HLl7musqZhxbTwykNvVa3kT+dctPaizM8mqZN3UAebcM4fHiht8NmKHQZqKJ6rJCSl46zqjtoqzGGf+SiSQJrn+FvqS2fISf8mGQj/AAHtvpNewe0uet/+yhmAZC/sxDhTXZaqYKp+79F2Pwz8PVRu2n/2NRROsn2jNg3LqIdI333lfH3mnsBvvZ9yRS1QcAb3/AdFf+WUBqEXeqLRUgctHwukQ5epxU12zBHoPE6VtJ9VOZmE9j5JNMc59ZENJceYP87BWShec0pmRl/fljAVOfF9A8jrbPxV/Ab8rWlSCL+v5LRLdpwo1G+Ck+L+0S70J4bYwzMrMQFcDTlkxUQGy07DOm20gyksIbP9Guq8KCzTuqAA+hRjYAJH0l+JVAus1wJBNtST41G0Yp/V7zh9TDps3znn87DzmfaFBI5uwgw4MdIrl3TvnA== X-Forefront-Antispam-Report: CIP:194.138.21.73;CTRY:DE;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:hybrid.siemens.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230001)(4636009)(36840700001)(46966006)(40470700004)(36860700001)(47076005)(6916009)(5660300002)(956004)(26005)(336012)(186003)(1076003)(2906002)(2616005)(107886003)(44832011)(356005)(16526019)(81166007)(82960400001)(36756003)(83380400001)(86362001)(40460700003)(82310400005)(498600001)(8936002)(4326008)(8676002)(70586007)(70206006)(6666004)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Apr 2022 06:35:48.7268 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bf9d30c4-6762-4017-cef1-08da1d17d8f9 X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=38ae3bcd-9579-4fd4-adda-b42e1495d55a;Ip=[194.138.21.73];Helo=[hybrid.siemens.com] X-MS-Exchange-CrossTenant-AuthSource: HE1EUR01FT107.eop-EUR01.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR10MB4716 X-TUID: 0oOmRfipkfdQ Hi, We have been running CI with shared sstate caches for some months now, in several downstream projects. This is the cache maintenance script that has evolved during that time. Detailed documentation is in the script itself. Main features: - upload cache artifacts to shared caches on filesystem, http, or s3 - clean old artifacts from shared caches - analyze in detail why cache misses happen (what has changed in the signatures) The last one is especially interesting, and has already yielded some improvements to the cacheability of Isar ([PATCH v2 0/4] Improve cacheability); analysis is still ongoing. This feature becomes more robust with a patch to bitbake (p2 of this series, also submitted upstream). One issue: testing! This is not easy, because it involves infrastructure, and artificial tests that provide decent coverage would be quite complex to design. If we declare that we sufficiently trust the sstate code, we could add a shared/persistent cache to the Isar CI infrastructure. This would further test the sstate feature and all steps involved in maintaining such a setup. In addition, it would significantly speed up CI builds. Adriaan Adriaan Schmidt (2): scripts: add isar-sstate bitbake-diffsigs: make finding of changed signatures more robust bitbake/lib/bb/siggen.py | 10 +- scripts/isar-sstate | 743 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 748 insertions(+), 5 deletions(-) create mode 100755 scripts/isar-sstate -- 2.30.2