From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7095678487044816896 X-Received: by 2002:a2e:a235:0:b0:24f:1209:fcbf with SMTP id i21-20020a2ea235000000b0024f1209fcbfmr10250207ljm.313.1652091389904; Mon, 09 May 2022 03:16:29 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:4ca:b0:24f:f52e:16e0 with SMTP id e10-20020a05651c04ca00b0024ff52e16e0ls2778235lji.3.gmail; Mon, 09 May 2022 03:16:28 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyAJP42osVzReOVPpyvi9bzQz0q0GMEus2TdSvfrKHiweGcyaevE5DdGgYEnnR9yTo0xjkP X-Received: by 2002:a2e:9953:0:b0:24f:2926:6a23 with SMTP id r19-20020a2e9953000000b0024f29266a23mr9958307ljj.312.1652091388750; Mon, 09 May 2022 03:16:28 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1652091388; cv=pass; d=google.com; s=arc-20160816; b=OPanQpwFOeKsCMgRqwUxGVxxbrprILfLePMiA09B1taq2HvmsQ6V86IjYprKZ8MVyK 63vhUG7r7WcuBCcSk1R83RwqqcV54eWaOk2v0Sd1dMgha3l9ZK3+p/0N+48G1W/A+Qrv k2Pqt9DQVzwWPI+suOM1bNIf0P1jEAYa68Tr+UMojbFj99C5a+tNqdF7Nq+XlK6oVLBZ Nxf+GuV9/rVloOIab5w00zcU6K2Jm4WRCeKlVGqG9Uhl3leWhnzo4UcbzthW3YK3JYzM UbPdyhBzxSQGgKSa8uoBVZYDtq2RoIOKIuuxA+Tn5QGkr7SmYoNdm2XeOWfcnic60UE6 Ql2g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dkim-signature; bh=LukLnJJ8bdM3RtizZ6zN9vcty59nn+4XHvtEn077ASw=; b=n5q7i5Ovzyw8EYBU+hoMunJbkNnW0tkjQkHRz/OdkViLJzr7gEX7+7UaXkzfxJjaZG XQTrSJU26Es8mgJ0NA+XHquGE3Skp56lVC+uwt0MFamvnLAd1/6bKCU4mG/I/u17+Zv6 16+tu0Cl5+Lj0LGbLvEfWdk7fiDpIbp52GLB4zYm3GoZ5CN+aVr/pkTDEIhmzNyBvVsT U5iodfQJcqsPS+fQbVb9PYxdTx9qUUowOIo8L71o2VHLKg+SESaHfJNhgWbKwEtntUDa cKUEGXRRM955r5QuuhYQy9J7/Xc3W3DVi4A3hc54zq281CNswZP3XyGtRyjKXJmMcPtf TNvg== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=GHDrniOf; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:7e1a::630 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on20630.outbound.protection.outlook.com. [2a01:111:f400:7e1a::630]) by gmr-mx.google.com with ESMTPS id j25-20020a2e8519000000b0024e33a076e7si465430lji.2.2022.05.09.03.16.28 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 May 2022 03:16:28 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:7e1a::630 as permitted sender) client-ip=2a01:111:f400:7e1a::630; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=GHDrniOf; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:7e1a::630 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SOYc5vrADTnoJQCouYOrH4s4/vrKC2jgnK9gWGKjivCJf0EysXETOUo1VZp1wk/gVb2Wm19ytjblbFQOXjMNF4O1QbZ7mxumVWyPDKGynNXjBf8uJ2hvcSbTS+CSfegIihrEl7w5sdWo7CS3D1xcu6dGGsb3lH6acHKZAePE8s99MqQyPbcLNa1FJ0yX4Gsz0cnt4XExvZH7bMV7DUm34hAx31MVfDlQ72PM9VTmhTRF5Abl2VWcrtTtxLuQwnXbmU29nyKBbGrfdb9me4QRMr3E9VkZJwsZdBltCq6D3Z3bHQ1G0xE08U5oJCR1RHbYSkrO+IiFA/f942NgGdga6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LukLnJJ8bdM3RtizZ6zN9vcty59nn+4XHvtEn077ASw=; b=cN3V8iy51L7MmhLozbicaUaOUvsKgHf10mxrzh9ZkpDthQ8hNWzRA0KP6qdlvoGE0I2M+Bti2TNQkYg8ZQkfepPgQItVIEGjD4/Tmxl1qdbqezdUvSeUx/9SQHI5bviRLiB+yLQdxcnVVWoc458xSaYeBqQ0PXu0AkxSx2PY7rax+V7SU3wc2+7YrgJcJo0FWWfagNP0UVMRme+s3g6yk62FOayEYnUoBOFL64HgbOlOwhAvZwziZNUzyX/RkBDffk8dMl6h6o0Zd+QevHGxsQ2LS92kffqeSBGCAFt4ZWHYflMZxnU/nEgHvVXzwtyHQnLUHIXmCX1V8ShiWOhLGA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 194.138.21.72) smtp.rcpttodomain=googlegroups.com smtp.mailfrom=siemens.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=siemens.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LukLnJJ8bdM3RtizZ6zN9vcty59nn+4XHvtEn077ASw=; b=GHDrniOfs6iWiYW95WNO93znmyCuEMvizLUihZMHS7TQsRSqdHpDJpOa6LZFEhLtwZPjs+fIGVX9h6e85wOa3mmt0yOVKVRHGAAjS5q5eX+B+uCz26bznZUsK571rRK9HMKH3+ICgLT9AQjDyPaL94rxy5aIM3135dzyLY3RUv9ZvVEghdIEgFKFPFCP0nA2rnZ0CfVCSuLOX0ejDmjdoXOlA4Pk80ioSHGgK+Pc1y7S2KUfEGEMZh1ZsGXIQTySfa/OKJyD1DrZWxwe9nbJbv5TJeO4I6dacnZo541M2eq31pVFZR1H1UjEseoT2m4cr/KzskE8YcdbmmwvRNs5/w== Received: from AS9PR04CA0090.eurprd04.prod.outlook.com (2603:10a6:20b:50e::11) by AM0PR10MB3090.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:208:15a::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.18; Mon, 9 May 2022 10:16:27 +0000 Received: from VE1EUR01FT070.eop-EUR01.prod.protection.outlook.com (2603:10a6:20b:50e:cafe::92) by AS9PR04CA0090.outlook.office365.com (2603:10a6:20b:50e::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.23 via Frontend Transport; Mon, 9 May 2022 10:16:27 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 194.138.21.72) smtp.mailfrom=siemens.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=siemens.com; Received-SPF: Pass (protection.outlook.com: domain of siemens.com designates 194.138.21.72 as permitted sender) receiver=protection.outlook.com; client-ip=194.138.21.72; helo=hybrid.siemens.com; Received: from hybrid.siemens.com (194.138.21.72) by VE1EUR01FT070.mail.protection.outlook.com (10.152.3.47) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5227.15 via Frontend Transport; Mon, 9 May 2022 10:16:27 +0000 Received: from DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) by DEMCHDC9SMA.ad011.siemens.net (194.138.21.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:26 +0200 Received: from random.ppmd.siemens.net (139.25.68.25) by DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:26 +0200 From: Adriaan Schmidt To: CC: Adriaan Schmidt Subject: [PATCH v2 0/7] Sstate maintenance script Date: Mon, 9 May 2022 12:15:57 +0200 Message-ID: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain Return-Path: adriaan.schmidt@siemens.com X-Originating-IP: [139.25.68.25] X-ClientProxiedBy: DEMCHDC89XA.ad011.siemens.net (139.25.226.103) To DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 84b338bb-2589-4e03-f448-08da31a4fa6d X-MS-TrafficTypeDiagnostic: AM0PR10MB3090:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: GktIj/wXDLrmjU6cC6hHNX34mlUVGnbStsY2wRxKtcX7QdqnGOtKD96urbp7YoF2QaezlO/XJyQK/RSPzlhRllmGxNT0VSwYQj7t3y9SYiYLeKxY71xkXhO6Nkg3800KIMO+QD+kfilvxFr+V9fTi/7QddGH3L4y9nvShS3vZJ1YnMN/IMMlyt/FqX+awSJNx54A6IIZRpWrpcNIXtFrj81hodpLhHAeVKjUHo42X6+KoiFJXDZOdX8OvWmUMB/IKBDLPbn8R/wrSW6J5E8OOMMW356zEQCF+mNrew/OG5GSUcmPi2mG8neu/gVjbtDDv+qB9NpcX9xlHEHBYjdEEWbxukSMa7SYLmjhs2Pm6WSbghlEmLY38Zy5I5ssuFPAqHkfrSdmCs7xVD1eWeJfiQJEtXvoaMEFar+200GK1ImQ3OhXhQtYkVyX/iO3ShjTDFZK2wgVJbhIJYwg4NdfOGkjDBARgt+gSBjCkGFU4wFC7PvCSmIIsx9TKMM46yXLtVRre1uMfiT4R1kPKAJmaFo0zMYiL4uPmuAK8wXbrVwr64BzyULweQTo+ET4onynhA5JXNMO+oIhxZUtLohNrAlrf3GTGBxskEuZtOXqxZ7+C458crJONi7nI3eSzBDAp/EWpow29lv4TKfEqSoHq6wtZpoarvBHxczkA91qKxnTsIUytSZ4eCyHIFAf9Upr6eIEFMn0kVSeAM8zkpfABw== X-Forefront-Antispam-Report: CIP:194.138.21.72;CTRY:DE;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:hybrid.siemens.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230001)(4636009)(46966006)(36840700001)(40470700004)(6666004)(2906002)(82310400005)(8936002)(4326008)(36756003)(8676002)(40460700003)(956004)(5660300002)(86362001)(26005)(81166007)(44832011)(70206006)(2616005)(70586007)(508600001)(47076005)(107886003)(1076003)(356005)(336012)(82960400001)(186003)(316002)(16526019)(36860700001)(6916009)(83380400001)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2022 10:16:27.1785 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 84b338bb-2589-4e03-f448-08da31a4fa6d X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=38ae3bcd-9579-4fd4-adda-b42e1495d55a;Ip=[194.138.21.72];Helo=[hybrid.siemens.com] X-MS-Exchange-CrossTenant-AuthSource: VE1EUR01FT070.eop-EUR01.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR10MB3090 X-TUID: 4G+mJq7CTkee We have been running CI with shared sstate caches for some months now, in several downstream projects. This is the cache maintenance script that has evolved during that time. Detailed documentation is in the script itself. Main features: - upload cache artifacts to shared caches on filesystem, http, or s3 - clean old artifacts from shared caches - analyze in detail why cache misses happen (what has changed in the signatures) - check the sstate signatures for absolute paths pointing to the build host The last two are especially interesting, and have already yielded some improvements to the cacheability of Isar, some already merged, and some more in "[PATCH 0/7] Further improve cachability of ISAR". p1 handles another absolute path in a variable (LAYERDIR_isar). p2..3 are minor patches to bitbake (both already upstream) that greatly improve accuracy and performance of the sstate analysis. p4 refactors handling of the apt_* tasks. This was motivated by the sstate analysis, but I think it also makes the code cleaner. p5..6 add the sstate maintenance script (2 authors, hence 2 patches). p7 includes a signature check into the sstate test case. This requires the changes from "[PATCH 0/7] Further improve cachability of ISAR" for the test to pass. One issue: testing! This is not easy, because it involves infrastructure, and artificial tests that provide decent coverage would be quite complex to design. If we declare that we sufficiently trust the sstate code, we could add a shared/persistent cache to the Isar CI infrastructure. This would further test the sstate feature and all steps involved in maintaining such a setup. In addition, it would significantly speed up CI builds. changes since v1: - generally improved script - analysis and cachability improvements in bitbake, dpkg-base, and meta-isar - added "sstate linting" to the testsuite Adriaan Schmidt (6): meta-isar: improve cachability bitbake-diffsigs: make finding of changed signatures more robust bitbake-diffsigs: break on first dependent task difference dpkg-base: refactor dependencies of apt_* tasks scripts: add isar-sstate testsuite: add cachability analysis to sstate test Felix Moessbauer (1): isar-sstate: add tool to check for caching issues bitbake/lib/bb/siggen.py | 11 +- meta-isar/conf/layer.conf | 1 + meta/classes/dpkg-base.bbclass | 14 +- meta/classes/dpkg.bbclass | 2 +- scripts/isar-sstate | 863 +++++++++++++++++++++++++++++++++ testsuite/cibase.py | 8 +- 6 files changed, 884 insertions(+), 15 deletions(-) create mode 100755 scripts/isar-sstate -- 2.30.2