From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7095678487044816896 X-Received: by 2002:a05:651c:158b:b0:250:a056:7e48 with SMTP id h11-20020a05651c158b00b00250a0567e48mr9962927ljq.64.1652091397140; Mon, 09 May 2022 03:16:37 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:b1c:b0:472:38f0:bc75 with SMTP id w28-20020a0565120b1c00b0047238f0bc75ls1798457lfu.0.gmail; Mon, 09 May 2022 03:16:36 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzEXWbBuawSWufUhU3CVrWXyAIslPhWvLuGgRNAdi7B6KK3HQeJDLroeZzv/nquptAAB0bU X-Received: by 2002:ac2:596f:0:b0:472:5c36:17f5 with SMTP id h15-20020ac2596f000000b004725c3617f5mr11862035lfp.142.1652091396070; Mon, 09 May 2022 03:16:36 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1652091396; cv=pass; d=google.com; s=arc-20160816; b=K+BqlmkFBcRk6cLoniq6f30JlVP2HEKmKnU9/D8urnrw5AnVwbx3FTS2U9huJytcEe monq/ssih7oMqAiqzfEZi8S6OJICDmd+bMfG6/KCxa6sEOhxfZt4yyqfOrNpCuDJDgLs m92w2svywVFyYi8OvNxBkmpED5lMFuA5g5NAX3DPp5KFPbEkO7IETgW6VqVcQBYK8cnh PqvvxIC80AISQiP+Bu6aFeXlROHszYzSEy+yvuDVgu8cljUmZxS4/zvtM/+bKJ1MTCBw xmSNwV9Gr+dbO5N5maHALj82vJi+UKRpC27Z1oijBBV2JVUflAr55RyX8XqtysZ5roQ+ YGRA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=xH794yLJ3URSQMMdAIjsC7aYSKXBnf4W12jdkCLlSXE=; b=tLZRvb+S8R7fgL+w9SDwqGsjK06jVf0oOIFrQISdBW0kGxB53ti9H0Xq/VZtjD/hWU +1JxYr8e8mhnE3tt2k47+/Zbr+6F6nAeeQmMRa0uvepTjyqK9SY0nIhoddPT6pC4H/+X RVaAPWvhb/2b1JBkbJ1j4CbBBfYsaqVQjl4zMIhiJcrgiyVCys5IMALhk1IyOAOX6xR8 WYYDO3jAMxhvbF1vr3VkHtz03jGTXBZdp7FkmWOSu9nTxlYLC/RRRNvV1b6DnBM7o+TO /dkY9UZ1eFhWZTy8RbmrGlPAPKIr/z6XlDAHvJeJ8vCVo82Ix538E9ROA+FRs1F58GTd 40GQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=w5YSqd7h; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0c::609 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-db3eur04on0609.outbound.protection.outlook.com. [2a01:111:f400:fe0c::609]) by gmr-mx.google.com with ESMTPS id e18-20020a2e8ed2000000b0024f0dcb32f8si465112ljl.5.2022.05.09.03.16.35 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 May 2022 03:16:35 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0c::609 as permitted sender) client-ip=2a01:111:f400:fe0c::609; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=w5YSqd7h; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe0c::609 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Egj455azM1dBT8dsKtb0W0cs0exJEqOjjLhSr/y2DjL/HC8kL4v3r3xTA/Ja2h3eu8k5bGkzsmReIF+Lvu1BQ0oVTYMjenN2iCcHP6KIIcDF+OnhpUhv0a4isx94TEDfppIAvOmXRvlvfutOj3bgsiRvnFxBNhiXt3Y+/lVhE4rcn+TpJy+kCowPLzqTnPrBvyfJHcoNTw7rzN/Xra5z9/f0p67fPKjOwRS9nkyRuRUVqzCXLDgwlzHKbsaoCsVy4Gx7CU7cFhG53dQflAjult/4y+HMU5iy+xk/7XHeI64RXaa9BWbTc9rpLcqw4+QRPSt0yIFVj3DDEAE8Iz626w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xH794yLJ3URSQMMdAIjsC7aYSKXBnf4W12jdkCLlSXE=; b=QzjlPl2Ajg4Ll52Niseixr6BAvELdxGlLAOCMhZXSyMDgNTGmgBJXoiFochLoGPL9MpRGiH4ht3fgQteT0NleoF1XiSnM10sjFwfKMU5Wnn9pcQqXAbAF8r/l2+onxJmkb38/Zqptib8apWKeXx10xI71evno/kLhRHu9XeUmz4QPPbyfbA6yhsJPdrUMMNj+zF8Bijazl2v2HA9nMXcOVceGA9QnX2QWbLzL5YXaa3JX55FgJ7kTzRrK5mo0/jXk/6Uj2svkwuGqxfx+coA1YRjwm9/qnLeDquw2+QDRubdNY43CS4dodFEOhhlTgz/JdbquKuEN2RyrOhP4FuUow== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 194.138.21.72) smtp.rcpttodomain=googlegroups.com smtp.mailfrom=siemens.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=siemens.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xH794yLJ3URSQMMdAIjsC7aYSKXBnf4W12jdkCLlSXE=; b=w5YSqd7h226Z95eOrWpa/b0oMhUBMeK+GVYajIsIYeNoOx+E3UZp9LA1yP5GdB3294JE3qbITfwt8zNnWPv9z6gT++HAnSNvlG1txprkXMn/zxsu+lu6g4IolLVDCixERmFFNQOSECmjGCEMjtu3IzY4qfDE1dLIWMhOi8SkLJit+c8sUtVStEfTtFlJgzZDeD/OQQ85mqpwuDxvhUhEh+bo4esoWV1Vw3B4M0GkFMSmHAuCIu2yY+g3TBUIeojbjcnfxgcwd5eFarmZykdzCqey/95aPIK8IuD2Nctxg5B0NDGBtJWDWsSNHUflJnXfmzwoihntLcoRunwb+glEYA== Received: from AM6P195CA0098.EURP195.PROD.OUTLOOK.COM (2603:10a6:209:86::39) by AS4PR10MB5695.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:4f2::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.22; Mon, 9 May 2022 10:16:34 +0000 Received: from VE1EUR01FT105.eop-EUR01.prod.protection.outlook.com (2603:10a6:209:86:cafe::8a) by AM6P195CA0098.outlook.office365.com (2603:10a6:209:86::39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.21 via Frontend Transport; Mon, 9 May 2022 10:16:34 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 194.138.21.72) smtp.mailfrom=siemens.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=siemens.com; Received-SPF: Pass (protection.outlook.com: domain of siemens.com designates 194.138.21.72 as permitted sender) receiver=protection.outlook.com; client-ip=194.138.21.72; helo=hybrid.siemens.com; Received: from hybrid.siemens.com (194.138.21.72) by VE1EUR01FT105.mail.protection.outlook.com (10.152.3.156) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5227.15 via Frontend Transport; Mon, 9 May 2022 10:16:34 +0000 Received: from DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) by DEMCHDC9SMA.ad011.siemens.net (194.138.21.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:33 +0200 Received: from random.ppmd.siemens.net (139.25.68.25) by DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Mon, 9 May 2022 12:16:32 +0200 From: Adriaan Schmidt To: CC: Felix Moessbauer Subject: [PATCH v2 6/7] isar-sstate: add tool to check for caching issues Date: Mon, 9 May 2022 12:16:03 +0200 Message-ID: <20220509101604.3249558-7-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> References: <20220509101604.3249558-1-adriaan.schmidt@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain Return-Path: adriaan.schmidt@siemens.com X-Originating-IP: [139.25.68.25] X-ClientProxiedBy: DEMCHDC89XA.ad011.siemens.net (139.25.226.103) To DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b43a4054-6333-4f7a-6321-08da31a4feb9 X-MS-TrafficTypeDiagnostic: AS4PR10MB5695:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: KjPGAMZoHBxZDGxiPtU4yX3KEFHc2rp5IoYvZ36M+W8nJmN58QyHCTcRuq1gmXorvtROG/w00ZdCsWcomuc3CyTFXRoETIh9MXmwqW1dG6KdRuiGpGXpUk6LzEEXKZSDz3VLhzPQErLL4Zx52IrlwVHSILUrPfNnW9Uv0c3ItqogOb2fwBvGEwDVyIKvKP+xB5InPpEbufp8PA/RTa4xbLNRB0x//RUbgSlIsYm2DAuH9yBVa91PGDuZpnxfixtUeX+V+pfIzxGrtMkYOKkWxxppAlUHtxkShl1QOTO5mavfkJZkZx6ogMUYAdZV9i1jd8r4XYLR8OvO7vHhvLVOoaxR3NOlA3n7gbMpsU3CE9RzZX+XahVfk3vPknPl9sakgYaaO9cyvv4sXIRCHmvnU1CP7NfNz58eVV4soRSn5X9f8xGlFIXVskcj7xdE5wA+FiaT0f2dFIhPCYeqxjeZB0EySElzSRBlZdF3B8EqQzoWcRhsEO5Wuj91+2Uvo2/980EZwOFvzEUOI7WJFYfj9m+RnKQ4+x15hTKTZXiOM50cxIs4GF644eynRfSqF6IVBdhM09Ho5MYlU/hFDu16pxwZ9MtNHFC/5FdsCWpG35RzcoEZJVyxFYUC43ykjouhssJmbSj91FQTU6BYW8t7vWeeFb1D6k36wzdof+XQ9VvuiLFb4WmUka6punXsYDhH6x7669BWZ7CD3V6MAD3C3g== X-Forefront-Antispam-Report: CIP:194.138.21.72;CTRY:DE;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:hybrid.siemens.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230001)(4636009)(40470700004)(36840700001)(46966006)(5660300002)(82310400005)(83380400001)(508600001)(81166007)(6916009)(36756003)(2906002)(26005)(70206006)(47076005)(16526019)(336012)(4326008)(40460700003)(44832011)(70586007)(86362001)(1076003)(316002)(6666004)(82960400001)(8676002)(356005)(36860700001)(956004)(2616005)(8936002)(107886003)(186003)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2022 10:16:34.3840 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b43a4054-6333-4f7a-6321-08da31a4feb9 X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=38ae3bcd-9579-4fd4-adda-b42e1495d55a;Ip=[194.138.21.72];Helo=[hybrid.siemens.com] X-MS-Exchange-CrossTenant-AuthSource: VE1EUR01FT105.eop-EUR01.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS4PR10MB5695 X-TUID: Ery9xECfkhVy From: Felix Moessbauer This patch adds the 'lint' command to the isar-sstate script that helps in finding cachability issues. Signed-off-by: Felix Moessbauer --- scripts/isar-sstate | 73 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 71 insertions(+), 2 deletions(-) diff --git a/scripts/isar-sstate b/scripts/isar-sstate index 8b541cf4..8ea85edc 100755 --- a/scripts/isar-sstate +++ b/scripts/isar-sstate @@ -62,6 +62,11 @@ to all potential matches in the remote cache, identified by matching architecture, recipe (`PN`), and task. This analysis has the same output format as `bitbake-diffsigs`. +### lint + +The `lint` command searches form common flaws that reduce the +cachability of a layer. + ## Backends ### Filesystem backend @@ -119,6 +124,7 @@ import shutil import sys from tempfile import NamedTemporaryFile import time +import pickle sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(__file__)), '..', 'bitbake', 'lib')) from bb.siggen import compare_sigfiles @@ -556,8 +562,8 @@ def arguments(): parser = argparse.ArgumentParser() parser.add_argument( 'command', type=str, metavar='command', - choices='info upload clean analyze'.split(), - help="command to execute (info, upload, clean, analyze)") + choices='info upload clean analyze lint'.split(), + help="command to execute (info, upload, clean, analyze, lint)") parser.add_argument( 'source', type=str, nargs='?', help="local sstate dir (for uploads or analysis)") @@ -572,6 +578,15 @@ def arguments(): parser.add_argument( '--max-sig-age', type=str, default=None, help="clean: remove siginfo files older than MAX_SIG_AGE (defaults to MAX_AGE)") + parser.add_argument( + '--sources-dir', type=str, default='/work/', + help="lint: absolute path to sources folder (e.g. layerbase)") + parser.add_argument( + '--build-dir', type=str, default='/build/tmp/', + help="lint: absolute path to build folder") + parser.add_argument( + '--exit-code', type=int, default=None, + help="lint: return this instead of number of found issues") args = parser.parse_args() if args.command in 'upload analyze'.split() and args.source is None: @@ -774,6 +789,60 @@ def sstate_analyze(source, target, **kwargs): print('\n'.join(out)) +def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, **kwargs): + ADDITIONAL_IGNORED_VARNAMES = 'PP'.split() + if not target.exists(): + print(f"ERROR: target {target} does not exist. Nothing to analyze.") + return -1 + + cache_sigs = {s.hash: s for s in target.list_all() if s.suffix.endswith('.siginfo')} + + hits_srcdir = 0 + hits_builddir = 0 + hits_other = 0 + for sig in cache_sigs.values(): + sig_file = target.download(sig.path) + with open(sig_file, 'rb') as f: + sigdata_raw = pickle.Unpickler(f) + sigdata = sigdata_raw.load() + + pn_issues = [] + for name, val in sigdata['varvals'].items(): + if not name[0].isupper(): + continue + if sigdata['basewhitelist'] and name in sigdata['basewhitelist'] or \ + sigdata['taskwhitelist'] and name in sigdata['taskwhitelist'] or \ + name in ADDITIONAL_IGNORED_VARNAMES: + continue + if not val or not val[0] == '/': + continue + task = sigdata['task'] + if val.startswith(build_dir): + pn_issues.append(f'\033[0;31m-> path in build-dir: {name} = "{val}"\033[0m') + hits_builddir += 1 + elif val.startswith(sources_dir): + pn_issues.append(f'\033[0;31m-> path in sources-dir: {name} = "{val}"\033[0m') + hits_srcdir += 1 + else: + hits_other += 1 + if verbose: + pn_issues.append(f'\033[0;34m-> other absolute path: {name} = "{val}"\033[0m') + if len(pn_issues) > 0: + print(f"\033[1;33m==== issues found in {sig.arch}:{sig.pn}:{sig.task} ({sig.hash[:8]}) ====\033[0m") + print('\n'.join(pn_issues)) + target.release(sig_file) + + sum_hits = hits_srcdir + hits_builddir + if sum_hits == 0: + print(f'no cachability issues found (scanned {len(cache_sigs)} signatures)') + else: + print(f'warning: found cachability issues (scanned {len(cache_sigs)} signatures)') + print(f'-> absolute paths: sources-dir {hits_srcdir}, build-dir {hits_builddir}, other {hits_other}') + if exit_code is not None: + return exit_code + return hits_srcdir + hits_builddir + + def main(): args = arguments() -- 2.30.2