From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7085973421677346816 X-Received: by 2002:a05:6512:486:b0:46d:11ca:18fc with SMTP id v6-20020a056512048600b0046d11ca18fcmr1235961lfq.102.1649924536302; Thu, 14 Apr 2022 01:22:16 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:5285:0:b0:44a:94d2:1d8f with SMTP id q5-20020ac25285000000b0044a94d21d8fls3926681lfm.0.gmail; Thu, 14 Apr 2022 01:22:15 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxG+viM/gJCUOoCaJDYwhbl+ytoHCItFYkW9uJJyJAJC0vcHY37AoegfZUyI/sjjL7TYty5 X-Received: by 2002:a05:6512:2622:b0:448:27b9:5299 with SMTP id bt34-20020a056512262200b0044827b95299mr1194936lfb.86.1649924535088; Thu, 14 Apr 2022 01:22:15 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1649924535; cv=pass; d=google.com; s=arc-20160816; b=lGZnPCjVNqnJxdEez7SCY4X8pcuMJ3qBbPx0qJqPu1BslmwhHCGRbnqNg5Dv3/T5vH V2b0W/O5I6X5cvQQu47sBQDBM8K+Wp2neyFwZoNZE5tu2715CLIBSHOdFeZmEdkxc9ne NK9fIUlvjL3cs3AxqEb2R3OfPDZEEAcqQ8lxAGb1lcJ+zS2CxY4MJSGEhzvSIGZnaK4t Ck4AtECRNZVQOXPP5apjBh4NoSCFsI6s9oGc0EePaWY6C7P9TqhJENh4yXoJ1oIaXsKm bClSOYNZVk2BcVMVzGq42N5WlNbByoYcJqWNKYNw8Am9hYf38jtgBt1Lp2RSrZfHflGJ jkVA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:references:in-reply-to :message-id:subject:cc:to:from:date:dkim-signature; bh=IbO4CCoqJUqRovUrL1ldnMj+/nOa7uABoIBtmG8DL6o=; b=KO6vLBK0cCwuKB9XdeXcmkEtOsCdkt7dCjn2wjeIoSisaN25yerjojyXH3L9V6uwBm ZO9StW2bMAf1KS9KFIFNwncminmmAKRb9NoBjwTaTLU+8lJUOCqyn+eY1ZxHZ4ipsIxt G2Qu+xhfRnSw66SeaAG1wFUDpH4qfXMj7IC7wwGuJ6BattSRCULJRgSxpyA5BlZZh9SN vaANSxJ6VrPTcbRRSyoNrz0b6sGAuneoHHCJHuTRrP2SFrX/HMHC7AIeFKGDZZMhHbAg jdSOUFXI7/lEZegbGGW9Ua0vquQefu0GDhxtJxu1ykGq5pPziXXoGVp6GZvUtmHbhfyV dakw== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b="GSV3Vx6/"; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe0d::60e as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-he1eur04on060e.outbound.protection.outlook.com. [2a01:111:f400:fe0d::60e]) by gmr-mx.google.com with ESMTPS id s27-20020ac2465b000000b0046c9d290537si225558lfo.9.2022.04.14.01.22.14 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Apr 2022 01:22:14 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe0d::60e as permitted sender) client-ip=2a01:111:f400:fe0d::60e; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b="GSV3Vx6/"; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe0d::60e as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ACzhVigzDzEwklPVilp1zCZMXfajF+g8lY0fuUgFbMVOL0el5eRxC86oMtxWZLtzMm626AZPoGS4y4dVT5fYkMCsxD9T5gCvDsf82gTDx4yw69O0W9kukXEdDEzGbHvUSE/JYESyW/rHckopPhkQnGMTszmo4rZfZzZngF9kW1pMw9k59g81699eqqaF17xr5f01AM6c15CFgRyROSwwSTpRvYwLgr/S3Jx2qNB3JuIBC/4u5Ok3Z4luowF1PRjfvoXXuu2MpJPj4AO4nuvhxZl6L2WB9c1VhFXQvWflV9DZ6860ePT9mNprt+Udg8AtRXLViBcQJMDT7nWH0untQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IbO4CCoqJUqRovUrL1ldnMj+/nOa7uABoIBtmG8DL6o=; b=HB72WX1MfJ7GbbhiYMuARB+NRnumpIsD/ttGxNIBPm/kAOIgDWZVCruqORutivRqlqRdZKpsmRp0OG/Daymo0FyZH1WqvvZRpBRrItdj8layhHsNijXaVT2rz8D/lmzsAxlOd+j41rWLDmdzHDk8bcpAr6c8TrCrg+pcY7b33/RIQNBvo+SUTYR8rEzotyk7ZMLrX7/YuoLXF8A2/irg/uhImeMs23IFSrlz0O51QgN/LCzk6wNcZr+qcvDVP4Icd1YIZe/hvVmvjyywmJuuuhbBLpDwV5mQCIs1Gw7ri5o7U/ABFOLT1n01/jvObcNP3fLBR/yvrAKLIVy29GDrJQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IbO4CCoqJUqRovUrL1ldnMj+/nOa7uABoIBtmG8DL6o=; b=GSV3Vx6/bmf05mDmzlzvOlwQTOpib2OUQFXVJZ9faTCKns/s2517M3k3zw6O/LlqMsCAmSoJMwjFuOueAYQ7ehfQfGP/kLnCaqFPRBA63eN9BqxM0SUVyg2ZnZClJQ7vpGdm5EoywLBv8sAJsdtILTIgh38Re3ZK/7eTHXQlpQDVqzakGmDaiknaWnACL0zT8xdr2nqXlQ3gpOFJGStM2ZsiwmO/gBVyZSXBCWTfzrvXEVEmAmCCL0oamerP1U97iPnKRRDjypX2/GmoEGUvRNQqjsYuZ1Pjhd5S4i/jMZ1jtfZD3BAno39ita2GuYNb71B99U2NoS0E9G540e0NVw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:269::8) by AM0PR10MB2548.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:208:115::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5164.20; Thu, 14 Apr 2022 08:22:12 +0000 Received: from PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM ([fe80::f4be:8b5d:4314:c2d8]) by PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM ([fe80::f4be:8b5d:4314:c2d8%4]) with mapi id 15.20.5144.028; Thu, 14 Apr 2022 08:22:12 +0000 Date: Thu, 14 Apr 2022 10:22:07 +0200 From: Henning Schild To: "Schmidt, Adriaan (T CED SES-DE)" Cc: "isar-users@googlegroups.com" Subject: Re: [PATCH 2/2] bitbake-diffsigs: make finding of changed signatures more robust Message-ID: <20220414102207.6e9cec4f@md1za8fc.ad001.siemens.net> In-Reply-To: References: <20220413063534.799526-1-adriaan.schmidt@siemens.com> <20220413063534.799526-3-adriaan.schmidt@siemens.com> <20220413101954.67aca2e0@md1za8fc.ad001.siemens.net> X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-ClientProxiedBy: AM6P193CA0080.EURP193.PROD.OUTLOOK.COM (2603:10a6:209:88::21) To PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:269::8) Return-Path: henning.schild@siemens.com MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 56177044-3f17-4cc5-9160-08da1defe089 X-MS-TrafficTypeDiagnostic: AM0PR10MB2548:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3muAPHM/NEezQ4+6Mfq24R7mKNswvtHqThXt4iAV+pa+6Bb72Qd2f1l72lC8Z/mzEfnpkF6xGbWhdUtAwJL/kwDAYWOd1SXpnOpjZ6h9rkgSiDlpYRo+mFGcFxY8vGNvAoZ/5JnKeSrGxq21UCv1FMeHw+N6jEmWgMNNhr7poIMZLYhW/Wgyoqno0cbKI4YojaEa5+DbQ/sVvtuVmG7AN+3QiEuSRX7S9UtU3AOauoxsMfrjAqivqfNgAE0UlH6gTxCV9y2XaC+w9yJfp+pBulgt/Un8sgPJxRpaVXZZiGwAE5GJjDvoF6v2cV2PXgHhbWhEXyr1pTGdtAR4vSTK5ucVZb7TRm/xVmOxrgt//p1U1icTFfWC6VPJpfB3cGMs5NpqBhgCxdWyXLC1Q6aY+jiafrwsWfixdHiHYEkj3SNolCSIaO6XdjnVWodWAbLhRaJd8LH3c8UPKPKXZ00uTPwo0POsjhZaUtJ24sMsQAVaUs0tW3hnA3khVe7502iaOzfB9bEaPJpagYRtLjs1QnusCgeSsKGdV+++mHuaSPUrhpk1A5KFZ7tc1J/Uzm6mKaOhkphZTyOBtFgWiT4/W1UmSPoh3bTMVVWhbUhlCHyVdWelDdP56z/rQuO8BSoJlBn9b8UzrmzsQAc/lAHjZeDKL+03PznUBpGhqaHTX/wbXoKCd4VVevHw1liHpcQutkLhWBlitGXwYX2zJKOkgBmUKgnSAEc9zUP44sPB4vg= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(82960400001)(8676002)(4326008)(86362001)(5660300002)(38100700002)(2906002)(6862004)(8936002)(44832011)(186003)(6666004)(9686003)(1076003)(6512007)(6506007)(6636002)(6486002)(498600001)(966005)(66946007)(66556008)(66476007)(83380400001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 2 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?NkyQH973JAhnYf4/IYmfpgNkjcp/+52/MuufI7T670e6FJD2HLlpT9fY5/XH?= =?us-ascii?Q?FD9Bq3vxJCaO3VTS0VND0ofW5wDkHQvRpTXMAi0hLtFeIM/fl725rAxIcMsq?= =?us-ascii?Q?zHRBMBbn/f4BjskM7az6K/Xah1DKtes3vSVUJPoW7eLsKzT74+bdCxJaqaHu?= =?us-ascii?Q?hgiTK15Y5THgyYqUHgMHdnG6lXd0rIgcDWXstLhkbw3PZxHKHr6UbkJJgzSJ?= =?us-ascii?Q?xscglLWaPAj2NC7bEPNDN4IL4qLdL0GCmRC5iX9WH6ai6e1yp4ktAYk1B+Ln?= =?us-ascii?Q?8sWtB6u+AHdL2pyL33zJQW0wD/1NgezCEIX7Xf9ItryhwdbStPdRkTRnC7yU?= =?us-ascii?Q?hOwpm8FukakRIP0Kuzz83FqJn4xew3iogvUcs40kbixIqBZcRxW5s8LBZ+AC?= =?us-ascii?Q?tsOadlfR8QToZD0+hF7EEilh3g+Kkp5m0a8FcRvuGGvc7bCPNs8QtY5p50QK?= =?us-ascii?Q?L9ueZOlWRDfL7kAvoegNoHlPnAEg1p+cj5/PkggfyGYpuGlM8oU5/98J/5rC?= =?us-ascii?Q?CPiRJpgO7aZ4Tx2Lu1OAAUBcD48LHEUVy2XR2/iwE5ZvD+1k6AmGOirPEreD?= =?us-ascii?Q?q2DjLj+zbuUuEJWK7i+4/taUysNhk823cH7p1TAHG9qcMjFJatBYI4GgVw3k?= =?us-ascii?Q?CZCyMdU56Fuuv7+fdwCXZevfPy8RW+OZTLf8bzrh/wz1wy2jqQUheGFq2cCV?= =?us-ascii?Q?/PylXiFbcPr+ZGVSoMJ9BPFy54Xn0EN6iMOTS7oZ3020mKcEbLWae0fV9kVp?= =?us-ascii?Q?A5CdHnBOsai3vjSYM4OOyO/TGDZFdRYRjKu8vXBUXgLiu9Sl0NmbAXsYcJj4?= =?us-ascii?Q?LY6VO/pgoJtCl//xvq0c8/0EcezBEDD8oXQOmcn4etxoF16aSoj6s10MSgoN?= =?us-ascii?Q?/Tlf/ihuq5u/YVSrMMqSyWhHtiAyFNXD98ggu2BDZQ8oohKaTd5PFF4QF1em?= =?us-ascii?Q?n4eCVKDtlFO9mYqCUxVz8pgEl/wyz6tM9iftJ4AiJUv+EpMCW2oZPcuJXz2R?= =?us-ascii?Q?z06vUXMbC1qwbwzwm2Dm+/o88AMkxF6co+RMbu2h4ZBzwBYCO2fAUkeRipLc?= =?us-ascii?Q?G9kdYVwmrKDyfwKnUIRvjVJ3U1jxUD4nkIdB116ZyMo9ce/Di0aMOXBAbthp?= =?us-ascii?Q?87qDpDTAFbKrA1t5DXTEXD2k6uPDV6+zL1BOGnZFcDI4397EgFseDJ52LXzL?= =?us-ascii?Q?xwJWlZBg1b2l5wV2/tb8Q/IwRk83q7SSP8q4fzNl9JDsP51WknieI6nGmoZu?= =?us-ascii?Q?qNCawVrD1lCQftZ/nfnbe39cTh6AZG35CXss1JhHCU8KgB6zWcueWde3vE7d?= =?us-ascii?Q?sMB0m9vPeA3Sr+VxARNoPf1uHUfH/QxTNqLSpbygOSaWp2R1n91WuT/0q2Ym?= =?us-ascii?Q?btYP3dAoatgU/nhEA2GFT8fxTCJIxNTT1fDsox98+46Y+2m+Rqz3Q9vqZbib?= =?us-ascii?Q?cjqxQWRKpcRMxQUFu+qPti1QbYdootUmk3rzGBj9mhF2FRsz51h1HQ0Fyep2?= =?us-ascii?Q?wq1fuMvPvEVj4OG8Ejr2jv7XHxTtUA5q6jOODNqMNSPuQx+ubKKPt45cXgZ+?= =?us-ascii?Q?8fJAfa178So/Hj857zLaW9RKmuqxADrb8kgzfR88ZsLj1UHLWF3lCgodgniN?= =?us-ascii?Q?Df4sHULD3lJhJensc4m4xNqP3Swx0py+h37Uv2iyvwXJgUdZd/WcoKuIi98H?= =?us-ascii?Q?pDCY10w3QKzJe4WMQmEmAFYtwxxMJvfaiLywHUuLoCA0sliLtG9JzLqMJub8?= =?us-ascii?Q?9rGcoR86IsM8/eEE4lAhw39roZipMAyfSKFkIMqQ6mT6lsp9k/Op61W6pmyP?= X-MS-Exchange-AntiSpam-MessageData-1: fnk2HdIgoSLDFxIgFEmwWSx6r9wtacrUsKw= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: 56177044-3f17-4cc5-9160-08da1defe089 X-MS-Exchange-CrossTenant-AuthSource: PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Apr 2022 08:22:12.8941 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Sqhu/tOewSVVW1vTFrSrzx02Z72D0HZ8sO1D9Zj4gIdhucYpXSyqFNQQGX8MFqRrMVmYoT4iTXgY5RZm6eibpRTlIu4NDtbZ8TER+D7F4Zc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR10MB2548 X-TUID: AQpCVdc7UNwA Am Thu, 14 Apr 2022 10:03:39 +0200 schrieb "Schmidt, Adriaan (T CED SES-DE)" : > Schild, Henning, Mittwoch, 13. April 2022 10:20 > > Am Wed, 13 Apr 2022 08:35:34 +0200 > > schrieb Adriaan Schmidt : > > > > > In `runtaskhashes`, the keys contain the absolute paths to the > > > recipe. When working with shared sstate caches (where these > > > absolute paths can be different) we see that compare_sigfiles > > > does not identifiy a changed hash of a dependent task as > > > "changed", but instead as "removed"&"added", preventing the > > > function from recursing and continuing the comparison. > > > > > > By calling `clean_basepaths` before comparing the `runtaskhashes` > > > dicts, we avoid this. > > > > > > Submitted upstream: > > > https://lists.openembedded.org/g/bitbake-devel/message/13603 > > > > Submitted does not count, we will have to wait till it is merged and > > then we can think about a backport. > > It's now on bitbake's `master-next`: > > https://git.openembedded.org/bitbake/commit/?h=master-next&id=01b2b300901dc8b93973318127f8eb3c29b9a168 Cool. I guess the commit message should be changed to include that upstream sha. Henning > > How important is that? If it is just "nice to have" i think we > > should wait and not even do the backport once merged into bitbake. > > In our specific setup on a GitLab K8s CI runner, we need this to get > useful analysis. Other setups might be fine without. > > Adriaan > > > Henning > > > > > Signed-off-by: Adriaan Schmidt > > > --- > > > bitbake/lib/bb/siggen.py | 10 +++++----- > > > 1 file changed, 5 insertions(+), 5 deletions(-) > > > > > > diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py > > > index 0d88c6ec..8b23fd04 100644 > > > --- a/bitbake/lib/bb/siggen.py > > > +++ b/bitbake/lib/bb/siggen.py > > > @@ -944,8 +944,8 @@ def compare_sigfiles(a, b, recursecb=None, > > > color=False, collapsed=False): > > > > > > if 'runtaskhashes' in a_data and 'runtaskhashes' in b_data: > > > - a = a_data['runtaskhashes'] > > > - b = b_data['runtaskhashes'] > > > + a = clean_basepaths(a_data['runtaskhashes']) > > > + b = clean_basepaths(b_data['runtaskhashes']) > > > changed, added, removed = dict_diff(a, b) > > > if added: > > > for dep in added: > > > @@ -956,7 +956,7 @@ def compare_sigfiles(a, b, recursecb=None, > > > color=False, collapsed=False): #output.append("Dependency on task > > > %s was replaced by %s with same hash" % (dep, bdep)) bdep_found = > > > True if not bdep_found: > > > - > > > output.append(color_format("{color_title}Dependency on task %s was > > > added{color_default} with hash %s") % (clean_basepath(dep), > > > b[dep])) + > > > output.append(color_format("{color_title}Dependency on task %s was > > > added{color_default} with hash %s") % (dep, b[dep])) if removed: > > > for dep in removed: adep_found = False > > > @@ -966,11 +966,11 @@ def compare_sigfiles(a, b, recursecb=None, > > > color=False, collapsed=False): #output.append("Dependency on task > > > %s was replaced by %s with same hash" % (adep, dep)) adep_found = > > > True if not adep_found: > > > - > > > output.append(color_format("{color_title}Dependency on task %s was > > > removed{color_default} with hash %s") % (clean_basepath(dep), > > > a[dep])) + > > > output.append(color_format("{color_title}Dependency on task %s was > > > removed{color_default} with hash %s") % (dep, a[dep])) if changed: > > > for dep in changed: if not collapsed: > > > - output.append(color_format("{color_title}Hash > > > for dependent task %s changed{color_default} from %s to %s") % > > > (clean_basepath(dep), a[dep], b[dep])) > > > + output.append(color_format("{color_title}Hash > > > for dependent task %s changed{color_default} from %s to %s") % > > > (dep, a[dep], b[dep])) if callable(recursecb): recout = > > > recursecb(dep, a[dep], b[dep]) if recout: >