From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7353323043876241408 X-Received: by 2002:a17:90a:680e:b0:2a2:2cbf:2144 with SMTP id p14-20020a17090a680e00b002a22cbf2144mr224083pjj.19.1712078942416; Tue, 02 Apr 2024 10:29:02 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:90a:7564:b0:2a0:4708:fd74 with SMTP id q91-20020a17090a756400b002a04708fd74ls29321pjk.0.-pod-prod-00-us; Tue, 02 Apr 2024 10:29:01 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE1h/mPy/P7JczBxT/TZgQlNrh5r+G2A3UzmEEI8cafqLxWAxk3xVNPAs0s1Hfd2blyuCDm X-Received: by 2002:a17:90b:4ccd:b0:2a0:78f4:2dc5 with SMTP id nd13-20020a17090b4ccd00b002a078f42dc5mr180371pjb.22.1712078941177; Tue, 02 Apr 2024 10:29:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1712078941; cv=none; d=google.com; s=arc-20160816; b=kHfsSKibzFun9/PwlZll3QtoVjqIc9N0CiCbTLbUcZVDRkeMSLVbs0siP9lqqrpJqV INq0NdQ9XJS+9XA3hSdYlNzRLJ5LD2Mut9OfmiOn4GlQmy91ZxMs1J9cqEQlZIOVHfMi 4JI0RNge/9PPL1xfpzfyzYkKPFXvYVoXCOqQRLcetBknszncz0hSP1uefMeFsrgigoLv JuUP6ghDR5Kn3nyTkN5B/ogrmIECnBgKlfjXiPdS1JHUDNegghuszlQDwgUQ3nVOQ7pA 8u5TW/BBS2iAVOM5rEnTaYkCP2vr+uSAC+Dcmfc8YnvP6wD5AH4FAH4T7x3vW8qxy11P BqQw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=pCp2/5apqyEvfPvSZ2O2t8CM9o6Ve9WNpL+a27IeD3k=; fh=oQsVBfLFKOCDpQI4v78XDUCp+c0vYPEMW+xCOFIgP4c=; b=ghj1ip9g3Ge8Qe1ZeTIoNISwZlTzeRF4mvQtj6e75YmqHvizko4Q3r8jolaoR6b5YO YM1QFy+qLq2wNwyHk9n3MqMvdR5b1lTquU7lRmMGYsuQOcRTm1bNLnIv+uVTBzSx3EgV B58WEAxLxrL91JvjEv6HFBaEC0BAm/nSsYTSRMIfmiLv4rnrIwh2KgL54LafMo9rQINm l2muG5c71ZK88gjY+xZYRngpTd7Tf7S+izCNz/YWYFkgtHnMWPka45K4hWmJ6dKW2biF /Q7RqZ6hBG8hZ8hLQKHUIpsXxfbCyZnlWQc8ozjGjhi7kt/76c/yNHOOu3VEX4qhOV0x +nEQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=gZqLPNCU; spf=pass (google.com: domain of kergoth@gmail.com designates 2607:f8b0:4864:20::102a as permitted sender) smtp.mailfrom=kergoth@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com. [2607:f8b0:4864:20::102a]) by gmr-mx.google.com with ESMTPS id d126-20020a633684000000b005f0a5113629si500562pga.3.2024.04.02.10.29.01 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 02 Apr 2024 10:29:01 -0700 (PDT) Received-SPF: pass (google.com: domain of kergoth@gmail.com designates 2607:f8b0:4864:20::102a as permitted sender) client-ip=2607:f8b0:4864:20::102a; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=gZqLPNCU; spf=pass (google.com: domain of kergoth@gmail.com designates 2607:f8b0:4864:20::102a as permitted sender) smtp.mailfrom=kergoth@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: by mail-pj1-x102a.google.com with SMTP id 98e67ed59e1d1-29f69710cbbso70557a91.1 for ; Tue, 02 Apr 2024 10:29:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712078941; x=1712683741; dara=google.com; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pCp2/5apqyEvfPvSZ2O2t8CM9o6Ve9WNpL+a27IeD3k=; b=gZqLPNCUV6/eg/jpjGa2kCXQ0VECODCGs12X9aDmMeiADEjiCseTXzk2aHnS2iCfJq 13xafcssFRjANmNdHgUZCeMtQ5ggqXty+Nsce/cZQcJG++CWHeMKAEVWjuJk5O1KuwN6 5XzKq4HuDsKOVjKNnNIlL9VPxBvVwosh18idzrIEj2uSPXrxmekEwydcGuUfFcYE5guu 2GEQjrFVYUmDtjrMsW3pxrB7EuWTt+PLCkmx3NVP455NEOS+r6/2WJpiDcv5b2aiThtM 4pQBXBWerIsjOa7MLMHWOQ+h6MmYscuvpPqSKnKfo8tIoXperJT1BOOD/02PEHatHeW2 oEAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712078941; x=1712683741; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pCp2/5apqyEvfPvSZ2O2t8CM9o6Ve9WNpL+a27IeD3k=; b=K9o01suXtzV0obOYKIwaJCJ3GYxojj41LkKkd+jMJtm4A5a4OdBhqAohcsW6flsGXR Pqk5++9MVT8fXfS4Cov4l3iW17C/bJHJyFVdmedNM61XiA9WyQcnjhB/Fy3/IZvkJsuo DbpL2mC05Qo/fnxTimnPhKS9tdBuMYKhTQ5huaW0pKE57CfDzKnBGDBdbBYyfovgTmJx 2IrXass2kEkJMYGdjSv9ZoAmRoQj7XpOVZlslow602EiC+ql5rbw8KTvKRHRYAOsx1Nx z/m13XVxS1suXS1Fdz84CwbVVw2CLAkARxLfBmbVaYZ7o8qng/qxYBBLjukvmHxHup5k /djA== X-Gm-Message-State: AOJu0YwAopN4gFG2Iok4YAX72Da6hT8THgSdYxy03Z43stHVhycr6rbX PGRd6dgdZYfEND03BwmL1+g9gRz0Ko3a99/5rAmguiPLDE6lAzzGY245Tmg1RUI= X-Received: by 2002:a17:90b:3712:b0:2a0:76b1:896 with SMTP id mg18-20020a17090b371200b002a076b10896mr224404pjb.12.1712078940778; Tue, 02 Apr 2024 10:29:00 -0700 (PDT) Return-Path: Received: from dc4670097af0.alm.mentorg.com (nat-mam.mentorg.com. [139.181.8.34]) by smtp.googlemail.com with ESMTPSA id bt19-20020a17090af01300b002a2313f62dcsm66705pjb.0.2024.04.02.10.28.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Apr 2024 10:29:00 -0700 (PDT) From: kergoth@gmail.com X-Google-Original-From: chris.larson@siemens.com To: isar-users@googlegroups.com Cc: Christopher Larson , Christopher Larson Subject: [PATCH 4/9] isar-sstate: lint: add support for checking stamps Date: Tue, 2 Apr 2024 17:28:08 +0000 Message-Id: <20240402172813.418770-5-chris.larson@siemens.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240402172813.418770-1-chris.larson@siemens.com> References: <20240402172813.418770-1-chris.larson@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TUID: 7qi3MPvWP1a5 From: Christopher Larson Bitbake supports writing signature data directly to the stamps directory without having to build, so we should add the ability to lint this signature data as well. This is useful for checking for cachability issues without having to complete a build. Signed-off-by: Christopher Larson --- scripts/isar-sstate | 34 +++++++++++++++++++++++----------- 1 file changed, 23 insertions(+), 11 deletions(-) diff --git a/scripts/isar-sstate b/scripts/isar-sstate index 9b20cb8e..b77f73eb 100755 --- a/scripts/isar-sstate +++ b/scripts/isar-sstate @@ -154,10 +154,12 @@ SstateCacheEntry = namedtuple( # "${PV}:${PR}:${SSTATE_PKGARCH}:${SSTATE_VERSION}:" # This regex extracts relevant fields: -SstateRegex = re.compile(r'sstate:(?P[^:]*):[^:]*:[^:]*:[^:]*:' +SstateRegex = re.compile(r'(.*/)?sstate:(?P[^:]*):[^:]*:[^:]*:[^:]*:' r'(?P[^:]*):[^:]*:(?P[0-9a-f]*)_' r'(?P[^\.]*)\.(?P.*)') - +StampsRegex = re.compile( + r"(.*/)?(?P[^/]+)/(?P[^/]+)/([^/]+)\.do_(?P[^/]+)\.(?Psigdata)\.(?P[0-9a-f]{64})" +) class SstateTargetBase(object): def __init__(self, path, cached=False): @@ -288,12 +290,13 @@ class SstateTargetBase(object): class SstateFileTarget(SstateTargetBase): - def __init__(self, path, **kwargs): + def __init__(self, path, regex=SstateRegex, **kwargs): super().__init__(path, **kwargs) if path.startswith('file://'): path = path[len('file://'):] self.path = path self.basepath = os.path.abspath(path) + self.regex = regex def __repr__(self): return f"file://{self.path}" @@ -334,12 +337,13 @@ class SstateFileTarget(SstateTargetBase): for subdir, dirs, files in os.walk(self.basepath): reldir = subdir[(len(self.basepath)+1):] for f in files: - m = SstateRegex.match(f) + relative = os.path.join(reldir, f) + m = self.regex.match(relative) if m is not None: islink = os.path.islink(os.path.join(subdir, f)) age = int(now - os.path.getmtime(os.path.join(subdir, f))) all_files.append(SstateCacheEntry( - path=os.path.join(reldir, f), + path=relative, size=os.path.getsize(os.path.join(subdir, f)), islink=islink, age=age, @@ -592,6 +596,9 @@ def arguments(): parser.add_argument( '--exit-code', type=int, default=None, help="lint: return this instead of number of found issues") + parser.add_argument( + '--lint-stamps', default=False, action='store_true', + help="lint: assume target is a stamps directory (target must be a local path)") args = parser.parse_args() if args.command in 'upload analyze'.split() and args.source is None: @@ -798,7 +805,7 @@ def sstate_analyze(source, target, **kwargs): print('\n'.join(out)) -def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, **kwargs): +def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, lint_stamps, **kwargs): ADDITIONAL_IGNORED_VARNAMES = 'PP'.split() # only list non-cacheable tasks here # note that these still can break caching of other tasks that depend on these. @@ -809,7 +816,10 @@ def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, ** print(f"WARNING: target {target} does not exist. Nothing to analyze.") return 0 - cache_sigs = {s.hash: s for s in target.list_all() if s.suffix.endswith('.siginfo')} + if lint_stamps: + cache_sigs = {s.hash: s for s in target.list_all()} + else: + cache_sigs = {s.hash: s for s in target.list_all() if s.suffix.endswith('.siginfo')} hits_srcdir = 0 hits_builddir = 0 @@ -891,10 +901,12 @@ def main(): target = SstateDavTarget(args.target) elif args.target.startswith('s3://'): target = SstateS3Target(args.target) - elif args.target.startswith('file://'): - target = SstateFileTarget(args.target) - else: # no protocol given, assume file:// - target = SstateFileTarget(args.target) + else: # Either file://, or no protocol given, assume file:// + target = SstateFileTarget(args.target, StampsRegex if args.lint_stamps else SstateRegex) + + if args.lint_stamps and not isinstance(target, SstateFileTarget): + print("ERROR: --lint-stamps only works with local file targets") + return 1 args.target = target return globals()[f'sstate_{args.command}'](**vars(args)) -- 2.39.2