From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7353323043876241408 X-Received: by 2002:a17:90a:fb48:b0:2a4:9836:aa2c with SMTP id iq8-20020a17090afb4800b002a49836aa2cmr167198pjb.28.1712334746780; Fri, 05 Apr 2024 09:32:26 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:90a:c20e:b0:2a2:9d0f:c728 with SMTP id e14-20020a17090ac20e00b002a29d0fc728ls1362034pjt.2.-pod-prod-06-us; Fri, 05 Apr 2024 09:32:25 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFLZ1h1qLoMP63bRCxdNEk5JeA84tmyvxnhuoAzESrzizeaq++fjwLbJLV9YccjZtLPgdcs X-Received: by 2002:a17:902:bc46:b0:1e2:3db0:1d84 with SMTP id t6-20020a170902bc4600b001e23db01d84mr1806315plz.32.1712334745446; Fri, 05 Apr 2024 09:32:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1712334745; cv=none; d=google.com; s=arc-20160816; b=t+4QURxjPXn2YGFsP5Kl8K5b1qcl/SkcKHh732OB46X5FkosUCXEMR9zyIXb6+Z1/y j8Q/jw4AQwb+rsPV1/FwtBh73p7P0H54e5LBc8sRv+v620O8qQFfRFwDSG5kb275Jg3J NIfL/QtPJ2HZluuXSp8Nw2ehKv/Ze51N7K/a5ltV+jjtyE/5E2gbAGvLuzztGLGQyryh 4JdWWJ80f5ulycaXbjW3PcP4Dnay10VFdR9n0QG8jML7YZgb8qPXMt4HFnupB2Y9C1Rg xqCccIrSYHbWpX5TvN3WYB8XaQJTrEmsKhlw96U8y0/Ut3F2s/XJZ1b0nTKz91uGAc/G i+IA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=lHo56J9PyTOzEGKYUsZ78jlLZiPMDdfI577GnBCiu5k=; fh=3bh9z2BWAzGAc1byRhMi0K+arMZmX5BQHE0sOxINdyU=; b=myyJkOFpUt/V7q38T4BnnTWmxjfJtAuMQyZT8XvjVNEzd5aPAI+on+EUczMnZXnsD3 dG1etMUc2HZCc79bo48LkJgqFEolv95XsJlHJUmJ8/RqLXNBPElWTconrO/TMm38Bsiq wO0+gRwU8GV0ws2KnUyQ5UkX/XRqYDPBecrzEBaAIcSwFO3UrgDZBi64B6Qz5aQQPxH0 B7lmuSUdZyfsr4+T3JqFLvtAN78aIT6DQMZgNHFISyEB4ESxZJBJVbQc/kJW+h5MEgpx sxuwfVrzT2NbyXrOEK/PyGoZJ9ubK9TK7KsY5Fzey0zvvSqa63UBKs8/hHahksF8MkoX 1f/Q==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=lCH2YKi5; spf=pass (google.com: domain of fm-1325773-2024040516322481f15bbe14d093a36e-wq6imq@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1325773-2024040516322481f15bbe14d093a36e-wQ6IMq@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Return-Path: Received: from mta-65-226.siemens.flowmailer.net (mta-65-226.siemens.flowmailer.net. [185.136.65.226]) by gmr-mx.google.com with ESMTPS id jc18-20020a17090325d200b001dd61b4ef8esi97963plb.12.2024.04.05.09.32.25 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 05 Apr 2024 09:32:25 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1325773-2024040516322481f15bbe14d093a36e-wq6imq@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) client-ip=185.136.65.226; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=lCH2YKi5; spf=pass (google.com: domain of fm-1325773-2024040516322481f15bbe14d093a36e-wq6imq@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1325773-2024040516322481f15bbe14d093a36e-wQ6IMq@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: by mta-65-226.siemens.flowmailer.net with ESMTPSA id 2024040516322481f15bbe14d093a36e for ; Fri, 05 Apr 2024 18:32:25 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=chris.larson@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=lHo56J9PyTOzEGKYUsZ78jlLZiPMDdfI577GnBCiu5k=; b=lCH2YKi5hEWdlO0kFceYIYYB26GeKbXq4VPNGBBbS+jv+WhyvWtiGAII1tRo+7IT45kwpJ ibG9FHE7nB3r2llDOwKPz0phtwRymHe37t4ExlnrTAG36WIm6zaDwHTqFQydpvctduo9Lrhr Ln9ll6CvZtRf/yTeBOVlX25yUWmP8=; From: chris.larson@siemens.com To: isar-users@googlegroups.com Cc: Christopher Larson Subject: [PATCH 4/9] isar-sstate: lint: add support for checking stamps Date: Fri, 5 Apr 2024 16:31:30 +0000 Message-Id: <20240405163135.2987489-5-chris.larson@siemens.com> In-Reply-To: <20240405163135.2987489-1-chris.larson@siemens.com> References: <20240405163135.2987489-1-chris.larson@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1325773:519-21489:flowmailer X-TUID: a7iOCuGRADXz From: Christopher Larson Bitbake supports writing signature data directly to the stamps directory without having to build, so we should add the ability to lint this signature data as well. This is useful for checking for cachability issues without having to complete a build. Submitted at https://groups.google.com/g/isar-users/c/2NB-PXyswq8/m/GMuGk61TAQAJ. Signed-off-by: Christopher Larson --- scripts/isar-sstate | 51 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 12 deletions(-) diff --git a/scripts/isar-sstate b/scripts/isar-sstate index 9b20cb8e..d68b8938 100755 --- a/scripts/isar-sstate +++ b/scripts/isar-sstate @@ -66,8 +66,16 @@ format as `bitbake-diffsigs`. The `lint` command searches for common flaws that reduce the cachability of a layer, e.g., signatures containing absolute paths from the host -environment. Issues found this way usually indicate errors in recipes -or in Isar itself. +environment. Issues found this way usually indicate errors in recipes or +in Isar itself. + +The `lint` command may be used to check a shared state cache folder, or, +with the use of the --lint-stamps argument, will check the tmp/stamps +folder and expects this as an argument. The signature data files must be +populated in the stamps folder, which requires either a completed build +or a `bitbake --dump-signatures=`. The value for this argument may be +`none`, `printdiff`, or others defined in the handler, but `none` is the +simplest to just generate signature data. ## Backends @@ -154,10 +162,19 @@ SstateCacheEntry = namedtuple( # "${PV}:${PR}:${SSTATE_PKGARCH}:${SSTATE_VERSION}:" # This regex extracts relevant fields: -SstateRegex = re.compile(r'sstate:(?P[^:]*):[^:]*:[^:]*:[^:]*:' +SstateRegex = re.compile(r'(.*/)?sstate:(?P[^:]*):[^:]*:[^:]*:[^:]*:' r'(?P[^:]*):[^:]*:(?P[0-9a-f]*)_' r'(?P[^\.]*)\.(?P.*)') +# The filename of stamps and associated signature data is defined in +# Isar (ith an added suffix of `..sigdata.` by BitBake): +# STAMPS_DIR ?= "${TMPDIR}/stamps" +# STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/${PV}-${PR}" + +# This regex extracts relevant fields: +StampsRegex = re.compile( + r"(.*/)?(?P[^/]+)/(?P[^/]+)/([^/]+)\.do_(?P[^/]+)\.(?Psigdata)\.(?P[0-9a-f]{64})" +) class SstateTargetBase(object): def __init__(self, path, cached=False): @@ -288,12 +305,13 @@ class SstateTargetBase(object): class SstateFileTarget(SstateTargetBase): - def __init__(self, path, **kwargs): + def __init__(self, path, regex=SstateRegex, **kwargs): super().__init__(path, **kwargs) if path.startswith('file://'): path = path[len('file://'):] self.path = path self.basepath = os.path.abspath(path) + self.regex = regex def __repr__(self): return f"file://{self.path}" @@ -334,12 +352,13 @@ class SstateFileTarget(SstateTargetBase): for subdir, dirs, files in os.walk(self.basepath): reldir = subdir[(len(self.basepath)+1):] for f in files: - m = SstateRegex.match(f) + relative = os.path.join(reldir, f) + m = self.regex.match(relative) if m is not None: islink = os.path.islink(os.path.join(subdir, f)) age = int(now - os.path.getmtime(os.path.join(subdir, f))) all_files.append(SstateCacheEntry( - path=os.path.join(reldir, f), + path=relative, size=os.path.getsize(os.path.join(subdir, f)), islink=islink, age=age, @@ -592,6 +611,9 @@ def arguments(): parser.add_argument( '--exit-code', type=int, default=None, help="lint: return this instead of number of found issues") + parser.add_argument( + '--lint-stamps', default=False, action='store_true', + help="lint: assume target is a stamps directory (target must be a local path)") args = parser.parse_args() if args.command in 'upload analyze'.split() and args.source is None: @@ -798,7 +820,7 @@ def sstate_analyze(source, target, **kwargs): print('\n'.join(out)) -def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, **kwargs): +def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, lint_stamps, **kwargs): ADDITIONAL_IGNORED_VARNAMES = 'PP'.split() # only list non-cacheable tasks here # note that these still can break caching of other tasks that depend on these. @@ -809,7 +831,10 @@ def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, ** print(f"WARNING: target {target} does not exist. Nothing to analyze.") return 0 - cache_sigs = {s.hash: s for s in target.list_all() if s.suffix.endswith('.siginfo')} + if lint_stamps: + cache_sigs = {s.hash: s for s in target.list_all()} + else: + cache_sigs = {s.hash: s for s in target.list_all() if s.suffix.endswith('.siginfo')} hits_srcdir = 0 hits_builddir = 0 @@ -891,10 +916,12 @@ def main(): target = SstateDavTarget(args.target) elif args.target.startswith('s3://'): target = SstateS3Target(args.target) - elif args.target.startswith('file://'): - target = SstateFileTarget(args.target) - else: # no protocol given, assume file:// - target = SstateFileTarget(args.target) + else: # Either file://, or no protocol given, assume file:// + target = SstateFileTarget(args.target, StampsRegex if args.lint_stamps else SstateRegex) + + if args.lint_stamps and not isinstance(target, SstateFileTarget): + print("ERROR: --lint-stamps only works with local file targets") + return 1 args.target = target return globals()[f'sstate_{args.command}'](**vars(args)) -- 2.39.2