From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 10 Apr 2025 11:17:40 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wr1-f55.google.com (mail-wr1-f55.google.com [209.85.221.55]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 53A9Hdqi010446 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Apr 2025 11:17:39 +0200 Received: by mail-wr1-f55.google.com with SMTP id ffacd0b85a97d-39126c3469fsf203988f8f.3 for ; Thu, 10 Apr 2025 02:17:39 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1744276654; cv=pass; d=google.com; s=arc-20240605; b=aCzZyW1/IisBJCPJEzoRhcIUXwtjjzc4M2tHLF14Fmx28y3HthRt8jlFlHUq1gMws5 CW2RhGFudqmAami66FtWkVLPwCtuVyjLkNKdTS7ip83Y/d5ujiX0BN5BI8ZoX4Pwa9A8 /YRulRLlLtfRIrV5LsMmqoFj79gyCO7lwgeK38PNa+HAWSQPV6zo9IDtPb2pAk/shqM/ 2rHAnWQ86M1iwnoHM/PeJOUjJYI2mruwm9DFmOpdEYnKiYey/qba+v2I5gt66g6I7H8e nbnnscIOkz02H6Na7nL4o/Av1M8R0Jzh2Joz3mr6XCttC46e8KTn1ZNPlgMt+F83raOK RHKw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=4iFpelOkEqvmcz3vYcn9jTb9tnaYovBUpGFiR3BAbVw=; fh=CMhBQqejffwxOm3kkeTugNMifxBm0vNctwazu8rymSw=; b=XG4YThOYg12y5xxT2Sxfh/8js+26rT1ucM4y8ObIjJEjX77Lu2T2koIsHLyg28MlMx y+myYoUAHW83GJezysLbiRbnaVM6j2lt2uv0jAXOVX1uMzGTzlLofRbNlORiHrkvzckX Fhth51seOnJ4LO/U4jmuqsK8Te60oDV3dp42t9XBU3UTiEg4v2XT1lmTL/nRBm3FpwER o5/5s0zDQ5qiu6567yCycy4imDZ+k99t9MeYhepg4MGvWiN4JvnnMpz54A5ncMq1YBLm U+Z+tApron05ebvRAck/Mz6D6HT9o62mgvFKhUI+xeA8/ynqXSe3LSsBKxgJDu3McpAI 4FiA==; darn=ilbers.de ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=MVV8y2Vk; spf=pass (google.com: domain of fm-1321639-20250410091729601c087d8b3d26f65a-_0saek@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1321639-20250410091729601c087d8b3d26f65a-_0sAeK@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1744276654; x=1744881454; darn=ilbers.de; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=4iFpelOkEqvmcz3vYcn9jTb9tnaYovBUpGFiR3BAbVw=; b=E8DD+JuE4NB/vhj8ijOm5AfbHjRhfmQqP6yNhC+1mU+LiL9c9xNEzKeem4OsRmHi4D NGFLAa0O9tgHrDXtH6u28S73tho/HRt05x3d67wtAWT8Kg07k4JCgNMJ+4wNipBG948F COgj0qiFTDceFBmDcc3xYOqzl2Vh78hz1Er504vCpSNOKE0wZjNHwvfPwqAsP2e/hI3z cyY6yfoepU1zZPxjIfP9cYL7qYU6zR8sPnzLNTkeQBTBTUcCe/J0wYnz4e4MiILikpaP X3Yagv9zT/okxduHm60fTUihz2LIttX1wxO3awxj3WTfD/Fa7wto/vI8DoQoIoLns4LH KsCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744276654; x=1744881454; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4iFpelOkEqvmcz3vYcn9jTb9tnaYovBUpGFiR3BAbVw=; b=YC1O43zDeZ6ffDuc+ZF92e2MrIosBbJTR1o0xiXzjrjlwgmSxcm5dBFnPlvPRKJpuf ADHz3zzn09Ex82mdfRCMMO4Zv6qHJMvt16C29Bowg5cjER7Ii4cHoUJFS8GvObuisqJk tWNTgLAYf0GYYVLbFo1l7+Iq6f6SVqrPfFyieTokuLgqvwGcX9bH411wevyaFbZCr57+ Gt9wmPWLTIUhMAG3RP3nfOgBb8kcQmbxMaEJZAcg7AKaXyntdsNW+2bssRNsarUJzQwu hlPVsQLxk50lxqAqFjYiA0nAj6mB0IhUgl/BA7DX3bYOGrn52rs16VhFZvF5Vmptb2Iy NUiA== X-Forwarded-Encrypted: i=2; AJvYcCVqO56tcgfWwZyTxZe3r4wWE0iDKH7OdWbq7PI93jyGdF07gpjdjnj67fkE+5TD0m1lAh8W@ilbers.de X-Gm-Message-State: AOJu0YzMfHHwm86x4mAmOKZ09g1KjFrb/KFvx941k9nVXQX3ssrkvHMc 8BsQSA4Eq4EFu5HcAn3Et8bNPHpm5ad8mRg3F8VAGu1G0SimOwg+ X-Google-Smtp-Source: AGHT+IEH2xKciOBRxX2XQlIkG+9jp+p5v6sxErC9RvjLu40gv6GaAqcsUSxNFt0Rfb43KmwBKRQgqA== X-Received: by 2002:a5d:5983:0:b0:38d:cf33:31d6 with SMTP id ffacd0b85a97d-39d8f27612emr1651199f8f.3.1744276652996; Thu, 10 Apr 2025 02:17:32 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h=ARLLPAKqfLgTBjs7SMFESNlgIbeqs4K9HovkQbPomHYo7oB/kQ== Received: by 2002:a05:600c:1d23:b0:43d:40b0:33 with SMTP id 5b1f17b1804b1-43f2c24f86als2056715e9.0.-pod-prod-01-eu; Thu, 10 Apr 2025 02:17:30 -0700 (PDT) X-Received: by 2002:a05:600c:3c91:b0:43b:cc3c:60bc with SMTP id 5b1f17b1804b1-43f2d7d6f10mr23615935e9.15.1744276649921; Thu, 10 Apr 2025 02:17:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1744276649; cv=none; d=google.com; s=arc-20240605; b=M3iZEDsHx/38FizU34wmzo5aKQQdEh00FrbaiH5kV/IraYfpRNIajqYoCnz0zkCLf7 bcZY9Rg3u5mMhBOE3OMyWBhuzZGOLJp2QEQ+Hd5MtcC9GwberrH77o9/+5xXoa+eyHST COmWnrGnG9pwbM/uLhogdCnN1Dq0UnGU8z0RjdtTae2JY9lbcuuQIqUxE+/jMcv1cX6S BjVAFT2ATc56AmVhVbzPybIDh+fjcjDbMRZSzg0vhEz8eKnKNq5CAgFvG6lMDod1NR4e Jv3NwSEQuGHxMEZSSap1I+ZMqK1HHcJtEnTJL61eS9RJJ5dG0Ws4fJG+bWeDyhUBTlpY kFpg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=MaNfNMIBig9Ef9e6U7xXxSwvqtqyzJ8O2h9d6hryhcE=; fh=OYA7Xj8Fgh6stKMJ6XJH0QE9OZQModMu4f2HyOuPUjs=; b=X4RA8RXRK8B/nE/8AlVIl7TJb1rXAYHghJIs3FAuxRdh2VwVT2nLs/d3d465BuHMGB b3l/hRtZ8sja0xrCmRaU12T9+4PHm8jhEIUsfvH0CSsazvuGtYBZ1ClOe7hJJ3idb5HC LOKsRz6vJbgRQIEmakvtPYBVOFexuekYNOZw+vV6jDRP/FWtdSgtyOfjAyf/W3lbl4kA znbryCx9zG6pGkFHIuad4yQF3EGH1OJt3LsBcyiNnaid5h+yjMJD3mT0ZzefGeeMiZK8 zzdj6pmIqO4WOMg2mc+ERr/3Yir1v2RYwQBZMq+KBwCRe9FZmfwTL+QUMLRGMsjlbb4d 1t7Q==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=MVV8y2Vk; spf=pass (google.com: domain of fm-1321639-20250410091729601c087d8b3d26f65a-_0saek@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1321639-20250410091729601c087d8b3d26f65a-_0sAeK@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-65-226.siemens.flowmailer.net (mta-65-226.siemens.flowmailer.net. [185.136.65.226]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-43f208e8436si2042235e9.0.2025.04.10.02.17.29 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 10 Apr 2025 02:17:29 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1321639-20250410091729601c087d8b3d26f65a-_0saek@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) client-ip=185.136.65.226; Received: by mta-65-226.siemens.flowmailer.net with ESMTPSA id 20250410091729601c087d8b3d26f65a for ; Thu, 10 Apr 2025 11:17:29 +0200 From: "'Felix Moessbauer' via isar-users" To: isar-users@googlegroups.com Cc: Felix Moessbauer , adriaan.schmidt@siemens.com Subject: [PATCH 1/1] isar-sstate: add filter parameter Date: Thu, 10 Apr 2025 11:16:45 +0200 Message-Id: <20250410091645.3840431-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-Original-Sender: felix.moessbauer@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm2 header.b=MVV8y2Vk; spf=pass (google.com: domain of fm-1321639-20250410091729601c087d8b3d26f65a-_0saek@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1321639-20250410091729601c087d8b3d26f65a-_0sAeK@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Felix Moessbauer Reply-To: Felix Moessbauer Content-Type: text/plain; charset="UTF-8" Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: c6jMByDl3cb+ Currently all sstate operations are always performend on the whole cache. This is problematic if some files should not be uploaded or a clean operation should only be applied to some files. We now add the filter parameter, which can be used in combination with any isar-sstate command. By that, the operation is limited to the files matching the filter. Negative matches are also supported. Signed-off-by: Felix Moessbauer --- scripts/isar-sstate | 35 ++++++++++++++++++++++++++++++----- 1 file changed, 30 insertions(+), 5 deletions(-) diff --git a/scripts/isar-sstate b/scripts/isar-sstate index 64511c44..8206a28e 100755 --- a/scripts/isar-sstate +++ b/scripts/isar-sstate @@ -31,6 +31,7 @@ and supports three remote backends (filesystem, http/webdav, AWS S3). The `upload` command pushes the contents of a local sstate cache to the remote location, uploading all files that don't already exist on the remote. +If some file should not be uploaded, use a negative filter like '(?!sbuild)'. ### clean @@ -620,6 +621,9 @@ def arguments(): parser.add_argument( '--excluded-tasks', type=str, default=DEFAULT_IGNORED_TASKS, help="lint: comma-separated list of tasks to ignore (default: %(default)s)") + parser.add_argument( + '--filter', type=str, default=None, + help="lint: filter tasks by PN (regex, default: all)") args = parser.parse_args() if args.command in 'upload analyze'.split() and args.source is None: @@ -632,13 +636,15 @@ def arguments(): return args -def sstate_upload(source, target, verbose, **kwargs): +def sstate_upload(source, target, verbose, filter, **kwargs): if not os.path.isdir(source): print(f"WARNING: source {source} does not exist. Not uploading.") return 0 if not target.exists() and not target.create(): print(f"WARNING: target {target} does not exist and could not be created. Not uploading.") return 0 + if filter: + reg_exp = re.compile(filter) print(f"INFO: uploading {source} to {target}") os.chdir(source) @@ -646,6 +652,10 @@ def sstate_upload(source, target, verbose, **kwargs): for subdir, dirs, files in os.walk('.'): target_dirs = subdir.split('/')[1:] for f in files: + if filter: + parts = f.split(':') + if len(parts) > 1 and not reg_exp.match(parts[1]): + continue file_path = (('/'.join(target_dirs) + '/') if len(target_dirs) > 0 else '') + f if target.exists(file_path): if verbose: @@ -664,7 +674,7 @@ def sstate_upload(source, target, verbose, **kwargs): return 0 -def sstate_clean(target, max_age, max_sig_age, verbose, **kwargs): +def sstate_clean(target, max_age, max_sig_age, verbose, filter, **kwargs): def convert_to_seconds(x): seconds_per_unit = {'s': 1, 'm': 60, 'h': 3600, 'd': 86400, 'w': 604800} m = re.match(r'^(\d+)(w|d|h|m|s)?', x) @@ -695,6 +705,10 @@ def sstate_clean(target, max_age, max_sig_age, verbose, **kwargs): print(f"NOTE: we have links: {links}") archive_files = [f for f in all_files if f.suffix in ['tgz', 'tar.zst']] siginfo_files = [f for f in all_files if f.suffix in ['tgz.siginfo', 'tar.zst.siginfo']] + if filter: + reg_exp = re.compile(filter) + archive_files = [f for f in archive_files if reg_exp.match(f.pn)] + siginfo_files = [f for f in all_files if reg_exp.match(f.pn)] del_archive_files = [f for f in archive_files if f.age >= max_age_seconds] del_archive_hashes = [f.hash for f in del_archive_files] del_siginfo_files = [f for f in siginfo_files if @@ -712,13 +726,16 @@ def sstate_clean(target, max_age, max_sig_age, verbose, **kwargs): return 0 -def sstate_info(target, verbose, **kwargs): +def sstate_info(target, verbose, filter, **kwargs): if not target.exists(): print(f"WARNING: cannot access target {target}. No info to show.") return 0 print(f"INFO: scanning {target}") all_files = target.list_all() + if filter: + reg_exp = re.compile(filter) + all_files = [f for f in all_files if reg_exp.match(f.pn)] size_gb = sum([x.size for x in all_files]) / 1024.0 / 1024.0 / 1024.0 print(f"INFO: found {len(all_files)} files ({size_gb:0.2f} GB)") @@ -753,7 +770,7 @@ def sstate_info(target, verbose, **kwargs): return 0 -def sstate_analyze(source, target, **kwargs): +def sstate_analyze(source, target, filter, **kwargs): if not os.path.isdir(source): print(f"WARNING: source {source} does not exist. Nothing to analyze.") return 0 @@ -768,6 +785,10 @@ def sstate_analyze(source, target, **kwargs): key_tasks = 'dpkg_build rootfs_install bootstrap'.split() + if filter: + reg_exp = re.compile(filter) + local_sigs = {k: v for k, v in local_sigs.items() if reg_exp.match(v.pn)} + check = [k for k, v in local_sigs.items() if v.task in key_tasks] for local_hash in check: s = local_sigs[local_hash] @@ -828,7 +849,7 @@ def sstate_analyze(source, target, **kwargs): def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, lint_stamps, - excluded_tasks, **kwargs): + excluded_tasks, filter, **kwargs): ADDITIONAL_IGNORED_VARNAMES = 'PP'.split() # only list non-cacheable tasks here # note that these still can break caching of other tasks that depend on these. @@ -844,6 +865,10 @@ def sstate_lint(target, verbose, sources_dir, build_dir, exit_code, pedantic, li else: cache_sigs = {s.hash: s for s in target.list_all() if s.suffix.endswith('.siginfo')} + if filter: + reg_exp = re.compile(filter) + cache_sigs = {k: v for k, v in cache_sigs.items() if reg_exp.match(v.pn)} + hits_srcdir = 0 hits_builddir = 0 hits_other = 0 -- 2.39.5 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250410091645.3840431-1-felix.moessbauer%40siemens.com.