From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7198551104298156032 X-Received: by 2002:a5d:6581:0:b0:2c5:49f0:e329 with SMTP id q1-20020a5d6581000000b002c549f0e329mr205720wru.357.1676056591474; Fri, 10 Feb 2023 11:16:31 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:1d06:b0:3cf:72dc:df8 with SMTP id l6-20020a05600c1d0600b003cf72dc0df8ls4870009wms.0.-pod-canary-gmail; Fri, 10 Feb 2023 11:16:30 -0800 (PST) X-Google-Smtp-Source: AK7set+EiFMICRlMT2iqTweDcylc4vxXHkuEDARa0KVmh38pdSyWMh8aPps6Ehvk+X4Qf2vhA1uz X-Received: by 2002:a05:600c:43d2:b0:3da:f5d1:c677 with SMTP id f18-20020a05600c43d200b003daf5d1c677mr13787603wmn.35.1676056590227; Fri, 10 Feb 2023 11:16:30 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1676056590; cv=pass; d=google.com; s=arc-20160816; b=F/fcv3e++BRxg2+efZw6imLpbu0Teu5rSmlhBCkHylqQWNQesXxspCSxDa20cf96gP mMlc3LYmz/9voGgrlKe4vxZSkp9Zzj3IDx4scgWoJDUuMKouqQ8SDUqjM1Nuch1y1nWp jxAXC5tXOCmLJfBbuLh3zor9WmpbB9buzB8s+qUqwKhdfCoSCa5WaeKg8iFsdbFFaxK6 wWzi1ASAC7a8JSqphSatzN5lDOsYdFbT7u7bGDbQavbI0gHK10dxNWv8PkjnBwmsPGRJ 2FVRmrloGy1O5gl+h2O95mJl8mM+f5WecGgOxv1qzPhzPUwgxDFuXKhk89GfsTeDLEEL LdvA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:references:in-reply-to :message-id:subject:cc:to:from:date:dkim-signature; bh=0+lw/yivlQsvx+FwTcd0aRf8rp8Un802irtz4wQ8xf8=; b=Vv6XlKkHvroiIhHep0ugxAAZooxvFjn5xecv1GcvJ0pWp1fUt0JlalEauqyJgHJ4s2 wNjZwyUUa2gECNc+3GxUB25LywX0LUV9IPNbRx2hN4OvUTcKf5s1PXZHAzxcLKpxADzd lteYKE3LHIA9+NFuuy3DsgVK0rwQnXb87otI0pU0xw14i7obAPLGWQp8liynHy1qA64s 9pAQvNiyIm5eIPmsvKgOViFP/Egs+CsjGGYL2falKZfqiS2lVQIE05Oq+MjPAvSWgDzR e+CMB2sEJ9tpc1VV523ZSjd346hLtTRKjvEjK546JnrHdN+QBnM8LuMMgYlhpECO0mC1 14tA== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=zFVx6WFK; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe1a::616 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR03-DBA-obe.outbound.protection.outlook.com (mail-dbaeur03on20616.outbound.protection.outlook.com. [2a01:111:f400:fe1a::616]) by gmr-mx.google.com with ESMTPS id bg31-20020a05600c3c9f00b003d9c73c820asi664729wmb.3.2023.02.10.11.16.29 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 10 Feb 2023 11:16:30 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe1a::616 as permitted sender) client-ip=2a01:111:f400:fe1a::616; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=zFVx6WFK; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe1a::616 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eUk9OEcZQwnKvfhhON0BRAt13VgYJEFGgQ1oc1ShPBWFLA2f5AL8187Nx53WvTgss9k4r1oRIjaJYtw+rkvMAH6aekk3JuAeHYlXl63YGdUzqe1Bv8ClAV7gsIkABetdnNHsX5DfAXZHsZsXiIxmmdyWcqVcWf/kxssUZuxsdeE3V/ItILB6jb1kus2loLCrtiSNra4pEet/Q3zvitMzz3O134bbDeV5BuMGDdgodsPDq3V+16fNs1xtXJdfYrj25QGy+c4BLhqGAIeRhF2cOHrOKjwkRb0E9jr0x/i9NECGhha3ez2usEXFvzR21DkWriw8+MY/XsyPd82HJCmnRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0+lw/yivlQsvx+FwTcd0aRf8rp8Un802irtz4wQ8xf8=; b=Q1fG5dfCiYeJcq7CxJBZBimjk6tCHA6pvQdA72fRByZ2lcxPpXCbYURS9cghpyhq8ppXO9C+olIv+2n0RHiLHvVwjYItPFjms9QxaDCl/E2YlYU5SaJFaVGYj6iWD2BmTrip/0K9ZdkXxUYFL6uwyKAA8mTd3W7DBygGBupdL/SaJpU8WdgClzm9vN8fIfTb7k+O1ZDMq2zFzS3OUZdXeceZPafppOEifXrxfmM2zcGuqqt9Og+CVBdENpyJyv23xkD5oUHKMlfJfvYGvmhxE1gdsuQY8QiO0QQRuKiNTNvzUCTUj7Z48EO4fwVh5XheVHqzlE9sYQPxC/+xz+04Sw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0+lw/yivlQsvx+FwTcd0aRf8rp8Un802irtz4wQ8xf8=; b=zFVx6WFK1/TnDpz9cvYRcCnwxuGvE+ReXC04uPyRLn/Sv1aXfEQbU+5ZVLle96hPtKwcLA1BBpz1jhl6snR9r5RSaMUgMFZ7l2jNwd+xqAl4SvWIgGfGZS1FZsVFHIeO5Q/cImOaL11ADT7hiFew/0wq1RyYPYt/YJrkUFRcoZCpTZ2zVkhgjdWX4+Fo182AqKtwYRSl0Mys6hTXVXSObnoZ9QmO4SPEzhYLqoknJ6wap+6PQeGWBY0+4FRGdjRRzPBVT3vON3DFViaAXuB2PfTbJCu1LM2iCHXEAwoBgOnEzeSc9LIzns+zKLt2Ojufy1jJAPG5FO68+Ldsf1nj0A== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:269::8) by PAWPR10MB7817.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:34e::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.36; Fri, 10 Feb 2023 19:16:29 +0000 Received: from PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM ([fe80::ee44:171b:4c40:d42c]) by PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM ([fe80::ee44:171b:4c40:d42c%6]) with mapi id 15.20.6086.020; Fri, 10 Feb 2023 19:16:28 +0000 Date: Fri, 10 Feb 2023 20:16:25 +0100 From: Henning Schild To: Adriaan Schmidt Cc: isar-users@googlegroups.com Subject: Re: [PATCH] fix(isar-sstate): also handle zst files Message-ID: <20230210201625.4fb0b4d5@md1za8fc.ad001.siemens.net> In-Reply-To: <20230210153434.1024604-1-adriaan.schmidt@siemens.com> References: <20230210153434.1024604-1-adriaan.schmidt@siemens.com> X-Mailer: Claws Mail 4.1.0 (GTK 3.24.35; x86_64-pc-linux-gnu) Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-ClientProxiedBy: FRYP281CA0008.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10::18) To PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:269::8) Return-Path: henning.schild@siemens.com MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PA4PR10MB5780:EE_|PAWPR10MB7817:EE_ X-MS-Office365-Filtering-Correlation-Id: f3fd544d-ce6c-47e5-4dfb-08db0b9b4f6f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Z1BWcseHpVpiT6N1Q3Xqsb9ClAz9KfBPveJ2Ky0nVHAdTG8SW43kBMZRq4j+h+yLiOBjaDixWl2EdVuqvRsF9yddLfmUwHvxe7VOLlvkin2fW6AZtRPwsZjHcn9H3f49m/55YC4ft+3afjJHpWt2dp0/M+SsygC0wZRrNJH8EOrVnNy3GBTnlF1BdgMtQjLe9oCP46WYIWTBGPv/ic9E1kCAQS/s81Ot0b4+TUyZ3S1G+2kxxJoRwAhgnnLIk1iajtoUNYwdsevOs9FC/pJzn1i32CDG8yKaUB9NdHucrjjUXDZJ2i17x9D/dkNGGf7KUZ61/F4PHhwspt0w2JwtwTlbmnGHJpVj8B84bGetRlHUwkiGVxiYnzYXfRwK55FAjisDAv4/iJ3ZtDLUblJ+gknn4cpoopt7NgLFz2AkiOx+Ipw6IldxRLQ7kgrNUYM0KTRLSzjkj4dbFfUNzMpAU/kzPdfTFfTzcaB7ysYfN24FCfw5Bo3hYBgMSv06IXjBY0bN6D9W7qVDqrfl63PAJMwy4DLQYj0/2e1d5AmFbXpV4YHp5AHVY4gO7rqlXntwwlSbu3op2JWVr5YpXKBFF2xbEetsCZebj1o41ek7cM/gZhnBiI5gO0Gz6hUP7Zo8kOhhJqM/NvxDc9gilNA7YA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230025)(4636009)(396003)(39860400002)(136003)(376002)(366004)(346002)(451199018)(316002)(6666004)(6486002)(83380400001)(82960400001)(6636002)(478600001)(4326008)(8676002)(66946007)(66476007)(66556008)(38100700002)(6862004)(8936002)(5660300002)(41300700001)(6506007)(1076003)(2906002)(9686003)(186003)(44832011)(6512007)(86362001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?4zLRRKdDVkp4XK7Dg4N5M4thrA/WofLuTdgVRkKTdMfAG4oErdmnBOGNIJYw?= =?us-ascii?Q?JIjYafh/bjaNbANOKinLv/TZzCIvmWNA2VcbC2BkRbOWJ8vOSuUai8IgOerh?= =?us-ascii?Q?CeHrfpS6EiatBbDh/+wsUdSo4KlhWUlbQ5/PY249RltzWdhwCTacpFJDvKzz?= =?us-ascii?Q?pn89s7BgLjpUETUG84BfiI5pKUi7+l+psKxPyHTxFtU1i/hHwjyWytGuszQ8?= =?us-ascii?Q?pqVt3564GLAWtbWG77WyN6fvL7rGcH2n3ASwgPnk0/InVFaiKZaivxcYX5Yr?= =?us-ascii?Q?i8h3QHzKJXwgRNDuwOIO+bGEdlBqIXPXV/ruZkRv4u1PPeLvg43VQ7JOlxVD?= =?us-ascii?Q?UnmgeXWLUAYoCPkZ55MS1gZQ9JPdmgKT3ZBUbtwzm+L9dYhQllk9vCdJD6Ye?= =?us-ascii?Q?LC+gHGAPDCb+tJktCBsPbVeloiTHXGd+5zVKIRGGxUZRGT8q7XjuhGq+90vk?= =?us-ascii?Q?r09BP9w1Njmud0Dn+6sggScULemtcpTBU/DVp4JRoVxc7ajTHtVAQ3Tf6xA0?= =?us-ascii?Q?wZFaEFipfHionWFdWNdvtphXD/0hl4XQIUYQJQhtsgGcJ2mykCHUVmoE0RTS?= =?us-ascii?Q?noOwYgS+ybeDhcsKWvR8kwEOfncsIlgMaqjOxEWxcZ4cIIA2OMbACyvs8ngp?= =?us-ascii?Q?br7Tmx6gRtma9158dGhu3oSxSYTn3AdeZOui8ZaUFpLYPmzZ4L9IG53tGGfP?= =?us-ascii?Q?gWsTjUeFE2Jr4HC8sZIadPylAhIysMVCG6gHo12nGiZb7zh2jYi3lrhVblZd?= =?us-ascii?Q?qiM0B0Fa7mA41tMH2Oey5XRCsOBU6mS+5LRNjUgVfioOMRsFt7At4hafd/wQ?= =?us-ascii?Q?gfkbLpB4ATI8pdpZJEcYNZKYdP1knExSbyswCG02eWWTvIbNYf1JIVudmTtn?= =?us-ascii?Q?hPtQ6To5ZbUvI25wyCD2K6xAESnpt/HaJ2DdzvcKYVS00PkyjVeHZW43vK8P?= =?us-ascii?Q?80iR7yHwXHHbL351T8As0f09NpVxwsXPlo0M5neTucQ9rit6AU0QEAa23PGg?= =?us-ascii?Q?BOVVoT8ALbO6AuZg8Gm8+nSBnjOK5OLtqefbBTVDwRfOI3U+G+GYyv5ZBfuA?= =?us-ascii?Q?dkX7jn/hY1hL9nNHY31Ci3VLAbKzeWF2EoZhzXA4IHwdJyzuGxJcy3rqcUNP?= =?us-ascii?Q?lT3p4IQLzk4lk78LBZ4gPjocIu6QTpoP4+KEbZkBg38nBTxyH24pIOu4MZSZ?= =?us-ascii?Q?DEEm23EI41ZU1eszXfCoE8DfCU7t+clo+laIEg+Qxd28pAYLCG3THueQIegk?= =?us-ascii?Q?g2xy33GiKfNEfLvIOMymJEKVsa11OiIzjX9Tj9gt8tV5WlN7feJPv4vCuL98?= =?us-ascii?Q?ZXxkf1WO7A4b0qMSFLPXOQN89cSuZqryrqOGIRrDsAtr8kbulyedaGkPSHyQ?= =?us-ascii?Q?AIVYIAEhAUoBZ/tKEvu4SnUvPhjcTVBRKX+BW8nBywWJwjcZlnjt3q6Y6ltU?= =?us-ascii?Q?XqZUMcIw0Cyho4K0Q/ZsEa8LYby5YEH09dLZ1hPBtX/zU71fK3kn3bC//chL?= =?us-ascii?Q?FWyamfirTxKDAcIvVOvoZkovPWu/M5dJmYqVT+Y4A1aPNY7YydwUYWWuHsAe?= =?us-ascii?Q?f21FcOdZc5GOyYZFiWBl7JIXda06ubUPmRYUsSNUKfLyBatPvr89iWFWnrXz?= =?us-ascii?Q?2jbVAcj03N5S7SujJWERJf5PKqnTLrFYjjmEIt6zfDyv/kbCIMg5u02OsVNy?= =?us-ascii?Q?dYXXeA=3D=3D?= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: f3fd544d-ce6c-47e5-4dfb-08db0b9b4f6f X-MS-Exchange-CrossTenant-AuthSource: PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Feb 2023 19:16:28.5775 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: shL9LVBFBW0ZKTq10DALD49s6dHoBkxHOwrJ7abW38hWY7w3NrkQma1OoWydBvbqEmf78hqVqUKpzluDu6T+2w/OniyinkH5GRLeJXHhZTk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAWPR10MB7817 X-TUID: h7clwLNgC2S1 ACK! fixes a bug in "next" that came with "bb2" and should be considered on a "fast path" sstate testing in Isar might have to be improved to cover this script after we test that sstate rebuild was faster, we could test that sstate clean age=1s gives us an "empty cache" ... "du -s" == small Henning Am Fri, 10 Feb 2023 16:34:34 +0100 schrieb Adriaan Schmidt : > With bitbake 2.0, sstate artifacts have changed from tgz to tar.zst. > Our isar-sstate script needs to scan for those as well. The > implementation is backwards-compatible. > > Signed-off-by: Adriaan Schmidt > --- > scripts/isar-sstate | 24 ++++++++++++------------ > 1 file changed, 12 insertions(+), 12 deletions(-) > > diff --git a/scripts/isar-sstate b/scripts/isar-sstate > index 53d0541f..c14c2843 100755 > --- a/scripts/isar-sstate > +++ b/scripts/isar-sstate > @@ -40,7 +40,7 @@ followed by one of `w`, `d`, `h`, `m`, or `s` (for > weeks, days, hours, minutes, seconds, respectively). > > `--max-age` specifies up to which age artifacts should be kept in > the cache. -Anything older will be removed. Note that this only > applies to the `.tgz` files +Anything older will be removed. Note > that this only applies to the archive files containing the actual > cached items, not the `.siginfo` files containing the cache metadata > (signatures and hashes). To permit analysis of caching details using > the `analyze` command, the siginfo @@ -576,7 +576,7 @@ def > arguments(): '-v', '--verbose', default=False, action='store_true') > parser.add_argument( > '--max-age', type=str, default='1d', > - help="clean: remove tgz files older than MAX_AGE (a number > followed by w|d|h|m|s)") > + help="clean: remove archive files older than MAX_AGE (a > number followed by w|d|h|m|s)") parser.add_argument( > '--max-sig-age', type=str, default=None, > help="clean: remove siginfo files older than MAX_SIG_AGE > (defaults to MAX_AGE)") @@ -664,21 +664,21 @@ def > sstate_clean(target, max_age, max_sig_age, verbose, **kwargs): links > = [f for f in all_files if f.islink] if links: > print(f"NOTE: we have links: {links}") > - tgz_files = [f for f in all_files if f.suffix == 'tgz'] > - siginfo_files = [f for f in all_files if f.suffix == > 'tgz.siginfo'] > - del_tgz_files = [f for f in tgz_files if f.age >= > max_age_seconds] > - del_tgz_hashes = [f.hash for f in del_tgz_files] > + archive_files = [f for f in all_files if f.suffix in ['tgz', > 'tar.zst']] > + siginfo_files = [f for f in all_files if f.suffix in > ['tgz.siginfo', 'tar.zst.siginfo']] > + del_archive_files = [f for f in archive_files if f.age >= > max_age_seconds] > + del_archive_hashes = [f.hash for f in del_archive_files] > del_siginfo_files = [f for f in siginfo_files if > - f.age >= max_sig_age_seconds or f.hash in > del_tgz_hashes] > - print(f"INFO: found {len(tgz_files)} tgz files, > {len(del_tgz_files)} of which are older than {max_age}") > + f.age >= max_sig_age_seconds or f.hash in > del_archive_hashes] > + print(f"INFO: found {len(archive_files)} archive files, > {len(del_archive_files)} of which are older than {max_age}") > print(f"INFO: found {len(siginfo_files)} siginfo files, > {len(del_siginfo_files)} of which " > - f"correspond to old tgz files or are older than > {max_sig_age}") > + f"correspond to old archive files or are older than > {max_sig_age}") > - for f in del_tgz_files + del_siginfo_files: > + for f in del_archive_files + del_siginfo_files: > if verbose: > print(f"[DELETE] {f.path}") > target.delete(f.path) > - freed_gb = sum([x.size for x in del_tgz_files + > del_siginfo_files]) / 1024.0 / 1024.0 / 1024.0 > + freed_gb = sum([x.size for x in del_archive_files + > del_siginfo_files]) / 1024.0 / 1024.0 / 1024.0 print(f"INFO: freed > {freed_gb:.02f} GB") return 0 > > @@ -716,7 +716,7 @@ def sstate_info(target, verbose, **kwargs): > for k, entries in recipes.items(): > print(f"Cache entries for {k}:") > for pn in entries: > - artifacts = [f for f in all_files if f.pn == pn and > f.task == key_task[k] and f.suffix == 'tgz'] > + artifacts = [f for f in all_files if f.pn == pn and > f.task == key_task[k] and f.suffix in ['tgz', 'tar.zst']] print(f" - > {pn}: {len(artifacts)} entries") print("Other cache entries:") > for pn in others: