From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7068176960994672640 X-Received: by 2002:a05:6638:358b:b0:311:b8d0:cc14 with SMTP id v11-20020a056638358b00b00311b8d0cc14mr2659232jal.58.1645718253907; Thu, 24 Feb 2022 07:57:33 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6638:22c7:b0:306:5406:6e59 with SMTP id j7-20020a05663822c700b0030654066e59ls726001jat.1.gmail; Thu, 24 Feb 2022 07:57:33 -0800 (PST) X-Google-Smtp-Source: ABdhPJzkuaJTXMTBPX3FTb8MKL2Lh24MuziGPowyDst7Swop2c6FsteOmECRWJLGKF25FeLXIUFF X-Received: by 2002:a02:c851:0:b0:30e:2296:4290 with SMTP id r17-20020a02c851000000b0030e22964290mr2765775jao.49.1645718253362; Thu, 24 Feb 2022 07:57:33 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1645718253; cv=pass; d=google.com; s=arc-20160816; b=POfy+jp3HLXQTFD2PTfi0XykczjWo7g1X3+mhgnOCtsJLwpj296D59svYtkjS94INg /PFv8eCkT+ugFqBbelX1WHxjPmz5xb76Ctzxz9/SAC7+qAbA8FNNt2P6yIa6hpNkkvxL lHdNnphYy+QKPjgx0lIS8lU7/G329pFiH21/gZze9Mscy2jZBzQIFySTdP1omTJ74EOg b+t77Fq+esQenm/Z1Op1P5qhkFsAnU5FTYcVWPJB1polw/RYTt2v9zWUCeeb1tN3lERo X5yT0MJ5xKQwZwgk9fwRsUa84HIpmIascUw5wUX06lAW5gRqlSNiDhQd9xCf+MwHQ5+7 h91A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:references:in-reply-to :message-id:subject:cc:to:from:date:dkim-signature; bh=FmYK07CnwtqOVgpC7BSkATpKaTbqPXKWlO8NWv1rnsQ=; b=T0mSBCFKuhFnxePfKIzjtQsydEvOgMuCc7RGXi4UrzJWNln30WgpfVhSQRyZc6NIIu XCXXfzgBtFIjzzL6p9aeN/HkSjbiCu5/5X26OHGi5NmuNodAGFEE3jLnRIU+bRJL2KkC tO2T9EoCr/4M2rCog7cQIPBw96+z9OZYi/XLLQcO7QyhHMpoabQ/rvQeu/A3+kA5lOx0 8o7RDxuTG4AS2BMM0OYlgv62Wiaw4/Y05kN7vDPDWYtQmKW0izJgaSpoKKTwWlrzkOaL rzv2oi8gibQJT6eUubAztAdzyRRxuCcpAu+hhs2pYHLOQVg475MIUL/mqXkKPxfglfXV G3Aw== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=gvkpMBzS; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe0a::62c as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR03-DB5-obe.outbound.protection.outlook.com (mail-db5eur03on062c.outbound.protection.outlook.com. [2a01:111:f400:fe0a::62c]) by gmr-mx.google.com with ESMTPS id x8si416727jaf.7.2022.02.24.07.57.32 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 Feb 2022 07:57:33 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe0a::62c as permitted sender) client-ip=2a01:111:f400:fe0a::62c; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=gvkpMBzS; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe0a::62c as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=huUx5e3INSCZjyG1Th4sKy8P3PRCR2qnk1dyowFutsCIQmJaY8jMbRNEkMhM5uFsTGpoMTY8OK3x9ySVy7oY25DvXInDTwGe47BOCXFmxFCoKD+BEWmtdWRUVIxKPMGHDGJ/XHNwSouohvPp2Q4J0971AsdnENIm637e/Gg/oyj/ruK6ijOM2S6B8AIEwCq3EllkIOmbBgPNaFDv6b/xsyyDGA1/hTzXQf9xjxZrFvzWN34nLOpvUuGsQymY39c4ViMPCn3fJzzsdyMAr2FAJLY2xKMSl7XDlahYYU6LUD0DOWUomsJs1pC9ZFXGqaBaywud6ztIE8kpSkZJgwfP4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FmYK07CnwtqOVgpC7BSkATpKaTbqPXKWlO8NWv1rnsQ=; b=Bh0gRsCy73iAUOa99E/EOI8fgsVQD3JG8CjZsDIMpKCJRn4MMFFTCELLo0wtR9PdbGbpYugKvSlnf0yuFr8egwOzSMpRen5nLkPih4qdPorc7Bx9BeUdQFapRSEy5a6n3u36PH8P8PlsypYg9kmrykUJ2dgRRo+TMLrfEbzYJuCwUVNt51qG3j91GvEkrvkldU/w3tT9pg90yoYUK1ZjlEent+Enh4w4j1UW2mnzEKSLvvrlt7hwxmWg/Pc2lbozr5RmXyoL0357c7XwlWV9tHSdvX0cQ8HvGYqlAu097bVnWW85WVymp1PHZBqHMr/+7h2lWe/CyU7Nq0rqjudN/A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FmYK07CnwtqOVgpC7BSkATpKaTbqPXKWlO8NWv1rnsQ=; b=gvkpMBzSME9d5+twbhDUAGii4/9yFxS7yYMnh8cWgxFHrHk3/z6lIfzWtrMoHUNoimk7gaYf1UuDdzfk0CWQvZoP1AmBXpiPVmxboHRuzvjN21ySclRzB3EDYVlkScMQjoQZhNKyTTXlVa/vcPiofq5Wm+F7QOrV0GTGqeTzdRX23vUSjHeS+f6JeZrlv4dvxTIzSHT5TsDZqnn1cSeX3kNBQYMixjq0RBxGtsEgJ/IieZsWz8qICFGIm3+73GS3DFQ8K1c0N3bGiv27qBgI5l/ijWdMH/guZU5/ArZpfhy5MntjQSX1UH5cVguL0xtsETNPR2+BxZ/plxq3w2Vbpg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:13d::17) by AM4PR1001MB1441.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:200:96::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.21; Thu, 24 Feb 2022 15:57:30 +0000 Received: from AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM ([fe80::a116:222:eb1b:efdf]) by AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM ([fe80::a116:222:eb1b:efdf%4]) with mapi id 15.20.5017.022; Thu, 24 Feb 2022 15:57:30 +0000 Date: Thu, 24 Feb 2022 16:57:26 +0100 From: Henning Schild To: "Schmidt, Adriaan" Cc: Uladzimir Bely , "isar-users@googlegroups.com" Subject: Re: [PATCH v2 0/1] sstate: do not keep intermediate sstate files Message-ID: <20220224165726.20387905@md1za8fc.ad001.siemens.net> In-Reply-To: References: <20220224073629.23717-1-ubely@ilbers.de> <3410387.dWV9SEqChM@home> X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-ClientProxiedBy: AS8PR04CA0029.eurprd04.prod.outlook.com (2603:10a6:20b:310::34) To AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:13d::17) Return-Path: henning.schild@siemens.com MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 028f49da-7fc3-4e5c-4655-08d9f7ae5cfd X-MS-TrafficTypeDiagnostic: AM4PR1001MB1441:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ofJ0Dwchhjd90mByzz6pg5YYuuRQAcnxCV19I2GZLgrR87cSV6vyZhoL+fx1F7AWMJmNPzZiwQZZ/l3Vt90Mk65YRw8ZtPnSbGwWTmIYAup1Ze9nk4YAaM6/kseU+DcPVzz/A5d4/P/9ULw9upt+N5uGcwMW1p99S8ET7P0XcgXE7jR5GRLNFV3Aod/k5Lap6pUp9/E3zdlZ0FhKhgiLcS6LLt1DJBJJV5NgpV5d467jX5GOW1NBYqxdp/YG16SmjjWOjQeTfhS/x+tKc67RhsayxVER5PKNgMVS+7eg4Ptu+OEYLs91XFD6ZsbyglKiSyDPPIiF4jYY36HFzMR66o6af+87DnfCSEhmWsGF+3tBsNdSGEaiWJ3Ytuy2VORtR4T7hbkr8mt0fb1YPQQwwmKZ9uFrvt1+mbZROT7ulnEpy5uMUHkNAygq7ihqY40XL3zssInupnAHEbYBQE9CYGp6n9fb+JgcEDYgYKs6lVpn4f2wCi3jnV8lhSRBnP77CI66946SVpJFoN83jbMOTCZ44ZuTY4i2Eo7ofdOTTJoeHZ/HdIkwxoMat4sP+W240EgdW/1Q/8vSyusBvw1UQfr4uO7pc2aEVrCOl3R/qVhpME0D4x7vShVfeBhVFFc9QrEeAn/Kr3qf1wA2XzSv6SNj5udDGhRZGaCGKIb9yqMMC1R0xzXIRl0lPAThCFdP3+rYjkYn3ysstMgMk+p5vXQ08r4SuQCGQ+s3609J0cc= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(44832011)(66946007)(66556008)(66476007)(8676002)(6862004)(4326008)(5660300002)(2906002)(8936002)(508600001)(6666004)(6506007)(966005)(6486002)(6512007)(9686003)(186003)(1076003)(83380400001)(86362001)(38100700002)(54906003)(6636002)(316002)(82960400001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?wy0nzKYOBHyxdMrr2gaMqpDuyRPoX3jLxsXL9roNF6fkAkLDt0NPr3sjsaPh?= =?us-ascii?Q?qPfs4e64ReBElmyVGbequv8fMr4/fgnYNMDgN6lFEnUXGQCb0kp1VejnfYvX?= =?us-ascii?Q?er24OLaRUeBNWXcUyVxneFZzL2i8OWHgJ2YvmH5Znxv6t/Dr5oOQZ/RzZ3kL?= =?us-ascii?Q?Cld3rOps1TWz/rIQjGFaBVHXXSTTRKFDvNg3MNQuldX4H/TEXzzZxCkpxahd?= =?us-ascii?Q?V+rxa+gZ5BVGt6vxrnKmxdCvor5DCbzckOVSHm+CI3FFJjljAKWsmc2aeChd?= =?us-ascii?Q?UMXrepGQHsiCvemXHki+QdIQX2bXoCf6dFGSB6EaBh5si+C3tzR8sP8N3Ryh?= =?us-ascii?Q?BWzCPB4gkm5w55mXGMWhCv95I0NnKhvLOYvoo38aMQcE+XzN8SpOpDnmX2pM?= =?us-ascii?Q?yz1yC/7GLZ04qXEdvXYBwSgLEvPzt5a/b0nq/YGSXhuDjWXPEyG743m7Dv63?= =?us-ascii?Q?W7RnuA6iUZUNAWHiuF9BlCh6FbqXpMf72lrYtAHdrfPfBe7Z134zb/j6B+us?= =?us-ascii?Q?7Wl7FkXeX4TqTTbmbUiRgl4wOgJrp43M51+shPQw00sskkaHLkKHzWug6sgU?= =?us-ascii?Q?m0OFIgwpIWdbKCTp8xvO2mFI56BE1KM2p5IYVYN8m21kD7jy7KQxTD9bKkCR?= =?us-ascii?Q?o1ObPX1ifsgeV8e3AOYgo5zx4vpE3l3e+u4srfODvnm/yR5x9REvfsJgalUp?= =?us-ascii?Q?5ntMXyLnqIekU8aXazqz8X3JAqpvwnHS7hJjz/QMsuym+1RdXg69Z0cTIVkL?= =?us-ascii?Q?qHmTwIoFXxBTqyGX+ISVCt2A3Oz4c2y9TW4Val0YEEQVridWhNT6phoWcc6L?= =?us-ascii?Q?nfhG/HWZt4hZgwNtisCtxmk3Zk91M4BBfqDAOJDuJT0A6u080ZEJYdIbl2XS?= =?us-ascii?Q?5UO7R6g7jLYGYhhy5kybsJiUN7IuayKPiCz2ZmOoAd9Hmv4vztbrh3xJfhK/?= =?us-ascii?Q?wpEOxybuoA0XXByHYlyfeWvpY0mgBBRYpRANZA2b72JWYoCBaL4jghHfxFce?= =?us-ascii?Q?zFaYPV05/sGMtyFwgUbRd2zqxJh8AFhKKl1DREmhBy9/IOEA1QWscVyMf3vo?= =?us-ascii?Q?L2NSoANbhJr5uQrZ9KbrrOvUK+5Uvq3roenNBu5Qm/j9Cs9bhGBVZ21T+yzc?= =?us-ascii?Q?smcZ2m7/4WWHIw+pUcBHHdgl64w+WMTsd6gh9C+Pw46w6IXZ++5K6PKOObaW?= =?us-ascii?Q?RUxPLvbExcgYuJ15HdLUE0fdmhS4BrLOed7YT+kgG2WDyM+sYrxmXep2Xboy?= =?us-ascii?Q?y3vzp73qaWrcGKG0jjWuPAfC5C9L14+J9yMPy1XuygxEfgVwo9Ln+tKsLp+4?= =?us-ascii?Q?Xm3UKnKOMwBJDjdi/wXhpYuKp5dTcPxBOG6XNufpOvzTVD/1Fo3f7F4XWnpN?= =?us-ascii?Q?GHBxNT7NQxWsc3GKmcccZah7OkMrrFRKTEZZykynKWp9jmCBDIkYa0nZZHe4?= =?us-ascii?Q?6w+YabXRG715qOEYeTtJzsFZNt1X1bY0y1YTC0Q8FVW+h2L2Pl1Jcpho6Kdq?= =?us-ascii?Q?kXY6ulaQD6yjpUzVvxx8i5QqXSBthO+ceBvHuiSZdMQyYsu/So6v62WN/da9?= =?us-ascii?Q?j/oH1pi3dCEixaq1fewEkv9qqp/HTR56z7ahLn5VCLDYTBKeq+GrHzU22Lf3?= =?us-ascii?Q?uhxPINU7eQUdN2ndNwXw+IcDLgbB68ZpJgybWqwNx/jJKeVjTeV7A7IS3qJD?= =?us-ascii?Q?lsfKJw+BlcB9m57qSFpf7hQHJbMYzxxdIT96CCJULk/RbdJ6EMrBaWubKTeM?= =?us-ascii?Q?kDHI/ucgpAPDMYD1XX9wzUlZdGhs4fs=3D?= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: 028f49da-7fc3-4e5c-4655-08d9f7ae5cfd X-MS-Exchange-CrossTenant-AuthSource: AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Feb 2022 15:57:30.7670 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: NfgU9o6qws78fEiebxbgv4R80hUjrmYLqyDjns72zVLIzLaZMzg4yojHMoYHzby8XE09rWvZE72ooVi3wcYh7WKbXcH0wICcG7H+DbshoTA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR1001MB1441 X-TUID: tWSo+H4Zpx3A Am Thu, 24 Feb 2022 09:04:40 +0000 schrieb "Schmidt, Adriaan" : > Uladzimir Bely , 24. Februar 2022 09:20: > > In the email from Thursday, 24 February 2022 10:56:48 +03 user > > Schmidt, Adriaan wrote: > > > Uladzimir Bely, 24. Februar 2022 08:36: > > > > Sstate is quite demanding for disk space. And it often causes > > > > failures in CI, related to 'out-of-space'. > > > > > > If those few GB cause problems, then you really need a larger > > > disk :) But I get the point, and I agree that we should try to > > > save space. > > > > These few GBs result to ~100GiB per CI job (when doing full builds) > > :) > > > > This patchset minimizes space usage by intermediate state files > > > > by removing them: > > > > - after sstate-cache populated: with separate task in task > > > > queue > > > > > > I would like to avoid adding extra tasks. There seems to be a hook > > > to insert functions (SSTATEPOSTCREATEFUNCS), but as sstate is not > > > really documented, I have not yet figured out how to make our use > > > case > > work. > > > > Yes, it would be better if there are some post-sstate functions we > > could use instead of inserting one more task after one we are > > handling... > > This seems to work for bootstrap: > === > SSTATETASKS += "do_bootstrap" > SSTATECREATEFUNCS += "bootstrap_sstate_prepare" > SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize" > > bootstrap_sstate_prepare() { > # this runs in SSTATE_BUILDDIR, which will be deleted > automatically sudo tar -C $(dirname "${ROOTFSDIR}") -cpf > "bootstrap.tar" --one-file-system $(basename "${ROOTFSDIR}") sudo > chown $(id -u):$(id -g) "bootstrap.tar" } > > bootstrap_sstate_finalize() { > # this runs in SSTATE_INSTDIR > # - after building the bootstrap, the tar won't be there, but we > also don't need to unpack # - after restoring from cache, there will > be a tar which we unpack and then delete if [ -f "bootstrap.tar" ]; > then sudo tar -C $(dirname "${ROOTFSDIR}") -xpf "bootstrap.tar" > sudo ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}" > rm "bootstrap.tar" > fi > } > > python do_bootstrap_setscene() { > sstate_setscene(d) > } > > addtask do_bootstrap_setscene > do_bootstrap_setscene[dirs] = "${DEPLOY_DIR_BOOTSTRAP}" > === > > It no longer uses the more-or-less documented interfaces, but > instead hooks deeper into the completely undocumented sstate logic, > so no `sstate-inteceptfuncs` and no explicitly created and cleaned-up > staging directories. I read code but did not look for docs, "sstate-inteceptfuncs" is used in OE like once so also nothing too common it seems. It might be worth double checking what has happened in more recent sstate. If what we end up with looks like it could survive a bump it might just be fine. > The same approach should work for rootfs, and could also improve dpkg > caching. I will do some more testing and prepare a patch. Cool! Thanks! Henning > > > > - after sstate-cache unpacked: by direct call of this task > > > > > > > > Some results shown below for three different cases: > > > > > > > > - sstate disabled; > > > > - sstate enabled; > > > > - sstate enabled + cleanup patch. > > > > > > > > bullseye-arm64 cross build with full IMAGE_INSTALL list: > > > > > > > > $ sudo du -sh build-* > > > > 5,7G build-nosstate > > > > 12G build-sstate > > > > 7,6G build-sstate-cleanup > > > > > > > > bullseye-arm64 cross build with empty IMAGE_INSTALL list: > > > > > > > > $ sudo du -sh build-* > > > > 2,5G build-nosstate > > > > 5,3G build-sstate > > > > 3,4G build-sstate-cleanup > > > > > > > > bullseye-amd64 native build with empty IMAGE_INSTALL list: > > > > > > > > $ sudo du -sh build-* > > > > 3,2G build-nosstate > > > > 6,4G build-sstate > > > > 4,2G build-sstate-cleanup > > > > > > > > > > > > Changes since v1: > > > > - patchset renamed from "sstate: compress rootfs sstate files"; > > > > - sstate intermediate files compression replaced by removing; > > > > - dpkg intermediate files are also handled. > > > > > > It's unnecessary for dpkg, because we use hardlinks to the *.deb > > > file, so no extra disk space is consumed for the sstate > > > preparation. > > > > Yes, I saw it, just implemented in the similar way. It may be > > useful if files are copied to another (slow) disk after build > > finishes - I'm afraid hardlinks won't work in this case. > > In this case we're only linking inside WORKDIR (not to the actual > sstate cache). But with the above approach all those intermediate > files/links will be handled in a better way. > > Adriaan > > > > > > > Adriaan > > > > > > > Uladzimir Bely (1): > > > > sstate: do not keep intermediate sstate files > > > > > > > > meta/classes/dpkg-base.bbclass | 7 +++++++ > > > > meta/classes/rootfs.bbclass | 7 +++++++ > > > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 +++++++ > > > > 3 files changed, 21 insertions(+) > > > > > > > > -- > > > > 2.20.1 > > > > > > > > -- > > > > You received this message because you are subscribed to the > > > > Google Groups "isar-users" group. > > > > To unsubscribe from this group and stop receiving emails from > > > > it, send an email to isar-users+unsubscribe@googlegroups.com. > > > > To view this discussion on the web visit > > > > https://groups.google.com/d/msgid/isar-users/20220224073629.23717-1-> > > > > > > > ubely%40ilbers.de. > > > > > > -- > > Uladzimir Bely > > > > >