From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7191847640499224576 X-Received: by 2002:a5d:5242:0:b0:2c5:55c6:38c9 with SMTP id k2-20020a5d5242000000b002c555c638c9mr168095wrc.237.1676551590852; Thu, 16 Feb 2023 04:46:30 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:1c0a:b0:3dc:5674:6707 with SMTP id j10-20020a05600c1c0a00b003dc56746707ls2584973wms.2.-pod-canary-gmail; Thu, 16 Feb 2023 04:46:29 -0800 (PST) X-Google-Smtp-Source: AK7set82yHUUWWS65QvZ6mSv0+Q8kk7dnLxYnMP1abDrZF0B09p+bnkpJWhA7P4jyM9HGtNaPMWD X-Received: by 2002:a05:600c:180a:b0:3e1:f8b3:6333 with SMTP id n10-20020a05600c180a00b003e1f8b36333mr4431660wmp.27.1676551589474; Thu, 16 Feb 2023 04:46:29 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1676551589; cv=pass; d=google.com; s=arc-20160816; b=QZTgRWSLAdXhSaZVe5Qf2lD9Lb+o+CsFsgILS66nWoLxh7Y8goM8li9UBj8bAwVDN/ ce8CQ/wG5Bl2W7JLl+UcHJD9DMy3WpUCeuQqE1zELPoRyqpOLbli7V2BrP3QCLEdIvWz aiU+2fWGHSjG2zK6pvxrEyy0cFDlx4MhzeWVd4lvCZgJpytGEYFtWPFowWmdxJE7JBBk +Sr+MGqvdFomOs80mANN1O6IJFKldZqXjgnMQFvejDOWzxioZfnyvypkjtrzQcX2C5/I +Ws70XO8ri8y8HuZJJGn8pPTbV/+0+38k32v/VKvS3DSCESq5H8oXHo/Gu+Ag2ocNtTz S9TQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:references:in-reply-to :message-id:subject:cc:to:from:date:dkim-signature; bh=ttVRMJ2Bfyy8e52aQBkkbX9YCZ/mSuaHZYato7uQY6I=; b=f47eMNnwmiuJZceTSRTe8FghgyFqrRMhou4vzBobWxcrvpO1ritCF6eL4YGFPS3rrQ fsvN9nJEtMrlseaIQhT/W5xvECU5IF2waFDAq5y3yBjyvOzDnRwbZNH02WiiEL93sJS7 GJJsN3dSUvyocIR0xdgGh/3y15P/QgBfsPIpm5zwiPHJDOBUAAZ5s/RU8ekLiy6izK2G pVnlTxq1Y+dyudpO8vXJAPv4fwm38d1rCWIurG8o3rjYf+ypWlVJNUVXe9rTtD6IZ4cN 5EFXeUpgsDz/Gxtb74saHIneuDMNm6AT6l6XF2pxr3zbCrsO1zQ2IsfKHLj5VxiZFcDn HueQ== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=HXejoLQg; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe0e::61c as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-vi1eur04on061c.outbound.protection.outlook.com. [2a01:111:f400:fe0e::61c]) by gmr-mx.google.com with ESMTPS id az19-20020a05600c601300b003e21b96f27asi13390wmb.2.2023.02.16.04.46.29 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 16 Feb 2023 04:46:29 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe0e::61c as permitted sender) client-ip=2a01:111:f400:fe0e::61c; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=HXejoLQg; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe0e::61c as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UbKPBISMsunFOVHgTuKPYvfwAjC1uhOz7V13RD+o3uS9Jyc9nLabxN0RjzsC3c2GNWGB8bUhIRmfPG31JLijOUD4KraOxAkHCbbFkDrlBKaqbNDVc0k2Rt+bV+u5kDfRqi7TplZ+g6yk5k0ZF56t1zPHIiJF9Iu0KiCZvhgX/EoRpl9LKjZGd3QEas31TLSKQ6+/GPTwZVlIVXQBr0Gc/L0nX1UL7RqIHSgDxow9e2pgHhse8E3Bko5POmWC+8ucpY5VuhjR/kDqOp5WQ2WrnhIIhus2zTRWqX+P5ygsaGv8dNMCHaG5oKUj2tb6943nw+UGcu46O1oqDpGhvdDMMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ttVRMJ2Bfyy8e52aQBkkbX9YCZ/mSuaHZYato7uQY6I=; b=cn5+ZCkBmw4aL98qytLyf78J+1GTLXIGRJcNcx0HWB2cr5iGzQYZICy9o78oFeWQ9v1h7RPhWFV98OF4FwRSffq9F0qfMoLJStdsp1LwSmAFAW0SlZdTVjoNILgoXj3tAujMstxPbvjfEjPZxTrnJlb38LAODdAFLd2hwUsy3YyAlDxV9PAWwCK+yk9ALg5an3gTT6rY5G2No+afmdSCNuZHDguXkpjW656wSq8d6U3N70Yw4/IjIQ6z/RG+AU0Khuamqu2rhpUGj7eZ5X/RGm99JCtIUQ195LUHK2/AoBMEDM9rGZE9IRhNUPx9A/NnLwsQ5qcGG92DXGo6epoMRg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ttVRMJ2Bfyy8e52aQBkkbX9YCZ/mSuaHZYato7uQY6I=; b=HXejoLQgTYaMU10VwX5mWWQaHzEXaMkxrUsUFztEYtlhSdG1eFrF8K8qERCcWo0s2Z5NfhnECqnazR1fdbo8CIYvb+dIDV+EQdarwPT2IdQUQamtVyKRcaqN0LYM2rMRoghhgZ9E8zLP7zAbRtKhOZfQzaTfELgmIF/hovvBx5eqxLRMOkEYCwBF23iLdiuldNMwuniuj9Qh3GGrDQ7y1Egs/MT3Zn63Wr0ywFmjVc0GOUdnvb7wdvMfoZa72FK0RWHJ/FKu4syq+XvTqaFaT/5ggEkbLoT7gao/R/ZFcPxTHIZXl32VBBHxp0IlPs7Pa1TaHCCp6kWMlO8wW1NerQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:269::8) by AS8PR10MB7375.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:615::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.24; Thu, 16 Feb 2023 12:46:28 +0000 Received: from PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM ([fe80::ee44:171b:4c40:d42c]) by PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM ([fe80::ee44:171b:4c40:d42c%5]) with mapi id 15.20.6086.026; Thu, 16 Feb 2023 12:46:28 +0000 Date: Thu, 16 Feb 2023 13:46:19 +0100 From: Henning Schild To: Michael Adler Cc: isar-users@googlegroups.com Subject: Re: [PATCH v3] feat: add zstd support to image conversions Message-ID: <20230216134619.3bc1e185@md1za8fc.ad001.siemens.net> In-Reply-To: <20230210143326.267033-1-michael.adler@siemens.com> References: <20230210143326.267033-1-michael.adler@siemens.com> X-Mailer: Claws Mail 4.1.0 (GTK 3.24.35; x86_64-pc-linux-gnu) Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-ClientProxiedBy: CH2PR18CA0005.namprd18.prod.outlook.com (2603:10b6:610:4f::15) To PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:269::8) Return-Path: henning.schild@siemens.com MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PA4PR10MB5780:EE_|AS8PR10MB7375:EE_ X-MS-Office365-Filtering-Correlation-Id: 397ed132-ce87-45e1-cf51-08db101bd203 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: cLBWwpkWEnvGvarAh1AMzeFmA+rRy7Ea7C2ilXlJVInFeXTD5gRu5UdXy4tqVW6AEV8WtCPPLcTU8r5Wrs+Z0/ny6jjJpme+Yy5SEgTH8EVEH0W9RZHJfxBrTnlHQ6y0zLk/PkfZ/m+fjcm9HThWvkcgaqCKlYfDmU+V3d6fB1BLMD9ykPDbjLdTSKKRW7rImbAN2vYfBr+KCrZecZdHOLMlfv4jLfS8Wo/K4C3Buvu+HYUNTE21onZfhbKNK634kDugxV/nvv9iepyALLdHyJtfmkhc068jF1qAIwpupI5Ycp1CHINUjIxM2N7RVFr47fnYS/N0yMN2++6qsPWqf14O/S8iPQe/KKiD2dctsK9f2tlktByw6LCRIqqXDgadIrObNOoM1fpWODXevsQ52nLtfOI/JCt1TmlAAeuz6E/hx0pZNkhlDczrmyzRf5WgOAug3JWLtRjaF4y3fx4l18yn0EctwGa+wyRwQxl/+tXIBBULJJQUfPyF77a1HXRgAHq/Rg0eydHJR6KPcdcCQn62hjYOH0/Q3jc9diiL+P/bKXh1MsNKwhw04G132QdobuUhqUcJ/EjGV1V4oY4VzPe0o8ZNe0NZSiAYbjwlexSx2irmCqyt8jEGNu+IirT1CGqIPIjT3mWRhIjSTSVVWGikNnqfVLQFJNORSlntr4g= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230025)(4636009)(366004)(346002)(39860400002)(396003)(136003)(376002)(451199018)(44832011)(6506007)(6636002)(1076003)(8676002)(86362001)(8936002)(82960400001)(966005)(41300700001)(478600001)(6486002)(38100700002)(66946007)(2906002)(83380400001)(66476007)(186003)(66556008)(6512007)(9686003)(4326008)(6666004)(5660300002)(316002)(6862004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?uJDLhZ4FttPn5ejiKlkgHDiw/IJ1JOUxnGUVJs0GE78jrEw5YO34raODikrl?= =?us-ascii?Q?6iYqoMPKgraO5wdyo04Mf6N6RygreLMTLPK89XhQ9CDH8g+KidhB5kOaPDeu?= =?us-ascii?Q?fJMXp31xedreaZXM1Im1xCnTTiZjqfYTcOYn/kAF7qaFj8Fb1wB3lCGomN3L?= =?us-ascii?Q?EObHnX7hucsOdXsE5V0j01pN8DJDC5oxPrS4hJPpPIWMzl/MZrca3IBtZTk8?= =?us-ascii?Q?V61TwlMBEE+cFpPyNMgEdccXZZm4/JxGitorcZkwO2d58DVap3KRq90dVZuB?= =?us-ascii?Q?KrxIl8Ct6wKpyOedmWCl2UMMFhdOM1rs17b8nsaYM3/0A8o7lAPhfMXnFMxE?= =?us-ascii?Q?r2gJkeEK+mKSjvq9O4M1R74Do6HLXUG1my41bSH9nzG7RzUBEwrZdteCPo5T?= =?us-ascii?Q?CUTOSc8CO5aSe+zaEntuCcZlL0EHMhOU8ufUAmL0PMym0MymkCRTEg379kwu?= =?us-ascii?Q?IxHygknsKmRohaVPDNof+ZJv5tlHopvg7oYRnZSkHd2XngDpbSskr7XoaiyU?= =?us-ascii?Q?uQHYmdcQMAH49GntSIanVe8O0o5vuWu92mmkSGDyRhbydwEDdk03L5lrnCW9?= =?us-ascii?Q?5P5wQc19xYAq+/VGO3O1AiqDDPEodf68eCnrwqFhjAYQ/Q3ngxaRTNoV7JHj?= =?us-ascii?Q?xbVSlwhxuMhWX3XEXxcUKX8r2lxb9cNeBT8g6eO3tfZoAn7mUZ8oTiXa/9bX?= =?us-ascii?Q?4Zb7UKCRuM5j97l/pBNqhijySiKaLTV/8hEBsn24Cmdf2yFhUcvjy1qXkiij?= =?us-ascii?Q?Qb9wXgs1GuGkTCc1aet2UsAwBKsr71DWLxqsWw3N2mWn4wI2Cx8Z6dfaHNqM?= =?us-ascii?Q?aySllL0iZ1XWCfC2YIVs0EPzzRqesSBf6fCFkKUqmbfVb+dPokQbzMVIazKE?= =?us-ascii?Q?g6zNKdrnAmyaYS8LGyZSHY0sVy0LzNEQDrOpC1g7ra/qmCKbx8uuGJHe+N+/?= =?us-ascii?Q?ZD4W5C29UvhIOwYe3PI/7rEVcmXWQOSg/QVaFQmoWgjVbGR1WLAm5oqp4Q9P?= =?us-ascii?Q?l8z76p73yaHiV+jQVawSA0kokwfXUCk+O/eEMQWq0EoPaijSfWSuTtYOkY4W?= =?us-ascii?Q?BtrAczsNZpJ6UCtOc3OVb3LzI+KP29HgIkWLDaB6bkmnhCbWAz7k2hTpozBY?= =?us-ascii?Q?kdxT69nw+XMXWFWam5gQ+4at1yTgtaHffNc9VndDAAbhF9YWYOivuDqXyQpo?= =?us-ascii?Q?x98dmABHqGCpUzi/FukQKEYKYRnZUgQGxiiHkFGFH4sd7lrr41k4DX6levHc?= =?us-ascii?Q?+LeQ2FViRToRBhRuX0qXaKL2edacvb1BcUyGueCxQGaV0cCrv+3n9xTh6Oas?= =?us-ascii?Q?UPv8QevjRWEEmpHlIk9WMbkOyHHi9+lWqwcvESCStZ1fjYJb35bT6jS9MiD8?= =?us-ascii?Q?uX0/S9QzBDYMpC7v+5YlSUisqM6E9nlwjKHO+wNTk/mFayB8iSikws65OgTK?= =?us-ascii?Q?LAQ0qmIWC0HGKTXnYDHNe6UaB7MkbjM91oFGqaFNoBGw2ocfLMDv2lRs1YNY?= =?us-ascii?Q?SWYmgGLzlF8xfcekIk08PAgtd/v0yCopWI1Ziu6rvCLE/QzbO4OrXzsPjA08?= =?us-ascii?Q?8bzvIq+1wzPXjG4S//fRhKDDckgJXAQh60H91Gg1tKz2R10pL+VArBcx//Fm?= =?us-ascii?Q?Ag8779S5Sf6pXB+brdnTED7ELtAxzPLn58dziSgR1f8vsd2sBL6x/taGBkbC?= =?us-ascii?Q?er+bcQ=3D=3D?= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: 397ed132-ce87-45e1-cf51-08db101bd203 X-MS-Exchange-CrossTenant-AuthSource: PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Feb 2023 12:46:27.9651 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: th8Kehqf5SLI4mpR2t0fqzl4bPgLuhCO5yuxImSfeSiLZBU4iDKlotd17qZ4llkxL+5sDuPqEnEqvbkCtznl+dnVP0DzeOoH10PW/rNxu5c= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR10MB7375 X-TUID: rpFcN60NafDG Thanks. After seeing this i asked myself if etcher can handle that, because that is the tool we often tell Windows/Mac and even Linux users to use for flashing. Seems it does not yet https://github.com/balena-io/etcher/issues/2856 Henning Am Fri, 10 Feb 2023 15:33:26 +0100 schrieb Michael Adler : > This patch enables ISAR to build zstd-compressed images. > > ISAR already supports building xz-compressed images but there are > several key differences between the two that may make one more > suitable than the other, depending on the specific use case: > > 1. Compression speed: zstd is generally faster than xz when it comes > to both compression and decompression times. > > 2. Compression ratio: "zstd and xz trade blows in their compression > ratio. Recompressing all [ArchLinux] packages to zstd with our > options yields a total ~0.8% increase in package size on all of our > packages combined, but the decompression time for all packages saw a > ~1300% speedup." [1] > > 3. Memory usage: zstd uses a smaller amount of memory compared to xz. > > [1] > https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/ > > Signed-off-by: Michael Adler > --- > meta/classes/imagetypes.bbclass | 5 ++++- > meta/conf/bitbake.conf | 3 +++ > 2 files changed, 7 insertions(+), 1 deletion(-) > > diff --git a/meta/classes/imagetypes.bbclass > b/meta/classes/imagetypes.bbclass index 0cff440..c2033fd 100644 > --- a/meta/classes/imagetypes.bbclass > +++ b/meta/classes/imagetypes.bbclass > @@ -93,10 +93,13 @@ IMAGE_CMD:ubi() { > IMAGE_CMD:ubi[depends] = "${PN}:do_transform_template" > > # image conversions > -IMAGE_CONVERSIONS = "gz xz" > +IMAGE_CONVERSIONS = "gz xz zst" > > CONVERSION_CMD:gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c > --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'" > CONVERSION_DEPS:gz = "gzip" > CONVERSION_CMD:xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS} > ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" CONVERSION_DEPS:xz = > "xz-utils" + > +CONVERSION_CMD:zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZSTD_DEFAULTS} > ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'" > +CONVERSION_DEPS:zst = "zstd" diff --git a/meta/conf/bitbake.conf > b/meta/conf/bitbake.conf index 522241a..c9f52a8 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -140,6 +140,9 @@ XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT > XZ_THREADS" # Default parallelism for zstd > ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > ZSTD_THREADS[vardepvalue] = "1" > +ZSTD_LEVEL ?= "19" > +ZSTD_DEFAULTS ?= "-${ZSTD_LEVEL} --threads=${ZSTD_THREADS}" > +ZSTD_DEFAULTS[vardepsexclude] += "ZSTD_LEVEL ZSTD_THREADS" > > BBINCLUDELOGS ??= "yes" >