From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7191847640499224576 X-Received: by 2002:a05:6000:1007:b0:2c3:e30d:4e92 with SMTP id a7-20020a056000100700b002c3e30d4e92mr550519wrx.534.1676039645841; Fri, 10 Feb 2023 06:34:05 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:3592:b0:3d9:bb72:6814 with SMTP id p18-20020a05600c359200b003d9bb726814ls2770369wmq.3.-pod-control-gmail; Fri, 10 Feb 2023 06:34:04 -0800 (PST) X-Google-Smtp-Source: AK7set+/NvcRz0gHZ2iIzdoqrfHakOgocgT6o+GsOKsWpro7ZCYfoCIxzvrcRyzlJgjsT8P7wKP1 X-Received: by 2002:a05:600c:18a0:b0:3df:b5ae:529f with SMTP id x32-20020a05600c18a000b003dfb5ae529fmr12817116wmp.31.1676039644594; Fri, 10 Feb 2023 06:34:04 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1676039644; cv=pass; d=google.com; s=arc-20160816; b=wNaBpnI2sJbO2WS+ip60uAI6nHbWLIrU/djfGHG8g9salsTYeV3mT+PXy15Vge1FTZ KfJVdDdix0k48aGKTJD8Sjk12tSGtJrw72l8nnShaaToYGSnSPbiSkrLJNYEeFIVx8Xc WANUeNlEHAIsaSgzXiSASA5Jb9ZRLZQERChpk7wFSeoupEVX8PPk4Hhp7lslBu61NH0K MyeyJZ59/neWl4st40yKmw3cKppuSGLtd8P3KVwKM5tqm783AFItCbFyKv1xBdqbGxqQ Bqg4DOb55oaglTx6XWOKnUGigZQdhc9u0pPU3QYdajeAMv+Lq6AO9bGUNCxeKXJF95tj 7TsA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:message-id:date:subject:cc :to:from:dkim-signature; bh=2UP3HrsSTySLwJInGW+7Jk9HEI0yYUbWxI6s1/my6v0=; b=Ms+UH98lGBgeYeGqCqLGO1RUK3mLHKW17ADoE2AiXNOka7MWXcNLwAtbvCdXs2NWFf xyD/PDNuzhy6SmcFp77Hq638mxZN86LngAq1C5gHd6vaj/nz527dcjK1GDfDifqUcj46 2h80PrCXsn4qnL6b3OXFB6jxKOwl4YaEsAvjMafvAWqriCqVLKJaxIQFYmHgewU1uI+A 9kmH4H8adjueBxqh8A/kQalfz5/C0JsIgq7P4WBDpNIyFi+Em1Z1jqBTCXBuh29tEHwk XX6Yh2rzv9xAp7uOOdhGFOklUKm1M3dX8JFKmrV6mbBqhzpTsSZMa1pnhpE6ZfuemV+F UMig== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=gBy04pvL; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of michael.adler@siemens.com designates 2a01:111:f400:fe0c::621 as permitted sender) smtp.mailfrom=michael.adler@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-db3eur04on0621.outbound.protection.outlook.com. [2a01:111:f400:fe0c::621]) by gmr-mx.google.com with ESMTPS id l39-20020a05600c1d2700b003d9dfe01039si634462wms.4.2023.02.10.06.34.04 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 10 Feb 2023 06:34:04 -0800 (PST) Received-SPF: pass (google.com: domain of michael.adler@siemens.com designates 2a01:111:f400:fe0c::621 as permitted sender) client-ip=2a01:111:f400:fe0c::621; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=gBy04pvL; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of michael.adler@siemens.com designates 2a01:111:f400:fe0c::621 as permitted sender) smtp.mailfrom=michael.adler@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LPfq5HjU10lB/NVgeTmlCpinQuFV3SyQhFOa6N3tRk7xN7eHnDrvHzWzSLh9QDvJ4DZ/H/Pkfzy/p+jTgDU0evdCOT1Ey1ROlZbmu6TMtnAfSQIJgFwwoHM8WtczoBHQkI7ykU9gaPtiqFljJk9hgxb3XrM3feIJOpj11mXmj/+YyWN52cEpFjvLVQXF3YLiIAKF6f1BVByICV4oNUJPzXMO4dbh9bidU/DX63qsFphk3zcdVidT0WUXP+WxOHUR3H0lQbDM6kwADpqQCD3UGmJ0GbtCoaRVGbK8Y7E3QpVrVdYYyJF8Gl/RgJ4K0Kil4MVlBFELR2CwzoP29eENbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2UP3HrsSTySLwJInGW+7Jk9HEI0yYUbWxI6s1/my6v0=; b=MBfijaJX5UJIe48WobevuFD2s8XpaCvM2XM3h/F7Ubk8I8BWLbEIqxhRtQQKkzwh+M0WIEv+cOQPjuub54HfNSppZjLA9ILSEPFdMHILL/U/Db2YEvjLWwWyk7q00xbuWpDUtilu8h2a/tps8VH602L3TQoQ91H5pBy3ofeqzraknu3lNyEeXYIiso+zuuCyl4kGbn8aBmS5DtKiR1ItvMdppNcfeR21QX+1bkP15qAAvcn9gCFOiR2V8+4L4G7/1Ig+Y8Ds1sjlz0sboM7yC7fBfVZS7CVgyaIaNJYoTSE+mwhpaBorRyfOZg11n7YKABQbGJdNU+W1HjIwQEO/cQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2UP3HrsSTySLwJInGW+7Jk9HEI0yYUbWxI6s1/my6v0=; b=gBy04pvLI7lI/x1LBt5/IvzUCzcbN8OoxN2dXdQT7uF/IM1CX/9HtcSR3awCsxCSXCqOaACSDXh4jvknVTJMassGPd/M9+lNnHUmgbqvFBV9vhnKJgRLyVMk15cl7gaMlVRfIWmH6cmd8zTn9j5oWMSVgXbGgShjiTiOeOFA8lRmf38xAHoThlSA+pTgkieim86VAKDxnqf110S+y9C4fVLVTK5/6E0hcKQE5nt7XCYxRlaSOsnbElkFc728w2bFGm5t3XYDvjD/klauk+0At5+WBFx7FwCnu7vV29WUBO5fGnCSTcXtbhq1WcmSg/DbRdx48TQAH0V0zEpw+5Mcbg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from PAXPR10MB4734.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:12e::15) by DU0PR10MB5849.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:10:3bf::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.17; Fri, 10 Feb 2023 14:34:03 +0000 Received: from PAXPR10MB4734.EURPRD10.PROD.OUTLOOK.COM ([fe80::ecad:4376:723d:76ee]) by PAXPR10MB4734.EURPRD10.PROD.OUTLOOK.COM ([fe80::ecad:4376:723d:76ee%7]) with mapi id 15.20.6086.019; Fri, 10 Feb 2023 14:34:03 +0000 From: Michael Adler To: isar-users@googlegroups.com Cc: Michael Adler Subject: [PATCH v3] feat: add zstd support to image conversions Date: Fri, 10 Feb 2023 15:33:26 +0100 Message-Id: <20230210143326.267033-1-michael.adler@siemens.com> X-Mailer: git-send-email 2.39.1 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: ZR0P278CA0110.CHEP278.PROD.OUTLOOK.COM (2603:10a6:910:20::7) To PAXPR10MB4734.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:12e::15) Return-Path: michael.adler@siemens.com MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR10MB4734:EE_|DU0PR10MB5849:EE_ X-MS-Office365-Filtering-Correlation-Id: 32b28041-c9cb-46f7-9b4d-08db0b73db57 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: HXLN3hUIGmuPkrfDDpv5iOrl+F1C8X0bn5h9emnMIWcQq4OOyaK8N2HFZNqPv4ZsI6LizBhV+rI5CvZFTrnECjNl5g3T6nqnro1X+0fYshgUV5NuvygQsVELkisReFGDoWtFRsjaJRlrIBjMZ4+AECCghsf4EEFtLTYYDwG6HTr15t0/GQloJNRehvtpPZdDjQuLlpMD7q3ACp2ZTPDP5OxySL6qiX87lKVO5DW6DZeoErXvQWtqophzgPrNHTE4k1Jv9G2U7bhIbvqhEYguSjLQnA5BgW/7zAq3SoQdIEYomNscJXS8O5R4XVfOE1scoL8gWgY8BlwiBkB7n691Guyr80tKJwrkgBJAj7vSTPT1ZNbHaZO77wQB8lXPP/rG2HMmIqII7mhXNoigxegRvqhUzBDw3gMTcw+aiCZ4tItpxZgbOd2czK3kjCaRlwl2v4GYjdvWA427daQbICzvR2XJe5L63jpRrQ9KVgHgEwnAA0Jews5S5VkfzuSY7qwNlhTiZZhH6UmMrBoTu4/EShokUsbTiTwd9yFMrPebczT3tHF0uDmUDE+MNENpYI1NBvrCrtiNjLbzKtLGoWwWkVk7JmtQECElepPvW/ARLLshltOwTTOGQlloEhKHucFuNEkGxrTPZagIf2iqJTEL89gejZcH7UT1/y8NVxoOAbo= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR10MB4734.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230025)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(451199018)(478600001)(6666004)(8676002)(107886003)(82960400001)(36756003)(6486002)(4326008)(1076003)(6506007)(316002)(966005)(66946007)(86362001)(66556008)(26005)(66476007)(6512007)(6916009)(186003)(38100700002)(44832011)(83380400001)(2906002)(8936002)(5660300002)(41300700001)(2616005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?E6cJTFwvfIvI9fAGHJ5NP/Q4i7d3lnZ5jCJmZ6qiRlCenNeAlhn877hS9rpd?= =?us-ascii?Q?b57ow1pb3QxWkoph0pUpe95yq4wE1wCKoYgiRkVoQ1I7KumjE5dxb3t6WeEO?= =?us-ascii?Q?5f2OH6yxCfYD+xCqwhKOThtsQd1sKRhLOBIfV6Lf5lXImuCjM/oXyLO/qjes?= =?us-ascii?Q?XxSUioSMuKP9qc/eT+wPGP/k2SwwWsNk3IkH4670HK3FbGWry02/Yc+VNgjc?= =?us-ascii?Q?IMNtWovuGVtQ1MU9FjwRV8aHUoA6tK5pzTjcEnL2936YI30GpW6WGTwK+sM4?= =?us-ascii?Q?geBdtOGCnIK4h2ZxUainuRiBt5GCcMlgWd5YCNFvEYHnpyhbrsecnBdeJEAg?= =?us-ascii?Q?hpGv9n7sa6kWz+/5tsyDfVnmigu23HT5qYLymnom0xybnNfnNc3rJZntF7Jf?= =?us-ascii?Q?cKNe512FJjtYwqgZF1efUqE7Ta8Yceu1BjD/6XK6/RhyJjaje3DMmyutLeVF?= =?us-ascii?Q?YO+kcMT63dyRDlH8GgyGK5CvlENl7MwY7M3Txp+EOlVGhyA6OIlOQhwye2ZD?= =?us-ascii?Q?z5vFbFICyFlre8uUAP/5F7x3Sqs8kzFo1CDZcKniGt2DKxI/EmQdwKaBzF+8?= =?us-ascii?Q?qgtnvhDQ3ItI5WyxPp8zwcvWB5RRgc1qP8jbHcEutlggqXPRNM21wi+8gewG?= =?us-ascii?Q?uPodk9v7E0/VaH3U/lMCSo5/LrY46mfC9ig63Ry8ep7iVSpoqo3o1cun3X4W?= =?us-ascii?Q?HVxn3UbJTlE37CUX41juh1lRThZAUzzG0sjNKoW0Ba4QWAzkzECmRmH2D+dO?= =?us-ascii?Q?m6doBi4nVC+q6QjPcaIgAlvY6aJJF85me38tcW3ozbsY68crQqyGs5ppH2d1?= =?us-ascii?Q?krT0P4DmykOUUy+r/ET9UgDltiqlHWG507xURcilqL4eL6c6DBfcGC+m8wkJ?= =?us-ascii?Q?r1RMARhzH8Pw3Hx5/CpJO6CK938teHQlWaMSLTlDYPGOV58ijxW6M0/8EiGX?= =?us-ascii?Q?xhZ8G+pVdmnu6ysrlP4BIzpoQelYwGi/AvUqqkV7p5V4iAal+HCS8AiKFyg6?= =?us-ascii?Q?trwtayXwhY0FeTFGSz4uucL2d+6a43igYefXKivG5ODJVynRMEmeKFYD6LAY?= =?us-ascii?Q?0gIQ01O+d8imSJGE6ibd0AA+aEYASWvAzeZhBbtGgVGdW55/WWbBOaB774gV?= =?us-ascii?Q?DADd2bbRXkfiJmdsWIUbAtSPfGaZLOyBX3GfrHjW7XXmg3Mt4dRfi4PyIu/0?= =?us-ascii?Q?d3ZVLnTyR0Hqr0JhgCrvFSjAoB6jlAJV3xz5cKbRc/Apde979Ug4SUKykrtH?= =?us-ascii?Q?7pH3tQyIhW7MlpHpkSCqt+B4PnAR8oDcN+MDipfO4jyoMXUSP0BgBDq3RJlx?= =?us-ascii?Q?q575abhpg8mR6wCpVbL1AsCYyOX5lHtjMzzewGgAygPFeDFtpkas6Hu9Hxlo?= =?us-ascii?Q?8DTKJoSVd9oKGmqsFc//PYtMAPB3VBgPAuTGTmZ4xdXnTUM4u7lWq/B2hkj1?= =?us-ascii?Q?sxoox3ukCRyE7b1HzDI66Bwdn9q9fgVK8ujCSd+nqep/khhvhDHeWw3SgwVZ?= =?us-ascii?Q?+rWA2GzhweEXBdIVvewbdIQUWCgbzvq2j861SpkmHNop0UMTPhwM8jACE7eG?= =?us-ascii?Q?FgnJObJA7xQ0hs9y7rJYIGYboqZfUjGSaigyDzDYi4a3J7NlgYA/FmAtrutg?= =?us-ascii?Q?pQ=3D=3D?= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: 32b28041-c9cb-46f7-9b4d-08db0b73db57 X-MS-Exchange-CrossTenant-AuthSource: PAXPR10MB4734.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Feb 2023 14:34:03.3090 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: zZGlKepSXWoTBPz9YeVtqvXNqtsr4ehzVKBCq3XR09uEmHNUzJ3C5mKkUlDVvbclJiyJ90o0cslUuJLXl8XF40+/D7/rWSTTgaV2bGBkWSs= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR10MB5849 X-TUID: yc9FsHUNhtwh This patch enables ISAR to build zstd-compressed images. ISAR already supports building xz-compressed images but there are several key differences between the two that may make one more suitable than the other, depending on the specific use case: 1. Compression speed: zstd is generally faster than xz when it comes to both compression and decompression times. 2. Compression ratio: "zstd and xz trade blows in their compression ratio. Recompressing all [ArchLinux] packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup." [1] 3. Memory usage: zstd uses a smaller amount of memory compared to xz. [1] https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/ Signed-off-by: Michael Adler --- meta/classes/imagetypes.bbclass | 5 ++++- meta/conf/bitbake.conf | 3 +++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/meta/classes/imagetypes.bbclass b/meta/classes/imagetypes.bbclass index 0cff440..c2033fd 100644 --- a/meta/classes/imagetypes.bbclass +++ b/meta/classes/imagetypes.bbclass @@ -93,10 +93,13 @@ IMAGE_CMD:ubi() { IMAGE_CMD:ubi[depends] = "${PN}:do_transform_template" # image conversions -IMAGE_CONVERSIONS = "gz xz" +IMAGE_CONVERSIONS = "gz xz zst" CONVERSION_CMD:gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'" CONVERSION_DEPS:gz = "gzip" CONVERSION_CMD:xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" CONVERSION_DEPS:xz = "xz-utils" + +CONVERSION_CMD:zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZSTD_DEFAULTS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'" +CONVERSION_DEPS:zst = "zstd" diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 522241a..c9f52a8 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -140,6 +140,9 @@ XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" # Default parallelism for zstd ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" ZSTD_THREADS[vardepvalue] = "1" +ZSTD_LEVEL ?= "19" +ZSTD_DEFAULTS ?= "-${ZSTD_LEVEL} --threads=${ZSTD_THREADS}" +ZSTD_DEFAULTS[vardepsexclude] += "ZSTD_LEVEL ZSTD_THREADS" BBINCLUDELOGS ??= "yes" -- 2.39.1