[PATCH v2] feat: add zstd support to image conversions

[PATCH v2] feat: add zstd support to image conversions

[Michael Adler]

This patch enables ISAR to build zstd-compressed images.

ISAR already supports building xz-compressed images but there are several key
differences between the two that may make one more suitable than the other,
depending on the specific use case:

1. Compression speed: zstd is generally faster than xz when it comes to both
compression and decompression times.

2. Compression ratio: "zstd and xz trade blows in their compression ratio.
Recompressing all [ArchLinux] packages to zstd with our options yields a total
~0.8% increase in package size on all of our packages combined, but the
decompression time for all packages saw a ~1300% speedup." [1]

3. Memory usage: zstd uses a smaller amount of memory compared to xz.

[1] https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/

Signed-off-by: Michael Adler <michael.adler@siemens.com>

---
 meta/classes/imagetypes.bbclass | 5 ++++-
 meta/conf/bitbake.conf          | 3 +++
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/meta/classes/imagetypes.bbclass b/meta/classes/imagetypes.bbclass
index 0cff440..f11ddc6 100644
--- a/meta/classes/imagetypes.bbclass
+++ b/meta/classes/imagetypes.bbclass
@@ -93,10 +93,13 @@ IMAGE_CMD:ubi() {
 IMAGE_CMD:ubi[depends] = "${PN}:do_transform_template"
 
 # image conversions
-IMAGE_CONVERSIONS = "gz xz"
+IMAGE_CONVERSIONS = "gz xz zst"
 
 CONVERSION_CMD:gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'"
 CONVERSION_DEPS:gz = "gzip"
 
 CONVERSION_CMD:xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'"
 CONVERSION_DEPS:xz = "xz-utils"
+
+CONVERSION_CMD:zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZSTD_OPTIONS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'"
+CONVERSION_DEPS:zst = "zstd"
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 522241a..a1bad71 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -140,6 +140,9 @@ XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
 # Default parallelism for zstd
 ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
 ZSTD_THREADS[vardepvalue] = "1"
+ZSTD_LEVEL ?= "19"
+ZSTD_OPTIONS ?= "-${ZSTD_LEVEL} --threads=${ZSTD_THREADS}"
+ZSTD_OPTIONS[vardepsexclude] += "ZSTD_LEVEL ZSTD_THREADS"
 
 BBINCLUDELOGS ??= "yes"
 
-- 
2.39.1

Re: [PATCH v2] feat: add zstd support to image conversions

[Moessbauer, Felix]

On Fri, 2023-02-10 at 14:45 +0100, Michael Adler wrote:
> This patch enables ISAR to build zstd-compressed images.
> 
> ISAR already supports building xz-compressed images but there are
> several key
> differences between the two that may make one more suitable than the
> other,
> depending on the specific use case:
> 
> 1. Compression speed: zstd is generally faster than xz when it comes
> to both
> compression and decompression times.
> 
> 2. Compression ratio: "zstd and xz trade blows in their compression
> ratio.
> Recompressing all [ArchLinux] packages to zstd with our options
> yields a total
> ~0.8% increase in package size on all of our packages combined, but
> the
> decompression time for all packages saw a ~1300% speedup." [1]
> 
> 3. Memory usage: zstd uses a smaller amount of memory compared to xz.
> 
> [1] 
> https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/
> 
> Signed-off-by: Michael Adler <michael.adler@siemens.com>
> 
> ---
>  meta/classes/imagetypes.bbclass | 5 ++++-
>  meta/conf/bitbake.conf          | 3 +++
>  2 files changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/imagetypes.bbclass
> b/meta/classes/imagetypes.bbclass
> index 0cff440..f11ddc6 100644
> --- a/meta/classes/imagetypes.bbclass
> +++ b/meta/classes/imagetypes.bbclass
> @@ -93,10 +93,13 @@ IMAGE_CMD:ubi() {
>  IMAGE_CMD:ubi[depends] = "${PN}:do_transform_template"
>  
>  # image conversions
> -IMAGE_CONVERSIONS = "gz xz"
> +IMAGE_CONVERSIONS = "gz xz zst"
>  
>  CONVERSION_CMD:gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --
> rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'"
>  CONVERSION_DEPS:gz = "gzip"
>  
>  CONVERSION_CMD:xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS}
> ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'"
>  CONVERSION_DEPS:xz = "xz-utils"
> +
> +CONVERSION_CMD:zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZSTD_OPTIONS}
> ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'"
> +CONVERSION_DEPS:zst = "zstd"
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 522241a..a1bad71 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -140,6 +140,9 @@ XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT
> XZ_THREADS"
>  # Default parallelism for zstd
>  ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
>  ZSTD_THREADS[vardepvalue] = "1"
> +ZSTD_LEVEL ?= "19"
> +ZSTD_OPTIONS ?= "-${ZSTD_LEVEL} --threads=${ZSTD_THREADS}"
> +ZSTD_OPTIONS[vardepsexclude] += "ZSTD_LEVEL ZSTD_THREADS"

This probably also needs to be renamed to ZSTD_DEFAULTS to be
consistent with XZ_DEFAULTS. The details have been discussed in
"imagetypes XZ_OPTIONS replaced by XZ_DEFAULTS defined in
bitbake.comf".

Please double check.

Felix

>  
>  BBINCLUDELOGS ??= "yes"
>  
> -- 
> 2.39.1
> 

Re: [PATCH v2] feat: add zstd support to image conversions

[Michael Adler]

Hi Felix,

> This probably also needs to be renamed to ZSTD_DEFAULTS to be
> consistent with XZ_DEFAULTS.

I agree that it probably makes sense to use the same naming scheme
(<algo>_DEFAULTS) for both xz and zstd. Thanks for the heads up, I will send
v3 in a moment.

> The details have been discussed in "imagetypes XZ_OPTIONS replaced by
> XZ_DEFAULTS defined in bitbake.comf".

well, but in my case it's correct because I use the same variable name
(ZSTD_OPTIONS) in both files :)

Kind Regards,
  Michael

-- 
Michael Adler

Siemens AG
T CED SES-DE
Otto-Hahn-Ring 6
81739 M�nchen, Deutschland

Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Jim Hagemann
Snabe; Vorstand: Roland Busch, Vorsitzender; Klaus Helmrich, Cedrik Neike,
Matthias Rebellius, Ralf P. Thomas, Judith Wiese; Sitz der Gesellschaft:
Berlin und M�nchen, Deutschland; Registergericht: Berlin-Charlottenburg, HRB
12300, M�nchen, HRB 6684; WEEE-Reg.-Nr. DE 23691322