[PATCH v3] feat: add zstd support to image conversions

[PATCH v3] feat: add zstd support to image conversions

[Michael Adler]

This patch enables ISAR to build zstd-compressed images.

ISAR already supports building xz-compressed images but there are several key
differences between the two that may make one more suitable than the other,
depending on the specific use case:

1. Compression speed: zstd is generally faster than xz when it comes to both
compression and decompression times.

2. Compression ratio: "zstd and xz trade blows in their compression ratio.
Recompressing all [ArchLinux] packages to zstd with our options yields a total
~0.8% increase in package size on all of our packages combined, but the
decompression time for all packages saw a ~1300% speedup." [1]

3. Memory usage: zstd uses a smaller amount of memory compared to xz.

[1] https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/

Signed-off-by: Michael Adler <michael.adler@siemens.com>
---
 meta/classes/imagetypes.bbclass | 5 ++++-
 meta/conf/bitbake.conf          | 3 +++
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/meta/classes/imagetypes.bbclass b/meta/classes/imagetypes.bbclass
index 0cff440..c2033fd 100644
--- a/meta/classes/imagetypes.bbclass
+++ b/meta/classes/imagetypes.bbclass
@@ -93,10 +93,13 @@ IMAGE_CMD:ubi() {
 IMAGE_CMD:ubi[depends] = "${PN}:do_transform_template"
 
 # image conversions
-IMAGE_CONVERSIONS = "gz xz"
+IMAGE_CONVERSIONS = "gz xz zst"
 
 CONVERSION_CMD:gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'"
 CONVERSION_DEPS:gz = "gzip"
 
 CONVERSION_CMD:xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'"
 CONVERSION_DEPS:xz = "xz-utils"
+
+CONVERSION_CMD:zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZSTD_DEFAULTS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'"
+CONVERSION_DEPS:zst = "zstd"
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 522241a..c9f52a8 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -140,6 +140,9 @@ XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
 # Default parallelism for zstd
 ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
 ZSTD_THREADS[vardepvalue] = "1"
+ZSTD_LEVEL ?= "19"
+ZSTD_DEFAULTS ?= "-${ZSTD_LEVEL} --threads=${ZSTD_THREADS}"
+ZSTD_DEFAULTS[vardepsexclude] += "ZSTD_LEVEL ZSTD_THREADS"
 
 BBINCLUDELOGS ??= "yes"
 
-- 
2.39.1

Re: [PATCH v3] feat: add zstd support to image conversions

[Uladzimir Bely]

In the email from Friday, 10 February 2023 17:33:26 +03 user Michael Adler 
wrote:
> This patch enables ISAR to build zstd-compressed images.
> 
> ISAR already supports building xz-compressed images but there are several
> key differences between the two that may make one more suitable than the
> other, depending on the specific use case:
> 
> 1. Compression speed: zstd is generally faster than xz when it comes to both
> compression and decompression times.
> 
> 2. Compression ratio: "zstd and xz trade blows in their compression ratio.
> Recompressing all [ArchLinux] packages to zstd with our options yields a
> total ~0.8% increase in package size on all of our packages combined, but
> the decompression time for all packages saw a ~1300% speedup." [1]
> 
> 3. Memory usage: zstd uses a smaller amount of memory compared to xz.
> 
> [1]
> https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-co
> mpression/
> 
> Signed-off-by: Michael Adler <michael.adler@siemens.com>
> ---
>  meta/classes/imagetypes.bbclass | 5 ++++-
>  meta/conf/bitbake.conf          | 3 +++
>  2 files changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/imagetypes.bbclass
> b/meta/classes/imagetypes.bbclass index 0cff440..c2033fd 100644
> --- a/meta/classes/imagetypes.bbclass
> +++ b/meta/classes/imagetypes.bbclass
> @@ -93,10 +93,13 @@ IMAGE_CMD:ubi() {
>  IMAGE_CMD:ubi[depends] = "${PN}:do_transform_template"
> 
>  # image conversions
> -IMAGE_CONVERSIONS = "gz xz"
> +IMAGE_CONVERSIONS = "gz xz zst"
> 
>  CONVERSION_CMD:gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable
> ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'" CONVERSION_DEPS:gz =
> "gzip"
> 
>  CONVERSION_CMD:xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS}
> ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" CONVERSION_DEPS:xz =
> "xz-utils"
> +
> +CONVERSION_CMD:zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZSTD_DEFAULTS}
> ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'" +CONVERSION_DEPS:zst =
> "zstd"
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 522241a..c9f52a8 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -140,6 +140,9 @@ XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
>  # Default parallelism for zstd
>  ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
>  ZSTD_THREADS[vardepvalue] = "1"
> +ZSTD_LEVEL ?= "19"
> +ZSTD_DEFAULTS ?= "-${ZSTD_LEVEL} --threads=${ZSTD_THREADS}"
> +ZSTD_DEFAULTS[vardepsexclude] += "ZSTD_LEVEL ZSTD_THREADS"
> 
>  BBINCLUDELOGS ??= "yes"

Applied to next, thanks.

Re: [PATCH v3] feat: add zstd support to image conversions

[Henning Schild]

Thanks. After seeing this i asked myself if etcher can handle that,
because that is the tool we often tell Windows/Mac and even Linux users
to use for flashing.

Seems it does not yet
https://github.com/balena-io/etcher/issues/2856

Henning

Am Fri, 10 Feb 2023 15:33:26 +0100
schrieb Michael Adler <michael.adler@siemens.com>:

> This patch enables ISAR to build zstd-compressed images.
> 
> ISAR already supports building xz-compressed images but there are
> several key differences between the two that may make one more
> suitable than the other, depending on the specific use case:
> 
> 1. Compression speed: zstd is generally faster than xz when it comes
> to both compression and decompression times.
> 
> 2. Compression ratio: "zstd and xz trade blows in their compression
> ratio. Recompressing all [ArchLinux] packages to zstd with our
> options yields a total ~0.8% increase in package size on all of our
> packages combined, but the decompression time for all packages saw a
> ~1300% speedup." [1]
> 
> 3. Memory usage: zstd uses a smaller amount of memory compared to xz.
> 
> [1]
> https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/
> 
> Signed-off-by: Michael Adler <michael.adler@siemens.com>
> ---
>  meta/classes/imagetypes.bbclass | 5 ++++-
>  meta/conf/bitbake.conf          | 3 +++
>  2 files changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/imagetypes.bbclass
> b/meta/classes/imagetypes.bbclass index 0cff440..c2033fd 100644
> --- a/meta/classes/imagetypes.bbclass
> +++ b/meta/classes/imagetypes.bbclass
> @@ -93,10 +93,13 @@ IMAGE_CMD:ubi() {
>  IMAGE_CMD:ubi[depends] = "${PN}:do_transform_template"
>  
>  # image conversions
> -IMAGE_CONVERSIONS = "gz xz"
> +IMAGE_CONVERSIONS = "gz xz zst"
>  
>  CONVERSION_CMD:gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c
> --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'"
> CONVERSION_DEPS:gz = "gzip" 
>  CONVERSION_CMD:xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS}
> ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" CONVERSION_DEPS:xz =
> "xz-utils" +
> +CONVERSION_CMD:zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZSTD_DEFAULTS}
> ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'"
> +CONVERSION_DEPS:zst = "zstd" diff --git a/meta/conf/bitbake.conf
> b/meta/conf/bitbake.conf index 522241a..c9f52a8 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -140,6 +140,9 @@ XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT
> XZ_THREADS" # Default parallelism for zstd
>  ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
>  ZSTD_THREADS[vardepvalue] = "1"
> +ZSTD_LEVEL ?= "19"
> +ZSTD_DEFAULTS ?= "-${ZSTD_LEVEL} --threads=${ZSTD_THREADS}"
> +ZSTD_DEFAULTS[vardepsexclude] += "ZSTD_LEVEL ZSTD_THREADS"
>  
>  BBINCLUDELOGS ??= "yes"
>