Re: [PATCH v2] feat: add zstd support to image conversions

["Moessbauer, Felix" <felix.moessbauer@siemens.com>]

On Fri, 2023-02-10 at 14:45 +0100, Michael Adler wrote:
> This patch enables ISAR to build zstd-compressed images.
> 
> ISAR already supports building xz-compressed images but there are
> several key
> differences between the two that may make one more suitable than the
> other,
> depending on the specific use case:
> 
> 1. Compression speed: zstd is generally faster than xz when it comes
> to both
> compression and decompression times.
> 
> 2. Compression ratio: "zstd and xz trade blows in their compression
> ratio.
> Recompressing all [ArchLinux] packages to zstd with our options
> yields a total
> ~0.8% increase in package size on all of our packages combined, but
> the
> decompression time for all packages saw a ~1300% speedup." [1]
> 
> 3. Memory usage: zstd uses a smaller amount of memory compared to xz.
> 
> [1] 
> https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/
> 
> Signed-off-by: Michael Adler <michael.adler@siemens.com>
> 
> ---
>  meta/classes/imagetypes.bbclass | 5 ++++-
>  meta/conf/bitbake.conf          | 3 +++
>  2 files changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/imagetypes.bbclass
> b/meta/classes/imagetypes.bbclass
> index 0cff440..f11ddc6 100644
> --- a/meta/classes/imagetypes.bbclass
> +++ b/meta/classes/imagetypes.bbclass
> @@ -93,10 +93,13 @@ IMAGE_CMD:ubi() {
>  IMAGE_CMD:ubi[depends] = "${PN}:do_transform_template"
>  
>  # image conversions
> -IMAGE_CONVERSIONS = "gz xz"
> +IMAGE_CONVERSIONS = "gz xz zst"
>  
>  CONVERSION_CMD:gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --
> rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'"
>  CONVERSION_DEPS:gz = "gzip"
>  
>  CONVERSION_CMD:xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS}
> ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'"
>  CONVERSION_DEPS:xz = "xz-utils"
> +
> +CONVERSION_CMD:zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZSTD_OPTIONS}
> ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'"
> +CONVERSION_DEPS:zst = "zstd"
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 522241a..a1bad71 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -140,6 +140,9 @@ XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT
> XZ_THREADS"
>  # Default parallelism for zstd
>  ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
>  ZSTD_THREADS[vardepvalue] = "1"
> +ZSTD_LEVEL ?= "19"
> +ZSTD_OPTIONS ?= "-${ZSTD_LEVEL} --threads=${ZSTD_THREADS}"
> +ZSTD_OPTIONS[vardepsexclude] += "ZSTD_LEVEL ZSTD_THREADS"

This probably also needs to be renamed to ZSTD_DEFAULTS to be
consistent with XZ_DEFAULTS. The details have been discussed in
"imagetypes XZ_OPTIONS replaced by XZ_DEFAULTS defined in
bitbake.comf".

Please double check.

Felix

>  
>  BBINCLUDELOGS ??= "yes"
>  
> -- 
> 2.39.1
>