[PATCH] feat: add zstd support to image conversions

[PATCH] feat: add zstd support to image conversions

[Michael Adler]

This patch enables ISAR to build zstd compressed images.

Signed-off-by: Michael Adler <michael.adler@siemens.com>
---
 meta/classes/imagetypes.bbclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/meta/classes/imagetypes.bbclass b/meta/classes/imagetypes.bbclass
index 1e8e223d..6950da7a 100644
--- a/meta/classes/imagetypes.bbclass
+++ b/meta/classes/imagetypes.bbclass
@@ -93,7 +93,7 @@ IMAGE_CMD_ubi() {
 IMAGE_CMD_ubi[depends] = "${PN}:do_transform_template"
 
 # image conversions
-IMAGE_CONVERSIONS = "gz xz"
+IMAGE_CONVERSIONS = "gz xz zst"
 
 CONVERSION_CMD_gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'"
 CONVERSION_DEPS_gz = "gzip"
@@ -105,3 +105,11 @@ XZ_OPTIONS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
 XZ_OPTIONS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
 CONVERSION_CMD_xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'"
 CONVERSION_DEPS_xz = "xz-utils"
+
+ZST_LEVEL ?= "19"
+ZST_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
+ZST_THREADS[vardepvalue] = "1"
+ZST_OPTIONS ?= "-${ZST_LEVEL} --threads=${ZST_THREADS}"
+ZST_OPTIONS[vardepsexclude] += "ZST_LEVEL ZST_THREADS"
+CONVERSION_CMD_zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZST_OPTIONS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'"
+CONVERSION_DEPS_zst = "zstd"
-- 
2.39.1

Re: [PATCH] feat: add zstd support to image conversions

[Jan Kiszka]

On 23.01.23 15:00, Michael Adler wrote:
> This patch enables ISAR to build zstd compressed images.
> 

I was briefly thinking you had solved the dpkg vs. zstd issue - just a
dream :)

Jan

> Signed-off-by: Michael Adler <michael.adler@siemens.com>
> ---
>  meta/classes/imagetypes.bbclass | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/imagetypes.bbclass b/meta/classes/imagetypes.bbclass
> index 1e8e223d..6950da7a 100644
> --- a/meta/classes/imagetypes.bbclass
> +++ b/meta/classes/imagetypes.bbclass
> @@ -93,7 +93,7 @@ IMAGE_CMD_ubi() {
>  IMAGE_CMD_ubi[depends] = "${PN}:do_transform_template"
>  
>  # image conversions
> -IMAGE_CONVERSIONS = "gz xz"
> +IMAGE_CONVERSIONS = "gz xz zst"
>  
>  CONVERSION_CMD_gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'"
>  CONVERSION_DEPS_gz = "gzip"
> @@ -105,3 +105,11 @@ XZ_OPTIONS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
>  XZ_OPTIONS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
>  CONVERSION_CMD_xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'"
>  CONVERSION_DEPS_xz = "xz-utils"
> +
> +ZST_LEVEL ?= "19"
> +ZST_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
> +ZST_THREADS[vardepvalue] = "1"
> +ZST_OPTIONS ?= "-${ZST_LEVEL} --threads=${ZST_THREADS}"
> +ZST_OPTIONS[vardepsexclude] += "ZST_LEVEL ZST_THREADS"
> +CONVERSION_CMD_zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZST_OPTIONS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'"
> +CONVERSION_DEPS_zst = "zstd"

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH] feat: add zstd support to image conversions

[Roberto A. Foglietta]

[-- Attachment #1: Type: text/plain, Size: 1461 bytes --]

Il Lun 23 Gen 2023, 20:22 Jan Kiszka <jan.kiszka@siemens.com> ha scritto:

> On 23.01.23 15:00, Michael Adler wrote:
> > This patch enables ISAR to build zstd compressed images.
> >
>
> I was briefly thinking you had solved the dpkg vs. zstd issue - just a
> dream :)
>

Hi Jan.

I think you are speaking about zstd compression for the debian package that
is 1% bigger but 13x times faster in expanding. [1]

Ubuntu and Arch adopted zstd but not debian which is still using xz and as
far as I have seen neither in parallel using option -T.

I think you should ask them to produce all package with

export XZ_DEFAULTS='-T 8'

and enabling dpkg to set internally

XZ_DEFAULTS='-T 0'

unless it is differently set. What Do you think about changing ISAR to use
zstd or xz -T without repackaging everything at fetch time?

It would be a solution: fetch, repackage with zstd and the downloads remain
for a long time like a permanent cache and also the benefits.

The counter effect is that those debian packages and those produced by ISAR
will be not anymore compatible with a standard debian and this could be a
nasty surprise in some cases for some users.

The alternative is to repackage everything with xz parallel. Usually a
modern laptop has 8 pipes but I have the sensation that with -T 0 it counts
just the cores because on my machine spawn 4 not 8.

Best regards, R-

[1]
https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/

[-- Attachment #2: Type: text/html, Size: 2303 bytes --]

Re: [PATCH] feat: add zstd support to image conversions

[Roberto A. Foglietta]

On Tue, 24 Jan 2023 at 04:17, Roberto A. Foglietta
<roberto.foglietta@gmail.com> wrote:
>
> Il Lun 23 Gen 2023, 20:22 Jan Kiszka <jan.kiszka@siemens.com> ha scritto:
>>
>> On 23.01.23 15:00, Michael Adler wrote:
>> > This patch enables ISAR to build zstd compressed images.
>> >
>>
>> I was briefly thinking you had solved the dpkg vs. zstd issue - just a
>> dream :)
>
>
> Hi Jan.
>
> I think you are speaking about zstd compression for the debian package that is 1% bigger but 13x times faster in expanding. [1]

[...]

>
> The alternative is to repackage everything with xz parallel. Usually a modern laptop has 8 pipes but I have the sensation that with -T 0 it counts just the cores because on my machine spawn 4 not 8.

Forget about the last 4 instead of 8 about xz. Every compressor is
doing the same on my laptop and it is a complete novelty. Possibly the
newer versions spawn with limitation to not clog the machine.

Yes, all 8 CPUs (4 cores X 2 threads) are online and account for every
command that gives access to them (lscpu, proc, nproc, htop, etc.).
Something similar on your machines?

Best regards, R-

Re: [PATCH] feat: add zstd support to image conversions

[Uladzimir Bely]

In the email from Monday, 23 January 2023 17:00:16 +03 user Michael Adler 
wrote:
> This patch enables ISAR to build zstd compressed images.
> 
> Signed-off-by: Michael Adler <michael.adler@siemens.com>
> ---
>  meta/classes/imagetypes.bbclass | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/imagetypes.bbclass
> b/meta/classes/imagetypes.bbclass index 1e8e223d..6950da7a 100644
> --- a/meta/classes/imagetypes.bbclass
> +++ b/meta/classes/imagetypes.bbclass
> @@ -93,7 +93,7 @@ IMAGE_CMD_ubi() {
>  IMAGE_CMD_ubi[depends] = "${PN}:do_transform_template"
> 
>  # image conversions
> -IMAGE_CONVERSIONS = "gz xz"
> +IMAGE_CONVERSIONS = "gz xz zst"
> 
>  CONVERSION_CMD_gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable
> ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'" CONVERSION_DEPS_gz =
> "gzip"
> @@ -105,3 +105,11 @@ XZ_OPTIONS ?= "--memlimit=${XZ_MEMLIMIT}
> --threads=${XZ_THREADS}" XZ_OPTIONS[vardepsexclude] += "XZ_MEMLIMIT
> XZ_THREADS"
>  CONVERSION_CMD_xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS}
> ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" CONVERSION_DEPS_xz =
> "xz-utils"
> +
> +ZST_LEVEL ?= "19"
> +ZST_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
> +ZST_THREADS[vardepvalue] = "1"
> +ZST_OPTIONS ?= "-${ZST_LEVEL} --threads=${ZST_THREADS}"
> +ZST_OPTIONS[vardepsexclude] += "ZST_LEVEL ZST_THREADS"
> +CONVERSION_CMD_zst = "${SUDO_CHROOT} sh -c 'zstd -c ${ZST_OPTIONS}

Since we switched to Bitbake 2.0, we need v2 with using new syntax.

Of course, in case we need this feature at all...

> ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'" +CONVERSION_DEPS_zst =
> "zstd"