[PATCH v2] use xz and gzip on host (outside chroot)

[PATCH v2] use xz and gzip on host (outside chroot)

[Felix Moessbauer]

From: Adriaan Schmidt <adriaan.schmidt@siemens.com>

This patch significantly speeds up the final image compression
as the compression itself is no longer emulated.

For gzip, no additional dep has to be installed on the system,
for xz the xz-utils package has to be installed but this will
be required from bitbake 2.0 on anyways.

Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
Acked-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 doc/user_manual.md              | 3 ++-
 meta/classes/imagetypes.bbclass | 8 ++++----
 2 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 546ae75c..932bfbe5 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -83,7 +83,8 @@ apt install \
   reprepro \
   sudo \
   git-buildpackage \
-  pristine-tar
+  pristine-tar \
+  xz-utils
 ```
 
 Additional setup is required since `sbuild` is now used for package build.
diff --git a/meta/classes/imagetypes.bbclass b/meta/classes/imagetypes.bbclass
index 1e8e223d..319e9dc8 100644
--- a/meta/classes/imagetypes.bbclass
+++ b/meta/classes/imagetypes.bbclass
@@ -95,13 +95,13 @@ IMAGE_CMD_ubi[depends] = "${PN}:do_transform_template"
 # image conversions
 IMAGE_CONVERSIONS = "gz xz"
 
-CONVERSION_CMD_gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'"
-CONVERSION_DEPS_gz = "gzip"
+CONVERSION_CMD_gz ?= "gzip -f -9 -n -c --rsyncable ${IMAGE_FILE_HOST} > ${IMAGE_FILE_HOST}.gz"
+CONVERSION_DEPS_gz ?= ""
 
 XZ_MEMLIMIT ?= "50%"
 XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
 XZ_THREADS[vardepvalue] = "1"
 XZ_OPTIONS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
 XZ_OPTIONS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
-CONVERSION_CMD_xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'"
-CONVERSION_DEPS_xz = "xz-utils"
+CONVERSION_CMD_xz ?= "xz -c ${XZ_OPTIONS} ${IMAGE_FILE_HOST} > ${IMAGE_FILE_HOST}.xz"
+CONVERSION_DEPS_xz ?= ""
-- 
2.30.2

Re: [PATCH v2] use xz and gzip on host (outside chroot)

[Henning Schild]

Am Mon,  8 Aug 2022 09:36:12 +0200
schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:

> From: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> 
> This patch significantly speeds up the final image compression
> as the compression itself is no longer emulated.
> 
> For gzip, no additional dep has to be installed on the system,
> for xz the xz-utils package has to be installed but this will
> be required from bitbake 2.0 on anyways.
> 
> Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> Acked-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>  doc/user_manual.md              | 3 ++-
>  meta/classes/imagetypes.bbclass | 8 ++++----
>  2 files changed, 6 insertions(+), 5 deletions(-)
> 
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 546ae75c..932bfbe5 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -83,7 +83,8 @@ apt install \
>    reprepro \
>    sudo \
>    git-buildpackage \
> -  pristine-tar
> +  pristine-tar \
> +  xz-utils
>  ```
>  
>  Additional setup is required since `sbuild` is now used for package
> build. diff --git a/meta/classes/imagetypes.bbclass
> b/meta/classes/imagetypes.bbclass index 1e8e223d..319e9dc8 100644
> --- a/meta/classes/imagetypes.bbclass
> +++ b/meta/classes/imagetypes.bbclass
> @@ -95,13 +95,13 @@ IMAGE_CMD_ubi[depends] =
> "${PN}:do_transform_template" # image conversions
>  IMAGE_CONVERSIONS = "gz xz"
>  
> -CONVERSION_CMD_gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c
> --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'"
> -CONVERSION_DEPS_gz = "gzip" +CONVERSION_CMD_gz ?= "gzip -f -9 -n -c
> --rsyncable ${IMAGE_FILE_HOST} > ${IMAGE_FILE_HOST}.gz"
> +CONVERSION_DEPS_gz ?= "" 

No. Please only add the ? so you can take the shortcut in your layer,
without changing the conservative and slow isar defaults.

Henning

>  XZ_MEMLIMIT ?= "50%"
>  XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
>  XZ_THREADS[vardepvalue] = "1"
>  XZ_OPTIONS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
>  XZ_OPTIONS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
> -CONVERSION_CMD_xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS}
> ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" -CONVERSION_DEPS_xz
> = "xz-utils" +CONVERSION_CMD_xz ?= "xz -c ${XZ_OPTIONS}
> ${IMAGE_FILE_HOST} > ${IMAGE_FILE_HOST}.xz" +CONVERSION_DEPS_xz ?= ""

Re: [PATCH v2] use xz and gzip on host (outside chroot)

[Baurzhan Ismagulov]

On Mon, Aug 08, 2022 at 05:30:49PM +0200, Henning Schild wrote:
> > diff --git a/meta/classes/imagetypes.bbclass b/meta/classes/imagetypes.bbclass index 1e8e223d..319e9dc8 100644
> > --- a/meta/classes/imagetypes.bbclass
> > +++ b/meta/classes/imagetypes.bbclass
> > @@ -95,13 +95,13 @@ IMAGE_CMD_ubi[depends] =
> > "${PN}:do_transform_template" # image conversions
> >  IMAGE_CONVERSIONS = "gz xz"
> >  
> > -CONVERSION_CMD_gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c
> > --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'"
> > -CONVERSION_DEPS_gz = "gzip" +CONVERSION_CMD_gz ?= "gzip -f -9 -n -c
> > --rsyncable ${IMAGE_FILE_HOST} > ${IMAGE_FILE_HOST}.gz"
> > +CONVERSION_DEPS_gz ?= "" 
> 
> No. Please only add the ? so you can take the shortcut in your layer,
> without changing the conservative and slow isar defaults.

I agree in principle that such cases occur from time to time. I've seen mke2fs
producing filesystems that were unreadable by older kernels due to some flags
and compression tool option differences (the latter not applying in this case).

In practice, I think such cases are quite rare; for this potential
compatibility, we'd pay performance penalty every day. So, I'd like to benefit
from this finding, in Isar, by default. If we can always switch back to the
slower variant, that sounds as Good Enough for me.

An alternative could be to compress from the host chroot regardless of the
cross-compilation setting. While at it, we could also evaluate pigz (it would
be interesting to see some benchmarks on the list) and recheck the reasons why
cross-compilation was disabled for the imager -- maybe it's solvable this time.

With kind regards,
Baurzhan

Re: [PATCH v2] use xz and gzip on host (outside chroot)

[Anton Mikanovich]

08.08.2022 10:36, Felix Moessbauer wrote:
> From: Adriaan Schmidt <adriaan.schmidt@siemens.com>
>
> This patch significantly speeds up the final image compression
> as the compression itself is no longer emulated.
>
> For gzip, no additional dep has to be installed on the system,
> for xz the xz-utils package has to be installed but this will
> be required from bitbake 2.0 on anyways.
>
> Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> Acked-by: Felix Moessbauer <felix.moessbauer@siemens.com>

Hello,

I've made some investigations to collect compression speed statistics on
qemuarm64-bullseye Isar target with 490Mb rootfs. Here are the results:

XZ:
Buildchroot xz (QEMU):   45.305s
Host xz:                 18.783s

GZIP:
Buildchroot gzip (QEMU):   4m30.155s
Host gzip:                 1m30.135s
Buildchroot gzip (native): 1m27.162s

Buildchroot pigz (QEMU):     32.265s
Host pigz:                   13.131s

So it looks reasonable to use native compression and we really need to think
about removing ISAR_CROSS_COMPILE="0" for imager.
We also can try to use parallelized pigz for gzip compression.

RE: [PATCH v2] use xz and gzip on host (outside chroot)

[Moessbauer, Felix]

> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
> Behalf Of Anton Mikanovich
> Sent: Tuesday, August 16, 2022 5:11 PM
> To: Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>;
> isar-users@googlegroups.com
> Cc: Schild, Henning (T CED SES-DE) <henning.schild@siemens.com>; Schmidt,
> Adriaan (T CED SES-DE) <adriaan.schmidt@siemens.com>; Baurzhan
> Ismagulov <ibr@ilbers.de>
> Subject: Re: [PATCH v2] use xz and gzip on host (outside chroot)
> 
> 08.08.2022 10:36, Felix Moessbauer wrote:
> > From: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> >
> > This patch significantly speeds up the final image compression as the
> > compression itself is no longer emulated.
> >
> > For gzip, no additional dep has to be installed on the system, for xz
> > the xz-utils package has to be installed but this will be required
> > from bitbake 2.0 on anyways.
> >
> > Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> > Acked-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> 
> Hello,
> 
> I've made some investigations to collect compression speed statistics on
> qemuarm64-bullseye Isar target with 490Mb rootfs. Here are the results:
> 
> XZ:
> Buildchroot xz (QEMU):   45.305s
> Host xz:                 18.783s
> 
> GZIP:
> Buildchroot gzip (QEMU):   4m30.155s
> Host gzip:                 1m30.135s
> Buildchroot gzip (native): 1m27.162s
> 
> Buildchroot pigz (QEMU):     32.265s
> Host pigz:                   13.131s

While this already shows the significant speedup, I saw much larger ones when using xz on multi GB images.

> 
> So it looks reasonable to use native compression and we really need to think
> about removing ISAR_CROSS_COMPILE="0" for imager.

I just gave that a try and it somehow magically worked out of the box (on my arm64 image).
Don't know why this was added in the first place.
Let's add Claudius to the discussion as he introduced that.

Just running the compression in the buildchroot-host is not as easy as many things are tightly coupled with the imager (like installing the compression utilities).
But just switching ISAR_CROSS_COMPILE is even simpler and further reduces the amount of work that has to be emulated.

Felix

> We also can try to use parallelized pigz for gzip compression.
>