[PATCH] use xz and gzip on host (outside chroot)

[PATCH] use xz and gzip on host (outside chroot)

[Felix Moessbauer]

From: Adriaan Schmidt <adriaan.schmidt@siemens.com>

This patch significantly speeds up the final image compression
as the compression itself is no longer emulated.

For gzip, no additional dep has to be installed on the system,
for xz the xz-utils package has to be installed but this will
be required from bitbake 2.0 on anyways.

Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
Acked-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 doc/user_manual.md              | 3 ++-
 meta/classes/imagetypes.bbclass | 6 ++----
 2 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 546ae75c..932bfbe5 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -83,7 +83,8 @@ apt install \
   reprepro \
   sudo \
   git-buildpackage \
-  pristine-tar
+  pristine-tar \
+  xz-utils
 ```
 
 Additional setup is required since `sbuild` is now used for package build.
diff --git a/meta/classes/imagetypes.bbclass b/meta/classes/imagetypes.bbclass
index 1e8e223d..c4b06ca6 100644
--- a/meta/classes/imagetypes.bbclass
+++ b/meta/classes/imagetypes.bbclass
@@ -95,13 +95,11 @@ IMAGE_CMD_ubi[depends] = "${PN}:do_transform_template"
 # image conversions
 IMAGE_CONVERSIONS = "gz xz"
 
-CONVERSION_CMD_gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'"
-CONVERSION_DEPS_gz = "gzip"
+CONVERSION_CMD_gz = "gzip -f -9 -n -c --rsyncable ${IMAGE_FILE_HOST} > ${IMAGE_FILE_HOST}.gz"
 
 XZ_MEMLIMIT ?= "50%"
 XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
 XZ_THREADS[vardepvalue] = "1"
 XZ_OPTIONS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
 XZ_OPTIONS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
-CONVERSION_CMD_xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'"
-CONVERSION_DEPS_xz = "xz-utils"
+CONVERSION_CMD_xz = "xz -c ${XZ_OPTIONS} ${IMAGE_FILE_HOST} > ${IMAGE_FILE_HOST}.xz"
-- 
2.30.2

Re: [PATCH] use xz and gzip on host (outside chroot)

[Henning Schild]

I am not sure i like that. The version of the tools one gets will be
out of control when doing that. While that might seem harmless ... it
is not!

I have seen bzip2 problems when the compressor was too new and the
extractor too old. Windows machines could not unpack or "etcher".

We have also seen several settle differences on other "super basic
tools that just act on storage", like filesystem and partition utils.

However i get the speed argument. So i would suggest to come up with a
"? = " pattern for the CMDs.
Or when you really just want to run a native compressor in the "target
version" we just need a cross buildchroot instead of a native
buildchroot. If at least one package builds cross we have that anyhow,
but we could switch all "imagers" to using that ... code for creating a
cross buildchroot is in place.

Henning

Am Wed,  3 Aug 2022 10:54:48 +0200
schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:

> From: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> 
> This patch significantly speeds up the final image compression
> as the compression itself is no longer emulated.
> 
> For gzip, no additional dep has to be installed on the system,
> for xz the xz-utils package has to be installed but this will
> be required from bitbake 2.0 on anyways.
> 
> Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> Acked-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>  doc/user_manual.md              | 3 ++-
>  meta/classes/imagetypes.bbclass | 6 ++----
>  2 files changed, 4 insertions(+), 5 deletions(-)
> 
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 546ae75c..932bfbe5 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -83,7 +83,8 @@ apt install \
>    reprepro \
>    sudo \
>    git-buildpackage \
> -  pristine-tar
> +  pristine-tar \
> +  xz-utils
>  ```
>  
>  Additional setup is required since `sbuild` is now used for package
> build. diff --git a/meta/classes/imagetypes.bbclass
> b/meta/classes/imagetypes.bbclass index 1e8e223d..c4b06ca6 100644
> --- a/meta/classes/imagetypes.bbclass
> +++ b/meta/classes/imagetypes.bbclass
> @@ -95,13 +95,11 @@ IMAGE_CMD_ubi[depends] =
> "${PN}:do_transform_template" # image conversions
>  IMAGE_CONVERSIONS = "gz xz"
>  
> -CONVERSION_CMD_gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c
> --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'"
> -CONVERSION_DEPS_gz = "gzip" +CONVERSION_CMD_gz = "gzip -f -9 -n -c
> --rsyncable ${IMAGE_FILE_HOST} > ${IMAGE_FILE_HOST}.gz" 
>  XZ_MEMLIMIT ?= "50%"
>  XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
>  XZ_THREADS[vardepvalue] = "1"
>  XZ_OPTIONS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
>  XZ_OPTIONS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
> -CONVERSION_CMD_xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS}
> ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" -CONVERSION_DEPS_xz
> = "xz-utils" +CONVERSION_CMD_xz = "xz -c ${XZ_OPTIONS}
> ${IMAGE_FILE_HOST} > ${IMAGE_FILE_HOST}.xz"

RE: [PATCH] use xz and gzip on host (outside chroot)

[Moessbauer, Felix]

> -----Original Message-----
> From: Schild, Henning (T CED SES-DE) <henning.schild@siemens.com>
> Sent: Friday, August 5, 2022 2:45 PM
> To: Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>
> Cc: isar-users@googlegroups.com; Schmidt, Adriaan (T CED SES-DE)
> <adriaan.schmidt@siemens.com>
> Subject: Re: [PATCH] use xz and gzip on host (outside chroot)
> 
> I am not sure i like that. The version of the tools one gets will be out of
> control when doing that. While that might seem harmless ... it is not!

Well... With this argument we could say no tools from the host should be used, but always a bootstrapped rootfs from ISAR (independent of the target rootfs).
And this brings us to the kas-container. There, we have a known host environment.

I actually prefer this in case there are compatibility issues instead of running things in the build chroot.
The external versions are also much more stable, especially as we nowadays often build images using Debian testing, which in turn brings us very recent versions of the compression tools.

> 
> I have seen bzip2 problems when the compressor was too new and the
> extractor too old. Windows machines could not unpack or "etcher".
> 
> We have also seen several settle differences on other "super basic tools that
> just act on storage", like filesystem and partition utils.
> 
> However i get the speed argument. So i would suggest to come up with a "?
> = " pattern for the CMDs.

Yes, I will send a v2 soon.

> Or when you really just want to run a native compressor in the "target
> version" we just need a cross buildchroot instead of a native buildchroot. If at
> least one package builds cross we have that anyhow, but we could switch all
> "imagers" to using that ... code for creating a cross buildchroot is in place.

Here, also argument 1 from above holds.
Apart from that, it will be very difficult to implement as we cannot be sure that there is always a cross buildchroot.

Felix

> 
> Henning
> 
> Am Wed,  3 Aug 2022 10:54:48 +0200
> schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:
> 
> > From: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> >
> > This patch significantly speeds up the final image compression as the
> > compression itself is no longer emulated.
> >
> > For gzip, no additional dep has to be installed on the system, for xz
> > the xz-utils package has to be installed but this will be required
> > from bitbake 2.0 on anyways.
> >
> > Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> > Acked-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> > ---
> >  doc/user_manual.md              | 3 ++-
> >  meta/classes/imagetypes.bbclass | 6 ++----
> >  2 files changed, 4 insertions(+), 5 deletions(-)
> >
> > diff --git a/doc/user_manual.md b/doc/user_manual.md index
> > 546ae75c..932bfbe5 100644
> > --- a/doc/user_manual.md
> > +++ b/doc/user_manual.md
> > @@ -83,7 +83,8 @@ apt install \
> >    reprepro \
> >    sudo \
> >    git-buildpackage \
> > -  pristine-tar
> > +  pristine-tar \
> > +  xz-utils
> >  ```
> >
> >  Additional setup is required since `sbuild` is now used for package
> > build. diff --git a/meta/classes/imagetypes.bbclass
> > b/meta/classes/imagetypes.bbclass index 1e8e223d..c4b06ca6 100644
> > --- a/meta/classes/imagetypes.bbclass
> > +++ b/meta/classes/imagetypes.bbclass
> > @@ -95,13 +95,11 @@ IMAGE_CMD_ubi[depends] =
> > "${PN}:do_transform_template" # image conversions
> IMAGE_CONVERSIONS =
> > "gz xz"
> >
> > -CONVERSION_CMD_gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c
> > --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'"
> > -CONVERSION_DEPS_gz = "gzip" +CONVERSION_CMD_gz = "gzip -f -9 -n -c
> > --rsyncable ${IMAGE_FILE_HOST} > ${IMAGE_FILE_HOST}.gz"
> >  XZ_MEMLIMIT ?= "50%"
> >  XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
> >  XZ_THREADS[vardepvalue] = "1"
> >  XZ_OPTIONS ?= "--memlimit=${XZ_MEMLIMIT} --
> threads=${XZ_THREADS}"
> >  XZ_OPTIONS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
> > -CONVERSION_CMD_xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS}
> > ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" -
> CONVERSION_DEPS_xz =
> > "xz-utils" +CONVERSION_CMD_xz = "xz -c ${XZ_OPTIONS}
> > ${IMAGE_FILE_HOST} > ${IMAGE_FILE_HOST}.xz"

Re: [PATCH] use xz and gzip on host (outside chroot)

[Henning Schild]

Am Mon, 8 Aug 2022 09:32:24 +0200
schrieb "Moessbauer, Felix (T CED SES-DE)"
<felix.moessbauer@siemens.com>:

> > -----Original Message-----
> > From: Schild, Henning (T CED SES-DE) <henning.schild@siemens.com>
> > Sent: Friday, August 5, 2022 2:45 PM
> > To: Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>
> > Cc: isar-users@googlegroups.com; Schmidt, Adriaan (T CED SES-DE)
> > <adriaan.schmidt@siemens.com>
> > Subject: Re: [PATCH] use xz and gzip on host (outside chroot)
> > 
> > I am not sure i like that. The version of the tools one gets will
> > be out of control when doing that. While that might seem harmless
> > ... it is not!  
> 
> Well... With this argument we could say no tools from the host should
> be used, but always a bootstrapped rootfs from ISAR (independent of
> the target rootfs).

I guess that would be nice but not too realistic. 

> And this brings us to the kas-container. There, we have a known host
> environment.

No! kas is optional on top if Isar, we must not rely on it solving
problems and it must not be an excuse to introduce problems.

> I actually prefer this in case there are compatibility issues instead
> of running things in the build chroot. The external versions are also
> much more stable, especially as we nowadays often build images using
> Debian testing, which in turn brings us very recent versions of the
> compression tools.

It is best to have all the tools from the time of the distro. I guess
even in OE you would get old native tools if you build on an old
release. If you allow wild mix and match you are asking for "works
for me" situations. If there is a real bug in an old one ... debian
would likely fix it or you would at least always get it. Combining host
with target is "no mans land".
And yes for a compressor that seems super harmless, but we have seen
cases. So i rather not allow such combination, or not make them
"default".

Henning

> > 
> > I have seen bzip2 problems when the compressor was too new and the
> > extractor too old. Windows machines could not unpack or "etcher".
> > 
> > We have also seen several settle differences on other "super basic
> > tools that just act on storage", like filesystem and partition
> > utils.
> > 
> > However i get the speed argument. So i would suggest to come up
> > with a "? = " pattern for the CMDs.  
> 
> Yes, I will send a v2 soon.
> 
> > Or when you really just want to run a native compressor in the
> > "target version" we just need a cross buildchroot instead of a
> > native buildchroot. If at least one package builds cross we have
> > that anyhow, but we could switch all "imagers" to using that ...
> > code for creating a cross buildchroot is in place.  
> 
> Here, also argument 1 from above holds.
> Apart from that, it will be very difficult to implement as we cannot
> be sure that there is always a cross buildchroot.
> 
> Felix
> 
> > 
> > Henning
> > 
> > Am Wed,  3 Aug 2022 10:54:48 +0200
> > schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:
> >   
> > > From: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> > >
> > > This patch significantly speeds up the final image compression as
> > > the compression itself is no longer emulated.
> > >
> > > For gzip, no additional dep has to be installed on the system,
> > > for xz the xz-utils package has to be installed but this will be
> > > required from bitbake 2.0 on anyways.
> > >
> > > Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> > > Acked-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> > > ---
> > >  doc/user_manual.md              | 3 ++-
> > >  meta/classes/imagetypes.bbclass | 6 ++----
> > >  2 files changed, 4 insertions(+), 5 deletions(-)
> > >
> > > diff --git a/doc/user_manual.md b/doc/user_manual.md index
> > > 546ae75c..932bfbe5 100644
> > > --- a/doc/user_manual.md
> > > +++ b/doc/user_manual.md
> > > @@ -83,7 +83,8 @@ apt install \
> > >    reprepro \
> > >    sudo \
> > >    git-buildpackage \
> > > -  pristine-tar
> > > +  pristine-tar \
> > > +  xz-utils
> > >  ```
> > >
> > >  Additional setup is required since `sbuild` is now used for
> > > package build. diff --git a/meta/classes/imagetypes.bbclass
> > > b/meta/classes/imagetypes.bbclass index 1e8e223d..c4b06ca6 100644
> > > --- a/meta/classes/imagetypes.bbclass
> > > +++ b/meta/classes/imagetypes.bbclass
> > > @@ -95,13 +95,11 @@ IMAGE_CMD_ubi[depends] =
> > > "${PN}:do_transform_template" # image conversions  
> > IMAGE_CONVERSIONS =  
> > > "gz xz"
> > >
> > > -CONVERSION_CMD_gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c
> > > --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'"
> > > -CONVERSION_DEPS_gz = "gzip" +CONVERSION_CMD_gz = "gzip -f -9 -n
> > > -c --rsyncable ${IMAGE_FILE_HOST} > ${IMAGE_FILE_HOST}.gz"
> > >  XZ_MEMLIMIT ?= "50%"
> > >  XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
> > >  XZ_THREADS[vardepvalue] = "1"
> > >  XZ_OPTIONS ?= "--memlimit=${XZ_MEMLIMIT} --  
> > threads=${XZ_THREADS}"  
> > >  XZ_OPTIONS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
> > > -CONVERSION_CMD_xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS}
> > > ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" -  
> > CONVERSION_DEPS_xz =  
> > > "xz-utils" +CONVERSION_CMD_xz = "xz -c ${XZ_OPTIONS}
> > > ${IMAGE_FILE_HOST} > ${IMAGE_FILE_HOST}.xz"  
>