RE: [PATCH] use xz and gzip on host (outside chroot)

["Moessbauer, Felix" <felix.moessbauer@siemens.com>]

> -----Original Message-----
> From: Schild, Henning (T CED SES-DE) <henning.schild@siemens.com>
> Sent: Friday, August 5, 2022 2:45 PM
> To: Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>
> Cc: isar-users@googlegroups.com; Schmidt, Adriaan (T CED SES-DE)
> <adriaan.schmidt@siemens.com>
> Subject: Re: [PATCH] use xz and gzip on host (outside chroot)
> 
> I am not sure i like that. The version of the tools one gets will be out of
> control when doing that. While that might seem harmless ... it is not!

Well... With this argument we could say no tools from the host should be used, but always a bootstrapped rootfs from ISAR (independent of the target rootfs).
And this brings us to the kas-container. There, we have a known host environment.

I actually prefer this in case there are compatibility issues instead of running things in the build chroot.
The external versions are also much more stable, especially as we nowadays often build images using Debian testing, which in turn brings us very recent versions of the compression tools.

> 
> I have seen bzip2 problems when the compressor was too new and the
> extractor too old. Windows machines could not unpack or "etcher".
> 
> We have also seen several settle differences on other "super basic tools that
> just act on storage", like filesystem and partition utils.
> 
> However i get the speed argument. So i would suggest to come up with a "?
> = " pattern for the CMDs.

Yes, I will send a v2 soon.

> Or when you really just want to run a native compressor in the "target
> version" we just need a cross buildchroot instead of a native buildchroot. If at
> least one package builds cross we have that anyhow, but we could switch all
> "imagers" to using that ... code for creating a cross buildchroot is in place.

Here, also argument 1 from above holds.
Apart from that, it will be very difficult to implement as we cannot be sure that there is always a cross buildchroot.

Felix

> 
> Henning
> 
> Am Wed,  3 Aug 2022 10:54:48 +0200
> schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:
> 
> > From: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> >
> > This patch significantly speeds up the final image compression as the
> > compression itself is no longer emulated.
> >
> > For gzip, no additional dep has to be installed on the system, for xz
> > the xz-utils package has to be installed but this will be required
> > from bitbake 2.0 on anyways.
> >
> > Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> > Acked-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> > ---
> >  doc/user_manual.md              | 3 ++-
> >  meta/classes/imagetypes.bbclass | 6 ++----
> >  2 files changed, 4 insertions(+), 5 deletions(-)
> >
> > diff --git a/doc/user_manual.md b/doc/user_manual.md index
> > 546ae75c..932bfbe5 100644
> > --- a/doc/user_manual.md
> > +++ b/doc/user_manual.md
> > @@ -83,7 +83,8 @@ apt install \
> >    reprepro \
> >    sudo \
> >    git-buildpackage \
> > -  pristine-tar
> > +  pristine-tar \
> > +  xz-utils
> >  ```
> >
> >  Additional setup is required since `sbuild` is now used for package
> > build. diff --git a/meta/classes/imagetypes.bbclass
> > b/meta/classes/imagetypes.bbclass index 1e8e223d..c4b06ca6 100644
> > --- a/meta/classes/imagetypes.bbclass
> > +++ b/meta/classes/imagetypes.bbclass
> > @@ -95,13 +95,11 @@ IMAGE_CMD_ubi[depends] =
> > "${PN}:do_transform_template" # image conversions
> IMAGE_CONVERSIONS =
> > "gz xz"
> >
> > -CONVERSION_CMD_gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c
> > --rsyncable ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'"
> > -CONVERSION_DEPS_gz = "gzip" +CONVERSION_CMD_gz = "gzip -f -9 -n -c
> > --rsyncable ${IMAGE_FILE_HOST} > ${IMAGE_FILE_HOST}.gz"
> >  XZ_MEMLIMIT ?= "50%"
> >  XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
> >  XZ_THREADS[vardepvalue] = "1"
> >  XZ_OPTIONS ?= "--memlimit=${XZ_MEMLIMIT} --
> threads=${XZ_THREADS}"
> >  XZ_OPTIONS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
> > -CONVERSION_CMD_xz = "${SUDO_CHROOT} sh -c 'xz -c ${XZ_OPTIONS}
> > ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.xz'" -
> CONVERSION_DEPS_xz =
> > "xz-utils" +CONVERSION_CMD_xz = "xz -c ${XZ_OPTIONS}
> > ${IMAGE_FILE_HOST} > ${IMAGE_FILE_HOST}.xz"