[PATCH v3 0/1] use xz and gzip on host (outside chroot)

[PATCH v3 0/1] use xz and gzip on host (outside chroot)

[Felix Moessbauer]

Changes since v1/v2:

instead of switching the compression logic,
this patch adds support to run the whole imager
in the buildchroot-host.

Felix Moessbauer (1):
  run imager in buildchroot-host on cross

 doc/user_manual.md                         | 11 +++++++++++
 meta/classes/image-tools-extension.bbclass |  3 ---
 2 files changed, 11 insertions(+), 3 deletions(-)

-- 
2.30.2

[PATCH v3 1/1] run imager in buildchroot-host on cross

[Felix Moessbauer]

When globally enabling ISAR_CROSS_COMPILE, the imager and compression
tasks run in the host buildchroot as well (instead of the target
buildchroot).

Similar to the cross support for DPKG recipes, this can be toggled
on a per-image basis by setting ISAR_CROSS_COMPILE in the image recipe.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 doc/user_manual.md                         | 11 +++++++++++
 meta/classes/image-tools-extension.bbclass |  3 ---
 2 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 5b2387d8..f38a133e 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -986,6 +986,17 @@ built for the compat arch need to be tagged individually by setting
 `PACKAGE_ARCH = "${COMPAT_DISTRO_ARCH}"` in the package recipe. Non-tagged
 packages will continue to be built for the primary target architecture.
 
+### Cross Support for Imagers
+
+If `ISAR_CROSS_COMPILE = "1"`, the imager and optional compression tasks
+run in the host buildchroot instead of the target buildchroot.
+This gives a significant speedup when compressing the generated image,
+as the compression is not emulated.
+
+In case your setup does not support cross-imaging, you can disable this
+just for the particular image by adding `ISAR_CROSS_COMPILE = "0"` to your
+image recipe.
+
 
 ## Examining and debugging package generation inside their buildchroot
 
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index b9968139..e13d4a3f 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -5,9 +5,6 @@
 #
 # This file extends the image.bbclass to supply tools for futher imager functions
 
-# Imager are expected to run natively, thus will use the target buildchroot.
-ISAR_CROSS_COMPILE = "0"
-
 inherit buildchroot
 
 IMAGER_INSTALL ??= ""
-- 
2.30.2

Re: [PATCH v3 1/1] run imager in buildchroot-host on cross

[Henning Schild]

I want a testcase for this where a whole wic gz/xz chain runs in CI with
a CROSS=1.

Henning

Am Tue, 13 Sep 2022 15:38:39 +0200
schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:

> When globally enabling ISAR_CROSS_COMPILE, the imager and compression
> tasks run in the host buildchroot as well (instead of the target
> buildchroot).
> 
> Similar to the cross support for DPKG recipes, this can be toggled
> on a per-image basis by setting ISAR_CROSS_COMPILE in the image
> recipe.
> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>  doc/user_manual.md                         | 11 +++++++++++
>  meta/classes/image-tools-extension.bbclass |  3 ---
>  2 files changed, 11 insertions(+), 3 deletions(-)
> 
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 5b2387d8..f38a133e 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -986,6 +986,17 @@ built for the compat arch need to be tagged
> individually by setting `PACKAGE_ARCH = "${COMPAT_DISTRO_ARCH}"` in
> the package recipe. Non-tagged packages will continue to be built for
> the primary target architecture. 
> +### Cross Support for Imagers
> +
> +If `ISAR_CROSS_COMPILE = "1"`, the imager and optional compression
> tasks +run in the host buildchroot instead of the target buildchroot.
> +This gives a significant speedup when compressing the generated
> image, +as the compression is not emulated.
> +
> +In case your setup does not support cross-imaging, you can disable
> this +just for the particular image by adding `ISAR_CROSS_COMPILE =
> "0"` to your +image recipe.
> +
>  
>  ## Examining and debugging package generation inside their
> buildchroot 
> diff --git a/meta/classes/image-tools-extension.bbclass
> b/meta/classes/image-tools-extension.bbclass index b9968139..e13d4a3f
> 100644 --- a/meta/classes/image-tools-extension.bbclass
> +++ b/meta/classes/image-tools-extension.bbclass
> @@ -5,9 +5,6 @@
>  #
>  # This file extends the image.bbclass to supply tools for futher
> imager functions 
> -# Imager are expected to run natively, thus will use the target
> buildchroot. -ISAR_CROSS_COMPILE = "0"
> -
>  inherit buildchroot
>  
>  IMAGER_INSTALL ??= ""

RE: [PATCH v3 1/1] run imager in buildchroot-host on cross

[Moessbauer, Felix]

> -----Original Message-----
> From: Schild, Henning (T CED SES-DE) <henning.schild@siemens.com>
> Sent: Wednesday, September 14, 2022 10:50 AM
> To: Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>
> Cc: isar-users@googlegroups.com; Kiszka, Jan (T CED)
> <jan.kiszka@siemens.com>; Schmidt, Adriaan (T CED SES-DE)
> <adriaan.schmidt@siemens.com>; amikan@ilbers.de; ibr@ilbers.de
> Subject: Re: [PATCH v3 1/1] run imager in buildchroot-host on cross
> 
> I want a testcase for this where a whole wic gz/xz chain runs in CI with a
> CROSS=1.

The WIC logic is already tested with CROSS=1 as the mc:qemuarm64-bullseye:isar-image-base is built in CI.
What is missing is the compression logic with CROSS=1.
Currently the only test that tests the compression logic at is the "container-amd64" machine.
However, these are not cross.

When trying to add the `wic.xz` to the qemuarm64 machine IMAGE_FSTYPES, we ran out of disk space on many systems as the generated images (and partitions) are rather big.
I'm going to send that as an individual patch so the maintainers can decide if that is really worth it.

Felix

> 
> Henning
> 
> Am Tue, 13 Sep 2022 15:38:39 +0200
> schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:
> 
> > When globally enabling ISAR_CROSS_COMPILE, the imager and
> compression
> > tasks run in the host buildchroot as well (instead of the target
> > buildchroot).
> >
> > Similar to the cross support for DPKG recipes, this can be toggled on
> > a per-image basis by setting ISAR_CROSS_COMPILE in the image recipe.
> >
> > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> > ---
> >  doc/user_manual.md                         | 11 +++++++++++
> >  meta/classes/image-tools-extension.bbclass |  3 ---
> >  2 files changed, 11 insertions(+), 3 deletions(-)
> >
> > diff --git a/doc/user_manual.md b/doc/user_manual.md index
> > 5b2387d8..f38a133e 100644
> > --- a/doc/user_manual.md
> > +++ b/doc/user_manual.md
> > @@ -986,6 +986,17 @@ built for the compat arch need to be tagged
> > individually by setting `PACKAGE_ARCH = "${COMPAT_DISTRO_ARCH}"` in
> > the package recipe. Non-tagged packages will continue to be built for
> > the primary target architecture.
> > +### Cross Support for Imagers
> > +
> > +If `ISAR_CROSS_COMPILE = "1"`, the imager and optional compression
> > tasks +run in the host buildchroot instead of the target buildchroot.
> > +This gives a significant speedup when compressing the generated
> > image, +as the compression is not emulated.
> > +
> > +In case your setup does not support cross-imaging, you can disable
> > this +just for the particular image by adding `ISAR_CROSS_COMPILE =
> > "0"` to your +image recipe.
> > +
> >
> >  ## Examining and debugging package generation inside their
> > buildchroot diff --git a/meta/classes/image-tools-extension.bbclass
> > b/meta/classes/image-tools-extension.bbclass index b9968139..e13d4a3f
> > 100644 --- a/meta/classes/image-tools-extension.bbclass
> > +++ b/meta/classes/image-tools-extension.bbclass
> > @@ -5,9 +5,6 @@
> >  #
> >  # This file extends the image.bbclass to supply tools for futher
> > imager functions -# Imager are expected to run natively, thus will use
> > the target buildchroot. -ISAR_CROSS_COMPILE = "0"
> > -
> >  inherit buildchroot
> >
> >  IMAGER_INSTALL ??= ""

Re: [PATCH v3 1/1] run imager in buildchroot-host on cross

[Henning Schild]

Am Wed, 14 Sep 2022 15:46:22 +0200
schrieb "Moessbauer, Felix (T CED SES-DE)"
<felix.moessbauer@siemens.com>:

> > -----Original Message-----
> > From: Schild, Henning (T CED SES-DE) <henning.schild@siemens.com>
> > Sent: Wednesday, September 14, 2022 10:50 AM
> > To: Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>
> > Cc: isar-users@googlegroups.com; Kiszka, Jan (T CED)
> > <jan.kiszka@siemens.com>; Schmidt, Adriaan (T CED SES-DE)
> > <adriaan.schmidt@siemens.com>; amikan@ilbers.de; ibr@ilbers.de
> > Subject: Re: [PATCH v3 1/1] run imager in buildchroot-host on cross
> > 
> > I want a testcase for this where a whole wic gz/xz chain runs in CI
> > with a CROSS=1.  
> 
> The WIC logic is already tested with CROSS=1 as the
> mc:qemuarm64-bullseye:isar-image-base is built in CI. What is missing
> is the compression logic with CROSS=1. Currently the only test that
> tests the compression logic at is the "container-amd64" machine.
> However, these are not cross.
> 
> When trying to add the `wic.xz` to the qemuarm64 machine
> IMAGE_FSTYPES, we ran out of disk space on many systems as the
> generated images (and partitions) are rather big. I'm going to send
> that as an individual patch so the maintainers can decide if that is
> really worth it.

Yes the test should be its own patch anyhow. It will be worth it ... HW
is dead cheap compared to what bad test coverage can cause.

Henning

> Felix
> 
> > 
> > Henning
> > 
> > Am Tue, 13 Sep 2022 15:38:39 +0200
> > schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:
> >   
> > > When globally enabling ISAR_CROSS_COMPILE, the imager and  
> > compression  
> > > tasks run in the host buildchroot as well (instead of the target
> > > buildchroot).
> > >
> > > Similar to the cross support for DPKG recipes, this can be
> > > toggled on a per-image basis by setting ISAR_CROSS_COMPILE in the
> > > image recipe.
> > >
> > > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> > > ---
> > >  doc/user_manual.md                         | 11 +++++++++++
> > >  meta/classes/image-tools-extension.bbclass |  3 ---
> > >  2 files changed, 11 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/doc/user_manual.md b/doc/user_manual.md index
> > > 5b2387d8..f38a133e 100644
> > > --- a/doc/user_manual.md
> > > +++ b/doc/user_manual.md
> > > @@ -986,6 +986,17 @@ built for the compat arch need to be tagged
> > > individually by setting `PACKAGE_ARCH = "${COMPAT_DISTRO_ARCH}"`
> > > in the package recipe. Non-tagged packages will continue to be
> > > built for the primary target architecture.
> > > +### Cross Support for Imagers
> > > +
> > > +If `ISAR_CROSS_COMPILE = "1"`, the imager and optional
> > > compression tasks +run in the host buildchroot instead of the
> > > target buildchroot. +This gives a significant speedup when
> > > compressing the generated image, +as the compression is not
> > > emulated. +
> > > +In case your setup does not support cross-imaging, you can
> > > disable this +just for the particular image by adding
> > > `ISAR_CROSS_COMPILE = "0"` to your +image recipe.
> > > +
> > >
> > >  ## Examining and debugging package generation inside their
> > > buildchroot diff --git
> > > a/meta/classes/image-tools-extension.bbclass
> > > b/meta/classes/image-tools-extension.bbclass index
> > > b9968139..e13d4a3f 100644 ---
> > > a/meta/classes/image-tools-extension.bbclass +++
> > > b/meta/classes/image-tools-extension.bbclass @@ -5,9 +5,6 @@ #
> > >  # This file extends the image.bbclass to supply tools for futher
> > > imager functions -# Imager are expected to run natively, thus
> > > will use the target buildchroot. -ISAR_CROSS_COMPILE = "0"
> > > -
> > >  inherit buildchroot
> > >
> > >  IMAGER_INSTALL ??= ""  
>