[PATCH v4 0/2] use xz and gzip on host (outside chroot)

[PATCH v4 0/2] use xz and gzip on host (outside chroot)

[Felix Moessbauer]

Changes since v3:

- add test for CROSS=1 wic image including compression
- integrate patch 2 into this series instead of standalone.

Please note, this series supersedes the "add wic.xz image to default qemuarm64 target" series
but does only requires little more disk space for the CI tests (compared to ISAR-next).
By that, the CI can still be executed on standard machines.

Changes since v1/v2:

instead of switching the compression logic, this patch adds support to run the whole imager in the buildchroot-host.


Felix Moessbauer (2):
  run imager in buildchroot-host on cross
  add wic.xz image to qemuarm64-bookworm

 doc/user_manual.md                                 | 11 +++++++++++
 meta-isar/conf/multiconfig/qemuarm64-bookworm.conf |  3 +++
 meta/classes/image-tools-extension.bbclass         |  3 ---
 testsuite/citest.py                                |  1 +
 4 files changed, 15 insertions(+), 3 deletions(-)

-- 
2.30.2

[PATCH v4 1/2] run imager in buildchroot-host on cross

[Felix Moessbauer]

When globally enabling ISAR_CROSS_COMPILE, the imager and compression
tasks run in the host buildchroot as well (instead of the target
buildchroot).

Similar to the cross support for DPKG recipes, this can be toggled
on a per-image basis by setting ISAR_CROSS_COMPILE in the image recipe.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 doc/user_manual.md                         | 11 +++++++++++
 meta/classes/image-tools-extension.bbclass |  3 ---
 2 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 5b2387d8..f38a133e 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -986,6 +986,17 @@ built for the compat arch need to be tagged individually by setting
 `PACKAGE_ARCH = "${COMPAT_DISTRO_ARCH}"` in the package recipe. Non-tagged
 packages will continue to be built for the primary target architecture.
 
+### Cross Support for Imagers
+
+If `ISAR_CROSS_COMPILE = "1"`, the imager and optional compression tasks
+run in the host buildchroot instead of the target buildchroot.
+This gives a significant speedup when compressing the generated image,
+as the compression is not emulated.
+
+In case your setup does not support cross-imaging, you can disable this
+just for the particular image by adding `ISAR_CROSS_COMPILE = "0"` to your
+image recipe.
+
 
 ## Examining and debugging package generation inside their buildchroot
 
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index b9968139..e13d4a3f 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -5,9 +5,6 @@
 #
 # This file extends the image.bbclass to supply tools for futher imager functions
 
-# Imager are expected to run natively, thus will use the target buildchroot.
-ISAR_CROSS_COMPILE = "0"
-
 inherit buildchroot
 
 IMAGER_INSTALL ??= ""
-- 
2.30.2

Re: [PATCH v4 1/2] run imager in buildchroot-host on cross

[Anton Mikanovich]

14.09.2022 23:21, Felix Moessbauer wrote:
> When globally enabling ISAR_CROSS_COMPILE, the imager and compression
> tasks run in the host buildchroot as well (instead of the target
> buildchroot).
>
> Similar to the cross support for DPKG recipes, this can be toggled
> on a per-image basis by setting ISAR_CROSS_COMPILE in the image recipe.
>
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>

Hello Felix,

This actually brakes SDK building for amd64 stretch targets because 
there is no
crossbuild-essential-amd64 there.

RE: [PATCH v4 1/2] run imager in buildchroot-host on cross

[Moessbauer, Felix]

> -----Original Message-----
> From: Anton Mikanovich <amikan@ilbers.de>
> Sent: Thursday, September 15, 2022 11:14 AM
> To: Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>; isar-
> users@googlegroups.com
> Cc: Schild, Henning (T CED SES-DE) <henning.schild@siemens.com>; Schmidt,
> Adriaan (T CED SES-DE) <adriaan.schmidt@siemens.com>; ibr@ilbers.de; Kiszka,
> Jan (T CED) <jan.kiszka@siemens.com>
> Subject: Re: [PATCH v4 1/2] run imager in buildchroot-host on cross
> 
> 14.09.2022 23:21, Felix Moessbauer wrote:
> > When globally enabling ISAR_CROSS_COMPILE, the imager and compression
> > tasks run in the host buildchroot as well (instead of the target
> > buildchroot).
> >
> > Similar to the cross support for DPKG recipes, this can be toggled on
> > a per-image basis by setting ISAR_CROSS_COMPILE in the image recipe.
> >
> > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> 
> Hello Felix,
> 
> This actually brakes SDK building for amd64 stretch targets because there is no
> crossbuild-essential-amd64 there.

Hi,

Interesting finding.
Are there any CI tests that catch this? At least I cannot find a CROSS test that builds the SDK.
Apart from that, Debian stretch is EOL since June 22.

I'm fine with keeping the stretch targets, but then we could simply disable CROSS by setting
ISAR_CROSS_COMPILE_debian-stretch = "0" in the image.bbclass.

If you could point me to a target for testing, I'm happy to add that.

Best regards,
Felix

Re: [PATCH v4 1/2] run imager in buildchroot-host on cross

[Anton Mikanovich]

15.09.2022 12:35, Moessbauer, Felix wrote:
> Hi,
>
> Interesting finding.
> Are there any CI tests that catch this? At least I cannot find a CROSS test that builds the SDK.
> Apart from that, Debian stretch is EOL since June 22.
>
> I'm fine with keeping the stretch targets, but then we could simply disable CROSS by setting
> ISAR_CROSS_COMPILE_debian-stretch = "0" in the image.bbclass.
>
> If you could point me to a target for testing, I'm happy to add that.
>
> Best regards,
> Felix
>
At least test_container_sdk test of our fast CI has do_populate_sdk 
target for
stretch. And we still need stretch support for downstreams, so adding
ISAR_CROSS_COMPILE override looks reasonable.

RE: [PATCH v4 1/2] run imager in buildchroot-host on cross

[Moessbauer, Felix]

> -----Original Message-----
> From: Anton Mikanovich <amikan@ilbers.de>
> Sent: Thursday, September 15, 2022 12:46 PM
> To: Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>; isar-
> users@googlegroups.com
> Cc: Schild, Henning (T CED SES-DE) <henning.schild@siemens.com>; Schmidt,
> Adriaan (T CED SES-DE) <adriaan.schmidt@siemens.com>; ibr@ilbers.de; Kiszka,
> Jan (T CED) <jan.kiszka@siemens.com>
> Subject: Re: [PATCH v4 1/2] run imager in buildchroot-host on cross
> 
> 15.09.2022 12:35, Moessbauer, Felix wrote:
> > Hi,
> >
> > Interesting finding.
> > Are there any CI tests that catch this? At least I cannot find a CROSS test that
> builds the SDK.
> > Apart from that, Debian stretch is EOL since June 22.
> >
> > I'm fine with keeping the stretch targets, but then we could simply
> > disable CROSS by setting ISAR_CROSS_COMPILE_debian-stretch = "0" in the
> image.bbclass.
> >
> > If you could point me to a target for testing, I'm happy to add that.
> >
> > Best regards,
> > Felix
> >
> At least test_container_sdk test of our fast CI has do_populate_sdk target for
> stretch. And we still need stretch support for downstreams, so adding
> ISAR_CROSS_COMPILE override looks reasonable.

Ok, will do that. But then using ` ISAR_CROSS_COMPILE_amd64_debian-stretch = "0"`

One last question before implementing this:
Could it be that this is in fact just a badly written test?
While cross-building for distro-arch == host-arch is technically possible (and uses a different toolchain), it does not look like it provides a reasonable value.
Could it be, that we better simply add ` perform_build_test(...,cross=False)` to the ContainerSdkTest as this is done for all other tests?

Best regards,
Felix

Re: [PATCH v4 1/2] run imager in buildchroot-host on cross

[Henning Schild]

Am Thu, 15 Sep 2022 13:45:35 +0300
schrieb Anton Mikanovich <amikan@ilbers.de>:

> 15.09.2022 12:35, Moessbauer, Felix wrote:
> > Hi,
> >
> > Interesting finding.
> > Are there any CI tests that catch this? At least I cannot find a
> > CROSS test that builds the SDK. Apart from that, Debian stretch is
> > EOL since June 22.
> >
> > I'm fine with keeping the stretch targets, but then we could simply
> > disable CROSS by setting ISAR_CROSS_COMPILE_debian-stretch = "0" in
> > the image.bbclass.
> >
> > If you could point me to a target for testing, I'm happy to add
> > that.
> >
> > Best regards,
> > Felix
> >  
> At least test_container_sdk test of our fast CI has do_populate_sdk 
> target for
> stretch. And we still need stretch support for downstreams, so adding
> ISAR_CROSS_COMPILE override looks reasonable.

Please take bullseye, not stretch. When we drop stretch we would
silently drop the cross/wic/compression test.

And in the long run we need a decoupling from testing multiconfigs to
example multiconfigs. Right now it is a little mixed so we might see
people copying testing bits because they think they are part of the
example, or testing being dropped by dropping a multiconfig that
contained some.

Henning

Re: [PATCH v4 1/2] run imager in buildchroot-host on cross

[Anton Mikanovich]

15.09.2022 14:06, Henning Schild wrote:
> Please take bullseye, not stretch. When we drop stretch we would
> silently drop the cross/wic/compression test.

Bullseye don't have any issues with SDK building, when stretch really do.
This is what Felix just asked and this is what my reply about.

Re: [PATCH v4 1/2] run imager in buildchroot-host on cross

[Henning Schild]

Am Thu, 15 Sep 2022 13:45:35 +0300
schrieb Anton Mikanovich <amikan@ilbers.de>:

> 15.09.2022 12:35, Moessbauer, Felix wrote:
> > Hi,
> >
> > Interesting finding.
> > Are there any CI tests that catch this? At least I cannot find a
> > CROSS test that builds the SDK. Apart from that, Debian stretch is
> > EOL since June 22.
> >
> > I'm fine with keeping the stretch targets, but then we could simply
> > disable CROSS by setting ISAR_CROSS_COMPILE_debian-stretch = "0" in
> > the image.bbclass.
> >
> > If you could point me to a target for testing, I'm happy to add
> > that.
> >
> > Best regards,
> > Felix
> >  
> At least test_container_sdk test of our fast CI has do_populate_sdk 
> target for
> stretch. And we still need stretch support for downstreams, so adding
> ISAR_CROSS_COMPILE override looks reasonable.

And that bookworm can not be crossed with SDK ... is that written down
somewhere? Maybe that should be a comment in those tests.

Henning

Re: [PATCH v4 1/2] run imager in buildchroot-host on cross

[Anton Mikanovich]

15.09.2022 14:08, Henning Schild wrote:
> And that bookworm can not be crossed with SDK ... is that written down
> somewhere? Maybe that should be a comment in those tests.

Why bookworm can't be crossed with SDK?

RE: [PATCH v4 1/2] run imager in buildchroot-host on cross

[Moessbauer, Felix]

> -----Original Message-----
> From: Anton Mikanovich <amikan@ilbers.de>
> Sent: Thursday, September 15, 2022 5:45 PM
> To: Schild, Henning (T CED SES-DE) <henning.schild@siemens.com>
> Cc: Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>; isar-
> users@googlegroups.com; Schmidt, Adriaan (T CED SES-DE)
> <adriaan.schmidt@siemens.com>; ibr@ilbers.de; Kiszka, Jan (T CED)
> <jan.kiszka@siemens.com>
> Subject: Re: [PATCH v4 1/2] run imager in buildchroot-host on cross
> 
> 15.09.2022 14:08, Henning Schild wrote:
> > And that bookworm can not be crossed with SDK ... is that written down
> > somewhere? Maybe that should be a comment in those tests.
> 
> Why bookworm can't be crossed with SDK?

I just checked and for me it worked without any issues.

Still, nobody answered my question about the distro-arch==host-arch question, huh!
Before sending a v5, I'll wait for a clarification.

Felix

Re: [PATCH v4 1/2] run imager in buildchroot-host on cross

[Anton Mikanovich]

15.09.2022 20:08, Moessbauer, Felix wrote:
> I just checked and for me it worked without any issues.
>
> Still, nobody answered my question about the distro-arch==host-arch question, huh!
> Before sending a v5, I'll wait for a clarification.

Running amd64 targets with enabled cross compile really don't have much 
sense,
but we have some logic in buildchroot.bbclass which makes Isar to use host
buildchroot in this case (and no such a logic for SDK - is that a bug?).

[PATCH v4 2/2] add wic.xz image to qemuarm64-bookworm

[Felix Moessbauer]

This patch adds generation of a wic.xz image for the mc:qemuarm64-bookworm target.
By that, the cross and non-cross imaging is automatically tested in CI.
Further, we add a qemuarm64-bookworm target to the cross build test.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta-isar/conf/multiconfig/qemuarm64-bookworm.conf | 3 +++
 testsuite/citest.py                                | 1 +
 2 files changed, 4 insertions(+)

diff --git a/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf b/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
index c6b92d2e..a04c9d1f 100644
--- a/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
+++ b/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
@@ -4,3 +4,6 @@
 
 MACHINE ?= "qemuarm64"
 DISTRO ?= "debian-bookworm"
+
+IMAGE_FSTYPES += "wic.xz"
+IMAGER_INSTALL += "${GRUB_BOOTLOADER_INSTALL}"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 6c8eb26e..630b4a65 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -72,6 +72,7 @@ class CrossTest(CIBaseTest):
             'mc:qemuarm-buster:isar-image-base',
             'mc:qemuarm-bullseye:isar-image-base',
             'mc:qemuarm64-bullseye:isar-image-base',
+            'mc:qemuarm64-bookworm:isar-image-base',
             'mc:de0-nano-soc-bullseye:isar-image-base',
             'mc:stm32mp15x-buster:isar-image-base'
                   ]
-- 
2.30.2