[PATCH v5 0/5] use xz and gzip on host (outside chroot)

[PATCH v5 0/5] use xz and gzip on host (outside chroot)

[Felix Moessbauer]

Changes since v4:

- add option WIC_DEPLOY_PARTITIONS = "0" to control if individual wic partitions are deployed (p3)
  Note, that the default set to disabled is important.
  This fixes the huge disk consumption when running the CI with WIC tests
- add test for WIC_DEPLOY_PARTITIONS logic (p4)
- add logic to handle cross-imaging when distro_arch == host_arch (similar to buildchroot-(target|host)) (p1)

Please note, that the additions around WIC_DEPLOY_PARTITIONS are standalone patches
which are only sent as part of this series to fix the disk space blow up in CI.

Changes since v3:

- add test for CROSS=1 wic image including compression
- integrate patch 2 into this series instead of standalone.

Please note, this series supersedes the "add wic.xz image to default qemuarm64 target" series but does only requires little more disk space for the CI tests (compared to ISAR-next).
By that, the CI can still be executed on standard machines.

Changes since v1/v2:

instead of switching the compression logic, this patch adds support to run the whole imager in the buildchroot-host.


Felix Moessbauer (5):
  do not crossbuild SDK for host-arch eq dist-arch
  run imager in buildchroot-host on cross
  add option to control deploy of raw wic partitions
  add test for wic partition deploy logic
  add wic.xz image to qemuarm64-bookworm

 RECIPE-API-CHANGELOG.md                       |  6 +++++
 doc/user_manual.md                            | 11 +++++++++
 meta-isar/conf/local.conf.sample              |  3 +++
 .../conf/multiconfig/qemuarm64-bookworm.conf  |  3 +++
 meta/classes/image-tools-extension.bbclass    |  3 ---
 meta/classes/imagetypes_wic.bbclass           |  5 ++++
 meta/classes/sdk.bbclass                      | 14 +++++++++--
 testsuite/cibase.py                           | 10 ++++++++
 testsuite/cibuilder.py                        |  5 +++-
 testsuite/citest.py                           | 24 +++++++++++++++++++
 10 files changed, 78 insertions(+), 6 deletions(-)

-- 
2.30.2

[PATCH v5 1/5] do not crossbuild SDK for host-arch eq dist-arch

[Felix Moessbauer]

This patch adds logic to detect when ISAR_CROSS_COMPILE
is set, but host-arch equals dist-arch.
In this case, we do not add the crossbuild-essential package,
but instead add the build-essential one.

To implement this, we transfer the existing logic into a
anonymous python function to make it better readable.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/sdk.bbclass | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sdk.bbclass b/meta/classes/sdk.bbclass
index 32ee2de7..f42a2ba1 100644
--- a/meta/classes/sdk.bbclass
+++ b/meta/classes/sdk.bbclass
@@ -41,8 +41,18 @@ SDK_PREINSTALL += " \
     devscripts \
     equivs"
 
-TOOLCHAIN = "${@'crossbuild-essential-${DISTRO_ARCH}' if d.getVar('ISAR_CROSS_COMPILE', True) == '1' else 'build-essential'}"
-TOOLCHAIN_append_compat-arch = " crossbuild-essential-${COMPAT_DISTRO_ARCH}"
+# Choose the correct toolchain: cross or native
+python __anonymous() {
+    mode = d.getVar('ISAR_CROSS_COMPILE', True)
+    distro_arch = d.getVar('DISTRO_ARCH')
+    if mode == "0" or d.getVar('HOST_ARCH') ==  distro_arch:
+        toolchain = "build-essential"
+    else:
+        toolchain = "crossbuild-essential-" + distro_arch
+    if d.getVar('ISAR_ENABLE_COMPAT_ARCH', True) == "1":
+        toolchain += " crossbuild-essential-" + d.getVar('COMPAT_DISTRO_ARCH')
+    d.setVar('TOOLCHAIN', toolchain)
+}
 
 # rootfs/image overrides for the SDK
 ROOTFS_ARCH_class-sdk = "${HOST_ARCH}"
-- 
2.30.2

[PATCH v5 2/5] run imager in buildchroot-host on cross

[Felix Moessbauer]

When globally enabling ISAR_CROSS_COMPILE, the imager and compression
tasks run in the host buildchroot as well (instead of the target
buildchroot).

Similar to the cross support for DPKG recipes, this can be toggled
on a per-image basis by setting ISAR_CROSS_COMPILE in the image recipe.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 doc/user_manual.md                         | 11 +++++++++++
 meta/classes/image-tools-extension.bbclass |  3 ---
 2 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 5b2387d8..f38a133e 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -986,6 +986,17 @@ built for the compat arch need to be tagged individually by setting
 `PACKAGE_ARCH = "${COMPAT_DISTRO_ARCH}"` in the package recipe. Non-tagged
 packages will continue to be built for the primary target architecture.
 
+### Cross Support for Imagers
+
+If `ISAR_CROSS_COMPILE = "1"`, the imager and optional compression tasks
+run in the host buildchroot instead of the target buildchroot.
+This gives a significant speedup when compressing the generated image,
+as the compression is not emulated.
+
+In case your setup does not support cross-imaging, you can disable this
+just for the particular image by adding `ISAR_CROSS_COMPILE = "0"` to your
+image recipe.
+
 
 ## Examining and debugging package generation inside their buildchroot
 
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index b9968139..e13d4a3f 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -5,9 +5,6 @@
 #
 # This file extends the image.bbclass to supply tools for futher imager functions
 
-# Imager are expected to run natively, thus will use the target buildchroot.
-ISAR_CROSS_COMPILE = "0"
-
 inherit buildchroot
 
 IMAGER_INSTALL ??= ""
-- 
2.30.2

[PATCH v5 3/5] add option to control deploy of raw wic partitions

[Felix Moessbauer]

This patch adds the WIC_DEPLOY_PARTITIONS option to control
if the raw partition files are deployed by wic.

We also set the default to not deploy to be closer to OE and to
optimize for the average case.
In addition, this significantly reduces the disk size required to
run the CI.

An API changelog entry is added.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 RECIPE-API-CHANGELOG.md             | 6 ++++++
 meta-isar/conf/local.conf.sample    | 3 +++
 meta/classes/imagetypes_wic.bbclass | 5 +++++
 3 files changed, 14 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 621d110e..0c34d9be 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -419,3 +419,9 @@ packages and will be lost after a given build session finishes.
 
 Any package build requirements for the rootfs should be satisfied in the
 Debian way via package dependencies.
+
+### Individual WIC partitions are no longer automatically deployed
+
+The individual wic partitions (e.g. *.wic.p1) are no longer automatically
+copied into the `DEPLOY_DIR_IMAGE`.
+To explicitly deploy these files (e.g. for swupdate), set `WIC_DEPLOY_PARTITIONS = "1"`.
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index ce7b58ee..57d06202 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -246,6 +246,9 @@ USER_isar[flags] += "clear-text-password"
 # Uncomment the below line to debug WIC.
 # WIC_CREATE_EXTRA_ARGS += "-D"
 
+# Uncomment this to also deploy each wic partition as separate file (e.g. for swupdate)
+#WIC_DEPLOY_PARTITIONS = "1"
+
 # Uncomment this to use ccache for custom packages
 #USE_CCACHE = "1"
 # Uncomment and set own top level ccache directory to share between builds
diff --git a/meta/classes/imagetypes_wic.bbclass b/meta/classes/imagetypes_wic.bbclass
index 61a74d4a..ba8aa15c 100644
--- a/meta/classes/imagetypes_wic.bbclass
+++ b/meta/classes/imagetypes_wic.bbclass
@@ -98,6 +98,7 @@ RECIPE_SYSROOT_NATIVE ?= "/"
 BUILDCHROOT_DIR = "${BUILDCHROOT_TARGET_DIR}"
 
 WIC_CREATE_EXTRA_ARGS ?= ""
+WIC_DEPLOY_PARTITIONS ?= "0"
 
 # taken from OE, do not touch directly
 WICVARS += "\
@@ -205,6 +206,10 @@ generate_wic_image() {
 
     sudo chown -R $(stat -c "%U" ${LAYERDIR_core}) ${LAYERDIR_core} ${LAYERDIR_isar} ${SCRIPTSDIR} || true
     sudo chown -R $(id -u):$(id -g) ${BUILDCHROOT_DIR}/${WICTMP}
+    # only keep partitions if requested (ending with .p<x>)
+    if [ "${WIC_DEPLOY_PARTITIONS}" -eq "0" ]; then
+        find ${BUILDCHROOT_DIR}/${WICTMP} -type f -regextype sed -regex ".*\.direct.*\.p[0-9]\{1,\}" -delete
+    fi
     find ${BUILDCHROOT_DIR}/${WICTMP} -type f -name "*.direct*" | while read f; do
         suffix=$(basename $f | sed 's/\(.*\)\(\.direct\)\(.*\)/\3/')
         mv -f ${f} "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic${suffix}"
-- 
2.30.2

[PATCH v5 4/5] add test for wic partition deploy logic

[Felix Moessbauer]

This patch adds tests to check if the wic partition files
are copied to the deploydir if requested.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 testsuite/cibase.py    | 10 ++++++++++
 testsuite/cibuilder.py |  5 ++++-
 testsuite/citest.py    | 23 +++++++++++++++++++++++
 3 files changed, 37 insertions(+), 1 deletion(-)

diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index 722d7bd2..6239b4de 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -17,6 +17,16 @@ class CIBaseTest(CIBuilder):
 
         self.bitbake(targets, **kwargs)
 
+    def perform_wic_partition_test(self, targets, wic_deploy_parts, **kwargs):
+        self.configure(wic_deploy_parts=wic_deploy_parts, **kwargs)
+        self.bitbake(targets, **kwargs)
+
+        partition_files = set(glob.glob(f'{self.build_dir}/tmp/deploy/images/*/*.wic.p1'))
+        if wic_deploy_parts and len(partition_files) == 0:
+            self.fail('Found raw wic partitions in DEPLOY_DIR')
+        if not wic_deploy_parts and len(partition_files) != 0:
+            self.fail('Did not find raw wic partitions in DEPLOY_DIR')
+
     def perform_repro_test(self, targets, signed=False, **kwargs):
         gpg_pub_key = os.path.dirname(__file__) + '/keys/base-apt/test_pub.key'
         gpg_priv_key = os.path.dirname(__file__) + '/keys/base-apt/test_priv.key'
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index dfb0a376..d5c83b22 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -54,7 +54,7 @@ class CIBuilder(Test):
 
     def configure(self, compat_arch=True, cross=None, debsrc_cache=False,
                   container=False, ccache=False, sstate=False, offline=False,
-                  gpg_pub_key=None, **kwargs):
+                  gpg_pub_key=None, wic_deploy_parts=False, **kwargs):
         # write configuration file and set bitbake_args
         # can run multiple times per test case
         self.check_init()
@@ -77,6 +77,7 @@ class CIBuilder(Test):
                       f'  ccache = {ccache}\n'
                       f'  sstate = {sstate}\n'
                       f'  gpg_pub_key = {gpg_pub_key}\n'
+                      f'  wic_deploy_parts = {wic_deploy_parts}\n'
                       f'===================================================')
 
         # determine bitbake_args
@@ -105,6 +106,8 @@ class CIBuilder(Test):
                 f.write('IMAGE_INSTALL_remove = "example-module-${KERNEL_NAME} enable-fsck"\n')
             if gpg_pub_key:
                 f.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
+            if wic_deploy_parts:
+                f.write('WIC_DEPLOY_PARTITIONS = "1"\n')
             if distro_apt_premir:
                 f.write('DISTRO_APT_PREMIRRORS = "%s"\n' % distro_apt_premir)
             if ccache:
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 6c8eb26e..8381f21c 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -223,6 +223,29 @@ class RebuildTest(CIBaseTest):
         finally:
             self.restorefile(dpkgbase_file)
 
+class WicTest(CIBaseTest):
+
+    """
+    Test creation of wic images
+
+    :avocado: tags=wic,full
+    """
+    def test_wic_nodeploy_partitions(self):
+        targets = ['mc:qemuarm64-bookworm:isar-image-base']
+
+        self.init()
+        self.delete_from_build_dir('tmp')
+        self.perform_wic_partition_test(targets,
+            wic_deploy_parts=False, debsrc_cache=True, compat_arch=False)
+
+    def test_wic_deploy_partitions(self):
+        targets = ['mc:qemuarm64-bookworm:isar-image-base']
+
+        self.init()
+        # reuse artifacts
+        self.perform_wic_partition_test(targets,
+            wic_deploy_parts=True, debsrc_cache=True, compat_arch=False)
+
 class ContainerImageTest(CIBaseTest):
 
     """
-- 
2.30.2

[PATCH v5 5/5] add wic.xz image to qemuarm64-bookworm

[Felix Moessbauer]

This patch adds generation of a wic.xz image for the mc:qemuarm64-bookworm target.
By that, the cross and non-cross imaging is automatically tested in CI.
Further, we add a qemuarm64-bookworm target to the cross build test.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta-isar/conf/multiconfig/qemuarm64-bookworm.conf | 3 +++
 testsuite/citest.py                                | 1 +
 2 files changed, 4 insertions(+)

diff --git a/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf b/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
index c6b92d2e..a04c9d1f 100644
--- a/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
+++ b/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
@@ -4,3 +4,6 @@
 
 MACHINE ?= "qemuarm64"
 DISTRO ?= "debian-bookworm"
+
+IMAGE_FSTYPES += "wic.xz"
+IMAGER_INSTALL += "${GRUB_BOOTLOADER_INSTALL}"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 8381f21c..0c0a6284 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -72,6 +72,7 @@ class CrossTest(CIBaseTest):
             'mc:qemuarm-buster:isar-image-base',
             'mc:qemuarm-bullseye:isar-image-base',
             'mc:qemuarm64-bullseye:isar-image-base',
+            'mc:qemuarm64-bookworm:isar-image-base',
             'mc:de0-nano-soc-bullseye:isar-image-base',
             'mc:stm32mp15x-buster:isar-image-base'
                   ]
-- 
2.30.2

Re: [PATCH v5 3/5] add option to control deploy of raw wic partitions

[Henning Schild]

Am Mon, 19 Sep 2022 13:20:33 +0200
schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:

> This patch adds the WIC_DEPLOY_PARTITIONS option to control
> if the raw partition files are deployed by wic.
> 
> We also set the default to not deploy to be closer to OE and to
> optimize for the average case.
> In addition, this significantly reduces the disk size required to
> run the CI.
> 
> An API changelog entry is added.
> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>  RECIPE-API-CHANGELOG.md             | 6 ++++++
>  meta-isar/conf/local.conf.sample    | 3 +++
>  meta/classes/imagetypes_wic.bbclass | 5 +++++
>  3 files changed, 14 insertions(+)
> 
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index 621d110e..0c34d9be 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -419,3 +419,9 @@ packages and will be lost after a given build
> session finishes. 
>  Any package build requirements for the rootfs should be satisfied in
> the Debian way via package dependencies.
> +
> +### Individual WIC partitions are no longer automatically deployed
> +
> +The individual wic partitions (e.g. *.wic.p1) are no longer
> automatically +copied into the `DEPLOY_DIR_IMAGE`.
> +To explicitly deploy these files (e.g. for swupdate), set
> `WIC_DEPLOY_PARTITIONS = "1"`. diff --git
> a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
> index ce7b58ee..57d06202 100644 --- a/meta-isar/conf/local.conf.sample
> +++ b/meta-isar/conf/local.conf.sample
> @@ -246,6 +246,9 @@ USER_isar[flags] += "clear-text-password"
>  # Uncomment the below line to debug WIC.
>  # WIC_CREATE_EXTRA_ARGS += "-D"
>  
> +# Uncomment this to also deploy each wic partition as separate file
> (e.g. for swupdate) +#WIC_DEPLOY_PARTITIONS = "1"
> +
>  # Uncomment this to use ccache for custom packages
>  #USE_CCACHE = "1"
>  # Uncomment and set own top level ccache directory to share between
> builds diff --git a/meta/classes/imagetypes_wic.bbclass
> b/meta/classes/imagetypes_wic.bbclass index 61a74d4a..ba8aa15c 100644
> --- a/meta/classes/imagetypes_wic.bbclass
> +++ b/meta/classes/imagetypes_wic.bbclass
> @@ -98,6 +98,7 @@ RECIPE_SYSROOT_NATIVE ?= "/"
>  BUILDCHROOT_DIR = "${BUILDCHROOT_TARGET_DIR}"
>  
>  WIC_CREATE_EXTRA_ARGS ?= ""
> +WIC_DEPLOY_PARTITIONS ?= "0"
>  
>  # taken from OE, do not touch directly
>  WICVARS += "\
> @@ -205,6 +206,10 @@ generate_wic_image() {
>  
>      sudo chown -R $(stat -c "%U" ${LAYERDIR_core}) ${LAYERDIR_core}
> ${LAYERDIR_isar} ${SCRIPTSDIR} || true sudo chown -R $(id -u):$(id
> -g) ${BUILDCHROOT_DIR}/${WICTMP}
> +    # only keep partitions if requested (ending with .p<x>)
> +    if [ "${WIC_DEPLOY_PARTITIONS}" -eq "0" ]; then
> +        find ${BUILDCHROOT_DIR}/${WICTMP} -type f -regextype sed
> -regex ".*\.direct.*\.p[0-9]\{1,\}" -delete

I just had a look at OE and they seem to do things differently. They
call wic with a -w "temporary workdir to use for intermediate files"
and later move only the ".direct" to DEPLOY/<image>.wic

So i would suggest to do that as well, and not delete valid files that
maybe a second run or another tool might end up missing. Say you
generate a .wic.xz and a .swu that swu will maybe need the partition
image where the .wic.xz itself would not deploy it.

Henning

> +    fi
>      find ${BUILDCHROOT_DIR}/${WICTMP} -type f -name "*.direct*" |
> while read f; do suffix=$(basename $f | sed
> 's/\(.*\)\(\.direct\)\(.*\)/\3/') mv -f ${f}
> "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic${suffix}"

RE: [PATCH v5 3/5] add option to control deploy of raw wic partitions

[Moessbauer, Felix]

> -----Original Message-----
> From: Schild, Henning (T CED SES-DE) <henning.schild@siemens.com>
> Sent: Monday, September 19, 2022 2:03 PM
> To: Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>
> Cc: isar-users@googlegroups.com; Schmidt, Adriaan (T CED SES-DE)
> <adriaan.schmidt@siemens.com>; Kiszka, Jan (T CED)
> <jan.kiszka@siemens.com>; ibr@ilbers.de; amikan@ilbers.de
> Subject: Re: [PATCH v5 3/5] add option to control deploy of raw wic partitions
> 
> Am Mon, 19 Sep 2022 13:20:33 +0200
> schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:
> 
> > This patch adds the WIC_DEPLOY_PARTITIONS option to control if the raw
> > partition files are deployed by wic.
> >
> > We also set the default to not deploy to be closer to OE and to
> > optimize for the average case.
> > In addition, this significantly reduces the disk size required to run
> > the CI.
> >
> > An API changelog entry is added.
> >
> > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> > ---
> >  RECIPE-API-CHANGELOG.md             | 6 ++++++
> >  meta-isar/conf/local.conf.sample    | 3 +++
> >  meta/classes/imagetypes_wic.bbclass | 5 +++++
> >  3 files changed, 14 insertions(+)
> >
> > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index
> > 621d110e..0c34d9be 100644
> > --- a/RECIPE-API-CHANGELOG.md
> > +++ b/RECIPE-API-CHANGELOG.md
> > @@ -419,3 +419,9 @@ packages and will be lost after a given build
> > session finishes.
> >  Any package build requirements for the rootfs should be satisfied in
> > the Debian way via package dependencies.
> > +
> > +### Individual WIC partitions are no longer automatically deployed
> > +
> > +The individual wic partitions (e.g. *.wic.p1) are no longer
> > automatically +copied into the `DEPLOY_DIR_IMAGE`.
> > +To explicitly deploy these files (e.g. for swupdate), set
> > `WIC_DEPLOY_PARTITIONS = "1"`. diff --git
> > a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
> > index ce7b58ee..57d06202 100644 --- a/meta-isar/conf/local.conf.sample
> > +++ b/meta-isar/conf/local.conf.sample
> > @@ -246,6 +246,9 @@ USER_isar[flags] += "clear-text-password"
> >  # Uncomment the below line to debug WIC.
> >  # WIC_CREATE_EXTRA_ARGS += "-D"
> >
> > +# Uncomment this to also deploy each wic partition as separate file
> > (e.g. for swupdate) +#WIC_DEPLOY_PARTITIONS = "1"
> > +
> >  # Uncomment this to use ccache for custom packages  #USE_CCACHE = "1"
> >  # Uncomment and set own top level ccache directory to share between
> > builds diff --git a/meta/classes/imagetypes_wic.bbclass
> > b/meta/classes/imagetypes_wic.bbclass index 61a74d4a..ba8aa15c 100644
> > --- a/meta/classes/imagetypes_wic.bbclass
> > +++ b/meta/classes/imagetypes_wic.bbclass
> > @@ -98,6 +98,7 @@ RECIPE_SYSROOT_NATIVE ?= "/"
> >  BUILDCHROOT_DIR = "${BUILDCHROOT_TARGET_DIR}"
> >
> >  WIC_CREATE_EXTRA_ARGS ?= ""
> > +WIC_DEPLOY_PARTITIONS ?= "0"
> >
> >  # taken from OE, do not touch directly  WICVARS += "\ @@ -205,6
> > +206,10 @@ generate_wic_image() {
> >
> >      sudo chown -R $(stat -c "%U" ${LAYERDIR_core}) ${LAYERDIR_core}
> > ${LAYERDIR_isar} ${SCRIPTSDIR} || true sudo chown -R $(id -u):$(id
> > -g) ${BUILDCHROOT_DIR}/${WICTMP}
> > +    # only keep partitions if requested (ending with .p<x>)
> > +    if [ "${WIC_DEPLOY_PARTITIONS}" -eq "0" ]; then
> > +        find ${BUILDCHROOT_DIR}/${WICTMP} -type f -regextype sed
> > -regex ".*\.direct.*\.p[0-9]\{1,\}" -delete
> 
> I just had a look at OE and they seem to do things differently. They call wic with
> a -w "temporary workdir to use for intermediate files"
> and later move only the ".direct" to DEPLOY/<image>.wic

I had a very careful look at the OE wic script as well and the only difference is that they do not copy any temporary files (like the partitions) to the deploy dir.
That's why they don't do this "find" but simply "move" the .direct file [1].
There had been proposed patches to change that (also from Siemens), but they never have been accepted.

The added "-w $tmp_wic" is indeed a divergence, but that is independent of the added / changed logic in this series.
In the end, the OE data in $tmp_wic is never re-used again.

> 
> So i would suggest to do that as well, and not delete valid files that maybe a
> second run or another tool might end up missing. Say you generate a .wic.xz and
> a .swu that swu will maybe need the partition image where the .wic.xz itself
> would not deploy it.

If you look carefully at the following lines after the "find", you will see that the
WICTMP is deleted anyways. And in between, there is no extension point where any other tool could integrate and read the temporary files.
By that, I assume that my approach is correct and reliable.

[1] https://git.openembedded.org/openembedded-core/tree/meta/classes/image_types_wic.bbclass?id=958ee0eede859bdba659e3343856b1c226207854#n68

Felix

> 
> Henning
> 
> > +    fi
> >      find ${BUILDCHROOT_DIR}/${WICTMP} -type f -name "*.direct*" |
> > while read f; do suffix=$(basename $f | sed
> > 's/\(.*\)\(\.direct\)\(.*\)/\3/') mv -f ${f}
> > "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic${suffix}"

Re: [PATCH v5 3/5] add option to control deploy of raw wic partitions

[Henning Schild]

Am Mon, 19 Sep 2022 13:20:33 +0200
schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:

> This patch adds the WIC_DEPLOY_PARTITIONS option to control
> if the raw partition files are deployed by wic.
> 
> We also set the default to not deploy to be closer to OE and to
> optimize for the average case.
> In addition, this significantly reduces the disk size required to
> run the CI.
> 
> An API changelog entry is added.

I think the deployment of the partitions was an accident and never
intended. Cause might be

https://github.com/ilbers/isar/commit/edc10d9361e79f36c8ea5488b1dab4c468213a8d

which wanted to deal only with compressed images when wic itself is
used for compression.

We should maybe just sharpen that to not include partitions, instead of
inventing a new interface to remove them again.

Henning

> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>  RECIPE-API-CHANGELOG.md             | 6 ++++++
>  meta-isar/conf/local.conf.sample    | 3 +++
>  meta/classes/imagetypes_wic.bbclass | 5 +++++
>  3 files changed, 14 insertions(+)
> 
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index 621d110e..0c34d9be 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -419,3 +419,9 @@ packages and will be lost after a given build
> session finishes. 
>  Any package build requirements for the rootfs should be satisfied in
> the Debian way via package dependencies.
> +
> +### Individual WIC partitions are no longer automatically deployed
> +
> +The individual wic partitions (e.g. *.wic.p1) are no longer
> automatically +copied into the `DEPLOY_DIR_IMAGE`.
> +To explicitly deploy these files (e.g. for swupdate), set
> `WIC_DEPLOY_PARTITIONS = "1"`. diff --git
> a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
> index ce7b58ee..57d06202 100644 --- a/meta-isar/conf/local.conf.sample
> +++ b/meta-isar/conf/local.conf.sample
> @@ -246,6 +246,9 @@ USER_isar[flags] += "clear-text-password"
>  # Uncomment the below line to debug WIC.
>  # WIC_CREATE_EXTRA_ARGS += "-D"
>  
> +# Uncomment this to also deploy each wic partition as separate file
> (e.g. for swupdate) +#WIC_DEPLOY_PARTITIONS = "1"
> +
>  # Uncomment this to use ccache for custom packages
>  #USE_CCACHE = "1"
>  # Uncomment and set own top level ccache directory to share between
> builds diff --git a/meta/classes/imagetypes_wic.bbclass
> b/meta/classes/imagetypes_wic.bbclass index 61a74d4a..ba8aa15c 100644
> --- a/meta/classes/imagetypes_wic.bbclass
> +++ b/meta/classes/imagetypes_wic.bbclass
> @@ -98,6 +98,7 @@ RECIPE_SYSROOT_NATIVE ?= "/"
>  BUILDCHROOT_DIR = "${BUILDCHROOT_TARGET_DIR}"
>  
>  WIC_CREATE_EXTRA_ARGS ?= ""
> +WIC_DEPLOY_PARTITIONS ?= "0"
>  
>  # taken from OE, do not touch directly
>  WICVARS += "\
> @@ -205,6 +206,10 @@ generate_wic_image() {
>  
>      sudo chown -R $(stat -c "%U" ${LAYERDIR_core}) ${LAYERDIR_core}
> ${LAYERDIR_isar} ${SCRIPTSDIR} || true sudo chown -R $(id -u):$(id
> -g) ${BUILDCHROOT_DIR}/${WICTMP}
> +    # only keep partitions if requested (ending with .p<x>)
> +    if [ "${WIC_DEPLOY_PARTITIONS}" -eq "0" ]; then
> +        find ${BUILDCHROOT_DIR}/${WICTMP} -type f -regextype sed
> -regex ".*\.direct.*\.p[0-9]\{1,\}" -delete
> +    fi
>      find ${BUILDCHROOT_DIR}/${WICTMP} -type f -name "*.direct*" |
> while read f; do suffix=$(basename $f | sed
> 's/\(.*\)\(\.direct\)\(.*\)/\3/') mv -f ${f}
> "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic${suffix}"

Re: [PATCH v5 3/5] add option to control deploy of raw wic partitions

[Jan Kiszka]

On 20.09.22 09:55, Henning Schild wrote:
> Am Mon, 19 Sep 2022 13:20:33 +0200
> schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:
> 
>> This patch adds the WIC_DEPLOY_PARTITIONS option to control
>> if the raw partition files are deployed by wic.
>>
>> We also set the default to not deploy to be closer to OE and to
>> optimize for the average case.
>> In addition, this significantly reduces the disk size required to
>> run the CI.
>>
>> An API changelog entry is added.
> 
> I think the deployment of the partitions was an accident and never
> intended. Cause might be

We might use that in the context of swu (SWUpdate) file generation.
Related recipes can be found in
https://gitlab.com/cip-project/cip-core/isar-cip-core/. But I didn't
check yet if this particular case is related to the SWUpdate, I just
know that we package individual partitions in that use case.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH v5 3/5] add option to control deploy of raw wic partitions

[Henning Schild]

Am Tue, 20 Sep 2022 09:55:43 +0200
schrieb Henning Schild <henning.schild@siemens.com>:

> Am Mon, 19 Sep 2022 13:20:33 +0200
> schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:
> 
> > This patch adds the WIC_DEPLOY_PARTITIONS option to control
> > if the raw partition files are deployed by wic.
> > 
> > We also set the default to not deploy to be closer to OE and to
> > optimize for the average case.
> > In addition, this significantly reduces the disk size required to
> > run the CI.
> > 
> > An API changelog entry is added.  
> 
> I think the deployment of the partitions was an accident and never
> intended. Cause might be
> 
> https://github.com/ilbers/isar/commit/edc10d9361e79f36c8ea5488b1dab4c468213a8d

I think this patch should simply be reverted, which would bring us in
line with OE. Where also only the ".direct" is taken into deploydir.
While wic does have internal compressor support, people seem to rely on
the external one.

So i suggest you simply revert this one.

Keep reading for a changelog suggestion.

> which wanted to deal only with compressed images when wic itself is
> used for compression.
> 
> We should maybe just sharpen that to not include partitions, instead
> of inventing a new interface to remove them again.
> 
> Henning
> 
> > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> > ---
> >  RECIPE-API-CHANGELOG.md             | 6 ++++++
> >  meta-isar/conf/local.conf.sample    | 3 +++
> >  meta/classes/imagetypes_wic.bbclass | 5 +++++
> >  3 files changed, 14 insertions(+)
> > 
> > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> > index 621d110e..0c34d9be 100644
> > --- a/RECIPE-API-CHANGELOG.md
> > +++ b/RECIPE-API-CHANGELOG.md
> > @@ -419,3 +419,9 @@ packages and will be lost after a given build
> > session finishes. 
> >  Any package build requirements for the rootfs should be satisfied
> > in the Debian way via package dependencies.
> > +
> > +### Individual WIC partitions are no longer automatically deployed

We used to copy all temporary WIC files, like the partitions, into the
deploy directory. That was intended actually only for compressed wic
images where wic itself would do the compression.
It was never intended to also deploy those partitions, so that will
also not be done anymore. For compressed wic images IMAGE_FSTYPES
should simply be extended with a compressed wic format, like "wic.xz".

Henning

> > +
> > +The individual wic partitions (e.g. *.wic.p1) are no longer
> > automatically +copied into the `DEPLOY_DIR_IMAGE`.
> > +To explicitly deploy these files (e.g. for swupdate), set
> > `WIC_DEPLOY_PARTITIONS = "1"`. diff --git
> > a/meta-isar/conf/local.conf.sample
> > b/meta-isar/conf/local.conf.sample index ce7b58ee..57d06202 100644
> > --- a/meta-isar/conf/local.conf.sample +++
> > b/meta-isar/conf/local.conf.sample @@ -246,6 +246,9 @@
> > USER_isar[flags] += "clear-text-password" # Uncomment the below
> > line to debug WIC. # WIC_CREATE_EXTRA_ARGS += "-D"
> >  
> > +# Uncomment this to also deploy each wic partition as separate file
> > (e.g. for swupdate) +#WIC_DEPLOY_PARTITIONS = "1"
> > +
> >  # Uncomment this to use ccache for custom packages
> >  #USE_CCACHE = "1"
> >  # Uncomment and set own top level ccache directory to share between
> > builds diff --git a/meta/classes/imagetypes_wic.bbclass
> > b/meta/classes/imagetypes_wic.bbclass index 61a74d4a..ba8aa15c
> > 100644 --- a/meta/classes/imagetypes_wic.bbclass
> > +++ b/meta/classes/imagetypes_wic.bbclass
> > @@ -98,6 +98,7 @@ RECIPE_SYSROOT_NATIVE ?= "/"
> >  BUILDCHROOT_DIR = "${BUILDCHROOT_TARGET_DIR}"
> >  
> >  WIC_CREATE_EXTRA_ARGS ?= ""
> > +WIC_DEPLOY_PARTITIONS ?= "0"
> >  
> >  # taken from OE, do not touch directly
> >  WICVARS += "\
> > @@ -205,6 +206,10 @@ generate_wic_image() {
> >  
> >      sudo chown -R $(stat -c "%U" ${LAYERDIR_core}) ${LAYERDIR_core}
> > ${LAYERDIR_isar} ${SCRIPTSDIR} || true sudo chown -R $(id -u):$(id
> > -g) ${BUILDCHROOT_DIR}/${WICTMP}
> > +    # only keep partitions if requested (ending with .p<x>)
> > +    if [ "${WIC_DEPLOY_PARTITIONS}" -eq "0" ]; then
> > +        find ${BUILDCHROOT_DIR}/${WICTMP} -type f -regextype sed
> > -regex ".*\.direct.*\.p[0-9]\{1,\}" -delete
> > +    fi
> >      find ${BUILDCHROOT_DIR}/${WICTMP} -type f -name "*.direct*" |
> > while read f; do suffix=$(basename $f | sed
> > 's/\(.*\)\(\.direct\)\(.*\)/\3/') mv -f ${f}
> > "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic${suffix}"  
> 

Re: [PATCH v5 3/5] add option to control deploy of raw wic partitions

[Henning Schild]

Am Tue, 20 Sep 2022 10:39:24 +0200
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 20.09.22 09:55, Henning Schild wrote:
> > Am Mon, 19 Sep 2022 13:20:33 +0200
> > schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:
> >   
> >> This patch adds the WIC_DEPLOY_PARTITIONS option to control
> >> if the raw partition files are deployed by wic.
> >>
> >> We also set the default to not deploy to be closer to OE and to
> >> optimize for the average case.
> >> In addition, this significantly reduces the disk size required to
> >> run the CI.
> >>
> >> An API changelog entry is added.  
> > 
> > I think the deployment of the partitions was an accident and never
> > intended. Cause might be  
> 
> We might use that in the context of swu (SWUpdate) file generation.
> Related recipes can be found in
> https://gitlab.com/cip-project/cip-core/isar-cip-core/. But I didn't
> check yet if this particular case is related to the SWUpdate, I just
> know that we package individual partitions in that use case.

Yes that use-case is "known" but still the patch was never created to
support that. I will happily assume that swupdate users never used
those partitions from that accident. But i still proposed a changelog
entry saying a word or two.

Henning

> Jan
>