RE: [PATCH v5 3/5] add option to control deploy of raw wic partitions

["Moessbauer, Felix" <felix.moessbauer@siemens.com>]

> -----Original Message-----
> From: Schild, Henning (T CED SES-DE) <henning.schild@siemens.com>
> Sent: Monday, September 19, 2022 2:03 PM
> To: Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>
> Cc: isar-users@googlegroups.com; Schmidt, Adriaan (T CED SES-DE)
> <adriaan.schmidt@siemens.com>; Kiszka, Jan (T CED)
> <jan.kiszka@siemens.com>; ibr@ilbers.de; amikan@ilbers.de
> Subject: Re: [PATCH v5 3/5] add option to control deploy of raw wic partitions
> 
> Am Mon, 19 Sep 2022 13:20:33 +0200
> schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:
> 
> > This patch adds the WIC_DEPLOY_PARTITIONS option to control if the raw
> > partition files are deployed by wic.
> >
> > We also set the default to not deploy to be closer to OE and to
> > optimize for the average case.
> > In addition, this significantly reduces the disk size required to run
> > the CI.
> >
> > An API changelog entry is added.
> >
> > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> > ---
> >  RECIPE-API-CHANGELOG.md             | 6 ++++++
> >  meta-isar/conf/local.conf.sample    | 3 +++
> >  meta/classes/imagetypes_wic.bbclass | 5 +++++
> >  3 files changed, 14 insertions(+)
> >
> > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index
> > 621d110e..0c34d9be 100644
> > --- a/RECIPE-API-CHANGELOG.md
> > +++ b/RECIPE-API-CHANGELOG.md
> > @@ -419,3 +419,9 @@ packages and will be lost after a given build
> > session finishes.
> >  Any package build requirements for the rootfs should be satisfied in
> > the Debian way via package dependencies.
> > +
> > +### Individual WIC partitions are no longer automatically deployed
> > +
> > +The individual wic partitions (e.g. *.wic.p1) are no longer
> > automatically +copied into the `DEPLOY_DIR_IMAGE`.
> > +To explicitly deploy these files (e.g. for swupdate), set
> > `WIC_DEPLOY_PARTITIONS = "1"`. diff --git
> > a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
> > index ce7b58ee..57d06202 100644 --- a/meta-isar/conf/local.conf.sample
> > +++ b/meta-isar/conf/local.conf.sample
> > @@ -246,6 +246,9 @@ USER_isar[flags] += "clear-text-password"
> >  # Uncomment the below line to debug WIC.
> >  # WIC_CREATE_EXTRA_ARGS += "-D"
> >
> > +# Uncomment this to also deploy each wic partition as separate file
> > (e.g. for swupdate) +#WIC_DEPLOY_PARTITIONS = "1"
> > +
> >  # Uncomment this to use ccache for custom packages  #USE_CCACHE = "1"
> >  # Uncomment and set own top level ccache directory to share between
> > builds diff --git a/meta/classes/imagetypes_wic.bbclass
> > b/meta/classes/imagetypes_wic.bbclass index 61a74d4a..ba8aa15c 100644
> > --- a/meta/classes/imagetypes_wic.bbclass
> > +++ b/meta/classes/imagetypes_wic.bbclass
> > @@ -98,6 +98,7 @@ RECIPE_SYSROOT_NATIVE ?= "/"
> >  BUILDCHROOT_DIR = "${BUILDCHROOT_TARGET_DIR}"
> >
> >  WIC_CREATE_EXTRA_ARGS ?= ""
> > +WIC_DEPLOY_PARTITIONS ?= "0"
> >
> >  # taken from OE, do not touch directly  WICVARS += "\ @@ -205,6
> > +206,10 @@ generate_wic_image() {
> >
> >      sudo chown -R $(stat -c "%U" ${LAYERDIR_core}) ${LAYERDIR_core}
> > ${LAYERDIR_isar} ${SCRIPTSDIR} || true sudo chown -R $(id -u):$(id
> > -g) ${BUILDCHROOT_DIR}/${WICTMP}
> > +    # only keep partitions if requested (ending with .p<x>)
> > +    if [ "${WIC_DEPLOY_PARTITIONS}" -eq "0" ]; then
> > +        find ${BUILDCHROOT_DIR}/${WICTMP} -type f -regextype sed
> > -regex ".*\.direct.*\.p[0-9]\{1,\}" -delete
> 
> I just had a look at OE and they seem to do things differently. They call wic with
> a -w "temporary workdir to use for intermediate files"
> and later move only the ".direct" to DEPLOY/<image>.wic

I had a very careful look at the OE wic script as well and the only difference is that they do not copy any temporary files (like the partitions) to the deploy dir.
That's why they don't do this "find" but simply "move" the .direct file [1].
There had been proposed patches to change that (also from Siemens), but they never have been accepted.

The added "-w $tmp_wic" is indeed a divergence, but that is independent of the added / changed logic in this series.
In the end, the OE data in $tmp_wic is never re-used again.

> 
> So i would suggest to do that as well, and not delete valid files that maybe a
> second run or another tool might end up missing. Say you generate a .wic.xz and
> a .swu that swu will maybe need the partition image where the .wic.xz itself
> would not deploy it.

If you look carefully at the following lines after the "find", you will see that the
WICTMP is deleted anyways. And in between, there is no extension point where any other tool could integrate and read the temporary files.
By that, I assume that my approach is correct and reliable.

[1] https://git.openembedded.org/openembedded-core/tree/meta/classes/image_types_wic.bbclass?id=958ee0eede859bdba659e3343856b1c226207854#n68

Felix

> 
> Henning
> 
> > +    fi
> >      find ${BUILDCHROOT_DIR}/${WICTMP} -type f -name "*.direct*" |
> > while read f; do suffix=$(basename $f | sed
> > 's/\(.*\)\(\.direct\)\(.*\)/\3/') mv -f ${f}
> > "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic${suffix}"