Re: [PATCH 1/1] add zchunk compression support

["Moessbauer, Felix" <felix.moessbauer@siemens.com>]

On Thu, 2023-04-27 at 06:18 +0000, Schmidt, Adriaan (T CED SES-DE)
wrote:
> 'Felix Moessbauer' via isar-users <isar-users@googlegroups.com>,
> Donnerstag, 27. April 2023 07:51:
> > This patch adds support to compress images with zchunk. The
> > compression
> > format is optimized for delta transfers, by ensuring that small
> > changes
> > on the input also only invalidate some blocks of the output.
> > 
> > There are not yet many configuration options available, but once
> > available, they can be added to the ZCK_DEFAULTS variable.
> > 
> > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> > ---
> >  meta/classes/imagetypes.bbclass | 5 ++++-
> >  meta/conf/bitbake.conf          | 3 +++
> >  2 files changed, 7 insertions(+), 1 deletion(-)
> > 
> > diff --git a/meta/classes/imagetypes.bbclass
> > b/meta/classes/imagetypes.bbclass
> > index a80a6da5..3639662b 100644
> > --- a/meta/classes/imagetypes.bbclass
> > +++ b/meta/classes/imagetypes.bbclass
> > @@ -93,7 +93,7 @@ IMAGE_CMD:ubi() {
> >  IMAGE_CMD:ubi[depends] = "${PN}:do_transform_template"
> > 
> >  # image conversions
> > -IMAGE_CONVERSIONS = "gz xz zst"
> > +IMAGE_CONVERSIONS = "gz xz zst zck"
> > 
> >  CONVERSION_CMD:gz = "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c --
> > rsyncable
> > ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'"
> >  CONVERSION_DEPS:gz = "gzip"
> > @@ -103,3 +103,6 @@ CONVERSION_DEPS:xz = "xz-utils"
> > 
> >  CONVERSION_CMD:zst = "${SUDO_CHROOT} sh -c 'zstd -c --sparse
> > ${ZSTD_DEFAULTS} ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.zst'"
> >  CONVERSION_DEPS:zst = "zstd"
> > +
> > +CONVERSION_CMD:zck = "${SUDO_CHROOT} sh -c 'cd $(dirname
> > ${IMAGE_FILE_CHROOT}); zck ${ZCK_DEFAULTS} ${IMAGE_FILE_CHROOT}'"
> 
> Hi Felix,
> 
> This looks like it compresses in-place, i.e., removes the original
> file.

It looks like that, but there zchunk behaves differently from the other
compressors. It actually just adds the compressed file next to the
input file (version 1.2.3). Also, it is not possible to tell zck to
write to stdout, hence the changedir logic.

I already thought about adding an issue in the upstream project [1]
about the CLI api. It is pretty non-standard. Another issue is, that
the underlying zstd compression level cannot be set and compression is
done sequentially. Also the documentation is really sparse.

Anyways, while the tool is in a pretty early state, it is still good to
have it to get some experience on delta updates.

[1] https://github.com/zchunk/zchunk

Felix

> That would break the imagetypes/conversion logic, where it is assumed
> that the original file still exists after conversion. That lets us
> define
> things like IMAGE_FSYTPES="wic wic.zck" if we want both files in the
> output.
> In case the uncompressed one is not needed (it's not in
> IMAGE_FSTYPES), it will
> be deleted explicitly by the imagetypes class.
> 
> Adriaan
> 
> > +CONVERSION_DEPS:zck = "zchunk"
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > index c660660a..f73e4838 100644
> > --- a/meta/conf/bitbake.conf
> > +++ b/meta/conf/bitbake.conf
> > @@ -144,6 +144,9 @@ ZSTD_LEVEL ?= "19"
> >  ZSTD_DEFAULTS ?= "--rsyncable -${ZSTD_LEVEL} --
> > threads=${ZSTD_THREADS}"
> >  ZSTD_DEFAULTS[vardepsexclude] += "ZSTD_LEVEL ZSTD_THREADS"
> > 
> > +# Default compression settings for zchunk
> > +ZCK_DEFAULTS ?= ""
> > +
> >  BBINCLUDELOGS ??= "yes"
> > 
> >  # Add event handlers for bitbake
> > --
> > 2.34.1
> > 
> > --
> > You received this message because you are subscribed to the Google
> > Groups
> > "isar-users" group.
> > To unsubscribe from this group and stop receiving emails from it,
> > send an
> > email to isar-users+unsubscribe@googlegroups.com.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/isar-users/20230427055118.1993072-1
> > -
> > felix.moessbauer%40siemens.com.