From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7226603299007365120 X-Received: by 2002:a05:620a:2a04:b0:75b:1b92:612c with SMTP id o4-20020a05620a2a0400b0075b1b92612cmr2509582qkp.4.1684901149502; Tue, 23 May 2023 21:05:49 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6214:1849:b0:623:9507:9d5b with SMTP id d9-20020a056214184900b0062395079d5bls9539558qvy.1.-pod-prod-08-us; Tue, 23 May 2023 21:05:48 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ64pZR8aqsM+DJiCRCYGKdH3crL4Lt7twqYuX87I8iNHh+n4LHNsZBfKF95iWUkH1+IFykz X-Received: by 2002:a05:6102:3c1:b0:439:e3c:8ef8 with SMTP id n1-20020a05610203c100b004390e3c8ef8mr4440524vsq.28.1684901148469; Tue, 23 May 2023 21:05:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684901148; cv=none; d=google.com; s=arc-20160816; b=zpemF/SBzxR8AFLVnKxHoT+FTbiJSDLi5YHWBxed3UEpCIGGUHzGCkt2iQe0/V7LNm 5zLeThTMmOa8kpujmTTjrlTV4x+yadPnkXKDBMuD3DaD8Fl8OW5C4C44LiLv6DnqZTKG 9kKAWqH4a5XOucD6twq4mQmbU9MDmCEbSjZ76brlkdlGtWTCF+GE7zK620K0QzDNEpJ8 m9bFYHGGMU/dIesWSqgoVmIzszTPGsLiWJmUzmw7rXbLV8p2FYoD+4kJuIVQ4MxIv/SN rO80ckAMyOnUwX2f0YP4AiLphU3EavkYqnwPsqCGexG/gxmlbj/YJ0jNgfb4WvDm0sdF N12g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id; bh=N6xOgQ5RUUB0eiTp/WmqzCfrZnHwWA1ECvO+/s67hnA=; b=mhQ+jRCNzlZgHIWxOgtm7XJQjCjTfR0X3TbmWihBprcsZOHL3CpzE/mk3n2G9MxUEk QkQGoHoDMVyATGdkkUkMXZx3b3Z3KWri1VPeFyfQr7vIYY9R+4b6lL8UI+Clff5kcE04 OCuSKeXrTLS2tJqchIVnUbn7jaUf465Eo8bXbnW3i2Mk2GYS41jKfD1TIs0PvlWZgvwF NROxpTwm5W8Go2Z9Oc9oQW0zcsHcJQxAjcqoNL1ZJ1jsrgU1flMMyO9k2iwzyepX9e/F 2GKDwqPZQVXCRtnizejW3FA0c+w/P+U4416/bxU1Gxb/Ihuz73s/UIxpG/0u90mt8fk9 m0xg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id s24-20020a056130021800b007836a48d143si1089780uac.1.2023.05.23.21.05.46 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 23 May 2023 21:05:48 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from [127.0.0.1] (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 34O45eA2000394 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 24 May 2023 06:05:41 +0200 Message-ID: Subject: Re: [PATCH 1/1] add zchunk compression support From: Uladzimir Bely To: "Moessbauer, Felix" , "Schmidt, Adriaan" , "isar-users@googlegroups.com" Cc: "Schild, Henning" Date: Wed, 24 May 2023 07:05:43 +0300 In-Reply-To: References: <20230427055118.1993072-1-felix.moessbauer@siemens.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.48.1 (by Flathub.org) MIME-Version: 1.0 X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: keikzLS/c6d7 On Thu, 2023-04-27 at 06:37 +0000, 'Moessbauer, Felix' via isar-users wrote: > On Thu, 2023-04-27 at 06:18 +0000, Schmidt, Adriaan (T CED SES-DE) > wrote: > > 'Felix Moessbauer' via isar-users , > > Donnerstag, 27. April 2023 07:51: > > > This patch adds support to compress images with zchunk. The > > > compression > > > format is optimized for delta transfers, by ensuring that small > > > changes > > > on the input also only invalidate some blocks of the output. > > >=20 > > > There are not yet many configuration options available, but once > > > available, they can be added to the ZCK_DEFAULTS variable. > > >=20 > > > Signed-off-by: Felix Moessbauer > > > --- > > > =C2=A0meta/classes/imagetypes.bbclass | 5 ++++- > > > =C2=A0meta/conf/bitbake.conf=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 | 3 +++ > > > =C2=A02 files changed, 7 insertions(+), 1 deletion(-) > > >=20 > > > diff --git a/meta/classes/imagetypes.bbclass > > > b/meta/classes/imagetypes.bbclass > > > index a80a6da5..3639662b 100644 > > > --- a/meta/classes/imagetypes.bbclass > > > +++ b/meta/classes/imagetypes.bbclass > > > @@ -93,7 +93,7 @@ IMAGE_CMD:ubi() { > > > =C2=A0IMAGE_CMD:ubi[depends] =3D "${PN}:do_transform_template" > > >=20 > > > =C2=A0# image conversions > > > -IMAGE_CONVERSIONS =3D "gz xz zst" > > > +IMAGE_CONVERSIONS =3D "gz xz zst zck" > > >=20 > > > =C2=A0CONVERSION_CMD:gz =3D "${SUDO_CHROOT} sh -c 'gzip -f -9 -n -c -= - > > > rsyncable > > > ${IMAGE_FILE_CHROOT} > ${IMAGE_FILE_CHROOT}.gz'" > > > =C2=A0CONVERSION_DEPS:gz =3D "gzip" > > > @@ -103,3 +103,6 @@ CONVERSION_DEPS:xz =3D "xz-utils" > > >=20 > > > =C2=A0CONVERSION_CMD:zst =3D "${SUDO_CHROOT} sh -c 'zstd -c --sparse > > > ${ZSTD_DEFAULTS} ${IMAGE_FILE_CHROOT} > > > > ${IMAGE_FILE_CHROOT}.zst'" > > > =C2=A0CONVERSION_DEPS:zst =3D "zstd" > > > + > > > +CONVERSION_CMD:zck =3D "${SUDO_CHROOT} sh -c 'cd $(dirname > > > ${IMAGE_FILE_CHROOT}); zck ${ZCK_DEFAULTS} ${IMAGE_FILE_CHROOT}'" > >=20 > > Hi Felix, > >=20 > > This looks like it compresses in-place, i.e., removes the original > > file. >=20 > It looks like that, but there zchunk behaves differently from the > other > compressors. It actually just adds the compressed file next to the > input file (version 1.2.3). Also, it is not possible to tell zck to > write to stdout, hence the changedir logic. >=20 > I already thought about adding an issue in the upstream project [1] > about the CLI api. It is pretty non-standard. Another issue is, that > the underlying zstd compression level cannot be set and compression > is > done sequentially. Also the documentation is really sparse. >=20 > Anyways, while the tool is in a pretty early state, it is still good > to > have it to get some experience on delta updates. >=20 > [1] https://github.com/zchunk/zchunk >=20 > Felix >=20 > > That would break the imagetypes/conversion logic, where it is > > assumed > > that the original file still exists after conversion. That lets us > > define > > things like IMAGE_FSYTPES=3D"wic wic.zck" if we want both files in > > the > > output. > > In case the uncompressed one is not needed (it's not in > > IMAGE_FSTYPES), it will > > be deleted explicitly by the imagetypes class. > >=20 > > Adriaan > >=20 > > > +CONVERSION_DEPS:zck =3D "zchunk" > > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > > > index c660660a..f73e4838 100644 > > > --- a/meta/conf/bitbake.conf > > > +++ b/meta/conf/bitbake.conf > > > @@ -144,6 +144,9 @@ ZSTD_LEVEL ?=3D "19" > > > =C2=A0ZSTD_DEFAULTS ?=3D "--rsyncable -${ZSTD_LEVEL} -- > > > threads=3D${ZSTD_THREADS}" > > > =C2=A0ZSTD_DEFAULTS[vardepsexclude] +=3D "ZSTD_LEVEL ZSTD_THREADS" > > >=20 > > > +# Default compression settings for zchunk > > > +ZCK_DEFAULTS ?=3D "" > > > + > > > =C2=A0BBINCLUDELOGS ??=3D "yes" > > >=20 > > > =C2=A0# Add event handlers for bitbake > > > -- > > > 2.34.1 > > >=20 > > > -- > > > You received this message because you are subscribed to the > > > Google > > > Groups > > > "isar-users" group. > > > To unsubscribe from this group and stop receiving emails from it, > > > send an > > > email to isar-users+unsubscribe@googlegroups.com. > > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/isar-users/20230427055118.1993072-1 > > > - > > > felix.moessbauer%40siemens.com. >=20 Bumping the topic to make a decision on merge. Patch itself passes fast/full CI, so if there are no other reasons against (except a bit non-standard compression behaviour), we could merge it.