* [PATCH v2 0/1] sstate: do not keep intermediate sstate files @ 2022-02-24 7:36 Uladzimir Bely 2022-02-24 7:36 ` [PATCH v2 1/1] " Uladzimir Bely 2022-02-24 7:56 ` [PATCH v2 0/1] " Schmidt, Adriaan 0 siblings, 2 replies; 6+ messages in thread From: Uladzimir Bely @ 2022-02-24 7:36 UTC (permalink / raw) To: isar-users Sstate is quite demanding for disk space. And it often causes failures in CI, related to 'out-of-space'. This patchset minimizes space usage by intermediate state files by removing them: - after sstate-cache populated: with separate task in task queue - after sstate-cache unpacked: by direct call of this task Some results shown below for three different cases: - sstate disabled; - sstate enabled; - sstate enabled + cleanup patch. bullseye-arm64 cross build with full IMAGE_INSTALL list: $ sudo du -sh build-* 5,7G build-nosstate 12G build-sstate 7,6G build-sstate-cleanup bullseye-arm64 cross build with empty IMAGE_INSTALL list: $ sudo du -sh build-* 2,5G build-nosstate 5,3G build-sstate 3,4G build-sstate-cleanup bullseye-amd64 native build with empty IMAGE_INSTALL list: $ sudo du -sh build-* 3,2G build-nosstate 6,4G build-sstate 4,2G build-sstate-cleanup Changes since v1: - patchset renamed from "sstate: compress rootfs sstate files"; - sstate intermediate files compression replaced by removing; - dpkg intermediate files are also handled. Uladzimir Bely (1): sstate: do not keep intermediate sstate files meta/classes/dpkg-base.bbclass | 7 +++++++ meta/classes/rootfs.bbclass | 7 +++++++ meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 +++++++ 3 files changed, 21 insertions(+) -- 2.20.1 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v2 1/1] sstate: do not keep intermediate sstate files 2022-02-24 7:36 [PATCH v2 0/1] sstate: do not keep intermediate sstate files Uladzimir Bely @ 2022-02-24 7:36 ` Uladzimir Bely 2022-02-24 7:56 ` [PATCH v2 0/1] " Schmidt, Adriaan 1 sibling, 0 replies; 6+ messages in thread From: Uladzimir Bely @ 2022-02-24 7:36 UTC (permalink / raw) To: isar-users This saves about 2...5 GiB of disk space per multiconfig depending on package list, cross-build mode, etc. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- meta/classes/dpkg-base.bbclass | 7 +++++++ meta/classes/rootfs.bbclass | 7 +++++++ meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 +++++++ 3 files changed, 21 insertions(+) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 928856a9..b5090d13 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -253,6 +253,7 @@ dpkg_build_sstate_finalize() { if [ -n "$(find ${DPKG_SSTATE} -maxdepth 1 -name '*.deb' -print -quit)" ]; then ln -f ${DPKG_SSTATE}/*.deb -t ${S}/.. fi + do_dpkg_build_sstate_cleanup } python do_dpkg_build_setscene() { @@ -263,6 +264,12 @@ python do_dpkg_build_setscene() { addtask dpkg_build_setscene do_dpkg_build_setscene[dirs] += "${S}/.." +do_dpkg_build_sstate_cleanup() { + rm -rf ${DPKG_SSTATE} +} + +addtask dpkg_build_sstate_cleanup before do_build after do_dpkg_build + KEEP_INSTALLED_ON_CLEAN ?= "0" CLEANFUNCS += "deb_clean" diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 2bdb3b6d..0c7b535c 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -305,6 +305,7 @@ do_rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock" rootfs_install_sstate_finalize() { sudo tar -C ${WORKDIR} -xpf ${ROOTFS_SSTATE}/rootfs.tar + do_rootfs_install_sstate_cleanup } python do_rootfs_install_setscene() { @@ -312,3 +313,9 @@ python do_rootfs_install_setscene() { bb.build.exec_func('rootfs_install_sstate_finalize', d) } addtask do_rootfs_install_setscene + +do_rootfs_install_sstate_cleanup() { + rm -rf ${ROOTFS_SSTATE} +} + +addtask rootfs_install_sstate_cleanup before do_rootfs_postprocess after do_rootfs_install diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 2f483f5a..16c04bf8 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -408,6 +408,7 @@ bootstrap_sstate_prepare() { bootstrap_sstate_finalize() { sudo tar -C $(dirname "${ROOTFSDIR}") -xpf ${BOOTSTRAP_SSTATE}/bootstrap.tar sudo ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}" + do_bootstrap_sstate_cleanup } python do_bootstrap_setscene() { @@ -418,6 +419,12 @@ python do_bootstrap_setscene() { addtask do_bootstrap_setscene do_bootstrap_setscene[dirs] = "${DEPLOY_DIR_BOOTSTRAP}" +do_bootstrap_sstate_cleanup() { + rm -rf ${BOOTSTRAP_SSTATE} +} + +addtask bootstrap_sstate_cleanup before do_build after do_bootstrap + CLEANFUNCS = "clean_deploy" clean_deploy() { rm -f "${DEPLOY_ISAR_BOOTSTRAP}" -- 2.20.1 ^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: [PATCH v2 0/1] sstate: do not keep intermediate sstate files 2022-02-24 7:36 [PATCH v2 0/1] sstate: do not keep intermediate sstate files Uladzimir Bely 2022-02-24 7:36 ` [PATCH v2 1/1] " Uladzimir Bely @ 2022-02-24 7:56 ` Schmidt, Adriaan 2022-02-24 8:19 ` Uladzimir Bely 1 sibling, 1 reply; 6+ messages in thread From: Schmidt, Adriaan @ 2022-02-24 7:56 UTC (permalink / raw) To: Uladzimir Bely, isar-users Uladzimir Bely, 24. Februar 2022 08:36: > Sstate is quite demanding for disk space. And it often causes failures > in CI, related to 'out-of-space'. If those few GB cause problems, then you really need a larger disk :) But I get the point, and I agree that we should try to save space. > This patchset minimizes space usage by intermediate state files by > removing them: > - after sstate-cache populated: with separate task in task queue I would like to avoid adding extra tasks. There seems to be a hook to insert functions (SSTATEPOSTCREATEFUNCS), but as sstate is not really documented, I have not yet figured out how to make our use case work. > - after sstate-cache unpacked: by direct call of this task > > Some results shown below for three different cases: > > - sstate disabled; > - sstate enabled; > - sstate enabled + cleanup patch. > > bullseye-arm64 cross build with full IMAGE_INSTALL list: > > $ sudo du -sh build-* > 5,7G build-nosstate > 12G build-sstate > 7,6G build-sstate-cleanup > > bullseye-arm64 cross build with empty IMAGE_INSTALL list: > > $ sudo du -sh build-* > 2,5G build-nosstate > 5,3G build-sstate > 3,4G build-sstate-cleanup > > bullseye-amd64 native build with empty IMAGE_INSTALL list: > > $ sudo du -sh build-* > 3,2G build-nosstate > 6,4G build-sstate > 4,2G build-sstate-cleanup > > > Changes since v1: > - patchset renamed from "sstate: compress rootfs sstate files"; > - sstate intermediate files compression replaced by removing; > - dpkg intermediate files are also handled. It's unnecessary for dpkg, because we use hardlinks to the *.deb file, so no extra disk space is consumed for the sstate preparation. Adriaan > > Uladzimir Bely (1): > sstate: do not keep intermediate sstate files > > meta/classes/dpkg-base.bbclass | 7 +++++++ > meta/classes/rootfs.bbclass | 7 +++++++ > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 +++++++ > 3 files changed, 21 insertions(+) > > -- > 2.20.1 > > -- > You received this message because you are subscribed to the Google Groups > "isar-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to isar-users+unsubscribe@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/isar-users/20220224073629.23717-1- > ubely%40ilbers.de. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2 0/1] sstate: do not keep intermediate sstate files 2022-02-24 7:56 ` [PATCH v2 0/1] " Schmidt, Adriaan @ 2022-02-24 8:19 ` Uladzimir Bely 2022-02-24 9:04 ` Schmidt, Adriaan 0 siblings, 1 reply; 6+ messages in thread From: Uladzimir Bely @ 2022-02-24 8:19 UTC (permalink / raw) To: isar-users; +Cc: Schmidt, Adriaan In the email from Thursday, 24 February 2022 10:56:48 +03 user Schmidt, Adriaan wrote: > Uladzimir Bely, 24. Februar 2022 08:36: > > Sstate is quite demanding for disk space. And it often causes failures > > in CI, related to 'out-of-space'. > > If those few GB cause problems, then you really need a larger disk :) > But I get the point, and I agree that we should try to save space. > These few GBs result to ~100GiB per CI job (when doing full builds) :) > > This patchset minimizes space usage by intermediate state files by > > removing them: > > - after sstate-cache populated: with separate task in task queue > > I would like to avoid adding extra tasks. There seems to be a hook > to insert functions (SSTATEPOSTCREATEFUNCS), but as sstate is not > really documented, I have not yet figured out how to make our use case work. Yes, it would be better if there are some post-sstate functions we could use instead of inserting one more task after one we are handling... > > - after sstate-cache unpacked: by direct call of this task > > > > Some results shown below for three different cases: > > > > - sstate disabled; > > - sstate enabled; > > - sstate enabled + cleanup patch. > > > > bullseye-arm64 cross build with full IMAGE_INSTALL list: > > > > $ sudo du -sh build-* > > 5,7G build-nosstate > > 12G build-sstate > > 7,6G build-sstate-cleanup > > > > bullseye-arm64 cross build with empty IMAGE_INSTALL list: > > > > $ sudo du -sh build-* > > 2,5G build-nosstate > > 5,3G build-sstate > > 3,4G build-sstate-cleanup > > > > bullseye-amd64 native build with empty IMAGE_INSTALL list: > > > > $ sudo du -sh build-* > > 3,2G build-nosstate > > 6,4G build-sstate > > 4,2G build-sstate-cleanup > > > > > > Changes since v1: > > - patchset renamed from "sstate: compress rootfs sstate files"; > > - sstate intermediate files compression replaced by removing; > > - dpkg intermediate files are also handled. > > It's unnecessary for dpkg, because we use hardlinks to the *.deb file, > so no extra disk space is consumed for the sstate preparation. Yes, I saw it, just implemented in the similar way. It may be useful if files are copied to another (slow) disk after build finishes - I'm afraid hardlinks won't work in this case. > > Adriaan > > > Uladzimir Bely (1): > > sstate: do not keep intermediate sstate files > > > > meta/classes/dpkg-base.bbclass | 7 +++++++ > > meta/classes/rootfs.bbclass | 7 +++++++ > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 +++++++ > > 3 files changed, 21 insertions(+) > > > > -- > > 2.20.1 > > > > -- > > You received this message because you are subscribed to the Google Groups > > "isar-users" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to isar-users+unsubscribe@googlegroups.com. > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/isar-users/20220224073629.23717-1-> > ubely%40ilbers.de. -- Uladzimir Bely ^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: [PATCH v2 0/1] sstate: do not keep intermediate sstate files 2022-02-24 8:19 ` Uladzimir Bely @ 2022-02-24 9:04 ` Schmidt, Adriaan 2022-02-24 15:57 ` Henning Schild 0 siblings, 1 reply; 6+ messages in thread From: Schmidt, Adriaan @ 2022-02-24 9:04 UTC (permalink / raw) To: Uladzimir Bely, isar-users Uladzimir Bely <ubely@ilbers.de>, 24. Februar 2022 09:20: > In the email from Thursday, 24 February 2022 10:56:48 +03 user Schmidt, > Adriaan wrote: > > Uladzimir Bely, 24. Februar 2022 08:36: > > > Sstate is quite demanding for disk space. And it often causes failures > > > in CI, related to 'out-of-space'. > > > > If those few GB cause problems, then you really need a larger disk :) > > But I get the point, and I agree that we should try to save space. > > > > These few GBs result to ~100GiB per CI job (when doing full builds) :) > > > > This patchset minimizes space usage by intermediate state files by > > > removing them: > > > - after sstate-cache populated: with separate task in task queue > > > > I would like to avoid adding extra tasks. There seems to be a hook > > to insert functions (SSTATEPOSTCREATEFUNCS), but as sstate is not > > really documented, I have not yet figured out how to make our use case > work. > > Yes, it would be better if there are some post-sstate functions we could use > instead of inserting one more task after one we are handling... This seems to work for bootstrap: === SSTATETASKS += "do_bootstrap" SSTATECREATEFUNCS += "bootstrap_sstate_prepare" SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize" bootstrap_sstate_prepare() { # this runs in SSTATE_BUILDDIR, which will be deleted automatically sudo tar -C $(dirname "${ROOTFSDIR}") -cpf "bootstrap.tar" --one-file-system $(basename "${ROOTFSDIR}") sudo chown $(id -u):$(id -g) "bootstrap.tar" } bootstrap_sstate_finalize() { # this runs in SSTATE_INSTDIR # - after building the bootstrap, the tar won't be there, but we also don't need to unpack # - after restoring from cache, there will be a tar which we unpack and then delete if [ -f "bootstrap.tar" ]; then sudo tar -C $(dirname "${ROOTFSDIR}") -xpf "bootstrap.tar" sudo ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}" rm "bootstrap.tar" fi } python do_bootstrap_setscene() { sstate_setscene(d) } addtask do_bootstrap_setscene do_bootstrap_setscene[dirs] = "${DEPLOY_DIR_BOOTSTRAP}" === It no longer uses the more-or-less documented interfaces, but instead hooks deeper into the completely undocumented sstate logic, so no `sstate-inteceptfuncs` and no explicitly created and cleaned-up staging directories. The same approach should work for rootfs, and could also improve dpkg caching. I will do some more testing and prepare a patch. > > > - after sstate-cache unpacked: by direct call of this task > > > > > > Some results shown below for three different cases: > > > > > > - sstate disabled; > > > - sstate enabled; > > > - sstate enabled + cleanup patch. > > > > > > bullseye-arm64 cross build with full IMAGE_INSTALL list: > > > > > > $ sudo du -sh build-* > > > 5,7G build-nosstate > > > 12G build-sstate > > > 7,6G build-sstate-cleanup > > > > > > bullseye-arm64 cross build with empty IMAGE_INSTALL list: > > > > > > $ sudo du -sh build-* > > > 2,5G build-nosstate > > > 5,3G build-sstate > > > 3,4G build-sstate-cleanup > > > > > > bullseye-amd64 native build with empty IMAGE_INSTALL list: > > > > > > $ sudo du -sh build-* > > > 3,2G build-nosstate > > > 6,4G build-sstate > > > 4,2G build-sstate-cleanup > > > > > > > > > Changes since v1: > > > - patchset renamed from "sstate: compress rootfs sstate files"; > > > - sstate intermediate files compression replaced by removing; > > > - dpkg intermediate files are also handled. > > > > It's unnecessary for dpkg, because we use hardlinks to the *.deb file, > > so no extra disk space is consumed for the sstate preparation. > > Yes, I saw it, just implemented in the similar way. It may be useful if files > are copied to another (slow) disk after build finishes - I'm afraid hardlinks > won't work in this case. In this case we're only linking inside WORKDIR (not to the actual sstate cache). But with the above approach all those intermediate files/links will be handled in a better way. Adriaan > > > > Adriaan > > > > > Uladzimir Bely (1): > > > sstate: do not keep intermediate sstate files > > > > > > meta/classes/dpkg-base.bbclass | 7 +++++++ > > > meta/classes/rootfs.bbclass | 7 +++++++ > > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 +++++++ > > > 3 files changed, 21 insertions(+) > > > > > > -- > > > 2.20.1 > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "isar-users" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to isar-users+unsubscribe@googlegroups.com. > > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/isar-users/20220224073629.23717-1-> > > ubely%40ilbers.de. > > > -- > Uladzimir Bely > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2 0/1] sstate: do not keep intermediate sstate files 2022-02-24 9:04 ` Schmidt, Adriaan @ 2022-02-24 15:57 ` Henning Schild 0 siblings, 0 replies; 6+ messages in thread From: Henning Schild @ 2022-02-24 15:57 UTC (permalink / raw) To: Schmidt, Adriaan; +Cc: Uladzimir Bely, isar-users Am Thu, 24 Feb 2022 09:04:40 +0000 schrieb "Schmidt, Adriaan" <adriaan.schmidt@siemens.com>: > Uladzimir Bely <ubely@ilbers.de>, 24. Februar 2022 09:20: > > In the email from Thursday, 24 February 2022 10:56:48 +03 user > > Schmidt, Adriaan wrote: > > > Uladzimir Bely, 24. Februar 2022 08:36: > > > > Sstate is quite demanding for disk space. And it often causes > > > > failures in CI, related to 'out-of-space'. > > > > > > If those few GB cause problems, then you really need a larger > > > disk :) But I get the point, and I agree that we should try to > > > save space. > > > > These few GBs result to ~100GiB per CI job (when doing full builds) > > :) > > > > This patchset minimizes space usage by intermediate state files > > > > by removing them: > > > > - after sstate-cache populated: with separate task in task > > > > queue > > > > > > I would like to avoid adding extra tasks. There seems to be a hook > > > to insert functions (SSTATEPOSTCREATEFUNCS), but as sstate is not > > > really documented, I have not yet figured out how to make our use > > > case > > work. > > > > Yes, it would be better if there are some post-sstate functions we > > could use instead of inserting one more task after one we are > > handling... > > This seems to work for bootstrap: > === > SSTATETASKS += "do_bootstrap" > SSTATECREATEFUNCS += "bootstrap_sstate_prepare" > SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize" > > bootstrap_sstate_prepare() { > # this runs in SSTATE_BUILDDIR, which will be deleted > automatically sudo tar -C $(dirname "${ROOTFSDIR}") -cpf > "bootstrap.tar" --one-file-system $(basename "${ROOTFSDIR}") sudo > chown $(id -u):$(id -g) "bootstrap.tar" } > > bootstrap_sstate_finalize() { > # this runs in SSTATE_INSTDIR > # - after building the bootstrap, the tar won't be there, but we > also don't need to unpack # - after restoring from cache, there will > be a tar which we unpack and then delete if [ -f "bootstrap.tar" ]; > then sudo tar -C $(dirname "${ROOTFSDIR}") -xpf "bootstrap.tar" > sudo ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}" > rm "bootstrap.tar" > fi > } > > python do_bootstrap_setscene() { > sstate_setscene(d) > } > > addtask do_bootstrap_setscene > do_bootstrap_setscene[dirs] = "${DEPLOY_DIR_BOOTSTRAP}" > === > > It no longer uses the more-or-less documented interfaces, but > instead hooks deeper into the completely undocumented sstate logic, > so no `sstate-inteceptfuncs` and no explicitly created and cleaned-up > staging directories. I read code but did not look for docs, "sstate-inteceptfuncs" is used in OE like once so also nothing too common it seems. It might be worth double checking what has happened in more recent sstate. If what we end up with looks like it could survive a bump it might just be fine. > The same approach should work for rootfs, and could also improve dpkg > caching. I will do some more testing and prepare a patch. Cool! Thanks! Henning > > > > - after sstate-cache unpacked: by direct call of this task > > > > > > > > Some results shown below for three different cases: > > > > > > > > - sstate disabled; > > > > - sstate enabled; > > > > - sstate enabled + cleanup patch. > > > > > > > > bullseye-arm64 cross build with full IMAGE_INSTALL list: > > > > > > > > $ sudo du -sh build-* > > > > 5,7G build-nosstate > > > > 12G build-sstate > > > > 7,6G build-sstate-cleanup > > > > > > > > bullseye-arm64 cross build with empty IMAGE_INSTALL list: > > > > > > > > $ sudo du -sh build-* > > > > 2,5G build-nosstate > > > > 5,3G build-sstate > > > > 3,4G build-sstate-cleanup > > > > > > > > bullseye-amd64 native build with empty IMAGE_INSTALL list: > > > > > > > > $ sudo du -sh build-* > > > > 3,2G build-nosstate > > > > 6,4G build-sstate > > > > 4,2G build-sstate-cleanup > > > > > > > > > > > > Changes since v1: > > > > - patchset renamed from "sstate: compress rootfs sstate files"; > > > > - sstate intermediate files compression replaced by removing; > > > > - dpkg intermediate files are also handled. > > > > > > It's unnecessary for dpkg, because we use hardlinks to the *.deb > > > file, so no extra disk space is consumed for the sstate > > > preparation. > > > > Yes, I saw it, just implemented in the similar way. It may be > > useful if files are copied to another (slow) disk after build > > finishes - I'm afraid hardlinks won't work in this case. > > In this case we're only linking inside WORKDIR (not to the actual > sstate cache). But with the above approach all those intermediate > files/links will be handled in a better way. > > Adriaan > > > > > > > Adriaan > > > > > > > Uladzimir Bely (1): > > > > sstate: do not keep intermediate sstate files > > > > > > > > meta/classes/dpkg-base.bbclass | 7 +++++++ > > > > meta/classes/rootfs.bbclass | 7 +++++++ > > > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 +++++++ > > > > 3 files changed, 21 insertions(+) > > > > > > > > -- > > > > 2.20.1 > > > > > > > > -- > > > > You received this message because you are subscribed to the > > > > Google Groups "isar-users" group. > > > > To unsubscribe from this group and stop receiving emails from > > > > it, send an email to isar-users+unsubscribe@googlegroups.com. > > > > To view this discussion on the web visit > > > > https://groups.google.com/d/msgid/isar-users/20220224073629.23717-1-> > > > > > > > ubely%40ilbers.de. > > > > > > -- > > Uladzimir Bely > > > > > ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-02-24 15:57 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-02-24 7:36 [PATCH v2 0/1] sstate: do not keep intermediate sstate files Uladzimir Bely 2022-02-24 7:36 ` [PATCH v2 1/1] " Uladzimir Bely 2022-02-24 7:56 ` [PATCH v2 0/1] " Schmidt, Adriaan 2022-02-24 8:19 ` Uladzimir Bely 2022-02-24 9:04 ` Schmidt, Adriaan 2022-02-24 15:57 ` Henning Schild
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox