Re: [PATCH 1/1] sstate: compress rootfs sstate files

[Uladzimir Bely <ubely@ilbers.de>]

In the email from Thursday, 24 February 2022 00:33:06 +03 user Henning Schild 
wrote:
> Am Wed, 23 Feb 2022 18:52:54 +0300
> 
> schrieb Uladzimir Bely <ubely@ilbers.de>:
> > In the email from Wednesday, 23 February 2022 18:43:21 +03 user
> > 
> > Henning Schild wrote:
> > > This is wrong. Everything you put into sstate is already compressed
> > > there.
> > > 
> > > So what you probably want is to "move" stuff into sstate and delete
> > > setscene after unpack so temporary tarballs maybe do not hang
> > > around.
> > > 
> > > That will save much more space and time, but i am not sure it would
> > > be "allowed". That is why i added Adriaan on cc.
> > 
> > Yes, it may be wrong because this tarballs might be needed to compare
> > hashes by sstate. But why not use them already compressed and save
> > space in workdirs?
> 
> The real way to save the space is to not keep sstate stuff at all (in
> WORKDIR). When putting things into sstate we should remove it from
> WORKDIR right after it went into sstate. When getting stuff out we
> should remove staging stuff at the end of setscene.
> 
> SSTATE_BUILDDIR seems to take care of some of the bits, i did not
> manage to write the correct code for all of that. But compressing bits
> we do not actually want to keep does not seem right.
> 

I also tried to remove these files, but didn't succeed. It's easy to do for 
getting stuff from sstate cache - we can just remove .tar file at the end of 
*_setscene(). But it's a bit more tricky, when we put stuff to sstate cache - 
it's the last stage and we can't execute any custom code. Even [postfuncs] are 
executed before sstate cache is created.

Probably, additional task for cleanup sstate dir could be used. For example,
do_bootstrap => (do_bootstrap_sstate_cleanup) => do_build.

> So a clear NACK from me.
> 
> Henning
> 
> > > Henning
> > > 
> > > Am Wed, 23 Feb 2022 10:17:49 +0100
> > > 
> > > schrieb Uladzimir Bely <ubely@ilbers.de>:
> > > > This saves about 1-2 GiB per multiconfig, at the expense of
> > > > approximately 20-40 seconds for packing the files.
> > > > 
> > > > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > > > ---
> > > > 
> > > >  meta/classes/rootfs.bbclass                         | 4 ++--
> > > >  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 4 ++--
> > > >  2 files changed, 4 insertions(+), 4 deletions(-)
> > > > 
> > > > diff --git a/meta/classes/rootfs.bbclass
> > > > b/meta/classes/rootfs.bbclass index 2bdb3b6d..15e87bec 100644
> > > > --- a/meta/classes/rootfs.bbclass
> > > > +++ b/meta/classes/rootfs.bbclass
> > > > @@ -298,13 +298,13 @@ do_rootfs_install[sstate-interceptfuncs] =
> > > > "rootfs_install_sstate_prepare" # the buildchroot is owned by
> > > > root, so we need some sudoing to pack and unpack
> > > > rootfs_install_sstate_prepare() { sudo mount --bind
> > > > ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
> > > > -    sudo tar -C ${WORKDIR}/mnt -cpf ${ROOTFS_SSTATE}/rootfs.tar
> > > > --one-file-system rootfs
> > > > +    sudo tar -C ${WORKDIR}/mnt -cpzf
> > > > ${ROOTFS_SSTATE}/rootfs.tar.gz --one-file-system rootfs sudo
> > > > umount ${WORKDIR}/mnt/rootfs
> > > > 
> > > >  }
> > > >  do_rootfs_install_sstate_prepare[lockfiles] =
> > > > 
> > > > "${REPO_ISAR_DIR}/isar.lock"
> > > > 
> > > >  rootfs_install_sstate_finalize() {
> > > > 
> > > > -    sudo tar -C ${WORKDIR} -xpf ${ROOTFS_SSTATE}/rootfs.tar
> > > > +    sudo tar -C ${WORKDIR} -xpzf ${ROOTFS_SSTATE}/rootfs.tar.gz
> > > > 
> > > >  }
> > > >  
> > > >  python do_rootfs_install_setscene() {
> > > > 
> > > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > > > 2f483f5a..99d1d945 100644 ---
> > > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -402,11
> > > > +402,11 @@ do_bootstrap[sstate-plaindirs] = "${BOOTSTRAP_SSTATE}"
> > > > do_bootstrap[sstate-interceptfuncs] = "bootstrap_sstate_prepare"
> > > > 
> > > >  bootstrap_sstate_prepare() {
> > > > 
> > > > -    sudo tar -C $(dirname "${ROOTFSDIR}") -cpf
> > > > ${BOOTSTRAP_SSTATE}/bootstrap.tar --one-file-system $(basename
> > > > "${ROOTFSDIR}")
> > > > +    sudo tar -C $(dirname "${ROOTFSDIR}") -cpzf
> > > > ${BOOTSTRAP_SSTATE}/bootstrap.tar.gz --one-file-system $(basename
> > > > "${ROOTFSDIR}") }
> > > > 
> > > >  bootstrap_sstate_finalize() {
> > > > 
> > > > -    sudo tar -C $(dirname "${ROOTFSDIR}") -xpf
> > > > ${BOOTSTRAP_SSTATE}/bootstrap.tar
> > > > +    sudo tar -C $(dirname "${ROOTFSDIR}") -xpzf
> > > > ${BOOTSTRAP_SSTATE}/bootstrap.tar.gz sudo ln -Tfsr "${ROOTFSDIR}"
> > > > "${DEPLOY_ISAR_BOOTSTRAP}" }


-- 
Uladzimir Bely