Re: [PATCH v2 0/1] sstate: do not keep intermediate sstate files

[Uladzimir Bely <ubely@ilbers.de>]

In the email from Thursday, 24 February 2022 10:56:48 +03 user Schmidt, 
Adriaan wrote:
> Uladzimir Bely, 24. Februar 2022 08:36:
> > Sstate is quite demanding for disk space. And it often causes failures
> > in CI, related to 'out-of-space'.
> 
> If those few GB cause problems, then you really need a larger disk :)
> But I get the point, and I agree that we should try to save space.
> 

These few GBs result to ~100GiB per CI job (when doing full builds) :)

> > This patchset minimizes space usage by intermediate state files by
> > removing them:
> > - after sstate-cache populated: with separate task in task queue
> 
> I would like to avoid adding extra tasks. There seems to be a hook
> to insert functions (SSTATEPOSTCREATEFUNCS), but as sstate is not
> really documented, I have not yet figured out how to make our use case work.

Yes, it would be better if there are some post-sstate functions we could use 
instead of inserting one more task after one we are handling...

> > - after sstate-cache unpacked: by direct call of this task
> > 
> > Some results shown below for three different cases:
> > 
> > - sstate disabled;
> > - sstate enabled;
> > - sstate enabled + cleanup patch.
> > 
> > bullseye-arm64 cross build with full IMAGE_INSTALL list:
> > 
> > $ sudo du -sh build-*
> > 5,7G    build-nosstate
> > 12G     build-sstate
> > 7,6G    build-sstate-cleanup
> > 
> > bullseye-arm64 cross build with empty IMAGE_INSTALL list:
> > 
> > $ sudo du -sh build-*
> > 2,5G    build-nosstate
> > 5,3G    build-sstate
> > 3,4G    build-sstate-cleanup
> > 
> > bullseye-amd64 native build with empty IMAGE_INSTALL list:
> > 
> > $ sudo du -sh build-*
> > 3,2G    build-nosstate
> > 6,4G    build-sstate
> > 4,2G    build-sstate-cleanup
> > 
> > 
> > Changes since v1:
> > - patchset renamed from "sstate: compress rootfs sstate files";
> > - sstate intermediate files compression replaced by removing;
> > - dpkg intermediate files are also handled.
> 
> It's unnecessary for dpkg, because we use hardlinks to the *.deb file,
> so no extra disk space is consumed for the sstate preparation.

Yes, I saw it, just implemented in the similar way. It may be useful if files 
are copied to another (slow) disk after build finishes - I'm afraid hardlinks 
won't work in this case.

> 
> Adriaan
> 
> > Uladzimir Bely (1):
> >   sstate: do not keep intermediate sstate files
> >  
> >  meta/classes/dpkg-base.bbclass                      | 7 +++++++
> >  meta/classes/rootfs.bbclass                         | 7 +++++++
> >  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 +++++++
> >  3 files changed, 21 insertions(+)
> > 
> > --
> > 2.20.1
> > 
> > --
> > You received this message because you are subscribed to the Google Groups
> > "isar-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to isar-users+unsubscribe@googlegroups.com.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/isar-users/20220224073629.23717-1-> > ubely%40ilbers.de.


-- 
Uladzimir Bely