Re: [PATCH v2 0/3] Improving apt cache

[Uladzimir Bely <ubely@ilbers.de>]

In the email from Monday, 9 January 2023 09:32:29 +03 user Moessbauer, Felix 
wrote:
> On Fri, 2023-01-06 at 07:48 +0100, Uladzimir Bely wrote:
> 
> > Currently, apt cache (e.g. `var/cache/apt/archives`) import and
> > export
> > functions are not optimal. Multiple files are imported from global
> > DL_DIR to package WORKDIR, increasing disk IO and size needed.
> > 
> > Also, various chroots (bootstrap, buildchroot, sbuild chroot) include
> > their apt caches to sstate cache files.
> > 
> > This patchset switches to hardlinks instead of copies and removes apt
> > cache from bootstrapped images ans sstate caches.
> > 
> > Currently (measured on qemuarm64-bullseye cross-compilation with
> > maximum 8 parallel tasks (e.g. 8 CPU cores)):
> > - build directory size reduced from 8906 to 6675 MiB
> > - runtime maximum disk usage reduced from 15965 to 8501 MiB
> > 
> > TODO:
> > - cleanup other rootfs's (sbuild-chroot, buildchroot).
> >   Actually, this won't bring much benefit, but why not cleanup final
> >   rootfs's apt cache when the build finishes?
> > - deal with additional copying in sbuild routines (patch 3).
> >   We could use hardlinks instead of copying packages between upper
> >   layer (where sbuild temporarly keeps them) and workdir rootfs,
> >   but it is not expected to bring much benefit, since upper layer
> >   temporary nature. Additinal measurements are required.
> 
> 
> Hi,
> 
> if the cache is 20GB, we still copy 20GB of data into the upper layer
> (per sbuild task).
> This has a huge impact and I would really appreciate if we could use
> hardlinks for that part.
> 
> Felix
> 

Hi.

I need to check if it really works. My doubts are related to overlayfs: if we 
are really able to create hardlinks between DL_DIR (that is common linux 
filesystem) and upper layer of overlayfs (that is technically the same 
partition, but it is a "mount option" of the different "overlay" filesystem).

> 
> > 
> > Changes since v1:
> >  - Simplified cleanup of apt cache in debootstrap rootfs.
> >  - Now "ln" instead of "cp -l" used.
> >  - Removed apt cache contents from sstate cache. The idea is proposed
> > in patch 3, but it was reworked and fixed. Firstly, CACHEDIR.TAG
> > can't
> > be just a file (e.g. created by 'touch'), it should include some
> > specific signature [1]. Secondly, it's easier to just create this tag
> > in bootstrapped rootfs and it will be automatically used in all
> > derivatives (sbuild-chroot/buildchroot/image). So, the original patch
> > from Roberto A. Foglietta was simplified.
> > 
> > This patchset includes (or absorbs) the logic from p1..p3 patches of
> > the series Roberto prosposed. What concerns additional patches, they
> > don't let us benefit much, but require quite significant changes
> > in Isar, so we should check twice if they are worth including.
> > 
> > [1] https://bford.info/cachedir/
> > - 
> > 
> > Roberto A. Foglietta (1):
> >   Changes for a faster build using less disk space
> > 
> > Uladzimir Bely (2):
> >   Clean apt cache from debootstrapped rootfs dirs
> >   Use hardlinks in deb-dl-dir functions
> > 
> >  meta/classes/deb-dl-dir.bbclass                     |  6 +++---
> >  meta/classes/rootfs.bbclass                         |  3 ++-
> >  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 10 +++++++++-
> >  3 files changed, 14 insertions(+), 5 deletions(-)
> > 
> > -- 
> > 2.20.1
> > 
> 
>