Re: [PATCH v2 0/3] Improving apt cache

["Moessbauer, Felix" <felix.moessbauer@siemens.com>]

On Fri, 2023-01-06 at 07:48 +0100, Uladzimir Bely wrote:
> Currently, apt cache (e.g. `var/cache/apt/archives`) import and
> export
> functions are not optimal. Multiple files are imported from global
> DL_DIR to package WORKDIR, increasing disk IO and size needed.
> 
> Also, various chroots (bootstrap, buildchroot, sbuild chroot) include
> their apt caches to sstate cache files.
> 
> This patchset switches to hardlinks instead of copies and removes apt
> cache from bootstrapped images ans sstate caches.
> 
> Currently (measured on qemuarm64-bullseye cross-compilation with
> maximum 8 parallel tasks (e.g. 8 CPU cores)):
> - build directory size reduced from 8906 to 6675 MiB
> - runtime maximum disk usage reduced from 15965 to 8501 MiB
> 
> TODO:
> - cleanup other rootfs's (sbuild-chroot, buildchroot).
>   Actually, this won't bring much benefit, but why not cleanup final
>   rootfs's apt cache when the build finishes?
> - deal with additional copying in sbuild routines (patch 3).
>   We could use hardlinks instead of copying packages between upper
>   layer (where sbuild temporarly keeps them) and workdir rootfs,
>   but it is not expected to bring much benefit, since upper layer
>   temporary nature. Additinal measurements are required.

Hi,

if the cache is 20GB, we still copy 20GB of data into the upper layer
(per sbuild task).
This has a huge impact and I would really appreciate if we could use
hardlinks for that part.

Felix

> 
> Changes since v1:
>  - Simplified cleanup of apt cache in debootstrap rootfs.
>  - Now "ln" instead of "cp -l" used.
>  - Removed apt cache contents from sstate cache. The idea is proposed
> in patch 3, but it was reworked and fixed. Firstly, CACHEDIR.TAG
> can't
> be just a file (e.g. created by 'touch'), it should include some
> specific signature [1]. Secondly, it's easier to just create this tag
> in bootstrapped rootfs and it will be automatically used in all
> derivatives (sbuild-chroot/buildchroot/image). So, the original patch
> from Roberto A. Foglietta was simplified.
> 
> This patchset includes (or absorbs) the logic from p1..p3 patches of
> the series Roberto prosposed. What concerns additional patches, they
> don't let us benefit much, but require quite significant changes
> in Isar, so we should check twice if they are worth including.
> 
> [1] https://bford.info/cachedir/
> - 
> 
> Roberto A. Foglietta (1):
>   Changes for a faster build using less disk space
> 
> Uladzimir Bely (2):
>   Clean apt cache from debootstrapped rootfs dirs
>   Use hardlinks in deb-dl-dir functions
> 
>  meta/classes/deb-dl-dir.bbclass                     |  6 +++---
>  meta/classes/rootfs.bbclass                         |  3 ++-
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 10 +++++++++-
>  3 files changed, 14 insertions(+), 5 deletions(-)
> 
> -- 
> 2.20.1
>