Re: [PATCH v2 0/3] Improving apt cache

[Uladzimir Bely <ubely@ilbers.de>]

In the email from Monday, 9 January 2023 10:39:00 +03 user Uladzimir Bely 
wrote:
> In the email from Monday, 9 January 2023 09:32:29 +03 user Moessbauer, Felix
> wrote:
> > On Fri, 2023-01-06 at 07:48 +0100, Uladzimir Bely wrote:
> > > Currently, apt cache (e.g. `var/cache/apt/archives`) import and
> > > export
> > > functions are not optimal. Multiple files are imported from global
> > > DL_DIR to package WORKDIR, increasing disk IO and size needed.
> > > 
> > > Also, various chroots (bootstrap, buildchroot, sbuild chroot) include
> > > their apt caches to sstate cache files.
> > > 
> > > This patchset switches to hardlinks instead of copies and removes apt
> > > cache from bootstrapped images ans sstate caches.
> > > 
> > > Currently (measured on qemuarm64-bullseye cross-compilation with
> > > maximum 8 parallel tasks (e.g. 8 CPU cores)):
> > > - build directory size reduced from 8906 to 6675 MiB
> > > - runtime maximum disk usage reduced from 15965 to 8501 MiB
> > > 
> > > TODO:
> > > - cleanup other rootfs's (sbuild-chroot, buildchroot).
> > > 
> > >   Actually, this won't bring much benefit, but why not cleanup final
> > >   rootfs's apt cache when the build finishes?
> > > 
> > > - deal with additional copying in sbuild routines (patch 3).
> > > 
> > >   We could use hardlinks instead of copying packages between upper
> > >   layer (where sbuild temporarly keeps them) and workdir rootfs,
> > >   but it is not expected to bring much benefit, since upper layer
> > >   temporary nature. Additinal measurements are required.
> > 
> > Hi,
> > 
> > if the cache is 20GB, we still copy 20GB of data into the upper layer
> > (per sbuild task).
> > This has a huge impact and I would really appreciate if we could use
> > hardlinks for that part.
> > 
> > Felix
> 
> Hi.
> 
> I need to check if it really works. My doubts are related to overlayfs: if
> we are really able to create hardlinks between DL_DIR (that is common linux
> filesystem) and upper layer of overlayfs (that is technically the same
> partition, but it is a "mount option" of the different "overlay"
> filesystem).

I got time to get back to this patches and checked this moment. And it really 
does not work as I expected.

Original patch includes the following:

-        --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/
*.deb -t ${deb_dir}/ || :" \
+        --chroot-setup-commands="ln -Pf ${ext_deb_dir}/*.deb -t ${deb_dir}/ 
2>/dev/null || :" \

This results to to failing "ReproTest" in CI with the following error (for 
libhello, for example):

sbuild-build-depends-dose3-dummy:armhf : Depends: dose-distcheck:amd64 but it 
is not installable
E: Unable to correct problems, you have held broken packages.

To debug it, I removed `2>/dev/null` and found, that hardlinks simply don't 
work and the following errors are now seen earlier:

ln: failed to create hard link '/var/cache/apt/archives/adduser_3.118_all.deb' 
=> '/home/builder/libhello/rootfs/var/cache/apt/archives/
adduser_3.118_all.deb': Invalid cross-device link
... #tons of similar errors...
ln: failed to create hard link '/var/cache/apt/archives/
zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.deb' => '/home/builder/libhello/
rootfs/var/cache/apt/archives/zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.deb': 
Invalid cross-device link

I: Finished running 'ln -Pf /home/builder/libhello/rootfs/var/cache/apt/
archives/*.deb -t /var/cache/apt/archives/ || :'.

So, it works for network builds (when missing packages always can be 
downloaded by apt), but it fails for local builds from apt-cache (when, at 
first network build, sbuild dependencies are simply not exported to download 
directory due to non-working hardlinks, plus with hidden stderr)

> > > Changes since v1:
> > >  - Simplified cleanup of apt cache in debootstrap rootfs.
> > >  - Now "ln" instead of "cp -l" used.
> > >  - Removed apt cache contents from sstate cache. The idea is proposed
> > > 
> > > in patch 3, but it was reworked and fixed. Firstly, CACHEDIR.TAG
> > > can't
> > > be just a file (e.g. created by 'touch'), it should include some
> > > specific signature [1]. Secondly, it's easier to just create this tag
> > > in bootstrapped rootfs and it will be automatically used in all
> > > derivatives (sbuild-chroot/buildchroot/image). So, the original patch
> > > from Roberto A. Foglietta was simplified.
> > > 
> > > This patchset includes (or absorbs) the logic from p1..p3 patches of
> > > the series Roberto prosposed. What concerns additional patches, they
> > > don't let us benefit much, but require quite significant changes
> > > in Isar, so we should check twice if they are worth including.
> > > 
> > > [1] https://bford.info/cachedir/
> > > -
> > > 
> > > Roberto A. Foglietta (1):
> > >   Changes for a faster build using less disk space
> > > 
> > > Uladzimir Bely (2):
> > >   Clean apt cache from debootstrapped rootfs dirs
> > >   Use hardlinks in deb-dl-dir functions
> > >  
> > >  meta/classes/deb-dl-dir.bbclass                     |  6 +++---
> > >  meta/classes/rootfs.bbclass                         |  3 ++-
> > >  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 10 +++++++++-
> > >  3 files changed, 14 insertions(+), 5 deletions(-)