Re: [PATCH v3 0/5] Improving apt cache

["Moessbauer, Felix" <felix.moessbauer@siemens.com>]

On Fri, 2023-01-20 at 08:31 +0100, Uladzimir Bely wrote:
> Currently, apt cache (e.g. `var/cache/apt/archives`) import and
> export
> functions are not optimal. Multiple files are copied from global
> DL_DIR to package WORKDIR, increasing disk IO and space needed.
> 
> Also, various chroots (bootstrap, buildchroot, sbuild chroot) include
> their apt caches to sstate cache files.
> 
> This patchset switches to hardlinks instead of copies and removes apt
> cache from bootstrapped images ans sstate caches.

I just saw that this pattern does NOT work in case the cache is on a
different filesystem. This unfortunately is the case for all CI systems
with locally mounted caches, as well as for kas-container builds with
DL_DIR outside the KAS_WORK_DIR.

In short: This breaks a LOT of use-cases.
I'm really sorry that we only found that by know.
But we definitely need a different solution.

Example output:
2023-01-30 08:38:51 - INFO     - | ln: failed to create hard link
'/builds/iiot-edge-device/foo/meta-iot2050-pg2-
foo/build/tmp/work/debian-bullseye-amd64/isar-bootstrap-target/1.0-
r0/rootfs/var/cache/apt/archives/gcc-arm-linux-gnueabihf_4%3a10.2.1-
1_arm64.deb' => '/local-cache/meta-iot2050-pg2-
foo/downloads/deb/debian-bullseye/gcc-arm-linux-gnueabihf_4%3a10.2.1-
1_arm64.deb': Invalid cross-device link

Best regards,
Felix

> 
> Changes since v2:
>  - Don't use CACHEDIR.TAG, simply exclude var/cache/apt directory
> when creating rootfs tarball for sstate-cache.
>  - Use symlinks when exchanging deb files between WORKDIR/rootfs
> and /var/cache/apt/archives inside sbuild chroot
> 
> Changes since v1:
>  - Simplified cleanup of apt cache in debootstrap rootfs.
>  - Now "ln" instead of "cp -l" used.
>  - Removed apt cache contents from sstate cache. The idea is proposed
> in patch 3, but it was reworked and fixed. Firstly, CACHEDIR.TAG
> can't
> be just a file (e.g. created by 'touch'), it should include some
> specific signature [1]. Secondly, it's easier to just create this tag
> in bootstrapped rootfs and it will be automatically used in all
> derivatives (sbuild-chroot/buildchroot/image). So, the original patch
> from Roberto A. Foglietta was simplified.
> 
> This patchset includes (or absorbs) the logic from p1..p3 patches of
> the series Roberto prosposed. What concerns additional patches, they
> don't let us benefit much, but require quite significant changes
> in Isar, so we should check twice if they are worth including.
> 
> Uladzimir Bely (5):
>   Clean apt cache from debootstrapped rootfs dirs
>   Use hardlinks in deb-dl-dir import/export
>   Exclude apt cache from sstate caches
>   Use symlinks when importing debian packages to sbuild chroot
>   Lightweight copy of rootfs directories if possible
> 
>  meta/classes/deb-dl-dir.bbclass                     | 4 ++--
>  meta/classes/dpkg.bbclass                           | 4 ++--
>  meta/classes/imagetypes_container.bbclass           | 2 +-
>  meta/classes/rootfs.bbclass                         | 7 ++++---
>  meta/classes/sdk.bbclass                            | 2 +-
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 6 +++++-
>  6 files changed, 15 insertions(+), 10 deletions(-)
> 
> -- 
> 2.20.1
>