[PATCH v3 0/5] Improving apt cache

[PATCH v3 0/5] Improving apt cache

[Uladzimir Bely]

Currently, apt cache (e.g. `var/cache/apt/archives`) import and export
functions are not optimal. Multiple files are copied from global
DL_DIR to package WORKDIR, increasing disk IO and space needed.

Also, various chroots (bootstrap, buildchroot, sbuild chroot) include
their apt caches to sstate cache files.

This patchset switches to hardlinks instead of copies and removes apt
cache from bootstrapped images ans sstate caches.

Changes since v2:
 - Don't use CACHEDIR.TAG, simply exclude var/cache/apt directory
when creating rootfs tarball for sstate-cache.
 - Use symlinks when exchanging deb files between WORKDIR/rootfs
and /var/cache/apt/archives inside sbuild chroot

Changes since v1:
 - Simplified cleanup of apt cache in debootstrap rootfs.
 - Now "ln" instead of "cp -l" used.
 - Removed apt cache contents from sstate cache. The idea is proposed
in patch 3, but it was reworked and fixed. Firstly, CACHEDIR.TAG can't
be just a file (e.g. created by 'touch'), it should include some
specific signature [1]. Secondly, it's easier to just create this tag
in bootstrapped rootfs and it will be automatically used in all
derivatives (sbuild-chroot/buildchroot/image). So, the original patch
from Roberto A. Foglietta was simplified.

This patchset includes (or absorbs) the logic from p1..p3 patches of
the series Roberto prosposed. What concerns additional patches, they
don't let us benefit much, but require quite significant changes
in Isar, so we should check twice if they are worth including.

Uladzimir Bely (5):
  Clean apt cache from debootstrapped rootfs dirs
  Use hardlinks in deb-dl-dir import/export
  Exclude apt cache from sstate caches
  Use symlinks when importing debian packages to sbuild chroot
  Lightweight copy of rootfs directories if possible

 meta/classes/deb-dl-dir.bbclass                     | 4 ++--
 meta/classes/dpkg.bbclass                           | 4 ++--
 meta/classes/imagetypes_container.bbclass           | 2 +-
 meta/classes/rootfs.bbclass                         | 7 ++++---
 meta/classes/sdk.bbclass                            | 2 +-
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 6 +++++-
 6 files changed, 15 insertions(+), 10 deletions(-)

-- 
2.20.1

[PATCH v3 1/5] Clean apt cache from debootstrapped rootfs dirs

[Uladzimir Bely]

This cleans `/var/cache/apt/archives` contents from debootstrapped
rootfs directories after exporting their contents to download
directory.

This reduces disk usage by both files and sstate caches.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index faba73fe..cb0079ec 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -392,6 +392,9 @@ do_bootstrap() {
         ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
 EOSUDO
     deb_dl_dir_export "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+
+    # Cleanup apt cache
+    sudo -Es chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean
 }
 
 addtask bootstrap before do_build after do_generate_keyrings
-- 
2.20.1

[PATCH v3 2/5] Use hardlinks in deb-dl-dir import/export

[Uladzimir Bely]

This saves disk space and reduces IO

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/deb-dl-dir.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 3b1517dc..7db25251 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -86,7 +86,7 @@ deb_dl_dir_import() {
         printenv | grep -q BB_VERBOSE_LOGS && set -x
 
         sudo find "${pc}" -type f -iname "*\.deb" -exec \
-            cp -n --no-preserve=owner -t "${rootfs}"/var/cache/apt/archives/ {} +
+            ln -Pf -t "${rootfs}"/var/cache/apt/archives/ {} +
     '
 }
 
@@ -109,7 +109,7 @@ deb_dl_dir_export() {
             if [ -n "$package" ]; then
                 cmp --silent "$package" "$p" && continue
             fi
-            sudo cp -n "${p}" "${pc}"
+            sudo ln -Pf "${p}" "${pc}"
         done
         sudo chown -R $(id -u):$(id -g) "${pc}"
     '
-- 
2.20.1

[PATCH v3 3/5] Exclude apt cache from sstate caches

[Uladzimir Bely]

By using `--exclude` option for tar we avoid placing apt cache
into sstate cache files.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/rootfs.bbclass                         | 3 ++-
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 786682d9..517185e8 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -308,7 +308,8 @@ rootfs_install_sstate_prepare() {
     # so we use some mount magic to prevent that
     mkdir -p ${WORKDIR}/mnt/rootfs
     sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
-    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system rootfs
+    lopts="--one-file-system --exclude=var/cache/apt/archives"
+    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar $lopts rootfs
     sudo umount ${WORKDIR}/mnt/rootfs
     sudo chown $(id -u):$(id -g) rootfs.tar
 }
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index cb0079ec..431ef2d3 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -405,7 +405,8 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
 
 bootstrap_sstate_prepare() {
     # this runs in SSTATE_BUILDDIR, which will be deleted automatically
-    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar --one-file-system $(basename "${ROOTFSDIR}")
+    lopts="--one-file-system --exclude=var/cache/apt/archives"
+    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts $(basename "${ROOTFSDIR}")
     sudo chown $(id -u):$(id -g) bootstrap.tar
 }
 
-- 
2.20.1

[PATCH v3 4/5] Use symlinks when importing debian packages to sbuild chroot

[Uladzimir Bely]

When building packages, sbuild downloads missing dependencies from
debian mirros. In order to reduce network consuming and have in DL_DIR
every package downloaded by sbuild, we additionally import/export
local debian packages to/from apt cache in schroot upper layer.

Since hardlinks between package WORKDIR and apt cache in schroot
don't work (different filesystems), will use symlinks when importing
debs to schroot (--chroot-setup-commands) and copy with dereference
when exporting them back (--finished-build-commands).

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 7822b14d..c1a8c77a 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -108,9 +108,9 @@ dpkg_runbuild() {
         --chroot-setup-commands="echo \"Package: *\nPin: release n=${DEBDISTRONAME}\nPin-Priority: 1000\" > /etc/apt/preferences.d/isar-apt" \
         --chroot-setup-commands="echo \"APT::Get::allow-downgrades 1;\" > /etc/apt/apt.conf.d/50isar-apt" \
         --chroot-setup-commands="rm -f /var/log/dpkg.log" \
-        --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
+        --chroot-setup-commands="ln -sf ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
         --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
-        --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
+        --finished-build-commands="cp -Ln --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
         --finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \
         --debbuildopts="--source-option=-I" \
         --build-dir=${WORKDIR} --dist="isar" ${DSC_FILE}
-- 
2.20.1

[PATCH v3 5/5] Lightweight copy of rootfs directories if possible

[Uladzimir Bely]

Some filesystem (such as BTFRS or XFS) support copy-on-write feature.
In this case, data blocks are not really copied until modified.

Using `--reflink=auto` enables this feature for `cp` and allows
to get a 2x..3x speed when copying files.

Some measurements on HDD and XFS (~900MiB rootfs from isar build):

```
real    0m31,828s
user    0m0,369s
sys     0m3,403s

real    0m13,611s
user    0m0,156s
sys     0m1,267s
```

The solution is taken from Roberto A. Foglietta patches and released
in a separate patch here.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/imagetypes_container.bbclass | 2 +-
 meta/classes/rootfs.bbclass               | 4 ++--
 meta/classes/sdk.bbclass                  | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/classes/imagetypes_container.bbclass b/meta/classes/imagetypes_container.bbclass
index 436a0051..50645aea 100644
--- a/meta/classes/imagetypes_container.bbclass
+++ b/meta/classes/imagetypes_container.bbclass
@@ -40,7 +40,7 @@ do_containerize() {
         "${oci_img_dir}_unpacked"
 
     # add root filesystem as the flesh of the skeleton
-    sudo cp -a "${rootfs}"/* "${oci_img_dir}_unpacked/rootfs/"
+    sudo cp --reflink=auto -a "${rootfs}"/* "${oci_img_dir}_unpacked/rootfs/"
     # clean-up temporary files
     sudo find "${oci_img_dir}_unpacked/rootfs/tmp" -mindepth 1 -delete
 
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 517185e8..19ce998f 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -78,7 +78,7 @@ BOOTSTRAP_SRC_${ROOTFS_ARCH} = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTF
 
 rootfs_prepare[weight] = "25"
 rootfs_prepare(){
-    sudo cp -Trpfx '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}'
+    sudo cp -Trpfx --reflink=auto '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}'
 }
 
 ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt"
@@ -218,7 +218,7 @@ cache_deb_src() {
     # Note: ISAR updates the apt state information(apt-get update) only once during bootstrap and
     # relies on that through out the build. Copy that state information instead of apt-get update
     # which generates a new state from upstream.
-    sudo cp -Trpn "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/"
+    sudo cp -Trpn --reflink=auto "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/"
 
     deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
     debsrc_download ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
diff --git a/meta/classes/sdk.bbclass b/meta/classes/sdk.bbclass
index 79e01a19..3feada21 100644
--- a/meta/classes/sdk.bbclass
+++ b/meta/classes/sdk.bbclass
@@ -82,7 +82,7 @@ do_rootfs_install[vardeps] += "${SDKROOTFSVARDEPS}"
 ROOTFS_CONFIGURE_COMMAND_append_class-sdk = " ${@'rootfs_configure_isar_apt_dir' if d.getVar('SDK_INCLUDE_ISAR_APT') == '1' else ''}"
 rootfs_configure_isar_apt_dir() {
     # Copy isar-apt instead of mounting:
-    sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${ROOTFSDIR}/isar-apt
+    sudo cp -Trpfx --reflink=auto ${REPO_ISAR_DIR}/${DISTRO} ${ROOTFSDIR}/isar-apt
 }
 
 ROOTFS_POSTPROCESS_COMMAND_prepend_class-sdk = "sdkchroot_configscript "
-- 
2.20.1

Re: [PATCH v3 0/5] Improving apt cache

[Moessbauer, Felix]

On Fri, 2023-01-20 at 08:31 +0100, Uladzimir Bely wrote:
> Currently, apt cache (e.g. `var/cache/apt/archives`) import and
> export
> functions are not optimal. Multiple files are copied from global
> DL_DIR to package WORKDIR, increasing disk IO and space needed.
> 
> Also, various chroots (bootstrap, buildchroot, sbuild chroot) include
> their apt caches to sstate cache files.
> 
> This patchset switches to hardlinks instead of copies and removes apt
> cache from bootstrapped images ans sstate caches.

I tested this series (v3) in two layers: amd64 (internal) and arm64
(meta-iot2050) and can confirm that it works. Also we get a significant
performance improvement and massively reduced disk consumption.

Thanks for implementing this!

Acked-by: Felix Moessbauer <felix.moessbauer@siemens.com>

Felix

> 
> Changes since v2:
>  - Don't use CACHEDIR.TAG, simply exclude var/cache/apt directory
> when creating rootfs tarball for sstate-cache.
>  - Use symlinks when exchanging deb files between WORKDIR/rootfs
> and /var/cache/apt/archives inside sbuild chroot
> 
> Changes since v1:
>  - Simplified cleanup of apt cache in debootstrap rootfs.
>  - Now "ln" instead of "cp -l" used.
>  - Removed apt cache contents from sstate cache. The idea is proposed
> in patch 3, but it was reworked and fixed. Firstly, CACHEDIR.TAG
> can't
> be just a file (e.g. created by 'touch'), it should include some
> specific signature [1]. Secondly, it's easier to just create this tag
> in bootstrapped rootfs and it will be automatically used in all
> derivatives (sbuild-chroot/buildchroot/image). So, the original patch
> from Roberto A. Foglietta was simplified.
> 
> This patchset includes (or absorbs) the logic from p1..p3 patches of
> the series Roberto prosposed. What concerns additional patches, they
> don't let us benefit much, but require quite significant changes
> in Isar, so we should check twice if they are worth including.
> 
> Uladzimir Bely (5):
>   Clean apt cache from debootstrapped rootfs dirs
>   Use hardlinks in deb-dl-dir import/export
>   Exclude apt cache from sstate caches
>   Use symlinks when importing debian packages to sbuild chroot
>   Lightweight copy of rootfs directories if possible
> 
>  meta/classes/deb-dl-dir.bbclass                     | 4 ++--
>  meta/classes/dpkg.bbclass                           | 4 ++--
>  meta/classes/imagetypes_container.bbclass           | 2 +-
>  meta/classes/rootfs.bbclass                         | 7 ++++---
>  meta/classes/sdk.bbclass                            | 2 +-
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 6 +++++-
>  6 files changed, 15 insertions(+), 10 deletions(-)
> 
> -- 
> 2.20.1
> 

Re: [PATCH v3 0/5] Improving apt cache

[Roberto A. Foglietta]

[-- Attachment #1: Type: text/plain, Size: 1620 bytes --]

On Fri, 20 Jan 2023 at 14:04, Moessbauer, Felix <
felix.moessbauer@siemens.com> wrote:

> On Fri, 2023-01-20 at 08:31 +0100, Uladzimir Bely wrote:
> > Currently, apt cache (e.g. `var/cache/apt/archives`) import and
> > export
> > functions are not optimal. Multiple files are copied from global
> > DL_DIR to package WORKDIR, increasing disk IO and space needed.
> >
> > Also, various chroots (bootstrap, buildchroot, sbuild chroot) include
> > their apt caches to sstate cache files.
> >
> > This patchset switches to hardlinks instead of copies and removes apt
> > cache from bootstrapped images ans sstate caches.
>
> I tested this series (v3) in two layers: amd64 (internal) and arm64
> (meta-iot2050) and can confirm that it works. Also we get a significant
> performance improvement and massively reduced disk consumption.
>
>
The candidate describes the following situation in plain straight words:

Q: input: ln -sf, output: cp -Ln
A: much ado for nothing (cit.)

Q: How does the candidate execute the tests on the system?
A: I don't but my gut feelings

Q: What alternative does the candidate propose?
A: an elegant solution because quality is relative but elegance is absolute
(cit.)

Q: summarise it in a nerd language
A:

mkdir tmp && ( cd tmp; mkdir a b
echo pippo > a/file.txt
ln -sf $PWD/a/file.txt b
cat a/file.txt b/file.txt
# pippo
# pippo
echo pluto >b/file.txt
cat a/file.txt b/file.txt
# pluto
# pluto
rm -f b/file.txt
echo ciao >b/file.txt
cat a/file.txt b/file.txt
# pluto
# ciao
cp -Ln b/file.txt a/file
cat a/file.txt b/file.txt
# pluto
# ciao
cd ..; rm -rf tmp ) # to repeat in loop

Enjoy, R-

[-- Attachment #2: Type: text/html, Size: 2379 bytes --]

Re: [PATCH v3 0/5] Improving apt cache

[Uladzimir Bely]

In mail from пятница, 20 января 2023 г. 10:31:01 +03 user Uladzimir Bely 
wrote:
> Currently, apt cache (e.g. `var/cache/apt/archives`) import and export
> functions are not optimal. Multiple files are copied from global
> DL_DIR to package WORKDIR, increasing disk IO and space needed.
> 
> Also, various chroots (bootstrap, buildchroot, sbuild chroot) include
> their apt caches to sstate cache files.
> 
> This patchset switches to hardlinks instead of copies and removes apt
> cache from bootstrapped images ans sstate caches.
> 
> Changes since v2:
>  - Don't use CACHEDIR.TAG, simply exclude var/cache/apt directory
> when creating rootfs tarball for sstate-cache.
>  - Use symlinks when exchanging deb files between WORKDIR/rootfs
> and /var/cache/apt/archives inside sbuild chroot
> 
> Changes since v1:
>  - Simplified cleanup of apt cache in debootstrap rootfs.
>  - Now "ln" instead of "cp -l" used.
>  - Removed apt cache contents from sstate cache. The idea is proposed
> in patch 3, but it was reworked and fixed. Firstly, CACHEDIR.TAG can't
> be just a file (e.g. created by 'touch'), it should include some
> specific signature [1]. Secondly, it's easier to just create this tag
> in bootstrapped rootfs and it will be automatically used in all
> derivatives (sbuild-chroot/buildchroot/image). So, the original patch
> from Roberto A. Foglietta was simplified.
> 
> This patchset includes (or absorbs) the logic from p1..p3 patches of
> the series Roberto prosposed. What concerns additional patches, they
> don't let us benefit much, but require quite significant changes
> in Isar, so we should check twice if they are worth including.
> 
> Uladzimir Bely (5):
>   Clean apt cache from debootstrapped rootfs dirs
>   Use hardlinks in deb-dl-dir import/export
>   Exclude apt cache from sstate caches
>   Use symlinks when importing debian packages to sbuild chroot
>   Lightweight copy of rootfs directories if possible
> 
>  meta/classes/deb-dl-dir.bbclass                     | 4 ++--
>  meta/classes/dpkg.bbclass                           | 4 ++--
>  meta/classes/imagetypes_container.bbclass           | 2 +-
>  meta/classes/rootfs.bbclass                         | 7 ++++---
>  meta/classes/sdk.bbclass                            | 2 +-
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 6 +++++-
>  6 files changed, 15 insertions(+), 10 deletions(-)

Applied to next.

Re: [PATCH v3 0/5] Improving apt cache

[Moessbauer, Felix]

On Fri, 2023-01-20 at 08:31 +0100, Uladzimir Bely wrote:
> Currently, apt cache (e.g. `var/cache/apt/archives`) import and
> export
> functions are not optimal. Multiple files are copied from global
> DL_DIR to package WORKDIR, increasing disk IO and space needed.
> 
> Also, various chroots (bootstrap, buildchroot, sbuild chroot) include
> their apt caches to sstate cache files.
> 
> This patchset switches to hardlinks instead of copies and removes apt
> cache from bootstrapped images ans sstate caches.

I just saw that this pattern does NOT work in case the cache is on a
different filesystem. This unfortunately is the case for all CI systems
with locally mounted caches, as well as for kas-container builds with
DL_DIR outside the KAS_WORK_DIR.

In short: This breaks a LOT of use-cases.
I'm really sorry that we only found that by know.
But we definitely need a different solution.

Example output:
2023-01-30 08:38:51 - INFO     - | ln: failed to create hard link
'/builds/iiot-edge-device/foo/meta-iot2050-pg2-
foo/build/tmp/work/debian-bullseye-amd64/isar-bootstrap-target/1.0-
r0/rootfs/var/cache/apt/archives/gcc-arm-linux-gnueabihf_4%3a10.2.1-
1_arm64.deb' => '/local-cache/meta-iot2050-pg2-
foo/downloads/deb/debian-bullseye/gcc-arm-linux-gnueabihf_4%3a10.2.1-
1_arm64.deb': Invalid cross-device link

Best regards,
Felix

> 
> Changes since v2:
>  - Don't use CACHEDIR.TAG, simply exclude var/cache/apt directory
> when creating rootfs tarball for sstate-cache.
>  - Use symlinks when exchanging deb files between WORKDIR/rootfs
> and /var/cache/apt/archives inside sbuild chroot
> 
> Changes since v1:
>  - Simplified cleanup of apt cache in debootstrap rootfs.
>  - Now "ln" instead of "cp -l" used.
>  - Removed apt cache contents from sstate cache. The idea is proposed
> in patch 3, but it was reworked and fixed. Firstly, CACHEDIR.TAG
> can't
> be just a file (e.g. created by 'touch'), it should include some
> specific signature [1]. Secondly, it's easier to just create this tag
> in bootstrapped rootfs and it will be automatically used in all
> derivatives (sbuild-chroot/buildchroot/image). So, the original patch
> from Roberto A. Foglietta was simplified.
> 
> This patchset includes (or absorbs) the logic from p1..p3 patches of
> the series Roberto prosposed. What concerns additional patches, they
> don't let us benefit much, but require quite significant changes
> in Isar, so we should check twice if they are worth including.
> 
> Uladzimir Bely (5):
>   Clean apt cache from debootstrapped rootfs dirs
>   Use hardlinks in deb-dl-dir import/export
>   Exclude apt cache from sstate caches
>   Use symlinks when importing debian packages to sbuild chroot
>   Lightweight copy of rootfs directories if possible
> 
>  meta/classes/deb-dl-dir.bbclass                     | 4 ++--
>  meta/classes/dpkg.bbclass                           | 4 ++--
>  meta/classes/imagetypes_container.bbclass           | 2 +-
>  meta/classes/rootfs.bbclass                         | 7 ++++---
>  meta/classes/sdk.bbclass                            | 2 +-
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 6 +++++-
>  6 files changed, 15 insertions(+), 10 deletions(-)
> 
> -- 
> 2.20.1
> 

Re: [PATCH v3 0/5] Improving apt cache

[Roberto A. Foglietta]

On Mon, 30 Jan 2023 at 09:45, Moessbauer, Felix
<felix.moessbauer@siemens.com> wrote:
>
> On Fri, 2023-01-20 at 08:31 +0100, Uladzimir Bely wrote:
> > Currently, apt cache (e.g. `var/cache/apt/archives`) import and
> > export
> > functions are not optimal. Multiple files are copied from global
> > DL_DIR to package WORKDIR, increasing disk IO and space needed.
> >
> > Also, various chroots (bootstrap, buildchroot, sbuild chroot) include
> > their apt caches to sstate cache files.
> >
> > This patchset switches to hardlinks instead of copies and removes apt
> > cache from bootstrapped images ans sstate caches.
>
> I just saw that this pattern does NOT work in case the cache is on a
> different filesystem. This unfortunately is the case for all CI systems
> with locally mounted caches, as well as for kas-container builds with
> DL_DIR outside the KAS_WORK_DIR.

The entire cache system needs to be reworked and this was pretty
clear. However, every step (hard link included) brings us nearer to
this conclusion and moreover it helps to reduce the building time (now
11m29s on complete image) that allows us to perform more tests in less
and lesser time. I am currently on this and I have a good feeling that
I will reach a general solution soon. As you can imagine the cash
system is not an easy piece to rework due to its implications in many
different starting points it can have. So, build and rebuild are just
two basic cases but variations of the top or ISAR layers that cause a
partial rebuilding and breaks in building are many other cases.
Obviously the main idea is to reach a reasonable goal AND then break
it in several small steps that can be integrated upstream because a
single large patch would be equivalent to fork another 'next' branch
which is possible but IHMO not acceptable for a long time. After all,
also the GPLv3 on composition is not a long-term way to go. So, in the
best case it makes sense that we reach the common goal to rationalise
and speed-up the cache subsystem.

Best regards, R-

Re: [PATCH v3 0/5] Improving apt cache

[Roberto A. Foglietta]

On Mon, 30 Jan 2023 at 10:56, Roberto A. Foglietta
<roberto.foglietta@gmail.com> wrote:
>
> On Mon, 30 Jan 2023 at 09:45, Moessbauer, Felix
> <felix.moessbauer@siemens.com> wrote:
> >
> > On Fri, 2023-01-20 at 08:31 +0100, Uladzimir Bely wrote:
> > > Currently, apt cache (e.g. `var/cache/apt/archives`) import and
> > > export
> > > functions are not optimal. Multiple files are copied from global
> > > DL_DIR to package WORKDIR, increasing disk IO and space needed.
> > >
> > > Also, various chroots (bootstrap, buildchroot, sbuild chroot) include
> > > their apt caches to sstate cache files.
> > >
> > > This patchset switches to hardlinks instead of copies and removes apt
> > > cache from bootstrapped images ans sstate caches.
> >
> > I just saw that this pattern does NOT work in case the cache is on a
> > different filesystem. This unfortunately is the case for all CI systems
> > with locally mounted caches, as well as for kas-container builds with
> > DL_DIR outside the KAS_WORK_DIR.
>
> The entire cache system needs to be reworked and this was pretty
> clear. However, every step (hard link included) brings us nearer to
> this conclusion and moreover it helps to reduce the building time (now
> 11m29s on complete image) that allows us to perform more tests in less
> and lesser time. I am currently on this and I have a good feeling that
> I will reach a general solution soon. As you can imagine the cash
> system is not an easy piece to rework due to its implications in many
> different starting points it can have. So, build and rebuild are just
> two basic cases but variations of the top or ISAR layers that cause a
> partial rebuilding and breaks in building are many other cases.
> Obviously the main idea is to reach a reasonable goal AND then break
> it in several small steps that can be integrated upstream because a
> single large patch would be equivalent to fork another 'next' branch
> which is possible but IHMO not acceptable for a long time. After all,
> also the GPLv3 on composition is not a long-term way to go. So, in the
> best case it makes sense that we reach the common goal to rationalise
> and speed-up the cache subsystem.
>

apt-cache and sstate-cache are two different subsystems: yes they are
but they are not separated much enough to allow a general
straightforward solution for apt-cache to work but just in dpkg.class.

Best regards, R-

Re: [PATCH v3 0/5] Improving apt cache

[Henning Schild]

Am Mon, 30 Jan 2023 09:45:45 +0100
schrieb "Moessbauer, Felix (T CED INW-CN)"
<felix.moessbauer@siemens.com>:

> On Fri, 2023-01-20 at 08:31 +0100, Uladzimir Bely wrote:
> > Currently, apt cache (e.g. `var/cache/apt/archives`) import and
> > export
> > functions are not optimal. Multiple files are copied from global
> > DL_DIR to package WORKDIR, increasing disk IO and space needed.
> >
> > Also, various chroots (bootstrap, buildchroot, sbuild chroot)
> > include their apt caches to sstate cache files.
> >
> > This patchset switches to hardlinks instead of copies and removes
> > apt cache from bootstrapped images ans sstate caches.  
> 
> I just saw that this pattern does NOT work in case the cache is on a
> different filesystem. This unfortunately is the case for all CI
> systems with locally mounted caches, as well as for kas-container
> builds with DL_DIR outside the KAS_WORK_DIR.

I remember discussing that issue with someone because that is always
what happens when you implement hard linking. Not sure it was on the
list because i kept away from this lively discussion.

It is like when someone adds overlayfs and then finds that it can not
be nested a few weeks later ;).

One way to go is to try hardlinking and fall back to copying or
mounting or whatever the slow/expensive case is.

I once wrote patches for wic doing a "if not hardlink, copy" that were
needed for Isar and its chroots.

But if the cases we care most about would always take the "slow path"
we better not have two paths.

Henning

> In short: This breaks a LOT of use-cases.
> I'm really sorry that we only found that by know.
> But we definitely need a different solution.
> 
> Example output:
> 2023-01-30 08:38:51 - INFO     - | ln: failed to create hard link
> '/builds/iiot-edge-device/foo/meta-iot2050-pg2-
> foo/build/tmp/work/debian-bullseye-amd64/isar-bootstrap-target/1.0-
> r0/rootfs/var/cache/apt/archives/gcc-arm-linux-gnueabihf_4%3a10.2.1-
> 1_arm64.deb' => '/local-cache/meta-iot2050-pg2-
> foo/downloads/deb/debian-bullseye/gcc-arm-linux-gnueabihf_4%3a10.2.1-
> 1_arm64.deb': Invalid cross-device link
> 
> Best regards,
> Felix
> 
> >
> > Changes since v2:
> >  - Don't use CACHEDIR.TAG, simply exclude var/cache/apt directory
> > when creating rootfs tarball for sstate-cache.
> >  - Use symlinks when exchanging deb files between WORKDIR/rootfs
> > and /var/cache/apt/archives inside sbuild chroot
> >
> > Changes since v1:
> >  - Simplified cleanup of apt cache in debootstrap rootfs.
> >  - Now "ln" instead of "cp -l" used.
> >  - Removed apt cache contents from sstate cache. The idea is
> > proposed in patch 3, but it was reworked and fixed. Firstly,
> > CACHEDIR.TAG can't
> > be just a file (e.g. created by 'touch'), it should include some
> > specific signature [1]. Secondly, it's easier to just create this
> > tag in bootstrapped rootfs and it will be automatically used in all
> > derivatives (sbuild-chroot/buildchroot/image). So, the original
> > patch from Roberto A. Foglietta was simplified.
> >
> > This patchset includes (or absorbs) the logic from p1..p3 patches of
> > the series Roberto prosposed. What concerns additional patches, they
> > don't let us benefit much, but require quite significant changes
> > in Isar, so we should check twice if they are worth including.
> >
> > Uladzimir Bely (5):
> >   Clean apt cache from debootstrapped rootfs dirs
> >   Use hardlinks in deb-dl-dir import/export
> >   Exclude apt cache from sstate caches
> >   Use symlinks when importing debian packages to sbuild chroot
> >   Lightweight copy of rootfs directories if possible
> >
> >  meta/classes/deb-dl-dir.bbclass                     | 4 ++--
> >  meta/classes/dpkg.bbclass                           | 4 ++--
> >  meta/classes/imagetypes_container.bbclass           | 2 +-
> >  meta/classes/rootfs.bbclass                         | 7 ++++---
> >  meta/classes/sdk.bbclass                            | 2 +-
> >  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 6 +++++-
> >  6 files changed, 15 insertions(+), 10 deletions(-)
> >
> > --
> > 2.20.1
> >  
> 

Re: [PATCH v3 0/5] Improving apt cache

[Roberto A. Foglietta]

On Mon, 30 Jan 2023 at 11:45, Henning Schild <henning.schild@siemens.com> wrote:
>
> Am Mon, 30 Jan 2023 09:45:45 +0100
> schrieb "Moessbauer, Felix (T CED INW-CN)"
> <felix.moessbauer@siemens.com>:
>
> > On Fri, 2023-01-20 at 08:31 +0100, Uladzimir Bely wrote:
> > > Currently, apt cache (e.g. `var/cache/apt/archives`) import and
> > > export
> > > functions are not optimal. Multiple files are copied from global
> > > DL_DIR to package WORKDIR, increasing disk IO and space needed.
> > >
> > > Also, various chroots (bootstrap, buildchroot, sbuild chroot)
> > > include their apt caches to sstate cache files.
> > >
> > > This patchset switches to hardlinks instead of copies and removes
> > > apt cache from bootstrapped images ans sstate caches.
> >
> > I just saw that this pattern does NOT work in case the cache is on a
> > different filesystem. This unfortunately is the case for all CI
> > systems with locally mounted caches, as well as for kas-container
> > builds with DL_DIR outside the KAS_WORK_DIR.
>
> I remember discussing that issue with someone because that is always
> what happens when you implement hard linking. Not sure it was on the
> list because i kept away from this lively discussion.
>
> It is like when someone adds overlayfs and then finds that it can not
> be nested a few weeks later ;).

The overlayfs is not the right way to go because a simpler approach
should prefered. The problem of going for straight simple solutions
(small patches) is that in the long run this brings us to an
overcomplicated system. So, from time to time, it is necessary to
approach the problem from a radical point of view. This is not because
a radical change would be necessary but because we need to re-think
the system and then see how options are viable and then take a choice
among them balancing our choice between various aspects.

Best regards, R-

Re: [PATCH v3 0/5] Improving apt cache

[Roberto A. Foglietta]

On Mon, 30 Jan 2023 at 09:45, Moessbauer, Felix
<felix.moessbauer@siemens.com> wrote:
>
> On Fri, 2023-01-20 at 08:31 +0100, Uladzimir Bely wrote:
> > Currently, apt cache (e.g. `var/cache/apt/archives`) import and
> > export
> > functions are not optimal. Multiple files are copied from global
> > DL_DIR to package WORKDIR, increasing disk IO and space needed.
> >
> > Also, various chroots (bootstrap, buildchroot, sbuild chroot) include
> > their apt caches to sstate cache files.
> >
> > This patchset switches to hardlinks instead of copies and removes apt
> > cache from bootstrapped images ans sstate caches.
>
> I just saw that this pattern does NOT work in case the cache is on a
> different filesystem. This unfortunately is the case for all CI systems
> with locally mounted caches, as well as for kas-container builds with
> DL_DIR outside the KAS_WORK_DIR.
>
> In short: This breaks a LOT of use-cases.
> I'm really sorry that we only found that by know.
> But we definitely need a different solution.
>
> Example output:
> 2023-01-30 08:38:51 - INFO     - | ln: failed to create hard link
> '/builds/iiot-edge-device/foo/meta-iot2050-pg2-
> foo/build/tmp/work/debian-bullseye-amd64/isar-bootstrap-target/1.0-
> r0/rootfs/var/cache/apt/archives/gcc-arm-linux-gnueabihf_4%3a10.2.1-
> 1_arm64.deb' => '/local-cache/meta-iot2050-pg2-
> foo/downloads/deb/debian-bullseye/gcc-arm-linux-gnueabihf_4%3a10.2.1-
> 1_arm64.deb': Invalid cross-device link
>

One more thing: as shown to Uladzimir, as long as the source of two
different mount points (not filesystems) exists on the same
filesystem, then lp -P complains but for some reason it works. I do
not grant that it works in any case but it works enough for which a
2>/dev/null makes the building complete and no one notices. So, if it
does not build, that's an issue. If it completes the build despite the
stderr messages, we are lucky. However, if confirmed - this weird
behaviour - is out of the standard, so being lucky is not enough in
the long run.

Best regards, R-