[PATCH 0/2] Improving apt cache

[PATCH 0/2] Improving apt cache

[Uladzimir Bely]

Currently, apt cache (e.g. `var/cache/apt/archives`) import and export
functions are not optimal. Multiple files are imported from global
DL_DIR to package WORKDIR, increasing disk IO and size needed.

Also, various chroots (bootstrap, buildchroot, sbuild chroot) include
their apt caches to sstate cache files.

This patchset switches to hardlinks instead of copies and removes apt
cache from bootstrapped images.

Currently (measured on qemuarm64-bullseye cross-compilation with
maximum 8 parallel tasks (e.g. 8 CPU cores)):
- build directory size reduced from 8878 to 7594 MiB
- runtime maximum disk usage reduced from 16018 to 12479 MiB

TODO:
- cleanup other rootfs's (sbuild-chroot, buildchroot)
- deal with additional copying in sbuild routines

Uladzimir Bely (2):
  Use hardlinks in deb-dl-dir functions
  Clean apt cache from debootstrapped rootfs dirs

 meta/classes/deb-dl-dir.bbclass                     | 4 ++--
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 6 ++++++
 2 files changed, 8 insertions(+), 2 deletions(-)

-- 
2.20.1

[PATCH 1/2] Use hardlinks in deb-dl-dir functions

[Uladzimir Bely]

This saves disk space and reduces IO

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/deb-dl-dir.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 3b1517dc..d846937a 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -86,7 +86,7 @@ deb_dl_dir_import() {
         printenv | grep -q BB_VERBOSE_LOGS && set -x
 
         sudo find "${pc}" -type f -iname "*\.deb" -exec \
-            cp -n --no-preserve=owner -t "${rootfs}"/var/cache/apt/archives/ {} +
+            cp -l -n --no-preserve=owner -t "${rootfs}"/var/cache/apt/archives/ {} +
     '
 }
 
@@ -109,7 +109,7 @@ deb_dl_dir_export() {
             if [ -n "$package" ]; then
                 cmp --silent "$package" "$p" && continue
             fi
-            sudo cp -n "${p}" "${pc}"
+            sudo cp -l -n "${p}" "${pc}"
         done
         sudo chown -R $(id -u):$(id -g) "${pc}"
     '
-- 
2.20.1

Re: [PATCH 1/2] Use hardlinks in deb-dl-dir functions

[Moessbauer, Felix]

On Fri, 2022-12-30 at 05:38 +0100, Uladzimir Bely wrote:
> This saves disk space and reduces IO
> 
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  meta/classes/deb-dl-dir.bbclass | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-
> dir.bbclass
> index 3b1517dc..d846937a 100644
> --- a/meta/classes/deb-dl-dir.bbclass
> +++ b/meta/classes/deb-dl-dir.bbclass
> @@ -86,7 +86,7 @@ deb_dl_dir_import() {
>          printenv | grep -q BB_VERBOSE_LOGS && set -x
>  
>          sudo find "${pc}" -type f -iname "*\.deb" -exec \
> -            cp -n --no-preserve=owner -t
> "${rootfs}"/var/cache/apt/archives/ {} +
> +            cp -l -n --no-preserve=owner -t
> "${rootfs}"/var/cache/apt/archives/ {} +
>      '

I just observed the following race condition on the cache:

| DEBUG: Python function sstate_hardcode_path finished
| DEBUG: Executing shell function rootfs_install_sstate_prepare
| tar: rootfs/var/cache/apt/archives/network-manager_1.30.6-
1+deb11u1_arm64.deb: file changed as we read it
| WARNING: exit code 1 from a shell command.
| DEBUG: Python function sstate_task_postfunc finished
| ERROR: ExecutionError('/build/tmp/work/debian-bullseye-amd64/sbuild-
chroot-target/1.0-r0/temp/run.rootfs_install_sstate_prepare.7023', 1,
None, None)
ERROR: Task (/build/../work/isar/meta/recipes-devtools/sbuild-
chroot/sbuild-chroot-target.bb:do_rootfs_install) failed with exit code
'1'

Could it be that we copied a file that had not been fully downloaded
yet?

Felix

>  }
>  
> @@ -109,7 +109,7 @@ deb_dl_dir_export() {
>              if [ -n "$package" ]; then
>                  cmp --silent "$package" "$p" && continue
>              fi
> -            sudo cp -n "${p}" "${pc}"
> +            sudo cp -l -n "${p}" "${pc}"
>          done
>          sudo chown -R $(id -u):$(id -g) "${pc}"
>      '
> -- 
> 2.20.1
> 

Re: [PATCH 1/2] Use hardlinks in deb-dl-dir functions

[Roberto A. Foglietta]

On Fri, 30 Dec 2022 at 08:30, Moessbauer, Felix
<felix.moessbauer@siemens.com> wrote:
>
> On Fri, 2022-12-30 at 05:38 +0100, Uladzimir Bely wrote:
> > This saves disk space and reduces IO
> >
>
> I just observed the following race condition on the cache:
>

Unfortunately, cp -l does not work as supposed to - it breaks things
while ln -P does not but I have no clue why, I just tried an
alternative at the first failure.

Best regards, R-

[PATCH 2/2] Clean apt cache from debootstrapped rootfs dirs

[Uladzimir Bely]

This cleans `/var/cache/apt/archives` contents from debootstrapped
rootfs directories after exporting their contents to download
directory.

This reduces disk usage by both files and sstate caches.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index b9ae16cd..b7619262 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -408,6 +408,12 @@ do_bootstrap() {
         ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
 EOSUDO
     deb_dl_dir_export "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+
+    # Cleanup apt cache
+    sudo -E -s <<'EOSUDO'
+        set -e
+        chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean
+EOSUDO
 }
 
 addtask bootstrap before do_build after do_generate_keyrings
-- 
2.20.1

Re: [PATCH 2/2] Clean apt cache from debootstrapped rootfs dirs

[Roberto A. Foglietta]

On Fri, 30 Dec 2022 at 05:39, Uladzimir Bely <ubely@ilbers.de> wrote:
>
> This cleans `/var/cache/apt/archives` contents from debootstrapped
> rootfs directories after exporting their contents to download
> directory.
>
> This reduces disk usage by both files and sstate caches.
>
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index b9ae16cd..b7619262 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -408,6 +408,12 @@ do_bootstrap() {
>          ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
>  EOSUDO
>      deb_dl_dir_export "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
> +
> +    # Cleanup apt cache
> +    sudo -E -s <<'EOSUDO'
> +        set -e
> +        chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean
> +EOSUDO
>  }
>

Hi Uladzmir, I have implemented this one as following

sudo -Es chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean

and in my case save the disk size equal to build/dowloads/deb

Best regards, R-

Re: [PATCH 2/2] Clean apt cache from debootstrapped rootfs dirs

[Henning Schild]

Am Fri, 30 Dec 2022 05:38:58 +0100
schrieb Uladzimir Bely <ubely@ilbers.de>:

> This cleans `/var/cache/apt/archives` contents from debootstrapped
> rootfs directories after exporting their contents to download
> directory.
> 
> This reduces disk usage by both files and sstate caches.
> 
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> b9ae16cd..b7619262 100644 ---
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -408,6
> +408,12 @@ do_bootstrap() { ln -Tfsr "${ROOTFSDIR}"
> "${DEPLOY_ISAR_BOOTSTRAP}" EOSUDO
>      deb_dl_dir_export "${ROOTFSDIR}"
> "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}" +
> +    # Cleanup apt cache
> +    sudo -E -s <<'EOSUDO'
> +        set -e
> +        chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean
> +EOSUDO

I guess the whole "EOSUDO" and "set -e" and "-s" is a little overkill
for one command.

Probably this will work as well.

 sudo -E chroot '${ROOTFSDIR}' /usr/bin/apt-get -y clean

Henning

>  }
>  
>  addtask bootstrap before do_build after do_generate_keyrings

Re: [PATCH 2/2] Clean apt cache from debootstrapped rootfs dirs

[Henning Schild]

Am Fri, 30 Dec 2022 05:38:58 +0100
schrieb Uladzimir Bely <ubely@ilbers.de>:

> This cleans `/var/cache/apt/archives` contents from debootstrapped
> rootfs directories after exporting their contents to download
> directory.
> 
> This reduces disk usage by both files and sstate caches.
> 
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> b9ae16cd..b7619262 100644 ---
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -408,6
> +408,12 @@ do_bootstrap() { ln -Tfsr "${ROOTFSDIR}"
> "${DEPLOY_ISAR_BOOTSTRAP}" EOSUDO
>      deb_dl_dir_export "${ROOTFSDIR}"
> "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}" +
> +    # Cleanup apt cache
> +    sudo -E -s <<'EOSUDO'
> +        set -e
> +        chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean
> +EOSUDO

I did not look into the details, but maybe that is what one might want
after most of not all calls to deb_dl_dir_export.

Meaning the clean should be put in there, and if not all we add an
argument to only clean sometimes.

Might be worth trying.

Henning

>  }
>  
>  addtask bootstrap before do_build after do_generate_keyrings

Re: [PATCH 2/2] Clean apt cache from debootstrapped rootfs dirs

[Roberto A. Foglietta]

[-- Attachment #1: Type: text/plain, Size: 1506 bytes --]

Il Lun 2 Gen 2023, 17:33 Henning Schild <henning.schild@siemens.com> ha
scritto:

> Am Fri, 30 Dec 2022 05:38:58 +0100
> schrieb Uladzimir Bely <ubely@ilbers.de>:
>
>         chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean
>
> I did not look into the details, but maybe that is what one might want
> after most of not all calls to deb_dl_dir_export.


Hi all,

There is a specific function/task that do that in rootfs post processing
which means that clean the apt cache BEFORE importing it (*).

Then with the schroot only patchset (which misaing the dpkg base class to
rip off buildchroot but it is straightforward to achieve) there is just one
need of this after inage finalisation (but just for aesthetic reasons due
to the use of ln -P).

(*) that task is currently also removing debian lists which is not intended
for and IHMO the two ROOTFS_FEATURE should be separated.

However, inserting at the end of deb_dl_export would simply the whole
architecture. Moreover, every import/export if packages should also do the
same with apt-get update lists in such a way it will be run faster also
with a slow Internet connection.

Another trick that matters to implement: wic stuff is produced into a local
schroot mount point and then moved outside causing a duplication of disk
I/O while using the final destination immediately will save that burden and
when the inage is about 10Gb, is not a little thing.

All these stuff can be seen in the branch 'schroot' of my fork plus some
fixes.

Best regards, R.

[-- Attachment #2: Type: text/html, Size: 2349 bytes --]