[PATCH v2 0/3] Improving apt cache

[PATCH v2 0/3] Improving apt cache

[Uladzimir Bely]

Currently, apt cache (e.g. `var/cache/apt/archives`) import and export
functions are not optimal. Multiple files are imported from global
DL_DIR to package WORKDIR, increasing disk IO and size needed.

Also, various chroots (bootstrap, buildchroot, sbuild chroot) include
their apt caches to sstate cache files.

This patchset switches to hardlinks instead of copies and removes apt
cache from bootstrapped images ans sstate caches.

Currently (measured on qemuarm64-bullseye cross-compilation with
maximum 8 parallel tasks (e.g. 8 CPU cores)):
- build directory size reduced from 8906 to 6675 MiB
- runtime maximum disk usage reduced from 15965 to 8501 MiB

TODO:
- cleanup other rootfs's (sbuild-chroot, buildchroot).
  Actually, this won't bring much benefit, but why not cleanup final
  rootfs's apt cache when the build finishes?
- deal with additional copying in sbuild routines (patch 3).
  We could use hardlinks instead of copying packages between upper
  layer (where sbuild temporarly keeps them) and workdir rootfs,
  but it is not expected to bring much benefit, since upper layer
  temporary nature. Additinal measurements are required.

Changes since v1:
 - Simplified cleanup of apt cache in debootstrap rootfs.
 - Now "ln" instead of "cp -l" used.
 - Removed apt cache contents from sstate cache. The idea is proposed
in patch 3, but it was reworked and fixed. Firstly, CACHEDIR.TAG can't
be just a file (e.g. created by 'touch'), it should include some
specific signature [1]. Secondly, it's easier to just create this tag
in bootstrapped rootfs and it will be automatically used in all
derivatives (sbuild-chroot/buildchroot/image). So, the original patch
from Roberto A. Foglietta was simplified.

This patchset includes (or absorbs) the logic from p1..p3 patches of
the series Roberto prosposed. What concerns additional patches, they
don't let us benefit much, but require quite significant changes
in Isar, so we should check twice if they are worth including.

[1] https://bford.info/cachedir/
- 

Roberto A. Foglietta (1):
  Changes for a faster build using less disk space

Uladzimir Bely (2):
  Clean apt cache from debootstrapped rootfs dirs
  Use hardlinks in deb-dl-dir functions

 meta/classes/deb-dl-dir.bbclass                     |  6 +++---
 meta/classes/rootfs.bbclass                         |  3 ++-
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 10 +++++++++-
 3 files changed, 14 insertions(+), 5 deletions(-)

-- 
2.20.1

[PATCH v2 1/3] Clean apt cache from debootstrapped rootfs dirs

[Uladzimir Bely]

This cleans `/var/cache/apt/archives` contents from debootstrapped
rootfs directories after exporting their contents to download
directory.

This reduces disk usage by both files and sstate caches.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index faba73fe..cb0079ec 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -392,6 +392,9 @@ do_bootstrap() {
         ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
 EOSUDO
     deb_dl_dir_export "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+
+    # Cleanup apt cache
+    sudo -Es chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean
 }
 
 addtask bootstrap before do_build after do_generate_keyrings
-- 
2.20.1

[PATCH v2 2/3] Use hardlinks in deb-dl-dir functions

[Uladzimir Bely]

This saves disk space and reduces IO

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/deb-dl-dir.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 3b1517dc..7db25251 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -86,7 +86,7 @@ deb_dl_dir_import() {
         printenv | grep -q BB_VERBOSE_LOGS && set -x
 
         sudo find "${pc}" -type f -iname "*\.deb" -exec \
-            cp -n --no-preserve=owner -t "${rootfs}"/var/cache/apt/archives/ {} +
+            ln -Pf -t "${rootfs}"/var/cache/apt/archives/ {} +
     '
 }
 
@@ -109,7 +109,7 @@ deb_dl_dir_export() {
             if [ -n "$package" ]; then
                 cmp --silent "$package" "$p" && continue
             fi
-            sudo cp -n "${p}" "${pc}"
+            sudo ln -Pf "${p}" "${pc}"
         done
         sudo chown -R $(id -u):$(id -g) "${pc}"
     '
-- 
2.20.1

[PATCH v2 3/3] Changes for a faster build using less disk space

[Uladzimir Bely]

From: "Roberto A. Foglietta" <roberto.foglietta@gmail.com>

Faster build for larger projects with a lot of debian packages

This patch leverages few ways to accomplish the task to optimise
the large buildings.

Put a CACHEDIR.TAG file in each cache folder in such a way every
tar that uses the option --exclude-caches could be avoided
to include debian packages saving a lot of time and disk space.

Signed-off-by: Roberto A. Foglietta <roberto.foglietta@gmail.com>
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/deb-dl-dir.bbclass                     | 2 +-
 meta/classes/rootfs.bbclass                         | 3 ++-
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 ++++++-
 3 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 7db25251..3afad4e8 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -102,7 +102,7 @@ deb_dl_dir_export() {
             -maxdepth 1 -type f -iname '*\.deb' |\
         while read p; do
             # skip files from a previous export
-            [ -f "${pc}/${p##*/}" ] && continue
+            [ -e "${pc}/${p##*/}" ] && continue
             # can not reuse bitbake function here, this is basically
             # "repo_contains_package"
             package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name ${p##*/})
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 786682d9..f274443d 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -308,7 +308,8 @@ rootfs_install_sstate_prepare() {
     # so we use some mount magic to prevent that
     mkdir -p ${WORKDIR}/mnt/rootfs
     sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
-    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system rootfs
+    lopts="--one-file-system --exclude-caches"
+    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar $lopts rootfs
     sudo umount ${WORKDIR}/mnt/rootfs
     sudo chown $(id -u):$(id -g) rootfs.tar
 }
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index cb0079ec..44019f95 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -395,6 +395,10 @@ EOSUDO
 
     # Cleanup apt cache
     sudo -Es chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean
+
+    # Don't include apt cache into sstate cache by marking the directory
+    # with a CACHEDIR.TAG file with a special signature
+    sudo -Es sh -c "echo Signature: 8a477f597d28d172789f06886806bc55 > ${ROOTFSDIR}/var/cache/apt/archives/CACHEDIR.TAG"
 }
 
 addtask bootstrap before do_build after do_generate_keyrings
@@ -405,7 +409,8 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
 
 bootstrap_sstate_prepare() {
     # this runs in SSTATE_BUILDDIR, which will be deleted automatically
-    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar --one-file-system $(basename "${ROOTFSDIR}")
+    lopts="--one-file-system --exclude-caches"
+    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts $(basename "${ROOTFSDIR}")
     sudo chown $(id -u):$(id -g) bootstrap.tar
 }
 
-- 
2.20.1

Re: [PATCH v2 3/3] Changes for a faster build using less disk space

[Henning Schild]

Am Fri,  6 Jan 2023 07:48:09 +0100
schrieb Uladzimir Bely <ubely@ilbers.de>:

> From: "Roberto A. Foglietta" <roberto.foglietta@gmail.com>
> 
> Faster build for larger projects with a lot of debian packages
> 
> This patch leverages few ways to accomplish the task to optimise
> the large buildings.
> 
> Put a CACHEDIR.TAG file in each cache folder in such a way every
> tar that uses the option --exclude-caches could be avoided
> to include debian packages saving a lot of time and disk space.
> 
> Signed-off-by: Roberto A. Foglietta <roberto.foglietta@gmail.com>
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  meta/classes/deb-dl-dir.bbclass                     | 2 +-
>  meta/classes/rootfs.bbclass                         | 3 ++-
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 ++++++-
>  3 files changed, 9 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes/deb-dl-dir.bbclass
> b/meta/classes/deb-dl-dir.bbclass index 7db25251..3afad4e8 100644
> --- a/meta/classes/deb-dl-dir.bbclass
> +++ b/meta/classes/deb-dl-dir.bbclass
> @@ -102,7 +102,7 @@ deb_dl_dir_export() {
>              -maxdepth 1 -type f -iname '*\.deb' |\
>          while read p; do
>              # skip files from a previous export
> -            [ -f "${pc}/${p##*/}" ] && continue
> +            [ -e "${pc}/${p##*/}" ] && continue
>              # can not reuse bitbake function here, this is basically
>              # "repo_contains_package"
>              package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name
> ${p##*/}) diff --git a/meta/classes/rootfs.bbclass
> b/meta/classes/rootfs.bbclass index 786682d9..f274443d 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -308,7 +308,8 @@ rootfs_install_sstate_prepare() {
>      # so we use some mount magic to prevent that
>      mkdir -p ${WORKDIR}/mnt/rootfs
>      sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
> -    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system
> rootfs
> +    lopts="--one-file-system --exclude-caches"
> +    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar $lopts rootfs
>      sudo umount ${WORKDIR}/mnt/rootfs
>      sudo chown $(id -u):$(id -g) rootfs.tar
>  }
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> cb0079ec..44019f95 100644 ---
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -395,6
> +395,10 @@ EOSUDO 
>      # Cleanup apt cache
>      sudo -Es chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean
> +
> +    # Don't include apt cache into sstate cache by marking the
> directory
> +    # with a CACHEDIR.TAG file with a special signature
> +    sudo -Es sh -c "echo Signature: 8a477f597d28d172789f06886806bc55

What is that "special signature" all about? Does anyone later care to
remove the file again, or do we risk it ending up in the final rootfs?

I find this a little weird and would prefer if the calls to tar would
simply exclude this by path and not by magic tool specific file.

It could be made a central function so we have all those tar calls in
one place.

In this patch i see three places ... hard to maintain.

Henning

> > ${ROOTFSDIR}/var/cache/apt/archives/CACHEDIR.TAG" }
>  
>  addtask bootstrap before do_build after do_generate_keyrings
> @@ -405,7 +409,8 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
>  
>  bootstrap_sstate_prepare() {
>      # this runs in SSTATE_BUILDDIR, which will be deleted
> automatically
> -    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar
> --one-file-system $(basename "${ROOTFSDIR}")
> +    lopts="--one-file-system --exclude-caches"
> +    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts
> $(basename "${ROOTFSDIR}") sudo chown $(id -u):$(id -g) bootstrap.tar
>  }
>  

Re: [PATCH v2 3/3] Changes for a faster build using less disk space

[Uladzimir Bely]

[-- Attachment #1: Type: text/plain, Size: 5040 bytes --]

In the email from Friday, 6 January 2023 18:58:26 +03 user Henning Schild 
wrote:
> Am Fri,  6 Jan 2023 07:48:09 +0100
> 
> schrieb Uladzimir Bely <ubely@ilbers.de>:
> > From: "Roberto A. Foglietta" <roberto.foglietta@gmail.com>
> > 
> > Faster build for larger projects with a lot of debian packages
> > 
> > This patch leverages few ways to accomplish the task to optimise
> > the large buildings.
> > 
> > Put a CACHEDIR.TAG file in each cache folder in such a way every
> > tar that uses the option --exclude-caches could be avoided
> > to include debian packages saving a lot of time and disk space.
> > 
> > Signed-off-by: Roberto A. Foglietta <roberto.foglietta@gmail.com>
> > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > ---
> > 
> >  meta/classes/deb-dl-dir.bbclass                     | 2 +-
> >  meta/classes/rootfs.bbclass                         | 3 ++-
> >  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 ++++++-
> >  3 files changed, 9 insertions(+), 3 deletions(-)
> > 
> > diff --git a/meta/classes/deb-dl-dir.bbclass
> > b/meta/classes/deb-dl-dir.bbclass index 7db25251..3afad4e8 100644
> > --- a/meta/classes/deb-dl-dir.bbclass
> > +++ b/meta/classes/deb-dl-dir.bbclass
> > @@ -102,7 +102,7 @@ deb_dl_dir_export() {
> > 
> >              -maxdepth 1 -type f -iname '*\.deb' |\
> >          
> >          while read p; do
> >          
> >              # skip files from a previous export
> > 
> > -            [ -f "${pc}/${p##*/}" ] && continue
> > +            [ -e "${pc}/${p##*/}" ] && continue
> > 
> >              # can not reuse bitbake function here, this is basically
> >              # "repo_contains_package"
> >              package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name
> > 
> > ${p##*/}) diff --git a/meta/classes/rootfs.bbclass
> > b/meta/classes/rootfs.bbclass index 786682d9..f274443d 100644
> > --- a/meta/classes/rootfs.bbclass
> > +++ b/meta/classes/rootfs.bbclass
> > @@ -308,7 +308,8 @@ rootfs_install_sstate_prepare() {
> > 
> >      # so we use some mount magic to prevent that
> >      mkdir -p ${WORKDIR}/mnt/rootfs
> >      sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
> > 
> > -    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system
> > rootfs
> > +    lopts="--one-file-system --exclude-caches"
> > +    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar $lopts rootfs
> > 
> >      sudo umount ${WORKDIR}/mnt/rootfs
> >      sudo chown $(id -u):$(id -g) rootfs.tar
> >  
> >  }
> > 
> > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > cb0079ec..44019f95 100644 ---
> > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -395,6
> > +395,10 @@ EOSUDO
> > 
> >      # Cleanup apt cache
> >      sudo -Es chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean
> > 
> > +
> > +    # Don't include apt cache into sstate cache by marking the
> > directory
> > +    # with a CACHEDIR.TAG file with a special signature
> > +    sudo -Es sh -c "echo Signature: 8a477f597d28d172789f06886806bc55
> 
> What is that "special signature" all about? Does anyone later care to
> remove the file again, or do we risk it ending up in the final rootfs?
> 
> I find this a little weird and would prefer if the calls to tar would
> simply exclude this by path and not by magic tool specific file.
> 

Original patch from Roberto included both `--exclude-caches` and `--
exclude=var/cache/apt/archives"` options for `tar`. First one was expected to 
automatically exclude all directories that contain special CACHEDIR.TAG file 
[1]. But it appeared not to work. Investigating the problem shown that it 
should not be empty file: GNU tar expects file starts with "Signature: 
8a477f597d28d172789f06886806bc55" line.

It means, all the stuff with multiple "touch CACHEDIR.TAG" was useless and the 
actually only option "--exclude=var/cache/apt/archives" did the trick.

I fixed first and removed second option. But you are right, we can do vice 
versa and it should look even simpler.

Final rootfs doesn't have this file since we do "apt-get clean" for it.

[1] https://bford.info/cachedir/

> It could be made a central function so we have all those tar calls in
> one place.
> 
> In this patch i see three places ... hard to maintain.
> 
> Henning
> 

> > > ${ROOTFSDIR}/var/cache/apt/archives/CACHEDIR.TAG" }
> >  
> >  addtask bootstrap before do_build after do_generate_keyrings
> > 
> > @@ -405,7 +409,8 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
> > 
> >  bootstrap_sstate_prepare() {
> >  
> >      # this runs in SSTATE_BUILDDIR, which will be deleted
> > 
> > automatically
> > -    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar
> > --one-file-system $(basename "${ROOTFSDIR}")
> > +    lopts="--one-file-system --exclude-caches"
> > +    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts
> > $(basename "${ROOTFSDIR}") sudo chown $(id -u):$(id -g) bootstrap.tar
> > 
> >  }


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v2 3/3] Changes for a faster build using less disk space

[Roberto A. Foglietta]

On Fri, 6 Jan 2023 at 18:38, Uladzimir Bely <ubely@ilbers.de> wrote:
>
> In the email from Friday, 6 January 2023 18:58:26 +03 user Henning Schild
> wrote:

> Original patch from Roberto included both `--exclude-caches` and `--
> exclude=var/cache/apt/archives"` options for `tar`. First one was expected to
> automatically exclude all directories that contain special CACHEDIR.TAG file
> [1]. But it appeared not to work. Investigating the problem shown that it
> should not be empty file: GNU tar expects file starts with "Signature:
> 8a477f597d28d172789f06886806bc55" line.
>
> It means, all the stuff with multiple "touch CACHEDIR.TAG" was useless and the
> actually only option "--exclude=var/cache/apt/archives" did the trick.
>
> I fixed first and removed second option. But you are right, we can do vice
> versa and it should look even simpler.
>

vice-versa means using --exclude because we know the folders in
advance and left --exclude-caches / --exclude-backups for the pain of
the users...
...as I did because it is simpler in terms of code changes and more
efficient at running time.
:-)

Bests, R-

Re: [PATCH v2 3/3] Changes for a faster build using less disk space

[Henning Schild]

Am Fri, 6 Jan 2023 20:38:05 +0300
schrieb Uladzimir Bely <ubely@ilbers.de>:

> In the email from Friday, 6 January 2023 18:58:26 +03 user Henning
> Schild wrote:
> > Am Fri,  6 Jan 2023 07:48:09 +0100
> > 
> > schrieb Uladzimir Bely <ubely@ilbers.de>:  
> > > From: "Roberto A. Foglietta" <roberto.foglietta@gmail.com>
> > > 
> > > Faster build for larger projects with a lot of debian packages
> > > 
> > > This patch leverages few ways to accomplish the task to optimise
> > > the large buildings.
> > > 
> > > Put a CACHEDIR.TAG file in each cache folder in such a way every
> > > tar that uses the option --exclude-caches could be avoided
> > > to include debian packages saving a lot of time and disk space.
> > > 
> > > Signed-off-by: Roberto A. Foglietta <roberto.foglietta@gmail.com>
> > > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > > ---
> > > 
> > >  meta/classes/deb-dl-dir.bbclass                     | 2 +-
> > >  meta/classes/rootfs.bbclass                         | 3 ++-
> > >  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 ++++++-
> > >  3 files changed, 9 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/meta/classes/deb-dl-dir.bbclass
> > > b/meta/classes/deb-dl-dir.bbclass index 7db25251..3afad4e8 100644
> > > --- a/meta/classes/deb-dl-dir.bbclass
> > > +++ b/meta/classes/deb-dl-dir.bbclass
> > > @@ -102,7 +102,7 @@ deb_dl_dir_export() {
> > > 
> > >              -maxdepth 1 -type f -iname '*\.deb' |\
> > >          
> > >          while read p; do
> > >          
> > >              # skip files from a previous export
> > > 
> > > -            [ -f "${pc}/${p##*/}" ] && continue
> > > +            [ -e "${pc}/${p##*/}" ] && continue
> > > 
> > >              # can not reuse bitbake function here, this is
> > > basically # "repo_contains_package"
> > >              package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name
> > > 
> > > ${p##*/}) diff --git a/meta/classes/rootfs.bbclass
> > > b/meta/classes/rootfs.bbclass index 786682d9..f274443d 100644
> > > --- a/meta/classes/rootfs.bbclass
> > > +++ b/meta/classes/rootfs.bbclass
> > > @@ -308,7 +308,8 @@ rootfs_install_sstate_prepare() {
> > > 
> > >      # so we use some mount magic to prevent that
> > >      mkdir -p ${WORKDIR}/mnt/rootfs
> > >      sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o
> > > ro
> > > 
> > > -    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system
> > > rootfs
> > > +    lopts="--one-file-system --exclude-caches"
> > > +    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar $lopts rootfs
> > > 
> > >      sudo umount ${WORKDIR}/mnt/rootfs
> > >      sudo chown $(id -u):$(id -g) rootfs.tar
> > >  
> > >  }
> > > 
> > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > > cb0079ec..44019f95 100644 ---
> > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -395,6
> > > +395,10 @@ EOSUDO
> > > 
> > >      # Cleanup apt cache
> > >      sudo -Es chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean
> > > 
> > > +
> > > +    # Don't include apt cache into sstate cache by marking the
> > > directory
> > > +    # with a CACHEDIR.TAG file with a special signature
> > > +    sudo -Es sh -c "echo Signature:
> > > 8a477f597d28d172789f06886806bc55  
> > 
> > What is that "special signature" all about? Does anyone later care
> > to remove the file again, or do we risk it ending up in the final
> > rootfs?
> > 
> > I find this a little weird and would prefer if the calls to tar
> > would simply exclude this by path and not by magic tool specific
> > file. 
> 
> Original patch from Roberto included both `--exclude-caches` and `--
> exclude=var/cache/apt/archives"` options for `tar`. First one was
> expected to automatically exclude all directories that contain
> special CACHEDIR.TAG file [1]. But it appeared not to work.
> Investigating the problem shown that it should not be empty file: GNU
> tar expects file starts with "Signature:
> 8a477f597d28d172789f06886806bc55" line.
> 
> It means, all the stuff with multiple "touch CACHEDIR.TAG" was
> useless and the actually only option
> "--exclude=var/cache/apt/archives" did the trick.
> 
> I fixed first and removed second option. But you are right, we can do
> vice versa and it should look even simpler.
> 
> Final rootfs doesn't have this file since we do "apt-get clean" for
> it.

I guess we got lucky. Who knows what future versions of apt-get might
do when strange files hang around.

Henning

> [1] https://bford.info/cachedir/
> 
> > It could be made a central function so we have all those tar calls
> > in one place.
> > 
> > In this patch i see three places ... hard to maintain.
> > 
> > Henning
> >   
> 
> > > > ${ROOTFSDIR}/var/cache/apt/archives/CACHEDIR.TAG" }  
> > >  
> > >  addtask bootstrap before do_build after do_generate_keyrings
> > > 
> > > @@ -405,7 +409,8 @@ SSTATEPOSTINSTFUNCS +=
> > > "bootstrap_sstate_finalize"
> > > 
> > >  bootstrap_sstate_prepare() {
> > >  
> > >      # this runs in SSTATE_BUILDDIR, which will be deleted
> > > 
> > > automatically
> > > -    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar
> > > --one-file-system $(basename "${ROOTFSDIR}")
> > > +    lopts="--one-file-system --exclude-caches"
> > > +    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar
> > > $lopts $(basename "${ROOTFSDIR}") sudo chown $(id -u):$(id -g)
> > > bootstrap.tar
> > > 
> > >  }  
> 

Re: [PATCH v2 0/3] Improving apt cache

[Moessbauer, Felix]

On Fri, 2023-01-06 at 07:48 +0100, Uladzimir Bely wrote:
> Currently, apt cache (e.g. `var/cache/apt/archives`) import and
> export
> functions are not optimal. Multiple files are imported from global
> DL_DIR to package WORKDIR, increasing disk IO and size needed.
> 
> Also, various chroots (bootstrap, buildchroot, sbuild chroot) include
> their apt caches to sstate cache files.
> 
> This patchset switches to hardlinks instead of copies and removes apt
> cache from bootstrapped images ans sstate caches.
> 
> Currently (measured on qemuarm64-bullseye cross-compilation with
> maximum 8 parallel tasks (e.g. 8 CPU cores)):
> - build directory size reduced from 8906 to 6675 MiB
> - runtime maximum disk usage reduced from 15965 to 8501 MiB
> 
> TODO:
> - cleanup other rootfs's (sbuild-chroot, buildchroot).
>   Actually, this won't bring much benefit, but why not cleanup final
>   rootfs's apt cache when the build finishes?
> - deal with additional copying in sbuild routines (patch 3).
>   We could use hardlinks instead of copying packages between upper
>   layer (where sbuild temporarly keeps them) and workdir rootfs,
>   but it is not expected to bring much benefit, since upper layer
>   temporary nature. Additinal measurements are required.

Hi,

if the cache is 20GB, we still copy 20GB of data into the upper layer
(per sbuild task).
This has a huge impact and I would really appreciate if we could use
hardlinks for that part.

Felix

> 
> Changes since v1:
>  - Simplified cleanup of apt cache in debootstrap rootfs.
>  - Now "ln" instead of "cp -l" used.
>  - Removed apt cache contents from sstate cache. The idea is proposed
> in patch 3, but it was reworked and fixed. Firstly, CACHEDIR.TAG
> can't
> be just a file (e.g. created by 'touch'), it should include some
> specific signature [1]. Secondly, it's easier to just create this tag
> in bootstrapped rootfs and it will be automatically used in all
> derivatives (sbuild-chroot/buildchroot/image). So, the original patch
> from Roberto A. Foglietta was simplified.
> 
> This patchset includes (or absorbs) the logic from p1..p3 patches of
> the series Roberto prosposed. What concerns additional patches, they
> don't let us benefit much, but require quite significant changes
> in Isar, so we should check twice if they are worth including.
> 
> [1] https://bford.info/cachedir/
> - 
> 
> Roberto A. Foglietta (1):
>   Changes for a faster build using less disk space
> 
> Uladzimir Bely (2):
>   Clean apt cache from debootstrapped rootfs dirs
>   Use hardlinks in deb-dl-dir functions
> 
>  meta/classes/deb-dl-dir.bbclass                     |  6 +++---
>  meta/classes/rootfs.bbclass                         |  3 ++-
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 10 +++++++++-
>  3 files changed, 14 insertions(+), 5 deletions(-)
> 
> -- 
> 2.20.1
> 

Re: [PATCH v2 0/3] Improving apt cache

[Uladzimir Bely]

In the email from Monday, 9 January 2023 09:32:29 +03 user Moessbauer, Felix 
wrote:
> On Fri, 2023-01-06 at 07:48 +0100, Uladzimir Bely wrote:
> 
> > Currently, apt cache (e.g. `var/cache/apt/archives`) import and
> > export
> > functions are not optimal. Multiple files are imported from global
> > DL_DIR to package WORKDIR, increasing disk IO and size needed.
> > 
> > Also, various chroots (bootstrap, buildchroot, sbuild chroot) include
> > their apt caches to sstate cache files.
> > 
> > This patchset switches to hardlinks instead of copies and removes apt
> > cache from bootstrapped images ans sstate caches.
> > 
> > Currently (measured on qemuarm64-bullseye cross-compilation with
> > maximum 8 parallel tasks (e.g. 8 CPU cores)):
> > - build directory size reduced from 8906 to 6675 MiB
> > - runtime maximum disk usage reduced from 15965 to 8501 MiB
> > 
> > TODO:
> > - cleanup other rootfs's (sbuild-chroot, buildchroot).
> >   Actually, this won't bring much benefit, but why not cleanup final
> >   rootfs's apt cache when the build finishes?
> > - deal with additional copying in sbuild routines (patch 3).
> >   We could use hardlinks instead of copying packages between upper
> >   layer (where sbuild temporarly keeps them) and workdir rootfs,
> >   but it is not expected to bring much benefit, since upper layer
> >   temporary nature. Additinal measurements are required.
> 
> 
> Hi,
> 
> if the cache is 20GB, we still copy 20GB of data into the upper layer
> (per sbuild task).
> This has a huge impact and I would really appreciate if we could use
> hardlinks for that part.
> 
> Felix
> 

Hi.

I need to check if it really works. My doubts are related to overlayfs: if we 
are really able to create hardlinks between DL_DIR (that is common linux 
filesystem) and upper layer of overlayfs (that is technically the same 
partition, but it is a "mount option" of the different "overlay" filesystem).

> 
> > 
> > Changes since v1:
> >  - Simplified cleanup of apt cache in debootstrap rootfs.
> >  - Now "ln" instead of "cp -l" used.
> >  - Removed apt cache contents from sstate cache. The idea is proposed
> > in patch 3, but it was reworked and fixed. Firstly, CACHEDIR.TAG
> > can't
> > be just a file (e.g. created by 'touch'), it should include some
> > specific signature [1]. Secondly, it's easier to just create this tag
> > in bootstrapped rootfs and it will be automatically used in all
> > derivatives (sbuild-chroot/buildchroot/image). So, the original patch
> > from Roberto A. Foglietta was simplified.
> > 
> > This patchset includes (or absorbs) the logic from p1..p3 patches of
> > the series Roberto prosposed. What concerns additional patches, they
> > don't let us benefit much, but require quite significant changes
> > in Isar, so we should check twice if they are worth including.
> > 
> > [1] https://bford.info/cachedir/
> > - 
> > 
> > Roberto A. Foglietta (1):
> >   Changes for a faster build using less disk space
> > 
> > Uladzimir Bely (2):
> >   Clean apt cache from debootstrapped rootfs dirs
> >   Use hardlinks in deb-dl-dir functions
> > 
> >  meta/classes/deb-dl-dir.bbclass                     |  6 +++---
> >  meta/classes/rootfs.bbclass                         |  3 ++-
> >  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 10 +++++++++-
> >  3 files changed, 14 insertions(+), 5 deletions(-)
> > 
> > -- 
> > 2.20.1
> > 
> 
> 

Re: [PATCH v2 0/3] Improving apt cache

[Uladzimir Bely]

In the email from Monday, 9 January 2023 10:39:00 +03 user Uladzimir Bely 
wrote:
> In the email from Monday, 9 January 2023 09:32:29 +03 user Moessbauer, Felix
> wrote:
> > On Fri, 2023-01-06 at 07:48 +0100, Uladzimir Bely wrote:
> > > Currently, apt cache (e.g. `var/cache/apt/archives`) import and
> > > export
> > > functions are not optimal. Multiple files are imported from global
> > > DL_DIR to package WORKDIR, increasing disk IO and size needed.
> > > 
> > > Also, various chroots (bootstrap, buildchroot, sbuild chroot) include
> > > their apt caches to sstate cache files.
> > > 
> > > This patchset switches to hardlinks instead of copies and removes apt
> > > cache from bootstrapped images ans sstate caches.
> > > 
> > > Currently (measured on qemuarm64-bullseye cross-compilation with
> > > maximum 8 parallel tasks (e.g. 8 CPU cores)):
> > > - build directory size reduced from 8906 to 6675 MiB
> > > - runtime maximum disk usage reduced from 15965 to 8501 MiB
> > > 
> > > TODO:
> > > - cleanup other rootfs's (sbuild-chroot, buildchroot).
> > > 
> > >   Actually, this won't bring much benefit, but why not cleanup final
> > >   rootfs's apt cache when the build finishes?
> > > 
> > > - deal with additional copying in sbuild routines (patch 3).
> > > 
> > >   We could use hardlinks instead of copying packages between upper
> > >   layer (where sbuild temporarly keeps them) and workdir rootfs,
> > >   but it is not expected to bring much benefit, since upper layer
> > >   temporary nature. Additinal measurements are required.
> > 
> > Hi,
> > 
> > if the cache is 20GB, we still copy 20GB of data into the upper layer
> > (per sbuild task).
> > This has a huge impact and I would really appreciate if we could use
> > hardlinks for that part.
> > 
> > Felix
> 
> Hi.
> 
> I need to check if it really works. My doubts are related to overlayfs: if
> we are really able to create hardlinks between DL_DIR (that is common linux
> filesystem) and upper layer of overlayfs (that is technically the same
> partition, but it is a "mount option" of the different "overlay"
> filesystem).

I got time to get back to this patches and checked this moment. And it really 
does not work as I expected.

Original patch includes the following:

-        --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/
*.deb -t ${deb_dir}/ || :" \
+        --chroot-setup-commands="ln -Pf ${ext_deb_dir}/*.deb -t ${deb_dir}/ 
2>/dev/null || :" \

This results to to failing "ReproTest" in CI with the following error (for 
libhello, for example):

sbuild-build-depends-dose3-dummy:armhf : Depends: dose-distcheck:amd64 but it 
is not installable
E: Unable to correct problems, you have held broken packages.

To debug it, I removed `2>/dev/null` and found, that hardlinks simply don't 
work and the following errors are now seen earlier:

ln: failed to create hard link '/var/cache/apt/archives/adduser_3.118_all.deb' 
=> '/home/builder/libhello/rootfs/var/cache/apt/archives/
adduser_3.118_all.deb': Invalid cross-device link
... #tons of similar errors...
ln: failed to create hard link '/var/cache/apt/archives/
zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.deb' => '/home/builder/libhello/
rootfs/var/cache/apt/archives/zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.deb': 
Invalid cross-device link

I: Finished running 'ln -Pf /home/builder/libhello/rootfs/var/cache/apt/
archives/*.deb -t /var/cache/apt/archives/ || :'.

So, it works for network builds (when missing packages always can be 
downloaded by apt), but it fails for local builds from apt-cache (when, at 
first network build, sbuild dependencies are simply not exported to download 
directory due to non-working hardlinks, plus with hidden stderr)

> > > Changes since v1:
> > >  - Simplified cleanup of apt cache in debootstrap rootfs.
> > >  - Now "ln" instead of "cp -l" used.
> > >  - Removed apt cache contents from sstate cache. The idea is proposed
> > > 
> > > in patch 3, but it was reworked and fixed. Firstly, CACHEDIR.TAG
> > > can't
> > > be just a file (e.g. created by 'touch'), it should include some
> > > specific signature [1]. Secondly, it's easier to just create this tag
> > > in bootstrapped rootfs and it will be automatically used in all
> > > derivatives (sbuild-chroot/buildchroot/image). So, the original patch
> > > from Roberto A. Foglietta was simplified.
> > > 
> > > This patchset includes (or absorbs) the logic from p1..p3 patches of
> > > the series Roberto prosposed. What concerns additional patches, they
> > > don't let us benefit much, but require quite significant changes
> > > in Isar, so we should check twice if they are worth including.
> > > 
> > > [1] https://bford.info/cachedir/
> > > -
> > > 
> > > Roberto A. Foglietta (1):
> > >   Changes for a faster build using less disk space
> > > 
> > > Uladzimir Bely (2):
> > >   Clean apt cache from debootstrapped rootfs dirs
> > >   Use hardlinks in deb-dl-dir functions
> > >  
> > >  meta/classes/deb-dl-dir.bbclass                     |  6 +++---
> > >  meta/classes/rootfs.bbclass                         |  3 ++-
> > >  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 10 +++++++++-
> > >  3 files changed, 14 insertions(+), 5 deletions(-)

Re: [PATCH v2 0/3] Improving apt cache

[Roberto A. Foglietta]

[-- Attachment #1: Type: text/plain, Size: 3541 bytes --]

On Thu, 19 Jan 2023 at 08:36, Uladzimir Bely <ubely@ilbers.de> wrote:

>
> I got time to get back to this patches and checked this moment. And it
> really
> does not work as I expected.
>
> Original patch includes the following:
>
> -        --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/
> *.deb -t ${deb_dir}/ || :" \
> +        --chroot-setup-commands="ln -Pf ${ext_deb_dir}/*.deb -t
> ${deb_dir}/
> 2>/dev/null || :" \
>
> This results to to failing "ReproTest" in CI with the following error (for
> libhello, for example):
>
> sbuild-build-depends-dose3-dummy:armhf : Depends: dose-distcheck:amd64 but
> it
> is not installable
> E: Unable to correct problems, you have held broken packages.
>
> To debug it, I removed `2>/dev/null` and found, that hardlinks simply
> don't
> work and the following errors are now seen earlier:
>
> ln: failed to create hard link
> '/var/cache/apt/archives/adduser_3.118_all.deb'
> => '/home/builder/libhello/rootfs/var/cache/apt/archives/
> adduser_3.118_all.deb': Invalid cross-device link
> ... #tons of similar errors...
> ln: failed to create hard link '/var/cache/apt/archives/
> zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.deb' => '/home/builder/libhello/
> rootfs/var/cache/apt/archives/zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.deb':
>
> Invalid cross-device link
>
> I: Finished running 'ln -Pf /home/builder/libhello/rootfs/var/cache/apt/
> archives/*.deb -t /var/cache/apt/archives/ || :'.
>
> So, it works for network builds (when missing packages always can be
> downloaded by apt), but it fails for local builds from apt-cache (when, at
> first network build, sbuild dependencies are simply not exported to
> download
> directory due to non-working hardlinks, plus with hidden stderr)
>

Hi, first of all I do not fully understand why we use ln -P instead of ln
-Pf in the exporting debs.

        --finished-build-commands="ln -P ${deb_dir}/*.deb -t
${ext_deb_dir}/ 2>/dev/null || :" \

This code's goal is to build a deb that should be exported otherwise we are
continuing using the old package. It seems that breaks things but - again -
I did fully not understand what and why but simply accepted the suggestion.
Now, I have reverted back to ln -Pf.

Second, those lines are supposed to fail - and obviously fail in both
directions: import and export - but the following

(a) if export fail we will not have the custom packages but we have them
(b) removing that lines and the code will always fail

These mean that ln -Pf complains to fail but make a difference and make the
difference that we want. However, I might not have understood the case in
which it fails completely so:

mv build/downloads .
rm -rf build
./build.sh basic-os (DONE)
./clean.sh isar
./build.sh (DONE)
./clean.sh all
./build.sh (DONE)

remove that two lines about ln -Pf that are supposed to do nothing than fail

./clean.sh all
./build.sh (DONE with the same building time)

rm -rf build/downloads
mv downloads build

./clean.sh isar
./build.sh complete (FAIL)

put back those two command lines with ln -Pf and

./clean.sh all
./build.sh (DONE)

I do not say you are wrong and I see ln -Pf complains on stderr but nothing
that tin my private fork cannot be solved using a stderr redirection to
/dev/null

However there are some other ways to do this thing:

1. sbuild uses schroot by default, using chroot instead
2. keep the default in schroot, elsewhere it uses upper folder that can be
populated in advance

I did not make any changes because I did not identified the issue.

Best regards, R-

[-- Attachment #2: Type: text/html, Size: 4631 bytes --]

Re: [PATCH v2 0/3] Improving apt cache

[Roberto A. Foglietta]

[-- Attachment #1: Type: text/plain, Size: 2085 bytes --]

On Thu, 19 Jan 2023 at 15:52, Roberto A. Foglietta <
roberto.foglietta@gmail.com> wrote:

>
> On Thu, 19 Jan 2023 at 08:36, Uladzimir Bely <ubely@ilbers.de> wrote:
>
>>
>> I got time to get back to this patches and checked this moment. And it
>> really
>> does not work as I expected.
>>
>> Original patch includes the following:
>>
>> -        --chroot-setup-commands="cp -n --no-preserve=owner
>> ${ext_deb_dir}/
>> *.deb -t ${deb_dir}/ || :" \
>> +        --chroot-setup-commands="ln -Pf ${ext_deb_dir}/*.deb -t
>> ${deb_dir}/
>> 2>/dev/null || :" \
>>
>> This results to to failing "ReproTest" in CI with the following error
>> (for
>> libhello, for example):
>>
>> sbuild-build-depends-dose3-dummy:armhf : Depends: dose-distcheck:amd64
>> but it
>> is not installable
>> E: Unable to correct problems, you have held broken packages.
>>
>> To debug it, I removed `2>/dev/null` and found, that hardlinks simply
>> don't
>> work and the following errors are now seen earlier:
>>
>> ln: failed to create hard link
>> '/var/cache/apt/archives/adduser_3.118_all.deb'
>> => '/home/builder/libhello/rootfs/var/cache/apt/archives/
>> adduser_3.118_all.deb': Invalid cross-device link
>> ... #tons of similar errors...
>> ln: failed to create hard link '/var/cache/apt/archives/
>> zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.deb' => '/home/builder/libhello/
>> rootfs/var/cache/apt/archives/zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.deb':
>>
>> Invalid cross-device link
>>
>> I: Finished running 'ln -Pf /home/builder/libhello/rootfs/var/cache/apt/
>> archives/*.deb -t /var/cache/apt/archives/ || :'.
>>
>> So, it works for network builds (when missing packages always can be
>> downloaded by apt), but it fails for local builds from apt-cache (when,
>> at
>> first network build, sbuild dependencies are simply not exported to
>> download
>> directory due to non-working hardlinks, plus with hidden stderr)
>>
>

Yes, I confirm that it is possible to do sbuild without using neither cp
nor ln, using an elegant solution.

So, whatever the problem was - it is not a problem anymore.

Best, R-

[-- Attachment #2: Type: text/html, Size: 2909 bytes --]

Re: [PATCH v2 0/3] Improving apt cache

[Uladzimir Bely]

In mail from четверг, 19 января 2023 г. 17:52:37 +03 user Roberto A. Foglietta 
wrote:
> On Thu, 19 Jan 2023 at 08:36, Uladzimir Bely <ubely@ilbers.de> wrote:
> > I got time to get back to this patches and checked this moment. And it
> > really
> > does not work as I expected.
> > 
> > Original patch includes the following:
> > 
> > -        --chroot-setup-commands="cp -n --no-preserve=owner
> > ${ext_deb_dir}/
> > *.deb -t ${deb_dir}/ || :" \
> > +        --chroot-setup-commands="ln -Pf ${ext_deb_dir}/*.deb -t
> > ${deb_dir}/
> > 2>/dev/null || :" \
> > 
> > This results to to failing "ReproTest" in CI with the following error (for
> > libhello, for example):
> > 
> > sbuild-build-depends-dose3-dummy:armhf : Depends: dose-distcheck:amd64 but
> > it
> > is not installable
> > E: Unable to correct problems, you have held broken packages.
> > 
> > To debug it, I removed `2>/dev/null` and found, that hardlinks simply
> > don't
> > work and the following errors are now seen earlier:
> > 
> > ln: failed to create hard link
> > '/var/cache/apt/archives/adduser_3.118_all.deb'
> > => '/home/builder/libhello/rootfs/var/cache/apt/archives/
> > adduser_3.118_all.deb': Invalid cross-device link
> > ... #tons of similar errors...
> > ln: failed to create hard link '/var/cache/apt/archives/
> > zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.deb' => '/home/builder/libhello/
> > rootfs/var/cache/apt/archives/zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.deb'
> > :
> > 
> > Invalid cross-device link
> > 
> > I: Finished running 'ln -Pf /home/builder/libhello/rootfs/var/cache/apt/
> > archives/*.deb -t /var/cache/apt/archives/ || :'.
> > 
> > So, it works for network builds (when missing packages always can be
> > downloaded by apt), but it fails for local builds from apt-cache (when, at
> > first network build, sbuild dependencies are simply not exported to
> > download
> > directory due to non-working hardlinks, plus with hidden stderr)
> 
> Hi, first of all I do not fully understand why we use ln -P instead of ln
> -Pf in the exporting debs.
> 
>         --finished-build-commands="ln -P ${deb_dir}/*.deb -t
> ${ext_deb_dir}/ 2>/dev/null || :" \
> 
> This code's goal is to build a deb that should be exported otherwise we are
> continuing using the old package. It seems that breaks things but - again -
> I did fully not understand what and why but simply accepted the suggestion.
> Now, I have reverted back to ln -Pf.
> 
> Second, those lines are supposed to fail - and obviously fail in both
> directions: import and export - but the following
> 
> (a) if export fail we will not have the custom packages but we have them
> (b) removing that lines and the code will always fail
> 
> These mean that ln -Pf complains to fail but make a difference and make the
> difference that we want. However, I might not have understood the case in
> which it fails completely so:
> 
> mv build/downloads .
> rm -rf build
> ./build.sh basic-os (DONE)
> ./clean.sh isar
> ./build.sh (DONE)
> ./clean.sh all
> ./build.sh (DONE)
> 
> remove that two lines about ln -Pf that are supposed to do nothing than fail
> 
> ./clean.sh all
> ./build.sh (DONE with the same building time)
> 
> rm -rf build/downloads
> mv downloads build
> 
> ./clean.sh isar
> ./build.sh complete (FAIL)
> 
> put back those two command lines with ln -Pf and
> 
> ./clean.sh all
> ./build.sh (DONE)
> 
> I do not say you are wrong and I see ln -Pf complains on stderr but nothing
> that tin my private fork cannot be solved using a stderr redirection to
> /dev/null
> 
> However there are some other ways to do this thing:
> 
> 1. sbuild uses schroot by default, using chroot instead
> 2. keep the default in schroot, elsewhere it uses upper folder that can be
> populated in advance
> 
> I did not make any changes because I did not identified the issue.
> 
> Best regards, R-

I'll try to make things clear. Isar supports local builds from 'base-apt' repo 
with.

First build user does as usual.

Second build is done with the following changes (at least):

ISAR_USE_CACHED_BASE_REPO = "1"
BB_NO_NETWORK = "1"

In this case every .deb we downloaded from Debian mirrors at first build goes 
to local 'base-apt' and second build use it, but not remote mirrors.

That's the reason of using this "cp/ln magic" inside of sbuild. While building 
some package, multipe dependencies might be downloaded inside of sbuild. And 
we need a way to extract them from "internal" apt-cache in sbuild schroot and 
put to the package workdir, where deb-dl-dir export will pass them to DL_DIR 
later.

When we replace copying to hardlinks, that actually don't work due to 
different filesystems, we come to the situation, when multiple debian packages 
(dependencies of what we are building or some packages that sbuild requires 
itself) are not present in DL_DIR. So, second "cached" build from local repo 
simply fails.

I see the following possible solutions that allows to save space.
1. Since hardlinks don't work, try to use symlinks.
2. Don't use import with sbuild at all, but only export. But this has a 
drawback, when all packages we build with sbuild might download the similar 
dependencies in parallel.

Re: [PATCH v2 0/3] Improving apt cache

[Moessbauer, Felix]

On Fri, 2023-01-20 at 07:44 +0300, Uladzimir Bely wrote:
> In mail from четверг, 19 января 2023 г. 17:52:37 +03 user Roberto A.
> Foglietta 
> wrote:
> > On Thu, 19 Jan 2023 at 08:36, Uladzimir Bely <ubely@ilbers.de>
> > wrote:
> > > I got time to get back to this patches and checked this moment.
> > > And it
> > > really
> > > does not work as I expected.
> > > 
> > > Original patch includes the following:
> > > 
> > > -        --chroot-setup-commands="cp -n --no-preserve=owner
> > > ${ext_deb_dir}/
> > > *.deb -t ${deb_dir}/ || :" \
> > > +        --chroot-setup-commands="ln -Pf ${ext_deb_dir}/*.deb -t
> > > ${deb_dir}/
> > > 2>/dev/null || :" \
> > > 
> > > This results to to failing "ReproTest" in CI with the following
> > > error (for
> > > libhello, for example):
> > > 
> > > sbuild-build-depends-dose3-dummy:armhf : Depends: dose-
> > > distcheck:amd64 but
> > > it
> > > is not installable
> > > E: Unable to correct problems, you have held broken packages.
> > > 
> > > To debug it, I removed `2>/dev/null` and found, that hardlinks
> > > simply
> > > don't
> > > work and the following errors are now seen earlier:
> > > 
> > > ln: failed to create hard link
> > > '/var/cache/apt/archives/adduser_3.118_all.deb'
> > > => '/home/builder/libhello/rootfs/var/cache/apt/archives/
> > > adduser_3.118_all.deb': Invalid cross-device link
> > > ... #tons of similar errors...
> > > ln: failed to create hard link '/var/cache/apt/archives/
> > > zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.deb' =>
> > > '/home/builder/libhello/
> > > rootfs/var/cache/apt/archives/zlib1g_1%3a1.2.11.dfsg-
> > > 2+deb11u2_mipsel.deb'
> > > :
> > > 
> > > Invalid cross-device link
> > > 
> > > I: Finished running 'ln -Pf
> > > /home/builder/libhello/rootfs/var/cache/apt/
> > > archives/*.deb -t /var/cache/apt/archives/ || :'.
> > > 
> > > So, it works for network builds (when missing packages always can
> > > be
> > > downloaded by apt), but it fails for local builds from apt-cache
> > > (when, at
> > > first network build, sbuild dependencies are simply not exported
> > > to
> > > download
> > > directory due to non-working hardlinks, plus with hidden stderr)
> > 
> > Hi, first of all I do not fully understand why we use ln -P instead
> > of ln
> > -Pf in the exporting debs.
> > 
> >         --finished-build-commands="ln -P ${deb_dir}/*.deb -t
> > ${ext_deb_dir}/ 2>/dev/null || :" \
> > 
> > This code's goal is to build a deb that should be exported
> > otherwise we are
> > continuing using the old package. It seems that breaks things but -
> > again -
> > I did fully not understand what and why but simply accepted the
> > suggestion.
> > Now, I have reverted back to ln -Pf.
> > 
> > Second, those lines are supposed to fail - and obviously fail in
> > both
> > directions: import and export - but the following
> > 
> > (a) if export fail we will not have the custom packages but we have
> > them
> > (b) removing that lines and the code will always fail
> > 
> > These mean that ln -Pf complains to fail but make a difference and
> > make the
> > difference that we want. However, I might not have understood the
> > case in
> > which it fails completely so:
> > 
> > mv build/downloads .
> > rm -rf build
> > ./build.sh basic-os (DONE)
> > ./clean.sh isar
> > ./build.sh (DONE)
> > ./clean.sh all
> > ./build.sh (DONE)
> > 
> > remove that two lines about ln -Pf that are supposed to do nothing
> > than fail
> > 
> > ./clean.sh all
> > ./build.sh (DONE with the same building time)
> > 
> > rm -rf build/downloads
> > mv downloads build
> > 
> > ./clean.sh isar
> > ./build.sh complete (FAIL)
> > 
> > put back those two command lines with ln -Pf and
> > 
> > ./clean.sh all
> > ./build.sh (DONE)
> > 
> > I do not say you are wrong and I see ln -Pf complains on stderr but
> > nothing
> > that tin my private fork cannot be solved using a stderr
> > redirection to
> > /dev/null
> > 
> > However there are some other ways to do this thing:
> > 
> > 1. sbuild uses schroot by default, using chroot instead
> > 2. keep the default in schroot, elsewhere it uses upper folder that
> > can be
> > populated in advance
> > 
> > I did not make any changes because I did not identified the issue.
> > 
> > Best regards, R-
> 
> I'll try to make things clear. Isar supports local builds from 'base-
> apt' repo 
> with.
> 
> First build user does as usual.
> 
> Second build is done with the following changes (at least):
> 
> ISAR_USE_CACHED_BASE_REPO = "1"
> BB_NO_NETWORK = "1"
> 
> In this case every .deb we downloaded from Debian mirrors at first
> build goes 
> to local 'base-apt' and second build use it, but not remote mirrors.
> 
> That's the reason of using this "cp/ln magic" inside of sbuild. While
> building 
> some package, multipe dependencies might be downloaded inside of
> sbuild. And 
> we need a way to extract them from "internal" apt-cache in sbuild
> schroot and 
> put to the package workdir, where deb-dl-dir export will pass them to
> DL_DIR 
> later.
> 
> When we replace copying to hardlinks, that actually don't work due to
> different filesystems, we come to the situation, when multiple debian
> packages 
> (dependencies of what we are building or some packages that sbuild
> requires 
> itself) are not present in DL_DIR. So, second "cached" build from
> local repo 
> simply fails.
> 
> I see the following possible solutions that allows to save space.
> 1. Since hardlinks don't work, try to use symlinks.

If that works, it clearly is the simplest solution.

> 2. Don't use import with sbuild at all, but only export. But this has
> a 
> drawback, when all packages we build with sbuild might download the
> similar 
> dependencies in parallel.

A third solution might be to mount the cache into the schroot, similar
to how we handle the sstate cache today.

Anyways, I would appreciate if we could merge this series even if it
still does not fully solve the quadratic behavior. It is very valuable
and "good enough" even for my biggest layers.

Felix

> 
> 

Re: [PATCH v2 0/3] Improving apt cache

[Uladzimir Bely]

In mail from пятница, 20 января 2023 г. 08:08:03 +03 user Moessbauer, Felix 
wrote:
> On Fri, 2023-01-20 at 07:44 +0300, Uladzimir Bely wrote:
> 
> > In mail from четверг, 19 января 2023 г. 17:52:37 +03 user Roberto A.
> > Foglietta 
> > wrote:
> > 
> > > On Thu, 19 Jan 2023 at 08:36, Uladzimir Bely <ubely@ilbers.de>
> > > wrote:
> > > 
> > > > I got time to get back to this patches and checked this moment.
> > > > And it
> > > > really
> > > > does not work as I expected.
> > > > 
> > > > Original patch includes the following:
> > > > 
> > > > -        --chroot-setup-commands="cp -n --no-preserve=owner
> > > > ${ext_deb_dir}/
> > > > *.deb -t ${deb_dir}/ || :" \
> > > > +        --chroot-setup-commands="ln -Pf ${ext_deb_dir}/*.deb -t
> > > > ${deb_dir}/
> > > > 2>/dev/null || :" \
> > > > 
> > > > This results to to failing "ReproTest" in CI with the following
> > > > error (for
> > > > libhello, for example):
> > > > 
> > > > sbuild-build-depends-dose3-dummy:armhf : Depends: dose-
> > > > distcheck:amd64 but
> > > > it
> > > > is not installable
> > > > E: Unable to correct problems, you have held broken packages.
> > > > 
> > > > To debug it, I removed `2>/dev/null` and found, that hardlinks
> > > > simply
> > > > don't
> > > > work and the following errors are now seen earlier:
> > > > 
> > > > ln: failed to create hard link
> > > > '/var/cache/apt/archives/adduser_3.118_all.deb'
> > > > => '/home/builder/libhello/rootfs/var/cache/apt/archives/
> > > > adduser_3.118_all.deb': Invalid cross-device link
> > > > ... #tons of similar errors...
> > > > ln: failed to create hard link '/var/cache/apt/archives/
> > > > zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.deb' =>
> > > > '/home/builder/libhello/
> > > > rootfs/var/cache/apt/archives/zlib1g_1%3a1.2.11.dfsg-
> > > > 2+deb11u2_mipsel.deb'
> > > > 
> > > > :
> > > > 
> > > > 
> > > > Invalid cross-device link
> > > > 
> > > > I: Finished running 'ln -Pf
> > > > /home/builder/libhello/rootfs/var/cache/apt/
> > > > archives/*.deb -t /var/cache/apt/archives/ || :'.
> > > > 
> > > > So, it works for network builds (when missing packages always can
> > > > be
> > > > downloaded by apt), but it fails for local builds from apt-cache
> > > > (when, at
> > > > first network build, sbuild dependencies are simply not exported
> > > > to
> > > > download
> > > > directory due to non-working hardlinks, plus with hidden stderr)
> > > 
> > > 
> > > Hi, first of all I do not fully understand why we use ln -P instead
> > > of ln
> > > -Pf in the exporting debs.
> > > 
> > >         --finished-build-commands="ln -P ${deb_dir}/*.deb -t
> > > ${ext_deb_dir}/ 2>/dev/null || :" \
> > > 
> > > This code's goal is to build a deb that should be exported
> > > otherwise we are
> > > continuing using the old package. It seems that breaks things but -
> > > again -
> > > I did fully not understand what and why but simply accepted the
> > > suggestion.
> > > Now, I have reverted back to ln -Pf.
> > > 
> > > Second, those lines are supposed to fail - and obviously fail in
> > > both
> > > directions: import and export - but the following
> > > 
> > > (a) if export fail we will not have the custom packages but we have
> > > them
> > > (b) removing that lines and the code will always fail
> > > 
> > > These mean that ln -Pf complains to fail but make a difference and
> > > make the
> > > difference that we want. However, I might not have understood the
> > > case in
> > > which it fails completely so:
> > > 
> > > mv build/downloads .
> > > rm -rf build
> > > ./build.sh basic-os (DONE)
> > > ./clean.sh isar
> > > ./build.sh (DONE)
> > > ./clean.sh all
> > > ./build.sh (DONE)
> > > 
> > > remove that two lines about ln -Pf that are supposed to do nothing
> > > than fail
> > > 
> > > ./clean.sh all
> > > ./build.sh (DONE with the same building time)
> > > 
> > > rm -rf build/downloads
> > > mv downloads build
> > > 
> > > ./clean.sh isar
> > > ./build.sh complete (FAIL)
> > > 
> > > put back those two command lines with ln -Pf and
> > > 
> > > ./clean.sh all
> > > ./build.sh (DONE)
> > > 
> > > I do not say you are wrong and I see ln -Pf complains on stderr but
> > > nothing
> > > that tin my private fork cannot be solved using a stderr
> > > redirection to
> > > /dev/null
> > > 
> > > However there are some other ways to do this thing:
> > > 
> > > 1. sbuild uses schroot by default, using chroot instead
> > > 2. keep the default in schroot, elsewhere it uses upper folder that
> > > can be
> > > populated in advance
> > > 
> > > I did not make any changes because I did not identified the issue.
> > > 
> > > Best regards, R-
> > 
> > 
> > I'll try to make things clear. Isar supports local builds from 'base-
> > apt' repo 
> > with.
> > 
> > First build user does as usual.
> > 
> > Second build is done with the following changes (at least):
> > 
> > ISAR_USE_CACHED_BASE_REPO = "1"
> > BB_NO_NETWORK = "1"
> > 
> > In this case every .deb we downloaded from Debian mirrors at first
> > build goes 
> > to local 'base-apt' and second build use it, but not remote mirrors.
> > 
> > That's the reason of using this "cp/ln magic" inside of sbuild. While
> > building 
> > some package, multipe dependencies might be downloaded inside of
> > sbuild. And 
> > we need a way to extract them from "internal" apt-cache in sbuild
> > schroot and 
> > put to the package workdir, where deb-dl-dir export will pass them to
> > DL_DIR 
> > later.
> > 
> > When we replace copying to hardlinks, that actually don't work due to
> > different filesystems, we come to the situation, when multiple debian
> > packages 
> > (dependencies of what we are building or some packages that sbuild
> > requires 
> > itself) are not present in DL_DIR. So, second "cached" build from
> > local repo 
> > simply fails.
> > 
> > I see the following possible solutions that allows to save space.
> > 1. Since hardlinks don't work, try to use symlinks.
> 
> 
> If that works, it clearly is the simplest solution.
> 

Yes, approach with symlinks inside sbuild chroot seems to be working, so I'm 
going to use it in patchset v3.

> 
> > 2. Don't use import with sbuild at all, but only export. But this has
> > a 
> > drawback, when all packages we build with sbuild might download the
> > similar 
> > dependencies in parallel.
> 
> 
> A third solution might be to mount the cache into the schroot, similar
> to how we handle the sstate cache today.
> 
> Anyways, I would appreciate if we could merge this series even if it
> still does not fully solve the quadratic behavior. It is very valuable
> and "good enough" even for my biggest layers.
> 
> Felix
> 
> 
> > 
> > 
> 
> 

Re: [PATCH v2 0/3] Improving apt cache

[Roberto A. Foglietta]

[-- Attachment #1: Type: text/plain, Size: 935 bytes --]

On Fri, 20 Jan 2023 at 08:00, Uladzimir Bely <ubely@ilbers.de> wrote:

> In mail from пятница, 20 января 2023 г. 08:08:03 +03 user Moessbauer,
> Felix
> wrote:
> > On Fri, 2023-01-20 at 07:44 +0300, Uladzimir Bely wrote:
> >
> > > In mail from четверг, 19 января 2023 г. 17:52:37 +03 user Roberto A.
>
> > >
> > > I see the following possible solutions that allows to save space.
> > > 1. Since hardlinks don't work, try to use symlinks.
> >
> >
> > If that works, it clearly is the simplest solution.
> >
>
> Yes, approach with symlinks inside sbuild chroot seems to be working, so
> I'm
> going to use it in patchset v3.
>

The symlink or software link does not work for exportation... I presume,
did you check?
Without changing anything it is easy to confuse an old-built package with
the nwer.

Best regards,
-- 
Roberto A. Foglietta
+49.176.274.75.661
+39.349.33.30.697

[-- Attachment #2: Type: text/html, Size: 1489 bytes --]

Re: [PATCH v2 0/3] Improving apt cache

[Uladzimir Bely]

In mail from пятница, 20 января 2023 г. 10:12:31 +03 user Roberto A. Foglietta 
wrote:
> On Fri, 20 Jan 2023 at 08:00, Uladzimir Bely <ubely@ilbers.de> wrote:
> > In mail from пятница, 20 января 2023 г. 08:08:03 +03 user Moessbauer,
> > Felix
> > 
> > wrote:
> > > On Fri, 2023-01-20 at 07:44 +0300, Uladzimir Bely wrote:
> > > > In mail from четверг, 19 января 2023 г. 17:52:37 +03 user Roberto A.
> > > > 
> > > > 
> > > > I see the following possible solutions that allows to save space.
> > > > 1. Since hardlinks don't work, try to use symlinks.
> > > 
> > > If that works, it clearly is the simplest solution.
> > 
> > Yes, approach with symlinks inside sbuild chroot seems to be working, so
> > I'm
> > going to use it in patchset v3.
> 
> The symlink or software link does not work for exportation... I presume,
> did you check?

Of course, if symlinks work inside sbuild chroot, whey won't work outside it 
(dl_dir import/export). So, that's how it's supposed to work:
- dl_dir_export: DL_DIR => hardlink => WORKDIR/rootfs
- sbuild chroot : WORKDIR/rootfs => symlink => /var/cache/apt/*.deb
- sbuild chroot: building the package, new .deb files downloaded
- sbuild chroot: /var/cache/apt/*.deb => copy => WORKIDIR/rootfs
- dl_dir_import: WORKDIR/rootfs => hardlink => DL_DIR
- remove WORKDIR/rootfs

> Without changing anything it is easy to confuse an old-built package with
> the nwer.
> 
> Best regards,