Re: [PATCH v2 3/3] Changes for a faster build using less disk space

[Henning Schild <henning.schild@siemens.com>]

Am Fri,  6 Jan 2023 07:48:09 +0100
schrieb Uladzimir Bely <ubely@ilbers.de>:

> From: "Roberto A. Foglietta" <roberto.foglietta@gmail.com>
> 
> Faster build for larger projects with a lot of debian packages
> 
> This patch leverages few ways to accomplish the task to optimise
> the large buildings.
> 
> Put a CACHEDIR.TAG file in each cache folder in such a way every
> tar that uses the option --exclude-caches could be avoided
> to include debian packages saving a lot of time and disk space.
> 
> Signed-off-by: Roberto A. Foglietta <roberto.foglietta@gmail.com>
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  meta/classes/deb-dl-dir.bbclass                     | 2 +-
>  meta/classes/rootfs.bbclass                         | 3 ++-
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 ++++++-
>  3 files changed, 9 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes/deb-dl-dir.bbclass
> b/meta/classes/deb-dl-dir.bbclass index 7db25251..3afad4e8 100644
> --- a/meta/classes/deb-dl-dir.bbclass
> +++ b/meta/classes/deb-dl-dir.bbclass
> @@ -102,7 +102,7 @@ deb_dl_dir_export() {
>              -maxdepth 1 -type f -iname '*\.deb' |\
>          while read p; do
>              # skip files from a previous export
> -            [ -f "${pc}/${p##*/}" ] && continue
> +            [ -e "${pc}/${p##*/}" ] && continue
>              # can not reuse bitbake function here, this is basically
>              # "repo_contains_package"
>              package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name
> ${p##*/}) diff --git a/meta/classes/rootfs.bbclass
> b/meta/classes/rootfs.bbclass index 786682d9..f274443d 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -308,7 +308,8 @@ rootfs_install_sstate_prepare() {
>      # so we use some mount magic to prevent that
>      mkdir -p ${WORKDIR}/mnt/rootfs
>      sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
> -    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system
> rootfs
> +    lopts="--one-file-system --exclude-caches"
> +    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar $lopts rootfs
>      sudo umount ${WORKDIR}/mnt/rootfs
>      sudo chown $(id -u):$(id -g) rootfs.tar
>  }
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> cb0079ec..44019f95 100644 ---
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -395,6
> +395,10 @@ EOSUDO 
>      # Cleanup apt cache
>      sudo -Es chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean
> +
> +    # Don't include apt cache into sstate cache by marking the
> directory
> +    # with a CACHEDIR.TAG file with a special signature
> +    sudo -Es sh -c "echo Signature: 8a477f597d28d172789f06886806bc55

What is that "special signature" all about? Does anyone later care to
remove the file again, or do we risk it ending up in the final rootfs?

I find this a little weird and would prefer if the calls to tar would
simply exclude this by path and not by magic tool specific file.

It could be made a central function so we have all those tar calls in
one place.

In this patch i see three places ... hard to maintain.

Henning

> > ${ROOTFSDIR}/var/cache/apt/archives/CACHEDIR.TAG" }
>  
>  addtask bootstrap before do_build after do_generate_keyrings
> @@ -405,7 +409,8 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
>  
>  bootstrap_sstate_prepare() {
>      # this runs in SSTATE_BUILDDIR, which will be deleted
> automatically
> -    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar
> --one-file-system $(basename "${ROOTFSDIR}")
> +    lopts="--one-file-system --exclude-caches"
> +    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts
> $(basename "${ROOTFSDIR}") sudo chown $(id -u):$(id -g) bootstrap.tar
>  }
>