Re: [PATCH 1/1] Reduce disk usage during build by avoiding copies of apt cache

["Roberto A. Foglietta" <roberto.foglietta@gmail.com>]

On Wed, 4 Jan 2023 at 03:31, Felix Moessbauer
<felix.moessbauer@siemens.com> wrote:
>
> From: "Roberto A. Foglietta" <roberto.foglietta@gmail.com>
>
> This patch leverages few ways to accomplish the task to optimise the large
> buildings. It uses hard physical links instead of copying files which is a
> faster way and saves disk spaces. Put a CACHEDIR.TAG file in each cache folder
> in such a way every tar that uses the option --exclude-caches could be avoided
> to include debian packages saving a lot of time and disk space.

Hi Felix,

The CACHEDIR is not the best but the following that uses --sxclude=<pattern>

+        sudo tar --one-file-system --exclude="var/log/*"
--exclude="var/cache/*" --exclude="var/backups/*" \
+            --exclude="var/tmp/*" --exclude="var/crash/*"
--exclude="var/spool/*" --exclude="var/lib/apt/*" \
+            --exclude-caches --exclude-backups -C ${WORKDIR}/mnt
-cpSf rootfs.tar rootfs

> In addition, I applied some small fixes to fix issues on rebuild.

The fixes are not in this patch.

> -
> +        sudo touch "${rootfs}"/var/cache/apt/archives/CACHEDIR.TAG
>          find "${rootfs}"/var/cache/apt/archives/ \
>              -maxdepth 1 -type f -iname '*\.deb' |\
>          while read p; do
>              # skip files from a previous export
> -            [ -f "${pc}/${p##*/}" ] && continue
> +            [ -e "${pc}/${p##*/}" ] && continue
>              # can not reuse bitbake function here, this is basically
>              # "repo_contains_package"
>              package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name ${p##*/})
>              if [ -n "$package" ]; then
>                  cmp --silent "$package" "$p" && continue
>              fi

This code is broken and even if it would work with /bin/dash, it is
useless. Moreover this function does not support the transfer of data
to minimize the time of apt-get update.

Another huge change is due to writing the .wic image (folder) directly
on the destination + the patches about schroot

Everything on my NEXT branch is valuable to be integrated

== FASTER AND DISK SAVING ==

* ee44d4e - deb_dl_dir_im/export nolists when USE_CCACHE is not active
* 2ad229b - sstate cache not anymore CACHEDIR.TAG but tar --exclude
* c442591 - changes for a faster build using less disk space, p.6
* 1e69b8f - changes for a faster build using less disk space, p.3 (v2)
* 997809a - changes for a faster build using less disk space, p.2
* fee43dc - changes for a faster build using less disk space, p.1

== BUG FIXES & Co. ==

* 6415204 - image tools ext. class defines ISAR_CROSS_COMPILE, if missing
* 06916be - dpkg class set default value for ISAR_CROSS_COMPILE, if missing

== FLEXIBILITY ==

* 7e72921 - dpkg class sbuild allows extra arguments by vars

== MIGRATION TO SCHROOT ==

* 79e97dd - dpkg base class: use schroot only not buildchroot anymore
* 662383a - events: Cleanup lost schroot sessions if any, v2
* 9750bd5 - imager: Move image types to schroot, v2
* 51b08db - imager: Migrate from buildchroot to schroot, v2
* 0fea6b1 - sbuild: Allow setting custom config paths, v2

I did the cherry picking before answering Felix, so the patchset is
not tested on next yet.

https://github.com/ilbers/isar/compare/next...robang74:isar:next

Best regards, R-