* Re: [PATCH 1/1] Reduce disk usage during build by avoiding copies of apt cache
2023-01-04 2:31 [PATCH 1/1] Reduce disk usage during build by avoiding copies of apt cache Felix Moessbauer
@ 2023-01-04 3:36 ` Roberto A. Foglietta
2023-01-04 3:45 ` Roberto A. Foglietta
2023-01-04 6:10 ` Schmidt, Adriaan
2023-01-04 8:15 ` Henning Schild
2 siblings, 1 reply; 8+ messages in thread
From: Roberto A. Foglietta @ 2023-01-04 3:36 UTC (permalink / raw)
To: Felix Moessbauer
Cc: isar-users, florian.bezdeka, ubely, henning.schild, jan.kiszka
On Wed, 4 Jan 2023 at 03:31, Felix Moessbauer
<felix.moessbauer@siemens.com> wrote:
>
> From: "Roberto A. Foglietta" <roberto.foglietta@gmail.com>
>
> This patch leverages few ways to accomplish the task to optimise the large
> buildings. It uses hard physical links instead of copying files which is a
> faster way and saves disk spaces. Put a CACHEDIR.TAG file in each cache folder
> in such a way every tar that uses the option --exclude-caches could be avoided
> to include debian packages saving a lot of time and disk space.
Hi Felix,
The CACHEDIR is not the best but the following that uses --sxclude=<pattern>
+ sudo tar --one-file-system --exclude="var/log/*"
--exclude="var/cache/*" --exclude="var/backups/*" \
+ --exclude="var/tmp/*" --exclude="var/crash/*"
--exclude="var/spool/*" --exclude="var/lib/apt/*" \
+ --exclude-caches --exclude-backups -C ${WORKDIR}/mnt
-cpSf rootfs.tar rootfs
> In addition, I applied some small fixes to fix issues on rebuild.
The fixes are not in this patch.
> -
> + sudo touch "${rootfs}"/var/cache/apt/archives/CACHEDIR.TAG
> find "${rootfs}"/var/cache/apt/archives/ \
> -maxdepth 1 -type f -iname '*\.deb' |\
> while read p; do
> # skip files from a previous export
> - [ -f "${pc}/${p##*/}" ] && continue
> + [ -e "${pc}/${p##*/}" ] && continue
> # can not reuse bitbake function here, this is basically
> # "repo_contains_package"
> package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name ${p##*/})
> if [ -n "$package" ]; then
> cmp --silent "$package" "$p" && continue
> fi
This code is broken and even if it would work with /bin/dash, it is
useless. Moreover this function does not support the transfer of data
to minimize the time of apt-get update.
Another huge change is due to writing the .wic image (folder) directly
on the destination + the patches about schroot
Everything on my NEXT branch is valuable to be integrated
== FASTER AND DISK SAVING ==
* ee44d4e - deb_dl_dir_im/export nolists when USE_CCACHE is not active
* 2ad229b - sstate cache not anymore CACHEDIR.TAG but tar --exclude
* c442591 - changes for a faster build using less disk space, p.6
* 1e69b8f - changes for a faster build using less disk space, p.3 (v2)
* 997809a - changes for a faster build using less disk space, p.2
* fee43dc - changes for a faster build using less disk space, p.1
== BUG FIXES & Co. ==
* 6415204 - image tools ext. class defines ISAR_CROSS_COMPILE, if missing
* 06916be - dpkg class set default value for ISAR_CROSS_COMPILE, if missing
== FLEXIBILITY ==
* 7e72921 - dpkg class sbuild allows extra arguments by vars
== MIGRATION TO SCHROOT ==
* 79e97dd - dpkg base class: use schroot only not buildchroot anymore
* 662383a - events: Cleanup lost schroot sessions if any, v2
* 9750bd5 - imager: Move image types to schroot, v2
* 51b08db - imager: Migrate from buildchroot to schroot, v2
* 0fea6b1 - sbuild: Allow setting custom config paths, v2
I did the cherry picking before answering Felix, so the patchset is
not tested on next yet.
https://github.com/ilbers/isar/compare/next...robang74:isar:next
Best regards, R-
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH 1/1] Reduce disk usage during build by avoiding copies of apt cache
2023-01-04 3:36 ` Roberto A. Foglietta
@ 2023-01-04 3:45 ` Roberto A. Foglietta
0 siblings, 0 replies; 8+ messages in thread
From: Roberto A. Foglietta @ 2023-01-04 3:45 UTC (permalink / raw)
To: Felix Moessbauer
Cc: isar-users, florian.bezdeka, ubely, henning.schild, jan.kiszka
On Wed, 4 Jan 2023 at 04:36, Roberto A. Foglietta
<roberto.foglietta@gmail.com> wrote:
> == BUG FIXES & Co. ==
>
and also this one
* 2ef5460 - image class bugfix: interruption does not break the rebuild
> * 6415204 - image tools ext. class defines ISAR_CROSS_COMPILE, if missing
> * 06916be - dpkg class set default value for ISAR_CROSS_COMPILE, if missing
>
> https://github.com/ilbers/isar/compare/next...robang74:isar:next
>
Best regards, R-
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: [PATCH 1/1] Reduce disk usage during build by avoiding copies of apt cache
2023-01-04 2:31 [PATCH 1/1] Reduce disk usage during build by avoiding copies of apt cache Felix Moessbauer
2023-01-04 3:36 ` Roberto A. Foglietta
@ 2023-01-04 6:10 ` Schmidt, Adriaan
2023-01-04 10:26 ` Roberto A. Foglietta
2023-01-04 8:15 ` Henning Schild
2 siblings, 1 reply; 8+ messages in thread
From: Schmidt, Adriaan @ 2023-01-04 6:10 UTC (permalink / raw)
To: MOESSBAUER, FELIX JONATHAN, isar-users
Cc: Bezdeka, Florian, roberto.foglietta, ubely, Schild, Henning,
Kiszka, Jan, MOESSBAUER, FELIX JONATHAN
Felix Moessbauer, Mittwoch, 4. Januar 2023 03:31:
> Subject: [PATCH 1/1] Reduce disk usage during build by avoiding copies of apt
> cache
>
> From: "Roberto A. Foglietta" <roberto.foglietta@gmail.com>
>
> This patch leverages few ways to accomplish the task to optimise the large
> buildings. It uses hard physical links instead of copying files which is a
> faster way and saves disk spaces. Put a CACHEDIR.TAG file in each cache
> folder
> in such a way every tar that uses the option --exclude-caches could be
> avoided
> to include debian packages saving a lot of time and disk space.
>
> Signed-off-by: Roberto A. Foglietta <roberto.foglietta@gmail.com>
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>
> This is a cleaned up and squashed version of the following commits
> from Robertos ISAR fork:
>
> - 83ce56580c39a2f708f9a3ede73439c82edac1be
> - 02bfafb20a8c43585561b9de306a2e23c5ff50bb
>
> In addition, I applied some small fixes to fix issues on rebuild.
> This is a compromise between the large patch series of Roberto and
> the too minimalistic one of Uladzimir. As the series from Roberto is
> not on this list, I want to send out this patch to have something which
> we can review.
>
> Maybe we can integrate that first to solve 90% of the issue and then
> continue improving it.
>
> Best regards,
> Felix
>
> meta/classes/deb-dl-dir.bbclass | 11 ++++++-----
> meta/classes/dpkg.bbclass | 6 +++---
> meta/classes/rootfs.bbclass | 8 ++++----
> meta/classes/sdk.bbclass | 2 +-
> meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 3 ++-
> 5 files changed, 16 insertions(+), 14 deletions(-)
>
> diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-
> dir.bbclass
> index 3b1517d..3a87390 100644
> --- a/meta/classes/deb-dl-dir.bbclass
> +++ b/meta/classes/deb-dl-dir.bbclass
> @@ -84,9 +84,9 @@ deb_dl_dir_import() {
> flock -s "${pc}".lock -c '
> set -e
> printenv | grep -q BB_VERBOSE_LOGS && set -x
> -
> + sudo touch "${rootfs}"/var/cache/apt/archives/CACHEDIR.TAG
> sudo find "${pc}" -type f -iname "*\.deb" -exec \
> - cp -n --no-preserve=owner -t "${rootfs}"/var/cache/apt/archives/
> {} +
> + ln -Pf -t "${rootfs}"/var/cache/apt/archives/ {} + 2>/dev/null
> || :
> '
> }
>
> @@ -97,20 +97,21 @@ deb_dl_dir_export() {
> flock "${pc}".lock -c '
> set -e
> printenv | grep -q BB_VERBOSE_LOGS && set -x
> -
> + sudo touch "${rootfs}"/var/cache/apt/archives/CACHEDIR.TAG
> find "${rootfs}"/var/cache/apt/archives/ \
> -maxdepth 1 -type f -iname '*\.deb' |\
> while read p; do
> # skip files from a previous export
> - [ -f "${pc}/${p##*/}" ] && continue
> + [ -e "${pc}/${p##*/}" ] && continue
> # can not reuse bitbake function here, this is basically
> # "repo_contains_package"
> package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name ${p##*/})
> if [ -n "$package" ]; then
> cmp --silent "$package" "$p" && continue
> fi
> - sudo cp -n "${p}" "${pc}"
> + sudo ln -P "${p}" "${pc}" 2>/dev/null || :
> done
> + sudo touch "${pc}/CACHEDIR.TAG"
> sudo chown -R $(id -u):$(id -g) "${pc}"
> '
> }
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index 7822b14..166ec16 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -108,10 +108,10 @@ dpkg_runbuild() {
> --chroot-setup-commands="echo \"Package: *\nPin: release
> n=${DEBDISTRONAME}\nPin-Priority: 1000\" > /etc/apt/preferences.d/isar-apt" \
> --chroot-setup-commands="echo \"APT::Get::allow-downgrades 1;\" >
> /etc/apt/apt.conf.d/50isar-apt" \
> --chroot-setup-commands="rm -f /var/log/dpkg.log" \
> - --chroot-setup-commands="cp -n --no-preserve=owner
> ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
> + --chroot-setup-commands="ln -Pf ${ext_deb_dir}/*.deb -t ${deb_dir}/
> 2>/dev/null || :" \
> --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-
> main-dummy_*.deb" \
> - --finished-build-commands="cp -n --no-preserve=owner
> ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
> - --finished-build-commands="cp /var/log/dpkg.log
> ${ext_root}/dpkg_partial.log" \
> + --finished-build-commands="ln -P ${deb_dir}/*.deb -t ${ext_deb_dir}/
> 2>/dev/null || :" \
> + --finished-build-commands="cp /var/log/dpkg.log
> ${ext_root}/dpkg_partial.log; touch ${ext_deb_dir}/CACHEDIR.TAG" \
> --debbuildopts="--source-option=-I" \
> --build-dir=${WORKDIR} --dist="isar" ${DSC_FILE}
>
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index 786682d..147f517 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -78,7 +78,7 @@ BOOTSTRAP_SRC_${ROOTFS_ARCH} =
> "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTF
>
> rootfs_prepare[weight] = "25"
> rootfs_prepare(){
> - sudo cp -Trpfx '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}'
> + sudo cp -Trpfx --reflink=auto '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}'
> }
>
> ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt"
> @@ -115,7 +115,6 @@ rootfs_configure_apt() {
> EOSUDO
> }
>
> -
> ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update"
> rootfs_install_pkgs_update[weight] = "5"
> rootfs_install_pkgs_update[isar-apt-lock] = "acquire-before"
> @@ -218,7 +217,7 @@ cache_deb_src() {
> # Note: ISAR updates the apt state information(apt-get update) only once
> during bootstrap and
> # relies on that through out the build. Copy that state information
> instead of apt-get update
> # which generates a new state from upstream.
> - sudo cp -Trpn "${BOOTSTRAP_SRC}/var/lib/apt/lists/"
> "${ROOTFSDIR}/var/lib/apt/lists/"
> + sudo cp -Trpn --reflink=auto "${BOOTSTRAP_SRC}/var/lib/apt/lists/"
> "${ROOTFSDIR}/var/lib/apt/lists/"
>
> deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-
> ${BASE_DISTRO_CODENAME}
> debsrc_download ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-
> ${BASE_DISTRO_CODENAME}
> @@ -308,7 +307,8 @@ rootfs_install_sstate_prepare() {
> # so we use some mount magic to prevent that
> mkdir -p ${WORKDIR}/mnt/rootfs
> sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
> - sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system rootfs
> + lopts="--one-file-system --exclude-caches --exclude=var/cache/apt/archives"
(several locations):
With the creation of CACHEDIR.TAG, and --exclude-caches, isn't the --exclude=
of that same path redundant?
Adriaan
> + sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar $lopts rootfs
> sudo umount ${WORKDIR}/mnt/rootfs
> sudo chown $(id -u):$(id -g) rootfs.tar
> }
> diff --git a/meta/classes/sdk.bbclass b/meta/classes/sdk.bbclass
> index 79e01a1..3feada2 100644
> --- a/meta/classes/sdk.bbclass
> +++ b/meta/classes/sdk.bbclass
> @@ -82,7 +82,7 @@ do_rootfs_install[vardeps] += "${SDKROOTFSVARDEPS}"
> ROOTFS_CONFIGURE_COMMAND_append_class-sdk = "
> ${@'rootfs_configure_isar_apt_dir' if d.getVar('SDK_INCLUDE_ISAR_APT') == '1'
> else ''}"
> rootfs_configure_isar_apt_dir() {
> # Copy isar-apt instead of mounting:
> - sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${ROOTFSDIR}/isar-apt
> + sudo cp -Trpfx --reflink=auto ${REPO_ISAR_DIR}/${DISTRO}
> ${ROOTFSDIR}/isar-apt
> }
>
> ROOTFS_POSTPROCESS_COMMAND_prepend_class-sdk = "sdkchroot_configscript "
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index b9ae16c..d53db4d 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -418,7 +418,8 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
>
> bootstrap_sstate_prepare() {
> # this runs in SSTATE_BUILDDIR, which will be deleted automatically
> - sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar --one-file-
> system $(basename "${ROOTFSDIR}")
> + lopts="--one-file-system --exclude-caches --
> exclude=var/cache/apt/archives"
> + sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts
> $(basename "${ROOTFSDIR}")
> sudo chown $(id -u):$(id -g) bootstrap.tar
> }
>
> --
> 2.34.1
>
> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-users/20230104023128.374390-1-
> felix.moessbauer%40siemens.com.
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH 1/1] Reduce disk usage during build by avoiding copies of apt cache
2023-01-04 6:10 ` Schmidt, Adriaan
@ 2023-01-04 10:26 ` Roberto A. Foglietta
2023-01-04 11:00 ` Bezdeka, Florian
0 siblings, 1 reply; 8+ messages in thread
From: Roberto A. Foglietta @ 2023-01-04 10:26 UTC (permalink / raw)
To: Schmidt, Adriaan
Cc: MOESSBAUER, FELIX JONATHAN, isar-users, Bezdeka, Florian, ubely,
Schild, Henning, Kiszka, Jan
On Wed, 4 Jan 2023 at 07:10, Schmidt, Adriaan
<adriaan.schmidt@siemens.com> wrote:
> > @@ -308,7 +307,8 @@ rootfs_install_sstate_prepare() {
> > # so we use some mount magic to prevent that
> > mkdir -p ${WORKDIR}/mnt/rootfs
> > sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
> > - sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system rootfs
> > + lopts="--one-file-system --exclude-caches --exclude=var/cache/apt/archives"
>
> (several locations):
> With the creation of CACHEDIR.TAG, and --exclude-caches, isn't the --exclude=
> of that same path redundant?
Hi Adriaan,
if you look at this, everything has been achieved using exclude option
sudo tar --one-file-system --exclude="var/log/*" \
--exclude="var/cache/*" --exclude="var/backups/*" \
--exclude="var/tmp/*" --exclude="var/crash/*" \
--exclude="var/spool/*" --exclude="var/lib/apt/*" \
--exclude-caches --exclude-backups -C ${WORKDIR}/mnt \
-cpSf rootfs.tar rootfs
however exclude backups and caches are still in place because the user
might define further ignoring locations using different ways.
P.S.: I have updated and rebased the cherry picking on my 'next'
branch and it works pretty well despite is just a subset of all the
patches I am experimenting.
Best regards, R-
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH 1/1] Reduce disk usage during build by avoiding copies of apt cache
2023-01-04 10:26 ` Roberto A. Foglietta
@ 2023-01-04 11:00 ` Bezdeka, Florian
2023-01-04 11:15 ` Roberto A. Foglietta
0 siblings, 1 reply; 8+ messages in thread
From: Bezdeka, Florian @ 2023-01-04 11:00 UTC (permalink / raw)
To: roberto.foglietta, Schmidt, Adriaan
Cc: ubely, isar-users, Kiszka, Jan, MOESSBAUER, FELIX JONATHAN,
Schild, Henning
On Wed, 2023-01-04 at 11:26 +0100, Roberto A. Foglietta wrote:
> On Wed, 4 Jan 2023 at 07:10, Schmidt, Adriaan
> <adriaan.schmidt@siemens.com> wrote:
>
> > > @@ -308,7 +307,8 @@ rootfs_install_sstate_prepare() {
> > > # so we use some mount magic to prevent that
> > > mkdir -p ${WORKDIR}/mnt/rootfs
> > > sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
> > > - sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system rootfs
> > > + lopts="--one-file-system --exclude-caches --exclude=var/cache/apt/archives"
> >
> > (several locations):
> > With the creation of CACHEDIR.TAG, and --exclude-caches, isn't the --exclude=
> > of that same path redundant?
>
> Hi Adriaan,
>
> if you look at this, everything has been achieved using exclude option
>
> sudo tar --one-file-system --exclude="var/log/*" \
> --exclude="var/cache/*" --exclude="var/backups/*" \
> --exclude="var/tmp/*" --exclude="var/crash/*" \
> --exclude="var/spool/*" --exclude="var/lib/apt/*" \
> --exclude-caches --exclude-backups -C ${WORKDIR}/mnt \
> -cpSf rootfs.tar rootfs
>
> however exclude backups and caches are still in place because the user
> might define further ignoring locations using different ways.
>
> P.S.: I have updated and rebased the cherry picking on my 'next'
> branch and it works pretty well despite is just a subset of all the
> patches I am experimenting.
Roberto,
nice to see you working on that, but: if you want us to look at
something (especially on code/patches) please send them - with the
right version tag - to the list. We can't go and check your repo every
time and follow all the activity (especially rebasing) that is going on
there.
Someone already mentioned that: We need to look at patches that are
based on current Isar next branch. Everything else is basically
unusable for us.
Best regards,
Florian
>
> Best regards, R-
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH 1/1] Reduce disk usage during build by avoiding copies of apt cache
2023-01-04 11:00 ` Bezdeka, Florian
@ 2023-01-04 11:15 ` Roberto A. Foglietta
0 siblings, 0 replies; 8+ messages in thread
From: Roberto A. Foglietta @ 2023-01-04 11:15 UTC (permalink / raw)
To: Bezdeka, Florian
Cc: Schmidt, Adriaan, ubely, isar-users, Kiszka, Jan, MOESSBAUER,
FELIX JONATHAN, Schild, Henning
On Wed, 4 Jan 2023 at 12:00, Bezdeka, Florian
<florian.bezdeka@siemens.com> wrote:
>
> On Wed, 2023-01-04 at 11:26 +0100, Roberto A. Foglietta wrote:
> > On Wed, 4 Jan 2023 at 07:10, Schmidt, Adriaan
> > <adriaan.schmidt@siemens.com> wrote:
> >
> > > > @@ -308,7 +307,8 @@ rootfs_install_sstate_prepare() {
> > > > # so we use some mount magic to prevent that
> > > > mkdir -p ${WORKDIR}/mnt/rootfs
> > > > sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
> > > > - sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system rootfs
> > > > + lopts="--one-file-system --exclude-caches --exclude=var/cache/apt/archives"
> > >
> > > (several locations):
> > > With the creation of CACHEDIR.TAG, and --exclude-caches, isn't the --exclude=
> > > of that same path redundant?
> >
> > Hi Adriaan,
> >
> > if you look at this, everything has been achieved using exclude option
> >
> > sudo tar --one-file-system --exclude="var/log/*" \
> > --exclude="var/cache/*" --exclude="var/backups/*" \
> > --exclude="var/tmp/*" --exclude="var/crash/*" \
> > --exclude="var/spool/*" --exclude="var/lib/apt/*" \
> > --exclude-caches --exclude-backups -C ${WORKDIR}/mnt \
> > -cpSf rootfs.tar rootfs
> >
> > however exclude backups and caches are still in place because the user
> > might define further ignoring locations using different ways.
> >
> > P.S.: I have updated and rebased the cherry picking on my 'next'
> > branch and it works pretty well despite is just a subset of all the
> > patches I am experimenting.
>
> Roberto,
>
> nice to see you working on that, but: if you want us to look at
> something (especially on code/patches) please send them - with the
> right version tag - to the list. We can't go and check your repo every
> time and follow all the activity (especially rebasing) that is going on
> there.
>
> Someone already mentioned that: We need to look at patches that are
> based on current Isar next branch. Everything else is basically
> unusable for us.
>
Hi Florian,
I am not going to push my will over you. I like that you will find
your own independent solution whatever the time it takes. After all,
my actual contribution is just for free aka I am just a hobbyist.
The 1st rule of a hobbyist is: if it works on my PC then, it works for
all my world! Sometimes also referred to as Robinson Crusoe 1st rule.
LOL
Just for sake of completeness: the basic-os was built in 8m33s, in the
current 'next' branch it takes 3m6s. It was requiring 43GB of disk
space but now less than 4GB.
The complete build was taking 20m13s but now is around 12m, sometimes
above and sometimes below. It was using 53GB of disk but now it is
just 29GB and I am still working on it to improve it.
Useless is such a big word, after all... ROTFL
Best regards, R-
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/1] Reduce disk usage during build by avoiding copies of apt cache
2023-01-04 2:31 [PATCH 1/1] Reduce disk usage during build by avoiding copies of apt cache Felix Moessbauer
2023-01-04 3:36 ` Roberto A. Foglietta
2023-01-04 6:10 ` Schmidt, Adriaan
@ 2023-01-04 8:15 ` Henning Schild
2 siblings, 0 replies; 8+ messages in thread
From: Henning Schild @ 2023-01-04 8:15 UTC (permalink / raw)
To: Felix Moessbauer
Cc: isar-users, florian.bezdeka, roberto.foglietta, ubely, jan.kiszka
Am Wed, 4 Jan 2023 02:31:28 +0000
schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:
> From: "Roberto A. Foglietta" <roberto.foglietta@gmail.com>
>
> This patch leverages few ways to accomplish the task to optimise the
> large buildings.
like the empire state building?
I find this patch hard to review, it seems to do multiple things at
once. Things that others have understood already so i will not review.
But if it needs to be send again, maybe we can split it up. If any of
that causes issues it will be hard to blame and revert. And the authors
might not understand it anymore in ... say a year or less.
Henning
> It uses hard physical links instead of copying files
> which is a faster way and saves disk spaces. Put a CACHEDIR.TAG file
> in each cache folder in such a way every tar that uses the option
> --exclude-caches could be avoided to include debian packages saving a
> lot of time and disk space.
>
> Signed-off-by: Roberto A. Foglietta <roberto.foglietta@gmail.com>
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>
> This is a cleaned up and squashed version of the following commits
> from Robertos ISAR fork:
>
> - 83ce56580c39a2f708f9a3ede73439c82edac1be
> - 02bfafb20a8c43585561b9de306a2e23c5ff50bb
>
> In addition, I applied some small fixes to fix issues on rebuild.
> This is a compromise between the large patch series of Roberto and
> the too minimalistic one of Uladzimir. As the series from Roberto is
> not on this list, I want to send out this patch to have something
> which we can review.
>
> Maybe we can integrate that first to solve 90% of the issue and then
> continue improving it.
>
> Best regards,
> Felix
>
> meta/classes/deb-dl-dir.bbclass | 11 ++++++-----
> meta/classes/dpkg.bbclass | 6 +++---
> meta/classes/rootfs.bbclass | 8 ++++----
> meta/classes/sdk.bbclass | 2 +-
> meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 3 ++-
> 5 files changed, 16 insertions(+), 14 deletions(-)
>
> diff --git a/meta/classes/deb-dl-dir.bbclass
> b/meta/classes/deb-dl-dir.bbclass index 3b1517d..3a87390 100644
> --- a/meta/classes/deb-dl-dir.bbclass
> +++ b/meta/classes/deb-dl-dir.bbclass
> @@ -84,9 +84,9 @@ deb_dl_dir_import() {
> flock -s "${pc}".lock -c '
> set -e
> printenv | grep -q BB_VERBOSE_LOGS && set -x
> -
> + sudo touch "${rootfs}"/var/cache/apt/archives/CACHEDIR.TAG
> sudo find "${pc}" -type f -iname "*\.deb" -exec \
> - cp -n --no-preserve=owner -t
> "${rootfs}"/var/cache/apt/archives/ {} +
> + ln -Pf -t "${rootfs}"/var/cache/apt/archives/ {} +
> 2>/dev/null || : '
> }
>
> @@ -97,20 +97,21 @@ deb_dl_dir_export() {
> flock "${pc}".lock -c '
> set -e
> printenv | grep -q BB_VERBOSE_LOGS && set -x
> -
> + sudo touch "${rootfs}"/var/cache/apt/archives/CACHEDIR.TAG
> find "${rootfs}"/var/cache/apt/archives/ \
> -maxdepth 1 -type f -iname '*\.deb' |\
> while read p; do
> # skip files from a previous export
> - [ -f "${pc}/${p##*/}" ] && continue
> + [ -e "${pc}/${p##*/}" ] && continue
> # can not reuse bitbake function here, this is basically
> # "repo_contains_package"
> package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name
> ${p##*/}) if [ -n "$package" ]; then
> cmp --silent "$package" "$p" && continue
> fi
> - sudo cp -n "${p}" "${pc}"
> + sudo ln -P "${p}" "${pc}" 2>/dev/null || :
> done
> + sudo touch "${pc}/CACHEDIR.TAG"
> sudo chown -R $(id -u):$(id -g) "${pc}"
> '
> }
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index 7822b14..166ec16 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -108,10 +108,10 @@ dpkg_runbuild() {
> --chroot-setup-commands="echo \"Package: *\nPin: release
> n=${DEBDISTRONAME}\nPin-Priority: 1000\" >
> /etc/apt/preferences.d/isar-apt" \ --chroot-setup-commands="echo
> \"APT::Get::allow-downgrades 1;\" > /etc/apt/apt.conf.d/50isar-apt" \
> --chroot-setup-commands="rm -f /var/log/dpkg.log" \
> - --chroot-setup-commands="cp -n --no-preserve=owner
> ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
> + --chroot-setup-commands="ln -Pf ${ext_deb_dir}/*.deb -t
> ${deb_dir}/ 2>/dev/null || :" \ --finished-build-commands="rm -f
> ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
> - --finished-build-commands="cp -n --no-preserve=owner
> ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
> - --finished-build-commands="cp /var/log/dpkg.log
> ${ext_root}/dpkg_partial.log" \
> + --finished-build-commands="ln -P ${deb_dir}/*.deb -t
> ${ext_deb_dir}/ 2>/dev/null || :" \
> + --finished-build-commands="cp /var/log/dpkg.log
> ${ext_root}/dpkg_partial.log; touch ${ext_deb_dir}/CACHEDIR.TAG" \
> --debbuildopts="--source-option=-I" \ --build-dir=${WORKDIR}
> --dist="isar" ${DSC_FILE}
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index 786682d..147f517 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -78,7 +78,7 @@ BOOTSTRAP_SRC_${ROOTFS_ARCH} =
> "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTF
> rootfs_prepare[weight] = "25"
> rootfs_prepare(){
> - sudo cp -Trpfx '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}'
> + sudo cp -Trpfx --reflink=auto '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}'
> }
>
> ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt"
> @@ -115,7 +115,6 @@ rootfs_configure_apt() {
> EOSUDO
> }
>
> -
> ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update"
> rootfs_install_pkgs_update[weight] = "5"
> rootfs_install_pkgs_update[isar-apt-lock] = "acquire-before"
> @@ -218,7 +217,7 @@ cache_deb_src() {
> # Note: ISAR updates the apt state information(apt-get update)
> only once during bootstrap and # relies on that through out the
> build. Copy that state information instead of apt-get update # which
> generates a new state from upstream.
> - sudo cp -Trpn "${BOOTSTRAP_SRC}/var/lib/apt/lists/"
> "${ROOTFSDIR}/var/lib/apt/lists/"
> + sudo cp -Trpn --reflink=auto
> "${BOOTSTRAP_SRC}/var/lib/apt/lists/"
> "${ROOTFSDIR}/var/lib/apt/lists/" deb_dl_dir_import ${ROOTFSDIR}
> ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME} debsrc_download
> ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME} @@ -308,7
> +307,8 @@ rootfs_install_sstate_prepare() { # so we use some mount
> magic to prevent that mkdir -p ${WORKDIR}/mnt/rootfs
> sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
> - sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system
> rootfs
> + lopts="--one-file-system --exclude-caches
> --exclude=var/cache/apt/archives"
> + sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar $lopts rootfs
> sudo umount ${WORKDIR}/mnt/rootfs
> sudo chown $(id -u):$(id -g) rootfs.tar
> }
> diff --git a/meta/classes/sdk.bbclass b/meta/classes/sdk.bbclass
> index 79e01a1..3feada2 100644
> --- a/meta/classes/sdk.bbclass
> +++ b/meta/classes/sdk.bbclass
> @@ -82,7 +82,7 @@ do_rootfs_install[vardeps] += "${SDKROOTFSVARDEPS}"
> ROOTFS_CONFIGURE_COMMAND_append_class-sdk = "
> ${@'rootfs_configure_isar_apt_dir' if
> d.getVar('SDK_INCLUDE_ISAR_APT') == '1' else ''}"
> rootfs_configure_isar_apt_dir() { # Copy isar-apt instead of mounting:
> - sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${ROOTFSDIR}/isar-apt
> + sudo cp -Trpfx --reflink=auto ${REPO_ISAR_DIR}/${DISTRO}
> ${ROOTFSDIR}/isar-apt }
>
> ROOTFS_POSTPROCESS_COMMAND_prepend_class-sdk =
> "sdkchroot_configscript " diff --git
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> b9ae16c..d53db4d 100644 ---
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -418,7
> +418,8 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
> bootstrap_sstate_prepare() { # this runs in SSTATE_BUILDDIR, which
> will be deleted automatically
> - sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar
> --one-file-system $(basename "${ROOTFSDIR}")
> + lopts="--one-file-system --exclude-caches
> --exclude=var/cache/apt/archives"
> + sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts
> $(basename "${ROOTFSDIR}") sudo chown $(id -u):$(id -g) bootstrap.tar
> }
>
^ permalink raw reply [flat|nested] 8+ messages in thread