* [PATCH v2 0/2] Support for ccache
@ 2021-09-29 9:58 Uladzimir Bely
2021-09-29 9:58 ` [PATCH v2 1/2] meta: Support for ccache for custom packages in buildchroot Uladzimir Bely
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Uladzimir Bely @ 2021-09-29 9:58 UTC (permalink / raw)
To: isar-users
Changes since v1:
- Simplified used variables
- Rebased on latest `next`
Some custom user packages built from sources may be written in C/C++.
Using ccache will help decrease build time in case they are rebuilt.
For example, building `mc:stm32mp15x-buster:linux-mainline` during the
test took 28 minutes at second build with ccache enabled in comparison
with 115 minutes of first build.
Uladzimir Bely (2):
meta: Support for ccache for custom packages in buildchroot
doc: Add section about ccache usage
doc/user_manual.md | 17 ++++++++++++++
meta-isar/conf/local.conf.sample | 5 ++++
meta/classes/buildchroot.bbclass | 6 +++++
meta/classes/ccache.bbclass | 23 +++++++++++++++++++
meta/classes/dpkg.bbclass | 3 ++-
.../buildchroot/buildchroot.inc | 1 +
.../buildchroot/files/build.sh | 6 +++++
.../buildchroot/files/common.sh | 1 +
8 files changed, 61 insertions(+), 1 deletion(-)
create mode 100644 meta/classes/ccache.bbclass
--
2.20.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH v2 1/2] meta: Support for ccache for custom packages in buildchroot
2021-09-29 9:58 [PATCH v2 0/2] Support for ccache Uladzimir Bely
@ 2021-09-29 9:58 ` Uladzimir Bely
2021-09-29 9:58 ` [PATCH v2 2/2] doc: Add section about ccache usage Uladzimir Bely
2021-10-28 15:41 ` [PATCH v2 0/2] Support for ccache Jan Kiszka
2 siblings, 0 replies; 5+ messages in thread
From: Uladzimir Bely @ 2021-09-29 9:58 UTC (permalink / raw)
To: isar-users
Add `INHERIT += "ccache"` to local.conf or package recipe
to globally activate the ccache functionality.
Add `CCACHE_DISABLE = '1'` to the recipe to disable ccache for it.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta-isar/conf/local.conf.sample | 5 ++++
meta/classes/buildchroot.bbclass | 6 +++++
meta/classes/ccache.bbclass | 23 +++++++++++++++++++
meta/classes/dpkg.bbclass | 3 ++-
.../buildchroot/buildchroot.inc | 1 +
.../buildchroot/files/build.sh | 6 +++++
.../buildchroot/files/common.sh | 1 +
7 files changed, 44 insertions(+), 1 deletion(-)
create mode 100644 meta/classes/ccache.bbclass
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 88a8175..f180f5a 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -224,3 +224,8 @@ USER_isar[flags] += "clear-text-password"
# Uncomment the below line to debug WIC.
# WIC_CREATE_EXTRA_ARGS += "-D"
+
+# Uncomment this to use ccache for custom packages
+#INHERIT += "ccache"
+# Uncomment and set own top level ccache directory to share between builds
+#CCACHE_TOP_DIR ?= "${TMPDIR}/ccache"
diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index e9eb9af..274ebdc 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -32,6 +32,12 @@ buildchroot_do_mounts() {
mount --bind '${REPO_ISAR_DIR}/${DISTRO}' '${BUILDCHROOT_DIR}/isar-apt'
mountpoint -q '${BUILDCHROOT_DIR}/downloads' ||
mount --bind '${DL_DIR}' '${BUILDCHROOT_DIR}/downloads'
+ if [ "${USE_CCACHE}" = "1" ]
+ then
+ mkdir -p '${BUILDCHROOT_DIR}/ccache'
+ mountpoint -q '${BUILDCHROOT_DIR}/ccache' ||
+ mount --bind '${CCACHE_DIR}' '${BUILDCHROOT_DIR}/ccache'
+ fi
mountpoint -q '${BUILDCHROOT_DIR}/dev' ||
mount --rbind /dev '${BUILDCHROOT_DIR}/dev'
mount --make-rslave '${BUILDCHROOT_DIR}/dev'
diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
new file mode 100644
index 0000000..0053dfa
--- /dev/null
+++ b/meta/classes/ccache.bbclass
@@ -0,0 +1,23 @@
+#
+# Usage:
+# - Enable ccache
+# Add the following line to a conffile such as conf/local.conf:
+# INHERIT += "ccache"
+#
+# - Disable ccache for a recipe
+# Add the following line to the recipe if it can't be built with ccache:
+# USE_CCACHE = "0"
+#
+# - Share ccache files between different builds
+# Set CCACHE_TOP_DIR to a shared dir
+# CCACHE_TOP_DIR = /path/to/shared_ccache/
+
+CCACHE_TOP_DIR ?= "${TMPDIR}/ccache"
+CCACHE_DIR ?= "${CCACHE_TOP_DIR}/${DISTRO}-${DISTRO_ARCH}"
+
+USE_CCACHE ?= "1"
+
+do_init_ccache[dirs] = "${CCACHE_DIR}"
+do_init_ccache() {
+}
+addtask do_init_ccache before do_rootfs
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 4e7c2f7..4269364 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -4,6 +4,7 @@
inherit dpkg-base
PACKAGE_ARCH ?= "${DISTRO_ARCH}"
+USE_CCACHE ??= "0"
# Install build dependencies for package
do_install_builddeps() {
@@ -33,5 +34,5 @@ dpkg_runbuild() {
E="${@ isar_export_proxies(d)}"
export PARALLEL_MAKE="${PARALLEL_MAKE}"
sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
- /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH}
+ /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH} ${USE_CCACHE}
}
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index 31524a1..726c7bb 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -32,6 +32,7 @@ BUILDCHROOT_PREINSTALL_COMMON = " \
make \
debhelper \
autotools-dev \
+ ccache \
dpkg \
locales \
docbook-to-man \
diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
index 101581d..b642f42 100644
--- a/meta/recipes-devtools/buildchroot/files/build.sh
+++ b/meta/recipes-devtools/buildchroot/files/build.sh
@@ -14,4 +14,10 @@ for i in configure aclocal.m4 Makefile.am Makefile.in; do
fi
done
+if [ "$use_ccache" == "1" ]
+then
+ export CCACHE_DIR=/ccache
+ export PATH=/usr/lib/ccache:$PATH
+fi
+
${GBP_PREFIX}dpkg-buildpackage -a$target_arch -d --source-option=-I
diff --git a/meta/recipes-devtools/buildchroot/files/common.sh b/meta/recipes-devtools/buildchroot/files/common.sh
index 0063a38..7085f44 100644
--- a/meta/recipes-devtools/buildchroot/files/common.sh
+++ b/meta/recipes-devtools/buildchroot/files/common.sh
@@ -17,6 +17,7 @@ fi
# Create human-readable names
target_arch=$2
+use_ccache=$3
set_arch="--host-arch $target_arch"
--
2.20.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH v2 2/2] doc: Add section about ccache usage
2021-09-29 9:58 [PATCH v2 0/2] Support for ccache Uladzimir Bely
2021-09-29 9:58 ` [PATCH v2 1/2] meta: Support for ccache for custom packages in buildchroot Uladzimir Bely
@ 2021-09-29 9:58 ` Uladzimir Bely
2021-10-28 15:41 ` [PATCH v2 0/2] Support for ccache Jan Kiszka
2 siblings, 0 replies; 5+ messages in thread
From: Uladzimir Bely @ 2021-09-29 9:58 UTC (permalink / raw)
To: isar-users
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
doc/user_manual.md | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/doc/user_manual.md b/doc/user_manual.md
index 01fe701..a553eeb 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -18,6 +18,7 @@ Copyright (C) 2016-2019, ilbers GmbH
- [Add a New Image Type](#add-a-new-image-type)
- [Add a Custom Application](#add-a-custom-application)
- [Enabling Cross-compilation](#isar-cross-compilation)
+ - [Using ccache for custom packages](#using-ccache-for-custom-packages)
- [Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)
- [Create a containerized Isar SDK root filesystem](#create-a-containerized-isar-sdk-root-filesystem)
- [Creation of local apt repo caching upstream Debian packages](#creation-of-local-apt-repo-caching-upstream-debian-packages)
@@ -912,6 +913,22 @@ package build. To invoke it, just call
`bitbake mc:${MACHINE}-${DISTRO}:<package_name> -c devshell`.
+## Using ccache for custom packages
+
+While base system is created from binary debian repositories, some user
+packages are built from sources. It's possible to reduce build time
+for such packages by enabling ccache.
+
+To enable global ccache functionality, `INHERIT += ccache` can be added
+to `local.conf`. If some package requires ccache to be always disabled,
+`USE_CCACHE = "0"` can be used in the recipe.
+
+By default, ccache directory is created inside `TMPDIR`, but it can be
+adjusted by `CCACHE_TOP_DIR` variable in `local.conf`. Ccache directory
+`CCACHE_DIR` default value is `"${CCACHE_TOP_DIR}/${DISTRO}-${DISTRO_ARCH}"`,
+that means caches for different distros and architectures are not overlapped.
+
+
## Create an ISAR SDK root filesystem
### Motivation
--
2.20.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v2 0/2] Support for ccache
2021-09-29 9:58 [PATCH v2 0/2] Support for ccache Uladzimir Bely
2021-09-29 9:58 ` [PATCH v2 1/2] meta: Support for ccache for custom packages in buildchroot Uladzimir Bely
2021-09-29 9:58 ` [PATCH v2 2/2] doc: Add section about ccache usage Uladzimir Bely
@ 2021-10-28 15:41 ` Jan Kiszka
2021-10-28 16:19 ` Uladzimir Bely
2 siblings, 1 reply; 5+ messages in thread
From: Jan Kiszka @ 2021-10-28 15:41 UTC (permalink / raw)
To: Uladzimir Bely, isar-users
On 29.09.21 11:58, Uladzimir Bely wrote:
> Changes since v1:
> - Simplified used variables
> - Rebased on latest `next`
>
> Some custom user packages built from sources may be written in C/C++.
> Using ccache will help decrease build time in case they are rebuilt.
>
> For example, building `mc:stm32mp15x-buster:linux-mainline` during the
> test took 28 minutes at second build with ccache enabled in comparison
> with 115 minutes of first build.
>
> Uladzimir Bely (2):
> meta: Support for ccache for custom packages in buildchroot
> doc: Add section about ccache usage
>
> doc/user_manual.md | 17 ++++++++++++++
> meta-isar/conf/local.conf.sample | 5 ++++
> meta/classes/buildchroot.bbclass | 6 +++++
> meta/classes/ccache.bbclass | 23 +++++++++++++++++++
> meta/classes/dpkg.bbclass | 3 ++-
> .../buildchroot/buildchroot.inc | 1 +
> .../buildchroot/files/build.sh | 6 +++++
> .../buildchroot/files/common.sh | 1 +
> 8 files changed, 61 insertions(+), 1 deletion(-)
> create mode 100644 meta/classes/ccache.bbclass
>
What's the status here? I think we still wanted to simplify/unify the
UI, aligning it to ISAR_CROSS_COMPILE.
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v2 0/2] Support for ccache
2021-10-28 15:41 ` [PATCH v2 0/2] Support for ccache Jan Kiszka
@ 2021-10-28 16:19 ` Uladzimir Bely
0 siblings, 0 replies; 5+ messages in thread
From: Uladzimir Bely @ 2021-10-28 16:19 UTC (permalink / raw)
To: isar-users, Jan Kiszka
In mail from четверг, 28 октября 2021 г. 18:41:05 +03 user Jan Kiszka wrote:
> On 29.09.21 11:58, Uladzimir Bely wrote:
> > Changes since v1:
> > - Simplified used variables
> > - Rebased on latest `next`
> >
> > Some custom user packages built from sources may be written in C/C++.
> > Using ccache will help decrease build time in case they are rebuilt.
> >
> > For example, building `mc:stm32mp15x-buster:linux-mainline` during the
> > test took 28 minutes at second build with ccache enabled in comparison
> > with 115 minutes of first build.
> >
> > Uladzimir Bely (2):
> > meta: Support for ccache for custom packages in buildchroot
> > doc: Add section about ccache usage
> >
> > doc/user_manual.md | 17 ++++++++++++++
> > meta-isar/conf/local.conf.sample | 5 ++++
> > meta/classes/buildchroot.bbclass | 6 +++++
> > meta/classes/ccache.bbclass | 23 +++++++++++++++++++
> > meta/classes/dpkg.bbclass | 3 ++-
> > .../buildchroot/buildchroot.inc | 1 +
> > .../buildchroot/files/build.sh | 6 +++++
> > .../buildchroot/files/common.sh | 1 +
> > 8 files changed, 61 insertions(+), 1 deletion(-)
> > create mode 100644 meta/classes/ccache.bbclass
>
> What's the status here? I think we still wanted to simplify/unify the
> UI, aligning it to ISAR_CROSS_COMPILE.
>
> Jan
The new patchset is prepared, but a bit stucked at internal review.
In short, changes in it are the following:
- USE_CCACHE="1" in `local.conf` is used in to enable the feature in general.
- USE_CCACHE="0" can be set in the recipe to avoid caching even if it's
enabled in general
- ccache.bbclass was removed the main ccache-related variables were placed to
bitbake.conf.
--
Uladzimir Bely
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-10-28 16:19 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-29 9:58 [PATCH v2 0/2] Support for ccache Uladzimir Bely
2021-09-29 9:58 ` [PATCH v2 1/2] meta: Support for ccache for custom packages in buildchroot Uladzimir Bely
2021-09-29 9:58 ` [PATCH v2 2/2] doc: Add section about ccache usage Uladzimir Bely
2021-10-28 15:41 ` [PATCH v2 0/2] Support for ccache Jan Kiszka
2021-10-28 16:19 ` Uladzimir Bely
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox