* [PATCH v3 1/3] meta: Support for ccache for custom packages
2021-10-29 13:56 [PATCH v3 0/3] Support of ccache Uladzimir Bely
@ 2021-10-29 13:56 ` Uladzimir Bely
2021-10-29 13:56 ` [PATCH v3 2/3] doc: Add section about ccache usage Uladzimir Bely
2021-10-29 13:57 ` [PATCH v3 3/3] ci: Add test for ccache Uladzimir Bely
2 siblings, 0 replies; 5+ messages in thread
From: Uladzimir Bely @ 2021-10-29 13:56 UTC (permalink / raw)
To: isar-users
Add `USE_CCACHE = "1"` to local.conf to globally activate
the ccache functionality for dpkg-based recipes.
Add `USE_CCACHE = "0"` to the recipe to disable ccache for it
even if globaly enabled.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta-isar/conf/local.conf.sample | 5 +++++
meta/classes/buildchroot.bbclass | 9 +++++++++
meta/classes/dpkg.bbclass | 2 +-
meta/conf/bitbake.conf | 5 +++++
meta/recipes-devtools/buildchroot/buildchroot.inc | 1 +
meta/recipes-devtools/buildchroot/files/build.sh | 5 +++++
meta/recipes-devtools/buildchroot/files/common.sh | 1 +
7 files changed, 27 insertions(+), 1 deletion(-)
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 1ab640d0..42a69aed 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -227,3 +227,8 @@ USER_isar[flags] += "clear-text-password"
# Uncomment the below line to debug WIC.
# WIC_CREATE_EXTRA_ARGS += "-D"
+
+# Uncomment this to use ccache for custom packages
+#USE_CCACHE = "1"
+# Uncomment and set own top level ccache directory to share between builds
+#CCACHE_TOP_DIR ?= "${TMPDIR}/ccache"
diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index e9eb9afa..dd8f4206 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -24,6 +24,10 @@ python __anonymous() {
MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}.lock"
buildchroot_do_mounts() {
+ if [ "${USE_CCACHE}" = "1" ]; then
+ mkdir -p ${CCACHE_DIR}
+ fi
+
sudo -s <<'EOSUDO'
( flock 9
set -e
@@ -32,6 +36,11 @@ buildchroot_do_mounts() {
mount --bind '${REPO_ISAR_DIR}/${DISTRO}' '${BUILDCHROOT_DIR}/isar-apt'
mountpoint -q '${BUILDCHROOT_DIR}/downloads' ||
mount --bind '${DL_DIR}' '${BUILDCHROOT_DIR}/downloads'
+ if [ "${USE_CCACHE}" = "1" ]; then
+ mkdir -p '${BUILDCHROOT_DIR}/ccache'
+ mountpoint -q '${BUILDCHROOT_DIR}/ccache' ||
+ mount --bind '${CCACHE_DIR}' '${BUILDCHROOT_DIR}/ccache'
+ fi
mountpoint -q '${BUILDCHROOT_DIR}/dev' ||
mount --rbind /dev '${BUILDCHROOT_DIR}/dev'
mount --make-rslave '${BUILDCHROOT_DIR}/dev'
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 4e7c2f77..86b47c28 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -33,5 +33,5 @@ dpkg_runbuild() {
E="${@ isar_export_proxies(d)}"
export PARALLEL_MAKE="${PARALLEL_MAKE}"
sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
- /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH}
+ /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH} ${USE_CCACHE}
}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 7f5901d7..dce467b2 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -115,6 +115,11 @@ BBINCLUDELOGS ??= "yes"
# Add event handlers for bitbake
INHERIT += "isar-events"
+# Default values for ccache
+USE_CCACHE ??= "0"
+CCACHE_TOP_DIR ?= "${TMPDIR}/ccache"
+CCACHE_DIR ?= "${CCACHE_TOP_DIR}/${DISTRO}-${DISTRO_ARCH}"
+
include conf/local.conf
include conf/multiconfig/${BB_CURRENT_MC}.conf
include conf/machine/${MACHINE}.conf
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index 31524a11..726c7bbd 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -32,6 +32,7 @@ BUILDCHROOT_PREINSTALL_COMMON = " \
make \
debhelper \
autotools-dev \
+ ccache \
dpkg \
locales \
docbook-to-man \
diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
index 101581db..8e6507c6 100644
--- a/meta/recipes-devtools/buildchroot/files/build.sh
+++ b/meta/recipes-devtools/buildchroot/files/build.sh
@@ -14,4 +14,9 @@ for i in configure aclocal.m4 Makefile.am Makefile.in; do
fi
done
+if [ "$use_ccache" == "1" ]; then
+ export CCACHE_DIR=/ccache
+ export PATH=/usr/lib/ccache:$PATH
+fi
+
${GBP_PREFIX}dpkg-buildpackage -a$target_arch -d --source-option=-I
diff --git a/meta/recipes-devtools/buildchroot/files/common.sh b/meta/recipes-devtools/buildchroot/files/common.sh
index 0063a38e..7085f443 100644
--- a/meta/recipes-devtools/buildchroot/files/common.sh
+++ b/meta/recipes-devtools/buildchroot/files/common.sh
@@ -17,6 +17,7 @@ fi
# Create human-readable names
target_arch=$2
+use_ccache=$3
set_arch="--host-arch $target_arch"
--
2.20.1
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH v3 2/3] doc: Add section about ccache usage
2021-10-29 13:56 [PATCH v3 0/3] Support of ccache Uladzimir Bely
2021-10-29 13:56 ` [PATCH v3 1/3] meta: Support for ccache for custom packages Uladzimir Bely
@ 2021-10-29 13:56 ` Uladzimir Bely
2021-10-29 13:57 ` [PATCH v3 3/3] ci: Add test for ccache Uladzimir Bely
2 siblings, 0 replies; 5+ messages in thread
From: Uladzimir Bely @ 2021-10-29 13:56 UTC (permalink / raw)
To: isar-users
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
doc/user_manual.md | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/doc/user_manual.md b/doc/user_manual.md
index 01fe7019..535d456a 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -18,6 +18,7 @@ Copyright (C) 2016-2019, ilbers GmbH
- [Add a New Image Type](#add-a-new-image-type)
- [Add a Custom Application](#add-a-custom-application)
- [Enabling Cross-compilation](#isar-cross-compilation)
+ - [Using ccache for custom packages](#using-ccache-for-custom-packages)
- [Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)
- [Create a containerized Isar SDK root filesystem](#create-a-containerized-isar-sdk-root-filesystem)
- [Creation of local apt repo caching upstream Debian packages](#creation-of-local-apt-repo-caching-upstream-debian-packages)
@@ -912,6 +913,22 @@ package build. To invoke it, just call
`bitbake mc:${MACHINE}-${DISTRO}:<package_name> -c devshell`.
+## Using ccache for custom packages
+
+While base system is created from binary Debian repositories, some user
+packages are built from sources. It's possible to reduce build time
+for such packages by enabling ccache.
+
+To enable global ccache functionality, `USE_CCACHE = "1"` can be added
+to `local.conf`. If some package requires ccache to be always disabled,
+`USE_CCACHE = "0"` can be used in the recipe despite global setup.
+
+By default, ccache directory is created inside `TMPDIR`, but it can be
+adjusted by `CCACHE_TOP_DIR` variable in `local.conf`. Ccache directory
+`CCACHE_DIR` default value is `"${CCACHE_TOP_DIR}/${DISTRO}-${DISTRO_ARCH}"`,
+that means caches for different distros and architectures are not overlapped.
+
+
## Create an ISAR SDK root filesystem
### Motivation
--
2.20.1
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH v3 3/3] ci: Add test for ccache
2021-10-29 13:56 [PATCH v3 0/3] Support of ccache Uladzimir Bely
2021-10-29 13:56 ` [PATCH v3 1/3] meta: Support for ccache for custom packages Uladzimir Bely
2021-10-29 13:56 ` [PATCH v3 2/3] doc: Add section about ccache usage Uladzimir Bely
@ 2021-10-29 13:57 ` Uladzimir Bely
2021-10-29 14:04 ` Uladzimir Bely
2 siblings, 1 reply; 5+ messages in thread
From: Uladzimir Bely @ 2021-10-29 13:57 UTC (permalink / raw)
To: isar-users
The test runs bitbake target twice. First run with ccache enabled
makes CCACHE_DIR be populated and the second run reuses it to
speedup build process.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
testsuite/build_test/build_test.py | 11 ++++++++++
testsuite/build_test/cibase.py | 33 ++++++++++++++++++++++++++++++
2 files changed, 44 insertions(+)
diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
index d39c10c0..05db4b78 100644
--- a/testsuite/build_test/build_test.py
+++ b/testsuite/build_test/build_test.py
@@ -206,3 +206,14 @@ class ContainerSdkTest(CIBaseTest):
targets = ['mc:container-amd64-stretch:isar-image-base']
self.perform_container_test(targets, 'do_populate_sdk')
+
+class CcacheTest(CIBaseTest):
+
+ """
+ Test rebuild speed improve with ccache
+
+ :avocado: tags=ccache,full
+ """
+ def test_ccache_rebuild(self):
+ targets = ['mc:de0-nano-soc-buster:isar-image-base']
+ self.perform_ccache_test(targets)
diff --git a/testsuite/build_test/cibase.py b/testsuite/build_test/cibase.py
index 78d7bdb7..0b053aa5 100644
--- a/testsuite/build_test/cibase.py
+++ b/testsuite/build_test/cibase.py
@@ -3,6 +3,7 @@
import os
import re
import tempfile
+import time
from cibuilder import CIBuilder
from avocado.utils import process
@@ -114,3 +115,35 @@ class CIBaseTest(CIBuilder):
self.bitbake(build_dir, targets, bitbake_cmd, bb_args)
+
+ def perform_ccache_test(self, targets):
+ build_dir, bb_args = self.prep('Isar ccache build', targets, 0, 0)
+
+ self.deletetmp(build_dir)
+ process.run('rm -rf ' + build_dir + '/ccache', sudo=True)
+
+ with open(build_dir + '/conf/ci_build.conf', 'a') as file:
+ file.write('USE_CCACHE = "1"\n')
+ file.write('CCACHE_TOP_DIR = "${TOPDIR}/ccache"')
+
+ self.log.info('Starting build and filling ccache dir...')
+ start = time.time()
+ self.bitbake(build_dir, targets, None, bb_args)
+ first_time = time.time() - start
+ self.log.info('Non-cached build: ' + str(round(first_time)) + 's')
+
+ self.deletetmp(build_dir)
+
+ self.log.info('Starting build and using ccache dir...')
+ start = time.time()
+ self.bitbake(build_dir, targets, None, bb_args)
+ second_time = time.time() - start
+ self.log.info('Cached build: ' + str(round(second_time)) + 's')
+
+ speedup_k = 1.1
+ if first_time / second_time < speedup_k:
+ self.fail('No speedup after rebuild with ccache')
+
+ # Cleanup
+ self.deletetmp(build_dir)
+ process.run('rm -rf ' + build_dir + '/ccache', sudo=True)
--
2.20.1
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH v3 3/3] ci: Add test for ccache
2021-10-29 13:57 ` [PATCH v3 3/3] ci: Add test for ccache Uladzimir Bely
@ 2021-10-29 14:04 ` Uladzimir Bely
0 siblings, 0 replies; 5+ messages in thread
From: Uladzimir Bely @ 2021-10-29 14:04 UTC (permalink / raw)
To: isar-users
In mail from пятница, 29 октября 2021 г. 16:57:00 +03 user Uladzimir Bely
wrote:
> The test runs bitbake target twice. First run with ccache enabled
> makes CCACHE_DIR be populated and the second run reuses it to
> speedup build process.
>
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
> testsuite/build_test/build_test.py | 11 ++++++++++
> testsuite/build_test/cibase.py | 33 ++++++++++++++++++++++++++++++
> 2 files changed, 44 insertions(+)
This patch for CI is currently tested only in manual mode but it is not
expected to have any problems when running with full chain of tests in CI.
Anyway, full test including this last patch will be run soon and the result
will be availiable by Monday.
--
Uladzimir Bely
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 5+ messages in thread