[PATCH 0/5] Improve debug infrastructure for build debugging

[PATCH 0/5] Improve debug infrastructure for build debugging

[Felix Moessbauer]

Patch 1: Fix crossbuild in devshell
Patch 2,3: Fix ccache integration in devshell
Patch 4: do not warn about variables in case CCACHE_DEBUG is enabled
Patch 5: allow to set CCACHE_DEBUG from calling environment

This series helps a lot to find and solve cachability issues in
kernel (cross) builds.

Best regards,
Felix

Felix Moessbauer (5):
  get architecture specific build deps for devshell
  move CCACHE_DISABLE env var to function
  prepend ccache wrappers in devshell
  do not warn on CCACHE related variables
  allow to switch CCACHE_DEBUG mode from environment

 meta/classes/base.bbclass      |  2 ++
 meta/classes/dpkg-base.bbclass | 10 ++++++++--
 meta/classes/dpkg.bbclass      |  5 +++--
 scripts/isar-buildenv-internal |  2 +-
 4 files changed, 14 insertions(+), 5 deletions(-)

-- 
2.30.2

[PATCH 1/5] get architecture specific build deps for devshell

[Felix Moessbauer]

When using the devshell for a cross-compile package,
the dependencies have to be downloaded for the host-arch,
not the build-arch (following debian terminology).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 64f0c26b..ef04ef31 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -314,10 +314,13 @@ python do_devshell() {
 
     schroot = d.getVar('SBUILD_CHROOT')
     isar_apt = d.getVar('ISAR_APT_REPO')
+    pkg_arch = d.getVar('PACKAGE_ARCH', True)
+    build_arch = d.getVar('SBUILD_HOST_ARCH', True)
     pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
 
-    install_deps = ":" if d.getVar('BB_CURRENTTASK') == "devshell_nodeps" else "mk-build-deps -i -t \
-        \"apt-get -y -q -o Debug::pkgProblemResolver=yes --no-install-recommends --allow-downgrades\" \
+    install_deps = ":" if d.getVar('BB_CURRENTTASK') == "devshell_nodeps" else f"mk-build-deps -i \
+        --host-arch {pkg_arch} --build-arch {build_arch}  \
+        -t \"apt-get -y -q -o Debug::pkgProblemResolver=yes --no-install-recommends --allow-downgrades\" \
         debian/control"
 
     termcmd = "schroot -d / -c {0} -u root -- sh -c ' \
-- 
2.30.2

[PATCH 2/5] move CCACHE_DISABLE env var to function

[Felix Moessbauer]

This patch moves the CCACHE_DISABLE environment variable
to the isar_export_ccache function to consolidate the code.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/base.bbclass | 2 ++
 meta/classes/dpkg.bbclass | 3 +--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 17dfeab9..8c874f31 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -177,6 +177,8 @@ def isar_export_ccache(d):
         if d.getVar('CCACHE_DEBUG') == '1':
             os.environ['CCACHE_DEBUG'] = '1'
             os.environ['CCACHE_DEBUGDIR'] = '/ccache/debug'
+    else:
+        os.environ['CCACHE_DISABLE'] = '1'
 
 do_fetch[dirs] = "${DL_DIR}"
 do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 67b5aac6..0f1112ff 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -34,6 +34,7 @@ dpkg_runbuild() {
         # Don't warn some variables
         [ "${var}" = "PARALLEL_MAKE" ] && continue
         [ "${var}" = "CCACHE_DIR" ] && continue
+        [ "${var}" = "CCACHE_DISABLE" ] && continue
         [ "${var}" = "PATH_PREPEND" ] && continue
         [ "${var}" = "DEB_BUILD_OPTIONS" ] && continue
 
@@ -62,8 +63,6 @@ dpkg_runbuild() {
 
     if [ ${USE_CCACHE} -eq 1 ]; then
         schroot_configure_ccache
-    else
-        sbuild_export CCACHE_DISABLE "1"
     fi
 
     profiles="${@ isar_deb_build_profiles(d)}"
-- 
2.30.2

[PATCH 3/5] prepend ccache wrappers in devshell

[Felix Moessbauer]

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index ef04ef31..12d18592 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -311,6 +311,8 @@ python do_devshell() {
     isar_export_proxies(d)
     isar_export_ccache(d)
     isar_export_build_settings(d)
+    if d.getVar('USE_CCACHE') == '1':
+        bb.build.exec_func('schroot_configure_ccache', d)
 
     schroot = d.getVar('SBUILD_CHROOT')
     isar_apt = d.getVar('ISAR_APT_REPO')
@@ -328,6 +330,7 @@ python do_devshell() {
         echo {2} > /etc/apt/sources.list.d/isar_apt.list; \
         apt-get -y -q update; \
         {3}; \
+        export PATH=$PATH_PREPEND:$PATH; \
         $SHELL -i \
     '"
     oe_terminal(termcmd.format(schroot, pp_pps, isar_apt, install_deps), "Isar devshell", d)
-- 
2.30.2

Re: [PATCH 3/5] prepend ccache wrappers in devshell

[Jan Kiszka]

Reasoning / impact of this change is missing. Does it fix something?

Jan

On 01.08.22 16:57, Felix Moessbauer wrote:
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>  meta/classes/dpkg-base.bbclass | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index ef04ef31..12d18592 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -311,6 +311,8 @@ python do_devshell() {
>      isar_export_proxies(d)
>      isar_export_ccache(d)
>      isar_export_build_settings(d)
> +    if d.getVar('USE_CCACHE') == '1':
> +        bb.build.exec_func('schroot_configure_ccache', d)
>  
>      schroot = d.getVar('SBUILD_CHROOT')
>      isar_apt = d.getVar('ISAR_APT_REPO')
> @@ -328,6 +330,7 @@ python do_devshell() {
>          echo {2} > /etc/apt/sources.list.d/isar_apt.list; \
>          apt-get -y -q update; \
>          {3}; \
> +        export PATH=$PATH_PREPEND:$PATH; \
>          $SHELL -i \
>      '"
>      oe_terminal(termcmd.format(schroot, pp_pps, isar_apt, install_deps), "Isar devshell", d)

-- 
Siemens AG, Technology
Competence Center Embedded Linux

[PATCH 3/5] prepend ccache wrappers in devshell

[Felix Moessbauer]

This fix prepends the ccache compiler wrappers also in the devshell.
By that, an invocation of a compiler in the devshell also profits from
ccache caching.

Prior to this patch, the build-tool (like meson) had to detect
if ccache is available and modify the invocation internally.
By prepending the ccache wrappers, ccache now works for all build
tools (and direct compiler invocations).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index ef04ef31..12d18592 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -311,6 +311,8 @@ python do_devshell() {
     isar_export_proxies(d)
     isar_export_ccache(d)
     isar_export_build_settings(d)
+    if d.getVar('USE_CCACHE') == '1':
+        bb.build.exec_func('schroot_configure_ccache', d)
 
     schroot = d.getVar('SBUILD_CHROOT')
     isar_apt = d.getVar('ISAR_APT_REPO')
@@ -328,6 +330,7 @@ python do_devshell() {
         echo {2} > /etc/apt/sources.list.d/isar_apt.list; \
         apt-get -y -q update; \
         {3}; \
+        export PATH=$PATH_PREPEND:$PATH; \
         $SHELL -i \
     '"
     oe_terminal(termcmd.format(schroot, pp_pps, isar_apt, install_deps), "Isar devshell", d)
-- 
2.30.2

[PATCH 4/5] do not warn on CCACHE related variables

[Felix Moessbauer]

This patch exludes all CCACHE related variables
from the environment warning logic for sbuild.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/dpkg.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 0f1112ff..c92ea7db 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -34,6 +34,8 @@ dpkg_runbuild() {
         # Don't warn some variables
         [ "${var}" = "PARALLEL_MAKE" ] && continue
         [ "${var}" = "CCACHE_DIR" ] && continue
+        [ "${var}" = "CCACHE_DEBUGDIR" ] && continue
+        [ "${var}" = "CCACHE_DEBUG" ] && continue
         [ "${var}" = "CCACHE_DISABLE" ] && continue
         [ "${var}" = "PATH_PREPEND" ] && continue
         [ "${var}" = "DEB_BUILD_OPTIONS" ] && continue
-- 
2.30.2

[PATCH 5/5] allow to switch CCACHE_DEBUG mode from environment

[Felix Moessbauer]

This patch whitelists the CCACHE_DEBUG variable to simplify
ccache debugging. It now can be set from the environment where bitbake
is invoked.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 scripts/isar-buildenv-internal | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/isar-buildenv-internal b/scripts/isar-buildenv-internal
index ec8e4377..d202e5f9 100755
--- a/scripts/isar-buildenv-internal
+++ b/scripts/isar-buildenv-internal
@@ -77,5 +77,5 @@ export PATH
 BBPATH="${BUILDDIR}"
 export BBPATH
 
-BB_ENV_EXTRAWHITE="BITBAKEDIR SCRIPTSDIR TESTSUITEDIR http_proxy https_proxy ftp_proxy no_proxy GNUPGHOME"
+BB_ENV_EXTRAWHITE="BITBAKEDIR SCRIPTSDIR TESTSUITEDIR http_proxy https_proxy ftp_proxy no_proxy GNUPGHOME CCACHE_DEBUG"
 export BB_ENV_EXTRAWHITE
-- 
2.30.2

Re: [PATCH 0/5] Improve debug infrastructure for build debugging

[Anton Mikanovich]

01.08.2022 17:57, Felix Moessbauer wrote:
> Patch 1: Fix crossbuild in devshell
> Patch 2,3: Fix ccache integration in devshell
> Patch 4: do not warn about variables in case CCACHE_DEBUG is enabled
> Patch 5: allow to set CCACHE_DEBUG from calling environment
>
> This series helps a lot to find and solve cachability issues in
> kernel (cross) builds.
>
> Best regards,
> Felix
>
> Felix Moessbauer (5):
>    get architecture specific build deps for devshell
>    move CCACHE_DISABLE env var to function
>    prepend ccache wrappers in devshell
>    do not warn on CCACHE related variables
>    allow to switch CCACHE_DEBUG mode from environment
>
>   meta/classes/base.bbclass      |  2 ++
>   meta/classes/dpkg-base.bbclass | 10 ++++++++--
>   meta/classes/dpkg.bbclass      |  5 +++--
>   scripts/isar-buildenv-internal |  2 +-
>   4 files changed, 14 insertions(+), 5 deletions(-)
>
Applied to next, thanks.