[PATCH 0/3] image and buildchroot tweaks

[PATCH 0/3] image and buildchroot tweaks

[Adriaan Schmidt]

Some minor adjustments to optimize signature calculation.

Adriaan Schmidt (3):
  image: set convenience variables in bitbake code
  buildchroot: delay expansion of variable BUILDCHROOT_DIR
  sstate: remove paths below TMPDIR from BB_HASHBASE_WHITELIST

 meta/classes/buildchroot.bbclass |  4 ++--
 meta/classes/image.bbclass       | 10 +++++-----
 meta/conf/bitbake.conf           |  4 +---
 3 files changed, 8 insertions(+), 10 deletions(-)

-- 
2.30.2

[PATCH 1/3] image: set convenience variables in bitbake code

[Adriaan Schmidt]

Moving these variable assignments from Python code into regular
bitbake variables gives us tracking of their dependencies in signatures.

Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
---
 meta/classes/image.bbclass | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 7eeed7db..0688b021 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -154,6 +154,11 @@ IMGCLASSES = "imagetypes imagetypes_wic imagetypes_vm imagetypes_container"
 IMGCLASSES += "${IMAGE_CLASSES}"
 inherit ${IMGCLASSES}
 
+# convenience variables to be used by CMDs
+IMAGE_FILE_HOST = "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.${type}"
+IMAGE_FILE_CHROOT = "${PP_DEPLOY}/${IMAGE_FULLNAME}.${type}"
+SUDO_CHROOT = "sudo chroot ${BUILDCHROOT_DIR}"
+
 # hook up IMAGE_CMD_*
 python() {
     image_types = (d.getVar('IMAGE_FSTYPES') or '').split()
@@ -217,11 +222,6 @@ python() {
         if any([d.getVar(arg) is None for arg in required_args]):
             bb.fatal("IMAGE_TYPE '%s' requires these arguments: %s" % (image_type, ', '.join(required_args)))
 
-        # convenience variables to be used by CMDs
-        localdata.setVar('IMAGE_FILE_HOST', '${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.${type}')
-        localdata.setVar('IMAGE_FILE_CHROOT', '${PP_DEPLOY}/${IMAGE_FULLNAME}.${type}')
-        localdata.setVar('SUDO_CHROOT', localdata.expand('sudo chroot ${BUILDCHROOT_DIR}'))
-
         # imager install
         for dep in (d.getVar('IMAGER_INSTALL_' + bt_clean) or '').split():
             imager_install.add(dep)
-- 
2.30.2

[PATCH 2/3] buildchroot: delay expansion of variable BUILDCHROOT_DIR

[Adriaan Schmidt]

The current implementation expands ${BUILDCHROOT_DIR_HOST|TARGET} during
assignment to BUILDCHROOT_DIR, which leads to absolute paths in signatures.
This can be observed in IMAGE_CMD_* and CONVERSION_CMD_*, and is bad for caching.

Thus, we delay expansion, and due to the way the do_image_* tasks are created,
the paths in our signatures will be relative to ${TMPDIR} (no expansion).

Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
---
 meta/classes/buildchroot.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index 11489661..f56c335b 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -12,10 +12,10 @@ python __anonymous() {
     if mode == "0" or d.getVar('HOST_ARCH') ==  distro_arch or \
        (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
         dep = "buildchroot-target:do_build"
-        rootfs = d.getVar('BUILDCHROOT_TARGET_DIR', True)
+        rootfs = d.getVar('BUILDCHROOT_TARGET_DIR', False)
     else:
         dep = "buildchroot-host:do_build"
-        rootfs = d.getVar('BUILDCHROOT_HOST_DIR', True)
+        rootfs = d.getVar('BUILDCHROOT_HOST_DIR', False)
 
     d.setVar('BUILDCHROOT_DEP', dep)
     d.setVar('BUILDCHROOT_DIR', rootfs)
-- 
2.30.2

[PATCH 3/3] sstate: remove paths below TMPDIR from BB_HASHBASE_WHITELIST

[Adriaan Schmidt]

This removes DEPLOY_DIR BUILDCHROOT_DIR, REPO_ISAR_DIR, REPO_ISAR_DB_DIR,
REPO_BASE_DIR, and REPO_BASE_DB_DIR from the list of variables ignored
during signature calculation.

Those paths are all below TMPDIR, which is still on the list, meaning
that no absolute paths end up in the signatures. But removing them
is cleaner, as it enables some more dependency tracking by bitbake.

Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
---
 meta/conf/bitbake.conf | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 6451cb59..c92fe1a5 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -102,9 +102,7 @@ BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI
     PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
     CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_NOHASHDIR LICENSE_PATH SDKPKGSUFFIX \
     WORKDIR STAMPCLEAN PKGDATA_DIR BUILD_ARCH SSTATE_PKGARCH \
-    BB_WORKERCONTEXT BB_LIMITEDDEPS DEPLOY_DIR BUILDCHROOT_DIR \
-    REPO_ISAR_DIR REPO_ISAR_DB_DIR REPO_BASE_DIR REPO_BASE_DB_DIR LAYERDIR_core \
-    SCRIPTSDIR TOPDIR"
+    BB_WORKERCONTEXT BB_LIMITEDDEPS LAYERDIR_core SCRIPTSDIR TOPDIR"
 BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SSH_AGENT_PID \
     SSH_AUTH_SOCK PSEUDO_BUILD BB_ENV_EXTRAWHITE DISABLE_SANITY_CHECKS \
     BB_NUMBER_THREADS BB_ORIGENV BB_INVALIDCONF BBINCLUDED \
-- 
2.30.2

Re: [PATCH 0/3] image and buildchroot tweaks

[Anton Mikanovich]

19.07.2022 15:39, Adriaan Schmidt wrote:
> Some minor adjustments to optimize signature calculation.
>
> Adriaan Schmidt (3):
>    image: set convenience variables in bitbake code
>    buildchroot: delay expansion of variable BUILDCHROOT_DIR
>    sstate: remove paths below TMPDIR from BB_HASHBASE_WHITELIST
>
>   meta/classes/buildchroot.bbclass |  4 ++--
>   meta/classes/image.bbclass       | 10 +++++-----
>   meta/conf/bitbake.conf           |  4 +---
>   3 files changed, 8 insertions(+), 10 deletions(-)
>
Applied to next, thanks.
p3 was rebased to be applied on the latest bitbake.conf changes.