[PATCH 0/7] Further improve cachability of ISAR

[PATCH 0/7] Further improve cachability of ISAR

[Felix Moessbauer]

This series changes which BB variables have an influence
of the signature of a task.
This helps to get high sstate cache hit rates, even if
the build folder is relocated (e.g. in a CI environment).

The series does not have any user-facing API changes.

Best regards,
Felix

Felix Moessbauer (7):
  signatures: exclude path to core layer from bb hashes
  signatures: whitelist external SCRIPTSDIR
  signatures: exclude BITBAKEDIR from WIC
  signatures: use TOPDIR instead of BUILDDIR in wic
  signatures: unify WKS_* variable exclusions with OE
  signatures: exclude TOPDIR from signatures
  signatures: do not expand SRC_URI

 meta/classes/dpkg-base.bbclass      | 2 +-
 meta/classes/dpkg-prebuilt.bbclass  | 2 +-
 meta/classes/imagetypes_wic.bbclass | 5 ++++-
 meta/conf/bitbake.conf              | 3 ++-
 scripts/isar-buildenv-internal      | 2 +-
 5 files changed, 9 insertions(+), 5 deletions(-)

-- 
2.30.2

[PATCH 1/7] signatures: exclude path to core layer from bb hashes

[Felix Moessbauer]

This patch excludes the LAYERDIR_core variable from
the BB hashes as it contains an absolute path.

Absolute path have to be avoided as this breaks sstate-cachability.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index db1180a7..496dcaad 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -101,7 +101,7 @@ BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI
     CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_NOHASHDIR LICENSE_PATH SDKPKGSUFFIX \
     WORKDIR STAMPCLEAN PKGDATA_DIR BUILD_ARCH SSTATE_PKGARCH \
     BB_WORKERCONTEXT BB_LIMITEDDEPS DEPLOY_DIR BUILDCHROOT_DIR \
-    REPO_ISAR_DIR REPO_ISAR_DB_DIR REPO_BASE_DIR REPO_BASE_DB_DIR"
+    REPO_ISAR_DIR REPO_ISAR_DB_DIR REPO_BASE_DIR REPO_BASE_DB_DIR LAYERDIR_core"
 BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SSH_AGENT_PID \
     SSH_AUTH_SOCK PSEUDO_BUILD BB_ENV_EXTRAWHITE DISABLE_SANITY_CHECKS \
     BB_NUMBER_THREADS BB_ORIGENV BB_INVALIDCONF BBINCLUDED \
-- 
2.30.2

[PATCH 2/7] signatures: whitelist external SCRIPTSDIR

[Felix Moessbauer]

This patch excludes the external SCRIPTSDIR from the signatures.
The directory is always relative to ISARROOT, but contains
a absolute path pointing into the build host.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/conf/bitbake.conf | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 496dcaad..9fce7c1c 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -101,7 +101,8 @@ BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI
     CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_NOHASHDIR LICENSE_PATH SDKPKGSUFFIX \
     WORKDIR STAMPCLEAN PKGDATA_DIR BUILD_ARCH SSTATE_PKGARCH \
     BB_WORKERCONTEXT BB_LIMITEDDEPS DEPLOY_DIR BUILDCHROOT_DIR \
-    REPO_ISAR_DIR REPO_ISAR_DB_DIR REPO_BASE_DIR REPO_BASE_DB_DIR LAYERDIR_core"
+    REPO_ISAR_DIR REPO_ISAR_DB_DIR REPO_BASE_DIR REPO_BASE_DB_DIR LAYERDIR_core \
+    SCRIPTSDIR"
 BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SSH_AGENT_PID \
     SSH_AUTH_SOCK PSEUDO_BUILD BB_ENV_EXTRAWHITE DISABLE_SANITY_CHECKS \
     BB_NUMBER_THREADS BB_ORIGENV BB_INVALIDCONF BBINCLUDED \
-- 
2.30.2

[PATCH 3/7] signatures: exclude BITBAKEDIR from WIC

[Felix Moessbauer]

This patch excludes the absolute BITBAKEDIR from the variables
that are used to compute the signature for wic tasks.
As bitbake is shipped with the CORE layer repo, no changes are
expected and have to be tracked.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/imagetypes_wic.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/imagetypes_wic.bbclass b/meta/classes/imagetypes_wic.bbclass
index cfcc94c7..db7bbe71 100644
--- a/meta/classes/imagetypes_wic.bbclass
+++ b/meta/classes/imagetypes_wic.bbclass
@@ -147,6 +147,7 @@ IMAGE_CMD_wic() {
     check_for_wic_warnings
 }
 
+wic_do_mounts[vardepsexclude] += "BITBAKEDIR"
 wic_do_mounts() {
     buildchroot_do_mounts
     sudo -s <<'EOSUDO'
@@ -162,6 +163,7 @@ wic_do_mounts() {
 EOSUDO
 }
 
+generate_wic_image[vardepsexclude] += "BITBAKEDIR"
 generate_wic_image() {
     export FAKEROOTCMD=${FAKEROOTCMD}
     export BUILDDIR=${BUILDDIR}
-- 
2.30.2

[PATCH 4/7] signatures: use TOPDIR instead of BUILDDIR in wic

[Felix Moessbauer]

This replaces the local externally passed BUILDDIR
with the wic variable TOPDIR. Both point to the same location,
but TOPDIR is excluded from the signatures.

This avoids an absolute path pointing into the build host.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/imagetypes_wic.bbclass | 2 +-
 scripts/isar-buildenv-internal      | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/imagetypes_wic.bbclass b/meta/classes/imagetypes_wic.bbclass
index db7bbe71..8bafe517 100644
--- a/meta/classes/imagetypes_wic.bbclass
+++ b/meta/classes/imagetypes_wic.bbclass
@@ -166,7 +166,7 @@ EOSUDO
 generate_wic_image[vardepsexclude] += "BITBAKEDIR"
 generate_wic_image() {
     export FAKEROOTCMD=${FAKEROOTCMD}
-    export BUILDDIR=${BUILDDIR}
+    export BUILDDIR=${TOPDIR}
     export MTOOLS_SKIP_CHECK=1
     mkdir -p ${IMAGE_ROOTFS}/../pseudo
     touch ${IMAGE_ROOTFS}/../pseudo/files.db
diff --git a/scripts/isar-buildenv-internal b/scripts/isar-buildenv-internal
index fe1e65fa..ec8e4377 100755
--- a/scripts/isar-buildenv-internal
+++ b/scripts/isar-buildenv-internal
@@ -77,5 +77,5 @@ export PATH
 BBPATH="${BUILDDIR}"
 export BBPATH
 
-BB_ENV_EXTRAWHITE="BUILDDIR BITBAKEDIR SCRIPTSDIR TESTSUITEDIR http_proxy https_proxy ftp_proxy no_proxy GNUPGHOME"
+BB_ENV_EXTRAWHITE="BITBAKEDIR SCRIPTSDIR TESTSUITEDIR http_proxy https_proxy ftp_proxy no_proxy GNUPGHOME"
 export BB_ENV_EXTRAWHITE
-- 
2.30.2

[PATCH 5/7] signatures: unify WKS_* variable exclusions with OE

[Felix Moessbauer]

This patch unifies the variables that are excluded from the signature
computation with its counterpart in OE.
The excluded variables contain absolute paths that point into the build
host. By excluding, we make the builddir relocatable.

This improves sstate cacheability.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/imagetypes_wic.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/imagetypes_wic.bbclass b/meta/classes/imagetypes_wic.bbclass
index 8bafe517..61a74d4a 100644
--- a/meta/classes/imagetypes_wic.bbclass
+++ b/meta/classes/imagetypes_wic.bbclass
@@ -10,6 +10,7 @@ WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}
 WKS_FILE ??= "sdimage-efi"
 
 do_copy_wks_template[file-checksums] += "${WKS_FILE_CHECKSUM}"
+do_copy_wks_template[vardepsexclude] += "WKS_TEMPLATE_PATH"
 do_copy_wks_template () {
     cp -f '${WKS_TEMPLATE_PATH}' '${WORKDIR}/${WKS_TEMPLATE_FILE}'
 }
@@ -163,7 +164,7 @@ wic_do_mounts() {
 EOSUDO
 }
 
-generate_wic_image[vardepsexclude] += "BITBAKEDIR"
+generate_wic_image[vardepsexclude] += "WKS_FULL_PATH BITBAKEDIR TOPDIR"
 generate_wic_image() {
     export FAKEROOTCMD=${FAKEROOTCMD}
     export BUILDDIR=${TOPDIR}
-- 
2.30.2

[PATCH 6/7] signatures: exclude TOPDIR from signatures

[Felix Moessbauer]

The TOPDIR variable contains the absolute path to the build directory.
As the general build environment must not depend on the path
where the build is executed from, we can safely exclude the
variable from the signatures.

This finally make the build-dir relocatable and improves caching.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 9fce7c1c..a05bd909 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -102,7 +102,7 @@ BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI
     WORKDIR STAMPCLEAN PKGDATA_DIR BUILD_ARCH SSTATE_PKGARCH \
     BB_WORKERCONTEXT BB_LIMITEDDEPS DEPLOY_DIR BUILDCHROOT_DIR \
     REPO_ISAR_DIR REPO_ISAR_DB_DIR REPO_BASE_DIR REPO_BASE_DB_DIR LAYERDIR_core \
-    SCRIPTSDIR"
+    SCRIPTSDIR TOPDIR"
 BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SSH_AGENT_PID \
     SSH_AUTH_SOCK PSEUDO_BUILD BB_ENV_EXTRAWHITE DISABLE_SANITY_CHECKS \
     BB_NUMBER_THREADS BB_ORIGENV BB_INVALIDCONF BBINCLUDED \
-- 
2.30.2

[PATCH 7/7] signatures: do not expand SRC_URI

[Felix Moessbauer]

This patch reworks modification to SRC_URI to avoid expanding
the variables in the string. By that, both cachability issues,
as well as information leaks (e.g. repo access tokens) are
mitigated.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/dpkg-base.bbclass     | 2 +-
 meta/classes/dpkg-prebuilt.bbclass | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 86933c57..163d678b 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -81,7 +81,7 @@ SRC_APT ?= ""
 
 # filter out all "apt://" URIs out of SRC_URI and stick them into SRC_APT
 python() {
-    src_uri = (d.getVar('SRC_URI') or "").split()
+    src_uri = (d.getVar('SRC_URI', False) or "").split()
 
     prefix = "apt://"
     new_src_uri = []
diff --git a/meta/classes/dpkg-prebuilt.bbclass b/meta/classes/dpkg-prebuilt.bbclass
index c32224bb..0a6daef7 100644
--- a/meta/classes/dpkg-prebuilt.bbclass
+++ b/meta/classes/dpkg-prebuilt.bbclass
@@ -7,7 +7,7 @@ inherit dpkg-base
 
 python do_unpack_prepend() {
     # enforce unpack=false
-    src_uri = (d.getVar('SRC_URI', True) or '').split()
+    src_uri = (d.getVar('SRC_URI', False) or '').split()
     if len(src_uri) == 0:
         return
     def ensure_unpack_false(uri):
-- 
2.30.2

Re: [PATCH 0/7] Further improve cachability of ISAR

[Anton Mikanovich]

04.05.2022 14:41, Felix Moessbauer wrote:
> This series changes which BB variables have an influence
> of the signature of a task.
> This helps to get high sstate cache hit rates, even if
> the build folder is relocated (e.g. in a CI environment).
>
> The series does not have any user-facing API changes.
>
> Best regards,
> Felix
>
> Felix Moessbauer (7):
>    signatures: exclude path to core layer from bb hashes
>    signatures: whitelist external SCRIPTSDIR
>    signatures: exclude BITBAKEDIR from WIC
>    signatures: use TOPDIR instead of BUILDDIR in wic
>    signatures: unify WKS_* variable exclusions with OE
>    signatures: exclude TOPDIR from signatures
>    signatures: do not expand SRC_URI
>
>   meta/classes/dpkg-base.bbclass      | 2 +-
>   meta/classes/dpkg-prebuilt.bbclass  | 2 +-
>   meta/classes/imagetypes_wic.bbclass | 5 ++++-
>   meta/conf/bitbake.conf              | 3 ++-
>   scripts/isar-buildenv-internal      | 2 +-
>   5 files changed, 9 insertions(+), 5 deletions(-)
>
Applied to next, thanks.