[PATCH 1/1] fix(bootstrap): make sstate updates of bootstrap atomic

["'Felix Moessbauer' via isar-users" <isar-users@googlegroups.com>]

When running under multiconfig, targets that share the same distro and
arch build the same bootstrap. As long as this is identical, this is not
problematic. However, the deploy of the artifact to the sstate cache is
not syncronized with with its consumers (the rootfs_prepare tasks) of
other machines running simultaneously. These races have been there ever
since, but with the deploy-via-sstate and other recent changes to
increase the caching, they became much more likely.

We fix this by adding a lock.

Fixes: 7ac2f931 ("bootstrap: directly deploy via sstate cache")
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
This continues the "Fix various race conditions on multiconfig" series.
As we aim towards more efficiently using the sstate cache, it also becomes
more likely to hit these races (despite that they have been there ever since).
In general, our multiconfig approach is problematic if two targets with
different machines share the same architecture and distro.

I'm also pretty sure they surface now because I run the testsuite with
sstate caching enabled, resulting in much closer timings than when building
without the cache (=> races occur more likely).

Best regards,
Felix Moessbauer

 meta/classes-recipe/rootfs.bbclass                    | 11 ++++++++++-
 meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc |  4 ++++
 2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index cf996f5c..b5cfac07 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -238,6 +238,12 @@ rootfs_do_qemu() {
 BOOTSTRAP_SRC = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}.tar.zst"
 BOOTSTRAP_SRC:${ROOTFS_ARCH} = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}.tar.zst"
 
+# Matches do_bootstrap[sstate-lockfile]: a shared read lock held only around the
+# untar below excludes a concurrent do_bootstrap of another multiconfig swapping
+# this tarball out mid-extraction.
+BOOTSTRAP_SRC_LOCK = "${DEPLOY_DIR}/bootstrap.${ROOTFS_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}.lock"
+BOOTSTRAP_SRC_LOCK:${ROOTFS_ARCH} = "${DEPLOY_DIR}/bootstrap.${ROOTFS_DISTRO}-${ROOTFS_ARCH}.lock"
+
 def rootfs_extra_import(d):
     bb.utils._context["rootfs_progress"] = __import__("rootfs_progress")
     return ""
@@ -247,10 +253,13 @@ ROOTFS_EXTRA_IMPORTED := "${@rootfs_extra_import(d)}"
 rootfs_prepare[weight] = "25"
 rootfs_prepare(){
     rm -rf ${ROOTFSDIR}
-    run_privileged_heredoc << 'EOF'
+    (
+        flock -s 9
+        run_privileged_heredoc << 'EOF'
         mkdir -p ${ROOTFSDIR}
         tar -xf "${BOOTSTRAP_SRC}" -C "${ROOTFSDIR}" --exclude="./dev/console"
 EOF
+    ) 9<> "${BOOTSTRAP_SRC_LOCK}"
 
     # setup chroot
     run_privileged "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"
diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
index 5d4ca1ef..f31affff 100644
--- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
+++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
@@ -136,6 +136,10 @@ do_bootstrap[sstate-outputdirs] = "${DEPLOY_DIR_BOOTSTRAP}"
 do_bootstrap[dirs] = "${BOOTSTRAP_TMPDIR} ${WORKDIR}/trusted.gpg.d ${WORKDIR}/sources.list.d"
 do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config"
 do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"
+# DEPLOY_DIR_BOOTSTRAP is shared by all multiconfigs. Keep the lock next
+# to, not inside, DEPLOY_DIR_BOOTSTRAP so sstate_clean_manifest() cannot sweep it.
+DEPLOY_DIR_BOOTSTRAP_LOCK = "${DEPLOY_DIR}/bootstrap.${DEPLOY_ISAR_BOOTSTRAP}.lock"
+do_bootstrap[sstate-lockfile] = "${DEPLOY_DIR_BOOTSTRAP_LOCK}"
 
 DEB_DL_LOCK ?= "${DEBDIR}/${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}.lock"
 
-- 
2.55.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260911150634.2086579-1-felix.moessbauer%40siemens.com.