* [PATCH 1/1] fix(bootstrap): make sstate updates of bootstrap atomic
@ 2026-09-11 15:06 'Felix Moessbauer' via isar-users
2026-09-15 8:33 ` Zhihang Wei
0 siblings, 1 reply; 2+ messages in thread
From: 'Felix Moessbauer' via isar-users @ 2026-09-11 15:06 UTC (permalink / raw)
To: isar-users; +Cc: Felix Moessbauer
When running under multiconfig, targets that share the same distro and
arch build the same bootstrap. As long as this is identical, this is not
problematic. However, the deploy of the artifact to the sstate cache is
not syncronized with with its consumers (the rootfs_prepare tasks) of
other machines running simultaneously. These races have been there ever
since, but with the deploy-via-sstate and other recent changes to
increase the caching, they became much more likely.
We fix this by adding a lock.
Fixes: 7ac2f931 ("bootstrap: directly deploy via sstate cache")
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
This continues the "Fix various race conditions on multiconfig" series.
As we aim towards more efficiently using the sstate cache, it also becomes
more likely to hit these races (despite that they have been there ever since).
In general, our multiconfig approach is problematic if two targets with
different machines share the same architecture and distro.
I'm also pretty sure they surface now because I run the testsuite with
sstate caching enabled, resulting in much closer timings than when building
without the cache (=> races occur more likely).
Best regards,
Felix Moessbauer
meta/classes-recipe/rootfs.bbclass | 11 ++++++++++-
meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 4 ++++
2 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index cf996f5c..b5cfac07 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -238,6 +238,12 @@ rootfs_do_qemu() {
BOOTSTRAP_SRC = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}.tar.zst"
BOOTSTRAP_SRC:${ROOTFS_ARCH} = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}.tar.zst"
+# Matches do_bootstrap[sstate-lockfile]: a shared read lock held only around the
+# untar below excludes a concurrent do_bootstrap of another multiconfig swapping
+# this tarball out mid-extraction.
+BOOTSTRAP_SRC_LOCK = "${DEPLOY_DIR}/bootstrap.${ROOTFS_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}.lock"
+BOOTSTRAP_SRC_LOCK:${ROOTFS_ARCH} = "${DEPLOY_DIR}/bootstrap.${ROOTFS_DISTRO}-${ROOTFS_ARCH}.lock"
+
def rootfs_extra_import(d):
bb.utils._context["rootfs_progress"] = __import__("rootfs_progress")
return ""
@@ -247,10 +253,13 @@ ROOTFS_EXTRA_IMPORTED := "${@rootfs_extra_import(d)}"
rootfs_prepare[weight] = "25"
rootfs_prepare(){
rm -rf ${ROOTFSDIR}
- run_privileged_heredoc << 'EOF'
+ (
+ flock -s 9
+ run_privileged_heredoc << 'EOF'
mkdir -p ${ROOTFSDIR}
tar -xf "${BOOTSTRAP_SRC}" -C "${ROOTFSDIR}" --exclude="./dev/console"
EOF
+ ) 9<> "${BOOTSTRAP_SRC_LOCK}"
# setup chroot
run_privileged "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"
diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
index 5d4ca1ef..f31affff 100644
--- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
+++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
@@ -136,6 +136,10 @@ do_bootstrap[sstate-outputdirs] = "${DEPLOY_DIR_BOOTSTRAP}"
do_bootstrap[dirs] = "${BOOTSTRAP_TMPDIR} ${WORKDIR}/trusted.gpg.d ${WORKDIR}/sources.list.d"
do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config"
do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"
+# DEPLOY_DIR_BOOTSTRAP is shared by all multiconfigs. Keep the lock next
+# to, not inside, DEPLOY_DIR_BOOTSTRAP so sstate_clean_manifest() cannot sweep it.
+DEPLOY_DIR_BOOTSTRAP_LOCK = "${DEPLOY_DIR}/bootstrap.${DEPLOY_ISAR_BOOTSTRAP}.lock"
+do_bootstrap[sstate-lockfile] = "${DEPLOY_DIR_BOOTSTRAP_LOCK}"
DEB_DL_LOCK ?= "${DEBDIR}/${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}.lock"
--
2.55.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260911150634.2086579-1-felix.moessbauer%40siemens.com.
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: [PATCH 1/1] fix(bootstrap): make sstate updates of bootstrap atomic
2026-09-11 15:06 [PATCH 1/1] fix(bootstrap): make sstate updates of bootstrap atomic 'Felix Moessbauer' via isar-users
@ 2026-09-15 8:33 ` Zhihang Wei
0 siblings, 0 replies; 2+ messages in thread
From: Zhihang Wei @ 2026-09-15 8:33 UTC (permalink / raw)
To: Felix Moessbauer, isar-users
Applied to next, thanks.
Zhihang
On 9/11/26 17:06, 'Felix Moessbauer' via isar-users wrote:
> When running under multiconfig, targets that share the same distro and
> arch build the same bootstrap. As long as this is identical, this is not
> problematic. However, the deploy of the artifact to the sstate cache is
> not syncronized with with its consumers (the rootfs_prepare tasks) of
> other machines running simultaneously. These races have been there ever
> since, but with the deploy-via-sstate and other recent changes to
> increase the caching, they became much more likely.
>
> We fix this by adding a lock.
>
> Fixes: 7ac2f931 ("bootstrap: directly deploy via sstate cache")
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
> This continues the "Fix various race conditions on multiconfig" series.
> As we aim towards more efficiently using the sstate cache, it also becomes
> more likely to hit these races (despite that they have been there ever since).
> In general, our multiconfig approach is problematic if two targets with
> different machines share the same architecture and distro.
>
> I'm also pretty sure they surface now because I run the testsuite with
> sstate caching enabled, resulting in much closer timings than when building
> without the cache (=> races occur more likely).
>
> Best regards,
> Felix Moessbauer
>
> meta/classes-recipe/rootfs.bbclass | 11 ++++++++++-
> meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 4 ++++
> 2 files changed, 14 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
> index cf996f5c..b5cfac07 100644
> --- a/meta/classes-recipe/rootfs.bbclass
> +++ b/meta/classes-recipe/rootfs.bbclass
> @@ -238,6 +238,12 @@ rootfs_do_qemu() {
> BOOTSTRAP_SRC = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}.tar.zst"
> BOOTSTRAP_SRC:${ROOTFS_ARCH} = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}.tar.zst"
>
> +# Matches do_bootstrap[sstate-lockfile]: a shared read lock held only around the
> +# untar below excludes a concurrent do_bootstrap of another multiconfig swapping
> +# this tarball out mid-extraction.
> +BOOTSTRAP_SRC_LOCK = "${DEPLOY_DIR}/bootstrap.${ROOTFS_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}.lock"
> +BOOTSTRAP_SRC_LOCK:${ROOTFS_ARCH} = "${DEPLOY_DIR}/bootstrap.${ROOTFS_DISTRO}-${ROOTFS_ARCH}.lock"
> +
> def rootfs_extra_import(d):
> bb.utils._context["rootfs_progress"] = __import__("rootfs_progress")
> return ""
> @@ -247,10 +253,13 @@ ROOTFS_EXTRA_IMPORTED := "${@rootfs_extra_import(d)}"
> rootfs_prepare[weight] = "25"
> rootfs_prepare(){
> rm -rf ${ROOTFSDIR}
> - run_privileged_heredoc << 'EOF'
> + (
> + flock -s 9
> + run_privileged_heredoc << 'EOF'
> mkdir -p ${ROOTFSDIR}
> tar -xf "${BOOTSTRAP_SRC}" -C "${ROOTFSDIR}" --exclude="./dev/console"
> EOF
> + ) 9<> "${BOOTSTRAP_SRC_LOCK}"
>
> # setup chroot
> run_privileged "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"
> diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> index 5d4ca1ef..f31affff 100644
> --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> @@ -136,6 +136,10 @@ do_bootstrap[sstate-outputdirs] = "${DEPLOY_DIR_BOOTSTRAP}"
> do_bootstrap[dirs] = "${BOOTSTRAP_TMPDIR} ${WORKDIR}/trusted.gpg.d ${WORKDIR}/sources.list.d"
> do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config"
> do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"
> +# DEPLOY_DIR_BOOTSTRAP is shared by all multiconfigs. Keep the lock next
> +# to, not inside, DEPLOY_DIR_BOOTSTRAP so sstate_clean_manifest() cannot sweep it.
> +DEPLOY_DIR_BOOTSTRAP_LOCK = "${DEPLOY_DIR}/bootstrap.${DEPLOY_ISAR_BOOTSTRAP}.lock"
> +do_bootstrap[sstate-lockfile] = "${DEPLOY_DIR_BOOTSTRAP_LOCK}"
>
> DEB_DL_LOCK ?= "${DEBDIR}/${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}.lock"
>
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/97a48fec-67a4-4527-9020-c8009877fc4d%40ilbers.de.
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-09-15 8:33 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-09-11 15:06 [PATCH 1/1] fix(bootstrap): make sstate updates of bootstrap atomic 'Felix Moessbauer' via isar-users
2026-09-15 8:33 ` Zhihang Wei
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox