Re: [PATCH 1/1] fix(bootstrap): make sstate updates of bootstrap atomic

[Zhihang Wei <wzh@ilbers.de>]

Applied to next, thanks.

Zhihang

On 9/11/26 17:06, 'Felix Moessbauer' via isar-users wrote:
> When running under multiconfig, targets that share the same distro and
> arch build the same bootstrap. As long as this is identical, this is not
> problematic. However, the deploy of the artifact to the sstate cache is
> not syncronized with with its consumers (the rootfs_prepare tasks) of
> other machines running simultaneously. These races have been there ever
> since, but with the deploy-via-sstate and other recent changes to
> increase the caching, they became much more likely.
>
> We fix this by adding a lock.
>
> Fixes: 7ac2f931 ("bootstrap: directly deploy via sstate cache")
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
> This continues the "Fix various race conditions on multiconfig" series.
> As we aim towards more efficiently using the sstate cache, it also becomes
> more likely to hit these races (despite that they have been there ever since).
> In general, our multiconfig approach is problematic if two targets with
> different machines share the same architecture and distro.
>
> I'm also pretty sure they surface now because I run the testsuite with
> sstate caching enabled, resulting in much closer timings than when building
> without the cache (=> races occur more likely).
>
> Best regards,
> Felix Moessbauer
>
>   meta/classes-recipe/rootfs.bbclass                    | 11 ++++++++++-
>   meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc |  4 ++++
>   2 files changed, 14 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
> index cf996f5c..b5cfac07 100644
> --- a/meta/classes-recipe/rootfs.bbclass
> +++ b/meta/classes-recipe/rootfs.bbclass
> @@ -238,6 +238,12 @@ rootfs_do_qemu() {
>   BOOTSTRAP_SRC = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}.tar.zst"
>   BOOTSTRAP_SRC:${ROOTFS_ARCH} = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}.tar.zst"
>   
> +# Matches do_bootstrap[sstate-lockfile]: a shared read lock held only around the
> +# untar below excludes a concurrent do_bootstrap of another multiconfig swapping
> +# this tarball out mid-extraction.
> +BOOTSTRAP_SRC_LOCK = "${DEPLOY_DIR}/bootstrap.${ROOTFS_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}.lock"
> +BOOTSTRAP_SRC_LOCK:${ROOTFS_ARCH} = "${DEPLOY_DIR}/bootstrap.${ROOTFS_DISTRO}-${ROOTFS_ARCH}.lock"
> +
>   def rootfs_extra_import(d):
>       bb.utils._context["rootfs_progress"] = __import__("rootfs_progress")
>       return ""
> @@ -247,10 +253,13 @@ ROOTFS_EXTRA_IMPORTED := "${@rootfs_extra_import(d)}"
>   rootfs_prepare[weight] = "25"
>   rootfs_prepare(){
>       rm -rf ${ROOTFSDIR}
> -    run_privileged_heredoc << 'EOF'
> +    (
> +        flock -s 9
> +        run_privileged_heredoc << 'EOF'
>           mkdir -p ${ROOTFSDIR}
>           tar -xf "${BOOTSTRAP_SRC}" -C "${ROOTFSDIR}" --exclude="./dev/console"
>   EOF
> +    ) 9<> "${BOOTSTRAP_SRC_LOCK}"
>   
>       # setup chroot
>       run_privileged "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"
> diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> index 5d4ca1ef..f31affff 100644
> --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> @@ -136,6 +136,10 @@ do_bootstrap[sstate-outputdirs] = "${DEPLOY_DIR_BOOTSTRAP}"
>   do_bootstrap[dirs] = "${BOOTSTRAP_TMPDIR} ${WORKDIR}/trusted.gpg.d ${WORKDIR}/sources.list.d"
>   do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config"
>   do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"
> +# DEPLOY_DIR_BOOTSTRAP is shared by all multiconfigs. Keep the lock next
> +# to, not inside, DEPLOY_DIR_BOOTSTRAP so sstate_clean_manifest() cannot sweep it.
> +DEPLOY_DIR_BOOTSTRAP_LOCK = "${DEPLOY_DIR}/bootstrap.${DEPLOY_ISAR_BOOTSTRAP}.lock"
> +do_bootstrap[sstate-lockfile] = "${DEPLOY_DIR_BOOTSTRAP_LOCK}"
>   
>   DEB_DL_LOCK ?= "${DEBDIR}/${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}.lock"
>   

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/97a48fec-67a4-4527-9020-c8009877fc4d%40ilbers.de.