[PATCH 0/2] Fix data races in sstate tasks

[PATCH 0/2] Fix data races in sstate tasks

['Felix Moessbauer' via isar-users]

While extensively testing the rootless builds in a CI running with
sstate cache, I was finally able to pin-point some sporadic issues.

These issues are completely unrelated to the rootless builds, though.
They have been around for quite some time, but were quite unlikely
to hit on normal (non multiconfig) builds.

Best regards,
Felix Moessbauer

Felix Moessbauer (2):
  fix(rootfs): protect rootfs_install sstate prepare with lock
  sstate: guard sstate funcs against execution in wrong context

 meta/classes-recipe/dpkg-base.bbclass                 | 4 ++++
 meta/classes-recipe/rootfs.bbclass                    | 6 +++++-
 meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 4 ++++
 3 files changed, 13 insertions(+), 1 deletion(-)

-- 
2.53.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260625150232.531442-1-felix.moessbauer%40siemens.com.

[PATCH 1/2] fix(rootfs): protect rootfs_install sstate prepare with lock

['Felix Moessbauer' via isar-users]

Previously the lock was a no-op, as it was attached to a task that does
not exist (the task is named do_rootfs_install_setscene). Instead attach
it to the function that is executed by bitbake on sstate prepare.

This might explain some sporadic data races on sstate tasks that
silently lead to the task being re-executed.

Fixes: 085efc7d ("sstate: add caching to rootfs")
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes-recipe/rootfs.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index a58ad031..8a4aad16 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -702,7 +702,7 @@ rootfs_install_sstate_prepare() {
     sudo umount ${WORKDIR}/mnt/rootfs
     sudo chown $(id -u):$(id -g) rootfs.tar
 }
-do_rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
+rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 
 rootfs_install_sstate_finalize() {
     # this runs in SSTATE_INSTDIR
-- 
2.53.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260625150232.531442-2-felix.moessbauer%40siemens.com.

[PATCH 2/2] sstate: guard sstate funcs against execution in wrong context

['Felix Moessbauer' via isar-users]

In sstate.bbclass, sstate_package() sets SSTATE_CURRTASK from
the sstate task being packaged, then runs every function listed
in SSTATECREATEFUNCS. This happens independent of the context the sstate
task is scoped to for all tasks in a single recipe.

While this pattern is not problematic in isar itself (we only have one
SSTATECREATEFUNC per recipe), it becomes problematic when downstream
users define further manual sstate tasks. In this case, the sstate tasks
are executed in the wrong context, leading to races and failures in all
colors.

We fix this by guarding each manually defined sstate func to only run in
its intended context.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes-recipe/dpkg-base.bbclass                 | 4 ++++
 meta/classes-recipe/rootfs.bbclass                    | 4 ++++
 meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 4 ++++
 3 files changed, 12 insertions(+)

diff --git a/meta/classes-recipe/dpkg-base.bbclass b/meta/classes-recipe/dpkg-base.bbclass
index f7a12302..85c37a5a 100644
--- a/meta/classes-recipe/dpkg-base.bbclass
+++ b/meta/classes-recipe/dpkg-base.bbclass
@@ -184,6 +184,8 @@ SSTATECREATEFUNCS += "dpkg_build_sstate_prepare"
 SSTATEPOSTINSTFUNCS += "dpkg_build_sstate_finalize"
 
 dpkg_build_sstate_prepare() {
+    [ "${SSTATE_CURRTASK}" = "dpkg_build" ] || return 0
+
     # this runs in SSTATE_BUILDDIR, which will be deleted automatically
     if [ -n "$(find ${WORKDIR} -maxdepth 1 -name '*.deb' -print -quit)" ]; then
         cp -f ${WORKDIR}/*.deb -t .
@@ -191,6 +193,8 @@ dpkg_build_sstate_prepare() {
 }
 
 dpkg_build_sstate_finalize() {
+    [ "${SSTATE_CURRTASK}" = "dpkg_build" ] || return 0
+
     # this runs in SSTATE_INSTDIR
     if [ -n "$(find . -maxdepth 1 -name '*.deb' -print -quit)" ]; then
         mv -f ./*.deb -t ${WORKDIR}/
diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index 8a4aad16..936f56fe 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -691,6 +691,8 @@ SSTATE_TAR_ATTR_FLAGS ?= "--xattrs --xattrs-include='*'"
 
 # the rootfs is owned by root, so we need some sudoing to pack and unpack
 rootfs_install_sstate_prepare() {
+    [ "${SSTATE_CURRTASK}" = "rootfs_install" ] || return 0
+
     # this runs in SSTATE_BUILDDIR, which will be deleted automatically
     # tar --one-file-system will cross bind-mounts to the same filesystem,
     # so we use some mount magic to prevent that
@@ -705,6 +707,8 @@ rootfs_install_sstate_prepare() {
 rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 
 rootfs_install_sstate_finalize() {
+    [ "${SSTATE_CURRTASK}" = "rootfs_install" ] || return 0
+
     # this runs in SSTATE_INSTDIR
     # - after building the rootfs, the tar won't be there, but we also don't need to unpack
     # - after restoring from cache, there will be a tar which we unpack and then delete
diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
index e746f469..98f6c090 100644
--- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
+++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
@@ -264,12 +264,16 @@ SSTATECREATEFUNCS += "bootstrap_sstate_prepare"
 SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
 
 bootstrap_sstate_prepare() {
+    [ "${SSTATE_CURRTASK}" = "bootstrap" ] || return 0
+
     # this runs in SSTATE_BUILDDIR, which will be deleted automatically
     sudo cp -a "${WORKDIR}/rootfs.tar.zst" ./bootstrap.tar.zst
     sudo chown $(id -u):$(id -g) bootstrap.tar.zst
 }
 
 bootstrap_sstate_finalize() {
+    [ "${SSTATE_CURRTASK}" = "bootstrap" ] || return 0
+
     # this runs in SSTATE_INSTDIR
     # we should restore symlinks after using tar
     if [ -f bootstrap.tar.zst ]; then
-- 
2.53.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260625150232.531442-3-felix.moessbauer%40siemens.com.

Re: [PATCH 2/2] sstate: guard sstate funcs against execution in wrong context

['Jan Kiszka' via isar-users]

On 25.06.26 17:02, 'Felix Moessbauer' via isar-users wrote:
> In sstate.bbclass, sstate_package() sets SSTATE_CURRTASK from
> the sstate task being packaged, then runs every function listed
> in SSTATECREATEFUNCS. This happens independent of the context the sstate
> task is scoped to for all tasks in a single recipe.
> 
> While this pattern is not problematic in isar itself (we only have one
> SSTATECREATEFUNC per recipe), it becomes problematic when downstream
> users define further manual sstate tasks. In this case, the sstate tasks
> are executed in the wrong context, leading to races and failures in all
> colors.
> 
> We fix this by guarding each manually defined sstate func to only run in
> its intended context.
> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>  meta/classes-recipe/dpkg-base.bbclass                 | 4 ++++
>  meta/classes-recipe/rootfs.bbclass                    | 4 ++++
>  meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 4 ++++
>  3 files changed, 12 insertions(+)
> 
> diff --git a/meta/classes-recipe/dpkg-base.bbclass b/meta/classes-recipe/dpkg-base.bbclass
> index f7a12302..85c37a5a 100644
> --- a/meta/classes-recipe/dpkg-base.bbclass
> +++ b/meta/classes-recipe/dpkg-base.bbclass
> @@ -184,6 +184,8 @@ SSTATECREATEFUNCS += "dpkg_build_sstate_prepare"
>  SSTATEPOSTINSTFUNCS += "dpkg_build_sstate_finalize"
>  
>  dpkg_build_sstate_prepare() {
> +    [ "${SSTATE_CURRTASK}" = "dpkg_build" ] || return 0
> +
>      # this runs in SSTATE_BUILDDIR, which will be deleted automatically
>      if [ -n "$(find ${WORKDIR} -maxdepth 1 -name '*.deb' -print -quit)" ]; then
>          cp -f ${WORKDIR}/*.deb -t .
> @@ -191,6 +193,8 @@ dpkg_build_sstate_prepare() {
>  }
>  
>  dpkg_build_sstate_finalize() {
> +    [ "${SSTATE_CURRTASK}" = "dpkg_build" ] || return 0
> +
>      # this runs in SSTATE_INSTDIR
>      if [ -n "$(find . -maxdepth 1 -name '*.deb' -print -quit)" ]; then
>          mv -f ./*.deb -t ${WORKDIR}/
> diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
> index 8a4aad16..936f56fe 100644
> --- a/meta/classes-recipe/rootfs.bbclass
> +++ b/meta/classes-recipe/rootfs.bbclass
> @@ -691,6 +691,8 @@ SSTATE_TAR_ATTR_FLAGS ?= "--xattrs --xattrs-include='*'"
>  
>  # the rootfs is owned by root, so we need some sudoing to pack and unpack
>  rootfs_install_sstate_prepare() {
> +    [ "${SSTATE_CURRTASK}" = "rootfs_install" ] || return 0
> +
>      # this runs in SSTATE_BUILDDIR, which will be deleted automatically
>      # tar --one-file-system will cross bind-mounts to the same filesystem,
>      # so we use some mount magic to prevent that
> @@ -705,6 +707,8 @@ rootfs_install_sstate_prepare() {
>  rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
>  
>  rootfs_install_sstate_finalize() {
> +    [ "${SSTATE_CURRTASK}" = "rootfs_install" ] || return 0
> +
>      # this runs in SSTATE_INSTDIR
>      # - after building the rootfs, the tar won't be there, but we also don't need to unpack
>      # - after restoring from cache, there will be a tar which we unpack and then delete
> diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> index e746f469..98f6c090 100644
> --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> @@ -264,12 +264,16 @@ SSTATECREATEFUNCS += "bootstrap_sstate_prepare"
>  SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
>  
>  bootstrap_sstate_prepare() {
> +    [ "${SSTATE_CURRTASK}" = "bootstrap" ] || return 0
> +
>      # this runs in SSTATE_BUILDDIR, which will be deleted automatically
>      sudo cp -a "${WORKDIR}/rootfs.tar.zst" ./bootstrap.tar.zst
>      sudo chown $(id -u):$(id -g) bootstrap.tar.zst
>  }
>  
>  bootstrap_sstate_finalize() {
> +    [ "${SSTATE_CURRTASK}" = "bootstrap" ] || return 0
> +
>      # this runs in SSTATE_INSTDIR
>      # we should restore symlinks after using tar
>      if [ -f bootstrap.tar.zst ]; then

Out of curiosity: Does OE have similar setup and solve them the same
way? Or are we doing something ... special here? This pattern looks like
it does not scale very well (easily forgotten when adding another
function in Isar and even more easily in downstream).

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/d8dcc7f5-01ef-47b8-8965-abaa82b08621%40siemens.com.

Re: [PATCH 2/2] sstate: guard sstate funcs against execution in wrong context

['MOESSBAUER, Felix' via isar-users]

On Thu, 2026-06-25 at 18:43 +0200, Jan Kiszka wrote:
> On 25.06.26 17:02, 'Felix Moessbauer' via isar-users wrote:
> > In sstate.bbclass, sstate_package() sets SSTATE_CURRTASK from
> > the sstate task being packaged, then runs every function listed
> > in SSTATECREATEFUNCS. This happens independent of the context the sstate
> > task is scoped to for all tasks in a single recipe.
> > 
> > While this pattern is not problematic in isar itself (we only have one
> > SSTATECREATEFUNC per recipe), it becomes problematic when downstream
> > users define further manual sstate tasks. In this case, the sstate tasks
> > are executed in the wrong context, leading to races and failures in all
> > colors.
> > 
> > We fix this by guarding each manually defined sstate func to only run in
> > its intended context.
> > 
> > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> > ---
> >  meta/classes-recipe/dpkg-base.bbclass                 | 4 ++++
> >  meta/classes-recipe/rootfs.bbclass                    | 4 ++++
> >  meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 4 ++++
> >  3 files changed, 12 insertions(+)
> > 
> > diff --git a/meta/classes-recipe/dpkg-base.bbclass b/meta/classes-recipe/dpkg-base.bbclass
> > index f7a12302..85c37a5a 100644
> > --- a/meta/classes-recipe/dpkg-base.bbclass
> > +++ b/meta/classes-recipe/dpkg-base.bbclass
> > @@ -184,6 +184,8 @@ SSTATECREATEFUNCS += "dpkg_build_sstate_prepare"
> >  SSTATEPOSTINSTFUNCS += "dpkg_build_sstate_finalize"
> >  
> >  dpkg_build_sstate_prepare() {
> > +    [ "${SSTATE_CURRTASK}" = "dpkg_build" ] || return 0
> > +
> >      # this runs in SSTATE_BUILDDIR, which will be deleted automatically
> >      if [ -n "$(find ${WORKDIR} -maxdepth 1 -name '*.deb' -print -quit)" ]; then
> >          cp -f ${WORKDIR}/*.deb -t .
> > @@ -191,6 +193,8 @@ dpkg_build_sstate_prepare() {
> >  }
> >  
> >  dpkg_build_sstate_finalize() {
> > +    [ "${SSTATE_CURRTASK}" = "dpkg_build" ] || return 0
> > +
> >      # this runs in SSTATE_INSTDIR
> >      if [ -n "$(find . -maxdepth 1 -name '*.deb' -print -quit)" ]; then
> >          mv -f ./*.deb -t ${WORKDIR}/
> > diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
> > index 8a4aad16..936f56fe 100644
> > --- a/meta/classes-recipe/rootfs.bbclass
> > +++ b/meta/classes-recipe/rootfs.bbclass
> > @@ -691,6 +691,8 @@ SSTATE_TAR_ATTR_FLAGS ?= "--xattrs --xattrs-include='*'"
> >  
> >  # the rootfs is owned by root, so we need some sudoing to pack and unpack
> >  rootfs_install_sstate_prepare() {
> > +    [ "${SSTATE_CURRTASK}" = "rootfs_install" ] || return 0
> > +
> >      # this runs in SSTATE_BUILDDIR, which will be deleted automatically
> >      # tar --one-file-system will cross bind-mounts to the same filesystem,
> >      # so we use some mount magic to prevent that
> > @@ -705,6 +707,8 @@ rootfs_install_sstate_prepare() {
> >  rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
> >  
> >  rootfs_install_sstate_finalize() {
> > +    [ "${SSTATE_CURRTASK}" = "rootfs_install" ] || return 0
> > +
> >      # this runs in SSTATE_INSTDIR
> >      # - after building the rootfs, the tar won't be there, but we also don't need to unpack
> >      # - after restoring from cache, there will be a tar which we unpack and then delete
> > diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> > index e746f469..98f6c090 100644
> > --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> > +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> > @@ -264,12 +264,16 @@ SSTATECREATEFUNCS += "bootstrap_sstate_prepare"
> >  SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
> >  
> >  bootstrap_sstate_prepare() {
> > +    [ "${SSTATE_CURRTASK}" = "bootstrap" ] || return 0
> > +
> >      # this runs in SSTATE_BUILDDIR, which will be deleted automatically
> >      sudo cp -a "${WORKDIR}/rootfs.tar.zst" ./bootstrap.tar.zst
> >      sudo chown $(id -u):$(id -g) bootstrap.tar.zst
> >  }
> >  
> >  bootstrap_sstate_finalize() {
> > +    [ "${SSTATE_CURRTASK}" = "bootstrap" ] || return 0
> > +
> >      # this runs in SSTATE_INSTDIR
> >      # we should restore symlinks after using tar
> >      if [ -f bootstrap.tar.zst ]; then
> 
> Out of curiosity: Does OE have similar setup and solve them the same
> way? Or are we doing something ... special here? 
> 

They use (or nowadays mostly used) this pattern as well. However, this
is only needed when implementing the low-level sstate functions
manually, which should be avoided. When using the task flags (sstate-
inputdirs / sstate-outputdirs) this is not needed. Unfortunately we
need to implement the rootfs part manually, as tarring and untarring
the rootfs can only be performed by root (due to the permissions) and
we also want to avoid having rootfs' tarballs lingering in the workdir.

My hope is that sometime in the future we can solely rely on the task
flags, but this requires more refactoring. The rootless story is part
of it, but does not fully solve this until we drop the old privileged
part (and more ...).

You can also call it technical dept.

> This pattern looks like
> it does not scale very well (easily forgotten when adding another
> function in Isar and even more easily in downstream).

Well... you rarely need to write sstate functions by yourself, but rely
on the inputdirs / outputdirs flags instead. Then it is not needed.

Felix

> 
> Jan
> 
> -- 
> Siemens AG, Foundational Technologies
> Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/fdba43d9e861fc5f4a8ced1e74afd2da3672ad04.camel%40siemens.com.

Re: [PATCH 0/2] Fix data races in sstate tasks

[Zhihang Wei]

Applied to next, thanks.

Zhihang

On 6/25/26 17:02, 'Felix Moessbauer' via isar-users wrote:
> While extensively testing the rootless builds in a CI running with
> sstate cache, I was finally able to pin-point some sporadic issues.
>
> These issues are completely unrelated to the rootless builds, though.
> They have been around for quite some time, but were quite unlikely
> to hit on normal (non multiconfig) builds.
>
> Best regards,
> Felix Moessbauer
>
> Felix Moessbauer (2):
>    fix(rootfs): protect rootfs_install sstate prepare with lock
>    sstate: guard sstate funcs against execution in wrong context
>
>   meta/classes-recipe/dpkg-base.bbclass                 | 4 ++++
>   meta/classes-recipe/rootfs.bbclass                    | 6 +++++-
>   meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 4 ++++
>   3 files changed, 13 insertions(+), 1 deletion(-)
>

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/1a756729-b92f-4eed-bb70-d4695bbbd2e2%40ilbers.de.