RE: [PATCH] fix: add currenttask-guards to SSTATE(CREATE|POSTINST)FUNCS

["'Schmidt, Adriaan' via isar-users" <isar-users@googlegroups.com>]

Baurzhan Ismagulov, Mittwoch, 16. April 2025 16:13:
> On 2025-04-11 16:33, 'Adriaan Schmidt' via isar-users wrote:
> > We use SSTATECREATEFUNCS and SSTATEPOSTINSTFUNCS to insert
> > code for preparation/postprocessing of sstate artifacts.
> > As it turns out, those are not per-job, but global, so if
> > a recipe has multiple sstate-cacheable tasks, all of those
> > functions are executed, and we need to make sure to bail out
> > of all unintended calls.
> 
> Thanks Adriaan for the patch,
> 
> we've had a look and have a few comments (numbering for easier referencing):
> 
> 1. For this use case, I'd like to have a testcase. Would you like to submit one
>    or provide quick hints what to add to trigger the behavior? We've started
>    checking some simple cases but haven't reproduced it so far.

It's actually not a problem in current Isar, as the behavior is only triggered when
we have a recipe that has more than one addition to SSTATECREATEFUNCS
or SSTATEPOSTINSTFUNCS.
Felix discovered the issue while working on his initrd patches.

> 
> > --- a/meta/classes/base.bbclass
> > +++ b/meta/classes/base.bbclass
> > @@ -204,6 +204,15 @@ def isar_export_ccache(d):
> >      else:
> >          os.environ['CCACHE_DISABLE'] = '1'
> >
> > +isar_check_current_task() {
> > +    if [ "do_${BB_CURRENTTASK}" = "$1" ]; then
> > +        return 0
> > +    else
> > +        echo "skipping because current task is not $1"
> > +        return 1
> > +    fi
> > +}
> 
> 2. Could we name "isar_check_current_task" to reflect when it returns what?
>    E.g., "fail_if_task_not" or something. (Minor but easy and helpful)

Sure, I also wasn't sure what to name the function.
The pattern was copied from OE. There they don't have a generic function,
but instead copy the check directly to all relevant locations.

> 3. Technically, this function itself doesn't skip, so the message might be
>    misleading if the caller makes a mistake. (Just a comment; I don't have a
>    better wording on the spot)

That's true, it relies on correct usage by the caller, but I don't have a better idea.
Even if we "correct" the wording, the caller still needs to do the right thing...
What that output addresses is that we see the invocation of all registered functions
in the log of the bitbake task. The output makes it clear that the function was
called but didn't do anything.

> 
> > --- a/meta/classes/dpkg-base.bbclass
> > +++ b/meta/classes/dpkg-base.bbclass
> > @@ -184,6 +184,7 @@ SSTATECREATEFUNCS += "dpkg_build_sstate_prepare"
> >  SSTATEPOSTINSTFUNCS += "dpkg_build_sstate_finalize"
> >
> >  dpkg_build_sstate_prepare() {
> > +    (isar_check_current_task "do_dpkg_build" || return) || true
> 
> 4. If isar_check_current_task succeeds, the execution should continue. If it
>    fails, we return. Why is || true needed? (Just double-checking)

If the check fails, bitbake not only returns from this one function, but terminates
the whole task.

>    Coincidentally, many || true occurrences needed fixing in the past, so I'm
>    not impartial towards those :) , although it's just a handier form of "if".

I tried to make the check a one-liner... But I'd also be ok with making it a
"normal" if/then/fi.

> 5. I wonder if there are ways to remove hard-coding of "do_dpkg_build" into the
>    function (something similar to overrides, flags, etc.). (Just brainstorming)

I don't think so... we're deep in (undocumented) sstate logic here...
The approach was taken from OE, e.g. https://github.com/openembedded/openembedded-core/blob/master/meta/classes-recipe/native.bbclass#L234

>    (I've always wanted to separate hard-coded cases like "i386", "stretch",
>    etc. from the code = separate generic mechanism from the specific machine,
>    distro, etc. -- haven't come to that so far)
> 
> 
> 6. Just for my understanding, what are some examples where *CREATE* and
>    *POSTINST* are useful?

We need to use those hooks because in some cases we don't produce "nice"
artifacts that can be given to the sstate logic directly:

- our rootfs needs to be tar'ed (with sudo) to preserve file permissions
- our dpkg jobs create (any number of) .deb packages, that we need to collect before giving them to sstate

In other instances (e.g. bootstrap) we already have a nice artifact, and 
don't need this additional processing.

Adriaan

> 7. Given that this touches central classes and sstate bugs could silently
>    produce incorrect deliverables, what are the opinions regarding applying
>    this after the 0.11 release? The use case doesn't sound (to me, in any case)
>    like it would affect the most downstreams, and they are using next anyway.
> 
> 
> With kind regards,
> Baurzhan
> 
> 
> >
> > Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> > ---
> >  meta/classes/base.bbclass                             | 9 +++++++++
> >  meta/classes/dpkg-base.bbclass                        | 2 ++
> >  meta/classes/rootfs.bbclass                           | 2 ++
> >  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc   | 2 ++
> >  meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 2 ++
> >  5 files changed, 17 insertions(+)
> >
> > diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> > index c730eec9..d5a3142b 100644
> > --- a/meta/classes/base.bbclass
> > +++ b/meta/classes/base.bbclass
> > @@ -204,6 +204,15 @@ def isar_export_ccache(d):
> >      else:
> >          os.environ['CCACHE_DISABLE'] = '1'
> >
> > +isar_check_current_task() {
> > +    if [ "do_${BB_CURRENTTASK}" = "$1" ]; then
> > +        return 0
> > +    else
> > +        echo "skipping because current task is not $1"
> > +        return 1
> > +    fi
> > +}
> > +
> >  do_fetch[dirs] = "${DL_DIR}"
> >  do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
> >  do_fetch[vardeps] += "SRCREV"
> > diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> > index 4468a49a..7f2dcd3f 100644
> > --- a/meta/classes/dpkg-base.bbclass
> > +++ b/meta/classes/dpkg-base.bbclass
> > @@ -184,6 +184,7 @@ SSTATECREATEFUNCS += "dpkg_build_sstate_prepare"
> >  SSTATEPOSTINSTFUNCS += "dpkg_build_sstate_finalize"
> >
> >  dpkg_build_sstate_prepare() {
> > +    (isar_check_current_task "do_dpkg_build" || return) || true
> >      # this runs in SSTATE_BUILDDIR, which will be deleted automatically
> >      if [ -n "$(find ${WORKDIR} -maxdepth 1 -name '*.deb' -print -quit)" ]; then
> >          cp -f ${WORKDIR}/*.deb -t .
> > @@ -191,6 +192,7 @@ dpkg_build_sstate_prepare() {
> >  }
> >
> >  dpkg_build_sstate_finalize() {
> > +    (isar_check_current_task "do_dpkg_build_setscene" || return) || true
> >      # this runs in SSTATE_INSTDIR
> >      if [ -n "$(find . -maxdepth 1 -name '*.deb' -print -quit)" ]; then
> >          mv -f ./*.deb -t ${WORKDIR}/
> > diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> > index cd8fd118..aed50a91 100644
> > --- a/meta/classes/rootfs.bbclass
> > +++ b/meta/classes/rootfs.bbclass
> > @@ -437,6 +437,7 @@ SSTATE_TAR_ATTR_FLAGS ?= "--xattrs --xattrs-
> include='*'"
> >
> >  # the rootfs is owned by root, so we need some sudoing to pack and unpack
> >  rootfs_install_sstate_prepare() {
> > +    (isar_check_current_task "do_rootfs_install" || return) || true
> >      # this runs in SSTATE_BUILDDIR, which will be deleted automatically
> >      # tar --one-file-system will cross bind-mounts to the same filesystem,
> >      # so we use some mount magic to prevent that
> > @@ -450,6 +451,7 @@ rootfs_install_sstate_prepare() {
> >  do_rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
> >
> >  rootfs_install_sstate_finalize() {
> > +    (isar_check_current_task "do_rootfs_install_setscene" || return) || true
> >      # this runs in SSTATE_INSTDIR
> >      # - after building the rootfs, the tar won't be there, but we also don't need to
> unpack
> >      # - after restoring from cache, there will be a tar which we unpack and then
> delete
> > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-
> core/isar-bootstrap/isar-bootstrap.inc
> > index 08d61a84..a3ab28ca 100644
> > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > @@ -219,12 +219,14 @@ SSTATECREATEFUNCS +=
> "bootstrap_sstate_prepare"
> >  SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
> >
> >  bootstrap_sstate_prepare() {
> > +    (isar_check_current_task "do_bootstrap" || return) || true
> >      # this runs in SSTATE_BUILDDIR, which will be deleted automatically
> >      sudo cp -a "${WORKDIR}/rootfs.tar.zst" ./bootstrap.tar.zst
> >      sudo chown $(id -u):$(id -g) bootstrap.tar.zst
> >  }
> >
> >  bootstrap_sstate_finalize() {
> > +    (isar_check_current_task "do_bootstrap_setscene" || return) || true
> >      # this runs in SSTATE_INSTDIR
> >      # we should restore symlinks after using tar
> >      if [ -f bootstrap.tar.zst ]; then
> > diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-
> core/isar-mmdebstrap/isar-mmdebstrap.inc
> > index 931f6f13..44775bc5 100644
> > --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> > +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> > @@ -222,12 +222,14 @@ SSTATECREATEFUNCS +=
> "bootstrap_sstate_prepare"
> >  SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
> >
> >  bootstrap_sstate_prepare() {
> > +    (isar_check_current_task "do_bootstrap" || return) || true
> >      # this runs in SSTATE_BUILDDIR, which will be deleted automatically
> >      sudo cp -a "${WORKDIR}/rootfs.tar.zst" ./bootstrap.tar.zst
> >      sudo chown $(id -u):$(id -g) bootstrap.tar.zst
> >  }
> >
> >  bootstrap_sstate_finalize() {
> > +    (isar_check_current_task "do_bootstrap_setscene" || return) || true
> >      # this runs in SSTATE_INSTDIR
> >      # we should restore symlinks after using tar
> >      if [ -f bootstrap.tar.zst ]; then
> > --
> > 2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/AS4PR10MB5318B4C7F92D5BEAD73EDCEAED842%40AS4PR10MB5318.EURPRD10.PROD.OUTLOOK.COM.