Re: [PATCH 5/7] meta: Update OE-libs to v5.0.3

["'MOESSBAUER, Felix' via isar-users" <isar-users@googlegroups.com>]

On Fri, 2026-06-19 at 09:21 +0200, Anton Mikanovich wrote:
> OE-core Revision: 236ac1b43308df722a78d3aa20aef065dfae5b2b.

Hi,

this change completely breaks isar if an external sstate cache is used
(e.g. with SSTATE_MIRRORS on S3). The oe-lib update has to be done in a
lockstep with an update of the sstate.bbclass. I'm currently checking
if we can just copy-in from oe-core, or if we need to adapt.

Error pattern:

ERROR: An uncaught exception occurred in runqueue
Traceback (most recent call last):
  File "/builds/ebsy/debian/meta-
realtime/isar/bitbake/lib/bb/runqueue.py", line 1637, in
execute_runqueue
    return self._execute_runqueue()
           ~~~~~~~~~~~~~~~~~~~~~~^^
  File "/builds/ebsy/debian/meta-
realtime/isar/bitbake/lib/bb/runqueue.py", line 1575, in
_execute_runqueue
    self.rqexe = RunQueueExecute(self)
                 ~~~~~~~~~~~~~~~^^^^^^
  File "/builds/ebsy/debian/meta-
realtime/isar/bitbake/lib/bb/runqueue.py", line 1918, in __init__
    update_scenequeue_data(self.sqdata.sq_revdeps, self.sqdata,
self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=True)
   
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/ebsy/debian/meta-
realtime/isar/bitbake/lib/bb/runqueue.py", line 3102, in
update_scenequeue_data
    sqdata.valid |= rq.validate_hashes(tocheck, cooker.data,
len(sqdata.stamppresent), False, summary=summary)
                   
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
  File "/builds/ebsy/debian/meta-
realtime/isar/bitbake/lib/bb/runqueue.py", line 1521, in
validate_hashes
    valid = self.validate_hash(sq_data, data, siginfo, currentcount,
summary)
  File "/builds/ebsy/debian/meta-
realtime/isar/bitbake/lib/bb/runqueue.py", line 1531, in validate_hash
    return bb.utils.better_eval(call, locs)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/builds/ebsy/debian/meta-
realtime/isar/bitbake/lib/bb/utils.py", line 434, in better_eval
    return eval(source, ctx, locals)
  File "<string>", line 1, in <module>
  File "/builds/ebsy/debian/meta-realtime/build/../isar/meta/classes-
global/sstate.bbclass", line 1028, in sstate_checkhashes
    pool = oe.utils.ThreadedPool(nproc, len(tasklist),
           ^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'oe.utils' has no attribute 'ThreadedPool'

Felix

> 
> Following changes are left because they are still needed in Isar:
> 
> 9a714f6e patch: Set dummy credentials for git notes
> 68f299d0 devshell: Use different termination test to avoid warnings
> 
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>  meta/lib/buildstats.py      |   4 +-
>  meta/lib/oe/classutils.py   |   2 +-
>  meta/lib/oe/data.py         |   2 +-
>  meta/lib/oe/gpg_sign.py     |   2 +-
>  meta/lib/oe/maketype.py     |   2 +-
>  meta/lib/oe/patch.py        |   2 +-
>  meta/lib/oe/path.py         |   2 +-
>  meta/lib/oe/reproducible.py |   2 +-
>  meta/lib/oe/sstatesig.py    |   2 +-
>  meta/lib/oe/terminal.py     |   2 +-
>  meta/lib/oe/types.py        |   2 +-
>  meta/lib/oe/utils.py        | 100 ++++++++++--------------------------
>  12 files changed, 39 insertions(+), 85 deletions(-)
> 
> diff --git a/meta/lib/buildstats.py b/meta/lib/buildstats.py
> index fe801a28..1ffe6798 100644
> --- a/meta/lib/buildstats.py
> +++ b/meta/lib/buildstats.py
> @@ -1,5 +1,5 @@
>  #
> -# Imported from openembedded-core
> +# Copyright OpenEmbedded Contributors
>  #
>  # SPDX-License-Identifier: GPL-2.0-only
>  #
> @@ -212,4 +212,4 @@ class SystemStats:
>                       b'\n')
>              self.last_disk_monitor = now
>              retval = True
> -        return retval
> +        return retval
> \ No newline at end of file
> diff --git a/meta/lib/oe/classutils.py b/meta/lib/oe/classutils.py
> index d879191d..ec3f6ad7 100644
> --- a/meta/lib/oe/classutils.py
> +++ b/meta/lib/oe/classutils.py
> @@ -1,5 +1,5 @@
>  #
> -# Imported from openembedded-core
> +# Copyright OpenEmbedded Contributors
>  #
>  # SPDX-License-Identifier: GPL-2.0-only
>  #
> diff --git a/meta/lib/oe/data.py b/meta/lib/oe/data.py
> index b832e9b3..37121cfa 100644
> --- a/meta/lib/oe/data.py
> +++ b/meta/lib/oe/data.py
> @@ -1,5 +1,5 @@
>  #
> -# Imported from openembedded-core
> +# Copyright OpenEmbedded Contributors
>  #
>  # SPDX-License-Identifier: GPL-2.0-only
>  #
> diff --git a/meta/lib/oe/gpg_sign.py b/meta/lib/oe/gpg_sign.py
> index 7a9cec94..ede6186c 100644
> --- a/meta/lib/oe/gpg_sign.py
> +++ b/meta/lib/oe/gpg_sign.py
> @@ -1,5 +1,5 @@
>  #
> -# Imported from openembedded-core
> +# Copyright OpenEmbedded Contributors
>  #
>  # SPDX-License-Identifier: GPL-2.0-only
>  #
> diff --git a/meta/lib/oe/maketype.py b/meta/lib/oe/maketype.py
> index a9a1dd7b..7a83bdf6 100644
> --- a/meta/lib/oe/maketype.py
> +++ b/meta/lib/oe/maketype.py
> @@ -1,5 +1,5 @@
>  #
> -# Imported from openembedded-core
> +# Copyright OpenEmbedded Contributors
>  #
>  # SPDX-License-Identifier: GPL-2.0-only
>  #
> diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
> index 35734a0d..60a0cc82 100644
> --- a/meta/lib/oe/patch.py
> +++ b/meta/lib/oe/patch.py
> @@ -1,5 +1,5 @@
>  #
> -# Imported from openembedded-core
> +# Copyright OpenEmbedded Contributors
>  #
>  # SPDX-License-Identifier: GPL-2.0-only
>  #
> diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
> index 68dcb595..5d21cdcb 100644
> --- a/meta/lib/oe/path.py
> +++ b/meta/lib/oe/path.py
> @@ -1,5 +1,5 @@
>  #
> -# Imported from openembedded-core
> +# Copyright OpenEmbedded Contributors
>  #
>  # SPDX-License-Identifier: GPL-2.0-only
>  #
> diff --git a/meta/lib/oe/reproducible.py b/meta/lib/oe/reproducible.py
> index 06a4b5fc..448befce 100644
> --- a/meta/lib/oe/reproducible.py
> +++ b/meta/lib/oe/reproducible.py
> @@ -1,5 +1,5 @@
>  #
> -# Imported from openembedded-core
> +# Copyright OpenEmbedded Contributors
>  #
>  # SPDX-License-Identifier: GPL-2.0-only
>  #
> diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
> index 63202204..a46e5502 100644
> --- a/meta/lib/oe/sstatesig.py
> +++ b/meta/lib/oe/sstatesig.py
> @@ -1,5 +1,5 @@
>  #
> -# Imported from openembedded-core
> +# Copyright OpenEmbedded Contributors
>  #
>  # SPDX-License-Identifier: GPL-2.0-only
>  #
> diff --git a/meta/lib/oe/terminal.py b/meta/lib/oe/terminal.py
> index 2ae7a45a..4412bc14 100644
> --- a/meta/lib/oe/terminal.py
> +++ b/meta/lib/oe/terminal.py
> @@ -1,5 +1,5 @@
>  #
> -# Imported from openembedded-core
> +# Copyright OpenEmbedded Contributors
>  #
>  # SPDX-License-Identifier: GPL-2.0-only
>  #
> diff --git a/meta/lib/oe/types.py b/meta/lib/oe/types.py
> index 4927fbc5..b929afb1 100644
> --- a/meta/lib/oe/types.py
> +++ b/meta/lib/oe/types.py
> @@ -1,5 +1,5 @@
>  #
> -# Imported from openembedded-core
> +# Copyright OpenEmbedded Contributors
>  #
>  # SPDX-License-Identifier: GPL-2.0-only
>  #
> diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
> index 9455aadb..14a7d07e 100644
> --- a/meta/lib/oe/utils.py
> +++ b/meta/lib/oe/utils.py
> @@ -1,5 +1,5 @@
>  #
> -# Imported from openembedded-core
> +# Copyright OpenEmbedded Contributors
>  #
>  # SPDX-License-Identifier: GPL-2.0-only
>  #
> @@ -7,6 +7,7 @@
>  import subprocess
>  import multiprocessing
>  import traceback
> +import errno
>  
>  def read_file(filename):
>      try:
> @@ -258,16 +259,23 @@ def execute_pre_post_process(d, cmds):
>      if cmds is None:
>          return
>  
> -    for cmd in cmds.strip().split(';'):
> -        cmd = cmd.strip()
> -        if cmd != '':
> -            bb.note("Executing %s ..." % cmd)
> -            bb.build.exec_func(cmd, d)
> +    cmds = cmds.replace(";", " ")
>  
> -# For each item in items, call the function 'target' with item as the first 
> +    for cmd in cmds.split():
> +        bb.note("Executing %s ..." % cmd)
> +        bb.build.exec_func(cmd, d)
> +
> +def get_bb_number_threads(d):
> +    return int(d.getVar("BB_NUMBER_THREADS") or os.cpu_count() or 1)
> +
> +def multiprocess_launch(target, items, d, extraargs=None):
> +    max_process = get_bb_number_threads(d)
> +    return multiprocess_launch_mp(target, items, max_process, extraargs)
> +
> +# For each item in items, call the function 'target' with item as the first
>  # argument, extraargs as the other arguments and handle any exceptions in the
>  # parent thread
> -def multiprocess_launch(target, items, d, extraargs=None):
> +def multiprocess_launch_mp(target, items, max_process, extraargs=None):
>  
>      class ProcessLaunch(multiprocessing.Process):
>          def __init__(self, *args, **kwargs):
> @@ -302,7 +310,6 @@ def multiprocess_launch(target, items, d, extraargs=None):
>              self.update()
>              return self._result
>  
> -    max_process = int(d.getVar("BB_NUMBER_THREADS") or os.cpu_count() or 1)
>      launched = []
>      errors = []
>      results = []
> @@ -475,70 +482,6 @@ def get_multilib_datastore(variant, d):
>          localdata.setVar("MLPREFIX", "")
>      return localdata
>  
> -#
> -# Python 2.7 doesn't have threaded pools (just multiprocessing)
> -# so implement a version here
> -#
> -
> -from queue import Queue
> -from threading import Thread
> -
> -class ThreadedWorker(Thread):
> -    """Thread executing tasks from a given tasks queue"""
> -    def __init__(self, tasks, worker_init, worker_end, name=None):
> -        Thread.__init__(self, name=name)
> -        self.tasks = tasks
> -        self.daemon = True
> -
> -        self.worker_init = worker_init
> -        self.worker_end = worker_end
> -
> -    def run(self):
> -        from queue import Empty
> -
> -        if self.worker_init is not None:
> -            self.worker_init(self)
> -
> -        while True:
> -            try:
> -                func, args, kargs = self.tasks.get(block=False)
> -            except Empty:
> -                if self.worker_end is not None:
> -                    self.worker_end(self)
> -                break
> -
> -            try:
> -                func(self, *args, **kargs)
> -            except Exception as e:
> -                # Eat all exceptions
> -                bb.mainlogger.debug("Worker task raised %s" % e, exc_info=e)
> -            finally:
> -                self.tasks.task_done()
> -
> -class ThreadedPool:
> -    """Pool of threads consuming tasks from a queue"""
> -    def __init__(self, num_workers, num_tasks, worker_init=None, worker_end=None, name="ThreadedPool-"):
> -        self.tasks = Queue(num_tasks)
> -        self.workers = []
> -
> -        for i in range(num_workers):
> -            worker = ThreadedWorker(self.tasks, worker_init, worker_end, name=name + str(i))
> -            self.workers.append(worker)
> -
> -    def start(self):
> -        for worker in self.workers:
> -            worker.start()
> -
> -    def add_task(self, func, *args, **kargs):
> -        """Add a task to the queue"""
> -        self.tasks.put((func, args, kargs))
> -
> -    def wait_completion(self):
> -        """Wait for completion of all the tasks in the queue"""
> -        self.tasks.join()
> -        for worker in self.workers:
> -            worker.join()
> -
>  class ImageQAFailed(Exception):
>      def __init__(self, description, name=None, logfile=None):
>          self.description = description
> @@ -586,3 +529,14 @@ def directory_size(root, blocksize=4096):
>          total += sum(roundup(getsize(os.path.join(root, name))) for name in files)
>          total += roundup(getsize(root))
>      return total
> +
> +# Update the mtime of a file, skip if permission/read-only issues
> +def touch(filename):
> +    try:
> +        os.utime(filename, None)
> +    except PermissionError:
> +        pass
> +    except OSError as e:
> +        # Handle read-only file systems gracefully
> +        if e.errno != errno.EROFS:
> +            raise e
> -- 
> 2.43.0
> 
> -- 
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260619072238.52120-6-amikan%40ilbers.de.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/6c650b4b20838cd07f9510f4dd9b8d49a0f1cb4f.camel%40siemens.com.