public inbox for isar-users@googlegroups.com
 help / color / mirror / Atom feed
From: "'Moessbauer, Felix' via isar-users" <isar-users@googlegroups.com>
To: "Kowalsky, Clara" <clara.kowalsky@siemens.com>,
	"isar-users@googlegroups.com" <isar-users@googlegroups.com>
Subject: Re: [PATCH 1/8] base: replace BitBake's outdated persist_data
Date: Tue, 10 Nov 2026 11:18:45 +0000	[thread overview]
Message-ID: <ecc2fbf80cc99b3bbc0956f4d65956b650c7d888.camel@siemens.com> (raw)
In-Reply-To: <20261109091802.202521-2-clara.kowalsky@siemens.com>

On Mon, 2026-11-09 at 10:17 +0100, 'Clara Kowalsky' via isar-users
wrote:
> Isar uses a persistent UUID to distinguish build-specific resources,
> including sbuild chroot names. 
> 

The only user I can find is schroot. This will hopefully go away rather
sooner than later. Given that, I'm wondering if we really need the unit
test.

Anyways, the change is fine.

Felix

> The UUID must remain stable throughout a
> BitBake run, so parallel builds using separate PID namespaces do not
> clash on external mountpoints. It must also be shared across
> multiconfigs and cleared when the build completes.
> Isar currently stores the UUID through bb.persist_data.persist, which
> is removed in BitBake v2.9.2 commit 681a7516 ("persist_data: Remove
> it") from October 2024.
> 
> Persist the UUID in a file under PERSISTENT_DIR instead. Use BitBake's
> file-locking helper when reading, creating and clearing it to ensure
> concurrent processes share the same UUID. Clear it on the default
> BuildCompleted event, so the next run receives a fresh UUID.
> 
> Add unittests for UUID reuse across calls and multiconfigs, cleanup and
> concurrent access.
> 
> Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com>
> ---
>  meta/classes-global/base.bbclass        | 25 +++++--
>  meta/classes-global/isar-events.bbclass |  2 +-
>  testsuite/unittests/test_build_uuid.py  | 89 +++++++++++++++++++++++++
>  3 files changed, 109 insertions(+), 7 deletions(-)
>  create mode 100644 testsuite/unittests/test_build_uuid.py
> 
> diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
> index 3d5d1c12..c61e59b4 100644
> --- a/meta/classes-global/base.bbclass
> +++ b/meta/classes-global/base.bbclass
> @@ -341,12 +341,25 @@ def base_set_filespath(path, d):
>  def calculate_build_uuid(d):
>      import uuid
>  
> -    uuid_data = bb.persist_data.persist('BB_ISAR_UUID_DATA', d)
> -    if "uuid" not in uuid_data or not uuid_data["uuid"]:
> -        # Generate new UUID
> -        uuid_data["uuid"] = str(uuid.uuid4())
> -
> -    return uuid_data["uuid"]
> +    uuid_file = os.path.join(d.getVar('PERSISTENT_DIR'), 'isar-build-uuid')
> +    with bb.utils.fileslocked([uuid_file + '.lock']):
> +        try:
> +            with open(uuid_file) as file:
> +                build_uuid = file.read().strip()
> +        except FileNotFoundError:
> +            build_uuid = ''
> +
> +        if not build_uuid:
> +            build_uuid = str(uuid.uuid4())
> +            with open(uuid_file, 'w') as file:
> +                file.write(build_uuid + '\n')
> +
> +        return build_uuid
> +
> +def clear_build_uuid(d):
> +    uuid_file = os.path.join(d.getVar('PERSISTENT_DIR'), 'isar-build-uuid')
> +    with bb.utils.fileslocked([uuid_file + '.lock']):
> +        bb.utils.remove(uuid_file)
>  
>  # Unique ID for this build, used to avoid name clashes on external mountpoints
>  # When running parallel builds in different PID namespaces
> diff --git a/meta/classes-global/isar-events.bbclass b/meta/classes-global/isar-events.bbclass
> index 15bfdb99..cfe6d7dd 100644
> --- a/meta/classes-global/isar-events.bbclass
> +++ b/meta/classes-global/isar-events.bbclass
> @@ -85,7 +85,7 @@ python build_completed() {
>                  bb.error(str(e))
>  
>      # Cleanup build UUID, the next bitbake run will generate new one
> -    bb.persist_data.persist('BB_ISAR_UUID_DATA', d).clear()
> +    bb.utils.exec_flat_python_func('clear_build_uuid', d)
>  }
>  
>  build_completed[eventmask] = "bb.event.BuildCompleted"
> diff --git a/testsuite/unittests/test_build_uuid.py b/testsuite/unittests/test_build_uuid.py
> new file mode 100644
> index 00000000..374cce64
> --- /dev/null
> +++ b/testsuite/unittests/test_build_uuid.py
> @@ -0,0 +1,89 @@
> +# This software is a part of Isar.
> +# Copyright (C) Siemens AG, 2026
> +#
> +# SPDX-License-Identifier: MIT
> +
> +import multiprocessing
> +import os
> +import sys
> +import tempfile
> +import unittest
> +import uuid
> +from pathlib import Path
> +
> +root = Path(__file__).resolve().parents[2]
> +sys.path.extend([
> +    str(root / 'bitbake/lib'),
> +    str(root / 'meta/lib'),
> +])
> +
> +import bb
> +import bb.data
> +import bb.utils
> +from bb.parse import handle
> +
> +
> +data = bb.data.init()
> +data.setVar('BBPATH', f'{root}/meta:{root}/bitbake')
> +data.setVar('__bbclasstype', 'global')
> +parsed = handle(str(root / 'meta/classes-global/base.bbclass'), data)
> +namespace = {'bb': bb, 'os': os}
> +exec(parsed['calculate_build_uuid'], namespace)
> +exec(parsed['clear_build_uuid'], namespace)
> +calculate_build_uuid = namespace['calculate_build_uuid']
> +clear_build_uuid = namespace['clear_build_uuid']
> +
> +
> +def calculate_in_worker(persistent_dir, start, results):
> +    data = bb.data.init()
> +    data.setVar('PERSISTENT_DIR', persistent_dir)
> +    start.wait()
> +    results.put(calculate_build_uuid(data))
> +
> +
> +class TestBuildUUID(unittest.TestCase):
> +    def setUp(self):
> +        directory = tempfile.TemporaryDirectory()
> +        self.addCleanup(directory.cleanup)
> +        self.persistent_dir = os.path.join(directory.name, 'cache')
> +        self.data = bb.data.init()
> +        self.data.setVar('PERSISTENT_DIR', self.persistent_dir)
> +
> +    def test_uuid_is_reused_across_calls_and_multiconfigs(self):
> +        build_uuid = calculate_build_uuid(self.data)
> +        self.assertEqual(str(uuid.UUID(build_uuid)), build_uuid)
> +        self.assertEqual(calculate_build_uuid(self.data), build_uuid)
> +        other_config = self.data.createCopy()
> +        other_config.setVar('BB_CURRENT_MC', 'other')
> +        self.assertEqual(calculate_build_uuid(other_config), build_uuid)
> +        clear_build_uuid(self.data)
> +        self.assertNotEqual(calculate_build_uuid(self.data), build_uuid)
> +
> +    def test_concurrent_workers_share_uuid(self):
> +        context = multiprocessing.get_context('fork')
> +        start = context.Event()
> +        results = context.Queue()
> +        workers = [
> +            context.Process(
> +                target=calculate_in_worker,
> +                args=(self.persistent_dir, start, results),
> +            )
> +            for _ in range(8)
> +        ]
> +        try:
> +            for worker in workers:
> +                worker.start()
> +            start.set()
> +            values = [results.get(timeout=15) for worker in workers]
> +            for worker in workers:
> +                worker.join(timeout=15)
> +                self.assertEqual(worker.exitcode, 0)
> +            self.assertEqual(len(set(values)), 1)
> +            self.assertEqual(values[0], calculate_build_uuid(self.data))
> +        finally:
> +            for worker in workers:
> +                if worker.is_alive():
> +                    worker.terminate()
> +                worker.join(timeout=15)
> +            results.close()
> +            results.join_thread()
> -- 
> 2.56.0
> 
> -- 
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/isar-users/20261109091802.202521-2-clara.kowalsky%40siemens.com.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/ecc2fbf80cc99b3bbc0956f4d65956b650c7d888.camel%40siemens.com.

  reply	other threads:[~2026-11-10 11:18 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-09  9:17 [PATCH 0/8] Update BitBake in Isar to v2.20.0 'Clara Kowalsky' via isar-users
2026-11-09  9:17 ` [PATCH 1/8] base: replace BitBake's outdated persist_data 'Clara Kowalsky' via isar-users
2026-11-10 11:18   ` 'Moessbauer, Felix' via isar-users [this message]
2026-11-11 14:37     ` 'Clara Kowalsky' via isar-users
2026-11-09  9:17 ` [PATCH 2/8] isar-events: preserve cleanup for default multiconfig 'Clara Kowalsky' via isar-users
2026-11-09  9:17 ` [PATCH 3/8] isar-events: close /proc/mounts after cleanup 'Clara Kowalsky' via isar-users
2026-11-10 11:22   ` 'Moessbauer, Felix' via isar-users
2026-11-11 14:38     ` 'Clara Kowalsky' via isar-users
2026-11-09  9:17 ` [PATCH 4/8] meta: migrate fetcher references to bb.fetch 'Clara Kowalsky' via isar-users
2026-11-09  9:17 ` [PATCH 5/8] wic: run the Python driver outside the imaging chroot 'Clara Kowalsky' via isar-users
2026-11-10 11:39   ` 'Moessbauer, Felix' via isar-users
2026-11-11 14:39     ` 'Clara Kowalsky' via isar-users
2026-11-09  9:17 ` [PATCH 6/8] bitbake.conf: explicitly exclude export and func from signatures 'Clara Kowalsky' via isar-users
2026-11-09  9:18 ` [PATCH 7/8] meta: add missing whitespace around assignment operators 'Clara Kowalsky' via isar-users
2026-11-09  9:18 ` [PATCH 8/8] bitbake: update to v2.20.0 'Clara Kowalsky' via isar-users
2026-11-13 14:41 ` [PATCH 0/8] Update BitBake in Isar " Anton Mikanovich
2026-11-13 17:19   ` 'Clara Kowalsky' via isar-users
2026-11-16  8:46     ` 'Moessbauer, Felix' via isar-users
2026-11-16 10:44       ` 'Clara Kowalsky' via isar-users
2026-11-16 12:12         ` 'Moessbauer, Felix' via isar-users

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ecc2fbf80cc99b3bbc0956f4d65956b650c7d888.camel@siemens.com \
    --to=isar-users@googlegroups.com \
    --cc=clara.kowalsky@siemens.com \
    --cc=felix.moessbauer@siemens.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox