public inbox for isar-users@googlegroups.com
 help / color / mirror / Atom feed
From: "'Clara Kowalsky' via isar-users" <isar-users@googlegroups.com>
To: "Moessbauer,
	Felix (FT RPD CPC OES-DE)" <felix.moessbauer@siemens.com>,
	"isar-users@googlegroups.com" <isar-users@googlegroups.com>
Subject: Re: [PATCH 1/8] base: replace BitBake's outdated persist_data
Date: Wed, 11 Nov 2026 15:37:45 +0100	[thread overview]
Message-ID: <0d7832ad-aa96-49c0-8e2d-502c5be233be@siemens.com> (raw)
In-Reply-To: <ecc2fbf80cc99b3bbc0956f4d65956b650c7d888.camel@siemens.com>



On 11/10/26 12:18, Moessbauer, Felix (FT RPD CPC OES-DE) wrote:
> On Mon, 2026-11-09 at 10:17 +0100, 'Clara Kowalsky' via isar-users
> wrote:
>> Isar uses a persistent UUID to distinguish build-specific resources,
>> including sbuild chroot names.
>>
> 
> The only user I can find is schroot. This will hopefully go away rather
> sooner than later. Given that, I'm wondering if we really need the unit
> test.
> 
> Anyways, the change is fine.
> 
> Felix

Yes, schroot is currently the only consumer. We can remove these bits 
when schroot support is dropped.

Clara
> 
>> The UUID must remain stable throughout a
>> BitBake run, so parallel builds using separate PID namespaces do not
>> clash on external mountpoints. It must also be shared across
>> multiconfigs and cleared when the build completes.
>> Isar currently stores the UUID through bb.persist_data.persist, which
>> is removed in BitBake v2.9.2 commit 681a7516 ("persist_data: Remove
>> it") from October 2024.
>>
>> Persist the UUID in a file under PERSISTENT_DIR instead. Use BitBake's
>> file-locking helper when reading, creating and clearing it to ensure
>> concurrent processes share the same UUID. Clear it on the default
>> BuildCompleted event, so the next run receives a fresh UUID.
>>
>> Add unittests for UUID reuse across calls and multiconfigs, cleanup and
>> concurrent access.
>>
>> Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com>
>> ---
>>   meta/classes-global/base.bbclass        | 25 +++++--
>>   meta/classes-global/isar-events.bbclass |  2 +-
>>   testsuite/unittests/test_build_uuid.py  | 89 +++++++++++++++++++++++++
>>   3 files changed, 109 insertions(+), 7 deletions(-)
>>   create mode 100644 testsuite/unittests/test_build_uuid.py
>>
>> diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
>> index 3d5d1c12..c61e59b4 100644
>> --- a/meta/classes-global/base.bbclass
>> +++ b/meta/classes-global/base.bbclass
>> @@ -341,12 +341,25 @@ def base_set_filespath(path, d):
>>   def calculate_build_uuid(d):
>>       import uuid
>>   
>> -    uuid_data = bb.persist_data.persist('BB_ISAR_UUID_DATA', d)
>> -    if "uuid" not in uuid_data or not uuid_data["uuid"]:
>> -        # Generate new UUID
>> -        uuid_data["uuid"] = str(uuid.uuid4())
>> -
>> -    return uuid_data["uuid"]
>> +    uuid_file = os.path.join(d.getVar('PERSISTENT_DIR'), 'isar-build-uuid')
>> +    with bb.utils.fileslocked([uuid_file + '.lock']):
>> +        try:
>> +            with open(uuid_file) as file:
>> +                build_uuid = file.read().strip()
>> +        except FileNotFoundError:
>> +            build_uuid = ''
>> +
>> +        if not build_uuid:
>> +            build_uuid = str(uuid.uuid4())
>> +            with open(uuid_file, 'w') as file:
>> +                file.write(build_uuid + '\n')
>> +
>> +        return build_uuid
>> +
>> +def clear_build_uuid(d):
>> +    uuid_file = os.path.join(d.getVar('PERSISTENT_DIR'), 'isar-build-uuid')
>> +    with bb.utils.fileslocked([uuid_file + '.lock']):
>> +        bb.utils.remove(uuid_file)
>>   
>>   # Unique ID for this build, used to avoid name clashes on external mountpoints
>>   # When running parallel builds in different PID namespaces
>> diff --git a/meta/classes-global/isar-events.bbclass b/meta/classes-global/isar-events.bbclass
>> index 15bfdb99..cfe6d7dd 100644
>> --- a/meta/classes-global/isar-events.bbclass
>> +++ b/meta/classes-global/isar-events.bbclass
>> @@ -85,7 +85,7 @@ python build_completed() {
>>                   bb.error(str(e))
>>   
>>       # Cleanup build UUID, the next bitbake run will generate new one
>> -    bb.persist_data.persist('BB_ISAR_UUID_DATA', d).clear()
>> +    bb.utils.exec_flat_python_func('clear_build_uuid', d)
>>   }
>>   
>>   build_completed[eventmask] = "bb.event.BuildCompleted"
>> diff --git a/testsuite/unittests/test_build_uuid.py b/testsuite/unittests/test_build_uuid.py
>> new file mode 100644
>> index 00000000..374cce64
>> --- /dev/null
>> +++ b/testsuite/unittests/test_build_uuid.py
>> @@ -0,0 +1,89 @@
>> +# This software is a part of Isar.
>> +# Copyright (C) Siemens AG, 2026
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +import multiprocessing
>> +import os
>> +import sys
>> +import tempfile
>> +import unittest
>> +import uuid
>> +from pathlib import Path
>> +
>> +root = Path(__file__).resolve().parents[2]
>> +sys.path.extend([
>> +    str(root / 'bitbake/lib'),
>> +    str(root / 'meta/lib'),
>> +])
>> +
>> +import bb
>> +import bb.data
>> +import bb.utils
>> +from bb.parse import handle
>> +
>> +
>> +data = bb.data.init()
>> +data.setVar('BBPATH', f'{root}/meta:{root}/bitbake')
>> +data.setVar('__bbclasstype', 'global')
>> +parsed = handle(str(root / 'meta/classes-global/base.bbclass'), data)
>> +namespace = {'bb': bb, 'os': os}
>> +exec(parsed['calculate_build_uuid'], namespace)
>> +exec(parsed['clear_build_uuid'], namespace)
>> +calculate_build_uuid = namespace['calculate_build_uuid']
>> +clear_build_uuid = namespace['clear_build_uuid']
>> +
>> +
>> +def calculate_in_worker(persistent_dir, start, results):
>> +    data = bb.data.init()
>> +    data.setVar('PERSISTENT_DIR', persistent_dir)
>> +    start.wait()
>> +    results.put(calculate_build_uuid(data))
>> +
>> +
>> +class TestBuildUUID(unittest.TestCase):
>> +    def setUp(self):
>> +        directory = tempfile.TemporaryDirectory()
>> +        self.addCleanup(directory.cleanup)
>> +        self.persistent_dir = os.path.join(directory.name, 'cache')
>> +        self.data = bb.data.init()
>> +        self.data.setVar('PERSISTENT_DIR', self.persistent_dir)
>> +
>> +    def test_uuid_is_reused_across_calls_and_multiconfigs(self):
>> +        build_uuid = calculate_build_uuid(self.data)
>> +        self.assertEqual(str(uuid.UUID(build_uuid)), build_uuid)
>> +        self.assertEqual(calculate_build_uuid(self.data), build_uuid)
>> +        other_config = self.data.createCopy()
>> +        other_config.setVar('BB_CURRENT_MC', 'other')
>> +        self.assertEqual(calculate_build_uuid(other_config), build_uuid)
>> +        clear_build_uuid(self.data)
>> +        self.assertNotEqual(calculate_build_uuid(self.data), build_uuid)
>> +
>> +    def test_concurrent_workers_share_uuid(self):
>> +        context = multiprocessing.get_context('fork')
>> +        start = context.Event()
>> +        results = context.Queue()
>> +        workers = [
>> +            context.Process(
>> +                target=calculate_in_worker,
>> +                args=(self.persistent_dir, start, results),
>> +            )
>> +            for _ in range(8)
>> +        ]
>> +        try:
>> +            for worker in workers:
>> +                worker.start()
>> +            start.set()
>> +            values = [results.get(timeout=15) for worker in workers]
>> +            for worker in workers:
>> +                worker.join(timeout=15)
>> +                self.assertEqual(worker.exitcode, 0)
>> +            self.assertEqual(len(set(values)), 1)
>> +            self.assertEqual(values[0], calculate_build_uuid(self.data))
>> +        finally:
>> +            for worker in workers:
>> +                if worker.is_alive():
>> +                    worker.terminate()
>> +                worker.join(timeout=15)
>> +            results.close()
>> +            results.join_thread()
>> -- 
>> 2.56.0
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups "isar-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
>> To view this discussion visit https://groups.google.com/d/msgid/isar-users/20261109091802.202521-2-clara.kowalsky%40siemens.com.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/0d7832ad-aa96-49c0-8e2d-502c5be233be%40siemens.com.

  reply	other threads:[~2026-11-11 14:38 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-09  9:17 [PATCH 0/8] Update BitBake in Isar to v2.20.0 'Clara Kowalsky' via isar-users
2026-11-09  9:17 ` [PATCH 1/8] base: replace BitBake's outdated persist_data 'Clara Kowalsky' via isar-users
2026-11-10 11:18   ` 'Moessbauer, Felix' via isar-users
2026-11-11 14:37     ` 'Clara Kowalsky' via isar-users [this message]
2026-11-09  9:17 ` [PATCH 2/8] isar-events: preserve cleanup for default multiconfig 'Clara Kowalsky' via isar-users
2026-11-09  9:17 ` [PATCH 3/8] isar-events: close /proc/mounts after cleanup 'Clara Kowalsky' via isar-users
2026-11-10 11:22   ` 'Moessbauer, Felix' via isar-users
2026-11-11 14:38     ` 'Clara Kowalsky' via isar-users
2026-11-09  9:17 ` [PATCH 4/8] meta: migrate fetcher references to bb.fetch 'Clara Kowalsky' via isar-users
2026-11-09  9:17 ` [PATCH 5/8] wic: run the Python driver outside the imaging chroot 'Clara Kowalsky' via isar-users
2026-11-10 11:39   ` 'Moessbauer, Felix' via isar-users
2026-11-11 14:39     ` 'Clara Kowalsky' via isar-users
2026-11-09  9:17 ` [PATCH 6/8] bitbake.conf: explicitly exclude export and func from signatures 'Clara Kowalsky' via isar-users
2026-11-09  9:18 ` [PATCH 7/8] meta: add missing whitespace around assignment operators 'Clara Kowalsky' via isar-users
2026-11-09  9:18 ` [PATCH 8/8] bitbake: update to v2.20.0 'Clara Kowalsky' via isar-users
2026-11-13 14:41 ` [PATCH 0/8] Update BitBake in Isar " Anton Mikanovich
2026-11-13 17:19   ` 'Clara Kowalsky' via isar-users
2026-11-16  8:46     ` 'Moessbauer, Felix' via isar-users
2026-11-16 10:44       ` 'Clara Kowalsky' via isar-users
2026-11-16 12:12         ` 'Moessbauer, Felix' via isar-users

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=0d7832ad-aa96-49c0-8e2d-502c5be233be@siemens.com \
    --to=isar-users@googlegroups.com \
    --cc=clara.kowalsky@siemens.com \
    --cc=felix.moessbauer@siemens.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox