[RFC] add uuid to schroot folder

[RFC] add uuid to schroot folder

[Anton Mikanovich]

PIDs are not unique across containers.
When running the build in a container (e.g. the kas container),
the PID of bitbake is likely the same across multiple simultaneously
running builds.

This is especially the case for CI runners, where it is common that
multiple jobs run in parallel.

This patch adds an additional UUID component stored till the end of
the build in bitbake persistend storage.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/base.bbclass        | 14 ++++++++++++++
 meta/classes/isar-events.bbclass |  3 +++
 meta/classes/sbuild.bbclass      |  2 +-
 meta/conf/bitbake.conf           |  2 +-
 4 files changed, 19 insertions(+), 2 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 4ec2c81..1d954de 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -278,3 +278,17 @@ def base_set_filespath(path, d):
             if p != "":
                 filespath.append(os.path.join(p, o))
     return ":".join(filespath)
+
+def calculate_build_uuid(d):
+    import uuid
+
+    uuid_data = bb.persist_data.persist('BB_ISAR_UUID_DATA', d)
+    if "uuid" not in uuid_data or not uuid_data["uuid"]:
+        # Generate new UUID
+        uuid_data["uuid"] = str(uuid.uuid4())
+
+    return uuid_data["uuid"]
+
+# Unique ID for this build, used to avoid name clashes on external mountpoints
+# When running parallel builds in different PID namespaces
+ISAR_BUILD_UUID = "${@ calculate_build_uuid(d)}"
diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
index 92aff20..f52b234 100644
--- a/meta/classes/isar-events.bbclass
+++ b/meta/classes/isar-events.bbclass
@@ -60,6 +60,9 @@ python build_completed() {
                     stdout=subprocess.DEVNULL,
                     stderr=subprocess.DEVNULL,
                 )
+
+    # Cleanup build UUID, the next bitbake run will generate new one
+    bb.persist_data.persist('BB_ISAR_UUID_DATA', d).clear()
 }
 
 build_completed[eventmask] = "bb.event.BuildCompleted"
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index f9bfaaf..7c39954 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -23,7 +23,7 @@ python __anonymous() {
     d.setVar('SCHROOT_DEP', dep)
 }
 
-SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${@os.getpid()}"
+SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${ISAR_BUILD_UUID}-${@os.getpid()}"
 
 SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
 SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 6451cb5..cadfbdd 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -104,7 +104,7 @@ BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI
     WORKDIR STAMPCLEAN PKGDATA_DIR BUILD_ARCH SSTATE_PKGARCH \
     BB_WORKERCONTEXT BB_LIMITEDDEPS DEPLOY_DIR BUILDCHROOT_DIR \
     REPO_ISAR_DIR REPO_ISAR_DB_DIR REPO_BASE_DIR REPO_BASE_DB_DIR LAYERDIR_core \
-    SCRIPTSDIR TOPDIR"
+    SCRIPTSDIR TOPDIR ISAR_BUILD_UUID"
 BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SSH_AGENT_PID \
     SSH_AUTH_SOCK PSEUDO_BUILD BB_ENV_EXTRAWHITE DISABLE_SANITY_CHECKS \
     BB_NUMBER_THREADS BB_ORIGENV BB_INVALIDCONF BBINCLUDED \
-- 
2.17.1

Re: [RFC] add uuid to schroot folder

[Anton Mikanovich]

[-- Attachment #1: Type: text/plain, Size: 748 bytes --]

11.07.2022 21:31, Anton Mikanovich wrote:
> PIDs are not unique across containers.
> When running the build in a container (e.g. the kas container),
> the PID of bitbake is likely the same across multiple simultaneously
> running builds.
>
> This is especially the case for CI runners, where it is common that
> multiple jobs run in parallel.
>
> This patch adds an additional UUID component stored till the end of
> the build in bitbake persistend storage.
>
> Signed-off-by: Anton Mikanovich<amikan@ilbers.de>
> Signed-off-by: Felix Moessbauer<felix.moessbauer@siemens.com>

Here is the version I propose as an alternative to original patch.
Main differences are:
1) placing everything inside bitbake
2) regeneration of UUID on every bitbake run

[-- Attachment #2: Type: text/html, Size: 1554 bytes --]

RE: [RFC] add uuid to schroot folder

[Moessbauer, Felix]

> From: Anton Mikanovich <amikan@ilbers.de> 
> Sent: Monday, July 11, 2022 8:35 PM
> To: isar-users@googlegroups.com; Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>
> Subject: Re: [RFC] add uuid to schroot folder
>
> 11.07.2022 21:31, Anton Mikanovich wrote:
> PIDs are not unique across containers.
> When running the build in a container (e.g. the kas container),
> the PID of bitbake is likely the same across multiple simultaneously
> running builds.
>
> This is especially the case for CI runners, where it is common that
> multiple jobs run in parallel.
>
> This patch adds an additional UUID component stored till the end of
> the build in bitbake persistend storage.
>
> Signed-off-by: Anton Mikanovich mailto:amikan@ilbers.de
> Signed-off-by: Felix Moessbauer mailto:felix.moessbauer@siemens.com
> Here is the version I propose as an alternative to original patch.
> Main differences are:
> 1) placing everything inside bitbake

I tried that as well but failed at writing the code in a way that BB does not complain about non-deterministic metadata.
If that works, your solution is much better.

Felix

> 2) regeneration of UUID on every bitbake run

Re: [RFC] add uuid to schroot folder

[Anton Mikanovich]

12.07.2022 16:38, Moessbauer, Felix wrote:
> I tried that as well but failed at writing the code in a way that BB does not complain about non-deterministic metadata.
> If that works, your solution is much better.
>
> Felix

So can I resend it as [PATCH v4] then?

RE: [RFC] add uuid to schroot folder

[Moessbauer, Felix]

> -----Original Message-----
> From: Anton Mikanovich <amikan@ilbers.de>
> Sent: Wednesday, July 13, 2022 3:23 PM
> To: Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>; isar-
> users@googlegroups.com
> Subject: Re: [RFC] add uuid to schroot folder
> 
> 12.07.2022 16:38, Moessbauer, Felix wrote:
> > I tried that as well but failed at writing the code in a way that BB does not
> complain about non-deterministic metadata.
> > If that works, your solution is much better.
> >
> > Felix
> 
> So can I resend it as [PATCH v4] then?

Yes, sure.
May I ask you to add this patch to a branch on Github as well?
As an outlook user, I have a hard time to apply patches from the ML (as outlook cannot generate mbox files).

Felix