[PATCH 07/13] bitbake: backport cooker: use BB_HASHSERVE_DB_DIR for hash server database location

["'Felix Moessbauer' via isar-users" <isar-users@googlegroups.com>]

Backport of upstream commit b339d05ad2b69a6518522ee4c46dd5f5a6e33f65.

If unset, the existing behavior is preseved.

The use case is sharing the database in bitbake-setup's driven builds
without having to set up/start/stop a common single server shared
between them (this is added to bitbake-setup in the next commit).

Also create the specified directory if it doesn't yet exist.

Add a check that the directory is not on a NFS path, and error out then,
guiding the user to set up a standalone hash equivalency server.

Note: the check runs 'stat' executable with a format string parameter,
ensuring it prints only the filesystem type and nothing else.
Python's standard library does not have a statfs() wrapper,
and using ctypes to call into it would've required delicate,
crash-prone data type definitions, full of magic numbers.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 .../bitbake-user-manual-ref-variables.rst     | 23 +++++++++++++++++++
 bitbake/lib/bb/cooker.py                      | 11 ++++++++-
 bitbake/lib/bb/utils.py                       |  9 ++++++++
 3 files changed, 42 insertions(+), 1 deletion(-)

diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
index 899e584f..c41df199 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
@@ -414,6 +414,29 @@ overview of their function and contents.
       equivalences that correspond to Share State caches that are
       only available on specific clients.
 
+   :term:`BB_HASHSERVE_DB_DIR`
+      When :term:`BB_HASHSERVE` is set to ``auto``, bitbake will use
+      a private location inside a particular build directory for the sqlite
+      database file that holds hash equivalency data.
+
+      This variable allows using a different path, which can be shared
+      between multiple build directories and bitbake instances
+      that operate on them. This enables using a common ``${SSTATE_DIR}``
+      together with common hash equivalency data for local builds, without having to
+      separately manage a hash equivalency server.
+
+      .. note::
+
+         This variable cannot be set to a NFS mount and bitbake will error out then.
+         The reason is that NFS implementations can have file locking issues, which
+         can cause data loss and corruption when there are multiple writers operating
+         on a file at the same time as explained in https://sqlite.org/faq.html#q5
+
+         If you'd like to share hash equivalency data between multiple computers, you
+         need to set up a hash equivalency server separately and point :term:`BB_HASHSERVE`
+         to it. See https://docs.yoctoproject.org/dev-manual/hashequivserver.html for
+         additional information.
+
    :term:`BB_HASHSERVE_UPSTREAM`
       Specifies an upstream Hash Equivalence server.
 
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 701cf51b..8ac5e07a 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -312,7 +312,16 @@ class BBCooker:
         if self.data.getVar("BB_HASHSERVE") == "auto":
             # Create a new hash server bound to a unix domain socket
             if not self.hashserv:
-                dbfile = (self.data.getVar("PERSISTENT_DIR") or self.data.getVar("CACHE")) + "/hashserv.db"
+                bb_hashserve_db_dir = self.data.getVar("BB_HASHSERVE_DB_DIR")
+                if bb_hashserve_db_dir and utils.is_path_on_nfs(bb_hashserve_db_dir):
+                    bb.fatal("""Hash equivalency database location (set via BB_HASHSERVE_DB_DIR to {})
+cannot be on a NFS mount due to potential NFS locking issues between sqlite clients, per https://sqlite.org/faq.html#q5
+
+If you need to share the database between several computers, set up a permanently running hash equivalency server
+according to https://docs.yoctoproject.org/dev-manual/hashequivserver.html""".format(bb_hashserve_db_dir))
+                dbdir = bb_hashserve_db_dir or self.data.getVar("PERSISTENT_DIR") or self.data.getVar("CACHE")
+                os.makedirs(dbdir, exist_ok=True)
+                dbfile = dbdir + "/hashserv.db"
                 upstream = self.data.getVar("BB_HASHSERVE_UPSTREAM") or None
                 if upstream:
                     import socket
diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index ebee65d3..3866c1b8 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -1866,3 +1866,12 @@ def lock_timeout(lock):
         yield held
     finally:
         lock.release()
+
+
+def is_path_on_nfs(path):
+    """
+    Returns True if ``path`` argument is on a NFS mount.
+    """
+    import bb.process
+    fstype = bb.process.run("stat -f -c %T {}".format(path))[0].strip()
+    return fstype == "nfs"
-- 
2.55.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20261002144936.246628-8-felix.moessbauer%40siemens.com.