[PATCH v3 0/3] Fix sporadic failures in do_wic_image

[PATCH v3 0/3] Fix sporadic failures in do_wic_image

[Felix Moessbauer]

Changes since v2:
- only lock current buildchroot (only for locking itself)
- do not umount: use mount cleanup handler to unmount all mountpoints
- add patch to show sub-progress of do_wic_image task (helpful for debugging)

Changes since v1:
- remove patches 3 and 4 (refactoring and breakup of do_rootfs_finalize task)
- use exclusive lock in do_wic_image

Best regards,
Felix


Felix Moessbauer (3):
  fix typo in do_rootfs_finalize
  do not unmount wic mounts to avoid over-locking
  display sub-task progress in wic-image

 meta/classes/image.bbclass   |  2 +-
 meta/classes/wic-img.bbclass | 24 ++++++++++++------------
 2 files changed, 13 insertions(+), 13 deletions(-)

-- 
2.30.2

[PATCH v3 1/3] fix typo in do_rootfs_finalize

[Felix Moessbauer]

This fixes a typo to make sure the same mountpoint is checked for
existence and unmounted.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/image.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index ec93cab..1f39690 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -200,7 +200,7 @@ do_rootfs_finalize() {
 
         mountpoint -q '${ROOTFSDIR}/dev' && \
             umount -l ${ROOTFSDIR}/dev
-        mountpoint -q '${ROOTFSDIR}/sys' && \
+        mountpoint -q '${ROOTFSDIR}/proc' && \
             umount -l ${ROOTFSDIR}/proc
         mountpoint -q '${ROOTFSDIR}/sys' && \
             umount -l ${ROOTFSDIR}/sys
-- 
2.30.2

[PATCH v3 2/3] do not unmount wic mounts to avoid over-locking

[Felix Moessbauer]

This patch reworks the mounting in the do_wic_image task to ensure
that mountpoints are not unmounted by a simultaneously running wic task.
The mounted directories will finally be cleaned up by the unmount
event handler.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/wic-img.bbclass | 15 +++++----------
 1 file changed, 5 insertions(+), 10 deletions(-)

diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass
index d849ad9..9df40a9 100644
--- a/meta/classes/wic-img.bbclass
+++ b/meta/classes/wic-img.bbclass
@@ -138,12 +138,13 @@ python check_for_wic_warnings() {
 
 do_wic_image[file-checksums] += "${WKS_FILE_CHECKSUM}"
 python do_wic_image() {
+    bb.build.exec_func("wic_do_mounts", d)
     bb.build.exec_func("generate_wic_image", d)
     bb.build.exec_func("check_for_wic_warnings", d)
 }
 addtask wic_image before do_image after do_image_tools
 
-generate_wic_image() {
+wic_do_mounts() {
     buildchroot_do_mounts
     sudo -s <<'EOSUDO'
         ( flock 9
@@ -155,6 +156,9 @@ generate_wic_image() {
         done
         ) 9>${MOUNT_LOCKFILE}
 EOSUDO
+}
+
+generate_wic_image() {
     export FAKEROOTCMD=${FAKEROOTCMD}
     export BUILDDIR=${BUILDDIR}
     export MTOOLS_SKIP_CHECK=1
@@ -200,13 +204,4 @@ EOSUDO
     done
     rm -rf ${BUILDCHROOT_DIR}/${WICTMP}
     rm -rf ${IMAGE_ROOTFS}/../pseudo
-    sudo -s <<'EOSUDO'
-        ( flock 9
-        for dir in ${BBLAYERS} ${STAGING_DIR} ${SCRIPTSDIR} ${BITBAKEDIR}; do
-            if mountpoint -q ${BUILDCHROOT_DIR}/$dir; then
-                umount ${BUILDCHROOT_DIR}/$dir
-            fi
-        done
-        ) 9>${MOUNT_LOCKFILE}
-EOSUDO
 }
-- 
2.30.2

[PATCH v3 3/3] display sub-task progress in wic-image

[Felix Moessbauer]

Displaying the sub-process helps to discover locking / dead-lock
issues as is exposes which task got the lock (the one with more progess).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/wic-img.bbclass | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass
index 9df40a9..caad1b1 100644
--- a/meta/classes/wic-img.bbclass
+++ b/meta/classes/wic-img.bbclass
@@ -138,9 +138,14 @@ python check_for_wic_warnings() {
 
 do_wic_image[file-checksums] += "${WKS_FILE_CHECKSUM}"
 python do_wic_image() {
-    bb.build.exec_func("wic_do_mounts", d)
-    bb.build.exec_func("generate_wic_image", d)
-    bb.build.exec_func("check_for_wic_warnings", d)
+    cmds = ['wic_do_mounts', 'generate_wic_image', 'check_for_wic_warnings']
+    weights = [5, 90, 5]
+    progress_reporter = bb.progress.MultiStageProgressReporter(d, weights)
+
+    for cmd in cmds:
+        progress_reporter.next_stage()
+        bb.build.exec_func(cmd, d)
+    progress_reporter.finish()
 }
 addtask wic_image before do_image after do_image_tools
 
-- 
2.30.2

Re: [PATCH v3 0/3] Fix sporadic failures in do_wic_image

[Anton Mikanovich]

On 27.09.21 16:14, Felix Moessbauer wrote:
> Changes since v2:
> - only lock current buildchroot (only for locking itself)
> - do not umount: use mount cleanup handler to unmount all mountpoints
> - add patch to show sub-progress of do_wic_image task (helpful for debugging)
>
> Changes since v1:
> - remove patches 3 and 4 (refactoring and breakup of do_rootfs_finalize task)
> - use exclusive lock in do_wic_image
>
> Best regards,
> Felix

Applied to next, thanks.

-- 
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov