[PATCH v2 0/5] isar: Rebuild mount logic

[PATCH v2 0/5] isar: Rebuild mount logic

[Anton Mikanovich]

Mounting is perfomed very inconsistent currently in Isar. A lot of mount
points are left after task execution and stays mounted until cleanup in
build_completed hook.

We already experienced issues with stuck forever on unmounting at
"Couldn't unmount, retrying..." error.
It is also quite easy to face with a lot of ramaining mounts after the
failed or interrupted manual builds.

This patchset implements unmounting paths after usage. In the cases of
mount sharing corresponding request counting is performed, so real mount
will be done only at first request, real unmount will be done only after
the last task will try to unmount the path.

Meantime build_completed will use emergency only, so following warning
added to show something goes wrong.

---
Changes since v1:
- Make all mounts be unmounted after use.
- Warn about lost mounts after finish.

Known issues:
- Full CI fails for qemuarm64-stretch.

Anton Mikanovich (5):
  dpkg: Make mount buildroot reliable
  buildchroot: Unmount buildchroot if not needed
  rootfs: Unmount rootfs mounts if not needed
  wic: Unmount dirs after usage
  events: Warn if mounted paths left

 meta/classes/buildchroot.bbclass              |  58 ++++++++
 meta/classes/dpkg-base.bbclass                | 127 +++++++++++-------
 meta/classes/dpkg-gbp.bbclass                 |   8 +-
 meta/classes/dpkg.bbclass                     |  14 +-
 meta/classes/ext4-img.bbclass                 |   2 +
 meta/classes/fit-img.bbclass                  |   2 +
 meta/classes/image-tools-extension.bbclass    |   2 +
 meta/classes/image.bbclass                    |  24 ++--
 meta/classes/initramfs.bbclass                |   2 +
 meta/classes/isar-events.bbclass              |   2 +
 meta/classes/rootfs.bbclass                   |  68 +++++++++-
 meta/classes/ubi-img.bbclass                  |   2 +
 meta/classes/ubifs-img.bbclass                |   2 +
 meta/classes/wic-img.bbclass                  |  10 ++
 .../buildchroot/buildchroot.inc               |   7 +
 15 files changed, 249 insertions(+), 81 deletions(-)

-- 
2.25.1

[PATCH v2 1/5] dpkg: Make mount buildroot reliable

[Anton Mikanovich]

We use mounting for several tasks. Mounting and unmounting is performed
in shell scripts. If a command fails before unmounting, the directories
remain mounted.

Implement exception handling around the scripts to ensure that
unmounting is performed also in case of script failure. The number of
unmounting attempts has been limited to avoid infinite loops.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 126 ++++++++++++++++++++-------------
 meta/classes/dpkg-gbp.bbclass  |   8 +--
 meta/classes/dpkg.bbclass      |  14 ++--
 3 files changed, 89 insertions(+), 59 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 97661a6..d5e70f1 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -61,12 +61,7 @@ addtask patch before do_adjust_git
 
 SRC_APT ?= ""
 
-do_apt_fetch() {
-    if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then
-        return 0
-    fi
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
+fetch_apt() {
     sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
         -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
         -o Dir::Etc::SourceParts="-" \
@@ -76,21 +71,25 @@ do_apt_fetch() {
         sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
             sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
     done
+}
+
+python do_apt_fetch() {
+    src_apt = d.getVar("SRC_APT", True)
+    if not src_apt:
+        return 0
 
-    dpkg_undo_mounts
+    dpkg_do_mounts(d)
+    try:
+        isar_export_proxies(d)
+        bb.build.exec_func("fetch_apt", d)
+    finally:
+        dpkg_undo_mounts(d)
 }
 
 addtask apt_fetch after do_unpack before do_apt_unpack
 do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
-do_apt_unpack() {
-    if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then
-        return 0
-    fi
-    rm -rf ${S}
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
-
+unpack_apt() {
     for uri in "${SRC_APT}"; do
         sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
             sh -c ' \
@@ -101,8 +100,25 @@ do_apt_unpack() {
                 dpkg-source -x "${dscfile}" "${PPS}"' \
                     my_script "${DISTRO}" "${uri}"
     done
+}
 
-    dpkg_undo_mounts
+python do_apt_unpack() {
+    import shutil
+
+    src_apt = d.getVar("SRC_APT", True)
+    if not src_apt:
+        return 0
+
+    srcsubdir = d.getVar('S', True)
+    if os.path.exists(srcsubdir):
+        shutil.rmtree(srcsubdir)
+
+    dpkg_do_mounts(d)
+    try:
+        isar_export_proxies(d)
+        bb.build.exec_func("unpack_apt", d)
+    finally:
+        dpkg_undo_mounts(d)
 }
 
 addtask apt_unpack after do_apt_fetch before do_patch
@@ -146,25 +162,37 @@ do_prepare_build[deptask] = "do_deploy_deb"
 
 BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
 
-dpkg_do_mounts() {
-    mkdir -p ${BUILDROOT}
-    sudo mount --bind ${WORKDIR} ${BUILDROOT}
-
-    buildchroot_do_mounts
-}
-
-dpkg_undo_mounts() {
-    i=1
-    while ! sudo umount ${BUILDROOT}; do
-        sleep 0.1
-        i=`expr $i + 1`
-        if [ $i -gt 100 ]; then
-            bbwarn "${BUILDROOT}: Couldn't unmount, retrying..."
-            i=1
-        fi
-    done
-    sudo rmdir ${BUILDROOT}
-}
+def ismount(path):
+    real = os.path.realpath(path)
+    with open('/proc/mounts') as f:
+        for line in f.readlines():
+            if len(line.split()) > 2 and real == line.split()[1]:
+                return True
+    return False
+
+def dpkg_do_mounts(d):
+    buildroot = d.getVar('BUILDROOT', True)
+    if ismount(buildroot):
+        bb.warn('Path %s already mounted!' % buildroot)
+        return
+    workdir = d.getVar('WORKDIR', True)
+    os.makedirs(buildroot, exist_ok=True)
+    os.system('sudo mount --bind %s %s' % (workdir, buildroot))
+    bb.build.exec_func("buildchroot_do_mounts", d)
+
+def dpkg_undo_mounts(d):
+    buildroot = d.getVar('BUILDROOT', True)
+    if not ismount(buildroot):
+        bb.warn('Path %s not mounted!' % buildroot)
+        return
+    for i in range(200):
+        if not os.system('sudo umount %s' % buildroot):
+            os.rmdir(buildroot)
+            return
+        if i % 100 == 0:
+            bb.warn("%s: Couldn't unmount, retrying..." % buildroot)
+        time.sleep(0.1)
+    bb.fatal("Couldn't unmount, exiting...")
 
 # Placeholder for actual dpkg_runbuild() implementation
 dpkg_runbuild() {
@@ -174,10 +202,12 @@ dpkg_runbuild() {
 python do_dpkg_build() {
     lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
                              shared=True)
-    bb.build.exec_func("dpkg_do_mounts", d)
-    bb.build.exec_func("dpkg_runbuild", d)
-    bb.build.exec_func("dpkg_undo_mounts", d)
-    bb.utils.unlockfile(lock)
+    dpkg_do_mounts(d)
+    try:
+        bb.build.exec_func("dpkg_runbuild", d)
+    finally:
+        dpkg_undo_mounts(d)
+        bb.utils.unlockfile(lock)
 }
 
 addtask dpkg_build before do_build
@@ -221,16 +251,16 @@ python do_devshell() {
     oe_lib_path = os.path.join(d.getVar('LAYERDIR_core'), 'lib')
     sys.path.insert(0, oe_lib_path)
 
-    bb.build.exec_func('dpkg_do_mounts', d)
-
-    isar_export_proxies(d)
-
-    buildchroot = d.getVar('BUILDCHROOT_DIR')
-    pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
-    termcmd = "sudo -E chroot {0} sh -c 'cd {1}; $SHELL -i'"
-    oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d)
+    dpkg_do_mounts(d)
+    try:
+        isar_export_proxies(d)
 
-    bb.build.exec_func('dpkg_undo_mounts', d)
+        buildchroot = d.getVar('BUILDCHROOT_DIR')
+        pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
+        termcmd = "sudo -E chroot {0} sh -c 'cd {1}; $SHELL -i'"
+        oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d)
+    finally:
+        dpkg_undo_mounts(d)
 }
 
 addtask devshell after do_prepare_build
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index d956e8c..20d2d4c 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -12,12 +12,7 @@ PATCHTOOL ?= "git"
 GBP_DEPENDS ?= "git-buildpackage pristine-tar"
 GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
-do_install_builddeps_append() {
-    dpkg_do_mounts
-    distro="${DISTRO}"
-    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-       distro="${HOST_DISTRO}"
-    fi
+builddeps_install_append() {
     deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
     sudo -E chroot ${BUILDCHROOT_DIR} \
         apt-get install -y -o Debug::pkgProblemResolver=yes \
@@ -26,7 +21,6 @@ do_install_builddeps_append() {
     sudo -E chroot ${BUILDCHROOT_DIR} \
         apt-get install -y -o Debug::pkgProblemResolver=yes \
                         --no-install-recommends ${GBP_DEPENDS}
-    dpkg_undo_mounts
 }
 
 dpkg_runbuild_prepend() {
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 4e7c2f7..29e2b89 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -6,9 +6,7 @@ inherit dpkg-base
 PACKAGE_ARCH ?= "${DISTRO_ARCH}"
 
 # Install build dependencies for package
-do_install_builddeps() {
-    dpkg_do_mounts
-    E="${@ isar_export_proxies(d)}"
+builddeps_install() {
     distro="${DISTRO}"
     if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
        distro="${HOST_DISTRO}"
@@ -19,7 +17,15 @@ do_install_builddeps() {
     deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
     sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
         ${PP}/${PPS} ${PACKAGE_ARCH}
-    dpkg_undo_mounts
+}
+
+python do_install_builddeps() {
+    dpkg_do_mounts(d)
+    isar_export_proxies(d)
+    try:
+        bb.build.exec_func("builddeps_install", d)
+    finally:
+        dpkg_undo_mounts(d)
 }
 
 addtask install_builddeps after do_prepare_build before do_dpkg_build
-- 
2.25.1

[PATCH v2 2/5] buildchroot: Unmount buildchroot if not needed

[Anton Mikanovich]

Count the usage of BUILDCHROOT_DIR mount and trigger unmount of it after
all tasks using chroot are finished.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/buildchroot.bbclass           | 58 ++++++++++++++++++++++
 meta/classes/dpkg-base.bbclass             |  1 +
 meta/classes/ext4-img.bbclass              |  2 +
 meta/classes/fit-img.bbclass               |  2 +
 meta/classes/image-tools-extension.bbclass |  2 +
 meta/classes/image.bbclass                 |  9 ++++
 meta/classes/isar-events.bbclass           |  1 +
 meta/classes/ubi-img.bbclass               |  2 +
 meta/classes/ubifs-img.bbclass             |  2 +
 meta/classes/wic-img.bbclass               |  1 +
 10 files changed, 80 insertions(+)

diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index 1a8ee13..1d50025 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -21,12 +21,45 @@ python __anonymous() {
     d.setVar('BUILDCHROOT_DIR', rootfs)
 }
 
+BUILDCHROOT_CNT_LOCKFILE = "${BUILDCHROOT_DIR}.mount_lock"
+
+buildchroot_mount_cnt_inc() {
+    sudo -s <<'EOSUDO'
+        ( flock 9
+        set -e
+        count="1"
+        if [ -f '${BUILDCHROOT_DIR}/.mount' ]; then
+            count=$(($(< '${BUILDCHROOT_DIR}/.mount') + 1))
+        fi
+        echo $count | tee '${BUILDCHROOT_DIR}/.mount'
+        ) 9>'${BUILDCHROOT_CNT_LOCKFILE}'
+EOSUDO
+}
+
+buildchroot_mount_cnt_dec() {
+    sudo -s <<'EOSUDO'
+        ( flock 9
+        set -e
+        if [ -f '${BUILDCHROOT_DIR}/.mount' ]; then
+            count=$(($(< '${BUILDCHROOT_DIR}/.mount') - 1))
+            echo $count | tee '${BUILDCHROOT_DIR}/.mount'
+        else
+            exit 1
+        fi
+        ) 9>'${BUILDCHROOT_CNT_LOCKFILE}'
+EOSUDO
+}
+
 MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}.lock"
 
 buildchroot_do_mounts() {
+    if [ $(buildchroot_mount_cnt_inc) -gt 1 ]; then
+        return
+    fi
     sudo -s <<'EOSUDO'
         ( flock 9
         set -e
+        mkdir -p '${BUILDCHROOT_DIR}/isar-apt'
         mountpoint -q '${BUILDCHROOT_DIR}/isar-apt' ||
             mount --bind '${REPO_ISAR_DIR}/${DISTRO}' '${BUILDCHROOT_DIR}/isar-apt'
         mountpoint -q '${BUILDCHROOT_DIR}/downloads' ||
@@ -59,3 +92,28 @@ buildchroot_do_mounts() {
         ) 9>'${MOUNT_LOCKFILE}'
 EOSUDO
 }
+
+buildchroot_undo_mounts() {
+    if [ $(buildchroot_mount_cnt_dec) -gt 0 ]; then
+        return
+    fi
+    sudo -s <<'EOSUDO'
+        ( flock 9
+        set -e
+        mountpoint -q '${BUILDCHROOT_DIR}/base-apt' && \
+            umount ${BUILDCHROOT_DIR}/base-apt && \
+            rmdir --ignore-fail-on-non-empty ${BUILDCHROOT_DIR}/base-apt
+        mountpoint -q '${BUILDCHROOT_DIR}/sys' && \
+            umount -R ${BUILDCHROOT_DIR}/sys
+        mountpoint -q '${BUILDCHROOT_DIR}/proc' && \
+            umount ${BUILDCHROOT_DIR}/proc
+        mountpoint -q '${BUILDCHROOT_DIR}/dev' && \
+            umount -R ${BUILDCHROOT_DIR}/dev
+        mountpoint -q '${BUILDCHROOT_DIR}/downloads' && \
+            umount ${BUILDCHROOT_DIR}/downloads
+        mountpoint -q '${BUILDCHROOT_DIR}/isar-apt' && \
+            umount ${BUILDCHROOT_DIR}/isar-apt && \
+            rmdir --ignore-fail-on-non-empty ${BUILDCHROOT_DIR}/isar-apt
+        ) 9>'${MOUNT_LOCKFILE}'
+EOSUDO
+}
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index d5e70f1..ed162b3 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -181,6 +181,7 @@ def dpkg_do_mounts(d):
     bb.build.exec_func("buildchroot_do_mounts", d)
 
 def dpkg_undo_mounts(d):
+    bb.build.exec_func("buildchroot_undo_mounts", d)
     buildroot = d.getVar('BUILDROOT', True)
     if not ismount(buildroot):
         bb.warn('Path %s not mounted!' % buildroot)
diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass
index 334dc64..26e0137 100644
--- a/meta/classes/ext4-img.bbclass
+++ b/meta/classes/ext4-img.bbclass
@@ -17,6 +17,8 @@ do_ext4_image() {
 
     sudo chroot ${BUILDCHROOT_DIR} /sbin/mke2fs ${MKE2FS_ARGS} \
                 -F -d '${PP_ROOTFS}' '${PP_DEPLOY}/${EXT4_IMAGE_FILE}'
+
+    image_undo_mounts
 }
 
 addtask ext4_image before do_image after do_image_tools
diff --git a/meta/classes/fit-img.bbclass b/meta/classes/fit-img.bbclass
index 221ac3f..da8e1da 100644
--- a/meta/classes/fit-img.bbclass
+++ b/meta/classes/fit-img.bbclass
@@ -23,5 +23,7 @@ do_fit_image() {
     # Create fit image using buildchroot tools
     sudo chroot ${BUILDCHROOT_DIR} /usr/bin/mkimage ${MKIMAGE_ARGS} \
                 -f '${PP_WORK}/${FIT_IMAGE_SOURCE}' '${PP_DEPLOY}/${FIT_IMAGE_FILE}'
+
+    image_undo_mounts
 }
 addtask fit_image before do_image after do_image_tools do_transform_template
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index 0b067ff..4738479 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -40,5 +40,7 @@ do_install_imager_deps() {
         apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
             --allow-unauthenticated --allow-downgrades install \
             ${IMAGER_INSTALL}'
+
+    buildchroot_undo_mounts
 }
 addtask install_imager_deps before do_image_tools
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index ec93cab..9f9b3f8 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -62,6 +62,15 @@ image_do_mounts() {
     buildchroot_do_mounts
 }
 
+image_undo_mounts() {
+    buildchroot_undo_mounts
+    sudo flock ${MOUNT_LOCKFILE} -c ' \
+        umount "${BUILDROOT_DEPLOY}"
+        umount "${BUILDROOT_ROOTFS}"
+        umount "${BUILDROOT_WORK}"
+    '
+}
+
 ROOTFSDIR = "${IMAGE_ROOTFS}"
 ROOTFS_FEATURES += "clean-package-cache generate-manifest export-dpkg-status"
 ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
index 4c49635..1deedb9 100644
--- a/meta/classes/isar-events.bbclass
+++ b/meta/classes/isar-events.bbclass
@@ -8,6 +8,7 @@ addhandler build_started
 
 python build_started() {
     bb.utils.remove(d.getVar('TMPDIR') + "/work/*/*/*/temp/once.*")
+    os.system('sudo rm %s/work/*/*/*/rootfs/.mount' % d.getVar('TMPDIR'))
 }
 build_started[eventmask] = "bb.event.BuildStarted"
 
diff --git a/meta/classes/ubi-img.bbclass b/meta/classes/ubi-img.bbclass
index 2178b50..0f4c4cb 100644
--- a/meta/classes/ubi-img.bbclass
+++ b/meta/classes/ubi-img.bbclass
@@ -26,5 +26,7 @@ do_ubi_image() {
     # Create ubi image using buildchroot tools
     sudo chroot ${BUILDCHROOT_DIR} /usr/sbin/ubinize ${UBINIZE_ARGS} \
                 -o '${PP_DEPLOY}/${UBI_IMAGE_FILE}' '${PP_WORK}/${UBINIZE_CFG}'
+
+    image_undo_mounts
 }
 addtask ubi_image before do_image after do_image_tools do_transform_template
diff --git a/meta/classes/ubifs-img.bbclass b/meta/classes/ubifs-img.bbclass
index 78926f6..e422b46 100644
--- a/meta/classes/ubifs-img.bbclass
+++ b/meta/classes/ubifs-img.bbclass
@@ -21,6 +21,8 @@ do_ubifs_image() {
     # Create ubifs image using buildchroot tools
     sudo chroot ${BUILDCHROOT_DIR} /usr/sbin/mkfs.ubifs ${MKUBIFS_ARGS} \
                 -r '${PP_ROOTFS}' '${PP_DEPLOY}/${UBIFS_IMAGE_FILE}'
+
+    image_undo_mounts
 }
 
 addtask ubifs_image before do_image after do_image_tools
diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass
index 76606f8..01baf89 100644
--- a/meta/classes/wic-img.bbclass
+++ b/meta/classes/wic-img.bbclass
@@ -200,4 +200,5 @@ EOSUDO
     done
     rm -rf ${BUILDCHROOT_DIR}/${WICTMP}
     rm -rf ${IMAGE_ROOTFS}/../pseudo
+    buildchroot_undo_mounts
 }
-- 
2.25.1

[PATCH v2 3/5] rootfs: Unmount rootfs mounts if not needed

[Anton Mikanovich]

Count the usage of rootfs mounts and trigger unmount only after all
corresponded tasks are finished.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/image.bbclass                    | 15 ----
 meta/classes/initramfs.bbclass                |  2 +
 meta/classes/rootfs.bbclass                   | 68 +++++++++++++++++--
 .../buildchroot/buildchroot.inc               |  7 ++
 4 files changed, 70 insertions(+), 22 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 9f9b3f8..12d1616 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -199,21 +199,6 @@ do_rootfs_finalize() {
             find "${ROOTFSDIR}/usr/bin" \
                 -maxdepth 1 -name 'qemu-*-static' -type f -delete
 
-        mountpoint -q '${ROOTFSDIR}/isar-apt' && \
-            umount -l ${ROOTFSDIR}/isar-apt
-        rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/isar-apt
-
-        mountpoint -q '${ROOTFSDIR}/base-apt' && \
-            umount -l ${ROOTFSDIR}/base-apt
-        rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/base-apt
-
-        mountpoint -q '${ROOTFSDIR}/dev' && \
-            umount -l ${ROOTFSDIR}/dev
-        mountpoint -q '${ROOTFSDIR}/sys' && \
-            umount -l ${ROOTFSDIR}/proc
-        mountpoint -q '${ROOTFSDIR}/sys' && \
-            umount -l ${ROOTFSDIR}/sys
-
         rm -f "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf"
 
         rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list"
diff --git a/meta/classes/initramfs.bbclass b/meta/classes/initramfs.bbclass
index 10a642b..e895afa 100644
--- a/meta/classes/initramfs.bbclass
+++ b/meta/classes/initramfs.bbclass
@@ -38,5 +38,7 @@ do_generate_initramfs() {
 
     rm -rf "${INITRAMFS_IMAGE_FILE}"
     cp "${INITRAMFS_ROOTFS}/initrd.img" "${INITRAMFS_IMAGE_FILE}"
+
+    rootfs_undo_mounts
 }
 addtask generate_initramfs after do_rootfs before do_build
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index f9151c5..7bb04e5 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -26,8 +26,41 @@ export LANG = "C"
 export LANGUAGE = "C"
 export LC_ALL = "C"
 
+ROOTFS_CNT_LOCKFILE = "${ROOTFSDIR}.mount_lock"
+
+rootfs_mount_cnt_inc() {
+    sudo -s <<'EOSUDO'
+        ( flock 9
+        set -e
+        count="1"
+        if [ -f '${ROOTFSDIR}/.mount' ]; then
+            count=$(($(< '${ROOTFSDIR}/.mount') + 1))
+        fi
+        echo $count | tee '${ROOTFSDIR}/.mount'
+        ) 9>'${ROOTFS_CNT_LOCKFILE}'
+EOSUDO
+}
+
+rootfs_mount_cnt_dec() {
+    sudo -s <<'EOSUDO'
+        ( flock 9
+        set -e
+        if [ -f '${ROOTFSDIR}/.mount' ]; then
+            count=$(($(< '${ROOTFSDIR}/.mount') - 1))
+            echo $count | tee '${ROOTFSDIR}/.mount'
+        else
+            exit 1
+        fi
+        ) 9>'${ROOTFS_CNT_LOCKFILE}'
+EOSUDO
+}
+
+
 rootfs_do_mounts[weight] = "3"
 rootfs_do_mounts() {
+    if [ $(rootfs_mount_cnt_inc) -gt 1 ]; then
+        return
+    fi
     sudo -s <<'EOSUDO'
         mountpoint -q '${ROOTFSDIR}/dev' || \
             mount --rbind /dev '${ROOTFSDIR}/dev'
@@ -59,6 +92,26 @@ rootfs_do_mounts() {
 EOSUDO
 }
 
+rootfs_undo_mounts() {
+    if [ $(rootfs_mount_cnt_dec) -gt 0 ]; then
+        return
+    fi
+    sudo -s <<'EOSUDO'
+        mountpoint -q '${ROOTFSDIR}/base-apt' && \
+            umount ${ROOTFSDIR}/base-apt && \
+            rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/base-apt
+        mountpoint -q '${ROOTFSDIR}/isar-apt' && \
+            umount ${ROOTFSDIR}/isar-apt && \
+            rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/isar-apt
+        mountpoint -q '${ROOTFSDIR}/sys' && \
+            umount -R ${ROOTFSDIR}/sys
+        mountpoint -q '${ROOTFSDIR}/proc' && \
+            umount ${ROOTFSDIR}/proc
+        mountpoint -q '${ROOTFSDIR}/dev' && \
+            umount -R ${ROOTFSDIR}/dev
+EOSUDO
+}
+
 rootfs_do_qemu() {
     if [ '${@repr(d.getVar('ROOTFS_ARCH') == d.getVar('HOST_ARCH'))}' = 'False' ]
     then
@@ -159,7 +212,7 @@ python do_rootfs_install() {
 
     # Mount after configure commands, so that they have time to copy
     # 'isar-apt' (sdkchroot):
-    cmds = ['rootfs_prepare'] + configure_cmds + ['rootfs_do_mounts'] + install_cmds
+    cmds = ['rootfs_prepare'] + configure_cmds + ['rootfs_do_mounts'] + install_cmds  + ['rootfs_undo_mounts']
 
     # NOTE: The weights specify how long each task takes in seconds and are used
     # by the MultiStageProgressReporter to render a progress bar for this task.
@@ -241,12 +294,13 @@ python do_rootfs_postprocess() {
     progress_reporter.update(0)
 
     cmds = d.getVar("ROOTFS_POSTPROCESS_COMMAND")
-    if cmds is None or not cmds.strip():
-        return
-    cmds = cmds.split()
-    for i, cmd in enumerate(cmds):
-        bb.build.exec_func(cmd, d)
-        progress_reporter.update(int(i / len(cmds) * 100))
+    if cmds is not None and cmds.strip():
+        cmds = cmds.split()
+        for i, cmd in enumerate(cmds):
+            bb.build.exec_func(cmd, d)
+            progress_reporter.update(int(i / len(cmds) * 100))
+
+    bb.build.exec_func('rootfs_undo_mounts', d)
 }
 addtask rootfs_postprocess before do_rootfs
 
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index 5a2befb..b80e703 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -49,6 +49,13 @@ rootfs_do_mounts_append() {
 EOSUDO
 }
 
+rootfs_undo_mounts_append() {
+    sudo -s <<'EOSUDO'
+    mountpoint -q '${BUILDCHROOT_DIR}/downloads' && \
+        umount ${BUILDCHROOT_DIR}/downloads
+EOSUDO
+}
+
 ROOTFS_POSTPROCESS_COMMAND =+ "buildchroot_install_files"
 buildchroot_install_files() {
     sudo mkdir -p "${BUILDCHROOT_DIR}/home/builder"
-- 
2.25.1

[PATCH v2 4/5] wic: Unmount dirs after usage

[Anton Mikanovich]

Unmount dirs mounted at generate_wic_image task after usage.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/wic-img.bbclass | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass
index 01baf89..a293fd3 100644
--- a/meta/classes/wic-img.bbclass
+++ b/meta/classes/wic-img.bbclass
@@ -200,5 +200,14 @@ EOSUDO
     done
     rm -rf ${BUILDCHROOT_DIR}/${WICTMP}
     rm -rf ${IMAGE_ROOTFS}/../pseudo
+    sudo -s <<'EOSUDO'
+        ( flock 9
+        for dir in ${BBLAYERS} ${STAGING_DIR} ${SCRIPTSDIR} ${BITBAKEDIR}; do
+            if mountpoint -q ${BUILDCHROOT_DIR}/$dir; then
+                umount ${BUILDCHROOT_DIR}/$dir
+            fi
+        done
+        ) 9>${MOUNT_LOCKFILE}
+EOSUDO
     buildchroot_undo_mounts
 }
-- 
2.25.1

[PATCH v2 5/5] events: Warn if mounted paths left

[Anton Mikanovich]

Output warnings in case any mounts are left after build.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/isar-events.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
index 1deedb9..d0c35b6 100644
--- a/meta/classes/isar-events.bbclass
+++ b/meta/classes/isar-events.bbclass
@@ -53,6 +53,7 @@ python build_completed() {
     with open('/proc/mounts') as f:
         for line in f.readlines():
             if basepath in line:
+                bb.warn('%s left mounted, unmounting...' % line.split()[1])
                 subprocess.call(
                     ["sudo", "umount", "-l", line.split()[1]],
                     stdout=subprocess.DEVNULL,
-- 
2.25.1

Re: [PATCH v2 3/5] rootfs: Unmount rootfs mounts if not needed

[Jan Kiszka]

On 21.04.21 16:58, Anton Mikanovich wrote:
> Count the usage of rootfs mounts and trigger unmount only after all
> corresponded tasks are finished.
> 
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>  meta/classes/image.bbclass                    | 15 ----
>  meta/classes/initramfs.bbclass                |  2 +
>  meta/classes/rootfs.bbclass                   | 68 +++++++++++++++++--
>  .../buildchroot/buildchroot.inc               |  7 ++
>  4 files changed, 70 insertions(+), 22 deletions(-)
> 
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 9f9b3f8..12d1616 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -199,21 +199,6 @@ do_rootfs_finalize() {
>              find "${ROOTFSDIR}/usr/bin" \
>                  -maxdepth 1 -name 'qemu-*-static' -type f -delete
>  
> -        mountpoint -q '${ROOTFSDIR}/isar-apt' && \
> -            umount -l ${ROOTFSDIR}/isar-apt
> -        rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/isar-apt
> -
> -        mountpoint -q '${ROOTFSDIR}/base-apt' && \
> -            umount -l ${ROOTFSDIR}/base-apt
> -        rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/base-apt
> -
> -        mountpoint -q '${ROOTFSDIR}/dev' && \
> -            umount -l ${ROOTFSDIR}/dev
> -        mountpoint -q '${ROOTFSDIR}/sys' && \
> -            umount -l ${ROOTFSDIR}/proc
> -        mountpoint -q '${ROOTFSDIR}/sys' && \
> -            umount -l ${ROOTFSDIR}/sys
> -
>          rm -f "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf"
>  
>          rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list"
> diff --git a/meta/classes/initramfs.bbclass b/meta/classes/initramfs.bbclass
> index 10a642b..e895afa 100644
> --- a/meta/classes/initramfs.bbclass
> +++ b/meta/classes/initramfs.bbclass
> @@ -38,5 +38,7 @@ do_generate_initramfs() {
>  
>      rm -rf "${INITRAMFS_IMAGE_FILE}"
>      cp "${INITRAMFS_ROOTFS}/initrd.img" "${INITRAMFS_IMAGE_FILE}"
> +
> +    rootfs_undo_mounts
>  }
>  addtask generate_initramfs after do_rootfs before do_build
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index f9151c5..7bb04e5 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -26,8 +26,41 @@ export LANG = "C"
>  export LANGUAGE = "C"
>  export LC_ALL = "C"
>  
> +ROOTFS_CNT_LOCKFILE = "${ROOTFSDIR}.mount_lock"
> +
> +rootfs_mount_cnt_inc() {
> +    sudo -s <<'EOSUDO'
> +        ( flock 9
> +        set -e
> +        count="1"
> +        if [ -f '${ROOTFSDIR}/.mount' ]; then
> +            count=$(($(< '${ROOTFSDIR}/.mount') + 1))
> +        fi
> +        echo $count | tee '${ROOTFSDIR}/.mount'
> +        ) 9>'${ROOTFS_CNT_LOCKFILE}'
> +EOSUDO
> +}
> +
> +rootfs_mount_cnt_dec() {
> +    sudo -s <<'EOSUDO'
> +        ( flock 9
> +        set -e
> +        if [ -f '${ROOTFSDIR}/.mount' ]; then
> +            count=$(($(< '${ROOTFSDIR}/.mount') - 1))
> +            echo $count | tee '${ROOTFSDIR}/.mount'
> +        else
> +            exit 1
> +        fi
> +        ) 9>'${ROOTFS_CNT_LOCKFILE}'
> +EOSUDO
> +}
> +
> +
>  rootfs_do_mounts[weight] = "3"
>  rootfs_do_mounts() {
> +    if [ $(rootfs_mount_cnt_inc) -gt 1 ]; then
> +        return
> +    fi
>      sudo -s <<'EOSUDO'
>          mountpoint -q '${ROOTFSDIR}/dev' || \
>              mount --rbind /dev '${ROOTFSDIR}/dev'
> @@ -59,6 +92,26 @@ rootfs_do_mounts() {
>  EOSUDO
>  }
>  
> +rootfs_undo_mounts() {
> +    if [ $(rootfs_mount_cnt_dec) -gt 0 ]; then
> +        return
> +    fi
> +    sudo -s <<'EOSUDO'
> +        mountpoint -q '${ROOTFSDIR}/base-apt' && \
> +            umount ${ROOTFSDIR}/base-apt && \
> +            rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/base-apt
> +        mountpoint -q '${ROOTFSDIR}/isar-apt' && \
> +            umount ${ROOTFSDIR}/isar-apt && \
> +            rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/isar-apt
> +        mountpoint -q '${ROOTFSDIR}/sys' && \
> +            umount -R ${ROOTFSDIR}/sys
> +        mountpoint -q '${ROOTFSDIR}/proc' && \
> +            umount ${ROOTFSDIR}/proc
> +        mountpoint -q '${ROOTFSDIR}/dev' && \
> +            umount -R ${ROOTFSDIR}/dev
> +EOSUDO
> +}
> +
>  rootfs_do_qemu() {
>      if [ '${@repr(d.getVar('ROOTFS_ARCH') == d.getVar('HOST_ARCH'))}' = 'False' ]
>      then
> @@ -159,7 +212,7 @@ python do_rootfs_install() {
>  
>      # Mount after configure commands, so that they have time to copy
>      # 'isar-apt' (sdkchroot):
> -    cmds = ['rootfs_prepare'] + configure_cmds + ['rootfs_do_mounts'] + install_cmds
> +    cmds = ['rootfs_prepare'] + configure_cmds + ['rootfs_do_mounts'] + install_cmds  + ['rootfs_undo_mounts']
>  
>      # NOTE: The weights specify how long each task takes in seconds and are used
>      # by the MultiStageProgressReporter to render a progress bar for this task.
> @@ -241,12 +294,13 @@ python do_rootfs_postprocess() {
>      progress_reporter.update(0)
>  
>      cmds = d.getVar("ROOTFS_POSTPROCESS_COMMAND")
> -    if cmds is None or not cmds.strip():
> -        return
> -    cmds = cmds.split()
> -    for i, cmd in enumerate(cmds):
> -        bb.build.exec_func(cmd, d)
> -        progress_reporter.update(int(i / len(cmds) * 100))
> +    if cmds is not None and cmds.strip():
> +        cmds = cmds.split()
> +        for i, cmd in enumerate(cmds):
> +            bb.build.exec_func(cmd, d)
> +            progress_reporter.update(int(i / len(cmds) * 100))
> +
> +    bb.build.exec_func('rootfs_undo_mounts', d)
>  }
>  addtask rootfs_postprocess before do_rootfs
>  
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
> index 5a2befb..b80e703 100644
> --- a/meta/recipes-devtools/buildchroot/buildchroot.inc
> +++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
> @@ -49,6 +49,13 @@ rootfs_do_mounts_append() {
>  EOSUDO
>  }
>  
> +rootfs_undo_mounts_append() {
> +    sudo -s <<'EOSUDO'
> +    mountpoint -q '${BUILDCHROOT_DIR}/downloads' && \
> +        umount ${BUILDCHROOT_DIR}/downloads
> +EOSUDO
> +}
> +
>  ROOTFS_POSTPROCESS_COMMAND =+ "buildchroot_install_files"
>  buildchroot_install_files() {
>      sudo mkdir -p "${BUILDCHROOT_DIR}/home/builder"
> 

This is simply from running next in CI:

ERROR: mc:de0-nano-soc-buster:buildchroot-host-1.0-r0 do_rootfs_postprocess: Error executing a python function in exec_python_func() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:

[cut useless backtrace]

buster-armhf/buildchroot-host/1.0-r0/temp/run.rootfs_undo_mounts.65491' failed with exit code 32:
umount: /builds/BLKE9aZ_/0/ebsy/debian/isar/build/tmp/work/debian-buster-armhf/buildchroot-host/1.0-r0/rootfs/sys/fs/cgroup: target is busy.
WARNING: exit code 32 from a shell command.
ERROR: Logfile of failure stored in: /builds/BLKE9aZ_/0/ebsy/debian/isar/build/tmp/work/debian-buster-armhf/buildchroot-host/1.0-r0/temp/log.do_rootfs_postprocess.65491
NOTE: recipe buildchroot-host-1.0-r0: task do_rootfs_postprocess: Failed
ERROR: Task (mc:de0-nano-soc-buster:/builds/BLKE9aZ_/0/ebsy/debian/isar/meta/recipes-devtools/buildchroot/buildchroot-host.bb:do_rootfs_postprocess) failed with exit code '1'

It's analogous to https://groups.google.com/d/msgid/isar-users/87cef6ee-411c-b611-4fb4-3957573d9455%40siemens.com, just clearly pointing to this series.

What is the status of fixing all this? I don't see patches on the list, 
while the umounting rework is negatively impacting next since April. 
With all theses regressions, we cannot properly test unrelated fixes and 
new features, not to speak of updating downstream layer to newer Isar.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [PATCH v2 3/5] rootfs: Unmount rootfs mounts if not needed

[Anton Mikanovich]

12.08.2021 08:39, Jan Kiszka wrote:
> This is simply from running next in CI:
>
> ERROR: mc:de0-nano-soc-buster:buildchroot-host-1.0-r0 do_rootfs_postprocess: Error executing a python function in exec_python_func() autogenerated:
> The stack trace of python calls that resulted in this exception/failure was:
>
> [cut useless backtrace]
>
> buster-armhf/buildchroot-host/1.0-r0/temp/run.rootfs_undo_mounts.65491' failed with exit code 32:
> umount: /builds/BLKE9aZ_/0/ebsy/debian/isar/build/tmp/work/debian-buster-armhf/buildchroot-host/1.0-r0/rootfs/sys/fs/cgroup: target is busy.
> WARNING: exit code 32 from a shell command.
> ERROR: Logfile of failure stored in: /builds/BLKE9aZ_/0/ebsy/debian/isar/build/tmp/work/debian-buster-armhf/buildchroot-host/1.0-r0/temp/log.do_rootfs_postprocess.65491
> NOTE: recipe buildchroot-host-1.0-r0: task do_rootfs_postprocess: Failed
> ERROR: Task (mc:de0-nano-soc-buster:/builds/BLKE9aZ_/0/ebsy/debian/isar/meta/recipes-devtools/buildchroot/buildchroot-host.bb:do_rootfs_postprocess) failed with exit code '1'
>
> It's analogous to https://groups.google.com/d/msgid/isar-users/87cef6ee-411c-b611-4fb4-3957573d9455%40siemens.com, just clearly pointing to this series.

Does this and 
https://groups.google.com/d/msgid/isar-users/87cef6ee-411c-b611-4fb4-3957573d9455%40siemens.com 
have happened on CI inside docker? Let me look into this issue anyway.

-- 
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH v2 3/5] rootfs: Unmount rootfs mounts if not needed

[Jan Kiszka]

On 12.08.21 11:09, Anton Mikanovich wrote:
> 12.08.2021 08:39, Jan Kiszka wrote:
>> This is simply from running next in CI:
>>
>> ERROR: mc:de0-nano-soc-buster:buildchroot-host-1.0-r0
>> do_rootfs_postprocess: Error executing a python function in
>> exec_python_func() autogenerated:
>> The stack trace of python calls that resulted in this
>> exception/failure was:
>>
>> [cut useless backtrace]
>>
>> buster-armhf/buildchroot-host/1.0-r0/temp/run.rootfs_undo_mounts.65491' failed
>> with exit code 32:
>> umount:
>> /builds/BLKE9aZ_/0/ebsy/debian/isar/build/tmp/work/debian-buster-armhf/buildchroot-host/1.0-r0/rootfs/sys/fs/cgroup:
>> target is busy.
>> WARNING: exit code 32 from a shell command.
>> ERROR: Logfile of failure stored in:
>> /builds/BLKE9aZ_/0/ebsy/debian/isar/build/tmp/work/debian-buster-armhf/buildchroot-host/1.0-r0/temp/log.do_rootfs_postprocess.65491
>>
>> NOTE: recipe buildchroot-host-1.0-r0: task do_rootfs_postprocess: Failed
>> ERROR: Task
>> (mc:de0-nano-soc-buster:/builds/BLKE9aZ_/0/ebsy/debian/isar/meta/recipes-devtools/buildchroot/buildchroot-host.bb:do_rootfs_postprocess)
>> failed with exit code '1'
>>
>> It's analogous to
>> https://groups.google.com/d/msgid/isar-users/87cef6ee-411c-b611-4fb4-3957573d9455%40siemens.com,
>> just clearly pointing to this series.
> 
> Does this and
> https://groups.google.com/d/msgid/isar-users/87cef6ee-411c-b611-4fb4-3957573d9455%40siemens.com
> have happened on CI inside docker? Let me look into this issue anyway.
> 

Yes, in gitlab-ci. That's how we are testing because that's Isar is
generally used.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [PATCH v2 3/5] rootfs: Unmount rootfs mounts if not needed

[Anton Mikanovich]

12.08.2021 12:25, Jan Kiszka wrote:
> On 12.08.21 11:09, Anton Mikanovich wrote:
>
>> Does this and
>> https://groups.google.com/d/msgid/isar-users/87cef6ee-411c-b611-4fb4-3957573d9455%40siemens.com
>> have happened on CI inside docker? Let me look into this issue anyway.
> Yes, in gitlab-ci. That's how we are testing because that's Isar is
> generally used.
>
> Jan
>
Ok, I've reproduced the issue inside both docker and chroot environment. 
Have added some additional debug, hope it will help to find the reason.
 From the first view it looks pretty much like the issue which forced 
Isar to use lazy unmounts (it was never reproduced in Jenkins before).
Will continue investigation to find the real root cause.

-- 
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH v2 3/5] rootfs: Unmount rootfs mounts if not needed

[Jan Kiszka]

On 13.08.21 18:27, Anton Mikanovich wrote:
> 12.08.2021 12:25, Jan Kiszka wrote:
>> On 12.08.21 11:09, Anton Mikanovich wrote:
>>
>>> Does this and
>>> https://groups.google.com/d/msgid/isar-users/87cef6ee-411c-b611-4fb4-3957573d9455%40siemens.com
>>>
>>> have happened on CI inside docker? Let me look into this issue anyway.
>> Yes, in gitlab-ci. That's how we are testing because that's Isar is
>> generally used.
>>
>> Jan
>>
> Ok, I've reproduced the issue inside both docker and chroot environment.
> Have added some additional debug, hope it will help to find the reason.
> From the first view it looks pretty much like the issue which forced
> Isar to use lazy unmounts (it was never reproduced in Jenkins before).
> Will continue investigation to find the real root cause.
> 

Good to hear that you can reproduce it. Maybe this allows to solve the
issue better this time, non-lazily. Even if not, it gives at least the
chance to document to reasons more clearly.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux