[PATCH 0/3] umount fixes and cleanups

[PATCH 0/3] umount fixes and cleanups

[Jan Kiszka]

Patches related to resolving the pending CI issues as well as
simplifying the umounts used during cleanup.

Jan

Jan Kiszka (3):
  ci: Wait for bitbake worker to finish before deleting artifacts
  isar-events: Improve umount handler
  Remove redundant recursive umounts

 meta/classes/isar-events.bbclass                   | 38 ++++++++--------------
 meta/classes/isar-image.bbclass                    |  4 +--
 .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  4 +--
 scripts/ci_build.sh                                |  1 +
 4 files changed, 18 insertions(+), 29 deletions(-)

-- 
2.16.4

[PATCH 1/3] ci: Wait for bitbake worker to finish before deleting artifacts

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

bitbake completes asynchronously to the worker daemon. Therefore, we
must not start deleting its tmp folder before the termination because it
may contain active mounts.

Wait for the socket that links frontend and backend to go away. This is
a reliably sign that the worker finished as well.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/ci_build.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 20b77b7..aae9084 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -92,6 +92,7 @@ if [ -n "$FAST_BUILD" ]; then
         multiconfig:qemuarm-stretch:isar-image-base \
         multiconfig:qemuarm64-stretch:isar-image-base \
         multiconfig:qemuamd64-stretch:isar-image-base
+    while [ -e bitbake.sock ]; do sleep 1; done
     sudo rm -rf tmp
     sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
     bitbake $BB_ARGS \
-- 
2.16.4

[PATCH 2/3] isar-events: Improve umount handler

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Using lazy umount avoids having to loop over the mount points.
Furthermore, also triggering on runQueueExitWait allows to perform the
umount earlier. Finally, filter for the events we actually want.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/isar-events.bbclass | 38 +++++++++++++-------------------------
 1 file changed, 13 insertions(+), 25 deletions(-)

diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
index b3ce434..05b27e5 100644
--- a/meta/classes/isar-events.bbclass
+++ b/meta/classes/isar-events.bbclass
@@ -6,34 +6,22 @@
 
 addhandler isar_handler
 
-python isar_handler () {
+python isar_handler() {
     import subprocess
-    import time
+    import bb.runqueue
 
-    def umount_all(basepath):
-        # '/proc/mounts' contains all the active mounts, so knowing basepath
-        # we can get the list of mounts for the specific multiconfig and
-        # clean them.
-        result = True
+    tmpdir = d.getVar('TMPDIR', True)
+    if not tmpdir:
+        return
+
+    basepath = tmpdir + '/work/'
+
+    with open(os.devnull, 'w') as devnull:
         with open('/proc/mounts', 'rU') as f:
             for line in f:
                 if basepath in line:
-                    if subprocess.call('sudo umount ' + line.split()[1],
-                                       stdout=devnull, stderr=devnull,
-                                       shell=True) != 0:
-                        result = False
-        return result
-
-    devnull = open(os.devnull, 'w')
-
-    if isinstance(e, bb.event.BuildCompleted):
-        tmpdir = d.getVar('TMPDIR', True)
-
-        if tmpdir:
-            basepath = tmpdir + '/work/'
-
-            while not umount_all(basepath):
-                time.sleep(1)
-
-    devnull.close()
+                    subprocess.call('sudo umount -l ' + line.split()[1],
+                                    stdout=devnull, stderr=devnull, shell=True)
 }
+
+isar_handler[eventmask] = "bb.runqueue.runQueueExitWait bb.event.BuildCompleted"
-- 
2.16.4

[PATCH 3/3] Remove redundant recursive umounts

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Lazily umounting also ensures that depending mounts are removed.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/isar-image.bbclass                     | 4 ++--
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/classes/isar-image.bbclass b/meta/classes/isar-image.bbclass
index 23b3a99..14b97dd 100644
--- a/meta/classes/isar-image.bbclass
+++ b/meta/classes/isar-image.bbclass
@@ -59,9 +59,9 @@ isar_image_cleanup() {
               -maxdepth 1 -name 'qemu-*-static' -type f -delete
     sudo umount -l ${IMAGE_ROOTFS}/isar-apt
     sudo rmdir ${IMAGE_ROOTFS}/isar-apt
-    sudo umount -R -l ${IMAGE_ROOTFS}/dev
+    sudo umount -l ${IMAGE_ROOTFS}/dev
     sudo umount -l ${IMAGE_ROOTFS}/proc
-    sudo umount -R -l ${IMAGE_ROOTFS}/sys
+    sudo umount -l ${IMAGE_ROOTFS}/sys
     sudo rm -f "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf"
     if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
         sudo umount -l ${IMAGE_ROOTFS}/base-apt
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index da077d0..811d50e 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -254,9 +254,9 @@ isar_bootstrap() {
             chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
                                   -o Debug::pkgProblemResolver=yes
 
-            umount -R -l "${ROOTFSDIR}/dev"
+            umount -l "${ROOTFSDIR}/dev"
             umount -l "${ROOTFSDIR}/proc"
-            umount -R -l "${ROOTFSDIR}/sys"
+            umount -l "${ROOTFSDIR}/sys"
             umount -l "${ROOTFSDIR}/base-apt" || true
 
             # Finalize debootstrap by setting the link in deploy
-- 
2.16.4

RE: [PATCH 1/3] ci: Wait for bitbake worker to finish before deleting artifacts

[Hombourger, Cedric]

s/reliably/reliable/
Looks good otherwise

-----Original Message-----
From: Jan Kiszka [mailto:jan.kiszka@siemens.com] 
Sent: Wednesday, December 5, 2018 12:30 PM
To: isar-users <isar-users@googlegroups.com>
Cc: Hombourger, Cedric <Cedric_Hombourger@mentor.com>
Subject: [PATCH 1/3] ci: Wait for bitbake worker to finish before deleting artifacts

From: Jan Kiszka <jan.kiszka@siemens.com>

bitbake completes asynchronously to the worker daemon. Therefore, we must not start deleting its tmp folder before the termination because it may contain active mounts.

Wait for the socket that links frontend and backend to go away. This is a reliably sign that the worker finished as well.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 scripts/ci_build.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh index 20b77b7..aae9084 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -92,6 +92,7 @@ if [ -n "$FAST_BUILD" ]; then
         multiconfig:qemuarm-stretch:isar-image-base \
         multiconfig:qemuarm64-stretch:isar-image-base \
         multiconfig:qemuamd64-stretch:isar-image-base
+    while [ -e bitbake.sock ]; do sleep 1; done
     sudo rm -rf tmp
     sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
     bitbake $BB_ARGS \
--
2.16.4

Re: [PATCH 1/3] ci: Wait for bitbake worker to finish before deleting artifacts

[Maxim Yu. Osipov]

On 12/5/18 12:29 PM, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> bitbake completes asynchronously to the worker daemon. Therefore, we
> must not start deleting its tmp folder before the termination because it
> may contain active mounts.
> 
> Wait for the socket that links frontend and backend to go away. This is
> a reliably sign that the worker finished as well.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>   scripts/ci_build.sh | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> index 20b77b7..aae9084 100755
> --- a/scripts/ci_build.sh
> +++ b/scripts/ci_build.sh
> @@ -92,6 +92,7 @@ if [ -n "$FAST_BUILD" ]; then
>           multiconfig:qemuarm-stretch:isar-image-base \
>           multiconfig:qemuarm64-stretch:isar-image-base \
>           multiconfig:qemuamd64-stretch:isar-image-base
> +    while [ -e bitbake.sock ]; do sleep 1; done

I'm OK with this change, but this doesn't solve problems
with hangs - when commands are launched by hand:

Command 1)
bitbake  -c cache_base_repo multiconfig:qemuarm-stretch:isar-image-base 
multiconfig:qemuarm64-stretch:isar-image-base multiconfig:qemuamd64-
stretch:isar-image-base

At this step I've doublechecked  that bitbake.sock is cleaned.
Command 2)
sudo rm -rf tmp

Command 3)
sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO 
?= "1"/g' conf/local.conf

No problems detected at this point - the same mounts etc.

The next command hangs on the last task (according strace bitbake tries 
to unmount /sys /dev)

Command 4)
bitbake multiconfig:qemuarm-stretch:isar-image-base 
multiconfig:qemuarm64-stretch:isar-image-base 
multiconfig:qemuamd64-stretch:isar-image-base

Maxim.

>       sudo rm -rf tmp
>       sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
>       bitbake $BB_ARGS \
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH 1/3] ci: Wait for bitbake worker to finish before deleting artifacts

[Jan Kiszka]

On 05.12.18 10:59, Maxim Yu. Osipov wrote:
> On 12/5/18 12:29 PM, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> bitbake completes asynchronously to the worker daemon. Therefore, we
>> must not start deleting its tmp folder before the termination because it
>> may contain active mounts.
>>
>> Wait for the socket that links frontend and backend to go away. This is
>> a reliably sign that the worker finished as well.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>   scripts/ci_build.sh | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
>> index 20b77b7..aae9084 100755
>> --- a/scripts/ci_build.sh
>> +++ b/scripts/ci_build.sh
>> @@ -92,6 +92,7 @@ if [ -n "$FAST_BUILD" ]; then
>>           multiconfig:qemuarm-stretch:isar-image-base \
>>           multiconfig:qemuarm64-stretch:isar-image-base \
>>           multiconfig:qemuamd64-stretch:isar-image-base
>> +    while [ -e bitbake.sock ]; do sleep 1; done
> 
> I'm OK with this change, but this doesn't solve problems
> with hangs - when commands are launched by hand:
> 
> Command 1)
> bitbake  -c cache_base_repo multiconfig:qemuarm-stretch:isar-image-base 
> multiconfig:qemuarm64-stretch:isar-image-base multiconfig:qemuamd64-
> stretch:isar-image-base
> 
> At this step I've doublechecked  that bitbake.sock is cleaned.
> Command 2)
> sudo rm -rf tmp
> 
> Command 3)
> sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= 
> "1"/g' conf/local.conf
> 
> No problems detected at this point - the same mounts etc.
> 
> The next command hangs on the last task (according strace bitbake tries to 
> unmount /sys /dev)

Again: What is the pstree of that? Where does the hang come from, what 
parameters were passed etc.? What is holding the mount point?

We really need to understand the problem, not just paper over it.

Jan

> 
> Command 4)
> bitbake multiconfig:qemuarm-stretch:isar-image-base 
> multiconfig:qemuarm64-stretch:isar-image-base 
> multiconfig:qemuamd64-stretch:isar-image-base
> 
> Maxim.
> 
>>       sudo rm -rf tmp
>>       sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO 
>> ?= "1"/g' conf/local.conf
>>       bitbake $BB_ARGS \
>>
> 
> 


-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 0/3] umount fixes and cleanups

[Maxim Yu. Osipov]

[-- Attachment #1: Type: text/plain, Size: 1500 bytes --]

On 12/5/18 12:29 PM, Jan Kiszka wrote:
> Patches related to resolving the pending CI issues as well as
> simplifying the umounts used during cleanup.

Tried to run in patch queue:

027b7cf Remove redundant recursive umounts
c1bdc33 isar-events: Improve umount handler
b354273 ci: Wait for bitbake worker to finish before deleting artifacts
9cf29e6 isar-bootstrap: Fix and cleanup bind mounting
b354026 isar-image: umount base-apt when doing offline build
e965c0d gitlab-ci: Switch to ci_build.sh
...

After execution of problematic test case (I rebooted PC and executed 
steps in clean tree):

my stretch Debian system was entered into unusable state
as many important mounts were disappeared (see log of mount points
before and after execution  of last command attached).

Maxim.


> Jan
> 
> Jan Kiszka (3):
>    ci: Wait for bitbake worker to finish before deleting artifacts
>    isar-events: Improve umount handler
>    Remove redundant recursive umounts
> 
>   meta/classes/isar-events.bbclass                   | 38 ++++++++--------------
>   meta/classes/isar-image.bbclass                    |  4 +--
>   .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  4 +--
>   scripts/ci_build.sh                                |  1 +
>   4 files changed, 18 insertions(+), 29 deletions(-)
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

[-- Attachment #2: mount_after.txt --]
[-- Type: text/plain, Size: 1260 bytes --]

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=4015904k,nr_inodes=1003976,mode=755)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=805840k,mode=755)
/dev/sda3 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=33,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15456)
/dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=ascii,shortname=mixed,quiet,utf8,errors=remount-ro)
/dev/sda5 on /home type ext4 (rw,relatime,data=ordered)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
tmpfs on /run/user/114 type tmpfs (rw,nosuid,nodev,relatime,size=805836k,mode=700,uid=114,gid=116)
gvfsd-fuse on /run/user/114/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=114,group_id=116)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=805836k,mode=700,uid=1000,gid=1000)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)

[-- Attachment #3: mount_before.txt --]
[-- Type: text/plain, Size: 2877 bytes --]

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=4015904k,nr_inodes=1003976,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=805840k,mode=755)
/dev/sda3 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=33,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15456)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
/dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=ascii,shortname=mixed,quiet,utf8,errors=remount-ro)
/dev/sda5 on /home type ext4 (rw,relatime,data=ordered)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
tmpfs on /run/user/114 type tmpfs (rw,nosuid,nodev,relatime,size=805836k,mode=700,uid=114,gid=116)
gvfsd-fuse on /run/user/114/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=114,group_id=116)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=805836k,mode=700,uid=1000,gid=1000)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)

RE: [PATCH 0/3] umount fixes and cleanups

[Hombourger, Cedric]

FWIW, here is an excerpt of the script being used in our (Mentor) CI

# ---------------------------------------------------------------------------------------------------------------------
# Clean previous build
# ---------------------------------------------------------------------------------------------------------------------

check_build_mounts() {
   local tries=0

   while :
   do
       mounts=$(mount|awk '{print $3;}'|grep ^${PWD}/ || true);
       [ -z "${mounts}" ] && break

       for m in ${mounts}
       do
           sudo umount ${m} 2>/dev/null || true
       done

       # Abort if some mounts remain after multiple attempts
       tries=$((${tries} + 1))
       if [ ${tries} -ge 3 ]
       then
          echo "# error: mounts within 'tmp' are busy!" >&2
          exit 1
       fi
       sleep 1
   done
}

# Make sure no mounts were left from a previous build before wiping "tmp" out
echo "# info: trying to remove mounts from previous build(s)..."
check_build_mounts

# Wipe out "tmp"
if [ "${CLEANBUILD}" -eq "1" ]; then
   echo "# info: deleting previous build(s)..."
   sudo rm -rf tmp
fi

-----Original Message-----
From: Maxim Yu. Osipov [mailto:mosipov@ilbers.de] 
Sent: Wednesday, December 5, 2018 2:12 PM
To: Jan Kiszka <jan.kiszka@siemens.com>; isar-users <isar-users@googlegroups.com>
Cc: Hombourger, Cedric <Cedric_Hombourger@mentor.com>
Subject: Re: [PATCH 0/3] umount fixes and cleanups

On 12/5/18 12:29 PM, Jan Kiszka wrote:
> Patches related to resolving the pending CI issues as well as 
> simplifying the umounts used during cleanup.

Tried to run in patch queue:

027b7cf Remove redundant recursive umounts
c1bdc33 isar-events: Improve umount handler
b354273 ci: Wait for bitbake worker to finish before deleting artifacts
9cf29e6 isar-bootstrap: Fix and cleanup bind mounting
b354026 isar-image: umount base-apt when doing offline build e965c0d gitlab-ci: Switch to ci_build.sh ...

After execution of problematic test case (I rebooted PC and executed steps in clean tree):

my stretch Debian system was entered into unusable state as many important mounts were disappeared (see log of mount points before and after execution  of last command attached).

Maxim.


> Jan
> 
> Jan Kiszka (3):
>    ci: Wait for bitbake worker to finish before deleting artifacts
>    isar-events: Improve umount handler
>    Remove redundant recursive umounts
> 
>   meta/classes/isar-events.bbclass                   | 38 ++++++++--------------
>   meta/classes/isar-image.bbclass                    |  4 +--
>   .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  4 +--
>   scripts/ci_build.sh                                |  1 +
>   4 files changed, 18 insertions(+), 29 deletions(-)
> 


--
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH 0/3] umount fixes and cleanups

[Jan Kiszka]

On 05.12.18 12:12, Maxim Yu. Osipov wrote:
> On 12/5/18 12:29 PM, Jan Kiszka wrote:
>> Patches related to resolving the pending CI issues as well as
>> simplifying the umounts used during cleanup.
> 
> Tried to run in patch queue:
> 
> 027b7cf Remove redundant recursive umounts
> c1bdc33 isar-events: Improve umount handler
> b354273 ci: Wait for bitbake worker to finish before deleting artifacts
> 9cf29e6 isar-bootstrap: Fix and cleanup bind mounting
> b354026 isar-image: umount base-apt when doing offline build
> e965c0d gitlab-ci: Switch to ci_build.sh
> ...
> 
> After execution of problematic test case (I rebooted PC and executed steps in 
> clean tree):
> 
> my stretch Debian system was entered into unusable state
> as many important mounts were disappeared (see log of mount points
> before and after execution  of last command attached).

OK, so I took the time and extracted the root cause. We may see a Debian 4.9 
kernel bug here:

# mkdir my-mnt
# mount --rbind /sys my-mnt
# mount --make-rslave my-mnt
# umount {-R,-l,-R -l} my-mnt
# rmdir my-mnt
rmdir: failed to remove 'my-mnt': Device or resource busy

This works fine on my Leap 15.0 kernel (4.12) as well as the 4.4 kernel (Ubuntu) 
used on our CI server.

*This* is information we can base commits on. Also, we can file a bug against 
Debian with this. Could you do that (keep my in CC)?

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

RE: [PATCH 0/3] umount fixes and cleanups

[Hombourger, Cedric]

I was able to use the following work-around for your test-case:

# run the following command until all mounts are gone
umount $(mount |grep $PWD/my-mnt|awk '{print $3}')

umount -R or -l indeed failed

Cedric

-----Original Message-----
From: Jan Kiszka [mailto:jan.kiszka@siemens.com] 
Sent: Wednesday, December 5, 2018 3:15 PM
To: Maxim Yu. Osipov <mosipov@ilbers.de>; isar-users <isar-users@googlegroups.com>
Cc: Hombourger, Cedric <Cedric_Hombourger@mentor.com>
Subject: Re: [PATCH 0/3] umount fixes and cleanups

On 05.12.18 12:12, Maxim Yu. Osipov wrote:
> On 12/5/18 12:29 PM, Jan Kiszka wrote:
>> Patches related to resolving the pending CI issues as well as 
>> simplifying the umounts used during cleanup.
> 
> Tried to run in patch queue:
> 
> 027b7cf Remove redundant recursive umounts
> c1bdc33 isar-events: Improve umount handler
> b354273 ci: Wait for bitbake worker to finish before deleting 
> artifacts
> 9cf29e6 isar-bootstrap: Fix and cleanup bind mounting
> b354026 isar-image: umount base-apt when doing offline build e965c0d 
> gitlab-ci: Switch to ci_build.sh ...
> 
> After execution of problematic test case (I rebooted PC and executed 
> steps in clean tree):
> 
> my stretch Debian system was entered into unusable state as many 
> important mounts were disappeared (see log of mount points before and 
> after execution  of last command attached).

OK, so I took the time and extracted the root cause. We may see a Debian 4.9 kernel bug here:

# mkdir my-mnt
# mount --rbind /sys my-mnt
# mount --make-rslave my-mnt
# umount {-R,-l,-R -l} my-mnt
# rmdir my-mnt
rmdir: failed to remove 'my-mnt': Device or resource busy

This works fine on my Leap 15.0 kernel (4.12) as well as the 4.4 kernel (Ubuntu) used on our CI server.

*This* is information we can base commits on. Also, we can file a bug against Debian with this. Could you do that (keep my in CC)?

Thanks,
Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux

Re: [PATCH 0/3] umount fixes and cleanups

[Jan Kiszka]

On 05.12.18 13:31, Hombourger, Cedric wrote:
> I was able to use the following work-around for your test-case:
> 
> # run the following command until all mounts are gone
> umount $(mount |grep $PWD/my-mnt|awk '{print $3}')

Does that listing always produce the right ordering for umount, or does the 
order not matter?

Just for me to remember: We do want the recursive mounts, don't we? Or, said 
differently, what is the downside of Maxim's revert for use cases you see?

Jan

> 
> umount -R or -l indeed failed
> 
> Cedric
> 
> -----Original Message-----
> From: Jan Kiszka [mailto:jan.kiszka@siemens.com]
> Sent: Wednesday, December 5, 2018 3:15 PM
> To: Maxim Yu. Osipov <mosipov@ilbers.de>; isar-users <isar-users@googlegroups.com>
> Cc: Hombourger, Cedric <Cedric_Hombourger@mentor.com>
> Subject: Re: [PATCH 0/3] umount fixes and cleanups
> 
> On 05.12.18 12:12, Maxim Yu. Osipov wrote:
>> On 12/5/18 12:29 PM, Jan Kiszka wrote:
>>> Patches related to resolving the pending CI issues as well as
>>> simplifying the umounts used during cleanup.
>>
>> Tried to run in patch queue:
>>
>> 027b7cf Remove redundant recursive umounts
>> c1bdc33 isar-events: Improve umount handler
>> b354273 ci: Wait for bitbake worker to finish before deleting
>> artifacts
>> 9cf29e6 isar-bootstrap: Fix and cleanup bind mounting
>> b354026 isar-image: umount base-apt when doing offline build e965c0d
>> gitlab-ci: Switch to ci_build.sh ...
>>
>> After execution of problematic test case (I rebooted PC and executed
>> steps in clean tree):
>>
>> my stretch Debian system was entered into unusable state as many
>> important mounts were disappeared (see log of mount points before and
>> after execution  of last command attached).
> 
> OK, so I took the time and extracted the root cause. We may see a Debian 4.9 kernel bug here:
> 
> # mkdir my-mnt
> # mount --rbind /sys my-mnt
> # mount --make-rslave my-mnt
> # umount {-R,-l,-R -l} my-mnt
> # rmdir my-mnt
> rmdir: failed to remove 'my-mnt': Device or resource busy
> 
> This works fine on my Leap 15.0 kernel (4.12) as well as the 4.4 kernel (Ubuntu) used on our CI server.
> 
> *This* is information we can base commits on. Also, we can file a bug against Debian with this. Could you do that (keep my in CC)?
> 
> Thanks,
> Jan
> 
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

RE: [PATCH 0/3] umount fixes and cleanups

[Hombourger, Cedric]

Hi Jan,

In my testing, the only mounts that were really needed for the go packages I was building are:

/dev		# was already mounted before our changes
/dev/pts	# was not mounted before
/sys		# was not mounted before

I have some doubts about /dev/shm; I cannot recall if that one was truly needed

Regarding ordering, this brute force umount command will produce errors for sub-mounts (e.g. when trying to umount /sys/fs/cgroup before /sys/fs/cgroup/blkio)
I seem to remember that in the early days Isar would just loop over its mounts until it got rid of everything

Cedric

-----Original Message-----
From: Jan Kiszka [mailto:jan.kiszka@siemens.com] 
Sent: Wednesday, December 5, 2018 3:36 PM
To: Hombourger, Cedric <Cedric_Hombourger@mentor.com>; Maxim Yu. Osipov <mosipov@ilbers.de>; isar-users <isar-users@googlegroups.com>
Subject: Re: [PATCH 0/3] umount fixes and cleanups

On 05.12.18 13:31, Hombourger, Cedric wrote:
> I was able to use the following work-around for your test-case:
> 
> # run the following command until all mounts are gone umount $(mount 
> |grep $PWD/my-mnt|awk '{print $3}')

Does that listing always produce the right ordering for umount, or does the order not matter?

Just for me to remember: We do want the recursive mounts, don't we? Or, said differently, what is the downside of Maxim's revert for use cases you see?

Jan

> 
> umount -R or -l indeed failed
> 
> Cedric
> 
> -----Original Message-----
> From: Jan Kiszka [mailto:jan.kiszka@siemens.com]
> Sent: Wednesday, December 5, 2018 3:15 PM
> To: Maxim Yu. Osipov <mosipov@ilbers.de>; isar-users 
> <isar-users@googlegroups.com>
> Cc: Hombourger, Cedric <Cedric_Hombourger@mentor.com>
> Subject: Re: [PATCH 0/3] umount fixes and cleanups
> 
> On 05.12.18 12:12, Maxim Yu. Osipov wrote:
>> On 12/5/18 12:29 PM, Jan Kiszka wrote:
>>> Patches related to resolving the pending CI issues as well as 
>>> simplifying the umounts used during cleanup.
>>
>> Tried to run in patch queue:
>>
>> 027b7cf Remove redundant recursive umounts
>> c1bdc33 isar-events: Improve umount handler
>> b354273 ci: Wait for bitbake worker to finish before deleting 
>> artifacts
>> 9cf29e6 isar-bootstrap: Fix and cleanup bind mounting
>> b354026 isar-image: umount base-apt when doing offline build e965c0d
>> gitlab-ci: Switch to ci_build.sh ...
>>
>> After execution of problematic test case (I rebooted PC and executed 
>> steps in clean tree):
>>
>> my stretch Debian system was entered into unusable state as many 
>> important mounts were disappeared (see log of mount points before and 
>> after execution  of last command attached).
> 
> OK, so I took the time and extracted the root cause. We may see a Debian 4.9 kernel bug here:
> 
> # mkdir my-mnt
> # mount --rbind /sys my-mnt
> # mount --make-rslave my-mnt
> # umount {-R,-l,-R -l} my-mnt
> # rmdir my-mnt
> rmdir: failed to remove 'my-mnt': Device or resource busy
> 
> This works fine on my Leap 15.0 kernel (4.12) as well as the 4.4 kernel (Ubuntu) used on our CI server.
> 
> *This* is information we can base commits on. Also, we can file a bug against Debian with this. Could you do that (keep my in CC)?
> 
> Thanks,
> Jan
> 
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate 
> Competence Center Embedded Linux
> 

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux

Re: [PATCH 0/3] umount fixes and cleanups

[Jan Kiszka]

On 05.12.18 13:48, Hombourger, Cedric wrote:
> Hi Jan,
> 
> In my testing, the only mounts that were really needed for the go packages I was building are:
> 
> /dev		# was already mounted before our changes
> /dev/pts	# was not mounted before
> /sys		# was not mounted before
> 
> I have some doubts about /dev/shm; I cannot recall if that one was truly needed

OK, but then I would suggest, when reverting the bind mount, to add /dev/pts and 
(to be safe) maybe also /dev/shm explicitly and umount them also explicitly.

And then we should finally consolidate common mount and umount for chroots into 
one central helper each.

> 
> Regarding ordering, this brute force umount command will produce errors for sub-mounts (e.g. when trying to umount /sys/fs/cgroup before /sys/fs/cgroup/blkio)
> I seem to remember that in the early days Isar would just loop over its mounts until it got rid of everything

OK, so this targets isar_handler() only. But we also have (and need) umounts 
elsewhere.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 0/3] umount fixes and cleanups

[Maxim Yu. Osipov]

On 12/5/18 3:14 PM, Jan Kiszka wrote:
> On 05.12.18 12:12, Maxim Yu. Osipov wrote:
>> On 12/5/18 12:29 PM, Jan Kiszka wrote:
>>> Patches related to resolving the pending CI issues as well as
>>> simplifying the umounts used during cleanup.
>>
>> Tried to run in patch queue:
>>
>> 027b7cf Remove redundant recursive umounts
>> c1bdc33 isar-events: Improve umount handler
>> b354273 ci: Wait for bitbake worker to finish before deleting artifacts
>> 9cf29e6 isar-bootstrap: Fix and cleanup bind mounting
>> b354026 isar-image: umount base-apt when doing offline build
>> e965c0d gitlab-ci: Switch to ci_build.sh
>> ...
>>
>> After execution of problematic test case (I rebooted PC and executed 
>> steps in clean tree):
>>
>> my stretch Debian system was entered into unusable state
>> as many important mounts were disappeared (see log of mount points
>> before and after execution  of last command attached).
> 
> OK, so I took the time and extracted the root cause. We may see a Debian 
> 4.9 kernel bug here:
> 
> # mkdir my-mnt
> # mount --rbind /sys my-mnt
> # mount --make-rslave my-mnt
> # umount {-R,-l,-R -l} my-mnt
> # rmdir my-mnt
> rmdir: failed to remove 'my-mnt': Device or resource busy
> 
> This works fine on my Leap 15.0 kernel (4.12) as well as the 4.4 kernel 
> (Ubuntu) used on our CI server.

This also works fine on my Debian 4.9 kernel (I've ran this sequence for 
every combination of {-R, -l , -R -l}):

> root@tiberius:~# uname -a
> Linux tiberius 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u6 (2018-10-08) x86_64 GNU/Linux
> root@tiberius:~# mkdir my-mnt
> root@tiberius:~# mount --rbind /sys my-mnt
> root@tiberius:~# mount --make-rslave my-mnt
> root@tiberius:~# umount -R my-mnt/
> root@tiberius:~# rmdir my-mnt/
> root@tiberius:~# mkdir my-mnt
> root@tiberius:~# mount --rbind /sys my-mnt
> root@tiberius:~# mount --make-rslave my-mnt
> root@tiberius:~# umount -l my-mnt
> root@tiberius:~# rmdir my-mnt/
> root@tiberius:~# mkdir my-mnt
> root@tiberius:~# mount --rbind /sys my-mnt
> root@tiberius:~# mount --make-rslave my-mnt
> root@tiberius:~# umount -R -l my-mnt
> root@tiberius:~# rmdir my-mnt/
> root@tiberius:~# 


Thanks,
Maxim.

> *This* is information we can base commits on. Also, we can file a bug 
> against Debian with this. Could you do that (keep my in CC)?



> Thanks,
> Jan
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH 0/3] umount fixes and cleanups

[Jan Kiszka]

On 05.12.18 15:26, Maxim Yu. Osipov wrote:
> On 12/5/18 3:14 PM, Jan Kiszka wrote:
>> On 05.12.18 12:12, Maxim Yu. Osipov wrote:
>>> On 12/5/18 12:29 PM, Jan Kiszka wrote:
>>>> Patches related to resolving the pending CI issues as well as
>>>> simplifying the umounts used during cleanup.
>>>
>>> Tried to run in patch queue:
>>>
>>> 027b7cf Remove redundant recursive umounts
>>> c1bdc33 isar-events: Improve umount handler
>>> b354273 ci: Wait for bitbake worker to finish before deleting artifacts
>>> 9cf29e6 isar-bootstrap: Fix and cleanup bind mounting
>>> b354026 isar-image: umount base-apt when doing offline build
>>> e965c0d gitlab-ci: Switch to ci_build.sh
>>> ...
>>>
>>> After execution of problematic test case (I rebooted PC and executed steps in 
>>> clean tree):
>>>
>>> my stretch Debian system was entered into unusable state
>>> as many important mounts were disappeared (see log of mount points
>>> before and after execution  of last command attached).
>>
>> OK, so I took the time and extracted the root cause. We may see a Debian 4.9 
>> kernel bug here:
>>
>> # mkdir my-mnt
>> # mount --rbind /sys my-mnt
>> # mount --make-rslave my-mnt
>> # umount {-R,-l,-R -l} my-mnt
>> # rmdir my-mnt
>> rmdir: failed to remove 'my-mnt': Device or resource busy
>>
>> This works fine on my Leap 15.0 kernel (4.12) as well as the 4.4 kernel 
>> (Ubuntu) used on our CI server.
> 
> This also works fine on my Debian 4.9 kernel (I've ran this sequence for every 
> combination of {-R, -l , -R -l}):
> 
>> root@tiberius:~# uname -a
>> Linux tiberius 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u6 (2018-10-08) 
>> x86_64 GNU/Linux

Hmpf, rechecking... I was still on 3.16 from the jessie that I upgraded from. 
Works fine with 4.9, indeed.

>> root@tiberius:~# mkdir my-mnt
>> root@tiberius:~# mount --rbind /sys my-mnt
>> root@tiberius:~# mount --make-rslave my-mnt
>> root@tiberius:~# umount -R my-mnt/
>> root@tiberius:~# rmdir my-mnt/
>> root@tiberius:~# mkdir my-mnt
>> root@tiberius:~# mount --rbind /sys my-mnt
>> root@tiberius:~# mount --make-rslave my-mnt
>> root@tiberius:~# umount -l my-mnt
>> root@tiberius:~# rmdir my-mnt/
>> root@tiberius:~# mkdir my-mnt
>> root@tiberius:~# mount --rbind /sys my-mnt
>> root@tiberius:~# mount --make-rslave my-mnt
>> root@tiberius:~# umount -R -l my-mnt
>> root@tiberius:~# rmdir my-mnt/
>> root@tiberius:~# 
> 

OK, then bind-mounting itself likely not the root cause, and we need to dig further.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 0/3] umount fixes and cleanups

[Jan Kiszka]

On 05.12.18 15:52, [ext] Jan Kiszka wrote:
> On 05.12.18 15:26, Maxim Yu. Osipov wrote:
>> On 12/5/18 3:14 PM, Jan Kiszka wrote:
>>> On 05.12.18 12:12, Maxim Yu. Osipov wrote:
>>>> On 12/5/18 12:29 PM, Jan Kiszka wrote:
>>>>> Patches related to resolving the pending CI issues as well as
>>>>> simplifying the umounts used during cleanup.
>>>>
>>>> Tried to run in patch queue:
>>>>
>>>> 027b7cf Remove redundant recursive umounts
>>>> c1bdc33 isar-events: Improve umount handler
>>>> b354273 ci: Wait for bitbake worker to finish before deleting artifacts
>>>> 9cf29e6 isar-bootstrap: Fix and cleanup bind mounting
>>>> b354026 isar-image: umount base-apt when doing offline build
>>>> e965c0d gitlab-ci: Switch to ci_build.sh
>>>> ...
>>>>
>>>> After execution of problematic test case (I rebooted PC and executed steps 
>>>> in clean tree):
>>>>
>>>> my stretch Debian system was entered into unusable state
>>>> as many important mounts were disappeared (see log of mount points
>>>> before and after execution  of last command attached).
>>>
>>> OK, so I took the time and extracted the root cause. We may see a Debian 4.9 
>>> kernel bug here:
>>>
>>> # mkdir my-mnt
>>> # mount --rbind /sys my-mnt
>>> # mount --make-rslave my-mnt
>>> # umount {-R,-l,-R -l} my-mnt
>>> # rmdir my-mnt
>>> rmdir: failed to remove 'my-mnt': Device or resource busy
>>>
>>> This works fine on my Leap 15.0 kernel (4.12) as well as the 4.4 kernel 
>>> (Ubuntu) used on our CI server.
>>
>> This also works fine on my Debian 4.9 kernel (I've ran this sequence for every 
>> combination of {-R, -l , -R -l}):
>>
>>> root@tiberius:~# uname -a
>>> Linux tiberius 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u6 (2018-10-08) 
>>> x86_64 GNU/Linux
> 
> Hmpf, rechecking... I was still on 3.16 from the jessie that I upgraded from. 
> Works fine with 4.9, indeed.
> 
>>> root@tiberius:~# mkdir my-mnt
>>> root@tiberius:~# mount --rbind /sys my-mnt
>>> root@tiberius:~# mount --make-rslave my-mnt
>>> root@tiberius:~# umount -R my-mnt/
>>> root@tiberius:~# rmdir my-mnt/
>>> root@tiberius:~# mkdir my-mnt
>>> root@tiberius:~# mount --rbind /sys my-mnt
>>> root@tiberius:~# mount --make-rslave my-mnt
>>> root@tiberius:~# umount -l my-mnt
>>> root@tiberius:~# rmdir my-mnt/
>>> root@tiberius:~# mkdir my-mnt
>>> root@tiberius:~# mount --rbind /sys my-mnt
>>> root@tiberius:~# mount --make-rslave my-mnt
>>> root@tiberius:~# umount -R -l my-mnt
>>> root@tiberius:~# rmdir my-mnt/
>>> root@tiberius:~# 
>>
> 
> OK, then bind-mounting itself likely not the root cause, and we need to dig 
> further.
> 

Here is the real solution, or at least a much better workaround, without
any reverts:

diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass
index 225463e..c66d208 100644
--- a/meta/classes/wic-img.bbclass
+++ b/meta/classes/wic-img.bbclass
@@ -90,7 +90,7 @@ do_wic_image() {
     for dir in ${BBLAYERS} ${STAGING_DIR} ${ISARROOT}/scripts; do
 	sudo mkdir -p ${BUILDCHROOT_DIR}/$dir
         mountpoint ${BUILDCHROOT_DIR}/$dir >/dev/null 2>&1 \
-        || sudo mount --bind $dir ${BUILDCHROOT_DIR}/$dir
+        || sudo mount --bind --make-private $dir ${BUILDCHROOT_DIR}/$dir
     done
     export FAKEROOTCMD=${FAKEROOTCMD}
     export BUILDDIR=${BUILDDIR}


I'll explain tomorrow, also what else is broken in do_wic_image.

Jan

Re: [PATCH 0/3] umount fixes and cleanups

[Maxim Yu. Osipov]

On 12/5/18 12:29 PM, Jan Kiszka wrote:
> Patches related to resolving the pending CI issues as well as
> simplifying the umounts used during cleanup.

Applied to the 'next',

Thanks,
Maxim.

> Jan
> 
> Jan Kiszka (3):
>    ci: Wait for bitbake worker to finish before deleting artifacts
>    isar-events: Improve umount handler
>    Remove redundant recursive umounts
> 
>   meta/classes/isar-events.bbclass                   | 38 ++++++++--------------
>   meta/classes/isar-image.bbclass                    |  4 +--
>   .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  4 +--
>   scripts/ci_build.sh                                |  1 +
>   4 files changed, 18 insertions(+), 29 deletions(-)
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov