Re: [PATCH] isar-bootstrap: Fix and cleanup bind mounting

["Maxim Yu. Osipov" <mosipov@ilbers.de>]

[-- Attachment #1: Type: text/plain, Size: 6468 bytes --]

On 12/3/18 3:59 PM, Jan Kiszka wrote:
> On 30.11.18 10:20, Maxim Yu. Osipov wrote:
>> Hi Jan,
>>
>> I've just tried this patch (on the 'next' with reverted patch 
>> d40a9ac0) and ran "fast" CI
>>
>> isar$mount | wc -l
>> 34
>>
>> isar$./scripts/ci_build.sh -q -f
>>
>> CI script hung on CI stage when dpkg-base is modified
>> causing rebuilding recipes based on dpkg-base.
>>
>> The mount reports less (!) mount points than before launching the script.
>>
>> mount | wc -l
>> 31
> 
> Any news on what's different on your side? Where exactly does your build 
> hang? Was your CI environment in a clean state when running this test? 
> Before the comment lots of things leaked.


On my stretch laptop (i7-6820HQ CPU @ 2.70GHz (8 cores) with SSD)  the 
reported problem is reproducible (I rerun 'ci_build.sh -q -f' several 
times in clean state) it hung and with the less mount points (the mount 
points before and after running are attached).

The strange thing that I observe two bitbake processes:

myo      26373  0.0  0.3 153116 29732 pts/0    Sl+  12:31   0:01 python3 
/home/myo/work/isar/src/trunk/isar/bitbake/bin/bitbake 
multiconfig:qemuarm-stretch:isar-image-base 
multiconfig:qemuarm64-stretch:isar-image-base 
multiconfig:qemuamd64-stretch:isar-image-base

myo      26379  2.5  0.6 328476 50028 ?        Sl   12:31   0:40 python3 
/home/myo/work/isar/src/trunk/isar/bitbake/bin/bitbake 
multiconfig:qemuarm-stretch:isar-image-base 
multiconfig:qemuarm64-stretch:isar-image-base 
multiconfig:qemuamd64-stretch:isar-image-base


When I turned on verbosity (called ci_build.sh without '-q' switch)
'scripts/ci_build -f' succeeded.

Build also fails on isar-build.org
http://isar-build.org:8080/job/isar_mosipov_next/98/console:
it complains  on
'rm: cannot remove 
'/workspace/build/isar_mosipov_next/98/build/tmp/work/debian-stretch-armhf/isar-bootstrap-target/rootfs/dev': 
Device or resource busy'

Note that ci-build.sh on isar-build.org is also called with '-q' switch.


Questions:

1) Did you launch ci_build with '-q' option?

2) Did you inspect mount points before and after launching ci_build? Are 
they identical?

3) Do you run this in docker?


Maxim.


> We still do not see problems with this bug fix, neither locally nor in 
> CI (which is ci_build-based now, too). And it's actually a precondition 
> to make "isar-image: umount base-apt when doing offline build" work as 
> expected.
> 
> Thanks,
> Jan
> 
>>
>> Maxim.
>>
>> On 11/29/18 9:39 PM, Henning Schild wrote:
>>> Maybe that is where the leftover mounts came from, and the reverts are
>>> not required.
>>>
>>> Henning
>>>
>>> Am Thu, 29 Nov 2018 17:29:17 +0100
>>> schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:
>>>
>>>> First, wrong variables were used (there is no BUILDCHROOT_DIR for
>>>> isar_bootstrap). And then we simply left the mount points active after
>>>> completing the bootstrap. The led to setup_root_file_system copying
>>>> all the mount point content over into the buildchroot or rootfs -
>>>> including base-apt when it was used.
>>>>
>>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>>> ---
>>>>
>>>> This is a precondition for "isar-image: umount base-apt when doing
>>>> offline build" to work properly. IOW, that patch reveal these bugs.
>>>>
>>>> Possibly, this also explains the increasing mount count that was
>>>> visible in CI.
>>>>
>>>>   meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 21
>>>> +++++++++++---------- 1 file changed, 11 insertions(+), 10
>>>> deletions(-)
>>>>
>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
>>>> 8ea3b08..da077d0 100644 ---
>>>> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -185,12
>>>> +185,8 @@ isar_bootstrap() { sudo -E flock "${ISAR_BOOTSTRAP_LOCK}"
>>>> -c "\ set -e
>>>>           if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then
>>>> -            if [ -e "${ROOTFSDIR}" ]; then
>>>> -               umount -R -l "${ROOTFSDIR}/dev" || true
>>>> -               umount -l "${ROOTFSDIR}/proc" || true
>>>> -               umount -R -l "${ROOTFSDIR}/sys" || true
>>>> -               rm -rf "${ROOTFSDIR}"
>>>> -            fi
>>>> +            rm -rf "${ROOTFSDIR}"
>>>> +
>>>>               if [ ${IS_HOST} ]; then
>>>>                   ${DEBOOTSTRAP} $debootstrap_args \
>>>>                                  ${@get_distro_components_argument(d,
>>>> True)} \ @@ -247,17 +243,22 @@ isar_bootstrap() {
>>>>               "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"
>>>>               # update APT
>>>> -            mount --rbind /dev ${BUILDCHROOT_DIR}/dev
>>>> -            mount --make-rslave ${BUILDCHROOT_DIR}/dev
>>>> +            mount --rbind /dev ${ROOTFSDIR}/dev
>>>> +            mount --make-rslave ${ROOTFSDIR}/dev
>>>>               mount -t proc none ${ROOTFSDIR}/proc
>>>> -            mount --rbind /sys ${BUILDCHROOT_DIR}/sys
>>>> -            mount --make-rslave ${BUILDCHROOT_DIR}/sys
>>>> +            mount --rbind /sys ${ROOTFSDIR}/sys
>>>> +            mount --make-rslave ${ROOTFSDIR}/sys
>>>>               export DEBIAN_FRONTEND=noninteractive
>>>>               chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
>>>>               chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
>>>>                                     -o Debug::pkgProblemResolver=yes
>>>> +            umount -R -l "${ROOTFSDIR}/dev"
>>>> +            umount -l "${ROOTFSDIR}/proc"
>>>> +            umount -R -l "${ROOTFSDIR}/sys"
>>>> +            umount -l "${ROOTFSDIR}/base-apt" || true
>>>> +
>>>>               # Finalize debootstrap by setting the link in deploy
>>>>               ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
>>>>           fi"
>>>
>>
>>
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

[-- Attachment #2: mount_after.txt --]
[-- Type: text/plain, Size: 5063 bytes --]

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=4015904k,nr_inodes=1003976,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=805840k,mode=755)
/dev/sda3 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=38,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=235)
/dev/sda5 on /home type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=ascii,shortname=mixed,quiet,utf8,errors=remount-ro)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
tmpfs on /run/user/114 type tmpfs (rw,nosuid,nodev,relatime,size=805836k,mode=700,uid=114,gid=116)
gvfsd-fuse on /run/user/114/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=114,group_id=116)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=805836k,mode=700,uid=1000,gid=1000)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
/dev/sda5 on /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-amd64/buildchroot-target/rootfs/home/myo/work/isar/src/trunk/isar/build/tmp type ext4 (rw,relatime,data=ordered)
udev on /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-amd64/buildchroot-target/rootfs/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-image-base/rootfs/dev type devtmpfs (rw,nosuid,relatime,size=4015904k,nr_inodes=1003976,mode=755)
devpts on /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-amd64/buildchroot-target/rootfs/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-image-base/rootfs/dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-amd64/buildchroot-target/rootfs/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-image-base/rootfs/dev/shm type tmpfs (rw,nosuid,nodev)
sysfs on /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-amd64/buildchroot-target/rootfs/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-image-base/rootfs/sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-amd64/buildchroot-target/rootfs/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-image-base/rootfs/sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-amd64/buildchroot-target/rootfs/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-image-base/rootfs/sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
udev on /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-amd64/buildchroot-target/rootfs/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-armhf/isar-image-base/rootfs/dev type devtmpfs (rw,nosuid,relatime,size=4015904k,nr_inodes=1003976,mode=755)
devpts on /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-amd64/buildchroot-target/rootfs/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-armhf/isar-image-base/rootfs/dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-amd64/buildchroot-target/rootfs/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-armhf/isar-image-base/rootfs/dev/shm type tmpfs (rw,nosuid,nodev)
sysfs on /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-amd64/buildchroot-target/rootfs/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-armhf/isar-image-base/rootfs/sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-amd64/buildchroot-target/rootfs/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-armhf/isar-image-base/rootfs/sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-amd64/buildchroot-target/rootfs/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-armhf/isar-image-base/rootfs/sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)

[-- Attachment #3: mount_before.txt --]
[-- Type: text/plain, Size: 2875 bytes --]

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=4015904k,nr_inodes=1003976,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=805840k,mode=755)
/dev/sda3 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=38,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=235)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
/dev/sda5 on /home type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=ascii,shortname=mixed,quiet,utf8,errors=remount-ro)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
tmpfs on /run/user/114 type tmpfs (rw,nosuid,nodev,relatime,size=805836k,mode=700,uid=114,gid=116)
gvfsd-fuse on /run/user/114/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=114,group_id=116)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=805836k,mode=700,uid=1000,gid=1000)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)