Re: [PATCH v3 0/5] Rebuild mount logic

[Jan Kiszka <jan.kiszka@siemens.com>]

On 08.05.21 08:25, Anton Mikanovich wrote:
> Mounting is perfomed very inconsistent currently in Isar. A lot of mount
> points are left after task execution and stays mounted until cleanup in
> build_completed hook.
> 
> We already experienced issues with stuck forever on unmounting at
> "Couldn't unmount, retrying..." error.
> It is also quite easy to face with a lot of ramaining mounts after the
> failed or interrupted manual builds.
> 
> This patchset implements unmounting paths after usage. In the cases of
> mount sharing corresponding request counting is performed, so real mount
> will be done only at first request, real unmount will be done only after
> the last task will try to unmount the path.
> 
> Meantime build_completed will use emergency only, so following warning
> added to show something goes wrong.
> 
> ---
> Changes since v2:
> - Prevent race conditions.
> - Fix building qemuarm64-stretch.
> - Minimize locks.
> - Cleanup counter files.
> - Unmount /proc/sys/fs/binfmt_misc also.
> Changes since v1:
> - Make all mounts be unmounted after use.
> - Warn about lost mounts after finish.
> 
> Anton Mikanovich (5):
>   dpkg: Make mount buildroot reliable
>   buildchroot: Unmount buildchroot mounts if not needed
>   rootfs: Unmount rootfs mounts if not needed
>   wic: Unmount dirs after usage
>   events: Warn if mounted paths left
> 
>  meta/classes/buildchroot.bbclass              |  47 +++++++
>  meta/classes/dpkg-base.bbclass                | 127 +++++++++++-------
>  meta/classes/dpkg-gbp.bbclass                 |   8 +-
>  meta/classes/dpkg.bbclass                     |  14 +-
>  meta/classes/ext4-img.bbclass                 |   2 +
>  meta/classes/fit-img.bbclass                  |   2 +
>  meta/classes/image-tools-extension.bbclass    |   2 +
>  meta/classes/image.bbclass                    |  24 ++--
>  meta/classes/initramfs.bbclass                |   2 +
>  meta/classes/isar-events.bbclass              |   1 +
>  meta/classes/rootfs.bbclass                   |  62 ++++++++-
>  meta/classes/ubi-img.bbclass                  |   2 +
>  meta/classes/ubifs-img.bbclass                |   2 +
>  meta/classes/wic-img.bbclass                  |  10 ++
>  .../buildchroot/buildchroot.inc               |   8 ++
>  15 files changed, 232 insertions(+), 81 deletions(-)
> 

This one is still not stable (reverting my claim "current next builds 
fine"):

NOTE: recipe buildchroot-target-1.0-r0: task do_rootfs_postprocess: Started
ERROR: mc:rpi-stretch:buildchroot-target-1.0-r0 do_rootfs_postprocess: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:do_rootfs_postprocess(d)
     0003:
File: '/builds/Vs1BETDQ/0/ebsy/debian/isar/meta/classes/rootfs.bbclass', lineno: 298, function: do_rootfs_postprocess
     0294:        for i, cmd in enumerate(cmds):
     0295:            bb.build.exec_func(cmd, d)
     0296:            progress_reporter.update(int(i / len(cmds) * 100))
     0297:
 *** 0298:    bb.build.exec_func('rootfs_undo_mounts', d)
     0299:}
     0300:addtask rootfs_postprocess before do_rootfs
     0301:
     0302:python do_rootfs() {
File: '/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/build.py', lineno: 251, function: exec_func
     0247:    with bb.utils.fileslocked(lockfiles):
     0248:        if ispython:
     0249:            exec_func_python(func, d, runfile, cwd=adir)
     0250:        else:
 *** 0251:            exec_func_shell(func, d, runfile, cwd=adir)
     0252:
     0253:    try:
     0254:        curcwd = os.getcwd()
     0255:    except:
File: '/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/build.py', lineno: 452, function: exec_func_shell
     0448:    with open(fifopath, 'r+b', buffering=0) as fifo:
     0449:        try:
     0450:            bb.debug(2, "Executing shell function %s" % func)
     0451:            with open(os.devnull, 'r+') as stdin, logfile:
 *** 0452:                bb.process.run(cmd, shell=False, stdin=stdin, log=logfile, extrafiles=[(fifo,readfifo)])
     0453:        finally:
     0454:            os.unlink(fifopath)
     0455:
     0456:    bb.debug(2, "Shell function %s finished" % func)
File: '/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/process.py', lineno: 182, function: run
     0178:        if not stderr is None:
     0179:            stderr = stderr.decode("utf-8")
     0180:
     0181:    if pipe.returncode != 0:
 *** 0182:        raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
     0183:    return stdout, stderr
Exception: bb.process.ExecutionError: Execution of '/builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/temp/run.rootfs_undo_mounts.19080' failed with exit code 32:
umount: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/sys/fs/cgroup: target is busy.
WARNING: exit code 32 from a shell command.


ERROR: Logfile of failure stored in: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/temp/log.do_rootfs_postprocess.19080
NOTE: recipe buildchroot-target-1.0-r0: task do_rootfs_postprocess: Failed
ERROR: Task (mc:rpi-stretch:/builds/Vs1BETDQ/0/ebsy/debian/isar/meta/recipes-dev


And then also 

NOTE: Tasks Summary: Attempted 1181 tasks of which 214 didn't need to be rerun and 1 failed.
WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev left mounted, unmounting...
WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev/pts left mounted, unmounting...
WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev/mqueue left mounted, unmounting...
WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev/termination-log left mounted, unmounting...
WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev/shm left mounted, unmounting...
WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/proc left mounted, unmounting...
WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/sys left mounted, unmounting...
WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/sys/fs/cgroup left mounted, unmounting...
WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/sys/fs/cgroup/cpuset left mounted, unmounting...
WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/downloads left mounted, unmounting...

Summary: 1 task failed:
  mc:rpi-stretch:/builds/Vs1BETDQ/0/ebsy/debian/isar/meta/recipes-devtools/buildchroot/buildchroot-target.bb:do_rootfs_postprocess


Though I suspect and hope the latter is related to the former.

Retrying, to see if we have a race (likely) or something stable.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux