Re: [PATCH v3 0/5] Rebuild mount logic

[Anton Mikanovich <amikan@ilbers.de>]

21.06.2021 18:54, Jan Kiszka wrote:
> This one is still not stable (reverting my claim "current next builds
> fine"):
>
> NOTE: recipe buildchroot-target-1.0-r0: task do_rootfs_postprocess: Started
> ERROR: mc:rpi-stretch:buildchroot-target-1.0-r0 do_rootfs_postprocess: Error executing a python function in exec_python_func() autogenerated:
>
> The stack trace of python calls that resulted in this exception/failure was:
> File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
>       0001:
>   *** 0002:do_rootfs_postprocess(d)
>       0003:
> File: '/builds/Vs1BETDQ/0/ebsy/debian/isar/meta/classes/rootfs.bbclass', lineno: 298, function: do_rootfs_postprocess
>       0294:        for i, cmd in enumerate(cmds):
>       0295:            bb.build.exec_func(cmd, d)
>       0296:            progress_reporter.update(int(i / len(cmds) * 100))
>       0297:
>   *** 0298:    bb.build.exec_func('rootfs_undo_mounts', d)
>       0299:}
>       0300:addtask rootfs_postprocess before do_rootfs
>       0301:
>       0302:python do_rootfs() {
> File: '/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/build.py', lineno: 251, function: exec_func
>       0247:    with bb.utils.fileslocked(lockfiles):
>       0248:        if ispython:
>       0249:            exec_func_python(func, d, runfile, cwd=adir)
>       0250:        else:
>   *** 0251:            exec_func_shell(func, d, runfile, cwd=adir)
>       0252:
>       0253:    try:
>       0254:        curcwd = os.getcwd()
>       0255:    except:
> File: '/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/build.py', lineno: 452, function: exec_func_shell
>       0448:    with open(fifopath, 'r+b', buffering=0) as fifo:
>       0449:        try:
>       0450:            bb.debug(2, "Executing shell function %s" % func)
>       0451:            with open(os.devnull, 'r+') as stdin, logfile:
>   *** 0452:                bb.process.run(cmd, shell=False, stdin=stdin, log=logfile, extrafiles=[(fifo,readfifo)])
>       0453:        finally:
>       0454:            os.unlink(fifopath)
>       0455:
>       0456:    bb.debug(2, "Shell function %s finished" % func)
> File: '/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/process.py', lineno: 182, function: run
>       0178:        if not stderr is None:
>       0179:            stderr = stderr.decode("utf-8")
>       0180:
>       0181:    if pipe.returncode != 0:
>   *** 0182:        raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
>       0183:    return stdout, stderr
> Exception: bb.process.ExecutionError: Execution of '/builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/temp/run.rootfs_undo_mounts.19080' failed with exit code 32:
> umount: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/sys/fs/cgroup: target is busy.
> WARNING: exit code 32 from a shell command.
>
>
> ERROR: Logfile of failure stored in: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/temp/log.do_rootfs_postprocess.19080
> NOTE: recipe buildchroot-target-1.0-r0: task do_rootfs_postprocess: Failed
> ERROR: Task (mc:rpi-stretch:/builds/Vs1BETDQ/0/ebsy/debian/isar/meta/recipes-dev
Looks like a race.
It was not a problem when we left mounts till the end of the build.
I will try to look for any possible reasons.
> And then also
>
> NOTE: Tasks Summary: Attempted 1181 tasks of which 214 didn't need to be rerun and 1 failed.
> WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev left mounted, unmounting...
> WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev/pts left mounted, unmounting...
> WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev/mqueue left mounted, unmounting...
> WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev/termination-log left mounted, unmounting...
> WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev/shm left mounted, unmounting...
> WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/proc left mounted, unmounting...
> WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/sys left mounted, unmounting...
> WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/sys/fs/cgroup left mounted, unmounting...
> WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/sys/fs/cgroup/cpuset left mounted, unmounting...
> WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/downloads left mounted, unmounting...
>
> Summary: 1 task failed:
>    mc:rpi-stretch:/builds/Vs1BETDQ/0/ebsy/debian/isar/meta/recipes-devtools/buildchroot/buildchroot-target.bb:do_rootfs_postprocess
>
>
> Though I suspect and hope the latter is related to the former.
Yes, it's just a 'visualization' of mount points cleanup at finalize 
task, called just after the build was failed.
It supposed to be empty for successful builds, but not for failed ones.
> Retrying, to see if we have a race (likely) or something stable.
>
> Jan

-- 
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov