Re: [HELP] tmp/work/debian-bullseye-arm64/isar-image-base-qemuarm64/1.0-r0/mnt/rootfs: target is busy

["cedric.hombourger@siemens.com" <cedric.hombourger@siemens.com>]

On Mon, 2024-02-26 at 13:27 +0200, Anton Mikanovich wrote:
> 26/02/2024 12:26, 'cedric.hombourger@siemens.com' via isar-users
> wrote:
> > Hello,
> > 
> > Seeing some sporadic failures with the test suite (but also with
> > builds
> > of our Isar-based product) when rootfs_install_sstate_prepare gets
> > executed:
> > 
> > DEBUG: Executing shell function rootfs_install_sstate_prepare
> > umount: /home/sutlej/isar/build/tmp/work/debian-bullseye-
> > arm64/isar-
> > image-base-qemuarm64/1.0-r0/mnt/rootfs: target is busy.
> > WARNING: exit code 32 from a shell command.
> > 
> > What's special about the machine I am running the builds on is that
> > /proc/cpuinfo reports 64 processors hence builds get massively
> > parallelized
> > 
> > I wasn't able to get to the bottom of this issue and understand why
> > the
> > bind mount is busy since rootfs_install_sstate_prepare creates,
> > uses
> > and removes that bind mount in the same function.
> > 
> > As a work-around, a lazy umount could be used but that annoys me as
> > I'd
> > like to understand what could cause this. Any ideas?
> > 
> > Cedric
> > 
> Hello Cedric,
> Does this issue happens with testsuite only or on manual build also?

I have unfortunately faced this issue with both

> How much targets and how much images for the same target are been 
> executed in
> this bitbake call?

The advocado logs suggest 4 different multiconfig builds being executed
in parallel but only 1 arm64 (so only 1 kid in town possibly using
tmp/work/debian-bullseye-arm64/isar-image-base-qemuarm64/1.0-
r0/mnt/rootfs if I am not mistaken)

Building ['mc:qemuamd64-bullseye:isar-image-ci', 'mc:qemuarm-
bullseye:isar-image-base', 'mc:qemuarm-bullseye:isar-image-
base:do_populate_sdk', 'mc:qemuarm64-bullseye:isar-image-base'


>  Which Isar commit are you using?

94ab67581ffdd96fd225772b0fd921eaaad86ff6
fix: clean apt-cache dirs in dpkg_runbuild as root

> We can try to reproduce the issue but need some details of your
> setup.

Sure thing. I first wanted to reach to check if anyone had faced a
similar issue. It does not sound like it.

> 
> The issue like that can be caused by double-mount scenario we already
> faced
> with: if some of the folders in Isar sources/builddir path is mounted
> inside
> itself or its parent, it can cause doubling the mount and umount will
> fail.
> 
> Here is an example of this scenario:
> $ mkdir dirA
> $ mkdir dirB
> $ echo "sudo umount ~/dirB" > dirA/um
> $ sudo mount --bind dirA dirB
> $ mount | grep dir
> /dev/sdb1 on /path/dirB type ext4 (rw,relatime)
> $ sudo mount --bind dirA dirB
> $ mount | grep dir
> /dev/sdb1 on /path/dirB type ext4 (rw,relatime)
> /dev/sdb1 on /path/dirB type ext4 (rw,relatime)
> /dev/sdb1 on /path/dirA type ext4 (rw,relatime)
> $ bash ~/dirA/um
> umount: /path/dirB: target is busy.
> 
> As a debug you can try to add some lsof/mount/findmnt inside the
> rootfs_install_sstate_prepare task before umount/

Yes that sounds like a good next step to get to the bottom of this
issue. Thanks!

> 
> P.S. lazy umount is not an option because it will only cover the
> issue which
> can cause other issues to appear later on.

100% agreed - I did not even dare to post that work-around to the
mailing-list for that exact reason :)

> 

-- 
Cedric Hombourger
Siemens AG
www.siemens.com