[HELP] tmp/work/debian-bullseye-arm64/isar-image-base-qemuarm64/1.0-r0/mnt/rootfs: target is busy

[HELP] tmp/work/debian-bullseye-arm64/isar-image-base-qemuarm64/1.0-r0/mnt/rootfs: target is busy

[cedric.hombourger]

Hello,

Seeing some sporadic failures with the test suite (but also with builds
of our Isar-based product) when rootfs_install_sstate_prepare gets
executed:

DEBUG: Executing shell function rootfs_install_sstate_prepare
umount: /home/sutlej/isar/build/tmp/work/debian-bullseye-arm64/isar-
image-base-qemuarm64/1.0-r0/mnt/rootfs: target is busy.
WARNING: exit code 32 from a shell command. 

What's special about the machine I am running the builds on is that
/proc/cpuinfo reports 64 processors hence builds get massively
parallelized

I wasn't able to get to the bottom of this issue and understand why the
bind mount is busy since rootfs_install_sstate_prepare creates, uses
and removes that bind mount in the same function.

As a work-around, a lazy umount could be used but that annoys me as I'd
like to understand what could cause this. Any ideas?

Cedric

Re: [HELP] tmp/work/debian-bullseye-arm64/isar-image-base-qemuarm64/1.0-r0/mnt/rootfs: target is busy

[Jan Kiszka]

On 26.02.24 11:26, 'cedric.hombourger@siemens.com' via isar-users wrote:
> 
> Hello,
> 
> Seeing some sporadic failures with the test suite (but also with builds
> of our Isar-based product) when rootfs_install_sstate_prepare gets
> executed:
> 
> DEBUG: Executing shell function rootfs_install_sstate_prepare
> umount: /home/sutlej/isar/build/tmp/work/debian-bullseye-arm64/isar-
> image-base-qemuarm64/1.0-r0/mnt/rootfs: target is busy.
> WARNING: exit code 32 from a shell command. 
> 
> What's special about the machine I am running the builds on is that
> /proc/cpuinfo reports 64 processors hence builds get massively
> parallelized
> 
> I wasn't able to get to the bottom of this issue and understand why the
> bind mount is busy since rootfs_install_sstate_prepare creates, uses
> and removes that bind mount in the same function.
> 
> As a work-around, a lazy umount could be used but that annoys me as I'd
> like to understand what could cause this. Any ideas?
> 

This is indeed weird: ${WORKDIR}/mnt/rootfs is only mounted by
rootfs_install_sstate_prepare in vanilla isar, and that happens under
isar.lock - close to impossible to have a race here. But did you check
in your recipes on top (assuming the issue is not reproducible for you
with vanilla isar alone) that there is no other usage of
${WORKDIR}/mnt/rootfs, possibly unlocked then?

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [HELP] tmp/work/debian-bullseye-arm64/isar-image-base-qemuarm64/1.0-r0/mnt/rootfs: target is busy

[cedric.hombourger]

On Mon, 2024-02-26 at 11:32 +0100, Jan Kiszka wrote:
> On 26.02.24 11:26, 'cedric.hombourger@siemens.com' via isar-users
> wrote:
> > 
> > Hello,
> > 
> > Seeing some sporadic failures with the test suite (but also with
> > builds
> > of our Isar-based product) when rootfs_install_sstate_prepare gets
> > executed:
> > 
> > DEBUG: Executing shell function rootfs_install_sstate_prepare
> > umount: /home/sutlej/isar/build/tmp/work/debian-bullseye-
> > arm64/isar-
> > image-base-qemuarm64/1.0-r0/mnt/rootfs: target is busy.
> > WARNING: exit code 32 from a shell command. 
> > 
> > What's special about the machine I am running the builds on is that
> > /proc/cpuinfo reports 64 processors hence builds get massively
> > parallelized
> > 
> > I wasn't able to get to the bottom of this issue and understand why
> > the
> > bind mount is busy since rootfs_install_sstate_prepare creates,
> > uses
> > and removes that bind mount in the same function.
> > 
> > As a work-around, a lazy umount could be used but that annoys me as
> > I'd
> > like to understand what could cause this. Any ideas?
> > 
> 
> This is indeed weird: ${WORKDIR}/mnt/rootfs is only mounted by
> rootfs_install_sstate_prepare in vanilla isar, and that happens under
> isar.lock - close to impossible to have a race here. But did you
> check
> in your recipes on top (assuming the issue is not reproducible for
> you
> with vanilla isar alone) that there is no other usage of
> ${WORKDIR}/mnt/rootfs, possibly unlocked then?

That's the thing. I ran into this issue twice this morning while using
Isar *without* our product layers. First with a small patch to Isar for
later submission. I then started a ci_build -T fast from a clean tree
and hit the exact same issue.

I wonder if it could be caused by processes run by our IT to scan the
file-system when files are added/modified (seeing a number of admin
processes running proprietary "security" tools). That's the only theory
I could come up with. I may need to create some sort of a script that
mimics what we do in rootfs_install_sstate_prepare while checking with
lsof in parallel for concurrent accesses to files from the bind mount

> 
> Jan
> 

Re: [HELP] tmp/work/debian-bullseye-arm64/isar-image-base-qemuarm64/1.0-r0/mnt/rootfs: target is busy

[Anton Mikanovich]

26/02/2024 12:26, 'cedric.hombourger@siemens.com' via isar-users wrote:
> Hello,
>
> Seeing some sporadic failures with the test suite (but also with builds
> of our Isar-based product) when rootfs_install_sstate_prepare gets
> executed:
>
> DEBUG: Executing shell function rootfs_install_sstate_prepare
> umount: /home/sutlej/isar/build/tmp/work/debian-bullseye-arm64/isar-
> image-base-qemuarm64/1.0-r0/mnt/rootfs: target is busy.
> WARNING: exit code 32 from a shell command.
>
> What's special about the machine I am running the builds on is that
> /proc/cpuinfo reports 64 processors hence builds get massively
> parallelized
>
> I wasn't able to get to the bottom of this issue and understand why the
> bind mount is busy since rootfs_install_sstate_prepare creates, uses
> and removes that bind mount in the same function.
>
> As a work-around, a lazy umount could be used but that annoys me as I'd
> like to understand what could cause this. Any ideas?
>
> Cedric
>
Hello Cedric,
Does this issue happens with testsuite only or on manual build also?
How much targets and how much images for the same target are been 
executed in
this bitbake call? Which Isar commit are you using?
We can try to reproduce the issue but need some details of your setup.

The issue like that can be caused by double-mount scenario we already faced
with: if some of the folders in Isar sources/builddir path is mounted inside
itself or its parent, it can cause doubling the mount and umount will fail.

Here is an example of this scenario:
$ mkdir dirA
$ mkdir dirB
$ echo "sudo umount ~/dirB" > dirA/um
$ sudo mount --bind dirA dirB
$ mount | grep dir
/dev/sdb1 on /path/dirB type ext4 (rw,relatime)
$ sudo mount --bind dirA dirB
$ mount | grep dir
/dev/sdb1 on /path/dirB type ext4 (rw,relatime)
/dev/sdb1 on /path/dirB type ext4 (rw,relatime)
/dev/sdb1 on /path/dirA type ext4 (rw,relatime)
$ bash ~/dirA/um
umount: /path/dirB: target is busy.

As a debug you can try to add some lsof/mount/findmnt inside the
rootfs_install_sstate_prepare task before umount/

P.S. lazy umount is not an option because it will only cover the issue which
can cause other issues to appear later on.

Re: [HELP] tmp/work/debian-bullseye-arm64/isar-image-base-qemuarm64/1.0-r0/mnt/rootfs: target is busy

[cedric.hombourger]

On Mon, 2024-02-26 at 13:27 +0200, Anton Mikanovich wrote:
> 26/02/2024 12:26, 'cedric.hombourger@siemens.com' via isar-users
> wrote:
> > Hello,
> > 
> > Seeing some sporadic failures with the test suite (but also with
> > builds
> > of our Isar-based product) when rootfs_install_sstate_prepare gets
> > executed:
> > 
> > DEBUG: Executing shell function rootfs_install_sstate_prepare
> > umount: /home/sutlej/isar/build/tmp/work/debian-bullseye-
> > arm64/isar-
> > image-base-qemuarm64/1.0-r0/mnt/rootfs: target is busy.
> > WARNING: exit code 32 from a shell command.
> > 
> > What's special about the machine I am running the builds on is that
> > /proc/cpuinfo reports 64 processors hence builds get massively
> > parallelized
> > 
> > I wasn't able to get to the bottom of this issue and understand why
> > the
> > bind mount is busy since rootfs_install_sstate_prepare creates,
> > uses
> > and removes that bind mount in the same function.
> > 
> > As a work-around, a lazy umount could be used but that annoys me as
> > I'd
> > like to understand what could cause this. Any ideas?
> > 
> > Cedric
> > 
> Hello Cedric,
> Does this issue happens with testsuite only or on manual build also?

I have unfortunately faced this issue with both

> How much targets and how much images for the same target are been 
> executed in
> this bitbake call?

The advocado logs suggest 4 different multiconfig builds being executed
in parallel but only 1 arm64 (so only 1 kid in town possibly using
tmp/work/debian-bullseye-arm64/isar-image-base-qemuarm64/1.0-
r0/mnt/rootfs if I am not mistaken)

Building ['mc:qemuamd64-bullseye:isar-image-ci', 'mc:qemuarm-
bullseye:isar-image-base', 'mc:qemuarm-bullseye:isar-image-
base:do_populate_sdk', 'mc:qemuarm64-bullseye:isar-image-base'


>  Which Isar commit are you using?

94ab67581ffdd96fd225772b0fd921eaaad86ff6
fix: clean apt-cache dirs in dpkg_runbuild as root

> We can try to reproduce the issue but need some details of your
> setup.

Sure thing. I first wanted to reach to check if anyone had faced a
similar issue. It does not sound like it.

> 
> The issue like that can be caused by double-mount scenario we already
> faced
> with: if some of the folders in Isar sources/builddir path is mounted
> inside
> itself or its parent, it can cause doubling the mount and umount will
> fail.
> 
> Here is an example of this scenario:
> $ mkdir dirA
> $ mkdir dirB
> $ echo "sudo umount ~/dirB" > dirA/um
> $ sudo mount --bind dirA dirB
> $ mount | grep dir
> /dev/sdb1 on /path/dirB type ext4 (rw,relatime)
> $ sudo mount --bind dirA dirB
> $ mount | grep dir
> /dev/sdb1 on /path/dirB type ext4 (rw,relatime)
> /dev/sdb1 on /path/dirB type ext4 (rw,relatime)
> /dev/sdb1 on /path/dirA type ext4 (rw,relatime)
> $ bash ~/dirA/um
> umount: /path/dirB: target is busy.
> 
> As a debug you can try to add some lsof/mount/findmnt inside the
> rootfs_install_sstate_prepare task before umount/

Yes that sounds like a good next step to get to the bottom of this
issue. Thanks!

> 
> P.S. lazy umount is not an option because it will only cover the
> issue which
> can cause other issues to appear later on.

100% agreed - I did not even dare to post that work-around to the
mailing-list for that exact reason :)

> 

-- 
Cedric Hombourger
Siemens AG
www.siemens.com