Re: schroot umount target is busy

["'cedric.hombourger@siemens.com' via isar-users" <isar-users@googlegroups.com>]

On Mon, 2024-07-08 at 17:13 +0200, Jan Kiszka wrote:
> On 08.07.24 16:17, Gökhan Çetin wrote:
> > Hi everyone,
> >
> > Similarly, we have been observing do_image tasks fail sporadically
> > in
> > CI/CD pipelines for a while.
> > You can see a few examples below:
> >
> > INFO - | E: 10mount: umount:
> > /run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570-
> > dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b: target is
> > busy.
> > INFO - | E: 10mount: rmdir: failed to remove
> > '/var/run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570-
> > dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b': Device
> > or resource busy
> > INFO - | E:
> > isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-
> > b359-48ad-91f4-2962f4bdf32b: Chroot setup failed: stage=setup-stop
> > INFO - NOTE: recipe .......: task do_image_swu: Failed
> > ERROR - ERROR: Task (/builds/.......bb:do_image_swu) failed with
> > exit
> > code '1'
> >
> > INFO - | E: 10mount: umount:
> > /run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade-
> > 57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82: target is
> > busy.
> > INFO - | E: 10mount: rmdir: failed to remove
> > '/var/run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade-
> > 57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82': Device
> > or resource busy
> > INFO - | E:
> > isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-
> > 8c19-4fc8-9b5e-d2ccb3d67d82: Chroot setup failed: stage=setup-stop
> > INFO - NOTE: recipe ......: task do_image_ova: Failed
> > ERROR - ERROR: Task (/builds/.....bb:do_image_ova) failed with exit
> > code '1'
> >
> > Do you have any information regarding the root cause?
>
> Trying once more to pull Cedric in (I think I used the wrong address
> last time): Weren't you trying to resolve that with a lazy umount? Or
> was it for a different corner of isar?

Correct. On our side, this issue is seen sporadically on servers
managed by our IT where they have some proprietary / black-box services
scanning the file-system when new mounts are discovered...

Have not found an elegant way to fix or work-around this (and obviously
no luck in getting IT to remove their software or configure it
differently)

>
> But even if we lazily umount, this won't help when the next command
> is
> trying to rmdir the mount point...
>
> Does anyone have a feeling when this started? Does it correlate with
> any
> major changes in isar?
>
> Jan
>
> >
> > Thanks,
> > Gokhan
> >
> > On Wednesday, June 5, 2024 at 11:55:34 AM UTC+3 Jan Kiszka wrote:
> >
> >     On 05.06.24 10:47, 'Schultschik, Sven' via isar-users wrote:
> >     > Hi all,
> >     >
> >     > Our ISAR setup is running smooth on many machines different
> > sizes.
> >     But
> >     > on our CI/CD pipeline it throws from time to time an error.
> >     Starting it
> >     > over again it works.
> >     >
> >     > Does someone knows this kind of error and where I have to
> > look or how
> >     > to fix?
> >     >
> >     > 2024-06-05 04:00:41 - INFO - | Processing triggers for man-db
> >     > (2.11.2-2) ...
> >     > 2024-06-05 04:00:41 - INFO - | e2fsck 1.47.0 (5-Feb-2023)
> >     > 2024-06-05 04:00:41 - INFO - | Pass 1: Checking inodes,
> > blocks, and
> >     > sizes
> >     > 2024-06-05 04:00:41 - INFO - | Pass 2: Checking directory
> > structure
> >     > 2024-06-05 04:00:41 - INFO - | Pass 3: Checking directory
> >     > connectivity
> >     > 2024-06-05 04:00:41 - INFO - | Pass 3A: Optimizing
> > directories
> >     > 2024-06-05 04:00:41 - INFO - | Pass 4: Checking reference
> > counts
> >     > 2024-06-05 04:00:41 - INFO - | Pass 5: Checking group summary
> >     > information
> >     > 2024-06-05 04:00:41 - INFO - |
> >     > 2024-06-05 04:00:41 - INFO - | /home/builder/edge-debug-
> > machine-
> >     > std/deploy/debug-machine-std.ext4: ***** FILE SYSTEM WAS
> > MODIFIED
> >     *****
> >     > 2024-06-05 04:00:41 - INFO - | /home/builder/debug-machine-
> >     > std/deploy/debug-machine-std.ext4: 25039/77600 files (0.1%
> > non-
> >     > contiguous), 292861/310076 blocks
> >     > 2024-06-05 04:00:41 - INFO - | E: 10mount: umount:
> >     > /run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd-
> > 11546112ec7c-
> >     > 982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3: target is busy.
> >     > 2024-06-05 04:00:41 - INFO - | E: 10mount: rmdir: failed to
> > remove
> >     > '/var/run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd-
> >     > 11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3':
> > Device or
> >     > resource busy
> >     > 2024-06-05 04:00:41 - INFO - | E: isar-builder-e7f8d532-aa0e-
> > 41cf-
> >     > 85dd-11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4-
> > f9629d1a55f3: Chroot
> >     > setup failed: stage=setup-stop
> >     > 2024-06-05 04:00:41 - INFO - NOTE: recipe debug-1.0-r0: task
> >     > do_image_ext4: Failed
> >     > 2024-06-05 04:00:41 - ERROR - ERROR: Task
> > (/build/../repo/recipes-
> >     > core/images/debug.bb:do_image_ext4) failed with exit code '1'
> >     > 2024-06-05 04:00:42 - INFO - NOTE: Tasks Summary: Attempted
> > 896
> >     > tasks of which 0 didn't need to be rerun and 1 failed.
> >     > 2024-06-05 04:00:43 - INFO -
> >     > 2024-06-05 04:00:43 - INFO - Summary: 1 task failed:
> >     > 2024-06-05 04:00:43 - INFO - /build/../repo/recipes-
> >     > core/images/debug.bb:do_image_ext4
> >     > 2024-06-05 04:00:43 - INFO - Summary: There was 1 WARNING
> > message.
> >     > 2024-06-05 04:00:43 - INFO - Summary: There was 1 ERROR
> > message,
> >     > returning a non-zero exit code.
> >     > 2024-06-05 04:00:43 - ERROR - Command "/build$
> >     > /work/isar/bitbake/bin/bitbake -c build debug" failed
> >     > --- Error summary ---
> >     > ERROR: debug-1.0-r0 do_image_ext4:
> >     > ExecutionError('/build/tmp/work/arm64/debug-machine-std/1.0-
> >     > r0/temp/run.do_image_ext4.982511', 1, None, None)
> >     > ERROR: Logfile of failure stored in:
> > /build/tmp/work/arm64/debug-
> >     > machine-std/1.0-r0/temp/log.do_image_ext4.982511
> >     > ERROR: Task
> >     (/build/../repo/recipes-core/images/debug.bb:do_image_ext4)
> >     > failed with exit code '1'
> >     >
> >
> >     Cedric, weren't you seeing something very similar?
> >
> >     Jan
> >
> >     --
> >     Siemens AG, Technology
> >     Linux Expert Center
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "isar-users" group.
> > To unsubscribe from this group and stop receiving emails from it,
> > send
> > an email to isar-users+unsubscribe@googlegroups.com
> > <mailto:isar-users+unsubscribe@googlegroups.com>.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/isar-users/d2508b62-d2fb-44a7-936f-aa2367347662n%40googlegroups.com
> >  <
> > https://groups.google.com/d/msgid/isar-users/d2508b62-d2fb-44a7-936f-aa2367347662n%40googlegroups.com?utm_medium=email&utm_source=footer
> > >.
>

--
Cedric Hombourger
Siemens AG
http://www.siemens.com/

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/10335bb9aa9443e71402b883db9ab30c92af6355.camel%40siemens.com.