On 08.07.24 16:17, Gökhan Çetin wrote:
> Hi everyone,
>
> Similarly, we have been observing do_image tasks fail sporadically in
> CI/CD pipelines for a while.
> You can see a few examples below:
>
> INFO - | E: 10mount: umount:
> /run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b: target is busy.
> INFO - | E: 10mount: rmdir: failed to remove
> '/var/run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b': Device or resource busy
> INFO - | E:
> isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b: Chroot setup failed: stage=setup-stop
> INFO - NOTE: recipe .......: task do_image_swu: Failed
> ERROR - ERROR: Task (/builds/.......bb:do_image_swu) failed with exit
> code '1'
>
> INFO - | E: 10mount: umount:
> /run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82: target is busy.
> INFO - | E: 10mount: rmdir: failed to remove
> '/var/run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82': Device or resource busy
> INFO - | E:
> isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82: Chroot setup failed: stage=setup-stop
> INFO - NOTE: recipe ......: task do_image_ova: Failed
> ERROR - ERROR: Task (/builds/.....bb:do_image_ova) failed with exit code '1'
>
> Do you have any information regarding the root cause?
Trying once more to pull Cedric in (I think I used the wrong address
last time): Weren't you trying to resolve that with a lazy umount? Or
was it for a different corner of isar?
But even if we lazily umount, this won't help when the next command is
trying to rmdir the mount point...
Does anyone have a feeling when this started? Does it correlate with any
major changes in isar?
Jan
>
> Thanks,
> Gokhan
>
> On Wednesday, June 5, 2024 at 11:55:34 AM UTC+3 Jan Kiszka wrote:
>
> On 05.06.24 10:47, 'Schultschik, Sven' via isar-users wrote:
> > Hi all,
> >
> > Our ISAR setup is running smooth on many machines different sizes.
> But
> > on our CI/CD pipeline it throws from time to time an error.
> Starting it
> > over again it works.
> >
> > Does someone knows this kind of error and where I have to look or how
> > to fix?
> >
> > 2024-06-05 04:00:41 - INFO - | Processing triggers for man-db
> > (2.11.2-2) ...
> > 2024-06-05 04:00:41 - INFO - | e2fsck 1.47.0 (5-Feb-2023)
> > 2024-06-05 04:00:41 - INFO - | Pass 1: Checking inodes, blocks, and
> > sizes
> > 2024-06-05 04:00:41 - INFO - | Pass 2: Checking directory structure
> > 2024-06-05 04:00:41 - INFO - | Pass 3: Checking directory
> > connectivity
> > 2024-06-05 04:00:41 - INFO - | Pass 3A: Optimizing directories
> > 2024-06-05 04:00:41 - INFO - | Pass 4: Checking reference counts
> > 2024-06-05 04:00:41 - INFO - | Pass 5: Checking group summary
> > information
> > 2024-06-05 04:00:41 - INFO - |
> > 2024-06-05 04:00:41 - INFO - | /home/builder/edge-debug-machine-
> > std/deploy/debug-machine-std.ext4: ***** FILE SYSTEM WAS MODIFIED
> *****
> > 2024-06-05 04:00:41 - INFO - | /home/builder/debug-machine-
> > std/deploy/debug-machine-std.ext4: 25039/77600 files (0.1% non-
> > contiguous), 292861/310076 blocks
> > 2024-06-05 04:00:41 - INFO - | E: 10mount: umount:
> > /run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd-11546112ec7c-
> > 982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3: target is busy.
> > 2024-06-05 04:00:41 - INFO - | E: 10mount: rmdir: failed to remove
> > '/var/run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd-
> > 11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3': Device or
> > resource busy
> > 2024-06-05 04:00:41 - INFO - | E: isar-builder-e7f8d532-aa0e-41cf-
> > 85dd-11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3: Chroot
> > setup failed: stage=setup-stop
> > 2024-06-05 04:00:41 - INFO - NOTE: recipe debug-1.0-r0: task
> > do_image_ext4: Failed
> > 2024-06-05 04:00:41 - ERROR - ERROR: Task (/build/../repo/recipes-
> > core/images/debug.bb:do_image_ext4) failed with exit code '1'
> > 2024-06-05 04:00:42 - INFO - NOTE: Tasks Summary: Attempted 896
> > tasks of which 0 didn't need to be rerun and 1 failed.
> > 2024-06-05 04:00:43 - INFO -
> > 2024-06-05 04:00:43 - INFO - Summary: 1 task failed:
> > 2024-06-05 04:00:43 - INFO - /build/../repo/recipes-
> > core/images/debug.bb:do_image_ext4
> > 2024-06-05 04:00:43 - INFO - Summary: There was 1 WARNING message.
> > 2024-06-05 04:00:43 - INFO - Summary: There was 1 ERROR message,
> > returning a non-zero exit code.
> > 2024-06-05 04:00:43 - ERROR - Command "/build$
> > /work/isar/bitbake/bin/bitbake -c build debug" failed
> > --- Error summary ---
> > ERROR: debug-1.0-r0 do_image_ext4:
> > ExecutionError('/build/tmp/work/arm64/debug-machine-std/1.0-
> > r0/temp/run.do_image_ext4.982511', 1, None, None)
> > ERROR: Logfile of failure stored in: /build/tmp/work/arm64/debug-
> > machine-std/1.0-r0/temp/log.do_image_ext4.982511
> > ERROR: Task
> (/build/../repo/recipes-core/images/debug.bb:do_image_ext4)
> > failed with exit code '1'
> >
>
> Cedric, weren't you seeing something very similar?
>
> Jan
>
> --
> Siemens AG, Technology
> Linux Expert Center
>
> --
> You received this message because you are subscribed to the Google
> Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to isar-users+...@googlegroups.com
> <mailto:isar-users+...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-users/d2508b62-d2fb-44a7-936f-aa2367347662n%40googlegroups.com <https://groups.google.com/d/msgid/isar-users/d2508b62-d2fb-44a7-936f-aa2367347662n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
Siemens AG, Technology
Linux Expert Center