* schroot umount target is busy @ 2024-06-05 8:47 Schultschik, Sven 2024-06-05 8:55 ` Jan Kiszka 0 siblings, 1 reply; 11+ messages in thread From: Schultschik, Sven @ 2024-06-05 8:47 UTC (permalink / raw) To: isar-users Hi all, Our ISAR setup is running smooth on many machines different sizes. But on our CI/CD pipeline it throws from time to time an error. Starting it over again it works. Does someone knows this kind of error and where I have to look or how to fix? 2024-06-05 04:00:41 - INFO - | Processing triggers for man-db (2.11.2-2) ... 2024-06-05 04:00:41 - INFO - | e2fsck 1.47.0 (5-Feb-2023) 2024-06-05 04:00:41 - INFO - | Pass 1: Checking inodes, blocks, and sizes 2024-06-05 04:00:41 - INFO - | Pass 2: Checking directory structure 2024-06-05 04:00:41 - INFO - | Pass 3: Checking directory connectivity 2024-06-05 04:00:41 - INFO - | Pass 3A: Optimizing directories 2024-06-05 04:00:41 - INFO - | Pass 4: Checking reference counts 2024-06-05 04:00:41 - INFO - | Pass 5: Checking group summary information 2024-06-05 04:00:41 - INFO - | 2024-06-05 04:00:41 - INFO - | /home/builder/edge-debug-machine- std/deploy/debug-machine-std.ext4: ***** FILE SYSTEM WAS MODIFIED ***** 2024-06-05 04:00:41 - INFO - | /home/builder/debug-machine- std/deploy/debug-machine-std.ext4: 25039/77600 files (0.1% non- contiguous), 292861/310076 blocks 2024-06-05 04:00:41 - INFO - | E: 10mount: umount: /run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd-11546112ec7c- 982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3: target is busy. 2024-06-05 04:00:41 - INFO - | E: 10mount: rmdir: failed to remove '/var/run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd- 11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3': Device or resource busy 2024-06-05 04:00:41 - INFO - | E: isar-builder-e7f8d532-aa0e-41cf- 85dd-11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3: Chroot setup failed: stage=setup-stop 2024-06-05 04:00:41 - INFO - NOTE: recipe debug-1.0-r0: task do_image_ext4: Failed 2024-06-05 04:00:41 - ERROR - ERROR: Task (/build/../repo/recipes- core/images/debug.bb:do_image_ext4) failed with exit code '1' 2024-06-05 04:00:42 - INFO - NOTE: Tasks Summary: Attempted 896 tasks of which 0 didn't need to be rerun and 1 failed. 2024-06-05 04:00:43 - INFO - 2024-06-05 04:00:43 - INFO - Summary: 1 task failed: 2024-06-05 04:00:43 - INFO - /build/../repo/recipes- core/images/debug.bb:do_image_ext4 2024-06-05 04:00:43 - INFO - Summary: There was 1 WARNING message. 2024-06-05 04:00:43 - INFO - Summary: There was 1 ERROR message, returning a non-zero exit code. 2024-06-05 04:00:43 - ERROR - Command "/build$ /work/isar/bitbake/bin/bitbake -c build debug" failed --- Error summary --- ERROR: debug-1.0-r0 do_image_ext4: ExecutionError('/build/tmp/work/arm64/debug-machine-std/1.0- r0/temp/run.do_image_ext4.982511', 1, None, None) ERROR: Logfile of failure stored in: /build/tmp/work/arm64/debug- machine-std/1.0-r0/temp/log.do_image_ext4.982511 ERROR: Task (/build/../repo/recipes-core/images/debug.bb:do_image_ext4) failed with exit code '1' Thank you br Sven ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: schroot umount target is busy 2024-06-05 8:47 schroot umount target is busy Schultschik, Sven @ 2024-06-05 8:55 ` Jan Kiszka 2024-07-08 14:17 ` Gökhan Çetin 0 siblings, 1 reply; 11+ messages in thread From: Jan Kiszka @ 2024-06-05 8:55 UTC (permalink / raw) To: Schultschik, Sven, isar-users, Cedric Hombourger On 05.06.24 10:47, 'Schultschik, Sven' via isar-users wrote: > Hi all, > > Our ISAR setup is running smooth on many machines different sizes. But > on our CI/CD pipeline it throws from time to time an error. Starting it > over again it works. > > Does someone knows this kind of error and where I have to look or how > to fix? > > 2024-06-05 04:00:41 - INFO - | Processing triggers for man-db > (2.11.2-2) ... > 2024-06-05 04:00:41 - INFO - | e2fsck 1.47.0 (5-Feb-2023) > 2024-06-05 04:00:41 - INFO - | Pass 1: Checking inodes, blocks, and > sizes > 2024-06-05 04:00:41 - INFO - | Pass 2: Checking directory structure > 2024-06-05 04:00:41 - INFO - | Pass 3: Checking directory > connectivity > 2024-06-05 04:00:41 - INFO - | Pass 3A: Optimizing directories > 2024-06-05 04:00:41 - INFO - | Pass 4: Checking reference counts > 2024-06-05 04:00:41 - INFO - | Pass 5: Checking group summary > information > 2024-06-05 04:00:41 - INFO - | > 2024-06-05 04:00:41 - INFO - | /home/builder/edge-debug-machine- > std/deploy/debug-machine-std.ext4: ***** FILE SYSTEM WAS MODIFIED ***** > 2024-06-05 04:00:41 - INFO - | /home/builder/debug-machine- > std/deploy/debug-machine-std.ext4: 25039/77600 files (0.1% non- > contiguous), 292861/310076 blocks > 2024-06-05 04:00:41 - INFO - | E: 10mount: umount: > /run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd-11546112ec7c- > 982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3: target is busy. > 2024-06-05 04:00:41 - INFO - | E: 10mount: rmdir: failed to remove > '/var/run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd- > 11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3': Device or > resource busy > 2024-06-05 04:00:41 - INFO - | E: isar-builder-e7f8d532-aa0e-41cf- > 85dd-11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3: Chroot > setup failed: stage=setup-stop > 2024-06-05 04:00:41 - INFO - NOTE: recipe debug-1.0-r0: task > do_image_ext4: Failed > 2024-06-05 04:00:41 - ERROR - ERROR: Task (/build/../repo/recipes- > core/images/debug.bb:do_image_ext4) failed with exit code '1' > 2024-06-05 04:00:42 - INFO - NOTE: Tasks Summary: Attempted 896 > tasks of which 0 didn't need to be rerun and 1 failed. > 2024-06-05 04:00:43 - INFO - > 2024-06-05 04:00:43 - INFO - Summary: 1 task failed: > 2024-06-05 04:00:43 - INFO - /build/../repo/recipes- > core/images/debug.bb:do_image_ext4 > 2024-06-05 04:00:43 - INFO - Summary: There was 1 WARNING message. > 2024-06-05 04:00:43 - INFO - Summary: There was 1 ERROR message, > returning a non-zero exit code. > 2024-06-05 04:00:43 - ERROR - Command "/build$ > /work/isar/bitbake/bin/bitbake -c build debug" failed > --- Error summary --- > ERROR: debug-1.0-r0 do_image_ext4: > ExecutionError('/build/tmp/work/arm64/debug-machine-std/1.0- > r0/temp/run.do_image_ext4.982511', 1, None, None) > ERROR: Logfile of failure stored in: /build/tmp/work/arm64/debug- > machine-std/1.0-r0/temp/log.do_image_ext4.982511 > ERROR: Task (/build/../repo/recipes-core/images/debug.bb:do_image_ext4) > failed with exit code '1' > Cedric, weren't you seeing something very similar? Jan -- Siemens AG, Technology Linux Expert Center ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: schroot umount target is busy 2024-06-05 8:55 ` Jan Kiszka @ 2024-07-08 14:17 ` Gökhan Çetin 2024-07-08 15:13 ` 'Jan Kiszka' via isar-users 0 siblings, 1 reply; 11+ messages in thread From: Gökhan Çetin @ 2024-07-08 14:17 UTC (permalink / raw) To: isar-users [-- Attachment #1.1: Type: text/plain, Size: 5306 bytes --] Hi everyone, Similarly, we have been observing do_image tasks fail sporadically in CI/CD pipelines for a while. You can see a few examples below: INFO - | E: 10mount: umount: /run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b: target is busy. INFO - | E: 10mount: rmdir: failed to remove '/var/run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b': Device or resource busy INFO - | E: isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b: Chroot setup failed: stage=setup-stop INFO - NOTE: recipe .......: task do_image_swu: Failed ERROR - ERROR: Task (/builds/.......bb:do_image_swu) failed with exit code '1' INFO - | E: 10mount: umount: /run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82: target is busy. INFO - | E: 10mount: rmdir: failed to remove '/var/run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82': Device or resource busy INFO - | E: isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82: Chroot setup failed: stage=setup-stop INFO - NOTE: recipe ......: task do_image_ova: Failed ERROR - ERROR: Task (/builds/.....bb:do_image_ova) failed with exit code '1' Do you have any information regarding the root cause? Thanks, Gokhan On Wednesday, June 5, 2024 at 11:55:34 AM UTC+3 Jan Kiszka wrote: > On 05.06.24 10:47, 'Schultschik, Sven' via isar-users wrote: > > Hi all, > > > > Our ISAR setup is running smooth on many machines different sizes. But > > on our CI/CD pipeline it throws from time to time an error. Starting it > > over again it works. > > > > Does someone knows this kind of error and where I have to look or how > > to fix? > > > > 2024-06-05 04:00:41 - INFO - | Processing triggers for man-db > > (2.11.2-2) ... > > 2024-06-05 04:00:41 - INFO - | e2fsck 1.47.0 (5-Feb-2023) > > 2024-06-05 04:00:41 - INFO - | Pass 1: Checking inodes, blocks, and > > sizes > > 2024-06-05 04:00:41 - INFO - | Pass 2: Checking directory structure > > 2024-06-05 04:00:41 - INFO - | Pass 3: Checking directory > > connectivity > > 2024-06-05 04:00:41 - INFO - | Pass 3A: Optimizing directories > > 2024-06-05 04:00:41 - INFO - | Pass 4: Checking reference counts > > 2024-06-05 04:00:41 - INFO - | Pass 5: Checking group summary > > information > > 2024-06-05 04:00:41 - INFO - | > > 2024-06-05 04:00:41 - INFO - | /home/builder/edge-debug-machine- > > std/deploy/debug-machine-std.ext4: ***** FILE SYSTEM WAS MODIFIED ***** > > 2024-06-05 04:00:41 - INFO - | /home/builder/debug-machine- > > std/deploy/debug-machine-std.ext4: 25039/77600 files (0.1% non- > > contiguous), 292861/310076 blocks > > 2024-06-05 04:00:41 - INFO - | E: 10mount: umount: > > /run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd-11546112ec7c- > > 982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3: target is busy. > > 2024-06-05 04:00:41 - INFO - | E: 10mount: rmdir: failed to remove > > '/var/run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd- > > 11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3': Device or > > resource busy > > 2024-06-05 04:00:41 - INFO - | E: isar-builder-e7f8d532-aa0e-41cf- > > 85dd-11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3: Chroot > > setup failed: stage=setup-stop > > 2024-06-05 04:00:41 - INFO - NOTE: recipe debug-1.0-r0: task > > do_image_ext4: Failed > > 2024-06-05 04:00:41 - ERROR - ERROR: Task (/build/../repo/recipes- > > core/images/debug.bb:do_image_ext4) failed with exit code '1' > > 2024-06-05 04:00:42 - INFO - NOTE: Tasks Summary: Attempted 896 > > tasks of which 0 didn't need to be rerun and 1 failed. > > 2024-06-05 04:00:43 - INFO - > > 2024-06-05 04:00:43 - INFO - Summary: 1 task failed: > > 2024-06-05 04:00:43 - INFO - /build/../repo/recipes- > > core/images/debug.bb:do_image_ext4 > > 2024-06-05 04:00:43 - INFO - Summary: There was 1 WARNING message. > > 2024-06-05 04:00:43 - INFO - Summary: There was 1 ERROR message, > > returning a non-zero exit code. > > 2024-06-05 04:00:43 - ERROR - Command "/build$ > > /work/isar/bitbake/bin/bitbake -c build debug" failed > > --- Error summary --- > > ERROR: debug-1.0-r0 do_image_ext4: > > ExecutionError('/build/tmp/work/arm64/debug-machine-std/1.0- > > r0/temp/run.do_image_ext4.982511', 1, None, None) > > ERROR: Logfile of failure stored in: /build/tmp/work/arm64/debug- > > machine-std/1.0-r0/temp/log.do_image_ext4.982511 > > ERROR: Task (/build/../repo/recipes-core/images/debug.bb:do_image_ext4) > > failed with exit code '1' > > > > Cedric, weren't you seeing something very similar? > > Jan > > -- > Siemens AG, Technology > Linux Expert Center > > -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/d2508b62-d2fb-44a7-936f-aa2367347662n%40googlegroups.com. [-- Attachment #1.2: Type: text/html, Size: 6409 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: schroot umount target is busy 2024-07-08 14:17 ` Gökhan Çetin @ 2024-07-08 15:13 ` 'Jan Kiszka' via isar-users 2024-07-09 6:28 ` Gökhan Çetin 2024-07-09 6:52 ` 'cedric.hombourger@siemens.com' via isar-users 0 siblings, 2 replies; 11+ messages in thread From: 'Jan Kiszka' via isar-users @ 2024-07-08 15:13 UTC (permalink / raw) To: Gökhan Çetin, isar-users, Cedric Hombourger, Schultschik, Sven Angelo (PD PA CI R&D 4) On 08.07.24 16:17, Gökhan Çetin wrote: > Hi everyone, > > Similarly, we have been observing do_image tasks fail sporadically in > CI/CD pipelines for a while. > You can see a few examples below: > > INFO - | E: 10mount: umount: > /run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b: target is busy. > INFO - | E: 10mount: rmdir: failed to remove > '/var/run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b': Device or resource busy > INFO - | E: > isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b: Chroot setup failed: stage=setup-stop > INFO - NOTE: recipe .......: task do_image_swu: Failed > ERROR - ERROR: Task (/builds/.......bb:do_image_swu) failed with exit > code '1' > > INFO - | E: 10mount: umount: > /run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82: target is busy. > INFO - | E: 10mount: rmdir: failed to remove > '/var/run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82': Device or resource busy > INFO - | E: > isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82: Chroot setup failed: stage=setup-stop > INFO - NOTE: recipe ......: task do_image_ova: Failed > ERROR - ERROR: Task (/builds/.....bb:do_image_ova) failed with exit code '1' > > Do you have any information regarding the root cause? Trying once more to pull Cedric in (I think I used the wrong address last time): Weren't you trying to resolve that with a lazy umount? Or was it for a different corner of isar? But even if we lazily umount, this won't help when the next command is trying to rmdir the mount point... Does anyone have a feeling when this started? Does it correlate with any major changes in isar? Jan > > Thanks, > Gokhan > > On Wednesday, June 5, 2024 at 11:55:34 AM UTC+3 Jan Kiszka wrote: > > On 05.06.24 10:47, 'Schultschik, Sven' via isar-users wrote: > > Hi all, > > > > Our ISAR setup is running smooth on many machines different sizes. > But > > on our CI/CD pipeline it throws from time to time an error. > Starting it > > over again it works. > > > > Does someone knows this kind of error and where I have to look or how > > to fix? > > > > 2024-06-05 04:00:41 - INFO - | Processing triggers for man-db > > (2.11.2-2) ... > > 2024-06-05 04:00:41 - INFO - | e2fsck 1.47.0 (5-Feb-2023) > > 2024-06-05 04:00:41 - INFO - | Pass 1: Checking inodes, blocks, and > > sizes > > 2024-06-05 04:00:41 - INFO - | Pass 2: Checking directory structure > > 2024-06-05 04:00:41 - INFO - | Pass 3: Checking directory > > connectivity > > 2024-06-05 04:00:41 - INFO - | Pass 3A: Optimizing directories > > 2024-06-05 04:00:41 - INFO - | Pass 4: Checking reference counts > > 2024-06-05 04:00:41 - INFO - | Pass 5: Checking group summary > > information > > 2024-06-05 04:00:41 - INFO - | > > 2024-06-05 04:00:41 - INFO - | /home/builder/edge-debug-machine- > > std/deploy/debug-machine-std.ext4: ***** FILE SYSTEM WAS MODIFIED > ***** > > 2024-06-05 04:00:41 - INFO - | /home/builder/debug-machine- > > std/deploy/debug-machine-std.ext4: 25039/77600 files (0.1% non- > > contiguous), 292861/310076 blocks > > 2024-06-05 04:00:41 - INFO - | E: 10mount: umount: > > /run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd-11546112ec7c- > > 982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3: target is busy. > > 2024-06-05 04:00:41 - INFO - | E: 10mount: rmdir: failed to remove > > '/var/run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd- > > 11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3': Device or > > resource busy > > 2024-06-05 04:00:41 - INFO - | E: isar-builder-e7f8d532-aa0e-41cf- > > 85dd-11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3: Chroot > > setup failed: stage=setup-stop > > 2024-06-05 04:00:41 - INFO - NOTE: recipe debug-1.0-r0: task > > do_image_ext4: Failed > > 2024-06-05 04:00:41 - ERROR - ERROR: Task (/build/../repo/recipes- > > core/images/debug.bb:do_image_ext4) failed with exit code '1' > > 2024-06-05 04:00:42 - INFO - NOTE: Tasks Summary: Attempted 896 > > tasks of which 0 didn't need to be rerun and 1 failed. > > 2024-06-05 04:00:43 - INFO - > > 2024-06-05 04:00:43 - INFO - Summary: 1 task failed: > > 2024-06-05 04:00:43 - INFO - /build/../repo/recipes- > > core/images/debug.bb:do_image_ext4 > > 2024-06-05 04:00:43 - INFO - Summary: There was 1 WARNING message. > > 2024-06-05 04:00:43 - INFO - Summary: There was 1 ERROR message, > > returning a non-zero exit code. > > 2024-06-05 04:00:43 - ERROR - Command "/build$ > > /work/isar/bitbake/bin/bitbake -c build debug" failed > > --- Error summary --- > > ERROR: debug-1.0-r0 do_image_ext4: > > ExecutionError('/build/tmp/work/arm64/debug-machine-std/1.0- > > r0/temp/run.do_image_ext4.982511', 1, None, None) > > ERROR: Logfile of failure stored in: /build/tmp/work/arm64/debug- > > machine-std/1.0-r0/temp/log.do_image_ext4.982511 > > ERROR: Task > (/build/../repo/recipes-core/images/debug.bb:do_image_ext4) > > failed with exit code '1' > > > > Cedric, weren't you seeing something very similar? > > Jan > > -- > Siemens AG, Technology > Linux Expert Center > > -- > You received this message because you are subscribed to the Google > Groups "isar-users" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to isar-users+unsubscribe@googlegroups.com > <mailto:isar-users+unsubscribe@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/isar-users/d2508b62-d2fb-44a7-936f-aa2367347662n%40googlegroups.com <https://groups.google.com/d/msgid/isar-users/d2508b62-d2fb-44a7-936f-aa2367347662n%40googlegroups.com?utm_medium=email&utm_source=footer>. -- Siemens AG, Technology Linux Expert Center -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/940834c5-ccfd-4a3a-b1fa-e8c49fba7a11%40siemens.com. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: schroot umount target is busy 2024-07-08 15:13 ` 'Jan Kiszka' via isar-users @ 2024-07-09 6:28 ` Gökhan Çetin 2024-07-09 6:48 ` Anton Mikanovich 2024-07-12 7:04 ` Anton Mikanovich 2024-07-09 6:52 ` 'cedric.hombourger@siemens.com' via isar-users 1 sibling, 2 replies; 11+ messages in thread From: Gökhan Çetin @ 2024-07-09 6:28 UTC (permalink / raw) To: isar-users [-- Attachment #1.1: Type: text/plain, Size: 7147 bytes --] Hi Jan, Thanks for pointing that out, I was not aware the fix on lazy umounts commited 2 weeks ago: https://github.com/ilbers/isar/commit/ed2f8fda7cc85309c4f9a1f49a3f4b96df9886bb Don't know if the issue on Sven's side has been resolved by this. Just trying out, will let you know if it will help us. Best, Gokhan On Monday, July 8, 2024 at 6:13:54 PM UTC+3 Jan Kiszka wrote: > On 08.07.24 16:17, Gökhan Çetin wrote: > > Hi everyone, > > > > Similarly, we have been observing do_image tasks fail sporadically in > > CI/CD pipelines for a while. > > You can see a few examples below: > > > > INFO - | E: 10mount: umount: > > > /run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b: > target is busy. > > INFO - | E: 10mount: rmdir: failed to remove > > > '/var/run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b': > Device or resource busy > > INFO - | E: > > > isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b: > Chroot setup failed: stage=setup-stop > > INFO - NOTE: recipe .......: task do_image_swu: Failed > > ERROR - ERROR: Task (/builds/.......bb:do_image_swu) failed with exit > > code '1' > > > > INFO - | E: 10mount: umount: > > > /run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82: > target is busy. > > INFO - | E: 10mount: rmdir: failed to remove > > > '/var/run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82': > Device or resource busy > > INFO - | E: > > > isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82: > Chroot setup failed: stage=setup-stop > > INFO - NOTE: recipe ......: task do_image_ova: Failed > > ERROR - ERROR: Task (/builds/.....bb:do_image_ova) failed with exit code > '1' > > > > Do you have any information regarding the root cause? > > Trying once more to pull Cedric in (I think I used the wrong address > last time): Weren't you trying to resolve that with a lazy umount? Or > was it for a different corner of isar? > > But even if we lazily umount, this won't help when the next command is > trying to rmdir the mount point... > > Does anyone have a feeling when this started? Does it correlate with any > major changes in isar? > > Jan > > > > > Thanks, > > Gokhan > > > > On Wednesday, June 5, 2024 at 11:55:34 AM UTC+3 Jan Kiszka wrote: > > > > On 05.06.24 10:47, 'Schultschik, Sven' via isar-users wrote: > > > Hi all, > > > > > > Our ISAR setup is running smooth on many machines different sizes. > > But > > > on our CI/CD pipeline it throws from time to time an error. > > Starting it > > > over again it works. > > > > > > Does someone knows this kind of error and where I have to look or how > > > to fix? > > > > > > 2024-06-05 04:00:41 - INFO - | Processing triggers for man-db > > > (2.11.2-2) ... > > > 2024-06-05 04:00:41 - INFO - | e2fsck 1.47.0 (5-Feb-2023) > > > 2024-06-05 04:00:41 - INFO - | Pass 1: Checking inodes, blocks, and > > > sizes > > > 2024-06-05 04:00:41 - INFO - | Pass 2: Checking directory structure > > > 2024-06-05 04:00:41 - INFO - | Pass 3: Checking directory > > > connectivity > > > 2024-06-05 04:00:41 - INFO - | Pass 3A: Optimizing directories > > > 2024-06-05 04:00:41 - INFO - | Pass 4: Checking reference counts > > > 2024-06-05 04:00:41 - INFO - | Pass 5: Checking group summary > > > information > > > 2024-06-05 04:00:41 - INFO - | > > > 2024-06-05 04:00:41 - INFO - | /home/builder/edge-debug-machine- > > > std/deploy/debug-machine-std.ext4: ***** FILE SYSTEM WAS MODIFIED > > ***** > > > 2024-06-05 04:00:41 - INFO - | /home/builder/debug-machine- > > > std/deploy/debug-machine-std.ext4: 25039/77600 files (0.1% non- > > > contiguous), 292861/310076 blocks > > > 2024-06-05 04:00:41 - INFO - | E: 10mount: umount: > > > /run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd-11546112ec7c- > > > 982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3: target is busy. > > > 2024-06-05 04:00:41 - INFO - | E: 10mount: rmdir: failed to remove > > > '/var/run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd- > > > 11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3': Device or > > > resource busy > > > 2024-06-05 04:00:41 - INFO - | E: isar-builder-e7f8d532-aa0e-41cf- > > > 85dd-11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3: Chroot > > > setup failed: stage=setup-stop > > > 2024-06-05 04:00:41 - INFO - NOTE: recipe debug-1.0-r0: task > > > do_image_ext4: Failed > > > 2024-06-05 04:00:41 - ERROR - ERROR: Task (/build/../repo/recipes- > > > core/images/debug.bb:do_image_ext4) failed with exit code '1' > > > 2024-06-05 04:00:42 - INFO - NOTE: Tasks Summary: Attempted 896 > > > tasks of which 0 didn't need to be rerun and 1 failed. > > > 2024-06-05 04:00:43 - INFO - > > > 2024-06-05 04:00:43 - INFO - Summary: 1 task failed: > > > 2024-06-05 04:00:43 - INFO - /build/../repo/recipes- > > > core/images/debug.bb:do_image_ext4 > > > 2024-06-05 04:00:43 - INFO - Summary: There was 1 WARNING message. > > > 2024-06-05 04:00:43 - INFO - Summary: There was 1 ERROR message, > > > returning a non-zero exit code. > > > 2024-06-05 04:00:43 - ERROR - Command "/build$ > > > /work/isar/bitbake/bin/bitbake -c build debug" failed > > > --- Error summary --- > > > ERROR: debug-1.0-r0 do_image_ext4: > > > ExecutionError('/build/tmp/work/arm64/debug-machine-std/1.0- > > > r0/temp/run.do_image_ext4.982511', 1, None, None) > > > ERROR: Logfile of failure stored in: /build/tmp/work/arm64/debug- > > > machine-std/1.0-r0/temp/log.do_image_ext4.982511 > > > ERROR: Task > > (/build/../repo/recipes-core/images/debug.bb:do_image_ext4) > > > failed with exit code '1' > > > > > > > Cedric, weren't you seeing something very similar? > > > > Jan > > > > -- > > Siemens AG, Technology > > Linux Expert Center > > > > -- > > You received this message because you are subscribed to the Google > > Groups "isar-users" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to isar-users+...@googlegroups.com > > <mailto:isar-users+...@googlegroups.com>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/isar-users/d2508b62-d2fb-44a7-936f-aa2367347662n%40googlegroups.com > < > https://groups.google.com/d/msgid/isar-users/d2508b62-d2fb-44a7-936f-aa2367347662n%40googlegroups.com?utm_medium=email&utm_source=footer > >. > > -- > Siemens AG, Technology > Linux Expert Center > > -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/f4495e54-a4f3-417d-8b4d-bd38b723a232n%40googlegroups.com. [-- Attachment #1.2: Type: text/html, Size: 9778 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: schroot umount target is busy 2024-07-09 6:28 ` Gökhan Çetin @ 2024-07-09 6:48 ` Anton Mikanovich 2024-07-12 7:04 ` Anton Mikanovich 1 sibling, 0 replies; 11+ messages in thread From: Anton Mikanovich @ 2024-07-09 6:48 UTC (permalink / raw) To: Gökhan Çetin, isar-users, Kiszka, Jan, Cedric Hombourger, Schultschik, Sven Angelo (PD PA CI R&D 4), Baurzhan Ismagulov 09/07/2024 09:28, Gökhan Çetin wrote: > Hi Jan, > > Thanks for pointing that out, I was not aware the fix on lazy umounts > commited 2 weeks > ago: https://github.com/ilbers/isar/commit/ed2f8fda7cc85309c4f9a1f49a3f4b96df9886bb > > Don't know if the issue on Sven's side has been resolved by this. Just > trying out, will let you know if it will help us. > > Best, > Gokhan Hello Gökhan, The issue you've observed is probably caused by some mounts still left mounted during schroot session stop phase (imager_run API). Lazy umount fix will not resolve those mounts, but makes them visible for debug. Schroot mounts are controlled by fstab, which is filled from SCHROOT_MOUNTS value. If there are some mounts performing manually during imager_run call, it should be also unmounted in the same call. -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/32cb6d50-4afd-44f3-8668-0c1ef7836d94%40ilbers.de. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: schroot umount target is busy 2024-07-09 6:28 ` Gökhan Çetin 2024-07-09 6:48 ` Anton Mikanovich @ 2024-07-12 7:04 ` Anton Mikanovich 2024-08-06 8:02 ` Anton Mikanovich 1 sibling, 1 reply; 11+ messages in thread From: Anton Mikanovich @ 2024-07-12 7:04 UTC (permalink / raw) To: Gökhan Çetin, isar-users, Kiszka, Jan, cedric.hombourger, Baurzhan Ismagulov Hello Gökhan, I've found an issue with isar-apt usage without lock protection in deb_dl_dir_export and imager_run implementations. This can influence on umount issues you've observed, so please try the fix from patchset: [PATCH 0/2] Protect isar-apt usages in shell -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/8a611b46-370e-46f4-8ebc-725f6dbfbef5%40ilbers.de. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: schroot umount target is busy 2024-07-12 7:04 ` Anton Mikanovich @ 2024-08-06 8:02 ` Anton Mikanovich 2024-08-06 8:19 ` Gökhan Çetin 0 siblings, 1 reply; 11+ messages in thread From: Anton Mikanovich @ 2024-08-06 8:02 UTC (permalink / raw) To: Gökhan Çetin, isar-users, Kiszka, Jan, cedric.hombourger, Baurzhan Ismagulov 12/07/2024 10:04, Anton Mikanovich wrote: > Hello Gökhan, > > I've found an issue with isar-apt usage without lock protection in > deb_dl_dir_export and imager_run implementations. This can influence > on umount > issues you've observed, so please try the fix from patchset: > > [PATCH 0/2] Protect isar-apt usages in shell Unfortunately this fix didn't help. Even with it we've got the same issue (it looks like there is the first time we've catched it in our CI): ERROR: Task (mc:qemuarm-bullseye:/workspace/build-auto/isar_next/1324/meta-test/recipes-core/images/isar-image-ci.bb:do_image_ext4) failed with exit code '1' E: 10mount: umount: /run/schroot/mount/isar-jenkins-0c03d5d2-da53-444c-93f6-fc9bdfafc290-2097734-1868b26c-f678-4e99-9e9a-360c7c05fcd3: target is busy. E: 10mount: rmdir: failed to remove '/var/run/schroot/mount/isar-jenkins-0c03d5d2-da53-444c-93f6-fc9bdfafc290-2097734-1868b26c-f678-4e99-9e9a-360c7c05fcd3': Device or resource busy E: isar-jenkins-0c03d5d2-da53-444c-93f6-fc9bdfafc290-2097734-1868b26c-f678-4e99-9e9a-360c7c05fcd3: Chroot setup failed: stage=setup-stop -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/a53a8580-854b-481c-a788-16fdab357e88%40ilbers.de. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: schroot umount target is busy 2024-08-06 8:02 ` Anton Mikanovich @ 2024-08-06 8:19 ` Gökhan Çetin 0 siblings, 0 replies; 11+ messages in thread From: Gökhan Çetin @ 2024-08-06 8:19 UTC (permalink / raw) To: isar-users [-- Attachment #1.1: Type: text/plain, Size: 2082 bytes --] Hi Anton, I haven't had a chance to try the patch you mentioned yet. But this fix <https://github.com/ilbers/isar/commit/ed2f8fda7cc85309c4f9a1f49a3f4b96df9886bb> has greatly reduced the error frequency on our side. As Cedric mentioned, we also have managed servers and several IT services that might cause such issues. That's why, I can say that we have achieved acceptable stability for now. Thanks, Gokhan On Tuesday, August 6, 2024 at 11:02:55 AM UTC+3 Anton Mikanovich wrote: > 12/07/2024 10:04, Anton Mikanovich wrote: > > Hello Gökhan, > > > > I've found an issue with isar-apt usage without lock protection in > > deb_dl_dir_export and imager_run implementations. This can influence > > on umount > > issues you've observed, so please try the fix from patchset: > > > > [PATCH 0/2] Protect isar-apt usages in shell > > Unfortunately this fix didn't help. > Even with it we've got the same issue (it looks like there is the first > time > we've catched it in our CI): > > ERROR: Task > (mc:qemuarm-bullseye:/workspace/build-auto/isar_next/1324/meta-test/recipes-core/images/isar-image-ci.bb:do_image_ext4) > > failed with exit code '1' > E: 10mount: umount: > /run/schroot/mount/isar-jenkins-0c03d5d2-da53-444c-93f6-fc9bdfafc290-2097734-1868b26c-f678-4e99-9e9a-360c7c05fcd3: > > target is busy. > E: 10mount: rmdir: failed to remove > '/var/run/schroot/mount/isar-jenkins-0c03d5d2-da53-444c-93f6-fc9bdfafc290-2097734-1868b26c-f678-4e99-9e9a-360c7c05fcd3': > > Device or resource busy > E: > isar-jenkins-0c03d5d2-da53-444c-93f6-fc9bdfafc290-2097734-1868b26c-f678-4e99-9e9a-360c7c05fcd3: > > Chroot setup failed: stage=setup-stop > > -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/f9dac9c3-e29e-4093-a3c0-18f01e5317c2n%40googlegroups.com. [-- Attachment #1.2: Type: text/html, Size: 2762 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: schroot umount target is busy 2024-07-08 15:13 ` 'Jan Kiszka' via isar-users 2024-07-09 6:28 ` Gökhan Çetin @ 2024-07-09 6:52 ` 'cedric.hombourger@siemens.com' via isar-users 2024-07-09 10:27 ` 'Jan Kiszka' via isar-users 1 sibling, 1 reply; 11+ messages in thread From: 'cedric.hombourger@siemens.com' via isar-users @ 2024-07-09 6:52 UTC (permalink / raw) To: Schultschik, Sven, isar-users, Kiszka, Jan, ctngkhn On Mon, 2024-07-08 at 17:13 +0200, Jan Kiszka wrote: > On 08.07.24 16:17, Gökhan Çetin wrote: > > Hi everyone, > > > > Similarly, we have been observing do_image tasks fail sporadically > > in > > CI/CD pipelines for a while. > > You can see a few examples below: > > > > INFO - | E: 10mount: umount: > > /run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570- > > dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b: target is > > busy. > > INFO - | E: 10mount: rmdir: failed to remove > > '/var/run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570- > > dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b': Device > > or resource busy > > INFO - | E: > > isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba- > > b359-48ad-91f4-2962f4bdf32b: Chroot setup failed: stage=setup-stop > > INFO - NOTE: recipe .......: task do_image_swu: Failed > > ERROR - ERROR: Task (/builds/.......bb:do_image_swu) failed with > > exit > > code '1' > > > > INFO - | E: 10mount: umount: > > /run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade- > > 57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82: target is > > busy. > > INFO - | E: 10mount: rmdir: failed to remove > > '/var/run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade- > > 57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82': Device > > or resource busy > > INFO - | E: > > isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b- > > 8c19-4fc8-9b5e-d2ccb3d67d82: Chroot setup failed: stage=setup-stop > > INFO - NOTE: recipe ......: task do_image_ova: Failed > > ERROR - ERROR: Task (/builds/.....bb:do_image_ova) failed with exit > > code '1' > > > > Do you have any information regarding the root cause? > > Trying once more to pull Cedric in (I think I used the wrong address > last time): Weren't you trying to resolve that with a lazy umount? Or > was it for a different corner of isar? Correct. On our side, this issue is seen sporadically on servers managed by our IT where they have some proprietary / black-box services scanning the file-system when new mounts are discovered... Have not found an elegant way to fix or work-around this (and obviously no luck in getting IT to remove their software or configure it differently) > > But even if we lazily umount, this won't help when the next command > is > trying to rmdir the mount point... > > Does anyone have a feeling when this started? Does it correlate with > any > major changes in isar? > > Jan > > > > > Thanks, > > Gokhan > > > > On Wednesday, June 5, 2024 at 11:55:34 AM UTC+3 Jan Kiszka wrote: > > > > On 05.06.24 10:47, 'Schultschik, Sven' via isar-users wrote: > > > Hi all, > > > > > > Our ISAR setup is running smooth on many machines different > > sizes. > > But > > > on our CI/CD pipeline it throws from time to time an error. > > Starting it > > > over again it works. > > > > > > Does someone knows this kind of error and where I have to > > look or how > > > to fix? > > > > > > 2024-06-05 04:00:41 - INFO - | Processing triggers for man-db > > > (2.11.2-2) ... > > > 2024-06-05 04:00:41 - INFO - | e2fsck 1.47.0 (5-Feb-2023) > > > 2024-06-05 04:00:41 - INFO - | Pass 1: Checking inodes, > > blocks, and > > > sizes > > > 2024-06-05 04:00:41 - INFO - | Pass 2: Checking directory > > structure > > > 2024-06-05 04:00:41 - INFO - | Pass 3: Checking directory > > > connectivity > > > 2024-06-05 04:00:41 - INFO - | Pass 3A: Optimizing > > directories > > > 2024-06-05 04:00:41 - INFO - | Pass 4: Checking reference > > counts > > > 2024-06-05 04:00:41 - INFO - | Pass 5: Checking group summary > > > information > > > 2024-06-05 04:00:41 - INFO - | > > > 2024-06-05 04:00:41 - INFO - | /home/builder/edge-debug- > > machine- > > > std/deploy/debug-machine-std.ext4: ***** FILE SYSTEM WAS > > MODIFIED > > ***** > > > 2024-06-05 04:00:41 - INFO - | /home/builder/debug-machine- > > > std/deploy/debug-machine-std.ext4: 25039/77600 files (0.1% > > non- > > > contiguous), 292861/310076 blocks > > > 2024-06-05 04:00:41 - INFO - | E: 10mount: umount: > > > /run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd- > > 11546112ec7c- > > > 982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3: target is busy. > > > 2024-06-05 04:00:41 - INFO - | E: 10mount: rmdir: failed to > > remove > > > '/var/run/schroot/mount/isar-builder-e7f8d532-aa0e-41cf-85dd- > > > 11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4-f9629d1a55f3': > > Device or > > > resource busy > > > 2024-06-05 04:00:41 - INFO - | E: isar-builder-e7f8d532-aa0e- > > 41cf- > > > 85dd-11546112ec7c-982511-2d2ada00-82dd-45bc-9bd4- > > f9629d1a55f3: Chroot > > > setup failed: stage=setup-stop > > > 2024-06-05 04:00:41 - INFO - NOTE: recipe debug-1.0-r0: task > > > do_image_ext4: Failed > > > 2024-06-05 04:00:41 - ERROR - ERROR: Task > > (/build/../repo/recipes- > > > core/images/debug.bb:do_image_ext4) failed with exit code '1' > > > 2024-06-05 04:00:42 - INFO - NOTE: Tasks Summary: Attempted > > 896 > > > tasks of which 0 didn't need to be rerun and 1 failed. > > > 2024-06-05 04:00:43 - INFO - > > > 2024-06-05 04:00:43 - INFO - Summary: 1 task failed: > > > 2024-06-05 04:00:43 - INFO - /build/../repo/recipes- > > > core/images/debug.bb:do_image_ext4 > > > 2024-06-05 04:00:43 - INFO - Summary: There was 1 WARNING > > message. > > > 2024-06-05 04:00:43 - INFO - Summary: There was 1 ERROR > > message, > > > returning a non-zero exit code. > > > 2024-06-05 04:00:43 - ERROR - Command "/build$ > > > /work/isar/bitbake/bin/bitbake -c build debug" failed > > > --- Error summary --- > > > ERROR: debug-1.0-r0 do_image_ext4: > > > ExecutionError('/build/tmp/work/arm64/debug-machine-std/1.0- > > > r0/temp/run.do_image_ext4.982511', 1, None, None) > > > ERROR: Logfile of failure stored in: > > /build/tmp/work/arm64/debug- > > > machine-std/1.0-r0/temp/log.do_image_ext4.982511 > > > ERROR: Task > > (/build/../repo/recipes-core/images/debug.bb:do_image_ext4) > > > failed with exit code '1' > > > > > > > Cedric, weren't you seeing something very similar? > > > > Jan > > > > -- > > Siemens AG, Technology > > Linux Expert Center > > > > -- > > You received this message because you are subscribed to the Google > > Groups "isar-users" group. > > To unsubscribe from this group and stop receiving emails from it, > > send > > an email to isar-users+unsubscribe@googlegroups.com > > <mailto:isar-users+unsubscribe@googlegroups.com>. > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/isar-users/d2508b62-d2fb-44a7-936f-aa2367347662n%40googlegroups.com > > < > > https://groups.google.com/d/msgid/isar-users/d2508b62-d2fb-44a7-936f-aa2367347662n%40googlegroups.com?utm_medium=email&utm_source=footer > > >. > -- Cedric Hombourger Siemens AG http://www.siemens.com/ -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/10335bb9aa9443e71402b883db9ab30c92af6355.camel%40siemens.com. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: schroot umount target is busy 2024-07-09 6:52 ` 'cedric.hombourger@siemens.com' via isar-users @ 2024-07-09 10:27 ` 'Jan Kiszka' via isar-users 0 siblings, 0 replies; 11+ messages in thread From: 'Jan Kiszka' via isar-users @ 2024-07-09 10:27 UTC (permalink / raw) To: Hombourger, Cedric (DI CTO FDS CES LX), Schultschik, Sven (DI PA DCP R&D 2), isar-users, ctngkhn On 09.07.24 08:52, Hombourger, Cedric (DI CTO FDS CES LX) wrote: > On Mon, 2024-07-08 at 17:13 +0200, Jan Kiszka wrote: >> On 08.07.24 16:17, Gökhan Çetin wrote: >>> Hi everyone, >>> >>> Similarly, we have been observing do_image tasks fail sporadically >>> in >>> CI/CD pipelines for a while. >>> You can see a few examples below: >>> >>> INFO - | E: 10mount: umount: >>> /run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570- >>> dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b: target is >>> busy. >>> INFO - | E: 10mount: rmdir: failed to remove >>> '/var/run/schroot/mount/isar-builder-31888f6e-6228-4bc8-9570- >>> dd4d69ad25ba-367048-331187ba-b359-48ad-91f4-2962f4bdf32b': Device >>> or resource busy >>> INFO - | E: >>> isar-builder-31888f6e-6228-4bc8-9570-dd4d69ad25ba-367048-331187ba- >>> b359-48ad-91f4-2962f4bdf32b: Chroot setup failed: stage=setup-stop >>> INFO - NOTE: recipe .......: task do_image_swu: Failed >>> ERROR - ERROR: Task (/builds/.......bb:do_image_swu) failed with >>> exit >>> code '1' >>> >>> INFO - | E: 10mount: umount: >>> /run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade- >>> 57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82: target is >>> busy. >>> INFO - | E: 10mount: rmdir: failed to remove >>> '/var/run/schroot/mount/isar-builder-5f43627d-fad6-488f-bade- >>> 57d0635ee0c6-408837-d15a1d4b-8c19-4fc8-9b5e-d2ccb3d67d82': Device >>> or resource busy >>> INFO - | E: >>> isar-builder-5f43627d-fad6-488f-bade-57d0635ee0c6-408837-d15a1d4b- >>> 8c19-4fc8-9b5e-d2ccb3d67d82: Chroot setup failed: stage=setup-stop >>> INFO - NOTE: recipe ......: task do_image_ova: Failed >>> ERROR - ERROR: Task (/builds/.....bb:do_image_ova) failed with exit >>> code '1' >>> >>> Do you have any information regarding the root cause? >> >> Trying once more to pull Cedric in (I think I used the wrong address >> last time): Weren't you trying to resolve that with a lazy umount? Or >> was it for a different corner of isar? > > Correct. On our side, this issue is seen sporadically on servers > managed by our IT where they have some proprietary / black-box services > scanning the file-system when new mounts are discovered... > > Have not found an elegant way to fix or work-around this (and obviously > no luck in getting IT to remove their software or configure it > differently) Hehe, this is the point where moving to a VM will both be more secure and faster. Seriously, all we could do for such a scenario is adding some finite retry loop. Messy. Jan -- Siemens AG, Technology Linux Expert Center -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/4e811a3d-fbfb-4030-ad02-f1cc0fa66799%40siemens.com. ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2024-08-06 8:19 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-06-05 8:47 schroot umount target is busy Schultschik, Sven 2024-06-05 8:55 ` Jan Kiszka 2024-07-08 14:17 ` Gökhan Çetin 2024-07-08 15:13 ` 'Jan Kiszka' via isar-users 2024-07-09 6:28 ` Gökhan Çetin 2024-07-09 6:48 ` Anton Mikanovich 2024-07-12 7:04 ` Anton Mikanovich 2024-08-06 8:02 ` Anton Mikanovich 2024-08-06 8:19 ` Gökhan Çetin 2024-07-09 6:52 ` 'cedric.hombourger@siemens.com' via isar-users 2024-07-09 10:27 ` 'Jan Kiszka' via isar-users
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox