Re: [PATCH 03/17] meta-isar: deploy-image: Change reboot logic

["'Jan Kiszka' via isar-users" <isar-users@googlegroups.com>]

On 13.03.25 14:10, Uladzimir Bely wrote:
> On Tue, 2024-07-02 at 15:38 +0200, 'Jan Kiszka' via isar-users wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> Pull the reboot out of the script. This allows for cleaner
>> integration
>> with different calling environment, may they be a systemd unit, an
>> initramfs script or simply a shell for testing purposes.
>>
>> And if the script exits with an error, wait a minute before rebooting
>> the system, rather than just trying to re-execute it. This permits to
>> inspect potential error as well.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>  .../recipes-installer/deploy-image/files/deploy-image-wic.sh    | 2
>> +-
>>  .../recipes-installer/deploy-image/files/install.override.conf  | 2
>> +-
>>  2 files changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta-isar/recipes-installer/deploy-image/files/deploy-
>> image-wic.sh b/meta-isar/recipes-installer/deploy-image/files/deploy-
>> image-wic.sh
>> index 8043aff1..12c1eea2 100644
>> --- a/meta-isar/recipes-installer/deploy-image/files/deploy-image-
>> wic.sh
>> +++ b/meta-isar/recipes-installer/deploy-image/files/deploy-image-
>> wic.sh
>> @@ -105,4 +105,4 @@ fi
>>  umount "$installdata"
>>  sync
>>  dialog --title "Reboot" --msgbox "Installation is successful. System
>> will be rebooted. Please remove the USB stick." 7 60
>> -reboot
>> +exit 0
>> diff --git a/meta-isar/recipes-installer/deploy-
>> image/files/install.override.conf b/meta-isar/recipes-
>> installer/deploy-image/files/install.override.conf
>> index 73874caa..357d8662 100644
>> --- a/meta-isar/recipes-installer/deploy-
>> image/files/install.override.conf
>> +++ b/meta-isar/recipes-installer/deploy-
>> image/files/install.override.conf
>> @@ -1,5 +1,5 @@
>>  [Service]
>>  ExecStart=
>> -ExecStart=/usr/bin/deploy-image-wic.sh
>> +ExecStart=/bin/sh -c "deploy-image-wic.sh || (echo 'Rebooting in 60
>> s'; sleep 60); reboot"
> 
> Hello Jan.
> 
> This change seems to cause a sort of race issue.
> 
> Few days ago "[PATCH v3 0/2] Cover installer image with tests" was
> merged. Tests were OK (and "next" is still OK) on two jenkins and one
> gitlab instances. But on the third jenkins instance, they were failing.
> 
> Boot log:
> 
> ```
> Got config:
>   installer_unattended=true
>   installer_image_uri=/install/isar-image-ci-debian-bookworm-
> qemuamd64.wic.zst
>   installer_target_dev=/dev/sda
>   installer_target_overwrite=OVERWRITE
> bmaptool: ERROR: An error occurred, here is the traceback:
> Traceback (most recent call last):
>   File "/usr/lib/python3/dist-packages/bmaptools/CLI.py", line 116, in
> open_block_device
>     descriptor = os.open(path, os.O_WRONLY | os.O_EXCL)
> 
> bmaptool: ERROR: cannot open block device '/dev/sda' in exclusive mode:
> [Errno 16] Device or resource busy: '/dev/sda'
> 
> Rebooting in 60 s
> ```
> 
> But either machine is not booted, or there are some "ext4" errors
> happen.
> 
> I couldn't understand few things:
>  - why this happens (unable to get exclusive access to block device)$
>  - why device is written even if there is such error in logs$
>  - why sometimes we don't see this error in logs$
>  - why this doesn't happen when "unattended mode" is off.
> 
> The reason is that we have these two "getty" instances overriden by
> custom one having the race. While "serial-getty@ttyS0.service.d" does
> the job, "getty@tty1.service.d" produces this error visible in logs and
> reboot machine in 60 seconds.
> 
> In some cases 60sec is enough for first service to finish writing to
> the disc, but on slower machines we result in incomplete copy and
> broken system.
> 
> Before this patch, when "reboot" calls were in the script, "failed"
> instance simply exited and didn't try to reboot the board.
> 
> As a quick fix, I would reverted this patch.

That won't be a fix, just papering over.

The race is apparently around the target device becoming ready for
bmaptool, or our script racing with some other service accessing the
device for a while. I suspect the service is lacking some dependency
here. So, let's identify the race partner first: How to reproduce locally?

Jan

> 
> Also, I think this behaviour may impact on bmap-tools "psplash" patches
> currently we have on maillist. I would expect that user won't see the
> progress bar on screen because it's somewhere on serial console.
> 
> 
>>  StandardInput=tty
>>  StandardOutput=tty
>> -- 
>> 2.43.0
>>
> 

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/fc56d0d1-359e-43d9-ab37-897c78e5916e%40siemens.com.