Re: [PATCH 03/17] meta-isar: deploy-image: Change reboot logic

["'Heinisch, Alexander' via isar-users" <isar-users@googlegroups.com>]

On Fri, 2025-03-14 at 09:25 +0300, Uladzimir Bely wrote:
> On Fri, 2025-03-14 at 06:54 +0100, Jan Kiszka wrote:
> > On 13.03.25 14:10, Uladzimir Bely wrote:
> > > On Tue, 2024-07-02 at 15:38 +0200, 'Jan Kiszka' via isar-users
> > > wrote:
> > > > From: Jan Kiszka <jan.kiszka@siemens.com>
> > > > 
> > > > Pull the reboot out of the script. This allows for cleaner
> > > > integration
> > > > with different calling environment, may they be a systemd unit,
> > > > an
> > > > initramfs script or simply a shell for testing purposes.
> > > > 
> > > > And if the script exits with an error, wait a minute before
> > > > rebooting
> > > > the system, rather than just trying to re-execute it. This
> > > > permits to
> > > > inspect potential error as well.
> > > > 
> > > > Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> > > > ---
> > > >  .../recipes-installer/deploy-image/files/deploy-image-
> > > > wic.sh   
> > > > > 2
> > > > +-
> > > >  .../recipes-installer/deploy-
> > > > image/files/install.override.conf 
> > > > > 2
> > > > +-
> > > >  2 files changed, 2 insertions(+), 2 deletions(-)
> > > > 
> > > > diff --git a/meta-isar/recipes-installer/deploy-
> > > > image/files/deploy-
> > > > image-wic.sh b/meta-isar/recipes-installer/deploy-
> > > > image/files/deploy-
> > > > image-wic.sh
> > > > index 8043aff1..12c1eea2 100644
> > > > --- a/meta-isar/recipes-installer/deploy-image/files/deploy-
> > > > image-
> > > > wic.sh
> > > > +++ b/meta-isar/recipes-installer/deploy-image/files/deploy-
> > > > image-
> > > > wic.sh
> > > > @@ -105,4 +105,4 @@ fi
> > > >  umount "$installdata"
> > > >  sync
> > > >  dialog --title "Reboot" --msgbox "Installation is successful.
> > > > System
> > > > will be rebooted. Please remove the USB stick." 7 60
> > > > -reboot
> > > > +exit 0
> > > > diff --git a/meta-isar/recipes-installer/deploy-
> > > > image/files/install.override.conf b/meta-isar/recipes-
> > > > installer/deploy-image/files/install.override.conf
> > > > index 73874caa..357d8662 100644
> > > > --- a/meta-isar/recipes-installer/deploy-
> > > > image/files/install.override.conf
> > > > +++ b/meta-isar/recipes-installer/deploy-
> > > > image/files/install.override.conf
> > > > @@ -1,5 +1,5 @@
> > > >  [Service]
> > > >  ExecStart=
> > > > -ExecStart=/usr/bin/deploy-image-wic.sh
> > > > +ExecStart=/bin/sh -c "deploy-image-wic.sh || (echo 'Rebooting
> > > > in
> > > > 60
> > > > s'; sleep 60); reboot"
> > > 
> > > Hello Jan.
> > > 
> > > This change seems to cause a sort of race issue.
> > > 
> > > Few days ago "[PATCH v3 0/2] Cover installer image with tests"
> > > was
> > > merged. Tests were OK (and "next" is still OK) on two jenkins and
> > > one
> > > gitlab instances. But on the third jenkins instance, they were
> > > failing.
> > > 
> > > Boot log:
> > > 
> > > ```
> > > Got config:
> > >   installer_unattended=true
> > >   installer_image_uri=/install/isar-image-ci-debian-bookworm-
> > > qemuamd64.wic.zst
> > >   installer_target_dev=/dev/sda
> > >   installer_target_overwrite=OVERWRITE
> > > bmaptool: ERROR: An error occurred, here is the traceback:
> > > Traceback (most recent call last):
> > >   File "/usr/lib/python3/dist-packages/bmaptools/CLI.py", line
> > > 116,
> > > in
> > > open_block_device
> > >     descriptor = os.open(path, os.O_WRONLY | os.O_EXCL)
> > > 
> > > bmaptool: ERROR: cannot open block device '/dev/sda' in exclusive
> > > mode:
> > > [Errno 16] Device or resource busy: '/dev/sda'
> > > 
> > > Rebooting in 60 s
> > > ```
> > > 
> > > But either machine is not booted, or there are some "ext4" errors
> > > happen.
> > > 
> > > I couldn't understand few things:
> > >  - why this happens (unable to get exclusive access to block
> > > device)$
> > >  - why device is written even if there is such error in logs$
> > >  - why sometimes we don't see this error in logs$
> > >  - why this doesn't happen when "unattended mode" is off.


It does not happen when "unattended mode" is off, because the race is
interrupted by the script waiting for user input at the very beginning.

This is a race condition between multiple instances of the install
script executed in parallel, dependent on how many gettys are active.
Since the number of gettys and further, which getty get's connected in
the end, via terminal, via serial, ... is kind of hard to predict, imo
the only option to resolve that race is by adding a dedicated
(singleton) service which is executed on startup.

I am currently preparing a patch set where a fix for that is included,
similar to this:

```
[Unit]
Description=Target-bootstrapper service

[Service]
User=root
ExecStart=/bin/sh -c "/usr/bin/target-bootstrapper.sh || (echo
'Rebooting in 60 s'; sleep 60); reboot"

StandardOutput=journal+console
TTYPath=/dev/console

[Install]
WantedBy=multi-user.target
``

BR Alexander

> > > 
> > > The reason is that we have these two "getty" instances overriden
> > > by
> > > custom one having the race. While "serial-getty@ttyS0.service.d"
> > > does
> > > the job, "getty@tty1.service.d" produces this error visible in
> > > logs
> > > and
> > > reboot machine in 60 seconds.
> > > 
> > > In some cases 60sec is enough for first service to finish writing
> > > to
> > > the disc, but on slower machines we result in incomplete copy and
> > > broken system.
> > > 
> > > Before this patch, when "reboot" calls were in the script,
> > > "failed"
> > > instance simply exited and didn't try to reboot the board.
> > > 
> > > As a quick fix, I would reverted this patch.
> > 
> > That won't be a fix, just papering over.
> > 
> > The race is apparently around the target device becoming ready for
> > bmaptool, or our script racing with some other service accessing
> > the
> > device for a while. I suspect the service is lacking some
> > dependency
> > here. So, let's identify the race partner first: How to reproduce
> > locally?
> > 
> > Jan
> > 
> 
> No, my finding is that getty@tty1.service.d conflicts with
> serial-getty@ttyS0.service.d one. Every of them tries to run bmaptool
> when unattended mode is active.
> 
> meta-isar/recipes-installer/deploy-image-service/deploy-image-
> service.bb:
> 
> ```
> do_install() {
>   install -m 0600 ${WORKDIR}/install.override.conf
> ${D}/usr/lib/systemd/system/getty@tty1.service.d/override.conf
>   install -m 0600 ${WORKDIR}/install.override.conf
> ${D}/usr/lib/systemd/system/serial-getty@ttyS0.service.d/override.con
> f
> }
> ```
> 
> It can be easy reproduced if we decrease timeout from 60s to e.g.
> 10s,
> to simulate slower system:
> 
> ```
> isar $ sed -i -e 's/60/10/g' meta-isar/recipes-installer/deploy-
> image-
> service/files/install.override.conf 
> isar $ ./kas/kas-container shell kas/isar.yaml 
> builder@b21a175d3b78:/build$ rm -rf conf/
> builder@b21a175d3b78:/build$ /repo/scripts/ci_build.sh -T installer
> ```
> 
> Failed instance of overriden getty reboots the system while the
> target
> is being populated with the not failed instance.
> 
> Reboot hangs at something like
> 
> ```
> [    9.323801] JBD2: no valid journal superblock found
> [    9.325117] EXT4-fs (sda2): error loading journal
> ```
> 
> 
> > > 
> > > Also, I think this behaviour may impact on bmap-tools "psplash"
> > > patches
> > > currently we have on maillist. I would expect that user won't see
> > > the
> > > progress bar on screen because it's somewhere on serial console.
> > > 
> > > 
> > > >  StandardInput=tty
> > > >  StandardOutput=tty
> > > > -- 
> > > > 2.43.0
> > > > 
> > > 
> > 
> 
> -- 
> Best regards,
> Uladzimir.
> 
> 
> 

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/be6af9287630689ae711b50cc247ccf1d4fcd498.camel%40siemens.com.