Re: [PATCH 03/17] meta-isar: deploy-image: Change reboot logic

[Uladzimir Bely <ubely@ilbers.de>]

On Fri, 2025-03-14 at 06:54 +0100, Jan Kiszka wrote:
> On 13.03.25 14:10, Uladzimir Bely wrote:
> > On Tue, 2024-07-02 at 15:38 +0200, 'Jan Kiszka' via isar-users
> > wrote:
> > > From: Jan Kiszka <jan.kiszka@siemens.com>
> > > 
> > > Pull the reboot out of the script. This allows for cleaner
> > > integration
> > > with different calling environment, may they be a systemd unit,
> > > an
> > > initramfs script or simply a shell for testing purposes.
> > > 
> > > And if the script exits with an error, wait a minute before
> > > rebooting
> > > the system, rather than just trying to re-execute it. This
> > > permits to
> > > inspect potential error as well.
> > > 
> > > Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> > > ---
> > >  .../recipes-installer/deploy-image/files/deploy-image-wic.sh   
> > > | 2
> > > +-
> > >  .../recipes-installer/deploy-image/files/install.override.conf 
> > > | 2
> > > +-
> > >  2 files changed, 2 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/meta-isar/recipes-installer/deploy-
> > > image/files/deploy-
> > > image-wic.sh b/meta-isar/recipes-installer/deploy-
> > > image/files/deploy-
> > > image-wic.sh
> > > index 8043aff1..12c1eea2 100644
> > > --- a/meta-isar/recipes-installer/deploy-image/files/deploy-
> > > image-
> > > wic.sh
> > > +++ b/meta-isar/recipes-installer/deploy-image/files/deploy-
> > > image-
> > > wic.sh
> > > @@ -105,4 +105,4 @@ fi
> > >  umount "$installdata"
> > >  sync
> > >  dialog --title "Reboot" --msgbox "Installation is successful.
> > > System
> > > will be rebooted. Please remove the USB stick." 7 60
> > > -reboot
> > > +exit 0
> > > diff --git a/meta-isar/recipes-installer/deploy-
> > > image/files/install.override.conf b/meta-isar/recipes-
> > > installer/deploy-image/files/install.override.conf
> > > index 73874caa..357d8662 100644
> > > --- a/meta-isar/recipes-installer/deploy-
> > > image/files/install.override.conf
> > > +++ b/meta-isar/recipes-installer/deploy-
> > > image/files/install.override.conf
> > > @@ -1,5 +1,5 @@
> > >  [Service]
> > >  ExecStart=
> > > -ExecStart=/usr/bin/deploy-image-wic.sh
> > > +ExecStart=/bin/sh -c "deploy-image-wic.sh || (echo 'Rebooting in
> > > 60
> > > s'; sleep 60); reboot"
> > 
> > Hello Jan.
> > 
> > This change seems to cause a sort of race issue.
> > 
> > Few days ago "[PATCH v3 0/2] Cover installer image with tests" was
> > merged. Tests were OK (and "next" is still OK) on two jenkins and
> > one
> > gitlab instances. But on the third jenkins instance, they were
> > failing.
> > 
> > Boot log:
> > 
> > ```
> > Got config:
> >   installer_unattended=true
> >   installer_image_uri=/install/isar-image-ci-debian-bookworm-
> > qemuamd64.wic.zst
> >   installer_target_dev=/dev/sda
> >   installer_target_overwrite=OVERWRITE
> > bmaptool: ERROR: An error occurred, here is the traceback:
> > Traceback (most recent call last):
> >   File "/usr/lib/python3/dist-packages/bmaptools/CLI.py", line 116,
> > in
> > open_block_device
> >     descriptor = os.open(path, os.O_WRONLY | os.O_EXCL)
> > 
> > bmaptool: ERROR: cannot open block device '/dev/sda' in exclusive
> > mode:
> > [Errno 16] Device or resource busy: '/dev/sda'
> > 
> > Rebooting in 60 s
> > ```
> > 
> > But either machine is not booted, or there are some "ext4" errors
> > happen.
> > 
> > I couldn't understand few things:
> >  - why this happens (unable to get exclusive access to block
> > device)$
> >  - why device is written even if there is such error in logs$
> >  - why sometimes we don't see this error in logs$
> >  - why this doesn't happen when "unattended mode" is off.
> > 
> > The reason is that we have these two "getty" instances overriden by
> > custom one having the race. While "serial-getty@ttyS0.service.d"
> > does
> > the job, "getty@tty1.service.d" produces this error visible in logs
> > and
> > reboot machine in 60 seconds.
> > 
> > In some cases 60sec is enough for first service to finish writing
> > to
> > the disc, but on slower machines we result in incomplete copy and
> > broken system.
> > 
> > Before this patch, when "reboot" calls were in the script, "failed"
> > instance simply exited and didn't try to reboot the board.
> > 
> > As a quick fix, I would reverted this patch.
> 
> That won't be a fix, just papering over.
> 
> The race is apparently around the target device becoming ready for
> bmaptool, or our script racing with some other service accessing the
> device for a while. I suspect the service is lacking some dependency
> here. So, let's identify the race partner first: How to reproduce
> locally?
> 
> Jan
> 

No, my finding is that getty@tty1.service.d conflicts with
serial-getty@ttyS0.service.d one. Every of them tries to run bmaptool
when unattended mode is active.

meta-isar/recipes-installer/deploy-image-service/deploy-image-
service.bb:

```
do_install() {
  install -m 0600 ${WORKDIR}/install.override.conf
${D}/usr/lib/systemd/system/getty@tty1.service.d/override.conf
  install -m 0600 ${WORKDIR}/install.override.conf
${D}/usr/lib/systemd/system/serial-getty@ttyS0.service.d/override.conf
}
```

It can be easy reproduced if we decrease timeout from 60s to e.g. 10s,
to simulate slower system:

```
isar $ sed -i -e 's/60/10/g' meta-isar/recipes-installer/deploy-image-
service/files/install.override.conf 
isar $ ./kas/kas-container shell kas/isar.yaml 
builder@b21a175d3b78:/build$ rm -rf conf/
builder@b21a175d3b78:/build$ /repo/scripts/ci_build.sh -T installer
```

Failed instance of overriden getty reboots the system while the target
is being populated with the not failed instance.

Reboot hangs at something like

```
[    9.323801] JBD2: no valid journal superblock found
[    9.325117] EXT4-fs (sda2): error loading journal
```


> > 
> > Also, I think this behaviour may impact on bmap-tools "psplash"
> > patches
> > currently we have on maillist. I would expect that user won't see
> > the
> > progress bar on screen because it's somewhere on serial console.
> > 
> > 
> > >  StandardInput=tty
> > >  StandardOutput=tty
> > > -- 
> > > 2.43.0
> > > 
> > 
> 

-- 
Best regards,
Uladzimir.



-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/f56884186356c514f66cbea793b521640be7ee1a.camel%40ilbers.de.