Re: [PATCH 03/17] meta-isar: deploy-image: Change reboot logic

[Uladzimir Bely <ubely@ilbers.de>]

On Fri, 2025-03-14 at 14:25 +0000, Heinisch, Alexander wrote:
> On Fri, 2025-03-14 at 09:25 +0300, Uladzimir Bely wrote:
> > On Fri, 2025-03-14 at 06:54 +0100, Jan Kiszka wrote:
> > > On 13.03.25 14:10, Uladzimir Bely wrote:
> > > > On Tue, 2024-07-02 at 15:38 +0200, 'Jan Kiszka' via isar-users
> > > > wrote:
> > > > > From: Jan Kiszka <jan.kiszka@siemens.com>
> > > > > 
> > > > > Pull the reboot out of the script. This allows for cleaner
> > > > > integration
> > > > > with different calling environment, may they be a systemd
> > > > > unit,
> > > > > an
> > > > > initramfs script or simply a shell for testing purposes.
> > > > > 
> > > > > And if the script exits with an error, wait a minute before
> > > > > rebooting
> > > > > the system, rather than just trying to re-execute it. This
> > > > > permits to
> > > > > inspect potential error as well.
> > > > > 
> > > > > Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> > > > > ---
> > > > >  .../recipes-installer/deploy-image/files/deploy-image-
> > > > > wic.sh   
> > > > > > 2
> > > > > +-
> > > > >  .../recipes-installer/deploy-
> > > > > image/files/install.override.conf 
> > > > > > 2
> > > > > +-
> > > > >  2 files changed, 2 insertions(+), 2 deletions(-)
> > > > > 
> > > > > diff --git a/meta-isar/recipes-installer/deploy-
> > > > > image/files/deploy-
> > > > > image-wic.sh b/meta-isar/recipes-installer/deploy-
> > > > > image/files/deploy-
> > > > > image-wic.sh
> > > > > index 8043aff1..12c1eea2 100644
> > > > > --- a/meta-isar/recipes-installer/deploy-image/files/deploy-
> > > > > image-
> > > > > wic.sh
> > > > > +++ b/meta-isar/recipes-installer/deploy-image/files/deploy-
> > > > > image-
> > > > > wic.sh
> > > > > @@ -105,4 +105,4 @@ fi
> > > > >  umount "$installdata"
> > > > >  sync
> > > > >  dialog --title "Reboot" --msgbox "Installation is
> > > > > successful.
> > > > > System
> > > > > will be rebooted. Please remove the USB stick." 7 60
> > > > > -reboot
> > > > > +exit 0
> > > > > diff --git a/meta-isar/recipes-installer/deploy-
> > > > > image/files/install.override.conf b/meta-isar/recipes-
> > > > > installer/deploy-image/files/install.override.conf
> > > > > index 73874caa..357d8662 100644
> > > > > --- a/meta-isar/recipes-installer/deploy-
> > > > > image/files/install.override.conf
> > > > > +++ b/meta-isar/recipes-installer/deploy-
> > > > > image/files/install.override.conf
> > > > > @@ -1,5 +1,5 @@
> > > > >  [Service]
> > > > >  ExecStart=
> > > > > -ExecStart=/usr/bin/deploy-image-wic.sh
> > > > > +ExecStart=/bin/sh -c "deploy-image-wic.sh || (echo
> > > > > 'Rebooting
> > > > > in
> > > > > 60
> > > > > s'; sleep 60); reboot"
> > > > 
> > > > Hello Jan.
> > > > 
> > > > This change seems to cause a sort of race issue.
> > > > 
> > > > Few days ago "[PATCH v3 0/2] Cover installer image with tests"
> > > > was
> > > > merged. Tests were OK (and "next" is still OK) on two jenkins
> > > > and
> > > > one
> > > > gitlab instances. But on the third jenkins instance, they were
> > > > failing.
> > > > 
> > > > Boot log:
> > > > 
> > > > ```
> > > > Got config:
> > > >   installer_unattended=true
> > > >   installer_image_uri=/install/isar-image-ci-debian-bookworm-
> > > > qemuamd64.wic.zst
> > > >   installer_target_dev=/dev/sda
> > > >   installer_target_overwrite=OVERWRITE
> > > > bmaptool: ERROR: An error occurred, here is the traceback:
> > > > Traceback (most recent call last):
> > > >   File "/usr/lib/python3/dist-packages/bmaptools/CLI.py", line
> > > > 116,
> > > > in
> > > > open_block_device
> > > >     descriptor = os.open(path, os.O_WRONLY | os.O_EXCL)
> > > > 
> > > > bmaptool: ERROR: cannot open block device '/dev/sda' in
> > > > exclusive
> > > > mode:
> > > > [Errno 16] Device or resource busy: '/dev/sda'
> > > > 
> > > > Rebooting in 60 s
> > > > ```
> > > > 
> > > > But either machine is not booted, or there are some "ext4"
> > > > errors
> > > > happen.
> > > > 
> > > > I couldn't understand few things:
> > > >  - why this happens (unable to get exclusive access to block
> > > > device)$
> > > >  - why device is written even if there is such error in logs$
> > > >  - why sometimes we don't see this error in logs$
> > > >  - why this doesn't happen when "unattended mode" is off.
> 
> 
> It does not happen when "unattended mode" is off, because the race is
> interrupted by the script waiting for user input at the very
> beginning.
> 
> This is a race condition between multiple instances of the install
> script executed in parallel, dependent on how many gettys are active.
> Since the number of gettys and further, which getty get's connected
> in
> the end, via terminal, via serial, ... is kind of hard to predict,
> imo
> the only option to resolve that race is by adding a dedicated
> (singleton) service which is executed on startup.
> 
> I am currently preparing a patch set where a fix for that is
> included,
> similar to this:

Yes, there were rhetorical questions that get ansered after later
debugging.

I wanted (but didn't have time to try) to do something like in the
script:

```
if ! $installer_unattended; then
    if [ "$(tty)" != "/dev/ttyS0" ]; then
        echo "Disable unattended mode on $(tty), it's active on
/dev/ttyS0"
        installer_unattended=0
    fi
fi
```

Futher, "/dev/ttyS0" could be not hardcoded but also passed from
installer configuration during build.

> 
> ```
> [Unit]
> Description=Target-bootstrapper service
> 
> [Service]
> User=root
> ExecStart=/bin/sh -c "/usr/bin/target-bootstrapper.sh || (echo
> 'Rebooting in 60 s'; sleep 60); reboot"
> 
> StandardOutput=journal+console
> TTYPath=/dev/console
> 
> [Install]
> WantedBy=multi-user.target
> ``
> 
> BR Alexander
> 
> > > > 
> > > > The reason is that we have these two "getty" instances
> > > > overriden
> > > > by
> > > > custom one having the race. While
> > > > "serial-getty@ttyS0.service.d"
> > > > does
> > > > the job, "getty@tty1.service.d" produces this error visible in
> > > > logs
> > > > and
> > > > reboot machine in 60 seconds.
> > > > 
> > > > In some cases 60sec is enough for first service to finish
> > > > writing
> > > > to
> > > > the disc, but on slower machines we result in incomplete copy
> > > > and
> > > > broken system.
> > > > 
> > > > Before this patch, when "reboot" calls were in the script,
> > > > "failed"
> > > > instance simply exited and didn't try to reboot the board.
> > > > 
> > > > As a quick fix, I would reverted this patch.
> > > 
> > > That won't be a fix, just papering over.
> > > 
> > > The race is apparently around the target device becoming ready
> > > for
> > > bmaptool, or our script racing with some other service accessing
> > > the
> > > device for a while. I suspect the service is lacking some
> > > dependency
> > > here. So, let's identify the race partner first: How to reproduce
> > > locally?
> > > 
> > > Jan
> > > 
> > 
> > No, my finding is that getty@tty1.service.d conflicts with
> > serial-getty@ttyS0.service.d one. Every of them tries to run
> > bmaptool
> > when unattended mode is active.
> > 
> > meta-isar/recipes-installer/deploy-image-service/deploy-image-
> > service.bb:
> > 
> > ```
> > do_install() {
> >   install -m 0600 ${WORKDIR}/install.override.conf
> > ${D}/usr/lib/systemd/system/getty@tty1.service.d/override.conf
> >   install -m 0600 ${WORKDIR}/install.override.conf
> > ${D}/usr/lib/systemd/system/serial-getty@ttyS0.service.d/override.c
> > on
> > f
> > }
> > ```
> > 
> > It can be easy reproduced if we decrease timeout from 60s to e.g.
> > 10s,
> > to simulate slower system:
> > 
> > ```
> > isar $ sed -i -e 's/60/10/g' meta-isar/recipes-installer/deploy-
> > image-
> > service/files/install.override.conf 
> > isar $ ./kas/kas-container shell kas/isar.yaml 
> > builder@b21a175d3b78:/build$ rm -rf conf/
> > builder@b21a175d3b78:/build$ /repo/scripts/ci_build.sh -T installer
> > ```
> > 
> > Failed instance of overriden getty reboots the system while the
> > target
> > is being populated with the not failed instance.
> > 
> > Reboot hangs at something like
> > 
> > ```
> > [    9.323801] JBD2: no valid journal superblock found
> > [    9.325117] EXT4-fs (sda2): error loading journal
> > ```
> > 
> > 
> > > > 
> > > > Also, I think this behaviour may impact on bmap-tools "psplash"
> > > > patches
> > > > currently we have on maillist. I would expect that user won't
> > > > see
> > > > the
> > > > progress bar on screen because it's somewhere on serial
> > > > console.
> > > > 
> > > > 
> > > > >  StandardInput=tty
> > > > >  StandardOutput=tty
> > > > > -- 
> > > > > 2.43.0
> > > > > 
> > > > 
> > > 
> > 
> > -- 
> > Best regards,
> > Uladzimir.
> > 
> > 
> > 
> 

-- 
Best regards,
Uladzimir.



-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/05793724b0dfe2ec6a986c0bad83476f12a2ccda.camel%40ilbers.de.