Re: expand-on-first-boot and surprising umounts

["Bezdeka, Florian" <florian.bezdeka@siemens.com>]

On Fri, 2022-10-21 at 09:48 +0200, Raphael Lisicki wrote:
> Hi everybody,
> 
> I am using a debian bullseye based system and use expand-on-first-boot 
> to expand the last partition. It is not the root file system but an 
> extra ext4 partition to be mounted under /data. The mounting happens 
> after expand-on-first boot has succeeded.
> 
> Sometimes, on some builds more often than others, /data gets (attempted 
> to be) umounted immediately after being mounted and subsequent units 
> will fail.

Hm... Nothing happens on build time. Everything takes place on the
"first" boot (until expand-on-first-boot disables itself). "more often"
is also confusing me because expand-on-first-boot should run exactly
once.

> 
> Removing expand-on-first-boot resolves the issue, so does adding 
> "ExecStartPost=/usr/bin/udevadm settle" to expand-on-first-boot but I am 
> not sure if this is only a sophisticated way of solving a race condition 
> with "sleep".
> 
> My gut feeling is that after expand-on-first boot finishes, the kernel
> still processes block device events, which systemd gets after /data has
> already been mounted and then cause it to be umounted, as systemd was
> already picky with umounting stuff in the past [1]. Unfortunately I have
> no idea how to test this hypothesis.

Nothing should be mounted (except the rootfs) until expand-on-first
completed. See below.

> 
> Did anyone already experience something like this? The racy-ness of the 
> issue makes creating a minimal reproduction hard.

Nope, have never seen that, but that doesn't mean that there is no
race.

We have

After=systemd-remount-fs.service
Before=local-fs-pre.target shutdown.target
ConditionPathIsReadWrite=/etc

inside the expand-on-first-boot.service file.

So we should be done before systemd tries to mount your /data
(according to Before=) and start after systemd did remounting in case
mount options (ro, rw, ...) have to be adjusted.

I guess you should check your startup order (systemd-analyze might
help) and report back if we missed something.

> 
> 
> 
> [1] https://github.com/systemd/systemd/issues/9869
> 
> 
> 
> best regards
> raphael
>