Re: expand-on-first-boot and surprising umounts

[Raphael Lisicki <raphael.lisicki@siemens.com>]

On 21.10.22 10:45, Bezdeka, Florian (T CED SES-DE) wrote:
> On Fri, 2022-10-21 at 09:48 +0200, Raphael Lisicki wrote:
>> Hi everybody,
>>
>> I am using a debian bullseye based system and use expand-on-first-boot
>> to expand the last partition. It is not the root file system but an
>> extra ext4 partition to be mounted under /data. The mounting happens
>> after expand-on-first boot has succeeded.
>>
>> Sometimes, on some builds more often than others, /data gets (attempted
>> to be) umounted immediately after being mounted and subsequent units
>> will fail.
> 
> Hm... Nothing happens on build time. Everything takes place on the
> "first" boot (until expand-on-first-boot disables itself). "more often"
> is also confusing me because expand-on-first-boot should run exactly
> once.

Yes, it runs on first boot, but I can use the very same original image 
multiple times (after restoring it). And I can also try the same with 
other builds (doing a first boot) and some builds seem to be more often 
affected than others. My guess from this is that the exact 
alignment/size/padding/moon phase of some things in the image also plays 
a role.


> 
>>
>> Removing expand-on-first-boot resolves the issue, so does adding
>> "ExecStartPost=/usr/bin/udevadm settle" to expand-on-first-boot but I am
>> not sure if this is only a sophisticated way of solving a race condition
>> with "sleep".
>>
>> My gut feeling is that after expand-on-first boot finishes, the kernel
>> still processes block device events, which systemd gets after /data has
>> already been mounted and then cause it to be umounted, as systemd was
>> already picky with umounting stuff in the past [1]. Unfortunately I have
>> no idea how to test this hypothesis.
> 
> Nothing should be mounted (except the rootfs) until expand-on-first
> completed. See below.
> 
>>
>> Did anyone already experience something like this? The racy-ness of the
>> issue makes creating a minimal reproduction hard.
> 
> Nope, have never seen that, but that doesn't mean that there is no
> race.
> 
> We have
> 
> After=systemd-remount-fs.service
> Before=local-fs-pre.target shutdown.target
> ConditionPathIsReadWrite=/etc
> 
> inside the expand-on-first-boot.service file.
> 
> So we should be done before systemd tries to mount your /data
> (according to Before=) and start after systemd did remounting in case
> mount options (ro, rw, ...) have to be adjusted.
> 
> I guess you should check your startup order (systemd-analyze might
> help) and report back if we missed something.

systemd-analyze shows exactly what you described: after 
expand-on-first-boot has finished, /data gets mounted (and immedeately 
umounted again).



best regards
raphael