Re: expand-on-first-boot and surprising umounts

[Jan Kiszka <jan.kiszka@siemens.com>]

On 21.10.22 11:41, Bezdeka, Florian wrote:
> On Fri, 2022-10-21 at 11:36 +0200, Raphael Lisicki wrote:
>>
>> On 21.10.22 10:45, Bezdeka, Florian (T CED SES-DE) wrote:
>>> On Fri, 2022-10-21 at 09:48 +0200, Raphael Lisicki wrote:
>>>> Hi everybody,
>>>>
>>>> I am using a debian bullseye based system and use expand-on-first-boot
>>>> to expand the last partition. It is not the root file system but an
>>>> extra ext4 partition to be mounted under /data. The mounting happens
>>>> after expand-on-first boot has succeeded.
>>>>
>>>> Sometimes, on some builds more often than others, /data gets (attempted
>>>> to be) umounted immediately after being mounted and subsequent units
>>>> will fail.
>>>
>>> Hm... Nothing happens on build time. Everything takes place on the
>>> "first" boot (until expand-on-first-boot disables itself). "more often"
>>> is also confusing me because expand-on-first-boot should run exactly
>>> once.
>>
>> Yes, it runs on first boot, but I can use the very same original image 
>> multiple times (after restoring it). And I can also try the same with 
>> other builds (doing a first boot) and some builds seem to be more often 
>> affected than others. My guess from this is that the exact 
>> alignment/size/padding/moon phase of some things in the image also plays 
>> a role.
>>
>>
>>>
>>>>
>>>> Removing expand-on-first-boot resolves the issue, so does adding
>>>> "ExecStartPost=/usr/bin/udevadm settle" to expand-on-first-boot but I am
>>>> not sure if this is only a sophisticated way of solving a race condition
>>>> with "sleep".
>>>>
>>>> My gut feeling is that after expand-on-first boot finishes, the kernel
>>>> still processes block device events, which systemd gets after /data has
>>>> already been mounted and then cause it to be umounted, as systemd was
>>>> already picky with umounting stuff in the past [1]. Unfortunately I have
>>>> no idea how to test this hypothesis.
>>>
>>> Nothing should be mounted (except the rootfs) until expand-on-first
>>> completed. See below.
>>>
>>>>
>>>> Did anyone already experience something like this? The racy-ness of the
>>>> issue makes creating a minimal reproduction hard.
>>>
>>> Nope, have never seen that, but that doesn't mean that there is no
>>> race.
>>>
>>> We have
>>>
>>> After=systemd-remount-fs.service
>>> Before=local-fs-pre.target shutdown.target
>>> ConditionPathIsReadWrite=/etc
>>>
>>> inside the expand-on-first-boot.service file.
>>>
>>> So we should be done before systemd tries to mount your /data
>>> (according to Before=) and start after systemd did remounting in case
>>> mount options (ro, rw, ...) have to be adjusted.
>>>
>>> I guess you should check your startup order (systemd-analyze might
>>> help) and report back if we missed something.
>>
>> systemd-analyze shows exactly what you described: after 
>> expand-on-first-boot has finished, /data gets mounted (and immedeately 
>> umounted again).
> 
> OK, so you have to figure out why this happens. Hopefully there are
> some logs available...
> 
> What file system is on your /data partition? What is the initial size
> and the full-blown size?
> 

This pattern rings a bell, I've seen something like this on the IOT2050
as well. I'm still looking for the hack that worked around it, maybe
Quirin recalls that more quickly. And I think that hack was obsoleted
when we moved to r/o-rootfs and overlays...

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux