Re: expand-on-first-boot and surprising umounts

["Bezdeka, Florian" <florian.bezdeka@siemens.com>]

On Fri, 2022-10-21 at 11:36 +0200, Raphael Lisicki wrote:
> 
> On 21.10.22 10:45, Bezdeka, Florian (T CED SES-DE) wrote:
> > On Fri, 2022-10-21 at 09:48 +0200, Raphael Lisicki wrote:
> > > Hi everybody,
> > > 
> > > I am using a debian bullseye based system and use expand-on-first-boot
> > > to expand the last partition. It is not the root file system but an
> > > extra ext4 partition to be mounted under /data. The mounting happens
> > > after expand-on-first boot has succeeded.
> > > 
> > > Sometimes, on some builds more often than others, /data gets (attempted
> > > to be) umounted immediately after being mounted and subsequent units
> > > will fail.
> > 
> > Hm... Nothing happens on build time. Everything takes place on the
> > "first" boot (until expand-on-first-boot disables itself). "more often"
> > is also confusing me because expand-on-first-boot should run exactly
> > once.
> 
> Yes, it runs on first boot, but I can use the very same original image 
> multiple times (after restoring it). And I can also try the same with 
> other builds (doing a first boot) and some builds seem to be more often 
> affected than others. My guess from this is that the exact 
> alignment/size/padding/moon phase of some things in the image also plays 
> a role.
> 
> 
> > 
> > > 
> > > Removing expand-on-first-boot resolves the issue, so does adding
> > > "ExecStartPost=/usr/bin/udevadm settle" to expand-on-first-boot but I am
> > > not sure if this is only a sophisticated way of solving a race condition
> > > with "sleep".
> > > 
> > > My gut feeling is that after expand-on-first boot finishes, the kernel
> > > still processes block device events, which systemd gets after /data has
> > > already been mounted and then cause it to be umounted, as systemd was
> > > already picky with umounting stuff in the past [1]. Unfortunately I have
> > > no idea how to test this hypothesis.
> > 
> > Nothing should be mounted (except the rootfs) until expand-on-first
> > completed. See below.
> > 
> > > 
> > > Did anyone already experience something like this? The racy-ness of the
> > > issue makes creating a minimal reproduction hard.
> > 
> > Nope, have never seen that, but that doesn't mean that there is no
> > race.
> > 
> > We have
> > 
> > After=systemd-remount-fs.service
> > Before=local-fs-pre.target shutdown.target
> > ConditionPathIsReadWrite=/etc
> > 
> > inside the expand-on-first-boot.service file.
> > 
> > So we should be done before systemd tries to mount your /data
> > (according to Before=) and start after systemd did remounting in case
> > mount options (ro, rw, ...) have to be adjusted.
> > 
> > I guess you should check your startup order (systemd-analyze might
> > help) and report back if we missed something.
> 
> systemd-analyze shows exactly what you described: after 
> expand-on-first-boot has finished, /data gets mounted (and immedeately 
> umounted again).

OK, so you have to figure out why this happens. Hopefully there are
some logs available...

What file system is on your /data partition? What is the initial size
and the full-blown size?

> 
> 
> 
> best regards
> raphael