[PATCH 0/3] Detect systemd services ordering cycle

[PATCH 0/3] Detect systemd services ordering cycle

[Uladzimir Bely]

Merging v3 of "expand-on-first-boot: Ensure that /tmp is writable"
led to systemd services ordering cycle. Systemd randomly drops one
of sycled services and sometimes it's "expand-on-first-boot" one.

This patchset reverts version 3 of the patch and returns to
previous version.

Additionally, we extend CI so that qemu run tests could always detect
any systemd services ordering cycles.

Clara Kowalsky (1):
  expand-on-first-boot: Add /tmp to ConditionPathIsReadWrite

Uladzimir Bely (2):
  Revert "expand-on-first-boot: Ensure that /tmp is writable"
  testsuite: Detect systemd services ordering cycle

 .../expand-on-first-boot/files/expand-on-first-boot.service  | 3 +--
 testsuite/cibuilder.py                                       | 5 +++++
 2 files changed, 6 insertions(+), 2 deletions(-)

-- 
2.44.2

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240816144333.27467-1-ubely%40ilbers.de.

[PATCH 1/3] Revert "expand-on-first-boot: Ensure that /tmp is writable"

[Uladzimir Bely]

This reverts commit 49073b9e716b2fb3203d26d9bd46b3bbd60e6016.
---
 .../expand-on-first-boot/files/expand-on-first-boot.service      | 1 -
 1 file changed, 1 deletion(-)

diff --git a/meta/recipes-support/expand-on-first-boot/files/expand-on-first-boot.service b/meta/recipes-support/expand-on-first-boot/files/expand-on-first-boot.service
index 8e76998b..90c92a39 100644
--- a/meta/recipes-support/expand-on-first-boot/files/expand-on-first-boot.service
+++ b/meta/recipes-support/expand-on-first-boot/files/expand-on-first-boot.service
@@ -16,7 +16,6 @@ Type=oneshot
 ExecStart=/usr/share/expand-on-first-boot/expand-last-partition.sh
 ExecStartPost=-/bin/systemctl disable expand-on-first-boot.service
 ExecStopPost=-/bin/systemctl disable expand-on-first-boot.service
-PrivateTmp=true
 
 [Install]
 WantedBy=sysinit.target
-- 
2.44.2

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240816144333.27467-2-ubely%40ilbers.de.

[PATCH 2/3] expand-on-first-boot: Add /tmp to ConditionPathIsReadWrite

[Uladzimir Bely]

From: Clara Kowalsky <clara.kowalsky@siemens.com>

Since mktemp is used and creates a temporary file, we need to ensure
that /tmp is writable.

Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com>
---
 .../expand-on-first-boot/files/expand-on-first-boot.service     | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-support/expand-on-first-boot/files/expand-on-first-boot.service b/meta/recipes-support/expand-on-first-boot/files/expand-on-first-boot.service
index 90c92a39..9daae719 100644
--- a/meta/recipes-support/expand-on-first-boot/files/expand-on-first-boot.service
+++ b/meta/recipes-support/expand-on-first-boot/files/expand-on-first-boot.service
@@ -9,7 +9,7 @@ DefaultDependencies=no
 Conflicts=shutdown.target
 After=systemd-remount-fs.service
 Before=local-fs-pre.target shutdown.target
-ConditionPathIsReadWrite=/etc
+ConditionPathIsReadWrite=/etc /tmp
 
 [Service]
 Type=oneshot
-- 
2.44.2

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240816144333.27467-3-ubely%40ilbers.de.

[PATCH 3/3] testsuite: Detect systemd services ordering cycle

[Uladzimir Bely]

This allows to recognize any cyclic dependencies between
systemd services.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/cibuilder.py | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 3f77cd42..1e2c265f 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -556,6 +556,8 @@ class CIBuilder(Test):
         # the printk of recipes-kernel/example-module
         module_output = b'Just an example'
         resize_output = None
+        # systemd service ordering cycle
+        ordering_cycle = b'Found ordering cycle'
         image_fstypes, wks_file, bbdistro = CIUtils.getVars(
             'IMAGE_FSTYPES', 'WKS_FILE', 'DISTRO', target=multiconfig
         )
@@ -582,6 +584,9 @@ class CIBuilder(Test):
                 else:
                     rc = 2
                     self.log.error("No example module output while expected")
+                if ordering_cycle in data:
+                    rc = 3
+                    self.log.error("Systemd services ordering cycle detected")
         return rc
 
     def vm_dump_dict(self, vm):
-- 
2.44.2

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240816144333.27467-4-ubely%40ilbers.de.

Re: [PATCH 0/3] Detect systemd services ordering cycle

[Uladzimir Bely]

On Fri, 2024-08-16 at 17:38 +0300, Uladzimir Bely wrote:
> Merging v3 of "expand-on-first-boot: Ensure that /tmp is writable"
> led to systemd services ordering cycle. Systemd randomly drops one
> of sycled services and sometimes it's "expand-on-first-boot" one.
> 
> This patchset reverts version 3 of the patch and returns to
> previous version.
> 
> Additionally, we extend CI so that qemu run tests could always detect
> any systemd services ordering cycles.
> 
> Clara Kowalsky (1):
>   expand-on-first-boot: Add /tmp to ConditionPathIsReadWrite
> 
> Uladzimir Bely (2):
>   Revert "expand-on-first-boot: Ensure that /tmp is writable"
>   testsuite: Detect systemd services ordering cycle
> 
>  .../expand-on-first-boot/files/expand-on-first-boot.service  | 3 +--
>  testsuite/cibuilder.py                                       | 5
> +++++
>  2 files changed, 6 insertions(+), 2 deletions(-)
> 
> -- 
> 2.44.2
> 

Hello all.

If there is no objection, we would like to merge this rather soon (in
~2 days) since currently CI is broken.

-- 
Best regards,
Uladzimir.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/048f5d185907d1049537daea9eaf1f95dbb403cc.camel%40ilbers.de.

Re: [PATCH 0/3] Detect systemd services ordering cycle

[Uladzimir Bely]

On Fri, 2024-08-16 at 17:38 +0300, Uladzimir Bely wrote:
> Merging v3 of "expand-on-first-boot: Ensure that /tmp is writable"
> led to systemd services ordering cycle. Systemd randomly drops one
> of sycled services and sometimes it's "expand-on-first-boot" one.
> 
> This patchset reverts version 3 of the patch and returns to
> previous version.
> 
> Additionally, we extend CI so that qemu run tests could always detect
> any systemd services ordering cycles.
> 
> Clara Kowalsky (1):
>   expand-on-first-boot: Add /tmp to ConditionPathIsReadWrite
> 
> Uladzimir Bely (2):
>   Revert "expand-on-first-boot: Ensure that /tmp is writable"
>   testsuite: Detect systemd services ordering cycle
> 
>  .../expand-on-first-boot/files/expand-on-first-boot.service  | 3 +--
>  testsuite/cibuilder.py                                       | 5
> +++++
>  2 files changed, 6 insertions(+), 2 deletions(-)
> 
> -- 
> 2.44.2
> 

Applied to next.

-- 
Best regards,
Uladzimir.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/3ef96fdd2414065059e49b74e908edf940be5934.camel%40ilbers.de.

Re: [PATCH 0/3] Detect systemd services ordering cycle

['Jan Kiszka' via isar-users]

On 21.08.24 06:49, Uladzimir Bely wrote:
> On Fri, 2024-08-16 at 17:38 +0300, Uladzimir Bely wrote:
>> Merging v3 of "expand-on-first-boot: Ensure that /tmp is writable"
>> led to systemd services ordering cycle. Systemd randomly drops one
>> of sycled services and sometimes it's "expand-on-first-boot" one.
>>
>> This patchset reverts version 3 of the patch and returns to
>> previous version.
>>
>> Additionally, we extend CI so that qemu run tests could always detect
>> any systemd services ordering cycles.
>>
>> Clara Kowalsky (1):
>>   expand-on-first-boot: Add /tmp to ConditionPathIsReadWrite
>>
>> Uladzimir Bely (2):
>>   Revert "expand-on-first-boot: Ensure that /tmp is writable"
>>   testsuite: Detect systemd services ordering cycle
>>
>>  .../expand-on-first-boot/files/expand-on-first-boot.service  | 3 +--
>>  testsuite/cibuilder.py                                       | 5
>> +++++
>>  2 files changed, 6 insertions(+), 2 deletions(-)
>>
>> -- 
>> 2.44.2
>>
> 
> Applied to next.
> 

Still broken:

root@isar:~# journalctl -u expand-on-first-boot
-- Boot 336ca3ef86c74a06ab8e11e40a321c84 --
Jun 16 09:44:32 isar systemd[1]: expand-on-first-boot.service - Expand last partition was skipped because of an unmet condition check (ConditionPathIsReadWrite=/etc /tmp).
root@isar:~# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-6.1.0-23-amd64 root=PARTLABEL=platform rw rootwait console=ttyS0,115200 console=tty0

Patch will follow.

Why didn't you test this?

Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/cb630784-3b27-483a-8bd2-5beed2462d2a%40siemens.com.

Re: [PATCH 0/3] Detect systemd services ordering cycle

[Uladzimir Bely]

On Sat, 2024-08-31 at 10:45 +0200, Jan Kiszka wrote:
> On 21.08.24 06:49, Uladzimir Bely wrote:
> > On Fri, 2024-08-16 at 17:38 +0300, Uladzimir Bely wrote:
> > > Merging v3 of "expand-on-first-boot: Ensure that /tmp is
> > > writable"
> > > led to systemd services ordering cycle. Systemd randomly drops
> > > one
> > > of sycled services and sometimes it's "expand-on-first-boot" one.
> > > 
> > > This patchset reverts version 3 of the patch and returns to
> > > previous version.
> > > 
> > > Additionally, we extend CI so that qemu run tests could always
> > > detect
> > > any systemd services ordering cycles.
> > > 
> > > Clara Kowalsky (1):
> > >   expand-on-first-boot: Add /tmp to ConditionPathIsReadWrite
> > > 
> > > Uladzimir Bely (2):
> > >   Revert "expand-on-first-boot: Ensure that /tmp is writable"
> > >   testsuite: Detect systemd services ordering cycle
> > > 
> > >  .../expand-on-first-boot/files/expand-on-first-boot.service  | 3
> > > +--
> > >  testsuite/cibuilder.py                                       | 5
> > > +++++
> > >  2 files changed, 6 insertions(+), 2 deletions(-)
> > > 
> > > -- 
> > > 2.44.2
> > > 
> > 
> > Applied to next.
> > 
> 
> Still broken:
> 
> root@isar:~# journalctl -u expand-on-first-boot
> -- Boot 336ca3ef86c74a06ab8e11e40a321c84 --
> Jun 16 09:44:32 isar systemd[1]: expand-on-first-boot.service -
> Expand last partition was skipped because of an unmet condition check
> (ConditionPathIsReadWrite=/etc /tmp).
> root@isar:~# cat /proc/cmdline 
> BOOT_IMAGE=/vmlinuz-6.1.0-23-amd64 root=PARTLABEL=platform rw
> rootwait console=ttyS0,115200 console=tty0
> 
> Patch will follow.
> 
> Why didn't you test this?
> 

Hello.

The patchset was autotested in CI several times, and, according to VM
logs (for example, with bullseye/amd64 it previusly was always failing)
everything is OK:

```
[    8.433834] systemd[1]: Starting Expand last partition...
[    8.467179] systemd[1]: Condition check resulted in Rebuild Hardware
Database being skipped.
[    8.481797] systemd[1]: Condition check resulted in Platform
Persistent Storage Archival being skipped.
[    8.492728] random: systemd: uninitialized urandom read (16 bytes
read)
[    8.520001] systemd[1]: Starting Load/Save Random Seed...
[    8.581835] systemd[1]: Starting Apply Kernel Variables...
[    8.655061] systemd[1]: Starting Create System Users...
[    8.738683] systemd[1]: Mounted FUSE Control File System.
[    8.786544] systemd[1]: Mounted Kernel Configuration File System.
[    8.837270] systemd[1]: Finished Apply Kernel Variables.
[    8.934155] systemd[1]: Started Journal Service.
[    9.096109] random: crng init done
[    9.098673] random: 46 urandom warning(s) missed due to ratelimiting
[    9.151174] systemd-journald[195]: Received client request to flush
runtime journal.
[    9.175233] systemd-journald[195]: Creating journal file
/var/log/journal/2cada36492d746358164b901eda86292/system.journal on a
btrfs file system, and copy-on-write is enabled. This is likely to slow
down journal access substantially, please consider turning off the
copy-on-write file attribute on the journal directory, using chattr +C.
[   10.292661] BTRFS info (device sda2): resize device /dev/sda2 (devid
1) from 1305477120 to 1573912576
```

Now I looked to all logs and found for "bookworm/arm" the following:

```
[   24.219166] systemd[1]: expand-on-first-boot.service - Expand last
partition was skipped because of an unmet condition check
(ConditionPathIsReadWrite=/etc /tmp).
```

We probably should to extend checks in testsuite, while currenaly only
images using "sdimage-efi-sd" and "sdimage-efi-btrfs" only are tested. 

Probably, this depend on systemd version and bullseye and lower
versions are not affected.

> Jan
> 

-- 
Best regards,
Uladzimir.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/58d072e0ac650f6480aedf69071a12efc92ff8f0.camel%40ilbers.de.