[PATCH v6 0/6] expand-on-first-boot btrfs and CI testing

[PATCH v6 0/6] expand-on-first-boot btrfs and CI testing

[henning.schild]

From: Henning Schild <henning.schild@siemens.com>

changes since v5:
 - drop dead code around mktemp subshell
 - revert cosmetic mktemp stuff back to v3
 - remove "Suggested-by" tag to hopefully improve signal to noise ratio
   in review process

changes since v4:
 - cosmetic fix in the way we call mktemp
 - add a patch that makes sure the service is run only once, also on
   error
 - bump package version after changes are done

changes since v3:
 - rebased on "expand-on-first-boot: switch back away from systemd-growfs"
 - implement btrfs resizing and CI testing it
 - put the padding wks entry into an include

changes since v2:
 - add python namespace prefix for the bitbake get functions
 - install only in images that use WKS_FILEs that have the "gap"

changes since v1:
 - complete rewrite to not focus only on the tests but rather the
   feature for all examples

This first was only about testing but meanwhile has a focus on enabling
the expand feature in all example images anyhow, and later look for
traces of that happening in CI test where we can.
We install the application in any image and in the wic images add some
scratch space at the end. That will enable improved interactive use of
all those wic images, so makes sense apart from any testing.
Later use the log reading feature of the qemu tests to look for traces
of successful resizing for non ubuntu images. In ubuntu the
console_loglevel is not verbose enough to find traces.

Henning Schild (6):
  expand-on-first-boot: really only do that once, especially on failure
  expand-on-first-boot: support resizing a btrfs
  meta-isar: introduce an example to use btrfs
  CI: improve cibuilder readability
  meta-isar: install expand-on-first-boot in most images and add space
  CI: expect a message about filesystem resize vom expand script

 meta-isar/conf/machine/qemuamd64.conf         |  6 ++++
 meta-isar/conf/machine/virtualbox.conf        |  1 +
 meta-isar/conf/machine/vmware.conf            |  1 +
 .../conf/multiconfig/qemuarm-bookworm.conf    |  1 +
 .../lib/wic/canned-wks/expand-padding.wks.inc |  5 +++
 .../lib/wic/canned-wks/sdimage-efi-btrfs.wks  | 11 +++++++
 .../lib/wic/canned-wks/sdimage-efi-sd.wks     |  2 ++
 .../lib/wic/canned-wks/sdimage-efi.wks        |  2 ++
 ...oot_1.3.bb => expand-on-first-boot_1.4.bb} |  0
 .../files/expand-last-partition.sh            | 31 ++++++++++++++++---
 .../files/expand-on-first-boot.service        |  1 +
 testsuite/cibuilder.py                        | 28 ++++++++++++++---
 12 files changed, 80 insertions(+), 9 deletions(-)
 create mode 100644 meta-isar/scripts/lib/wic/canned-wks/expand-padding.wks.inc
 create mode 100644 meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-btrfs.wks
 rename meta/recipes-support/expand-on-first-boot/{expand-on-first-boot_1.3.bb => expand-on-first-boot_1.4.bb} (100%)

-- 
2.37.4

[PATCH v6 1/6] expand-on-first-boot: really only do that once, especially on failure

[henning.schild]

From: Henning Schild <henning.schild@siemens.com>

We do hope the expand will work but we also know very well that it can
fail. The script is written in a way that currently only supports one
try. And not a retry if it failed in an earlier boot.
So make sure to disable the service also on failure.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 .../expand-on-first-boot/files/expand-on-first-boot.service      | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-support/expand-on-first-boot/files/expand-on-first-boot.service b/meta/recipes-support/expand-on-first-boot/files/expand-on-first-boot.service
index fda50016e9fe..90c92a396b27 100644
--- a/meta/recipes-support/expand-on-first-boot/files/expand-on-first-boot.service
+++ b/meta/recipes-support/expand-on-first-boot/files/expand-on-first-boot.service
@@ -15,6 +15,7 @@ ConditionPathIsReadWrite=/etc
 Type=oneshot
 ExecStart=/usr/share/expand-on-first-boot/expand-last-partition.sh
 ExecStartPost=-/bin/systemctl disable expand-on-first-boot.service
+ExecStopPost=-/bin/systemctl disable expand-on-first-boot.service
 
 [Install]
 WantedBy=sysinit.target
-- 
2.37.4

[PATCH v6 2/6] expand-on-first-boot: support resizing a btrfs

[henning.schild]

From: Henning Schild <henning.schild@siemens.com>

This adds support for resizing btrfs filesystems if they are in that
last partition. It also prepares for potentially other filesystems to
come in the future by introducing a switch-case.

The mounting logic is taken from the systemd-growfs patches we had to
revert again. Some filesystems need to online for resizing, but in order
to find the filesystem of a partition (without udev) mounting it and
letting the kernel detect seems a good idea.

Suggested-by: Joe MacDonald <joe_macdonald@mentor.com>
Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 ...oot_1.3.bb => expand-on-first-boot_1.4.bb} |  0
 .../files/expand-last-partition.sh            | 31 ++++++++++++++++---
 2 files changed, 27 insertions(+), 4 deletions(-)
 rename meta/recipes-support/expand-on-first-boot/{expand-on-first-boot_1.3.bb => expand-on-first-boot_1.4.bb} (100%)

diff --git a/meta/recipes-support/expand-on-first-boot/expand-on-first-boot_1.3.bb b/meta/recipes-support/expand-on-first-boot/expand-on-first-boot_1.4.bb
similarity index 100%
rename from meta/recipes-support/expand-on-first-boot/expand-on-first-boot_1.3.bb
rename to meta/recipes-support/expand-on-first-boot/expand-on-first-boot_1.4.bb
diff --git a/meta/recipes-support/expand-on-first-boot/files/expand-last-partition.sh b/meta/recipes-support/expand-on-first-boot/files/expand-last-partition.sh
index 93eddda2a3b7..7d658864924b 100755
--- a/meta/recipes-support/expand-on-first-boot/files/expand-last-partition.sh
+++ b/meta/recipes-support/expand-on-first-boot/files/expand-last-partition.sh
@@ -62,8 +62,31 @@ if grep -q x-systemd.growfs /etc/fstab; then
 	exit 0
 fi
 
-# Do not fail resize2fs if no mtab entry is found, e.g.,
-# when using systemd mount units.
-export EXT2FS_NO_MTAB_OK=1
+# some filesystems need to be mounted i.e. btrfs, but mounting also helps
+# detect the filesystem type without having to wait for udev
+# mount $LAST_PART out of tree, so we won't conflict with other mounts
+MOUNT_POINT=$(mktemp -d -p "" "$(basename "$0").XXXXXXXXXX")
+mount "${LAST_PART}" "${MOUNT_POINT}"
 
-resize2fs "${LAST_PART}"
+ret=0
+# Determine the filesystem type and perform the appropriate resize function
+FS_TYPE=$(findmnt -fno FSTYPE "${MOUNT_POINT}" )
+case ${FS_TYPE} in
+ext*)
+	# Do not fail resize2fs if no mtab entry is found, e.g.,
+	# when using systemd mount units.
+	export EXT2FS_NO_MTAB_OK=1
+	resize2fs "${LAST_PART}"
+	;;
+btrfs)
+	btrfs filesystem resize max "${MOUNT_POINT}"
+	;;
+*)
+	echo "Unrecognized filesystem type ${FS_TYPE} - no resize performed"
+	ret=1
+	;;
+esac
+
+umount "${MOUNT_POINT}"
+rmdir "${MOUNT_POINT}"
+exit $ret
-- 
2.37.4

[PATCH v6 3/6] meta-isar: introduce an example to use btrfs

[henning.schild]

From: Henning Schild <henning.schild@siemens.com>

Switch debian bullseye over to btrfs rootfs and also to btrfs in last
partition to have a target where we test expand-on-first-boot for btrfs.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/conf/machine/qemuamd64.conf                    | 5 +++++
 .../scripts/lib/wic/canned-wks/sdimage-efi-btrfs.wks     | 9 +++++++++
 2 files changed, 14 insertions(+)
 create mode 100644 meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-btrfs.wks

diff --git a/meta-isar/conf/machine/qemuamd64.conf b/meta-isar/conf/machine/qemuamd64.conf
index eca2628c4ae5..6b40cb50702e 100644
--- a/meta-isar/conf/machine/qemuamd64.conf
+++ b/meta-isar/conf/machine/qemuamd64.conf
@@ -13,6 +13,11 @@ IMAGER_INSTALL += "${GRUB_BOOTLOADER_INSTALL}"
 WKS_FILE_ubuntu-focal ?= "sdimage-efi-sd"
 IMAGER_INSTALL_remove_ubuntu-focal = "${GRUB_BOOTLOADER_INSTALL}"
 
+WKS_FILE_debian-bullseye ?= "sdimage-efi-btrfs"
+IMAGER_INSTALL_remove_debian-bullseye = "${GRUB_BOOTLOADER_INSTALL}"
+IMAGER_INSTALL_append_debian-bullseye = " ${SYSTEMD_BOOTLOADER_INSTALL} btrfs-progs"
+IMAGE_PREINSTALL_append_debian-bullseye = " btrfs-progs"
+
 IMAGE_INSTALL += "sshd-regen-keys"
 
 QEMU_ARCH ?= "x86_64"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-btrfs.wks b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-btrfs.wks
new file mode 100644
index 000000000000..014ecca61f0c
--- /dev/null
+++ b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-btrfs.wks
@@ -0,0 +1,9 @@
+# short-description: Create an EFI disk image without any swap and btrfs
+# long-description: Creates a partitioned EFI disk image without any swap that
+# the user can directly dd to boot media, where the last partition is btrfs.
+
+part /boot --source bootimg-efi-isar --sourceparams "loader=systemd-boot" --ondisk sda --label efi --part-type EF00 --align 1024
+
+part / --source rootfs --ondisk sda --fstype btrfs --label platform --align 1024 --use-uuid --exclude-path boot/
+
+bootloader --ptable gpt --timeout 3 --append "rootwait console=ttyS0,115200 console=tty0"
-- 
2.37.4

[PATCH v6 4/6] CI: improve cibuilder readability

[henning.schild]

From: Henning Schild <henning.schild@siemens.com>

What was called "servive_prompt" is not a prompt nor a service. Document
what that is and rename the variable.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 testsuite/cibuilder.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index d5c83b222d50..07a9edc56d9c 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -229,7 +229,8 @@ class CIBuilder(Test):
         self.log.info('QEMU boot line: ' + str(cmdline))
 
         login_prompt = b'isar login:'
-        service_prompt = b'Just an example'
+        # the printk of recipes-kernel/example-module
+        module_output = b'Just an example'
 
         timeout = time.time() + int(time_to_wait)
 
@@ -263,7 +264,7 @@ class CIBuilder(Test):
         if os.path.exists(output_file) and os.path.getsize(output_file) > 0:
             with open(output_file, "rb") as f1:
                 data = f1.read()
-                if service_prompt in data and login_prompt in data:
+                if module_output in data and login_prompt in data:
                     return
                 else:
                     app_log.error(data.decode(errors='replace'))
-- 
2.37.4

[PATCH v6 5/6] meta-isar: install expand-on-first-boot in most images and add space

[henning.schild]

From: Henning Schild <henning.schild@siemens.com>

Here we take the most common and generic images and add some empty space
at the end of then.
This will enable better interactive use of our example images since
people will have some space to install some more packages. While the
space seems fixed it really is open end if the mass storage happens to be
bigger, because we install expand-on-first-boot as well.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/conf/machine/qemuamd64.conf                       | 1 +
 meta-isar/conf/machine/virtualbox.conf                      | 1 +
 meta-isar/conf/machine/vmware.conf                          | 1 +
 meta-isar/conf/multiconfig/qemuarm-bookworm.conf            | 1 +
 meta-isar/scripts/lib/wic/canned-wks/expand-padding.wks.inc | 5 +++++
 meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-btrfs.wks  | 2 ++
 meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks     | 2 ++
 meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks        | 2 ++
 8 files changed, 15 insertions(+)
 create mode 100644 meta-isar/scripts/lib/wic/canned-wks/expand-padding.wks.inc

diff --git a/meta-isar/conf/machine/qemuamd64.conf b/meta-isar/conf/machine/qemuamd64.conf
index 6b40cb50702e..a1be5e5b113b 100644
--- a/meta-isar/conf/machine/qemuamd64.conf
+++ b/meta-isar/conf/machine/qemuamd64.conf
@@ -19,6 +19,7 @@ IMAGER_INSTALL_append_debian-bullseye = " ${SYSTEMD_BOOTLOADER_INSTALL} btrfs-pr
 IMAGE_PREINSTALL_append_debian-bullseye = " btrfs-progs"
 
 IMAGE_INSTALL += "sshd-regen-keys"
+IMAGE_INSTALL += "expand-on-first-boot"
 
 QEMU_ARCH ?= "x86_64"
 QEMU_MACHINE ?= "q35"
diff --git a/meta-isar/conf/machine/virtualbox.conf b/meta-isar/conf/machine/virtualbox.conf
index de3d04c3c4b4..8c3e9bacbf9b 100644
--- a/meta-isar/conf/machine/virtualbox.conf
+++ b/meta-isar/conf/machine/virtualbox.conf
@@ -10,6 +10,7 @@ KERNEL_NAME ?= "amd64"
 WKS_FILE ?= "sdimage-efi"
 
 IMAGER_INSTALL += "${GRUB_BOOTLOADER_INSTALL}"
+IMAGE_INSTALL += "expand-on-first-boot"
 
 VMDK_SUBFORMAT = "monolithicSparse"
 IMAGE_FSTYPES ?= "ova"
diff --git a/meta-isar/conf/machine/vmware.conf b/meta-isar/conf/machine/vmware.conf
index fba639b19350..42503374742c 100644
--- a/meta-isar/conf/machine/vmware.conf
+++ b/meta-isar/conf/machine/vmware.conf
@@ -12,6 +12,7 @@ WKS_FILE ?= "sdimage-efi"
 IMAGER_INSTALL += "${GRUB_BOOTLOADER_INSTALL}"
 
 OVF_TEMPLATE_FILE ?= "vm-img-vmware.ovf.tmpl"
+IMAGE_INSTALL += "expand-on-first-boot"
 
 VMDK_SUBFORMAT = "streamOptimized"
 IMAGE_FSTYPES ?= "ova"
diff --git a/meta-isar/conf/multiconfig/qemuarm-bookworm.conf b/meta-isar/conf/multiconfig/qemuarm-bookworm.conf
index 126935b06fda..0b9ecb664545 100644
--- a/meta-isar/conf/multiconfig/qemuarm-bookworm.conf
+++ b/meta-isar/conf/multiconfig/qemuarm-bookworm.conf
@@ -7,3 +7,4 @@ IMAGE_FSTYPES_append = " wic"
 WKS_FILE ?= "sdimage-efi-sd"
 
 IMAGER_INSTALL += "${SYSTEMD_BOOTLOADER_INSTALL}"
+IMAGE_INSTALL += "expand-on-first-boot"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/expand-padding.wks.inc b/meta-isar/scripts/lib/wic/canned-wks/expand-padding.wks.inc
new file mode 100644
index 000000000000..701fba764176
--- /dev/null
+++ b/meta-isar/scripts/lib/wic/canned-wks/expand-padding.wks.inc
@@ -0,0 +1,5 @@
+# some extra space we put in the end after the last partition
+# that allows testing expand-on-first-boot while at the same time
+# making our images more useful because they have space to play around
+
+part --source empty --no-table --ondisk sda --size 256M
diff --git a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-btrfs.wks b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-btrfs.wks
index 014ecca61f0c..e22dba2f83f8 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-btrfs.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-btrfs.wks
@@ -6,4 +6,6 @@ part /boot --source bootimg-efi-isar --sourceparams "loader=systemd-boot" --ondi
 
 part / --source rootfs --ondisk sda --fstype btrfs --label platform --align 1024 --use-uuid --exclude-path boot/
 
+include expand-padding.wks.inc
+
 bootloader --ptable gpt --timeout 3 --append "rootwait console=ttyS0,115200 console=tty0"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks
index 754fbc46f8e4..7fe2953f5ff2 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks
@@ -6,4 +6,6 @@ part /boot --source bootimg-efi-isar --sourceparams "loader=systemd-boot" --ondi
 
 part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/
 
+include expand-padding.wks.inc
+
 bootloader --ptable gpt --timeout 3 --append "rootwait console=ttyS0,115200 console=tty0"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
index f3addbc7515d..80c3a5359b1f 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
@@ -6,4 +6,6 @@ part /boot --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk s
 
 part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/
 
+include expand-padding.wks.inc
+
 bootloader --ptable gpt --timeout 3 --append "rootwait console=ttyS0,115200 console=tty0"
-- 
2.37.4

[PATCH v6 6/6] CI: expect a message about filesystem resize vom expand script

[henning.schild]

From: Henning Schild <henning.schild@siemens.com>

We have added the expand-on-first-boot recipe and some space to grow to
our example images. So now any image using wic should "expand" on its
first boot. And the kernel actually leaves a message for us to read, at
least on debian but not on ubuntu.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 testsuite/cibuilder.py | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 07a9edc56d9c..8b67815442a8 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -231,6 +231,20 @@ class CIBuilder(Test):
         login_prompt = b'isar login:'
         # the printk of recipes-kernel/example-module
         module_output = b'Just an example'
+        resize_output = None
+
+        bb_output = start_vm.get_bitbake_env(arch, distro).decode()
+        wks_file = start_vm.get_bitbake_var(bb_output, 'WKS_FILE')
+        if wks_file:
+            bbdistro = start_vm.get_bitbake_var(bb_output, 'DISTRO')
+            # ubuntu is less verbose so we do not see the message
+            # /etc/sysctl.d/10-console-messages.conf
+            if bbdistro and "ubuntu" not in bbdistro:
+                if "sdimage-efi-sd" in wks_file:
+                    # output we see when expand-on-first-boot runs on ext4
+                    resize_output = b'resized filesystem to'
+                if "sdimage-efi-btrfs" in wks_file:
+                    resize_output = b': resize device '
 
         timeout = time.time() + int(time_to_wait)
 
@@ -265,8 +279,11 @@ class CIBuilder(Test):
             with open(output_file, "rb") as f1:
                 data = f1.read()
                 if module_output in data and login_prompt in data:
-                    return
-                else:
-                    app_log.error(data.decode(errors='replace'))
+                    if resize_output:
+                        if resize_output in data:
+                            return
+                    else:
+                        return
+                app_log.error(data.decode(errors='replace'))
 
         self.fail('Log ' + output_file)
-- 
2.37.4

Re: [PATCH v6 0/6] expand-on-first-boot btrfs and CI testing

[Henning Schild]

In case we keep seeing more trolling on that one i think we should only
consider the CI bits and drop btrfs.

btrfs has been proposed by two downstream users who already have their
own patches and might not care too much about upstream.

I am trying hard to cater for them. But one might not care too much,
while the other should likely be ignored because the demand and all the
reviews seem more like an act of sabotage than actually wanting to help.

Please review this if you care about this in a truly relevant use-case.
Please stop reviewing this if you are only looking into Isar as a hobby
or revenge for something that went south for you.

Henning

Am Thu, 15 Dec 2022 23:16:00 +0100
schrieb henning.schild@siemens.com:

> From: Henning Schild <henning.schild@siemens.com>
> 
> changes since v5:
>  - drop dead code around mktemp subshell
>  - revert cosmetic mktemp stuff back to v3
>  - remove "Suggested-by" tag to hopefully improve signal to noise
> ratio in review process
> 
> changes since v4:
>  - cosmetic fix in the way we call mktemp
>  - add a patch that makes sure the service is run only once, also on
>    error
>  - bump package version after changes are done
> 
> changes since v3:
>  - rebased on "expand-on-first-boot: switch back away from
> systemd-growfs"
>  - implement btrfs resizing and CI testing it
>  - put the padding wks entry into an include
> 
> changes since v2:
>  - add python namespace prefix for the bitbake get functions
>  - install only in images that use WKS_FILEs that have the "gap"
> 
> changes since v1:
>  - complete rewrite to not focus only on the tests but rather the
>    feature for all examples
> 
> This first was only about testing but meanwhile has a focus on
> enabling the expand feature in all example images anyhow, and later
> look for traces of that happening in CI test where we can.
> We install the application in any image and in the wic images add some
> scratch space at the end. That will enable improved interactive use of
> all those wic images, so makes sense apart from any testing.
> Later use the log reading feature of the qemu tests to look for traces
> of successful resizing for non ubuntu images. In ubuntu the
> console_loglevel is not verbose enough to find traces.
> 
> Henning Schild (6):
>   expand-on-first-boot: really only do that once, especially on
> failure expand-on-first-boot: support resizing a btrfs
>   meta-isar: introduce an example to use btrfs
>   CI: improve cibuilder readability
>   meta-isar: install expand-on-first-boot in most images and add space
>   CI: expect a message about filesystem resize vom expand script
> 
>  meta-isar/conf/machine/qemuamd64.conf         |  6 ++++
>  meta-isar/conf/machine/virtualbox.conf        |  1 +
>  meta-isar/conf/machine/vmware.conf            |  1 +
>  .../conf/multiconfig/qemuarm-bookworm.conf    |  1 +
>  .../lib/wic/canned-wks/expand-padding.wks.inc |  5 +++
>  .../lib/wic/canned-wks/sdimage-efi-btrfs.wks  | 11 +++++++
>  .../lib/wic/canned-wks/sdimage-efi-sd.wks     |  2 ++
>  .../lib/wic/canned-wks/sdimage-efi.wks        |  2 ++
>  ...oot_1.3.bb => expand-on-first-boot_1.4.bb} |  0
>  .../files/expand-last-partition.sh            | 31
> ++++++++++++++++--- .../files/expand-on-first-boot.service        |
> 1 + testsuite/cibuilder.py                        | 28
> ++++++++++++++--- 12 files changed, 80 insertions(+), 9 deletions(-)
>  create mode 100644
> meta-isar/scripts/lib/wic/canned-wks/expand-padding.wks.inc create
> mode 100644
> meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-btrfs.wks rename
> meta/recipes-support/expand-on-first-boot/{expand-on-first-boot_1.3.bb
> => expand-on-first-boot_1.4.bb} (100%)
> 

Re: [PATCH v6 0/6] expand-on-first-boot btrfs and CI testing

[Moessbauer, Felix]

Hi,

I tested this on an Siemens IoT2050 board and can confirm that it
properly works.

Acked-by: Felix Moessbauer <felix.moessbauer@siemens.com>

Felix

On Thu, 2022-12-15 at 23:16 +0100, henning.schild@siemens.com wrote:
> From: Henning Schild <henning.schild@siemens.com>
> 
> changes since v5:
>  - drop dead code around mktemp subshell
>  - revert cosmetic mktemp stuff back to v3
>  - remove "Suggested-by" tag to hopefully improve signal to noise
> ratio
>    in review process
> 
> changes since v4:
>  - cosmetic fix in the way we call mktemp
>  - add a patch that makes sure the service is run only once, also on
>    error
>  - bump package version after changes are done
> 
> changes since v3:
>  - rebased on "expand-on-first-boot: switch back away from systemd-
> growfs"
>  - implement btrfs resizing and CI testing it
>  - put the padding wks entry into an include
> 
> changes since v2:
>  - add python namespace prefix for the bitbake get functions
>  - install only in images that use WKS_FILEs that have the "gap"
> 
> changes since v1:
>  - complete rewrite to not focus only on the tests but rather the
>    feature for all examples
> 
> This first was only about testing but meanwhile has a focus on
> enabling
> the expand feature in all example images anyhow, and later look for
> traces of that happening in CI test where we can.
> We install the application in any image and in the wic images add
> some
> scratch space at the end. That will enable improved interactive use
> of
> all those wic images, so makes sense apart from any testing.
> Later use the log reading feature of the qemu tests to look for
> traces
> of successful resizing for non ubuntu images. In ubuntu the
> console_loglevel is not verbose enough to find traces.
> 
> Henning Schild (6):
>   expand-on-first-boot: really only do that once, especially on
> failure
>   expand-on-first-boot: support resizing a btrfs
>   meta-isar: introduce an example to use btrfs
>   CI: improve cibuilder readability
>   meta-isar: install expand-on-first-boot in most images and add
> space
>   CI: expect a message about filesystem resize vom expand script
> 
>  meta-isar/conf/machine/qemuamd64.conf         |  6 ++++
>  meta-isar/conf/machine/virtualbox.conf        |  1 +
>  meta-isar/conf/machine/vmware.conf            |  1 +
>  .../conf/multiconfig/qemuarm-bookworm.conf    |  1 +
>  .../lib/wic/canned-wks/expand-padding.wks.inc |  5 +++
>  .../lib/wic/canned-wks/sdimage-efi-btrfs.wks  | 11 +++++++
>  .../lib/wic/canned-wks/sdimage-efi-sd.wks     |  2 ++
>  .../lib/wic/canned-wks/sdimage-efi.wks        |  2 ++
>  ...oot_1.3.bb => expand-on-first-boot_1.4.bb} |  0
>  .../files/expand-last-partition.sh            | 31 ++++++++++++++++-
> --
>  .../files/expand-on-first-boot.service        |  1 +
>  testsuite/cibuilder.py                        | 28 ++++++++++++++---
>  12 files changed, 80 insertions(+), 9 deletions(-)
>  create mode 100644 meta-isar/scripts/lib/wic/canned-wks/expand-
> padding.wks.inc
>  create mode 100644 meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-
> btrfs.wks
>  rename meta/recipes-support/expand-on-first-boot/{expand-on-first-
> boot_1.3.bb => expand-on-first-boot_1.4.bb} (100%)
> 
> -- 
> 2.37.4
> 

Re: [PATCH v6 0/6] expand-on-first-boot btrfs and CI testing

[Anton Mikanovich]

16.12.2022 01:16, henning.schild@siemens.com wrote:
> From: Henning Schild <henning.schild@siemens.com>
>
> changes since v5:
>   - drop dead code around mktemp subshell
>   - revert cosmetic mktemp stuff back to v3
>   - remove "Suggested-by" tag to hopefully improve signal to noise ratio
>     in review process
>
> changes since v4:
>   - cosmetic fix in the way we call mktemp
>   - add a patch that makes sure the service is run only once, also on
>     error
>   - bump package version after changes are done
>
> changes since v3:
>   - rebased on "expand-on-first-boot: switch back away from systemd-growfs"
>   - implement btrfs resizing and CI testing it
>   - put the padding wks entry into an include
>
> changes since v2:
>   - add python namespace prefix for the bitbake get functions
>   - install only in images that use WKS_FILEs that have the "gap"
>
> changes since v1:
>   - complete rewrite to not focus only on the tests but rather the
>     feature for all examples
>
> This first was only about testing but meanwhile has a focus on enabling
> the expand feature in all example images anyhow, and later look for
> traces of that happening in CI test where we can.
> We install the application in any image and in the wic images add some
> scratch space at the end. That will enable improved interactive use of
> all those wic images, so makes sense apart from any testing.
> Later use the log reading feature of the qemu tests to look for traces
> of successful resizing for non ubuntu images. In ubuntu the
> console_loglevel is not verbose enough to find traces.
>
> Henning Schild (6):
>    expand-on-first-boot: really only do that once, especially on failure
>    expand-on-first-boot: support resizing a btrfs
>    meta-isar: introduce an example to use btrfs
>    CI: improve cibuilder readability
>    meta-isar: install expand-on-first-boot in most images and add space
>    CI: expect a message about filesystem resize vom expand script
>
>   meta-isar/conf/machine/qemuamd64.conf         |  6 ++++
>   meta-isar/conf/machine/virtualbox.conf        |  1 +
>   meta-isar/conf/machine/vmware.conf            |  1 +
>   .../conf/multiconfig/qemuarm-bookworm.conf    |  1 +
>   .../lib/wic/canned-wks/expand-padding.wks.inc |  5 +++
>   .../lib/wic/canned-wks/sdimage-efi-btrfs.wks  | 11 +++++++
>   .../lib/wic/canned-wks/sdimage-efi-sd.wks     |  2 ++
>   .../lib/wic/canned-wks/sdimage-efi.wks        |  2 ++
>   ...oot_1.3.bb => expand-on-first-boot_1.4.bb} |  0
>   .../files/expand-last-partition.sh            | 31 ++++++++++++++++---
>   .../files/expand-on-first-boot.service        |  1 +
>   testsuite/cibuilder.py                        | 28 ++++++++++++++---
>   12 files changed, 80 insertions(+), 9 deletions(-)
>   create mode 100644 meta-isar/scripts/lib/wic/canned-wks/expand-padding.wks.inc
>   create mode 100644 meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-btrfs.wks
>   rename meta/recipes-support/expand-on-first-boot/{expand-on-first-boot_1.3.bb => expand-on-first-boot_1.4.bb} (100%)
>
Hello Henning,

This patchset look really good, but still fails on arm-bookworm startvm 
test of
our fast CI. The issue can be reproduced by just building qemuarm-bookworm
target with bitbake (even outside testsuite) and then execute
VmBootTestFast.test_arm_bookworm test case.

So my proposal is to apply revert expand commit now without waiting for this
patchset to be ready.

Re: [PATCH v6 0/6] expand-on-first-boot btrfs and CI testing

[Henning Schild]

Am Tue, 20 Dec 2022 22:36:03 +0300
schrieb Anton Mikanovich <amikan@ilbers.de>:

> 16.12.2022 01:16, henning.schild@siemens.com wrote:
> > From: Henning Schild <henning.schild@siemens.com>
> >
> > changes since v5:
> >   - drop dead code around mktemp subshell
> >   - revert cosmetic mktemp stuff back to v3
> >   - remove "Suggested-by" tag to hopefully improve signal to noise
> > ratio in review process
> >
> > changes since v4:
> >   - cosmetic fix in the way we call mktemp
> >   - add a patch that makes sure the service is run only once, also
> > on error
> >   - bump package version after changes are done
> >
> > changes since v3:
> >   - rebased on "expand-on-first-boot: switch back away from
> > systemd-growfs"
> >   - implement btrfs resizing and CI testing it
> >   - put the padding wks entry into an include
> >
> > changes since v2:
> >   - add python namespace prefix for the bitbake get functions
> >   - install only in images that use WKS_FILEs that have the "gap"
> >
> > changes since v1:
> >   - complete rewrite to not focus only on the tests but rather the
> >     feature for all examples
> >
> > This first was only about testing but meanwhile has a focus on
> > enabling the expand feature in all example images anyhow, and later
> > look for traces of that happening in CI test where we can.
> > We install the application in any image and in the wic images add
> > some scratch space at the end. That will enable improved
> > interactive use of all those wic images, so makes sense apart from
> > any testing. Later use the log reading feature of the qemu tests to
> > look for traces of successful resizing for non ubuntu images. In
> > ubuntu the console_loglevel is not verbose enough to find traces.
> >
> > Henning Schild (6):
> >    expand-on-first-boot: really only do that once, especially on
> > failure expand-on-first-boot: support resizing a btrfs
> >    meta-isar: introduce an example to use btrfs
> >    CI: improve cibuilder readability
> >    meta-isar: install expand-on-first-boot in most images and add
> > space CI: expect a message about filesystem resize vom expand script
> >
> >   meta-isar/conf/machine/qemuamd64.conf         |  6 ++++
> >   meta-isar/conf/machine/virtualbox.conf        |  1 +
> >   meta-isar/conf/machine/vmware.conf            |  1 +
> >   .../conf/multiconfig/qemuarm-bookworm.conf    |  1 +
> >   .../lib/wic/canned-wks/expand-padding.wks.inc |  5 +++
> >   .../lib/wic/canned-wks/sdimage-efi-btrfs.wks  | 11 +++++++
> >   .../lib/wic/canned-wks/sdimage-efi-sd.wks     |  2 ++
> >   .../lib/wic/canned-wks/sdimage-efi.wks        |  2 ++
> >   ...oot_1.3.bb => expand-on-first-boot_1.4.bb} |  0
> >   .../files/expand-last-partition.sh            | 31
> > ++++++++++++++++--- .../files/expand-on-first-boot.service        |
> >  1 + testsuite/cibuilder.py                        | 28
> > ++++++++++++++--- 12 files changed, 80 insertions(+), 9 deletions(-)
> >   create mode 100644
> > meta-isar/scripts/lib/wic/canned-wks/expand-padding.wks.inc create
> > mode 100644
> > meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-btrfs.wks rename
> > meta/recipes-support/expand-on-first-boot/{expand-on-first-boot_1.3.bb
> > => expand-on-first-boot_1.4.bb} (100%) 
> Hello Henning,
> 
> This patchset look really good, but still fails on arm-bookworm
> startvm test of
> our fast CI. The issue can be reproduced by just building
> qemuarm-bookworm target with bitbake (even outside testsuite) and
> then execute VmBootTestFast.test_arm_bookworm test case.

I saw that when i ran ilbers CI once. Was hoping it to be unrelated and
maybe a known issue. But if you confirm that these patches introduced
it, i will have a look

> So my proposal is to apply revert expand commit now without waiting
> for this patchset to be ready.

Sure. The "revert" patch is important. This stuff can wait.

Henning
 

Re: [PATCH v6 0/6] expand-on-first-boot btrfs and CI testing

[Henning Schild]

Am Wed, 21 Dec 2022 09:22:21 +0100
schrieb Henning Schild <henning.schild@siemens.com>:

> Am Tue, 20 Dec 2022 22:36:03 +0300
> schrieb Anton Mikanovich <amikan@ilbers.de>:
> 
> > 16.12.2022 01:16, henning.schild@siemens.com wrote:
> > > From: Henning Schild <henning.schild@siemens.com>
> > >
> > > changes since v5:
> > >   - drop dead code around mktemp subshell
> > >   - revert cosmetic mktemp stuff back to v3
> > >   - remove "Suggested-by" tag to hopefully improve signal to noise
> > > ratio in review process
> > >
> > > changes since v4:
> > >   - cosmetic fix in the way we call mktemp
> > >   - add a patch that makes sure the service is run only once, also
> > > on error
> > >   - bump package version after changes are done
> > >
> > > changes since v3:
> > >   - rebased on "expand-on-first-boot: switch back away from
> > > systemd-growfs"
> > >   - implement btrfs resizing and CI testing it
> > >   - put the padding wks entry into an include
> > >
> > > changes since v2:
> > >   - add python namespace prefix for the bitbake get functions
> > >   - install only in images that use WKS_FILEs that have the "gap"
> > >
> > > changes since v1:
> > >   - complete rewrite to not focus only on the tests but rather the
> > >     feature for all examples
> > >
> > > This first was only about testing but meanwhile has a focus on
> > > enabling the expand feature in all example images anyhow, and
> > > later look for traces of that happening in CI test where we can.
> > > We install the application in any image and in the wic images add
> > > some scratch space at the end. That will enable improved
> > > interactive use of all those wic images, so makes sense apart from
> > > any testing. Later use the log reading feature of the qemu tests
> > > to look for traces of successful resizing for non ubuntu images.
> > > In ubuntu the console_loglevel is not verbose enough to find
> > > traces.
> > >
> > > Henning Schild (6):
> > >    expand-on-first-boot: really only do that once, especially on
> > > failure expand-on-first-boot: support resizing a btrfs
> > >    meta-isar: introduce an example to use btrfs
> > >    CI: improve cibuilder readability
> > >    meta-isar: install expand-on-first-boot in most images and add
> > > space CI: expect a message about filesystem resize vom expand
> > > script
> > >
> > >   meta-isar/conf/machine/qemuamd64.conf         |  6 ++++
> > >   meta-isar/conf/machine/virtualbox.conf        |  1 +
> > >   meta-isar/conf/machine/vmware.conf            |  1 +
> > >   .../conf/multiconfig/qemuarm-bookworm.conf    |  1 +
> > >   .../lib/wic/canned-wks/expand-padding.wks.inc |  5 +++
> > >   .../lib/wic/canned-wks/sdimage-efi-btrfs.wks  | 11 +++++++
> > >   .../lib/wic/canned-wks/sdimage-efi-sd.wks     |  2 ++
> > >   .../lib/wic/canned-wks/sdimage-efi.wks        |  2 ++
> > >   ...oot_1.3.bb => expand-on-first-boot_1.4.bb} |  0
> > >   .../files/expand-last-partition.sh            | 31
> > > ++++++++++++++++--- .../files/expand-on-first-boot.service
> > > | 1 + testsuite/cibuilder.py                        | 28
> > > ++++++++++++++--- 12 files changed, 80 insertions(+), 9
> > > deletions(-) create mode 100644
> > > meta-isar/scripts/lib/wic/canned-wks/expand-padding.wks.inc create
> > > mode 100644
> > > meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-btrfs.wks rename
> > > meta/recipes-support/expand-on-first-boot/{expand-on-first-boot_1.3.bb
> > > => expand-on-first-boot_1.4.bb} (100%) 
> > Hello Henning,
> > 
> > This patchset look really good, but still fails on arm-bookworm
> > startvm test of
> > our fast CI. The issue can be reproduced by just building
> > qemuarm-bookworm target with bitbake (even outside testsuite) and
> > then execute VmBootTestFast.test_arm_bookworm test case.
> 
> I saw that when i ran ilbers CI once. Was hoping it to be unrelated
> and maybe a known issue. But if you confirm that these patches
> introduced it, i will have a look

Found it, reason is that i added test code which expects resize traces
in logs when a target used a specific WKS_FILE.

Now that qemuarm-bookworm has
 IMAGE_FSTYPES="ext4 wic"

And the CI test does test [0] so it does test ext4 and not wic, in
which case we really can not find resize traces.

That was all kind of introduced by
8b17d91cc758d173aca9d97961208a17a95f71a2

I will change that last commit to also check IMAGE_FSTYPES and match
[0] to be wic

Henning

> > So my proposal is to apply revert expand commit now without waiting
> > for this patchset to be ready.
> 
> Sure. The "revert" patch is important. This stuff can wait.
> 
> Henning
>  
>