[PATCH v2 0/4] wic: Update to the latest revision

[PATCH v2 0/4] wic: Update to the latest revision

[Felix Moessbauer]

Changes since v1:

- add example how to use splitted rootfs
- generate such an image in the ci
- test if the partition layout looks like what we expect

Best regards,
Felix

Felix Moessbauer (4):
  wic: Update to the latest revision
  meta-isar: add example how to split rootfs across partitions
  use multipart image for qemuamd64-bookworm target
  ci: test partition layout of splitted-rootfs image

 .../conf/multiconfig/qemuamd64-bookworm.conf  |  1 +
 .../lib/wic/canned-wks/multipart-efi.wks      |  8 +++
 scripts/lib/wic/misc.py                       |  1 +
 scripts/lib/wic/plugins/imager/direct.py      |  2 +-
 scripts/lib/wic/plugins/source/rootfs.py      |  5 +-
 testsuite/cibase.py                           | 58 +++++++++++++++++++
 testsuite/cibuilder.py                        |  6 +-
 testsuite/citest.py                           | 14 +++++
 8 files changed, 92 insertions(+), 3 deletions(-)
 create mode 100644 meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks

-- 
2.30.2

[PATCH v2 1/4] wic: Update to the latest revision

[Felix Moessbauer]

Update to the latest wic from OE-core.

OE-core Revision: b8878cf0d35cf3d1ac30576d9b9943a7761c011b

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 scripts/lib/wic/misc.py                  | 1 +
 scripts/lib/wic/plugins/imager/direct.py | 2 +-
 scripts/lib/wic/plugins/source/rootfs.py | 5 ++++-
 3 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py
index 3e118229..3bc165fd 100644
--- a/scripts/lib/wic/misc.py
+++ b/scripts/lib/wic/misc.py
@@ -36,6 +36,7 @@ NATIVE_RECIPES = {"bmaptool": "bmap-tools",
                   "mkdosfs": "dosfstools",
                   "mkisofs": "cdrtools",
                   "mkfs.btrfs": "btrfs-tools",
+                  "mkfs.erofs": "erofs-utils",
                   "mkfs.ext2": "e2fsprogs",
                   "mkfs.ext3": "e2fsprogs",
                   "mkfs.ext4": "e2fsprogs",
diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index 35fff7c1..4d0b836e 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -259,7 +259,7 @@ class DirectPlugin(ImagerPlugin):
             if part.mountpoint == "/":
                 if part.uuid:
                     return "PARTUUID=%s" % part.uuid
-                elif part.label:
+                elif part.label and self.ptable_format != 'msdos':
                     return "PARTLABEL=%s" % part.label
                 else:
                     suffix = 'p' if part.disk.startswith('mmcblk') else ''
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 2e34e715..25bb41dd 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -50,7 +50,7 @@ class RootfsPlugin(SourcePlugin):
 
     @staticmethod
     def __get_rootfs_dir(rootfs_dir):
-        if os.path.isdir(rootfs_dir):
+        if rootfs_dir and os.path.isdir(rootfs_dir):
             return os.path.realpath(rootfs_dir)
 
         image_rootfs_dir = get_bitbake_var("IMAGE_ROOTFS", rootfs_dir)
@@ -96,6 +96,9 @@ class RootfsPlugin(SourcePlugin):
         part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
         part.has_fstab = os.path.exists(os.path.join(part.rootfs_dir, "etc/fstab"))
         pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo")
+        if not os.path.lexists(pseudo_dir):
+            pseudo_dir = os.path.join(cls.__get_rootfs_dir(None), '../pseudo')
+
         if not os.path.lexists(pseudo_dir):
             logger.warn("%s folder does not exist. "
                         "Usernames and permissions will be invalid " % pseudo_dir)
-- 
2.30.2

[PATCH v2 2/4] meta-isar: add example how to split rootfs across partitions

[Felix Moessbauer]

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks | 8 ++++++++
 1 file changed, 8 insertions(+)
 create mode 100644 meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks

diff --git a/meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks b/meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks
new file mode 100644
index 00000000..c8d876ae
--- /dev/null
+++ b/meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks
@@ -0,0 +1,8 @@
+# Example showing how to split a single rootfs across partitions
+part /boot --ondisk sda --source bootimg-efi-isar --sourceparams="loader=grub-efi" --label boot --active --align 1024
+# when excluding paths, just specify --exclude-path once and separate paths using spaces
+part /     --ondisk sda --source rootfs --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --exclude-path boot home --extra-space 1G
+# put home last to support auto-expand of this partition
+part /home --ondisk sda --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/home --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --exclude-path boot home
+
+bootloader --ptable gpt --timeout 3 --append "rootwait console=ttyS0,115200 console=tty0"
-- 
2.30.2

[PATCH v2 3/4] use multipart image for qemuamd64-bookworm target

[Felix Moessbauer]

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta-isar/conf/multiconfig/qemuamd64-bookworm.conf | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta-isar/conf/multiconfig/qemuamd64-bookworm.conf b/meta-isar/conf/multiconfig/qemuamd64-bookworm.conf
index a5ad814f..591918de 100644
--- a/meta-isar/conf/multiconfig/qemuamd64-bookworm.conf
+++ b/meta-isar/conf/multiconfig/qemuamd64-bookworm.conf
@@ -2,3 +2,4 @@
 
 MACHINE ?= "qemuamd64"
 DISTRO ?= "debian-bookworm"
+WKS_FILE ?= "multipart-efi.wks"
-- 
2.30.2

[PATCH v2 4/4] ci: test partition layout of splitted-rootfs image

[Felix Moessbauer]

This adds a test that checks if splitting the rootfs
across multiple partitions correctly works.

The following is checked:

file-permissions (based on wic internal check):
WIC already has a built-in check for correct
file permissions. In case the permission db is missing,
a WIC warning is emitted. We simply check for this warning.

partition layout and excluded paths:
Specifying the path exclusions is error prone.
Hence, we check that all generated partitions do not contain
excluded paths. Further, we check that expected path are there.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 testsuite/cibase.py    | 58 ++++++++++++++++++++++++++++++++++++++++++
 testsuite/cibuilder.py |  6 ++++-
 testsuite/citest.py    | 14 ++++++++++
 3 files changed, 77 insertions(+), 1 deletion(-)

diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index b25c356e..fd272ef5 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -164,3 +164,61 @@ class CIBaseTest(CIBuilder):
                     ['do_rootfs_install_setscene', '!do_rootfs_install'])
             ]):
             self.fail("Failed rebuild package and image")
+
+    def perform_partition_layout_test(self, mc_target, **kwargs):
+        def _mount(device, mountpoint):
+            process.run(f'mount -o ro {device} {mountpoint}', sudo=True, ignore_status=False)
+
+        def _umount(mountpoint):
+            process.run(f'umount -l {mountpoint}', sudo=True, ignore_status=True)
+
+        def check_output_for_warnings(target, machine):
+            wic_output_files = glob.glob(f'{self.build_dir}/tmp/work/*/{target}-{machine}/*/temp/log.do_image_wic')
+            if len(wic_output_files) == 0:
+                self.fail('could not find WIC output file')
+            with open(wic_output_files[0], 'r') as output:
+                # in case we had permission problems, a WIC warning is in the logs
+                for line in output:
+                    token = line.split()
+                    if len(token) > 0 and token[0] == 'WARNING:':
+                        self.fail(f'WIC issue found: {line}')
+
+        def check_part_layout(target, machine):
+            pattern = f'{self.build_dir}/tmp/deploy/images/{machine}/{target}-*-{machine}.wic.p*'
+            partitions = sorted(glob.glob(pattern))
+            if len(partitions) != 3:
+                self.fail(f'expected 3 partitions, but got {len(partitions)} (in {pattern})')
+            mountpoints = [os.path.join(self.build_dir, mp) for mp in 'mnt_efi mnt_root mnt_home'.split()]
+            for i in range(3):
+                os.makedirs(mountpoints[i], exist_ok=True)
+                _umount(mountpoints[i])
+                _mount(partitions[i], mountpoints[i])
+
+            # in boot partition, we expect /boot
+            if not os.path.isdir(os.path.join(mountpoints[0], 'EFI')):
+                self.fail('boot partition does not provide /EFI')
+            # in root partition, boot and home should be excluded
+            if not os.path.isdir(os.path.join(mountpoints[1], 'etc')) or \
+               os.path.isdir(os.path.join(mountpoints[1], 'home')) or \
+               os.path.isdir(os.path.join(mountpoints[1], 'boot')):
+                self.fail('root partition does not contain expected dirs')
+            # home partition should contain home of user "user"
+            if not os.path.isdir(os.path.join(mountpoints[2], 'user')):
+                self.fail('home partition does not contain home of user')
+
+            [_umount(mnt) for mnt in mountpoints]
+
+        mc_spec = mc_target.split(':')
+        target = mc_spec[2]
+        machine = mc_spec[1].split('-')[0]
+
+        self.configure(
+            sstate=False,
+            compat_arch=False,
+            interactive_user=True,
+            **kwargs)
+
+        self.bitbake(mc_target, **kwargs)
+
+        check_output_for_warnings(target, machine)
+        check_part_layout(target, machine)
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index bc48d47f..e3fbb859 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -54,7 +54,7 @@ class CIBuilder(Test):
 
     def configure(self, compat_arch=True, cross=None, debsrc_cache=False,
                   container=False, ccache=False, sstate=False, offline=False,
-                  gpg_pub_key=None, **kwargs):
+                  gpg_pub_key=None, interactive_user=False, **kwargs):
         # write configuration file and set bitbake_args
         # can run multiple times per test case
         self.check_init()
@@ -107,6 +107,10 @@ class CIBuilder(Test):
                 f.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
             if distro_apt_premir:
                 f.write('DISTRO_APT_PREMIRRORS = "%s"\n' % distro_apt_premir)
+            if interactive_user:
+                f.write('USERS += "user"\n')
+                f.write('USER_user[home] = "/home/user"\n')
+                f.write('USER_user[flags] = "create-home"\n')
 
         # include ci_build.conf in local.conf
         with open(self.build_dir + '/conf/local.conf', 'r+') as f:
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 8d00054a..26101010 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -318,3 +318,17 @@ class VmBootTestFull(CIBaseTest):
     def test_amd64_focal(self):
         self.init()
         self.vm_start('amd64','focal')
+
+class SplittedRootfsTest(CIBaseTest):
+
+    """
+    Test partition layout of splitted rootfs image
+
+    :avocado: tags=splittedrootfs,full
+    """
+
+    def test_wic_partition_layout(self):
+        mc_target = 'mc:qemuamd64-bookworm:isar-image-base'
+
+        self.init('build-wic-rootfs')
+        self.perform_partition_layout_test(mc_target)
-- 
2.30.2

Re: [PATCH v2 0/4] wic: Update to the latest revision

[Henning Schild]

Cool, i really like that in depth testing.

@Tobias this has an example with a non-rootfs last partition that your
growfs bits could also be tested against, also a chance to maybe
"simply" making that home btrfs to show how growfs can hopefully handle
that.

Others stay tunes, Tobias is currently looking into systemd growfs and
repart.

Henning

Am Thu, 12 May 2022 11:45:20 +0200
schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:

> Changes since v1:
> 
> - add example how to use splitted rootfs
> - generate such an image in the ci
> - test if the partition layout looks like what we expect
> 
> Best regards,
> Felix
> 
> Felix Moessbauer (4):
>   wic: Update to the latest revision
>   meta-isar: add example how to split rootfs across partitions
>   use multipart image for qemuamd64-bookworm target
>   ci: test partition layout of splitted-rootfs image
> 
>  .../conf/multiconfig/qemuamd64-bookworm.conf  |  1 +
>  .../lib/wic/canned-wks/multipart-efi.wks      |  8 +++
>  scripts/lib/wic/misc.py                       |  1 +
>  scripts/lib/wic/plugins/imager/direct.py      |  2 +-
>  scripts/lib/wic/plugins/source/rootfs.py      |  5 +-
>  testsuite/cibase.py                           | 58
> +++++++++++++++++++ testsuite/cibuilder.py                        |
> 6 +- testsuite/citest.py                           | 14 +++++
>  8 files changed, 92 insertions(+), 3 deletions(-)
>  create mode 100644
> meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks
> 

Re: [PATCH v2 2/4] meta-isar: add example how to split rootfs across partitions

[Bezdeka, Florian]

On Thu, 2022-05-12 at 11:45 +0200, Felix Moessbauer wrote:
> +# put home last to support auto-expand of this partition
> +part /home --ondisk sda --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/home --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --exclude-path boot home

"home" is part of the --exclude-path for /home? Is that really right?

How are the paths given in --exclude-path interpreted when --rootfs-dir
is given? Relative to --rotfs-dir? Maybe we can simply remove 
--exclude-path?

RE: [PATCH v2 2/4] meta-isar: add example how to split rootfs across partitions

[Moessbauer, Felix]

> From: Bezdeka, Florian (T CED SES-DE) <florian.bezdeka@siemens.com>
> Sent: Thursday, May 12, 2022 12:55 PM
> To: isar-users@googlegroups.com; Moessbauer, Felix (T CED SES-DE)
> <felix.moessbauer@siemens.com>
> Cc: Kiszka, Jan (T CED) <jan.kiszka@siemens.com>; Schild, Henning (T CED SES-
> DE) <henning.schild@siemens.com>
> Subject: Re: [PATCH v2 2/4] meta-isar: add example how to split rootfs across
> partitions
> 
> On Thu, 2022-05-12 at 11:45 +0200, Felix Moessbauer wrote:
> > +# put home last to support auto-expand of this partition part /home
> > +--ondisk sda --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/home
> > +--fstype=ext4 --mkfs-extraopts "-T default" --label platform --align
> > +1024 --exclude-path boot home
> 
> "home" is part of the --exclude-path for /home? Is that really right?

Good catch. On /home, we don't need the exclude path.
Will send a v3.

> 
> How are the paths given in --exclude-path interpreted when --rootfs-dir is given?
> Relative to --rotfs-dir? Maybe we can simply remove --exclude-path?
> 

The exclude paths are relative to the --rootfs-dir. These have to be specified as relative dirs and if the path ends in '/', that means the path is created but all child paths are excluded.
A common mistake I saw in downstream layers is that --rootfs-dir was specified multiple times, providing one path each.
This just excludes a single path (ATM I don't even know which one) but copies all others into the partition.

Felix