[PATCH v4 0/5] wic: Update to the latest revision

[PATCH v4 0/5] wic: Update to the latest revision

[Felix Moessbauer]

Changes since v3:

- improved mount / unmount logic in partition test
- upload test logs in ci

Changes since v2:

- removed redundant paths exclusion for home (reported by Florian Bezdeka, thanks)
- improved commit messages

Changes since v1:

- add example how to use splitted rootfs
- generate such an image in the ci
- test if the partition layout looks like what we expect

Best regards,
Felix

Felix Moessbauer (5):
  wic: Update to the latest revision
  meta-isar: add example how to split rootfs across partitions
  use multipart image for qemuamd64-bookworm target
  ci: test partition layout of splitted-rootfs image
  ci: upload all logs from testsuite

 .gitlab-ci.yml                                |  4 +-
 .../conf/multiconfig/qemuamd64-bookworm.conf  |  1 +
 .../lib/wic/canned-wks/multipart-efi.wks      |  8 +++
 scripts/lib/wic/misc.py                       |  1 +
 scripts/lib/wic/plugins/imager/direct.py      |  2 +-
 scripts/lib/wic/plugins/source/rootfs.py      |  5 +-
 testsuite/cibase.py                           | 59 +++++++++++++++++++
 testsuite/cibuilder.py                        |  6 +-
 testsuite/citest.py                           | 14 +++++
 9 files changed, 95 insertions(+), 5 deletions(-)
 create mode 100644 meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks

-- 
2.30.2

[PATCH v4 1/5] wic: Update to the latest revision

[Felix Moessbauer]

Update to the latest wic from OE-core.

OE-core Revision: b8878cf0d35cf3d1ac30576d9b9943a7761c011b

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 scripts/lib/wic/misc.py                  | 1 +
 scripts/lib/wic/plugins/imager/direct.py | 2 +-
 scripts/lib/wic/plugins/source/rootfs.py | 5 ++++-
 3 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py
index 3e118229..3bc165fd 100644
--- a/scripts/lib/wic/misc.py
+++ b/scripts/lib/wic/misc.py
@@ -36,6 +36,7 @@ NATIVE_RECIPES = {"bmaptool": "bmap-tools",
                   "mkdosfs": "dosfstools",
                   "mkisofs": "cdrtools",
                   "mkfs.btrfs": "btrfs-tools",
+                  "mkfs.erofs": "erofs-utils",
                   "mkfs.ext2": "e2fsprogs",
                   "mkfs.ext3": "e2fsprogs",
                   "mkfs.ext4": "e2fsprogs",
diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index 35fff7c1..4d0b836e 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -259,7 +259,7 @@ class DirectPlugin(ImagerPlugin):
             if part.mountpoint == "/":
                 if part.uuid:
                     return "PARTUUID=%s" % part.uuid
-                elif part.label:
+                elif part.label and self.ptable_format != 'msdos':
                     return "PARTLABEL=%s" % part.label
                 else:
                     suffix = 'p' if part.disk.startswith('mmcblk') else ''
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 2e34e715..25bb41dd 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -50,7 +50,7 @@ class RootfsPlugin(SourcePlugin):
 
     @staticmethod
     def __get_rootfs_dir(rootfs_dir):
-        if os.path.isdir(rootfs_dir):
+        if rootfs_dir and os.path.isdir(rootfs_dir):
             return os.path.realpath(rootfs_dir)
 
         image_rootfs_dir = get_bitbake_var("IMAGE_ROOTFS", rootfs_dir)
@@ -96,6 +96,9 @@ class RootfsPlugin(SourcePlugin):
         part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
         part.has_fstab = os.path.exists(os.path.join(part.rootfs_dir, "etc/fstab"))
         pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo")
+        if not os.path.lexists(pseudo_dir):
+            pseudo_dir = os.path.join(cls.__get_rootfs_dir(None), '../pseudo')
+
         if not os.path.lexists(pseudo_dir):
             logger.warn("%s folder does not exist. "
                         "Usernames and permissions will be invalid " % pseudo_dir)
-- 
2.30.2

Re: [PATCH v4 1/5] wic: Update to the latest revision

[Schmidl, Tobias]

Hi all,

Am Samstag, dem 28.05.2022 um 16:50 +0200 schrieb Felix Moessbauer:
> Update to the latest wic from OE-core.
> 
> OE-core Revision: b8878cf0d35cf3d1ac30576d9b9943a7761c011b
> 

how is the status of this patch? I have a pending patch that is dependent
on this one.

Kind regards,

Tobias

Re: [PATCH v4 1/5] wic: Update to the latest revision

[Anton Mikanovich]

01.06.2022 15:22, Schmidl, Tobias wrote:
> Hi all,
>
> Am Samstag, dem 28.05.2022 um 16:50 +0200 schrieb Felix Moessbauer:
>> Update to the latest wic from OE-core.
>>
>> OE-core Revision: b8878cf0d35cf3d1ac30576d9b9943a7761c011b
>>
> how is the status of this patch? I have a pending patch that is dependent
> on this one.
>
> Kind regards,
>
> Tobias
>
Hello,

This patch is ok to merge, but we have some issues with the new CI test case
introduced in the same patchset. It is currently fails on one of our CI 
servers
because of non working loop files mounting inside CI chroot.

I'm also not sure about do we need to introduce privileged operations like
'mount' inside testsuite. We need to get rid of sudo one day, and every new
occurrence moves us away from the goal.

I think the most safe way is to use some tool like '7z l -ba <filename>' to
list partition file content without mounting. But this will require to 
add it
to Isar host tools requirements / kas image.

RE: [PATCH v4 1/5] wic: Update to the latest revision

[Moessbauer, Felix]

> -----Original Message-----
> From: Anton Mikanovich <amikan@ilbers.de>
> Sent: Wednesday, June 1, 2022 4:10 PM
> To: Schmidl, Tobias (T CED SES-DE) <tobiasschmidl@siemens.com>;
> Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>; isar-
> users@googlegroups.com; Baurzhan Ismagulov <ibr@ilbers.de>
> Subject: Re: [PATCH v4 1/5] wic: Update to the latest revision
> 
> 01.06.2022 15:22, Schmidl, Tobias wrote:
> > Hi all,
> >
> > Am Samstag, dem 28.05.2022 um 16:50 +0200 schrieb Felix Moessbauer:
> >> Update to the latest wic from OE-core.
> >>
> >> OE-core Revision: b8878cf0d35cf3d1ac30576d9b9943a7761c011b
> >>
> > how is the status of this patch? I have a pending patch that is
> > dependent on this one.
> >
> > Kind regards,
> >
> > Tobias
> >
> Hello,
> 
> This patch is ok to merge, but we have some issues with the new CI test case
> introduced in the same patchset. It is currently fails on one of our CI servers
> because of non working loop files mounting inside CI chroot.
> 
> I'm also not sure about do we need to introduce privileged operations like
> 'mount' inside testsuite. We need to get rid of sudo one day, and every new
> occurrence moves us away from the goal.

Maybe we should just drop these mounting tests until we have a proper solution.
It's just a patch and it should really not block the integration of the actual fix.

For me it is also ok to simply not include patch 5.

Felix

> 
> I think the most safe way is to use some tool like '7z l -ba <filename>' to list
> partition file content without mounting. But this will require to add it to Isar host
> tools requirements / kas image.

Re: [PATCH v4 1/5] wic: Update to the latest revision

[Anton Mikanovich]

02.06.2022 10:44, Moessbauer, Felix wrote:
> Maybe we should just drop these mounting tests until we have a proper solution.
> It's just a patch and it should really not block the integration of the actual fix.
>
> For me it is also ok to simply not include patch 5.
>
> Felix

Ok, let's merge p1-p3 and left p4-p5 to the separate patchset.

[PATCH v4 2/5] meta-isar: add example how to split rootfs across partitions

[Felix Moessbauer]

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks | 8 ++++++++
 1 file changed, 8 insertions(+)
 create mode 100644 meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks

diff --git a/meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks b/meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks
new file mode 100644
index 00000000..b3abb3d0
--- /dev/null
+++ b/meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks
@@ -0,0 +1,8 @@
+# Example showing how to split a single rootfs across partitions
+part /boot --ondisk sda --source bootimg-efi-isar --sourceparams="loader=grub-efi" --label boot --active --align 1024
+# when excluding paths, just specify --exclude-path once and separate paths using spaces
+part /     --ondisk sda --source rootfs --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --exclude-path boot home --extra-space 1G
+# put home last to support auto-expand of this partition
+part /home --ondisk sda --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/home --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024
+
+bootloader --ptable gpt --timeout 3 --append "rootwait console=ttyS0,115200 console=tty0"
-- 
2.30.2

[PATCH v4 3/5] use multipart image for qemuamd64-bookworm target

[Felix Moessbauer]

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta-isar/conf/multiconfig/qemuamd64-bookworm.conf | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta-isar/conf/multiconfig/qemuamd64-bookworm.conf b/meta-isar/conf/multiconfig/qemuamd64-bookworm.conf
index a5ad814f..591918de 100644
--- a/meta-isar/conf/multiconfig/qemuamd64-bookworm.conf
+++ b/meta-isar/conf/multiconfig/qemuamd64-bookworm.conf
@@ -2,3 +2,4 @@
 
 MACHINE ?= "qemuamd64"
 DISTRO ?= "debian-bookworm"
+WKS_FILE ?= "multipart-efi.wks"
-- 
2.30.2

[PATCH v4 4/5] ci: test partition layout of splitted-rootfs image

[Felix Moessbauer]

This adds a test that checks if splitting the rootfs
across multiple partitions correctly works.

The following is checked:

file-permissions (based on wic internal check):
WIC already has a built-in check for correct
file permissions. In case the permission db is missing,
a WIC warning is emitted. We simply check for this warning.

partition layout and excluded paths:
Specifying exclude paths is error prone.
Hence, we check that all generated partitions do not contain
excluded paths. Further, we check that the expected paths are there.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 testsuite/cibase.py    | 59 ++++++++++++++++++++++++++++++++++++++++++
 testsuite/cibuilder.py |  6 ++++-
 testsuite/citest.py    | 14 ++++++++++
 3 files changed, 78 insertions(+), 1 deletion(-)

diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index 2ffb8191..fddf9af5 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -167,3 +167,62 @@ class CIBaseTest(CIBuilder):
                     ['do_rootfs_install_setscene', '!do_rootfs_install'])
             ]):
             self.fail("Failed rebuild package and image")
+
+    def perform_partition_layout_test(self, mc_target, **kwargs):
+        def _mount(device, mountpoint):
+            process.run(f'mount -o ro {device} {mountpoint}', sudo=True, ignore_status=False)
+
+        def _umount(mountpoint):
+            process.run(f'umount {mountpoint}', sudo=True, ignore_status=False)
+
+        def check_output_for_warnings(target, machine):
+            wic_output_files = glob.glob(f'{self.build_dir}/tmp/work/*/{target}-{machine}/*/temp/log.do_image_wic')
+            if len(wic_output_files) == 0:
+                self.fail('could not find WIC output file')
+            with open(wic_output_files[0], 'r') as output:
+                # in case we had permission problems, a WIC warning is in the logs
+                for line in output:
+                    token = line.split()
+                    if len(token) > 0 and token[0] == 'WARNING:':
+                        self.fail(f'WIC issue found: {line}')
+
+        def check_part_layout(target, machine):
+            pattern = f'{self.build_dir}/tmp/deploy/images/{machine}/{target}-*-{machine}.wic.p*'
+            partitions = sorted(glob.glob(pattern))
+            if len(partitions) != 3:
+                self.fail(f'expected 3 partitions, but got {len(partitions)} (in {pattern})')
+            mountpoints = [os.path.join(self.build_dir, mp) for mp in 'mnt_efi mnt_root mnt_home'.split()]
+
+            try:
+                [os.makedirs(mp) for mp in mountpoints]
+                [_mount(p,mp) for p,mp in zip(partitions, mountpoints)]
+
+                # in boot partition, we expect /EFI
+                if not os.path.isdir(os.path.join(mountpoints[0], 'EFI')):
+                    self.fail('boot partition does not provide /EFI')
+                # in root partition, boot and home should be excluded
+                if not os.path.isdir(os.path.join(mountpoints[1], 'etc')) or \
+                os.path.isdir(os.path.join(mountpoints[1], 'home')) or \
+                os.path.isdir(os.path.join(mountpoints[1], 'boot')):
+                    self.fail('root partition does not contain expected dirs')
+                # home partition should contain home of user "user"
+                if not os.path.isdir(os.path.join(mountpoints[2], 'user')):
+                    self.fail('home partition does not contain home of user')
+            finally:
+                [_umount(p) for p in partitions]
+                [os.removedirs(mp) for mp in mountpoints]
+
+        mc_spec = mc_target.split(':')
+        target = mc_spec[2]
+        machine = mc_spec[1].split('-')[0]
+
+        self.configure(
+            sstate=False,
+            compat_arch=False,
+            interactive_user=True,
+            **kwargs)
+
+        self.bitbake(mc_target, **kwargs)
+
+        check_output_for_warnings(target, machine)
+        check_part_layout(target, machine)
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index bc48d47f..e3fbb859 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -54,7 +54,7 @@ class CIBuilder(Test):
 
     def configure(self, compat_arch=True, cross=None, debsrc_cache=False,
                   container=False, ccache=False, sstate=False, offline=False,
-                  gpg_pub_key=None, **kwargs):
+                  gpg_pub_key=None, interactive_user=False, **kwargs):
         # write configuration file and set bitbake_args
         # can run multiple times per test case
         self.check_init()
@@ -107,6 +107,10 @@ class CIBuilder(Test):
                 f.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
             if distro_apt_premir:
                 f.write('DISTRO_APT_PREMIRRORS = "%s"\n' % distro_apt_premir)
+            if interactive_user:
+                f.write('USERS += "user"\n')
+                f.write('USER_user[home] = "/home/user"\n')
+                f.write('USER_user[flags] = "create-home"\n')
 
         # include ci_build.conf in local.conf
         with open(self.build_dir + '/conf/local.conf', 'r+') as f:
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 16e38d07..1f3e4abc 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -318,3 +318,17 @@ class VmBootTestFull(CIBaseTest):
     def test_amd64_focal(self):
         self.init()
         self.vm_start('amd64','focal')
+
+class SplittedRootfsTest(CIBaseTest):
+
+    """
+    Test partition layout of splitted rootfs image
+
+    :avocado: tags=splittedrootfs,full
+    """
+
+    def test_wic_partition_layout(self):
+        mc_target = 'mc:qemuamd64-bookworm:isar-image-base'
+
+        self.init('build-wic-rootfs')
+        self.perform_partition_layout_test(mc_target)
-- 
2.30.2

[PATCH v4 5/5] ci: upload all logs from testsuite

[Felix Moessbauer]

This patch also uploads the bitbake logs from the sstate
and the wic test.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 .gitlab-ci.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index d0488a0c..dccdacdc 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -14,8 +14,8 @@ variables:
   artifacts:
     name: "logs-$CI_JOB_ID"
     paths:
-      - build/tmp/work/*/*/*/temp
-      - build/job-results
+      - build*/tmp/work/*/*/*/temp
+      - build*/job-results
     reports:
       junit:
         - build/job-results/job*/results.xml
-- 
2.30.2

Re: [PATCH v4 0/5] wic: Update to the latest revision

[Anton Mikanovich]

28.05.2022 17:50, Felix Moessbauer wrote:
> Changes since v3:
>
> - improved mount / unmount logic in partition test
> - upload test logs in ci
>
> Changes since v2:
>
> - removed redundant paths exclusion for home (reported by Florian Bezdeka, thanks)
> - improved commit messages
>
> Changes since v1:
>
> - add example how to use splitted rootfs
> - generate such an image in the ci
> - test if the partition layout looks like what we expect
>
> Best regards,
> Felix
>
> Felix Moessbauer (5):
>    wic: Update to the latest revision
>    meta-isar: add example how to split rootfs across partitions
>    use multipart image for qemuamd64-bookworm target
>    ci: test partition layout of splitted-rootfs image
>    ci: upload all logs from testsuite
>
>   .gitlab-ci.yml                                |  4 +-
>   .../conf/multiconfig/qemuamd64-bookworm.conf  |  1 +
>   .../lib/wic/canned-wks/multipart-efi.wks      |  8 +++
>   scripts/lib/wic/misc.py                       |  1 +
>   scripts/lib/wic/plugins/imager/direct.py      |  2 +-
>   scripts/lib/wic/plugins/source/rootfs.py      |  5 +-
>   testsuite/cibase.py                           | 59 +++++++++++++++++++
>   testsuite/cibuilder.py                        |  6 +-
>   testsuite/citest.py                           | 14 +++++
>   9 files changed, 95 insertions(+), 5 deletions(-)
>   create mode 100644 meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks
>
p1-p3 applied to next, thanks.