[PATCH v2 0/7] re-fork wic pcbios and efi plugins

[PATCH v2 0/7] re-fork wic pcbios and efi plugins

[Henning Schild]

changes since v1:
  - efi plugin forked as well
  - systemd-boot support in efi plugin enabled
  - common functionality in utility library
  - test case for system-boot
  - "cp -a" moved to "find exec cp" because of ubuntu
  - changed wks files to exclude boot from root and mount it

The forked plugins have gotten out of sync with the last wic version
bumps. And the original fork was not exactly minimal or made for easy
maintenance.

This series does a re-fork of the two plugins with the aim to come up
with something readable, minimal and maintainable.

There used to be a special case for grub-efi where the actual kernel and
initrd would remain in the root partition, which kind of allowed kernel
updates with apt-get. Now all three bootloaders (systemd-boot now works
as well) place bootloader, config and boot artifacts in a
boot-partition.

Kernel updates with apt-get are now consistantly "broken". That
consistency very likely is not too bad. A generic solution for this
feature (if wanted) will need to be found. Covering not just these three
bootloaders but possibly also u-boot and efibootguard.

Henning Schild (7):
  wic: reset our plugin forks to OE upstream for re-forking
  wic: add utility library for common bits of isar plugins
  wic: apply the actual fork changes to our pcbios plugin fork
  wic: clean up wic class in terms of isar variables
  wic: apply the actual fork changes to our efi plugin fork
  wic: mount /boot and exlude it from root for efi
  meta-isar: use "systemd-boot" for one test target

 RECIPE-API-CHANGELOG.md                       |   9 +
 meta-isar/conf/machine/qemuamd64.conf         |   3 +
 .../scripts/lib/wic/canned-wks/hikey.wks      |   4 +-
 .../lib/wic/canned-wks/sdimage-efi-sd.wks     |   9 +
 .../lib/wic/canned-wks/sdimage-efi.wks        |   4 +-
 meta/classes/wic-img.bbclass                  |   6 +-
 .../scripts/lib/wic/plugins/isarpluginbase.py |  39 ++++
 .../wic/plugins/source/bootimg-efi-isar.py    | 200 ++++++++++++++----
 .../wic/plugins/source/bootimg-pcbios-isar.py | 139 ++++++------
 9 files changed, 290 insertions(+), 123 deletions(-)
 create mode 100644 meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks
 create mode 100644 meta/scripts/lib/wic/plugins/isarpluginbase.py

-- 
2.32.0

[PATCH v2 1/7] wic: reset our plugin forks to OE upstream for re-forking

[Henning Schild]

Note that this commit alone will not work. It is just to get back to a
starting point for clean history.

It takes the original files from our current OE version in this tree as
basis for later changes.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 .../wic/plugins/source/bootimg-efi-isar.py    | 242 +++++++++++-------
 .../wic/plugins/source/bootimg-pcbios-isar.py | 124 ++++-----
 2 files changed, 199 insertions(+), 167 deletions(-)

diff --git a/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py b/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py
index 8094e8f74890..cdc72543c200 100644
--- a/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py
+++ b/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py
@@ -1,24 +1,10 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
 # Copyright (c) 2014, Intel Corporation.
-# All rights reserved.
 #
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
 #
 # DESCRIPTION
-# This implements the 'bootimg-efi-isar' source plugin class for 'wic'
+# This implements the 'bootimg-efi' source plugin class for 'wic'
 #
 # AUTHORS
 # Tom Zanussi <tom.zanussi (at] linux.intel.com>
@@ -27,11 +13,15 @@
 import logging
 import os
 import shutil
+import re
+
+from glob import glob
 
 from wic import WicError
 from wic.engine import get_custom_config
 from wic.pluginbase import SourcePlugin
-from wic.misc import (exec_cmd, get_bitbake_var, BOOTDD_EXTRA_SPACE)
+from wic.misc import (exec_cmd, exec_native_cmd,
+                      get_bitbake_var, BOOTDD_EXTRA_SPACE)
 
 logger = logging.getLogger('wic')
 
@@ -41,10 +31,10 @@ class BootimgEFIPlugin(SourcePlugin):
     This plugin supports GRUB 2 and systemd-boot bootloaders.
     """
 
-    name = 'bootimg-efi-isar'
+    name = 'bootimg-efi'
 
     @classmethod
-    def do_configure_grubefi(cls, creator, cr_workdir, bootpart):
+    def do_configure_grubefi(cls, hdddir, creator, cr_workdir, source_params):
         """
         Create loader-specific (grub-efi) config
         """
@@ -61,29 +51,52 @@ class BootimgEFIPlugin(SourcePlugin):
                 raise WicError("configfile is specified but failed to "
                                "get it from %s." % configfile)
 
+        initrd = source_params.get('initrd')
+
+        if initrd:
+            bootimg_dir = get_bitbake_var("DEPLOY_DIR_IMAGE")
+            if not bootimg_dir:
+                raise WicError("Couldn't find DEPLOY_DIR_IMAGE, exiting")
+
+            initrds = initrd.split(';')
+            for rd in initrds:
+                cp_cmd = "cp %s/%s %s" % (bootimg_dir, rd, hdddir)
+                exec_cmd(cp_cmd, True)
+        else:
+            logger.debug("Ignoring missing initrd")
+
         if not custom_cfg:
             # Create grub configuration using parameters from wks file
             bootloader = creator.ks.bootloader
+            title = source_params.get('title')
 
-            kernel_initrd_path = "/"
-            if get_bitbake_var("DISTRO").startswith("ubuntu"):
-                kernel_initrd_path = "/boot/"
-
-            grubefi_conf =  "serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1\n"
-            grubefi_conf += "terminal_input --append serial\n"
-            grubefi_conf += "terminal_output --append serial\n"
-            grubefi_conf += "\n"
+            grubefi_conf = ""
+            grubefi_conf += "serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1\n"
             grubefi_conf += "default=boot\n"
             grubefi_conf += "timeout=%s\n" % bootloader.timeout
-            for part in creator.parts:
-                if part.mountpoint == "/":
-                    grubefi_conf += "regexp --set bootdisk '(hd[0-9]*),' $prefix\n"
-                    grubefi_conf += "set root=$bootdisk',gpt%d'\n" % part.realnum
-            grubefi_conf += "\n"
-            grubefi_conf += "menuentry 'boot'{\n"
-            grubefi_conf += "    linux %svmlinuz root=%s rootwait %s\n" \
-                            % (kernel_initrd_path, creator.rootdev, bootloader.append or "")
-            grubefi_conf += "    initrd %sinitrd.img\n" % kernel_initrd_path
+            grubefi_conf += "menuentry '%s'{\n" % (title if title else "boot")
+
+            kernel = get_bitbake_var("KERNEL_IMAGETYPE")
+            if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1":
+                if get_bitbake_var("INITRAMFS_IMAGE"):
+                    kernel = "%s-%s.bin" % \
+                        (get_bitbake_var("KERNEL_IMAGETYPE"), get_bitbake_var("INITRAMFS_LINK_NAME"))
+
+            label = source_params.get('label')
+            label_conf = "root=%s" % creator.rootdev
+            if label:
+                label_conf = "LABEL=%s" % label
+
+            grubefi_conf += "linux /%s %s rootwait %s\n" \
+                % (kernel, label_conf, bootloader.append)
+
+            if initrd:
+                initrds = initrd.split(';')
+                grubefi_conf += "initrd"
+                for rd in initrds:
+                    grubefi_conf += " /%s" % rd
+                grubefi_conf += "\n"
+
             grubefi_conf += "}\n"
 
         logger.debug("Writing grubefi config %s/hdd/boot/EFI/BOOT/grub.cfg",
@@ -117,8 +130,10 @@ class BootimgEFIPlugin(SourcePlugin):
             if not bootimg_dir:
                 raise WicError("Couldn't find DEPLOY_DIR_IMAGE, exiting")
 
-            cp_cmd = "cp %s/%s %s" % (bootimg_dir, initrd, hdddir)
-            exec_cmd(cp_cmd, True)
+            initrds = initrd.split(';')
+            for rd in initrds:
+                cp_cmd = "cp %s/%s %s" % (bootimg_dir, rd, hdddir)
+                exec_cmd(cp_cmd, True)
         else:
             logger.debug("Ignoring missing initrd")
 
@@ -143,16 +158,30 @@ class BootimgEFIPlugin(SourcePlugin):
 
         if not custom_cfg:
             # Create systemd-boot configuration using parameters from wks file
-            kernel = "/vmlinuz"
+            kernel = get_bitbake_var("KERNEL_IMAGETYPE")
+            if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1":
+                if get_bitbake_var("INITRAMFS_IMAGE"):
+                    kernel = "%s-%s.bin" % \
+                        (get_bitbake_var("KERNEL_IMAGETYPE"), get_bitbake_var("INITRAMFS_LINK_NAME"))
+
+            title = source_params.get('title')
 
             boot_conf = ""
-            boot_conf += "title boot\n"
-            boot_conf += "linux %s\n" % kernel
-            boot_conf += "options LABEL=Boot root=%s %s\n" % \
-                             (creator.rootdev, bootloader.append or "")
+            boot_conf += "title %s\n" % (title if title else "boot")
+            boot_conf += "linux /%s\n" % kernel
+
+            label = source_params.get('label')
+            label_conf = "LABEL=Boot root=%s" % creator.rootdev
+            if label:
+                label_conf = "LABEL=%s" % label
+
+            boot_conf += "options %s %s\n" % \
+                             (label_conf, bootloader.append)
 
             if initrd:
-                boot_conf += "initrd /%s\n" % initrd
+                initrds = initrd.split(';')
+                for rd in initrds:
+                    boot_conf += "initrd /%s\n" % rd
 
         logger.debug("Writing systemd-boot config "
                      "%s/hdd/boot/loader/entries/boot.conf", cr_workdir)
@@ -175,14 +204,65 @@ class BootimgEFIPlugin(SourcePlugin):
 
         try:
             if source_params['loader'] == 'grub-efi':
-                cls.do_configure_grubefi(creator, cr_workdir, part)
+                cls.do_configure_grubefi(hdddir, creator, cr_workdir, source_params)
             elif source_params['loader'] == 'systemd-boot':
                 cls.do_configure_systemdboot(hdddir, creator, cr_workdir, source_params)
             else:
-                raise WicError("unrecognized bootimg-efi-isar loader: %s" % source_params['loader'])
+                raise WicError("unrecognized bootimg-efi loader: %s" % source_params['loader'])
         except KeyError:
-            raise WicError("bootimg-efi-isar requires a loader, none specified")
+            raise WicError("bootimg-efi requires a loader, none specified")
 
+        if get_bitbake_var("IMAGE_EFI_BOOT_FILES") is None:
+            logger.debug('No boot files defined in IMAGE_EFI_BOOT_FILES')
+        else:
+            boot_files = None
+            for (fmt, id) in (("_uuid-%s", part.uuid), ("_label-%s", part.label), (None, None)):
+                if fmt:
+                    var = fmt % id
+                else:
+                    var = ""
+
+                boot_files = get_bitbake_var("IMAGE_EFI_BOOT_FILES" + var)
+                if boot_files:
+                    break
+
+            logger.debug('Boot files: %s', boot_files)
+
+            # list of tuples (src_name, dst_name)
+            deploy_files = []
+            for src_entry in re.findall(r'[\w;\-\./\*]+', boot_files):
+                if ';' in src_entry:
+                    dst_entry = tuple(src_entry.split(';'))
+                    if not dst_entry[0] or not dst_entry[1]:
+                        raise WicError('Malformed boot file entry: %s' % src_entry)
+                else:
+                    dst_entry = (src_entry, src_entry)
+
+                logger.debug('Destination entry: %r', dst_entry)
+                deploy_files.append(dst_entry)
+
+            cls.install_task = [];
+            for deploy_entry in deploy_files:
+                src, dst = deploy_entry
+                if '*' in src:
+                    # by default install files under their basename
+                    entry_name_fn = os.path.basename
+                    if dst != src:
+                        # unless a target name was given, then treat name
+                        # as a directory and append a basename
+                        entry_name_fn = lambda name: \
+                                        os.path.join(dst,
+                                                     os.path.basename(name))
+
+                    srcs = glob(os.path.join(kernel_dir, src))
+
+                    logger.debug('Globbed sources: %s', ', '.join(srcs))
+                    for entry in srcs:
+                        src = os.path.relpath(entry, kernel_dir)
+                        entry_dst_name = entry_name_fn(entry)
+                        cls.install_task.append((src, entry_dst_name))
+                else:
+                    cls.install_task.append((src, dst))
 
     @classmethod
     def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
@@ -202,6 +282,23 @@ class BootimgEFIPlugin(SourcePlugin):
 
         hdddir = "%s/hdd/boot" % cr_workdir
 
+        kernel = get_bitbake_var("KERNEL_IMAGETYPE")
+        if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1":
+            if get_bitbake_var("INITRAMFS_IMAGE"):
+                kernel = "%s-%s.bin" % \
+                    (get_bitbake_var("KERNEL_IMAGETYPE"), get_bitbake_var("INITRAMFS_LINK_NAME"))
+
+        install_cmd = "install -m 0644 %s/%s %s/%s" % \
+            (staging_kernel_dir, kernel, hdddir, kernel)
+        exec_cmd(install_cmd)
+
+        if get_bitbake_var("IMAGE_EFI_BOOT_FILES"):
+            for src_path, dst_path in cls.install_task:
+                install_cmd = "install -m 0644 -D %s %s" \
+                              % (os.path.join(kernel_dir, src_path),
+                                 os.path.join(hdddir, dst_path))
+                exec_cmd(install_cmd)
+
         try:
             if source_params['loader'] == 'grub-efi':
                 shutil.copyfile("%s/hdd/boot/EFI/BOOT/grub.cfg" % cr_workdir,
@@ -211,52 +308,15 @@ class BootimgEFIPlugin(SourcePlugin):
                     exec_cmd(cp_cmd, True)
                 shutil.move("%s/grub.cfg" % cr_workdir,
                             "%s/hdd/boot/EFI/BOOT/grub.cfg" % cr_workdir)
-
-                distro_arch = get_bitbake_var("DISTRO_ARCH")
-                if not distro_arch:
-                    raise WicError("Couldn't find target architecture")
-
-                if distro_arch == "amd64":
-                    grub_target = 'x86_64-efi'
-                    grub_image = "bootx64.efi"
-                    grub_modules = "multiboot efi_uga iorw ata "
-                elif distro_arch == "i386":
-                    grub_target = 'i386-efi'
-                    grub_image = "bootia32.efi"
-                    grub_modules = "multiboot efi_uga iorw ata "
-                elif distro_arch == "arm64":
-                    grub_target = 'arm64-efi'
-                    grub_image = "bootaa64.efi"
-                    grub_modules = ""
-                else:
-                    raise WicError("grub-efi is incompatible with target %s" %
-                                   distro_arch)
-
-                bootimg_dir = "%s/hdd/boot" % cr_workdir
-                if not os.path.isfile("%s/EFI/BOOT/%s" \
-                                      % (bootimg_dir, grub_image)):
-
-                    # TODO: check that grub-mkimage is available
-                    grub_cmd = "grub-mkimage -p /EFI/BOOT "
-                    grub_cmd += "-O %s -o %s/EFI/BOOT/%s " \
-                                % (grub_target, bootimg_dir, grub_image)
-                    grub_cmd += "part_gpt part_msdos ntfs ntfscomp fat ext2 "
-                    grub_cmd += "normal chain boot configfile linux "
-                    grub_cmd += "search efi_gop font gfxterm gfxmenu "
-                    grub_cmd += "terminal minicmd test loadenv echo help "
-                    grub_cmd += "reboot serial terminfo iso9660 loopback tar "
-                    grub_cmd += "memdisk ls search_fs_uuid udf btrfs xfs lvm "
-                    grub_cmd += "reiserfs regexp " + grub_modules
-                    exec_cmd(grub_cmd)
             elif source_params['loader'] == 'systemd-boot':
                 for mod in [x for x in os.listdir(kernel_dir) if x.startswith("systemd-")]:
                     cp_cmd = "cp %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, hdddir, mod[8:])
                     exec_cmd(cp_cmd, True)
             else:
-                raise WicError("unrecognized bootimg-efi-isar loader: %s" %
+                raise WicError("unrecognized bootimg-efi loader: %s" %
                                source_params['loader'])
         except KeyError:
-            raise WicError("bootimg-efi-isar requires a loader, none specified")
+            raise WicError("bootimg-efi requires a loader, none specified")
 
         startup = os.path.join(kernel_dir, "startup.nsh")
         if os.path.exists(startup):
@@ -280,12 +340,14 @@ class BootimgEFIPlugin(SourcePlugin):
         # dosfs image, created by mkdosfs
         bootimg = "%s/boot.img" % cr_workdir
 
-        dosfs_cmd = "mkdosfs -n efi -i %s -C %s %d" % \
-                    (part.fsuuid, bootimg, blocks)
-        exec_cmd(dosfs_cmd)
+        label = part.label if part.label else "ESP"
+
+        dosfs_cmd = "mkdosfs -n %s -i %s -C %s %d" % \
+                    (label, part.fsuuid, bootimg, blocks)
+        exec_native_cmd(dosfs_cmd, native_sysroot)
 
         mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir)
-        exec_cmd(mcopy_cmd, True)
+        exec_native_cmd(mcopy_cmd, native_sysroot)
 
         chmod_cmd = "chmod 644 %s" % bootimg
         exec_cmd(chmod_cmd)
diff --git a/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py b/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py
index 2dc90bd0cd30..f2639e700493 100644
--- a/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py
+++ b/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py
@@ -1,24 +1,10 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
 # Copyright (c) 2014, Intel Corporation.
-# All rights reserved.
 #
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
 #
 # DESCRIPTION
-# This implements the 'bootimg-pcbios-isar' source plugin class for 'wic'
+# This implements the 'bootimg-pcbios' source plugin class for 'wic'
 #
 # AUTHORS
 # Tom Zanussi <tom.zanussi (at] linux.intel.com>
@@ -26,36 +12,40 @@
 
 import logging
 import os
+import re
 
 from wic import WicError
 from wic.engine import get_custom_config
 from wic.pluginbase import SourcePlugin
 from wic.misc import (exec_cmd, exec_native_cmd,
-                            get_bitbake_var, BOOTDD_EXTRA_SPACE)
+                      get_bitbake_var, BOOTDD_EXTRA_SPACE)
 
 logger = logging.getLogger('wic')
 
-class BootimgPcbiosIsarPlugin(SourcePlugin):
+class BootimgPcbiosPlugin(SourcePlugin):
     """
     Create MBR boot partition and install syslinux on it.
     """
 
-    name = 'bootimg-pcbios-isar'
+    name = 'bootimg-pcbios'
 
     @classmethod
-    def _get_syslinux_dir(cls, bootimg_dir):
+    def _get_bootimg_dir(cls, bootimg_dir, dirname):
         """
-        Get path to syslinux from either default bootimg_dir
-        or wic-tools STAGING_DIR.
+        Check if dirname exists in default bootimg_dir or in STAGING_DIR.
         """
-        for path in (bootimg_dir, get_bitbake_var("STAGING_DATADIR", "wic-tools")):
-            if not path:
-                continue
-            syslinux_dir = os.path.join(path, 'syslinux')
-            if os.path.exists(syslinux_dir):
-                return syslinux_dir
+        staging_datadir = get_bitbake_var("STAGING_DATADIR")
+        for result in (bootimg_dir, staging_datadir):
+            if os.path.exists("%s/%s" % (result, dirname)):
+                return result
+
+        # STAGING_DATADIR is expanded with MLPREFIX if multilib is enabled
+        # but dependency syslinux is still populated to original STAGING_DATADIR
+        nonarch_datadir = re.sub('/[^/]*recipe-sysroot', '/recipe-sysroot', staging_datadir)
+        if os.path.exists(os.path.join(nonarch_datadir, dirname)):
+            return nonarch_datadir
 
-        raise WicError("Couldn't find syslinux directory, exiting")
+        raise WicError("Couldn't find correct bootimg_dir, exiting")
 
     @classmethod
     def do_install_disk(cls, disk, disk_name, creator, workdir, oe_builddir,
@@ -64,11 +54,12 @@ class BootimgPcbiosIsarPlugin(SourcePlugin):
         Called after all partitions have been prepared and assembled into a
         disk image.  In this case, we install the MBR.
         """
-        syslinux_dir = cls._get_syslinux_dir(bootimg_dir)
+        bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux')
+        mbrfile = "%s/syslinux/" % bootimg_dir
         if creator.ptable_format == 'msdos':
-            mbrfile = os.path.join(syslinux_dir, "mbr/mbr.bin")
+            mbrfile += "mbr.bin"
         elif creator.ptable_format == 'gpt':
-            mbrfile = os.path.join(syslinux_dir, "mbr/gptmbr.bin")
+            mbrfile += "gptmbr.bin"
         else:
             raise WicError("Unsupported partition table: %s" %
                            creator.ptable_format)
@@ -131,20 +122,11 @@ class BootimgPcbiosIsarPlugin(SourcePlugin):
             syslinux_conf += "DEFAULT boot\n"
             syslinux_conf += "LABEL boot\n"
 
-            kernel_file = get_bitbake_var("KERNEL_FILE")
-            rootfs_dir = get_bitbake_var("IMAGE_ROOTFS")
-            kernel = os.path.basename(os.path.realpath(os.path.join(rootfs_dir, kernel_file)))
-            kernel_parts = kernel.split("-")
-            kernel_suffix = "-".join(kernel_parts[1:])
-            initrd = "initrd.img"
-
-            if kernel_suffix:
-                initrd += "-%s" % kernel_suffix
-
+            kernel = "/vmlinuz"
             syslinux_conf += "KERNEL " + kernel + "\n"
 
-            syslinux_conf += "APPEND label=boot root=%s initrd=%s %s\n" % \
-                             (creator.rootdev, initrd, bootloader.append)
+            syslinux_conf += "APPEND label=boot root=%s %s\n" % \
+                             (creator.rootdev, bootloader.append)
 
         logger.debug("Writing syslinux config %s/hdd/boot/syslinux.cfg",
                      cr_workdir)
@@ -161,40 +143,28 @@ class BootimgPcbiosIsarPlugin(SourcePlugin):
         'prepares' the partition to be incorporated into the image.
         In this case, prepare content for legacy bios boot partition.
         """
-        syslinux_dir = cls._get_syslinux_dir(bootimg_dir)
-
-        kernel_file = get_bitbake_var("KERNEL_FILE")
-        rootfs_dir = rootfs_dir['ROOTFS_DIR']
-        kernel = os.path.basename(os.path.realpath(os.path.join(rootfs_dir, kernel_file)))
-        kernel_parts = kernel.split("-")
-        kernel_suffix = "-".join(kernel_parts[1:])
-        initrd = "initrd.img"
-        config = "config"
-        mapfile = "System.map"
-
-        if kernel_suffix:
-            initrd += "-%s" % kernel_suffix
-            config += "-%s" % kernel_suffix
-            mapfile += "-%s" % kernel_suffix
+        bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux')
+
+        staging_kernel_dir = kernel_dir
 
         hdddir = "%s/hdd/boot" % cr_workdir
 
-        cmds = ("install -m 0644 %s/%s/%s %s/%s" %
-                (rootfs_dir, "boot", kernel, hdddir, kernel),
-                "install -m 0644 %s/%s/%s %s/%s" %
-                (rootfs_dir, "boot", initrd, hdddir, initrd),
-                "install -m 0644 %s/%s/%s %s/%s" %
-                (rootfs_dir, "boot", config, hdddir, config),
-                "install -m 0644 %s/%s/%s %s/%s" %
-                (rootfs_dir, "boot", mapfile, hdddir, mapfile),
-                "install -m 444 %s/modules/bios/ldlinux.c32 %s/ldlinux.c32" %
-                (syslinux_dir, hdddir),
-                "install -m 0644 %s/modules/bios/vesamenu.c32 %s/vesamenu.c32" %
-                (syslinux_dir, hdddir),
-                "install -m 444 %s/modules/bios/libcom32.c32 %s/libcom32.c32" %
-                (syslinux_dir, hdddir),
-                "install -m 444 %s/modules/bios/libutil.c32 %s/libutil.c32" %
-                (syslinux_dir, hdddir))
+        kernel = get_bitbake_var("KERNEL_IMAGETYPE")
+        if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1":
+            if get_bitbake_var("INITRAMFS_IMAGE"):
+                kernel = "%s-%s.bin" % \
+                    (get_bitbake_var("KERNEL_IMAGETYPE"), get_bitbake_var("INITRAMFS_LINK_NAME"))
+
+        cmds = ("install -m 0644 %s/%s %s/vmlinuz" %
+                (staging_kernel_dir, kernel, hdddir),
+                "install -m 444 %s/syslinux/ldlinux.sys %s/ldlinux.sys" %
+                (bootimg_dir, hdddir),
+                "install -m 0644 %s/syslinux/vesamenu.c32 %s/vesamenu.c32" %
+                (bootimg_dir, hdddir),
+                "install -m 444 %s/syslinux/libcom32.c32 %s/libcom32.c32" %
+                (bootimg_dir, hdddir),
+                "install -m 444 %s/syslinux/libutil.c32 %s/libutil.c32" %
+                (bootimg_dir, hdddir))
 
         for install_cmd in cmds:
             exec_cmd(install_cmd)
@@ -214,14 +184,14 @@ class BootimgPcbiosIsarPlugin(SourcePlugin):
                      extra_blocks, part.mountpoint, blocks)
 
         # dosfs image, created by mkdosfs
-        bootimg = "%s/boot.img" % cr_workdir
+        bootimg = "%s/boot%s.img" % (cr_workdir, part.lineno)
 
         dosfs_cmd = "mkdosfs -n boot -i %s -S 512 -C %s %d" % \
                     (part.fsuuid, bootimg, blocks)
         exec_native_cmd(dosfs_cmd, native_sysroot)
 
         mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir)
-        exec_cmd(mcopy_cmd, native_sysroot)
+        exec_native_cmd(mcopy_cmd, native_sysroot)
 
         syslinux_cmd = "syslinux %s" % bootimg
         exec_native_cmd(syslinux_cmd, native_sysroot)
-- 
2.32.0

[PATCH v2 2/7] wic: add utility library for common bits of isar plugins

[Henning Schild]

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 .../scripts/lib/wic/plugins/isarpluginbase.py | 39 +++++++++++++++++++
 1 file changed, 39 insertions(+)
 create mode 100644 meta/scripts/lib/wic/plugins/isarpluginbase.py

diff --git a/meta/scripts/lib/wic/plugins/isarpluginbase.py b/meta/scripts/lib/wic/plugins/isarpluginbase.py
new file mode 100644
index 000000000000..185e2ea14fc7
--- /dev/null
+++ b/meta/scripts/lib/wic/plugins/isarpluginbase.py
@@ -0,0 +1,39 @@
+#
+# Copyright (c) Siemens AG, 2021
+#
+# SPDX-License-Identifier: MIT
+#
+# DESCRIPTION
+# This implements a few utility functions to be used in isar-specific plugins.
+
+import os
+
+def isar_populate_boot_cmd(rootfs_dir, hdddir):
+    # copy all files from rootfs/boot into boot partition
+    # no not copy symlinks (ubuntu places them here) because targetfs is fat
+    return "find %s/boot -type f -exec cp -a {} %s ;" % (rootfs_dir, hdddir)
+
+def isar_get_filenames(rootfs_dir):
+    # figure out the real filename in /boot by following debian symlinks
+    for kernel_symlink in ["vmlinuz", "vmlinux" ]:
+        kernel_file_abs = os.path.join(rootfs_dir, kernel_symlink)
+        if os.path.isfile(kernel_file_abs):
+            break
+        # in ubuntu those symlinks could be in /boot
+        kernel_file_abs = os.path.join(rootfs_dir, "boot", kernel_symlink)
+        if os.path.isfile(kernel_file_abs):
+            break
+
+    kernel = os.path.basename(os.path.realpath(kernel_file_abs))
+    initrd =  "initrd.img"
+    kernel_parts = kernel.split("-")
+    kernel_suffix = "-".join(kernel_parts[1:])
+    if kernel_suffix:
+        initrd += "-%s" % kernel_suffix
+
+    if not os.path.isfile(os.path.join(rootfs_dir, "boot", kernel)):
+        raise WicError("kernel %s not found" % (os.path.join(rootfs_dir, "boot", kernel)))
+    if not os.path.isfile(os.path.join(rootfs_dir, "boot", initrd)):
+        raise WicError("initrd %s not found" % (os.path.join(rootfs_dir, "boot", initrd)))
+
+    return kernel, initrd
-- 
2.32.0

[PATCH v2 3/7] wic: apply the actual fork changes to our pcbios plugin fork

[Henning Schild]

This aims to be a rather minimal one-commit fork to do isar specific
changes to the forked class. It does things in a way that results in
somehow strange code but pretty clean patching that aims to be easy to
read and mostly about adding lines to get things done. The hope is that
it will make maintaining the fork easier than before.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 .../wic/plugins/source/bootimg-pcbios-isar.py | 31 +++++++++++++++++--
 1 file changed, 28 insertions(+), 3 deletions(-)

diff --git a/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py b/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py
index f2639e700493..f493890fae31 100644
--- a/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py
+++ b/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py
@@ -4,7 +4,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 #
 # DESCRIPTION
-# This implements the 'bootimg-pcbios' source plugin class for 'wic'
+# This implements the 'bootimg-pcbios-isar' source plugin class for 'wic'
 #
 # AUTHORS
 # Tom Zanussi <tom.zanussi (at] linux.intel.com>
@@ -20,14 +20,18 @@ from wic.pluginbase import SourcePlugin
 from wic.misc import (exec_cmd, exec_native_cmd,
                       get_bitbake_var, BOOTDD_EXTRA_SPACE)
 
+import sys
+sys.path[0] = os.path.dirname(os.path.abspath(__file__)) + "/.."
+from isarpluginbase import (isar_get_filenames, isar_populate_boot_cmd)
+
 logger = logging.getLogger('wic')
 
-class BootimgPcbiosPlugin(SourcePlugin):
+class BootimgPcbiosIsarPlugin(SourcePlugin):
     """
     Create MBR boot partition and install syslinux on it.
     """
 
-    name = 'bootimg-pcbios'
+    name = 'bootimg-pcbios-isar'
 
     @classmethod
     def _get_bootimg_dir(cls, bootimg_dir, dirname):
@@ -56,6 +60,10 @@ class BootimgPcbiosPlugin(SourcePlugin):
         """
         bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux')
         mbrfile = "%s/syslinux/" % bootimg_dir
+
+        # files have different prefix in debian
+        mbrfile += "mbr/"
+
         if creator.ptable_format == 'msdos':
             mbrfile += "mbr.bin"
         elif creator.ptable_format == 'gpt':
@@ -123,11 +131,18 @@ class BootimgPcbiosPlugin(SourcePlugin):
             syslinux_conf += "LABEL boot\n"
 
             kernel = "/vmlinuz"
+
+            kernel, initrd = isar_get_filenames(get_bitbake_var("IMAGE_ROOTFS"))
+
             syslinux_conf += "KERNEL " + kernel + "\n"
 
             syslinux_conf += "APPEND label=boot root=%s %s\n" % \
                              (creator.rootdev, bootloader.append)
 
+            # we are using an initrd, smuggle it in
+            syslinux_conf = syslinux_conf.replace(" root=%s " % (creator.rootdev),
+                                                  " root=%s initrd=%s " % (creator.rootdev, initrd))
+
         logger.debug("Writing syslinux config %s/hdd/boot/syslinux.cfg",
                      cr_workdir)
         cfg = open("%s/hdd/boot/syslinux.cfg" % cr_workdir, "w")
@@ -166,7 +181,17 @@ class BootimgPcbiosPlugin(SourcePlugin):
                 "install -m 444 %s/syslinux/libutil.c32 %s/libutil.c32" %
                 (bootimg_dir, hdddir))
 
+        install_cmd = isar_populate_boot_cmd(rootfs_dir['ROOTFS_DIR'], hdddir)
+        exec_cmd(install_cmd)
+
         for install_cmd in cmds:
+            # skip the kernel install from OE parts, we copy boot from debian
+            if install_cmd == cmds[0]:
+                continue
+            # one file has a different suffix in debian
+            install_cmd = install_cmd.replace("ldlinux.sys", "ldlinux.c32")
+            # files have different prefix in debian
+            install_cmd = install_cmd.replace("/syslinux/", "/syslinux/modules/bios/")
             exec_cmd(install_cmd)
 
         du_cmd = "du -bks %s" % hdddir
-- 
2.32.0

[PATCH v2 4/7] wic: clean up wic class in terms of isar variables

[Henning Schild]

Remove meanwhile unused variables from that append and add a comment
to prevent further "messing" with forked bits.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/wic-img.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass
index 170b83ed3416..d849ad9fae5f 100644
--- a/meta/classes/wic-img.bbclass
+++ b/meta/classes/wic-img.bbclass
@@ -92,14 +92,14 @@ BUILDCHROOT_DIR = "${BUILDCHROOT_TARGET_DIR}"
 
 WIC_CREATE_EXTRA_ARGS ?= ""
 
+# taken from OE, do not touch directly
 WICVARS += "\
            BBLAYERS IMGDEPLOYDIR DEPLOY_DIR_IMAGE FAKEROOTCMD IMAGE_BASENAME IMAGE_BOOT_FILES \
            IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD INITRD_LIVE ISODIR RECIPE_SYSROOT_NATIVE \
-           ROOTFS_SIZE STAGING_DATADIR STAGING_DIR STAGING_LIBDIR TARGET_SYS TRANSLATED_TARGET_ARCH \
-           KERNEL_FILE"
+           ROOTFS_SIZE STAGING_DATADIR STAGING_DIR STAGING_LIBDIR TARGET_SYS TRANSLATED_TARGET_ARCH"
 
 # Isar specific vars used in our plugins
-WICVARS += "KERNEL_IMAGE INITRD_IMAGE DISTRO DISTRO_ARCH"
+WICVARS += "DISTRO DISTRO_ARCH"
 
 python do_rootfs_wicenv () {
     wicvars = d.getVar('WICVARS', True)
-- 
2.32.0

[PATCH v2 5/7] wic: apply the actual fork changes to our efi plugin fork

[Henning Schild]

This aims to be a rather minimal one-commit fork to do isar specific
changes to the forked class. It does things in a way that results in
somehow strange code but pretty clean patching that aims to be easy to
read and mostly about adding lines to get things done. The hope is that
it will make maintaining the fork easier than before.

Note that it changes the way we deal with grub, we now do it like OE and
consistent with systemd-boot and syslinux. Meaning we have the
bootloader, its config and kernel+initrd in the /boot partition.
While that breaks kernel updates with "apt-get" it gets things
consistent and less hacky. Such kernel updates anyways only worked with
grub-efi and should rather be solved differently covering all possible
bootloaders.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 RECIPE-API-CHANGELOG.md                       |  9 +++
 .../wic/plugins/source/bootimg-efi-isar.py    | 62 +++++++++++++++++--
 2 files changed, 65 insertions(+), 6 deletions(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 806c2914f13f..7312d4d4baa9 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -292,3 +292,12 @@ Migrate your patches so they can be applied with "git am", or
 
 Kernel image name for arm64 platforms is vmlinux now. Image format was
 not changed (uncompressed executable) but now it is named correctly.
+
+### wic plugins for efi and pcbios use seperate /boot partition
+
+It used to depend on the bootloader whether stuff was in in the root partition or in the boot partition, now it will always be in the boot partition.
+
+Kernel update with "apt-get" will not work since bootloader configuration will
+not be updated. It used to "kind of work" for grub and efi, that hack is gone.
+
+When using the plugins it is advised to name the partition "/boot" and to exclude boot from the follwing rootfs to not waste space.
diff --git a/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py b/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py
index cdc72543c200..d97125481155 100644
--- a/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py
+++ b/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py
@@ -4,7 +4,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 #
 # DESCRIPTION
-# This implements the 'bootimg-efi' source plugin class for 'wic'
+# This implements the 'bootimg-efi-isar' source plugin class for 'wic'
 #
 # AUTHORS
 # Tom Zanussi <tom.zanussi (at] linux.intel.com>
@@ -23,6 +23,10 @@ from wic.pluginbase import SourcePlugin
 from wic.misc import (exec_cmd, exec_native_cmd,
                       get_bitbake_var, BOOTDD_EXTRA_SPACE)
 
+import sys
+sys.path[0] = os.path.dirname(os.path.abspath(__file__)) + "/.."
+from isarpluginbase import (isar_get_filenames, isar_populate_boot_cmd)
+
 logger = logging.getLogger('wic')
 
 class BootimgEFIPlugin(SourcePlugin):
@@ -31,7 +35,7 @@ class BootimgEFIPlugin(SourcePlugin):
     This plugin supports GRUB 2 and systemd-boot bootloaders.
     """
 
-    name = 'bootimg-efi'
+    name = 'bootimg-efi-isar'
 
     @classmethod
     def do_configure_grubefi(cls, hdddir, creator, cr_workdir, source_params):
@@ -72,6 +76,9 @@ class BootimgEFIPlugin(SourcePlugin):
 
             grubefi_conf = ""
             grubefi_conf += "serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1\n"
+            grubefi_conf += "terminal_input --append serial\n"
+            grubefi_conf += "terminal_output --append serial\n"
+            grubefi_conf += "\n"
             grubefi_conf += "default=boot\n"
             grubefi_conf += "timeout=%s\n" % bootloader.timeout
             grubefi_conf += "menuentry '%s'{\n" % (title if title else "boot")
@@ -87,6 +94,7 @@ class BootimgEFIPlugin(SourcePlugin):
             if label:
                 label_conf = "LABEL=%s" % label
 
+            kernel, initrd = isar_get_filenames(get_bitbake_var("IMAGE_ROOTFS"))
             grubefi_conf += "linux /%s %s rootwait %s\n" \
                 % (kernel, label_conf, bootloader.append)
 
@@ -166,6 +174,8 @@ class BootimgEFIPlugin(SourcePlugin):
 
             title = source_params.get('title')
 
+            kernel, initrd = isar_get_filenames(get_bitbake_var("IMAGE_ROOTFS"))
+
             boot_conf = ""
             boot_conf += "title %s\n" % (title if title else "boot")
             boot_conf += "linux /%s\n" % kernel
@@ -208,9 +218,9 @@ class BootimgEFIPlugin(SourcePlugin):
             elif source_params['loader'] == 'systemd-boot':
                 cls.do_configure_systemdboot(hdddir, creator, cr_workdir, source_params)
             else:
-                raise WicError("unrecognized bootimg-efi loader: %s" % source_params['loader'])
+                raise WicError("unrecognized bootimg-efi-isar loader: %s" % source_params['loader'])
         except KeyError:
-            raise WicError("bootimg-efi requires a loader, none specified")
+            raise WicError("bootimg-efi-isar requires a loader, none specified")
 
         if get_bitbake_var("IMAGE_EFI_BOOT_FILES") is None:
             logger.debug('No boot files defined in IMAGE_EFI_BOOT_FILES')
@@ -290,6 +300,8 @@ class BootimgEFIPlugin(SourcePlugin):
 
         install_cmd = "install -m 0644 %s/%s %s/%s" % \
             (staging_kernel_dir, kernel, hdddir, kernel)
+
+        install_cmd = isar_populate_boot_cmd(rootfs_dir['ROOTFS_DIR'], hdddir)
         exec_cmd(install_cmd)
 
         if get_bitbake_var("IMAGE_EFI_BOOT_FILES"):
@@ -308,15 +320,53 @@ class BootimgEFIPlugin(SourcePlugin):
                     exec_cmd(cp_cmd, True)
                 shutil.move("%s/grub.cfg" % cr_workdir,
                             "%s/hdd/boot/EFI/BOOT/grub.cfg" % cr_workdir)
+
+                distro_arch = get_bitbake_var("DISTRO_ARCH")
+                if not distro_arch:
+                    raise WicError("Couldn't find target architecture")
+
+                if distro_arch == "amd64":
+                    grub_target = 'x86_64-efi'
+                    grub_image = "bootx64.efi"
+                    grub_modules = "multiboot efi_uga iorw ata "
+                elif distro_arch == "i386":
+                    grub_target = 'i386-efi'
+                    grub_image = "bootia32.efi"
+                    grub_modules = "multiboot efi_uga iorw ata "
+                elif distro_arch == "arm64":
+                    grub_target = 'arm64-efi'
+                    grub_image = "bootaa64.efi"
+                    grub_modules = ""
+                else:
+                    raise WicError("grub-efi is incompatible with target %s" %
+                                   distro_arch)
+
+                bootimg_dir = "%s/hdd/boot" % cr_workdir
+                if not os.path.isfile("%s/EFI/BOOT/%s" \
+                                      % (bootimg_dir, grub_image)):
+
+                    # TODO: check that grub-mkimage is available
+                    grub_cmd = "grub-mkimage -p /EFI/BOOT "
+                    grub_cmd += "-O %s -o %s/EFI/BOOT/%s " \
+                                % (grub_target, bootimg_dir, grub_image)
+                    grub_cmd += "part_gpt part_msdos ntfs ntfscomp fat ext2 "
+                    grub_cmd += "normal chain boot configfile linux "
+                    grub_cmd += "search efi_gop font gfxterm gfxmenu "
+                    grub_cmd += "terminal minicmd test loadenv echo help "
+                    grub_cmd += "reboot serial terminfo iso9660 loopback tar "
+                    grub_cmd += "memdisk ls search_fs_uuid udf btrfs xfs lvm "
+                    grub_cmd += "reiserfs regexp " + grub_modules
+                    exec_cmd(grub_cmd)
             elif source_params['loader'] == 'systemd-boot':
+                kernel_dir = os.path.join(rootfs_dir['ROOTFS_DIR'], "usr/lib/systemd/boot/efi/")
                 for mod in [x for x in os.listdir(kernel_dir) if x.startswith("systemd-")]:
                     cp_cmd = "cp %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, hdddir, mod[8:])
                     exec_cmd(cp_cmd, True)
             else:
-                raise WicError("unrecognized bootimg-efi loader: %s" %
+                raise WicError("unrecognized bootimg-efi-isar loader: %s" %
                                source_params['loader'])
         except KeyError:
-            raise WicError("bootimg-efi requires a loader, none specified")
+            raise WicError("bootimg-efi-isar requires a loader, none specified")
 
         startup = os.path.join(kernel_dir, "startup.nsh")
         if os.path.exists(startup):
-- 
2.32.0

[PATCH v2 6/7] wic: mount /boot and exlude it from root for efi

[Henning Schild]

We now have everything in a boot-partition also for efi, so mount that
and exclude it from root to save space.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/scripts/lib/wic/canned-wks/hikey.wks       | 4 ++--
 meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks
index 1b7a64e1b55c..0c966e65622d 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks
@@ -3,8 +3,8 @@
 #
 # SPDX-License-Identifier: MIT
 
-part --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk mmcblk1 --label efi --part-type EF00 --align 1024
+part /boot --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk mmcblk1 --label efi --part-type EF00 --align 1024
 
-part / --source rootfs --ondisk mmcblk1 --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid
+part / --source rootfs --ondisk mmcblk1 --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/
 
 bootloader --ptable gpt --timeout=3
diff --git a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
index be468430a7d6..f3addbc7515d 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
@@ -2,8 +2,8 @@
 # long-description: Creates a partitioned EFI disk image without any swap that
 # the user can directly dd to boot media.
 
-part --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk sda --label efi --part-type EF00 --align 1024
+part /boot --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk sda --label efi --part-type EF00 --align 1024
 
-part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid
+part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/
 
 bootloader --ptable gpt --timeout 3 --append "rootwait console=ttyS0,115200 console=tty0"
-- 
2.32.0

[PATCH v2 7/7] meta-isar: use "systemd-boot" for one test target

[Henning Schild]

An earlier commit changed the efi plugin fork so that its systemd
support will also work in isar. This commit introduces a wks file and
switches one example target to the alternative efi bootloader.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/conf/machine/qemuamd64.conf                   | 3 +++
 meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks | 9 +++++++++
 2 files changed, 12 insertions(+)
 create mode 100644 meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks

diff --git a/meta-isar/conf/machine/qemuamd64.conf b/meta-isar/conf/machine/qemuamd64.conf
index 238bc90720ea..dab4f0942a52 100644
--- a/meta-isar/conf/machine/qemuamd64.conf
+++ b/meta-isar/conf/machine/qemuamd64.conf
@@ -10,6 +10,9 @@ IMAGE_TYPE ?= "wic-img"
 WKS_FILE ?= "sdimage-efi"
 IMAGER_INSTALL += "${GRUB_BOOTLOADER_INSTALL}"
 
+WKS_FILE_ubuntu-focal ?= "sdimage-efi-sd"
+IMAGER_INSTALL_remove_ubuntu-focal = "${GRUB_BOOTLOADER_INSTALL}"
+
 IMAGE_INSTALL += "sshd-regen-keys"
 
 QEMU_ARCH ?= "x86_64"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks
new file mode 100644
index 000000000000..754fbc46f8e4
--- /dev/null
+++ b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks
@@ -0,0 +1,9 @@
+# short-description: Create an EFI disk image without any swap
+# long-description: Creates a partitioned EFI disk image without any swap that
+# the user can directly dd to boot media.
+
+part /boot --source bootimg-efi-isar --sourceparams "loader=systemd-boot" --ondisk sda --label efi --part-type EF00 --align 1024
+
+part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/
+
+bootloader --ptable gpt --timeout 3 --append "rootwait console=ttyS0,115200 console=tty0"
-- 
2.32.0

Re: [PATCH v2 6/7] wic: mount /boot and exlude it from root for efi

[Henning Schild]

Am Fri, 3 Sep 2021 14:53:54 +0200
schrieb Henning Schild <henning.schild@siemens.com>:

> We now have everything in a boot-partition also for efi, so mount that
> and exclude it from root to save space.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>  meta-isar/scripts/lib/wic/canned-wks/hikey.wks       | 4 ++--
>  meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks | 4 ++--
>  2 files changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks
> b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks index
> 1b7a64e1b55c..0c966e65622d 100644 ---
> a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks +++
> b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks @@ -3,8 +3,8 @@
>  #
>  # SPDX-License-Identifier: MIT
>  
> -part --source bootimg-efi-isar --sourceparams "loader=grub-efi"
> --ondisk mmcblk1 --label efi --part-type EF00 --align 1024 
> +part /boot --source bootimg-efi-isar --sourceparams "loader=grub-efi"
> --ondisk mmcblk1 --label efi --part-type EF00 --align 1024

This and the other similar one could probably use a "--use-uuid". So
that we could also boot this in qemu where mmcblk1 would become vda.

Henning

> -part /
> --source rootfs --ondisk mmcblk1 --fstype ext4 --mkfs-extraopts "-T
> default" --label platform --align 1024 --use-uuid +part / --source
> rootfs --ondisk mmcblk1 --fstype ext4 --mkfs-extraopts "-T default"
> --label platform --align 1024 --use-uuid --exclude-path boot/
> bootloader --ptable gpt --timeout=3 diff --git
> a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
> b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks index
> be468430a7d6..f3addbc7515d 100644 ---
> a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks +++
> b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks @@ -2,8 +2,8
> @@ # long-description: Creates a partitioned EFI disk image without
> any swap that # the user can directly dd to boot media. -part
> --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk
> sda --label efi --part-type EF00 --align 1024 +part /boot --source
> bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk sda
> --label efi --part-type EF00 --align 1024 -part / --source rootfs
> --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label
> platform --align 1024 --use-uuid +part / --source rootfs --ondisk sda
> --fstype ext4 --mkfs-extraopts "-T default" --label platform --align
> 1024 --use-uuid --exclude-path boot/ bootloader --ptable gpt
> --timeout 3 --append "rootwait console=ttyS0,115200 console=tty0"

Re: [PATCH v2 0/7] re-fork wic pcbios and efi plugins

[Jan Kiszka]

On 03.09.21 14:53, Henning Schild wrote:
> changes since v1:
>   - efi plugin forked as well
>   - systemd-boot support in efi plugin enabled
>   - common functionality in utility library
>   - test case for system-boot
>   - "cp -a" moved to "find exec cp" because of ubuntu
>   - changed wks files to exclude boot from root and mount it
> 
> The forked plugins have gotten out of sync with the last wic version
> bumps. And the original fork was not exactly minimal or made for easy
> maintenance.
> 
> This series does a re-fork of the two plugins with the aim to come up
> with something readable, minimal and maintainable.
> 

I already pointed this out to Baurzhan: We should create some developer
doc somewhere that list the update procedures for forked elements. Which
files come from upstream? Which are supposed to be vanilla? Which
contain changes and should be revisited when upstream is updated? With
such a file, both maintainers and contributors can check what to do when
the want to update affected files / components or see patches touching them.

Maybe start that file with covering wic and its plugins?

> There used to be a special case for grub-efi where the actual kernel and
> initrd would remain in the root partition, which kind of allowed kernel
> updates with apt-get. Now all three bootloaders (systemd-boot now works
> as well) place bootloader, config and boot artifacts in a
> boot-partition.
> 
> Kernel updates with apt-get are now consistantly "broken". That
> consistency very likely is not too bad. A generic solution for this
> feature (if wanted) will need to be found. Covering not just these three
> bootloaders but possibly also u-boot and efibootguard.

I assume that this only applies to efi and pcbios images, not
u-boot-script or anything else that manages kernel updates more or less
properly already.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [PATCH v2 0/7] re-fork wic pcbios and efi plugins

[Henning Schild]

Am Mon, 6 Sep 2021 07:05:17 +0200
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 03.09.21 14:53, Henning Schild wrote:
> > changes since v1:
> >   - efi plugin forked as well
> >   - systemd-boot support in efi plugin enabled
> >   - common functionality in utility library
> >   - test case for system-boot
> >   - "cp -a" moved to "find exec cp" because of ubuntu
> >   - changed wks files to exclude boot from root and mount it
> > 
> > The forked plugins have gotten out of sync with the last wic version
> > bumps. And the original fork was not exactly minimal or made for
> > easy maintenance.
> > 
> > This series does a re-fork of the two plugins with the aim to come
> > up with something readable, minimal and maintainable.
> >   
> 
> I already pointed this out to Baurzhan: We should create some
> developer doc somewhere that list the update procedures for forked
> elements. Which files come from upstream? Which are supposed to be
> vanilla? Which contain changes and should be revisited when upstream
> is updated? With such a file, both maintainers and contributors can
> check what to do when the want to update affected files / components
> or see patches touching them.
> 
> Maybe start that file with covering wic and its plugins?

This is a good idea but i would not want to include a patch into this
series. Something to do on top.
The main reason is that we also have half-forked bits like the wks
files and especially meta/classes/wic-img.bbclass which probably has
not been checked in a long time. 

> > There used to be a special case for grub-efi where the actual
> > kernel and initrd would remain in the root partition, which kind of
> > allowed kernel updates with apt-get. Now all three bootloaders
> > (systemd-boot now works as well) place bootloader, config and boot
> > artifacts in a boot-partition.
> > 
> > Kernel updates with apt-get are now consistantly "broken". That
> > consistency very likely is not too bad. A generic solution for this
> > feature (if wanted) will need to be found. Covering not just these
> > three bootloaders but possibly also u-boot and efibootguard.  
> 
> I assume that this only applies to efi and pcbios images, not
> u-boot-script or anything else that manages kernel updates more or
> less properly already.

Right. The u-boot plugin is not a fork but our own. It might contain
copied bits but i did not look at it at all. And it is currently the
only bootloader bringing an update-script to generate bootloader conf
when updating the kernel with apt-get.

At a later point we can see if we can enable "apt-get update <kernel>"
bootloader indep. I think that should be possible very similar to how
we do things for u-boot.

But that is a story for the future, after this series has been merged.
I see it as a win to move from "mostly broken" to "consistently
broken". The only case that sometimes worked was grub ... depending on
which wic version and whether one would mount <boot> at /boot.

Henning

> Jan
> 

Re: [PATCH v2 0/7] re-fork wic pcbios and efi plugins

[Anton Mikanovich]

03.09.2021 15:53, Henning Schild wrote:
> changes since v1:
>    - efi plugin forked as well
>    - systemd-boot support in efi plugin enabled
>    - common functionality in utility library
>    - test case for system-boot
>    - "cp -a" moved to "find exec cp" because of ubuntu
>    - changed wks files to exclude boot from root and mount it

Should it actually be v3 with changes since v2?
There was v2 received already.

-- 
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH v2 0/7] re-fork wic pcbios and efi plugins

[Henning Schild]

Am Mon, 6 Sep 2021 12:48:12 +0300
schrieb Anton Mikanovich <amikan@ilbers.de>:

> 03.09.2021 15:53, Henning Schild wrote:
> > changes since v1:
> >    - efi plugin forked as well
> >    - systemd-boot support in efi plugin enabled
> >    - common functionality in utility library
> >    - test case for system-boot
> >    - "cp -a" moved to "find exec cp" because of ubuntu
> >    - changed wks files to exclude boot from root and mount it  
> 
> Should it actually be v3 with changes since v2?
> There was v2 received already.

Oh right. But i think that is not too bad of a mistake, since it does
not affect the code.
In case we need another run i will make that v4.

Henning

Re: [PATCH v2 0/7] re-fork wic pcbios and efi plugins

[Bezdeka, Florian]

On Fri, 2021-09-03 at 14:53 +0200, Henning Schild wrote:
> changes since v1:
>   - efi plugin forked as well
>   - systemd-boot support in efi plugin enabled
>   - common functionality in utility library
>   - test case for system-boot
>   - "cp -a" moved to "find exec cp" because of ubuntu
>   - changed wks files to exclude boot from root and mount it
> 
> The forked plugins have gotten out of sync with the last wic version
> bumps. And the original fork was not exactly minimal or made for easy
> maintenance.
> 
> This series does a re-fork of the two plugins with the aim to come up
> with something readable, minimal and maintainable.
> 
> There used to be a special case for grub-efi where the actual kernel and
> initrd would remain in the root partition, which kind of allowed kernel
> updates with apt-get. Now all three bootloaders (systemd-boot now works
> as well) place bootloader, config and boot artifacts in a
> boot-partition.
> 
> Kernel updates with apt-get are now consistantly "broken". That
> consistency very likely is not too bad. A generic solution for this
> feature (if wanted) will need to be found. Covering not just these three
> bootloaders but possibly also u-boot and efibootguard.

I had a quick look and asked myself if it would be possible to
implement the ISAR specific parts in sub-classes of such plugins. I
assume that would require some upstream work first, followed by a re-
sync and adding the ISAR stuff in plugins that inherit from the
upstream ones.

As far as I can see all affected plugins are python classes. I'm not a
python expert but I guess that should be doable and would completely
isolate the ISAR parts from the upstream stuff.
 
> 
> Henning Schild (7):
>   wic: reset our plugin forks to OE upstream for re-forking
>   wic: add utility library for common bits of isar plugins
>   wic: apply the actual fork changes to our pcbios plugin fork
>   wic: clean up wic class in terms of isar variables
>   wic: apply the actual fork changes to our efi plugin fork
>   wic: mount /boot and exlude it from root for efi
>   meta-isar: use "systemd-boot" for one test target
> 
>  RECIPE-API-CHANGELOG.md                       |   9 +
>  meta-isar/conf/machine/qemuamd64.conf         |   3 +
>  .../scripts/lib/wic/canned-wks/hikey.wks      |   4 +-
>  .../lib/wic/canned-wks/sdimage-efi-sd.wks     |   9 +
>  .../lib/wic/canned-wks/sdimage-efi.wks        |   4 +-
>  meta/classes/wic-img.bbclass                  |   6 +-
>  .../scripts/lib/wic/plugins/isarpluginbase.py |  39 ++++
>  .../wic/plugins/source/bootimg-efi-isar.py    | 200 ++++++++++++++----
>  .../wic/plugins/source/bootimg-pcbios-isar.py | 139 ++++++------
>  9 files changed, 290 insertions(+), 123 deletions(-)
>  create mode 100644 meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks
>  create mode 100644 meta/scripts/lib/wic/plugins/isarpluginbase.py
> 

Re: [PATCH v2 0/7] re-fork wic pcbios and efi plugins

[Henning Schild]

Am Tue, 7 Sep 2021 09:15:36 +0200
schrieb "Bezdeka, Florian (T RDA IOT SES-DE)"
<florian.bezdeka@siemens.com>:

> On Fri, 2021-09-03 at 14:53 +0200, Henning Schild wrote:
> > changes since v1:
> >   - efi plugin forked as well
> >   - systemd-boot support in efi plugin enabled
> >   - common functionality in utility library
> >   - test case for system-boot
> >   - "cp -a" moved to "find exec cp" because of ubuntu
> >   - changed wks files to exclude boot from root and mount it
> >
> > The forked plugins have gotten out of sync with the last wic version
> > bumps. And the original fork was not exactly minimal or made for
> > easy maintenance.
> >
> > This series does a re-fork of the two plugins with the aim to come
> > up with something readable, minimal and maintainable.
> >
> > There used to be a special case for grub-efi where the actual
> > kernel and initrd would remain in the root partition, which kind of
> > allowed kernel updates with apt-get. Now all three bootloaders
> > (systemd-boot now works as well) place bootloader, config and boot
> > artifacts in a boot-partition.
> >
> > Kernel updates with apt-get are now consistantly "broken". That
> > consistency very likely is not too bad. A generic solution for this
> > feature (if wanted) will need to be found. Covering not just these
> > three bootloaders but possibly also u-boot and efibootguard.  
> 
> I had a quick look and asked myself if it would be possible to
> implement the ISAR specific parts in sub-classes of such plugins. I
> assume that would require some upstream work first, followed by a re-
> sync and adding the ISAR stuff in plugins that inherit from the
> upstream ones.

My original goal was in fact to reduce it to only bits that can be
upstreamed. In general we need to hook into generation of the config,
because we use another name for that kernel binary and because we use
an initrd.

After that the installation of the binaries differs mainly in where
they come from. That looks really simple for systemd-boot, a little
more involved for syslinux and really very different for grub.

In addition the population of boot with kernel initrd and misc files
differs in isar.

I would not easily find a way to make that generic enough to try and
motivate OE to change those classes to be "more generic". It is in bits
just pretty different.

> As far as I can see all affected plugins are python classes. I'm not a
> python expert but I guess that should be doable and would completely
> isolate the ISAR parts from the upstream stuff.

Yes an no. Once forked they are only there to serve isar. Any kind of
"monkey patching" would likely require more code and still be
patching/forking. If even possible i would not know how.

As we can see in this series, a regular fork inherits useful bits from
the original. I was really surprised how easy it was to enable
systemd-boot. And i tend to prefer that over grub, because the
bootloader is installed from the rootfs and not buildchroot ... and is
therefore also part of the Bill Of Materials.

But yes, such a plugin just needs to implement the interface of such a
class. But it seems like deriving from the original results in way less
isar-specific code.

In fact we do have a fork of "bootimg-biosplusefi" in one of our
layers, that one is only about very little renaming. Basically calling
the efi and the pcbios plugins. That very likely only works so well
because our plugins are forked and so very similar.

I have nothing against further improvements in the future. But as it
stands this series remains with "forking" but doing so in a hopefully
easier to maintain way. And it aligns our forks with the originals.
IMHO that is worth merging and whether further steps will be needed in
the future remains to be seen.

The upstream classes do not change too often and not too dramatic. So
the biggest risk in isar is lack of disciple/care when bumping wic and
low test coverage. Both points that we might want to improve.
We have the suggestion from Jan to write a documents where we list
"forked" bits, and i have a bunch more patches to increase test
coverage. i.e. arm64 efi booting

https://github.com/henning-schild-work/isar/tree/henning/staging9

regards,
Henning

> >
> > Henning Schild (7):
> >   wic: reset our plugin forks to OE upstream for re-forking
> >   wic: add utility library for common bits of isar plugins
> >   wic: apply the actual fork changes to our pcbios plugin fork
> >   wic: clean up wic class in terms of isar variables
> >   wic: apply the actual fork changes to our efi plugin fork
> >   wic: mount /boot and exlude it from root for efi
> >   meta-isar: use "systemd-boot" for one test target
> >
> >  RECIPE-API-CHANGELOG.md                       |   9 +
> >  meta-isar/conf/machine/qemuamd64.conf         |   3 +
> >  .../scripts/lib/wic/canned-wks/hikey.wks      |   4 +-
> >  .../lib/wic/canned-wks/sdimage-efi-sd.wks     |   9 +
> >  .../lib/wic/canned-wks/sdimage-efi.wks        |   4 +-
> >  meta/classes/wic-img.bbclass                  |   6 +-
> >  .../scripts/lib/wic/plugins/isarpluginbase.py |  39 ++++
> >  .../wic/plugins/source/bootimg-efi-isar.py    | 200
> > ++++++++++++++---- .../wic/plugins/source/bootimg-pcbios-isar.py |
> > 139 ++++++------ 9 files changed, 290 insertions(+), 123
> > deletions(-) create mode 100644
> > meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks create mode
> > 100644 meta/scripts/lib/wic/plugins/isarpluginbase.py 
> 

Re: [PATCH v2 0/7] re-fork wic pcbios and efi plugins

[Anton Mikanovich]

03.09.2021 15:53, Henning Schild wrote:
> changes since v1:
>    - efi plugin forked as well
>    - systemd-boot support in efi plugin enabled
>    - common functionality in utility library
>    - test case for system-boot
>    - "cp -a" moved to "find exec cp" because of ubuntu
>    - changed wks files to exclude boot from root and mount it
>
> The forked plugins have gotten out of sync with the last wic version
> bumps. And the original fork was not exactly minimal or made for easy
> maintenance.
>
> This series does a re-fork of the two plugins with the aim to come up
> with something readable, minimal and maintainable.
>
> There used to be a special case for grub-efi where the actual kernel and
> initrd would remain in the root partition, which kind of allowed kernel
> updates with apt-get. Now all three bootloaders (systemd-boot now works
> as well) place bootloader, config and boot artifacts in a
> boot-partition.
>
> Kernel updates with apt-get are now consistantly "broken". That
> consistency very likely is not too bad. A generic solution for this
> feature (if wanted) will need to be found. Covering not just these three
> bootloaders but possibly also u-boot and efibootguard.
>
> Henning Schild (7):
>    wic: reset our plugin forks to OE upstream for re-forking
>    wic: add utility library for common bits of isar plugins
>    wic: apply the actual fork changes to our pcbios plugin fork
>    wic: clean up wic class in terms of isar variables
>    wic: apply the actual fork changes to our efi plugin fork
>    wic: mount /boot and exlude it from root for efi
>    meta-isar: use "systemd-boot" for one test target
>
>   RECIPE-API-CHANGELOG.md                       |   9 +
>   meta-isar/conf/machine/qemuamd64.conf         |   3 +
>   .../scripts/lib/wic/canned-wks/hikey.wks      |   4 +-
>   .../lib/wic/canned-wks/sdimage-efi-sd.wks     |   9 +
>   .../lib/wic/canned-wks/sdimage-efi.wks        |   4 +-
>   meta/classes/wic-img.bbclass                  |   6 +-
>   .../scripts/lib/wic/plugins/isarpluginbase.py |  39 ++++
>   .../wic/plugins/source/bootimg-efi-isar.py    | 200 ++++++++++++++----
>   .../wic/plugins/source/bootimg-pcbios-isar.py | 139 ++++++------
>   9 files changed, 290 insertions(+), 123 deletions(-)
>   create mode 100644 meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks
>   create mode 100644 meta/scripts/lib/wic/plugins/isarpluginbase.py
>
Applied to next, thanks.

-- 
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov