[PATCH v2 0/3] wic: Update to the latest revision

[PATCH v2 0/3] wic: Update to the latest revision

[Uladzimir Bely]

This patchset updates wic in isar to the OE-core revision b85a09e.
This is the latest revision that is compatible with bitbake
version in isar.
Before updating to even newer wic we need first to update bitbake.

Changes since v1:
 - Reverted Y2038 patch since it's already in the new wic
 - Updated isar wks files

Anton Mikanovich (1):
  wic: Update to the latest revision

Florian Bezdeka (1):
  Revert "wic: Warn if an ext filesystem affected by the Y2038 problem
    is used"

Uladzimir Bely (1):
  wic: Force mke2fs to use Y2038-compatible inode size

 .../scripts/lib/wic/canned-wks/bananapi.wks   |  2 +-
 .../lib/wic/canned-wks/common-isar.wks.inc    |  2 +-
 .../lib/wic/canned-wks/de0-nano-soc.wks.in    |  2 +-
 .../scripts/lib/wic/canned-wks/hikey.wks      |  2 +-
 .../lib/wic/canned-wks/nanopi-neo.wks.in      |  2 +-
 .../lib/wic/canned-wks/sdimage-efi.wks        |  2 +-
 .../lib/wic/canned-wks/sifive-fu540.wks       |  2 +-
 .../lib/wic/canned-wks/stm32mp15x.wks.in      |  2 +-
 scripts/lib/wic/canned-wks/common.wks.inc     |  2 +-
 scripts/lib/wic/canned-wks/directdisk-gpt.wks |  2 +-
 scripts/lib/wic/canned-wks/mkefidisk.wks      |  2 +-
 scripts/lib/wic/ksparser.py                   | 17 +++++
 scripts/lib/wic/misc.py                       |  8 +-
 scripts/lib/wic/partition.py                  | 74 +++++++++++++------
 scripts/lib/wic/plugins/imager/direct.py      | 61 ++++++++-------
 scripts/lib/wic/plugins/source/bootimg-efi.py |  8 +-
 scripts/lib/wic/plugins/source/empty.py       | 32 ++++++++
 .../wic/plugins/source/isoimage-isohybrid.py  | 14 ++++
 scripts/lib/wic/plugins/source/rootfs.py      | 33 +++++++--
 scripts/wic                                   |  2 +
 20 files changed, 195 insertions(+), 76 deletions(-)
 create mode 100644 scripts/lib/wic/plugins/source/empty.py

-- 
2.20.1

[PATCH v2 1/3] Revert "wic: Warn if an ext filesystem affected by the Y2038 problem is used"

[Uladzimir Bely]

From: Florian Bezdeka <florian.bezdeka@siemens.com>

This reverts commit aa5dd75a7314798bba5b413eea2886983449bd28.

Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 scripts/lib/wic/partition.py | 38 ------------------------------------
 1 file changed, 38 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 4a5a31e..85eb15c 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -270,8 +270,6 @@ class Partition():
         mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs)
         exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
 
-        self.check_for_Y2038_problem(rootfs, native_sysroot)
-
     def prepare_rootfs_btrfs(self, rootfs, oe_builddir, rootfs_dir,
                              native_sysroot, pseudo):
         """
@@ -358,8 +356,6 @@ class Partition():
             (self.fstype, extraopts, label_str, self.fsuuid, rootfs)
         exec_native_cmd(mkfs_cmd, native_sysroot)
 
-        self.check_for_Y2038_problem(rootfs, native_sysroot)
-
     def prepare_empty_partition_btrfs(self, rootfs, oe_builddir,
                                       native_sysroot):
         """
@@ -421,37 +417,3 @@ class Partition():
 
         mkswap_cmd = "mkswap %s -U %s %s" % (label_str, self.fsuuid, path)
         exec_native_cmd(mkswap_cmd, native_sysroot)
-
-    def check_for_Y2038_problem(self, rootfs, native_sysroot):
-        """
-        Check if the filesystem is affected by the Y2038 problem
-        (Y2038 problem = 32 bit time_t overflow in January 2038)
-        """
-        def get_err_str(part):
-            err = "The {} filesystem {} has no Y2038 support."
-            if part.mountpoint:
-                args = [part.fstype, "mounted at %s" % part.mountpoint]
-            elif part.label:
-                args = [part.fstype, "labeled %s" % part.label]
-            elif part.part_name:
-                args = [part.fstype, "in partition %s" % part.part_name]
-            else:
-                args = [part.fstype, ""]
-            return err.format(*args)
-
-        ret, out = exec_native_cmd("dumpe2fs %s" % rootfs, native_sysroot)
-
-        # ext2 and ext3 are always affected by the Y2038 problem
-        if self.fstype in ["ext2", "ext3"]:
-            logger.warn(get_err_str(self))
-            return
-
-        # if ext4 is affected by the Y2038 problem depends on the inode size
-        # Remember: inode size depends on the file system size
-        for line in out.splitlines():
-            if line.startswith("Inode size:"):
-                size = int(line.split(":")[1].strip())
-                if size < 256:
-                    logger.warn("%s Inodes (of size %d) are too small." % \
-                                (get_err_str(self), size))
-                break
\ No newline at end of file
-- 
2.20.1

[PATCH v2 2/3] wic: Update to the latest revision

[Uladzimir Bely]

From: Anton Mikanovich <amikan@ilbers.de>

Update to the latest wic from OE-core.

OE-core Revision: b85a09ea450a5e8f49418f4a930805fbb88dc83b

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 scripts/lib/wic/canned-wks/common.wks.inc     |  2 +-
 scripts/lib/wic/canned-wks/directdisk-gpt.wks |  2 +-
 scripts/lib/wic/canned-wks/mkefidisk.wks      |  2 +-
 scripts/lib/wic/ksparser.py                   | 17 ++++
 scripts/lib/wic/misc.py                       |  8 +-
 scripts/lib/wic/partition.py                  | 96 ++++++++++++++++---
 scripts/lib/wic/plugins/imager/direct.py      | 61 ++++++------
 scripts/lib/wic/plugins/source/bootimg-efi.py |  8 +-
 scripts/lib/wic/plugins/source/empty.py       | 32 +++++++
 .../wic/plugins/source/isoimage-isohybrid.py  | 14 +++
 scripts/lib/wic/plugins/source/rootfs.py      | 33 +++++--
 scripts/wic                                   |  2 +
 12 files changed, 217 insertions(+), 60 deletions(-)
 create mode 100644 scripts/lib/wic/plugins/source/empty.py

diff --git a/scripts/lib/wic/canned-wks/common.wks.inc b/scripts/lib/wic/canned-wks/common.wks.inc
index 89880b4..4fd29fa 100644
--- a/scripts/lib/wic/canned-wks/common.wks.inc
+++ b/scripts/lib/wic/canned-wks/common.wks.inc
@@ -1,3 +1,3 @@
 # This file is included into 3 canned wks files from this directory
 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
-part / --source rootfs --use-uuid --fstype=ext4 --label platform --align 1024
+part / --source rootfs --use-uuid --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024
diff --git a/scripts/lib/wic/canned-wks/directdisk-gpt.wks b/scripts/lib/wic/canned-wks/directdisk-gpt.wks
index 8d7d8de..cf16c0c 100644
--- a/scripts/lib/wic/canned-wks/directdisk-gpt.wks
+++ b/scripts/lib/wic/canned-wks/directdisk-gpt.wks
@@ -4,7 +4,7 @@
 
 
 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
-part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
+part / --source rootfs --ondisk sda --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid
 
 bootloader  --ptable gpt --timeout=0  --append="rootwait rootfstype=ext4 video=vesafb vga=0x318 console=tty0 console=ttyS0,115200n8"
 
diff --git a/scripts/lib/wic/canned-wks/mkefidisk.wks b/scripts/lib/wic/canned-wks/mkefidisk.wks
index 9f534fe..d1878e2 100644
--- a/scripts/lib/wic/canned-wks/mkefidisk.wks
+++ b/scripts/lib/wic/canned-wks/mkefidisk.wks
@@ -4,7 +4,7 @@
 
 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
 
-part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
+part / --source rootfs --ondisk sda --fstype=ext4 --mkfs-extraopts "-T default"  --label platform --align 1024 --use-uuid
 
 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
 
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 913e328..3eb669d 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -229,6 +229,23 @@ class KickStart():
                                 err = "%s:%d: SquashFS does not support LABEL" \
                                        % (confpath, lineno)
                                 raise KickStartError(err)
+                        if parsed.fstype == 'msdos' or parsed.fstype == 'vfat':
+                            if parsed.fsuuid:
+                                if parsed.fsuuid.upper().startswith('0X'):
+                                    if len(parsed.fsuuid) > 10:
+                                        err = "%s:%d: fsuuid %s given in wks kickstart file " \
+                                              "exceeds the length limit for %s filesystem. " \
+                                              "It should be in the form of a 32 bit hexadecimal" \
+                                              "number (for example, 0xABCD1234)." \
+                                              % (confpath, lineno, parsed.fsuuid, parsed.fstype)
+                                        raise KickStartError(err)
+                                elif len(parsed.fsuuid) > 8:
+                                    err = "%s:%d: fsuuid %s given in wks kickstart file " \
+                                          "exceeds the length limit for %s filesystem. " \
+                                          "It should be in the form of a 32 bit hexadecimal" \
+                                          "number (for example, 0xABCD1234)." \
+                                          % (confpath, lineno, parsed.fsuuid, parsed.fstype)
+                                    raise KickStartError(err)
                         if parsed.use_label and not parsed.label:
                             err = "%s:%d: Must set the label with --label" \
                                   % (confpath, lineno)
diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py
index 4b08d64..57c042c 100644
--- a/scripts/lib/wic/misc.py
+++ b/scripts/lib/wic/misc.py
@@ -26,6 +26,7 @@ logger = logging.getLogger('wic')
 
 # executable -> recipe pairs for exec_native_cmd
 NATIVE_RECIPES = {"bmaptool": "bmap-tools",
+                  "dumpe2fs": "e2fsprogs",
                   "grub-mkimage": "grub-efi",
                   "isohybrid": "syslinux",
                   "mcopy": "mtools",
@@ -138,9 +139,12 @@ def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
     if pseudo:
         cmd_and_args = pseudo + cmd_and_args
 
-    native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin" % \
+    hosttools_dir = get_bitbake_var("HOSTTOOLS_DIR")
+
+    native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin:%s" % \
                    (native_sysroot, native_sysroot,
-                    native_sysroot, native_sysroot)
+                    native_sysroot, native_sysroot,
+                    hosttools_dir)
 
     native_cmd_and_args = "export PATH=%s:$PATH;%s" % \
                    (native_paths, cmd_and_args)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 85eb15c..76d144d 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -54,6 +54,9 @@ class Partition():
         self.uuid = args.uuid
         self.fsuuid = args.fsuuid
         self.type = args.type
+        self.updated_fstab_path = None
+        self.has_fstab = False
+        self.update_fstab_in_rootfs = False
 
         self.lineno = lineno
         self.source_file = ""
@@ -118,11 +121,15 @@ class Partition():
         return self.fixed_size if self.fixed_size else self.size
 
     def prepare(self, creator, cr_workdir, oe_builddir, rootfs_dir,
-                bootimg_dir, kernel_dir, native_sysroot):
+                bootimg_dir, kernel_dir, native_sysroot, updated_fstab_path):
         """
         Prepare content for individual partitions, depending on
         partition command parameters.
         """
+        self.updated_fstab_path = updated_fstab_path
+        if self.updated_fstab_path and not (self.fstype.startswith("ext") or self.fstype == "msdos"):
+            self.update_fstab_in_rootfs = True
+
         if not self.source:
             if not self.size and not self.fixed_size:
                 raise WicError("The %s partition has a size of zero. Please "
@@ -199,21 +206,33 @@ class Partition():
 
         Currently handles ext2/3/4, btrfs, vfat and squashfs.
         """
+
+        rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
+                                         self.lineno, self.fstype)
+        if os.path.isfile(rootfs):
+            os.remove(rootfs)
+
         p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
         if (pseudo_dir):
+            # Canonicalize the ignore paths. This corresponds to
+            # calling oe.path.canonicalize(), which is used in bitbake.conf.
+            ignore_paths = [rootfs] + (get_bitbake_var("PSEUDO_IGNORE_PATHS") or "").split(",")
+            canonical_paths = []
+            for path in ignore_paths:
+                if "$" not in path:
+                    trailing_slash = path.endswith("/") and "/" or ""
+                    canonical_paths.append(os.path.realpath(path) + trailing_slash)
+            ignore_paths = ",".join(canonical_paths)
+
             pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
             pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
             pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
             pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+            pseudo += "export PSEUDO_IGNORE_PATHS=%s;" % ignore_paths
             pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
         else:
             pseudo = None
 
-        rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
-                                         self.lineno, self.fstype)
-        if os.path.isfile(rootfs):
-            os.remove(rootfs)
-
         if not self.size and real_rootfs:
             # The rootfs size is not set in .ks file so try to get it
             # from bitbake variable
@@ -235,7 +254,7 @@ class Partition():
 
         prefix = "ext" if self.fstype.startswith("ext") else self.fstype
         method = getattr(self, "prepare_rootfs_" + prefix)
-        method(rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo)
+        method(rootfs, cr_workdir, oe_builddir, rootfs_dir, native_sysroot, pseudo)
         self.source_file = rootfs
 
         # get the rootfs size in the right units for kickstart (kB)
@@ -243,7 +262,7 @@ class Partition():
         out = exec_cmd(du_cmd)
         self.size = int(out.split()[0])
 
-    def prepare_rootfs_ext(self, rootfs, oe_builddir, rootfs_dir,
+    def prepare_rootfs_ext(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
                            native_sysroot, pseudo):
         """
         Prepare content for an ext2/3/4 rootfs partition.
@@ -267,10 +286,21 @@ class Partition():
             (self.fstype, extraopts, rootfs, label_str, self.fsuuid, rootfs_dir)
         exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
 
+        if self.updated_fstab_path and self.has_fstab:
+            debugfs_script_path = os.path.join(cr_workdir, "debugfs_script")
+            with open(debugfs_script_path, "w") as f:
+                f.write("cd etc\n")
+                f.write("rm fstab\n")
+                f.write("write %s fstab\n" % (self.updated_fstab_path))
+            debugfs_cmd = "debugfs -w -f %s %s" % (debugfs_script_path, rootfs)
+            exec_native_cmd(debugfs_cmd, native_sysroot)
+
         mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs)
         exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
 
-    def prepare_rootfs_btrfs(self, rootfs, oe_builddir, rootfs_dir,
+        self.check_for_Y2038_problem(rootfs, native_sysroot)
+
+    def prepare_rootfs_btrfs(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
                              native_sysroot, pseudo):
         """
         Prepare content for a btrfs rootfs partition.
@@ -293,7 +323,7 @@ class Partition():
              self.mkfs_extraopts, self.fsuuid, rootfs)
         exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
 
-    def prepare_rootfs_msdos(self, rootfs, oe_builddir, rootfs_dir,
+    def prepare_rootfs_msdos(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
                              native_sysroot, pseudo):
         """
         Prepare content for a msdos/vfat rootfs partition.
@@ -309,8 +339,6 @@ class Partition():
             label_str = "-n %s" % self.label
 
         size_str = ""
-        if self.fstype == 'msdos':
-            size_str = "-F 16" # FAT 16
 
         extraopts = self.mkfs_extraopts or '-S 512'
 
@@ -322,12 +350,16 @@ class Partition():
         mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir)
         exec_native_cmd(mcopy_cmd, native_sysroot)
 
+        if self.updated_fstab_path and self.has_fstab:
+            mcopy_cmd = "mcopy -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path)
+            exec_native_cmd(mcopy_cmd, native_sysroot)
+
         chmod_cmd = "chmod 644 %s" % rootfs
         exec_cmd(chmod_cmd)
 
     prepare_rootfs_vfat = prepare_rootfs_msdos
 
-    def prepare_rootfs_squashfs(self, rootfs, oe_builddir, rootfs_dir,
+    def prepare_rootfs_squashfs(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
                                 native_sysroot, pseudo):
         """
         Prepare content for a squashfs rootfs partition.
@@ -356,6 +388,8 @@ class Partition():
             (self.fstype, extraopts, label_str, self.fsuuid, rootfs)
         exec_native_cmd(mkfs_cmd, native_sysroot)
 
+        self.check_for_Y2038_problem(rootfs, native_sysroot)
+
     def prepare_empty_partition_btrfs(self, rootfs, oe_builddir,
                                       native_sysroot):
         """
@@ -386,8 +420,6 @@ class Partition():
             label_str = "-n %s" % self.label
 
         size_str = ""
-        if self.fstype == 'msdos':
-            size_str = "-F 16" # FAT 16
 
         extraopts = self.mkfs_extraopts or '-S 512'
 
@@ -417,3 +449,37 @@ class Partition():
 
         mkswap_cmd = "mkswap %s -U %s %s" % (label_str, self.fsuuid, path)
         exec_native_cmd(mkswap_cmd, native_sysroot)
+
+    def check_for_Y2038_problem(self, rootfs, native_sysroot):
+        """
+        Check if the filesystem is affected by the Y2038 problem
+        (Y2038 problem = 32 bit time_t overflow in January 2038)
+        """
+        def get_err_str(part):
+            err = "The {} filesystem {} has no Y2038 support."
+            if part.mountpoint:
+                args = [part.fstype, "mounted at %s" % part.mountpoint]
+            elif part.label:
+                args = [part.fstype, "labeled '%s'" % part.label]
+            elif part.part_name:
+                args = [part.fstype, "in partition '%s'" % part.part_name]
+            else:
+                args = [part.fstype, "in partition %s" % part.num]
+            return err.format(*args)
+
+        # ext2 and ext3 are always affected by the Y2038 problem
+        if self.fstype in ["ext2", "ext3"]:
+            logger.warn(get_err_str(self))
+            return
+
+        ret, out = exec_native_cmd("dumpe2fs %s" % rootfs, native_sysroot)
+
+        # if ext4 is affected by the Y2038 problem depends on the inode size
+        for line in out.splitlines():
+            if line.startswith("Inode size:"):
+                size = int(line.split(":")[1].strip())
+                if size < 256:
+                    logger.warn("%s Inodes (of size %d) are too small." %
+                                (get_err_str(self), size))
+                break
+
diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index 55db826..ea709e8 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -54,15 +54,16 @@ class DirectPlugin(ImagerPlugin):
         self.native_sysroot = native_sysroot
         self.oe_builddir = oe_builddir
 
+        self.debug = options.debug
         self.outdir = options.outdir
         self.compressor = options.compressor
         self.bmap = options.bmap
         self.no_fstab_update = options.no_fstab_update
-        self.original_fstab = None
+        self.updated_fstab_path = None
 
         self.name = "%s-%s" % (os.path.splitext(os.path.basename(wks_file))[0],
                                strftime("%Y%m%d%H%M"))
-        self.workdir = tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.')
+        self.workdir = self.setup_workdir(options.workdir)
         self._image = None
         self.ptable_format = self.ks.bootloader.ptable
         self.parts = self.ks.partitions
@@ -78,6 +79,16 @@ class DirectPlugin(ImagerPlugin):
         self._image = PartitionedImage(image_path, self.ptable_format,
                                        self.parts, self.native_sysroot)
 
+    def setup_workdir(self, workdir):
+        if workdir:
+            if os.path.exists(workdir):
+                raise WicError("Internal workdir '%s' specified in wic arguments already exists!" % (workdir))
+
+            os.makedirs(workdir)
+            return workdir
+        else:
+            return tempfile.mkdtemp(dir=self.outdir, prefix='tmp.wic.')
+
     def do_create(self):
         """
         Plugin entry point.
@@ -90,11 +101,8 @@ class DirectPlugin(ImagerPlugin):
         finally:
             self.cleanup()
 
-    def _write_fstab(self, image_rootfs):
-        """overriden to generate fstab (temporarily) in rootfs. This is called
-        from _create, make sure it doesn't get called from
-        BaseImage.create()
-        """
+    def update_fstab(self, image_rootfs):
+        """Assume partition order same as in wks"""
         if not image_rootfs:
             return
 
@@ -104,18 +112,9 @@ class DirectPlugin(ImagerPlugin):
 
         with open(fstab_path) as fstab:
             fstab_lines = fstab.readlines()
-            self.original_fstab = fstab_lines.copy()
-
-        if self._update_fstab(fstab_lines, self.parts):
-            with open(fstab_path, "w") as fstab:
-                fstab.writelines(fstab_lines)
-        else:
-            self.original_fstab = None
 
-    def _update_fstab(self, fstab_lines, parts):
-        """Assume partition order same as in wks"""
         updated = False
-        for part in parts:
+        for part in self.parts:
             if not part.realnum or not part.mountpoint \
                or part.mountpoint == "/":
                 continue
@@ -144,7 +143,10 @@ class DirectPlugin(ImagerPlugin):
             fstab_lines.append(line)
             updated = True
 
-        return updated
+        if updated:
+            self.updated_fstab_path = os.path.join(self.workdir, "fstab")
+            with open(self.updated_fstab_path, "w") as f:
+                f.writelines(fstab_lines)
 
     def _full_path(self, path, name, extention):
         """ Construct full file path to a file we generate. """
@@ -160,7 +162,7 @@ class DirectPlugin(ImagerPlugin):
         a partitioned image.
         """
         if not self.no_fstab_update:
-            self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
+            self.update_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
 
         for part in self.parts:
             # get rootfs size from bitbake variable if it's not set in .ks file
@@ -273,14 +275,9 @@ class DirectPlugin(ImagerPlugin):
             if os.path.isfile(path):
                 shutil.move(path, os.path.join(self.outdir, fname))
 
-        #Restore original fstab
-        if self.original_fstab:
-            fstab_path = self.rootfs_dir.get("ROOTFS_DIR") + "/etc/fstab"
-            with open(fstab_path, "w") as fstab:
-                fstab.writelines(self.original_fstab)
-
-        # remove work directory
-        shutil.rmtree(self.workdir, ignore_errors=True)
+        # remove work directory when it is not in debugging mode
+        if not self.debug:
+            shutil.rmtree(self.workdir, ignore_errors=True)
 
 # Overhead of the MBR partitioning scheme (just one sector)
 MBR_OVERHEAD = 1
@@ -343,6 +340,13 @@ class PartitionedImage():
                     part.fsuuid = '0x' + str(uuid.uuid4())[:8].upper()
                 else:
                     part.fsuuid = str(uuid.uuid4())
+            else:
+                #make sure the fsuuid for vfat/msdos align with format 0xYYYYYYYY
+                if part.fstype == 'vfat' or part.fstype == 'msdos':
+                    if part.fsuuid.upper().startswith("0X"):
+                        part.fsuuid = '0x' + part.fsuuid.upper()[2:].rjust(8,"0")
+                    else:
+                        part.fsuuid = '0x' + part.fsuuid.upper().rjust(8,"0")
 
     def prepare(self, imager):
         """Prepare an image. Call prepare method of all image partitions."""
@@ -351,7 +355,8 @@ class PartitionedImage():
             # sizes before we can add them and do the layout.
             part.prepare(imager, imager.workdir, imager.oe_builddir,
                          imager.rootfs_dir, imager.bootimg_dir,
-                         imager.kernel_dir, imager.native_sysroot)
+                         imager.kernel_dir, imager.native_sysroot,
+                         imager.updated_fstab_path)
 
             # Converting kB to sectors for parted
             part.size_sec = part.disk_size * 1024 // self.sector_size
diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
index 14c1723..cdc7254 100644
--- a/scripts/lib/wic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
@@ -212,8 +212,8 @@ class BootimgEFIPlugin(SourcePlugin):
         except KeyError:
             raise WicError("bootimg-efi requires a loader, none specified")
 
-        if get_bitbake_var("IMAGE_BOOT_FILES") is None:
-            logger.debug('No boot files defined in IMAGE_BOOT_FILES')
+        if get_bitbake_var("IMAGE_EFI_BOOT_FILES") is None:
+            logger.debug('No boot files defined in IMAGE_EFI_BOOT_FILES')
         else:
             boot_files = None
             for (fmt, id) in (("_uuid-%s", part.uuid), ("_label-%s", part.label), (None, None)):
@@ -222,7 +222,7 @@ class BootimgEFIPlugin(SourcePlugin):
                 else:
                     var = ""
 
-                boot_files = get_bitbake_var("IMAGE_BOOT_FILES" + var)
+                boot_files = get_bitbake_var("IMAGE_EFI_BOOT_FILES" + var)
                 if boot_files:
                     break
 
@@ -292,7 +292,7 @@ class BootimgEFIPlugin(SourcePlugin):
             (staging_kernel_dir, kernel, hdddir, kernel)
         exec_cmd(install_cmd)
 
-        if get_bitbake_var("IMAGE_BOOT_FILES"):
+        if get_bitbake_var("IMAGE_EFI_BOOT_FILES"):
             for src_path, dst_path in cls.install_task:
                 install_cmd = "install -m 0644 -D %s %s" \
                               % (os.path.join(kernel_dir, src_path),
diff --git a/scripts/lib/wic/plugins/source/empty.py b/scripts/lib/wic/plugins/source/empty.py
new file mode 100644
index 0000000..041617d
--- /dev/null
+++ b/scripts/lib/wic/plugins/source/empty.py
@@ -0,0 +1,32 @@
+#
+# SPDX-License-Identifier: MIT
+#
+
+# The empty wic plugin is used to create unformatted empty partitions for wic
+# images.
+# To use it you must pass "empty" as argument for the "--source" parameter in
+# the wks file. For example:
+# part foo --source empty --ondisk sda --size="1024" --align 1024
+
+import logging
+
+from wic.pluginbase import SourcePlugin
+
+logger = logging.getLogger('wic')
+
+class EmptyPartitionPlugin(SourcePlugin):
+    """
+    Populate unformatted empty partition.
+    """
+
+    name = 'empty'
+
+    @classmethod
+    def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
+                             oe_builddir, bootimg_dir, kernel_dir,
+                             rootfs_dir, native_sysroot):
+        """
+        Called to do the actual content population for a partition i.e. it
+        'prepares' the partition to be incorporated into the image.
+        """
+        return
diff --git a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
index 11326a2..afc9ea0 100644
--- a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
+++ b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
@@ -216,6 +216,18 @@ class IsoImagePlugin(SourcePlugin):
             creator.name = source_params['image_name'].strip()
             logger.debug("The name of the image is: %s", creator.name)
 
+    @staticmethod
+    def _install_payload(source_params, iso_dir):
+        """
+        Copies contents of payload directory (as specified in 'payload_dir' param) into iso_dir
+        """
+
+        if source_params.get('payload_dir'):
+            payload_dir = source_params['payload_dir']
+
+            logger.debug("Payload directory: %s", payload_dir)
+            shutil.copytree(payload_dir, iso_dir, symlinks=True, dirs_exist_ok=True)
+
     @classmethod
     def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
                              oe_builddir, bootimg_dir, kernel_dir,
@@ -228,6 +240,8 @@ class IsoImagePlugin(SourcePlugin):
 
         isodir = "%s/ISO" % cr_workdir
 
+        cls._install_payload(source_params, isodir)
+
         if part.rootfs_dir is None:
             if not 'ROOTFS_DIR' in rootfs_dir:
                 raise WicError("Couldn't find --rootfs-dir, exiting.")
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index f1db83f..96d940a 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -94,6 +94,7 @@ class RootfsPlugin(SourcePlugin):
                                "it is not a valid path, exiting" % part.rootfs_dir)
 
         part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
+        part.has_fstab = os.path.exists(os.path.join(part.rootfs_dir, "etc/fstab"))
         pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo")
         if not os.path.lexists(pseudo_dir):
             logger.warn("%s folder does not exist. "
@@ -103,9 +104,9 @@ class RootfsPlugin(SourcePlugin):
         new_rootfs = None
         new_pseudo = None
         # Handle excluded paths.
-        if part.exclude_path or part.include_path or part.change_directory:
-            # We need a new rootfs directory we can delete files from. Copy to
-            # workdir.
+        if part.exclude_path or part.include_path or part.change_directory or part.update_fstab_in_rootfs:
+            # We need a new rootfs directory we can safely modify without
+            # interfering with other tasks. Copy to workdir.
             new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
 
             if os.path.lexists(new_rootfs):
@@ -199,17 +200,33 @@ class RootfsPlugin(SourcePlugin):
                 if not os.path.lexists(full_path):
                     continue
 
+                if new_pseudo:
+                    pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo)
+                else:
+                    pseudo = None
                 if path.endswith(os.sep):
                     # Delete content only.
                     for entry in os.listdir(full_path):
                         full_entry = os.path.join(full_path, entry)
-                        if os.path.isdir(full_entry) and not os.path.islink(full_entry):
-                            shutil.rmtree(full_entry)
-                        else:
-                            os.remove(full_entry)
+                        rm_cmd = "rm -rf %s" % (full_entry)
+                        exec_native_cmd(rm_cmd, native_sysroot, pseudo)
                 else:
                     # Delete whole directory.
-                    shutil.rmtree(full_path)
+                    rm_cmd = "rm -rf %s" % (full_path)
+                    exec_native_cmd(rm_cmd, native_sysroot, pseudo)
+
+            # Update part.has_fstab here as fstab may have been added or
+            # removed by the above modifications.
+            part.has_fstab = os.path.exists(os.path.join(new_rootfs, "etc/fstab"))
+            if part.update_fstab_in_rootfs and part.has_fstab:
+                fstab_path = os.path.join(new_rootfs, "etc/fstab")
+                # Assume that fstab should always be owned by root with fixed permissions
+                install_cmd = "install -m 0644 %s %s" % (part.updated_fstab_path, fstab_path)
+                if new_pseudo:
+                    pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo)
+                else:
+                    pseudo = None
+                exec_native_cmd(install_cmd, native_sysroot, pseudo)
 
         part.prepare_rootfs(cr_workdir, oe_builddir,
                             new_rootfs or part.rootfs_dir, native_sysroot,
diff --git a/scripts/wic b/scripts/wic
index 24700f3..a741aed 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -312,6 +312,8 @@ def wic_init_parser_create(subparser):
 
     subparser.add_argument("-o", "--outdir", dest="outdir", default='.',
                       help="name of directory to create image in")
+    subparser.add_argument("-w", "--workdir",
+                      help="temporary workdir to use for intermediate files")
     subparser.add_argument("-e", "--image-name", dest="image_name",
                       help="name of the image to use the artifacts from "
                            "e.g. core-image-sato")
-- 
2.20.1

Re: [PATCH v2 2/3] wic: Update to the latest revision

[Henning Schild]

Am Tue, 29 Jun 2021 11:22:24 +0200
schrieb Uladzimir Bely <ubely@ilbers.de>:

> From: Anton Mikanovich <amikan@ilbers.de>
> 
> Update to the latest wic from OE-core.
> 
> OE-core Revision: b85a09ea450a5e8f49418f4a930805fbb88dc83b
> 
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  scripts/lib/wic/canned-wks/common.wks.inc     |  2 +-
>  scripts/lib/wic/canned-wks/directdisk-gpt.wks |  2 +-
>  scripts/lib/wic/canned-wks/mkefidisk.wks      |  2 +-
>  scripts/lib/wic/ksparser.py                   | 17 ++++
>  scripts/lib/wic/misc.py                       |  8 +-
>  scripts/lib/wic/partition.py                  | 96
> ++++++++++++++++--- scripts/lib/wic/plugins/imager/direct.py      |
> 61 ++++++------ scripts/lib/wic/plugins/source/bootimg-efi.py |  8 +-

Our fork of that plugin now needs to sync up again to only differ in
the parts we need it to differ in.

meta/scripts/wic/plugins/source/bootimg-efi-isar.py

Henning

>  scripts/lib/wic/plugins/source/empty.py       | 32 +++++++
>  .../wic/plugins/source/isoimage-isohybrid.py  | 14 +++
>  scripts/lib/wic/plugins/source/rootfs.py      | 33 +++++--
>  scripts/wic                                   |  2 +
>  12 files changed, 217 insertions(+), 60 deletions(-)
>  create mode 100644 scripts/lib/wic/plugins/source/empty.py
> 
> diff --git a/scripts/lib/wic/canned-wks/common.wks.inc
> b/scripts/lib/wic/canned-wks/common.wks.inc index 89880b4..4fd29fa
> 100644 --- a/scripts/lib/wic/canned-wks/common.wks.inc
> +++ b/scripts/lib/wic/canned-wks/common.wks.inc
> @@ -1,3 +1,3 @@
>  # This file is included into 3 canned wks files from this directory
>  part /boot --source bootimg-pcbios --ondisk sda --label boot
> --active --align 1024 -part / --source rootfs --use-uuid
> --fstype=ext4 --label platform --align 1024 +part / --source rootfs
> --use-uuid --fstype=ext4 --mkfs-extraopts "-T default" --label
> platform --align 1024 diff --git
> a/scripts/lib/wic/canned-wks/directdisk-gpt.wks
> b/scripts/lib/wic/canned-wks/directdisk-gpt.wks index
> 8d7d8de..cf16c0c 100644 ---
> a/scripts/lib/wic/canned-wks/directdisk-gpt.wks +++
> b/scripts/lib/wic/canned-wks/directdisk-gpt.wks @@ -4,7 +4,7 @@ 
>  part /boot --source bootimg-pcbios --ondisk sda --label boot
> --active --align 1024 -part / --source rootfs --ondisk sda
> --fstype=ext4 --label platform --align 1024 --use-uuid +part /
> --source rootfs --ondisk sda --fstype=ext4 --mkfs-extraopts "-T
> default" --label platform --align 1024 --use-uuid bootloader
> --ptable gpt --timeout=0  --append="rootwait rootfstype=ext4
> video=vesafb vga=0x318 console=tty0 console=ttyS0,115200n8" diff
> --git a/scripts/lib/wic/canned-wks/mkefidisk.wks
> b/scripts/lib/wic/canned-wks/mkefidisk.wks index 9f534fe..d1878e2
> 100644 --- a/scripts/lib/wic/canned-wks/mkefidisk.wks +++
> b/scripts/lib/wic/canned-wks/mkefidisk.wks @@ -4,7 +4,7 @@
>  
>  part /boot --source bootimg-efi --sourceparams="loader=grub-efi"
> --ondisk sda --label msdos --active --align 1024 
> -part / --source rootfs --ondisk sda --fstype=ext4 --label platform
> --align 1024 --use-uuid +part / --source rootfs --ondisk sda
> --fstype=ext4 --mkfs-extraopts "-T default"  --label platform --align
> 1024 --use-uuid part swap --ondisk sda --size 44 --label swap1
> --fstype=swap 
> diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> index 913e328..3eb669d 100644
> --- a/scripts/lib/wic/ksparser.py
> +++ b/scripts/lib/wic/ksparser.py
> @@ -229,6 +229,23 @@ class KickStart():
>                                  err = "%s:%d: SquashFS does not
> support LABEL" \ % (confpath, lineno)
>                                  raise KickStartError(err)
> +                        if parsed.fstype == 'msdos' or parsed.fstype
> == 'vfat':
> +                            if parsed.fsuuid:
> +                                if
> parsed.fsuuid.upper().startswith('0X'):
> +                                    if len(parsed.fsuuid) > 10:
> +                                        err = "%s:%d: fsuuid %s
> given in wks kickstart file " \
> +                                              "exceeds the length
> limit for %s filesystem. " \
> +                                              "It should be in the
> form of a 32 bit hexadecimal" \
> +                                              "number (for example,
> 0xABCD1234)." \
> +                                              % (confpath, lineno,
> parsed.fsuuid, parsed.fstype)
> +                                        raise KickStartError(err)
> +                                elif len(parsed.fsuuid) > 8:
> +                                    err = "%s:%d: fsuuid %s given in
> wks kickstart file " \
> +                                          "exceeds the length limit
> for %s filesystem. " \
> +                                          "It should be in the form
> of a 32 bit hexadecimal" \
> +                                          "number (for example,
> 0xABCD1234)." \
> +                                          % (confpath, lineno,
> parsed.fsuuid, parsed.fstype)
> +                                    raise KickStartError(err)
>                          if parsed.use_label and not parsed.label:
>                              err = "%s:%d: Must set the label with
> --label" \ % (confpath, lineno)
> diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py
> index 4b08d64..57c042c 100644
> --- a/scripts/lib/wic/misc.py
> +++ b/scripts/lib/wic/misc.py
> @@ -26,6 +26,7 @@ logger = logging.getLogger('wic')
>  
>  # executable -> recipe pairs for exec_native_cmd
>  NATIVE_RECIPES = {"bmaptool": "bmap-tools",
> +                  "dumpe2fs": "e2fsprogs",
>                    "grub-mkimage": "grub-efi",
>                    "isohybrid": "syslinux",
>                    "mcopy": "mtools",
> @@ -138,9 +139,12 @@ def exec_native_cmd(cmd_and_args,
> native_sysroot, pseudo=""): if pseudo:
>          cmd_and_args = pseudo + cmd_and_args
>  
> -    native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin" % \
> +    hosttools_dir = get_bitbake_var("HOSTTOOLS_DIR")
> +
> +    native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin:%s" % \
>                     (native_sysroot, native_sysroot,
> -                    native_sysroot, native_sysroot)
> +                    native_sysroot, native_sysroot,
> +                    hosttools_dir)
>  
>      native_cmd_and_args = "export PATH=%s:$PATH;%s" % \
>                     (native_paths, cmd_and_args)
> diff --git a/scripts/lib/wic/partition.py
> b/scripts/lib/wic/partition.py index 85eb15c..76d144d 100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -54,6 +54,9 @@ class Partition():
>          self.uuid = args.uuid
>          self.fsuuid = args.fsuuid
>          self.type = args.type
> +        self.updated_fstab_path = None
> +        self.has_fstab = False
> +        self.update_fstab_in_rootfs = False
>  
>          self.lineno = lineno
>          self.source_file = ""
> @@ -118,11 +121,15 @@ class Partition():
>          return self.fixed_size if self.fixed_size else self.size
>  
>      def prepare(self, creator, cr_workdir, oe_builddir, rootfs_dir,
> -                bootimg_dir, kernel_dir, native_sysroot):
> +                bootimg_dir, kernel_dir, native_sysroot,
> updated_fstab_path): """
>          Prepare content for individual partitions, depending on
>          partition command parameters.
>          """
> +        self.updated_fstab_path = updated_fstab_path
> +        if self.updated_fstab_path and not
> (self.fstype.startswith("ext") or self.fstype == "msdos"):
> +            self.update_fstab_in_rootfs = True
> +
>          if not self.source:
>              if not self.size and not self.fixed_size:
>                  raise WicError("The %s partition has a size of zero.
> Please " @@ -199,21 +206,33 @@ class Partition():
>  
>          Currently handles ext2/3/4, btrfs, vfat and squashfs.
>          """
> +
> +        rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
> +                                         self.lineno, self.fstype)
> +        if os.path.isfile(rootfs):
> +            os.remove(rootfs)
> +
>          p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" %
> native_sysroot) if (pseudo_dir):
> +            # Canonicalize the ignore paths. This corresponds to
> +            # calling oe.path.canonicalize(), which is used in
> bitbake.conf.
> +            ignore_paths = [rootfs] +
> (get_bitbake_var("PSEUDO_IGNORE_PATHS") or "").split(",")
> +            canonical_paths = []
> +            for path in ignore_paths:
> +                if "$" not in path:
> +                    trailing_slash = path.endswith("/") and "/" or ""
> +                    canonical_paths.append(os.path.realpath(path) +
> trailing_slash)
> +            ignore_paths = ",".join(canonical_paths)
> +
>              pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
>              pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
>              pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
>              pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
> +            pseudo += "export PSEUDO_IGNORE_PATHS=%s;" % ignore_paths
>              pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
>          else:
>              pseudo = None
>  
> -        rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
> -                                         self.lineno, self.fstype)
> -        if os.path.isfile(rootfs):
> -            os.remove(rootfs)
> -
>          if not self.size and real_rootfs:
>              # The rootfs size is not set in .ks file so try to get it
>              # from bitbake variable
> @@ -235,7 +254,7 @@ class Partition():
>  
>          prefix = "ext" if self.fstype.startswith("ext") else
> self.fstype method = getattr(self, "prepare_rootfs_" + prefix)
> -        method(rootfs, oe_builddir, rootfs_dir, native_sysroot,
> pseudo)
> +        method(rootfs, cr_workdir, oe_builddir, rootfs_dir,
> native_sysroot, pseudo) self.source_file = rootfs
>  
>          # get the rootfs size in the right units for kickstart (kB)
> @@ -243,7 +262,7 @@ class Partition():
>          out = exec_cmd(du_cmd)
>          self.size = int(out.split()[0])
>  
> -    def prepare_rootfs_ext(self, rootfs, oe_builddir, rootfs_dir,
> +    def prepare_rootfs_ext(self, rootfs, cr_workdir, oe_builddir,
> rootfs_dir, native_sysroot, pseudo):
>          """
>          Prepare content for an ext2/3/4 rootfs partition.
> @@ -267,10 +286,21 @@ class Partition():
>              (self.fstype, extraopts, rootfs, label_str, self.fsuuid,
> rootfs_dir) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
>  
> +        if self.updated_fstab_path and self.has_fstab:
> +            debugfs_script_path = os.path.join(cr_workdir,
> "debugfs_script")
> +            with open(debugfs_script_path, "w") as f:
> +                f.write("cd etc\n")
> +                f.write("rm fstab\n")
> +                f.write("write %s fstab\n" %
> (self.updated_fstab_path))
> +            debugfs_cmd = "debugfs -w -f %s %s" %
> (debugfs_script_path, rootfs)
> +            exec_native_cmd(debugfs_cmd, native_sysroot)
> +
>          mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs)
>          exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
>  
> -    def prepare_rootfs_btrfs(self, rootfs, oe_builddir, rootfs_dir,
> +        self.check_for_Y2038_problem(rootfs, native_sysroot)
> +
> +    def prepare_rootfs_btrfs(self, rootfs, cr_workdir, oe_builddir,
> rootfs_dir, native_sysroot, pseudo):
>          """
>          Prepare content for a btrfs rootfs partition.
> @@ -293,7 +323,7 @@ class Partition():
>               self.mkfs_extraopts, self.fsuuid, rootfs)
>          exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
>  
> -    def prepare_rootfs_msdos(self, rootfs, oe_builddir, rootfs_dir,
> +    def prepare_rootfs_msdos(self, rootfs, cr_workdir, oe_builddir,
> rootfs_dir, native_sysroot, pseudo):
>          """
>          Prepare content for a msdos/vfat rootfs partition.
> @@ -309,8 +339,6 @@ class Partition():
>              label_str = "-n %s" % self.label
>  
>          size_str = ""
> -        if self.fstype == 'msdos':
> -            size_str = "-F 16" # FAT 16
>  
>          extraopts = self.mkfs_extraopts or '-S 512'
>  
> @@ -322,12 +350,16 @@ class Partition():
>          mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir)
>          exec_native_cmd(mcopy_cmd, native_sysroot)
>  
> +        if self.updated_fstab_path and self.has_fstab:
> +            mcopy_cmd = "mcopy -i %s %s ::/etc/fstab" % (rootfs,
> self.updated_fstab_path)
> +            exec_native_cmd(mcopy_cmd, native_sysroot)
> +
>          chmod_cmd = "chmod 644 %s" % rootfs
>          exec_cmd(chmod_cmd)
>  
>      prepare_rootfs_vfat = prepare_rootfs_msdos
>  
> -    def prepare_rootfs_squashfs(self, rootfs, oe_builddir,
> rootfs_dir,
> +    def prepare_rootfs_squashfs(self, rootfs, cr_workdir,
> oe_builddir, rootfs_dir, native_sysroot, pseudo):
>          """
>          Prepare content for a squashfs rootfs partition.
> @@ -356,6 +388,8 @@ class Partition():
>              (self.fstype, extraopts, label_str, self.fsuuid, rootfs)
>          exec_native_cmd(mkfs_cmd, native_sysroot)
>  
> +        self.check_for_Y2038_problem(rootfs, native_sysroot)
> +
>      def prepare_empty_partition_btrfs(self, rootfs, oe_builddir,
>                                        native_sysroot):
>          """
> @@ -386,8 +420,6 @@ class Partition():
>              label_str = "-n %s" % self.label
>  
>          size_str = ""
> -        if self.fstype == 'msdos':
> -            size_str = "-F 16" # FAT 16
>  
>          extraopts = self.mkfs_extraopts or '-S 512'
>  
> @@ -417,3 +449,37 @@ class Partition():
>  
>          mkswap_cmd = "mkswap %s -U %s %s" % (label_str, self.fsuuid,
> path) exec_native_cmd(mkswap_cmd, native_sysroot)
> +
> +    def check_for_Y2038_problem(self, rootfs, native_sysroot):
> +        """
> +        Check if the filesystem is affected by the Y2038 problem
> +        (Y2038 problem = 32 bit time_t overflow in January 2038)
> +        """
> +        def get_err_str(part):
> +            err = "The {} filesystem {} has no Y2038 support."
> +            if part.mountpoint:
> +                args = [part.fstype, "mounted at %s" %
> part.mountpoint]
> +            elif part.label:
> +                args = [part.fstype, "labeled '%s'" % part.label]
> +            elif part.part_name:
> +                args = [part.fstype, "in partition '%s'" %
> part.part_name]
> +            else:
> +                args = [part.fstype, "in partition %s" % part.num]
> +            return err.format(*args)
> +
> +        # ext2 and ext3 are always affected by the Y2038 problem
> +        if self.fstype in ["ext2", "ext3"]:
> +            logger.warn(get_err_str(self))
> +            return
> +
> +        ret, out = exec_native_cmd("dumpe2fs %s" % rootfs,
> native_sysroot) +
> +        # if ext4 is affected by the Y2038 problem depends on the
> inode size
> +        for line in out.splitlines():
> +            if line.startswith("Inode size:"):
> +                size = int(line.split(":")[1].strip())
> +                if size < 256:
> +                    logger.warn("%s Inodes (of size %d) are too
> small." %
> +                                (get_err_str(self), size))
> +                break
> +
> diff --git a/scripts/lib/wic/plugins/imager/direct.py
> b/scripts/lib/wic/plugins/imager/direct.py index 55db826..ea709e8
> 100644 --- a/scripts/lib/wic/plugins/imager/direct.py
> +++ b/scripts/lib/wic/plugins/imager/direct.py
> @@ -54,15 +54,16 @@ class DirectPlugin(ImagerPlugin):
>          self.native_sysroot = native_sysroot
>          self.oe_builddir = oe_builddir
>  
> +        self.debug = options.debug
>          self.outdir = options.outdir
>          self.compressor = options.compressor
>          self.bmap = options.bmap
>          self.no_fstab_update = options.no_fstab_update
> -        self.original_fstab = None
> +        self.updated_fstab_path = None
>  
>          self.name = "%s-%s" %
> (os.path.splitext(os.path.basename(wks_file))[0],
> strftime("%Y%m%d%H%M"))
> -        self.workdir = tempfile.mkdtemp(dir=self.outdir,
> prefix='tmp.wic.')
> +        self.workdir = self.setup_workdir(options.workdir)
>          self._image = None
>          self.ptable_format = self.ks.bootloader.ptable
>          self.parts = self.ks.partitions
> @@ -78,6 +79,16 @@ class DirectPlugin(ImagerPlugin):
>          self._image = PartitionedImage(image_path,
> self.ptable_format, self.parts, self.native_sysroot)
>  
> +    def setup_workdir(self, workdir):
> +        if workdir:
> +            if os.path.exists(workdir):
> +                raise WicError("Internal workdir '%s' specified in
> wic arguments already exists!" % (workdir)) +
> +            os.makedirs(workdir)
> +            return workdir
> +        else:
> +            return tempfile.mkdtemp(dir=self.outdir,
> prefix='tmp.wic.') +
>      def do_create(self):
>          """
>          Plugin entry point.
> @@ -90,11 +101,8 @@ class DirectPlugin(ImagerPlugin):
>          finally:
>              self.cleanup()
>  
> -    def _write_fstab(self, image_rootfs):
> -        """overriden to generate fstab (temporarily) in rootfs. This
> is called
> -        from _create, make sure it doesn't get called from
> -        BaseImage.create()
> -        """
> +    def update_fstab(self, image_rootfs):
> +        """Assume partition order same as in wks"""
>          if not image_rootfs:
>              return
>  
> @@ -104,18 +112,9 @@ class DirectPlugin(ImagerPlugin):
>  
>          with open(fstab_path) as fstab:
>              fstab_lines = fstab.readlines()
> -            self.original_fstab = fstab_lines.copy()
> -
> -        if self._update_fstab(fstab_lines, self.parts):
> -            with open(fstab_path, "w") as fstab:
> -                fstab.writelines(fstab_lines)
> -        else:
> -            self.original_fstab = None
>  
> -    def _update_fstab(self, fstab_lines, parts):
> -        """Assume partition order same as in wks"""
>          updated = False
> -        for part in parts:
> +        for part in self.parts:
>              if not part.realnum or not part.mountpoint \
>                 or part.mountpoint == "/":
>                  continue
> @@ -144,7 +143,10 @@ class DirectPlugin(ImagerPlugin):
>              fstab_lines.append(line)
>              updated = True
>  
> -        return updated
> +        if updated:
> +            self.updated_fstab_path = os.path.join(self.workdir,
> "fstab")
> +            with open(self.updated_fstab_path, "w") as f:
> +                f.writelines(fstab_lines)
>  
>      def _full_path(self, path, name, extention):
>          """ Construct full file path to a file we generate. """
> @@ -160,7 +162,7 @@ class DirectPlugin(ImagerPlugin):
>          a partitioned image.
>          """
>          if not self.no_fstab_update:
> -            self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
> +            self.update_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
>  
>          for part in self.parts:
>              # get rootfs size from bitbake variable if it's not set
> in .ks file @@ -273,14 +275,9 @@ class DirectPlugin(ImagerPlugin):
>              if os.path.isfile(path):
>                  shutil.move(path, os.path.join(self.outdir, fname))
>  
> -        #Restore original fstab
> -        if self.original_fstab:
> -            fstab_path = self.rootfs_dir.get("ROOTFS_DIR") +
> "/etc/fstab"
> -            with open(fstab_path, "w") as fstab:
> -                fstab.writelines(self.original_fstab)
> -
> -        # remove work directory
> -        shutil.rmtree(self.workdir, ignore_errors=True)
> +        # remove work directory when it is not in debugging mode
> +        if not self.debug:
> +            shutil.rmtree(self.workdir, ignore_errors=True)
>  
>  # Overhead of the MBR partitioning scheme (just one sector)
>  MBR_OVERHEAD = 1
> @@ -343,6 +340,13 @@ class PartitionedImage():
>                      part.fsuuid = '0x' +
> str(uuid.uuid4())[:8].upper() else:
>                      part.fsuuid = str(uuid.uuid4())
> +            else:
> +                #make sure the fsuuid for vfat/msdos align with
> format 0xYYYYYYYY
> +                if part.fstype == 'vfat' or part.fstype == 'msdos':
> +                    if part.fsuuid.upper().startswith("0X"):
> +                        part.fsuuid = '0x' +
> part.fsuuid.upper()[2:].rjust(8,"0")
> +                    else:
> +                        part.fsuuid = '0x' +
> part.fsuuid.upper().rjust(8,"0") 
>      def prepare(self, imager):
>          """Prepare an image. Call prepare method of all image
> partitions.""" @@ -351,7 +355,8 @@ class PartitionedImage():
>              # sizes before we can add them and do the layout.
>              part.prepare(imager, imager.workdir, imager.oe_builddir,
>                           imager.rootfs_dir, imager.bootimg_dir,
> -                         imager.kernel_dir, imager.native_sysroot)
> +                         imager.kernel_dir, imager.native_sysroot,
> +                         imager.updated_fstab_path)
>  
>              # Converting kB to sectors for parted
>              part.size_sec = part.disk_size * 1024 // self.sector_size
> diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py
> b/scripts/lib/wic/plugins/source/bootimg-efi.py index
> 14c1723..cdc7254 100644 ---
> a/scripts/lib/wic/plugins/source/bootimg-efi.py +++
> b/scripts/lib/wic/plugins/source/bootimg-efi.py @@ -212,8 +212,8 @@
> class BootimgEFIPlugin(SourcePlugin): except KeyError:
>              raise WicError("bootimg-efi requires a loader, none
> specified") 
> -        if get_bitbake_var("IMAGE_BOOT_FILES") is None:
> -            logger.debug('No boot files defined in IMAGE_BOOT_FILES')
> +        if get_bitbake_var("IMAGE_EFI_BOOT_FILES") is None:
> +            logger.debug('No boot files defined in
> IMAGE_EFI_BOOT_FILES') else:
>              boot_files = None
>              for (fmt, id) in (("_uuid-%s", part.uuid), ("_label-%s",
> part.label), (None, None)): @@ -222,7 +222,7 @@ class
> BootimgEFIPlugin(SourcePlugin): else:
>                      var = ""
>  
> -                boot_files = get_bitbake_var("IMAGE_BOOT_FILES" +
> var)
> +                boot_files = get_bitbake_var("IMAGE_EFI_BOOT_FILES"
> + var) if boot_files:
>                      break
>  
> @@ -292,7 +292,7 @@ class BootimgEFIPlugin(SourcePlugin):
>              (staging_kernel_dir, kernel, hdddir, kernel)
>          exec_cmd(install_cmd)
>  
> -        if get_bitbake_var("IMAGE_BOOT_FILES"):
> +        if get_bitbake_var("IMAGE_EFI_BOOT_FILES"):
>              for src_path, dst_path in cls.install_task:
>                  install_cmd = "install -m 0644 -D %s %s" \
>                                % (os.path.join(kernel_dir, src_path),
> diff --git a/scripts/lib/wic/plugins/source/empty.py
> b/scripts/lib/wic/plugins/source/empty.py new file mode 100644
> index 0000000..041617d
> --- /dev/null
> +++ b/scripts/lib/wic/plugins/source/empty.py
> @@ -0,0 +1,32 @@
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +# The empty wic plugin is used to create unformatted empty
> partitions for wic +# images.
> +# To use it you must pass "empty" as argument for the "--source"
> parameter in +# the wks file. For example:
> +# part foo --source empty --ondisk sda --size="1024" --align 1024
> +
> +import logging
> +
> +from wic.pluginbase import SourcePlugin
> +
> +logger = logging.getLogger('wic')
> +
> +class EmptyPartitionPlugin(SourcePlugin):
> +    """
> +    Populate unformatted empty partition.
> +    """
> +
> +    name = 'empty'
> +
> +    @classmethod
> +    def do_prepare_partition(cls, part, source_params, cr,
> cr_workdir,
> +                             oe_builddir, bootimg_dir, kernel_dir,
> +                             rootfs_dir, native_sysroot):
> +        """
> +        Called to do the actual content population for a partition
> i.e. it
> +        'prepares' the partition to be incorporated into the image.
> +        """
> +        return
> diff --git a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
> b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py index
> 11326a2..afc9ea0 100644 ---
> a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py +++
> b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py @@ -216,6
> +216,18 @@ class IsoImagePlugin(SourcePlugin): creator.name =
> source_params['image_name'].strip() logger.debug("The name of the
> image is: %s", creator.name) 
> +    @staticmethod
> +    def _install_payload(source_params, iso_dir):
> +        """
> +        Copies contents of payload directory (as specified in
> 'payload_dir' param) into iso_dir
> +        """
> +
> +        if source_params.get('payload_dir'):
> +            payload_dir = source_params['payload_dir']
> +
> +            logger.debug("Payload directory: %s", payload_dir)
> +            shutil.copytree(payload_dir, iso_dir, symlinks=True,
> dirs_exist_ok=True) +
>      @classmethod
>      def do_prepare_partition(cls, part, source_params, creator,
> cr_workdir, oe_builddir, bootimg_dir, kernel_dir,
> @@ -228,6 +240,8 @@ class IsoImagePlugin(SourcePlugin):
>  
>          isodir = "%s/ISO" % cr_workdir
>  
> +        cls._install_payload(source_params, isodir)
> +
>          if part.rootfs_dir is None:
>              if not 'ROOTFS_DIR' in rootfs_dir:
>                  raise WicError("Couldn't find --rootfs-dir,
> exiting.") diff --git a/scripts/lib/wic/plugins/source/rootfs.py
> b/scripts/lib/wic/plugins/source/rootfs.py index f1db83f..96d940a
> 100644 --- a/scripts/lib/wic/plugins/source/rootfs.py
> +++ b/scripts/lib/wic/plugins/source/rootfs.py
> @@ -94,6 +94,7 @@ class RootfsPlugin(SourcePlugin):
>                                 "it is not a valid path, exiting" %
> part.rootfs_dir) 
>          part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
> +        part.has_fstab =
> os.path.exists(os.path.join(part.rootfs_dir, "etc/fstab")) pseudo_dir
> = os.path.join(part.rootfs_dir, "../pseudo") if not
> os.path.lexists(pseudo_dir): logger.warn("%s folder does not exist. "
> @@ -103,9 +104,9 @@ class RootfsPlugin(SourcePlugin):
>          new_rootfs = None
>          new_pseudo = None
>          # Handle excluded paths.
> -        if part.exclude_path or part.include_path or
> part.change_directory:
> -            # We need a new rootfs directory we can delete files
> from. Copy to
> -            # workdir.
> +        if part.exclude_path or part.include_path or
> part.change_directory or part.update_fstab_in_rootfs:
> +            # We need a new rootfs directory we can safely modify
> without
> +            # interfering with other tasks. Copy to workdir.
>              new_rootfs = os.path.realpath(os.path.join(cr_workdir,
> "rootfs%d" % part.lineno)) 
>              if os.path.lexists(new_rootfs):
> @@ -199,17 +200,33 @@ class RootfsPlugin(SourcePlugin):
>                  if not os.path.lexists(full_path):
>                      continue
>  
> +                if new_pseudo:
> +                    pseudo = cls.__get_pseudo(native_sysroot,
> new_rootfs, new_pseudo)
> +                else:
> +                    pseudo = None
>                  if path.endswith(os.sep):
>                      # Delete content only.
>                      for entry in os.listdir(full_path):
>                          full_entry = os.path.join(full_path, entry)
> -                        if os.path.isdir(full_entry) and not
> os.path.islink(full_entry):
> -                            shutil.rmtree(full_entry)
> -                        else:
> -                            os.remove(full_entry)
> +                        rm_cmd = "rm -rf %s" % (full_entry)
> +                        exec_native_cmd(rm_cmd, native_sysroot,
> pseudo) else:
>                      # Delete whole directory.
> -                    shutil.rmtree(full_path)
> +                    rm_cmd = "rm -rf %s" % (full_path)
> +                    exec_native_cmd(rm_cmd, native_sysroot, pseudo)
> +
> +            # Update part.has_fstab here as fstab may have been
> added or
> +            # removed by the above modifications.
> +            part.has_fstab = os.path.exists(os.path.join(new_rootfs,
> "etc/fstab"))
> +            if part.update_fstab_in_rootfs and part.has_fstab:
> +                fstab_path = os.path.join(new_rootfs, "etc/fstab")
> +                # Assume that fstab should always be owned by root
> with fixed permissions
> +                install_cmd = "install -m 0644 %s %s" %
> (part.updated_fstab_path, fstab_path)
> +                if new_pseudo:
> +                    pseudo = cls.__get_pseudo(native_sysroot,
> new_rootfs, new_pseudo)
> +                else:
> +                    pseudo = None
> +                exec_native_cmd(install_cmd, native_sysroot, pseudo)
>  
>          part.prepare_rootfs(cr_workdir, oe_builddir,
>                              new_rootfs or part.rootfs_dir,
> native_sysroot, diff --git a/scripts/wic b/scripts/wic
> index 24700f3..a741aed 100755
> --- a/scripts/wic
> +++ b/scripts/wic
> @@ -312,6 +312,8 @@ def wic_init_parser_create(subparser):
>  
>      subparser.add_argument("-o", "--outdir", dest="outdir",
> default='.', help="name of directory to create image in")
> +    subparser.add_argument("-w", "--workdir",
> +                      help="temporary workdir to use for
> intermediate files") subparser.add_argument("-e", "--image-name",
> dest="image_name", help="name of the image to use the artifacts from "
>                             "e.g. core-image-sato")

Re: [PATCH v2 2/3] wic: Update to the latest revision

[Uladzimir Bely]

[-- Attachment #1.1: Type: text/plain, Size: 26373 bytes --]

Original file (scripts/lib/wic/plugins/source/bootimg-efi.py) with latest 
update switched from IMAGE_BOOT_FILES usage to IMAGE_EFI_BOOT_FILES.
None of these variables is used in forked plugin, so that it didn't get 
updates

On Monday, August 9, 2021 at 4:53:43 PM UTC+3 Henning Schild wrote:

> Am Tue, 29 Jun 2021 11:22:24 +0200
> schrieb Uladzimir Bely <ub...@ilbers.de>:
>
> > From: Anton Mikanovich <ami...@ilbers.de>
> > 
> > Update to the latest wic from OE-core.
> > 
> > OE-core Revision: b85a09ea450a5e8f49418f4a930805fbb88dc83b
> > 
> > Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
> > Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
> > ---
> > scripts/lib/wic/canned-wks/common.wks.inc | 2 +-
> > scripts/lib/wic/canned-wks/directdisk-gpt.wks | 2 +-
> > scripts/lib/wic/canned-wks/mkefidisk.wks | 2 +-
> > scripts/lib/wic/ksparser.py | 17 ++++
> > scripts/lib/wic/misc.py | 8 +-
> > scripts/lib/wic/partition.py | 96
> > ++++++++++++++++--- scripts/lib/wic/plugins/imager/direct.py |
> > 61 ++++++------ scripts/lib/wic/plugins/source/bootimg-efi.py | 8 +-
>
> Our fork of that plugin now needs to sync up again to only differ in
> the parts we need it to differ in.
>
> meta/scripts/wic/plugins/source/bootimg-efi-isar.py
>
> Henning
>
> > scripts/lib/wic/plugins/source/empty.py | 32 +++++++
> > .../wic/plugins/source/isoimage-isohybrid.py | 14 +++
> > scripts/lib/wic/plugins/source/rootfs.py | 33 +++++--
> > scripts/wic | 2 +
> > 12 files changed, 217 insertions(+), 60 deletions(-)
> > create mode 100644 scripts/lib/wic/plugins/source/empty.py
> > 
> > diff --git a/scripts/lib/wic/canned-wks/common.wks.inc
> > b/scripts/lib/wic/canned-wks/common.wks.inc index 89880b4..4fd29fa
> > 100644 --- a/scripts/lib/wic/canned-wks/common.wks.inc
> > +++ b/scripts/lib/wic/canned-wks/common.wks.inc
> > @@ -1,3 +1,3 @@
> > # This file is included into 3 canned wks files from this directory
> > part /boot --source bootimg-pcbios --ondisk sda --label boot
> > --active --align 1024 -part / --source rootfs --use-uuid
> > --fstype=ext4 --label platform --align 1024 +part / --source rootfs
> > --use-uuid --fstype=ext4 --mkfs-extraopts "-T default" --label
> > platform --align 1024 diff --git
> > a/scripts/lib/wic/canned-wks/directdisk-gpt.wks
> > b/scripts/lib/wic/canned-wks/directdisk-gpt.wks index
> > 8d7d8de..cf16c0c 100644 ---
> > a/scripts/lib/wic/canned-wks/directdisk-gpt.wks +++
> > b/scripts/lib/wic/canned-wks/directdisk-gpt.wks @@ -4,7 +4,7 @@ 
> > part /boot --source bootimg-pcbios --ondisk sda --label boot
> > --active --align 1024 -part / --source rootfs --ondisk sda
> > --fstype=ext4 --label platform --align 1024 --use-uuid +part /
> > --source rootfs --ondisk sda --fstype=ext4 --mkfs-extraopts "-T
> > default" --label platform --align 1024 --use-uuid bootloader
> > --ptable gpt --timeout=0 --append="rootwait rootfstype=ext4
> > video=vesafb vga=0x318 console=tty0 console=ttyS0,115200n8" diff
> > --git a/scripts/lib/wic/canned-wks/mkefidisk.wks
> > b/scripts/lib/wic/canned-wks/mkefidisk.wks index 9f534fe..d1878e2
> > 100644 --- a/scripts/lib/wic/canned-wks/mkefidisk.wks +++
> > b/scripts/lib/wic/canned-wks/mkefidisk.wks @@ -4,7 +4,7 @@
> > 
> > part /boot --source bootimg-efi --sourceparams="loader=grub-efi"
> > --ondisk sda --label msdos --active --align 1024 
> > -part / --source rootfs --ondisk sda --fstype=ext4 --label platform
> > --align 1024 --use-uuid +part / --source rootfs --ondisk sda
> > --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align
> > 1024 --use-uuid part swap --ondisk sda --size 44 --label swap1
> > --fstype=swap 
> > diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> > index 913e328..3eb669d 100644
> > --- a/scripts/lib/wic/ksparser.py
> > +++ b/scripts/lib/wic/ksparser.py
> > @@ -229,6 +229,23 @@ class KickStart():
> > err = "%s:%d: SquashFS does not
> > support LABEL" \ % (confpath, lineno)
> > raise KickStartError(err)
> > + if parsed.fstype == 'msdos' or parsed.fstype
> > == 'vfat':
> > + if parsed.fsuuid:
> > + if
> > parsed.fsuuid.upper().startswith('0X'):
> > + if len(parsed.fsuuid) > 10:
> > + err = "%s:%d: fsuuid %s
> > given in wks kickstart file " \
> > + "exceeds the length
> > limit for %s filesystem. " \
> > + "It should be in the
> > form of a 32 bit hexadecimal" \
> > + "number (for example,
> > 0xABCD1234)." \
> > + % (confpath, lineno,
> > parsed.fsuuid, parsed.fstype)
> > + raise KickStartError(err)
> > + elif len(parsed.fsuuid) > 8:
> > + err = "%s:%d: fsuuid %s given in
> > wks kickstart file " \
> > + "exceeds the length limit
> > for %s filesystem. " \
> > + "It should be in the form
> > of a 32 bit hexadecimal" \
> > + "number (for example,
> > 0xABCD1234)." \
> > + % (confpath, lineno,
> > parsed.fsuuid, parsed.fstype)
> > + raise KickStartError(err)
> > if parsed.use_label and not parsed.label:
> > err = "%s:%d: Must set the label with
> > --label" \ % (confpath, lineno)
> > diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py
> > index 4b08d64..57c042c 100644
> > --- a/scripts/lib/wic/misc.py
> > +++ b/scripts/lib/wic/misc.py
> > @@ -26,6 +26,7 @@ logger = logging.getLogger('wic')
> > 
> > # executable -> recipe pairs for exec_native_cmd
> > NATIVE_RECIPES = {"bmaptool": "bmap-tools",
> > + "dumpe2fs": "e2fsprogs",
> > "grub-mkimage": "grub-efi",
> > "isohybrid": "syslinux",
> > "mcopy": "mtools",
> > @@ -138,9 +139,12 @@ def exec_native_cmd(cmd_and_args,
> > native_sysroot, pseudo=""): if pseudo:
> > cmd_and_args = pseudo + cmd_and_args
> > 
> > - native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin" % \
> > + hosttools_dir = get_bitbake_var("HOSTTOOLS_DIR")
> > +
> > + native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin:%s" % \
> > (native_sysroot, native_sysroot,
> > - native_sysroot, native_sysroot)
> > + native_sysroot, native_sysroot,
> > + hosttools_dir)
> > 
> > native_cmd_and_args = "export PATH=%s:$PATH;%s" % \
> > (native_paths, cmd_and_args)
> > diff --git a/scripts/lib/wic/partition.py
> > b/scripts/lib/wic/partition.py index 85eb15c..76d144d 100644
> > --- a/scripts/lib/wic/partition.py
> > +++ b/scripts/lib/wic/partition.py
> > @@ -54,6 +54,9 @@ class Partition():
> > self.uuid = args.uuid
> > self.fsuuid = args.fsuuid
> > self.type = args.type
> > + self.updated_fstab_path = None
> > + self.has_fstab = False
> > + self.update_fstab_in_rootfs = False
> > 
> > self.lineno = lineno
> > self.source_file = ""
> > @@ -118,11 +121,15 @@ class Partition():
> > return self.fixed_size if self.fixed_size else self.size
> > 
> > def prepare(self, creator, cr_workdir, oe_builddir, rootfs_dir,
> > - bootimg_dir, kernel_dir, native_sysroot):
> > + bootimg_dir, kernel_dir, native_sysroot,
> > updated_fstab_path): """
> > Prepare content for individual partitions, depending on
> > partition command parameters.
> > """
> > + self.updated_fstab_path = updated_fstab_path
> > + if self.updated_fstab_path and not
> > (self.fstype.startswith("ext") or self.fstype == "msdos"):
> > + self.update_fstab_in_rootfs = True
> > +
> > if not self.source:
> > if not self.size and not self.fixed_size:
> > raise WicError("The %s partition has a size of zero.
> > Please " @@ -199,21 +206,33 @@ class Partition():
> > 
> > Currently handles ext2/3/4, btrfs, vfat and squashfs.
> > """
> > +
> > + rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
> > + self.lineno, self.fstype)
> > + if os.path.isfile(rootfs):
> > + os.remove(rootfs)
> > +
> > p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" %
> > native_sysroot) if (pseudo_dir):
> > + # Canonicalize the ignore paths. This corresponds to
> > + # calling oe.path.canonicalize(), which is used in
> > bitbake.conf.
> > + ignore_paths = [rootfs] +
> > (get_bitbake_var("PSEUDO_IGNORE_PATHS") or "").split(",")
> > + canonical_paths = []
> > + for path in ignore_paths:
> > + if "$" not in path:
> > + trailing_slash = path.endswith("/") and "/" or ""
> > + canonical_paths.append(os.path.realpath(path) +
> > trailing_slash)
> > + ignore_paths = ",".join(canonical_paths)
> > +
> > pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
> > pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
> > pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
> > pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
> > + pseudo += "export PSEUDO_IGNORE_PATHS=%s;" % ignore_paths
> > pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
> > else:
> > pseudo = None
> > 
> > - rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
> > - self.lineno, self.fstype)
> > - if os.path.isfile(rootfs):
> > - os.remove(rootfs)
> > -
> > if not self.size and real_rootfs:
> > # The rootfs size is not set in .ks file so try to get it
> > # from bitbake variable
> > @@ -235,7 +254,7 @@ class Partition():
> > 
> > prefix = "ext" if self.fstype.startswith("ext") else
> > self.fstype method = getattr(self, "prepare_rootfs_" + prefix)
> > - method(rootfs, oe_builddir, rootfs_dir, native_sysroot,
> > pseudo)
> > + method(rootfs, cr_workdir, oe_builddir, rootfs_dir,
> > native_sysroot, pseudo) self.source_file = rootfs
> > 
> > # get the rootfs size in the right units for kickstart (kB)
> > @@ -243,7 +262,7 @@ class Partition():
> > out = exec_cmd(du_cmd)
> > self.size = int(out.split()[0])
> > 
> > - def prepare_rootfs_ext(self, rootfs, oe_builddir, rootfs_dir,
> > + def prepare_rootfs_ext(self, rootfs, cr_workdir, oe_builddir,
> > rootfs_dir, native_sysroot, pseudo):
> > """
> > Prepare content for an ext2/3/4 rootfs partition.
> > @@ -267,10 +286,21 @@ class Partition():
> > (self.fstype, extraopts, rootfs, label_str, self.fsuuid,
> > rootfs_dir) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
> > 
> > + if self.updated_fstab_path and self.has_fstab:
> > + debugfs_script_path = os.path.join(cr_workdir,
> > "debugfs_script")
> > + with open(debugfs_script_path, "w") as f:
> > + f.write("cd etc\n")
> > + f.write("rm fstab\n")
> > + f.write("write %s fstab\n" %
> > (self.updated_fstab_path))
> > + debugfs_cmd = "debugfs -w -f %s %s" %
> > (debugfs_script_path, rootfs)
> > + exec_native_cmd(debugfs_cmd, native_sysroot)
> > +
> > mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs)
> > exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
> > 
> > - def prepare_rootfs_btrfs(self, rootfs, oe_builddir, rootfs_dir,
> > + self.check_for_Y2038_problem(rootfs, native_sysroot)
> > +
> > + def prepare_rootfs_btrfs(self, rootfs, cr_workdir, oe_builddir,
> > rootfs_dir, native_sysroot, pseudo):
> > """
> > Prepare content for a btrfs rootfs partition.
> > @@ -293,7 +323,7 @@ class Partition():
> > self.mkfs_extraopts, self.fsuuid, rootfs)
> > exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
> > 
> > - def prepare_rootfs_msdos(self, rootfs, oe_builddir, rootfs_dir,
> > + def prepare_rootfs_msdos(self, rootfs, cr_workdir, oe_builddir,
> > rootfs_dir, native_sysroot, pseudo):
> > """
> > Prepare content for a msdos/vfat rootfs partition.
> > @@ -309,8 +339,6 @@ class Partition():
> > label_str = "-n %s" % self.label
> > 
> > size_str = ""
> > - if self.fstype == 'msdos':
> > - size_str = "-F 16" # FAT 16
> > 
> > extraopts = self.mkfs_extraopts or '-S 512'
> > 
> > @@ -322,12 +350,16 @@ class Partition():
> > mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir)
> > exec_native_cmd(mcopy_cmd, native_sysroot)
> > 
> > + if self.updated_fstab_path and self.has_fstab:
> > + mcopy_cmd = "mcopy -i %s %s ::/etc/fstab" % (rootfs,
> > self.updated_fstab_path)
> > + exec_native_cmd(mcopy_cmd, native_sysroot)
> > +
> > chmod_cmd = "chmod 644 %s" % rootfs
> > exec_cmd(chmod_cmd)
> > 
> > prepare_rootfs_vfat = prepare_rootfs_msdos
> > 
> > - def prepare_rootfs_squashfs(self, rootfs, oe_builddir,
> > rootfs_dir,
> > + def prepare_rootfs_squashfs(self, rootfs, cr_workdir,
> > oe_builddir, rootfs_dir, native_sysroot, pseudo):
> > """
> > Prepare content for a squashfs rootfs partition.
> > @@ -356,6 +388,8 @@ class Partition():
> > (self.fstype, extraopts, label_str, self.fsuuid, rootfs)
> > exec_native_cmd(mkfs_cmd, native_sysroot)
> > 
> > + self.check_for_Y2038_problem(rootfs, native_sysroot)
> > +
> > def prepare_empty_partition_btrfs(self, rootfs, oe_builddir,
> > native_sysroot):
> > """
> > @@ -386,8 +420,6 @@ class Partition():
> > label_str = "-n %s" % self.label
> > 
> > size_str = ""
> > - if self.fstype == 'msdos':
> > - size_str = "-F 16" # FAT 16
> > 
> > extraopts = self.mkfs_extraopts or '-S 512'
> > 
> > @@ -417,3 +449,37 @@ class Partition():
> > 
> > mkswap_cmd = "mkswap %s -U %s %s" % (label_str, self.fsuuid,
> > path) exec_native_cmd(mkswap_cmd, native_sysroot)
> > +
> > + def check_for_Y2038_problem(self, rootfs, native_sysroot):
> > + """
> > + Check if the filesystem is affected by the Y2038 problem
> > + (Y2038 problem = 32 bit time_t overflow in January 2038)
> > + """
> > + def get_err_str(part):
> > + err = "The {} filesystem {} has no Y2038 support."
> > + if part.mountpoint:
> > + args = [part.fstype, "mounted at %s" %
> > part.mountpoint]
> > + elif part.label:
> > + args = [part.fstype, "labeled '%s'" % part.label]
> > + elif part.part_name:
> > + args = [part.fstype, "in partition '%s'" %
> > part.part_name]
> > + else:
> > + args = [part.fstype, "in partition %s" % part.num]
> > + return err.format(*args)
> > +
> > + # ext2 and ext3 are always affected by the Y2038 problem
> > + if self.fstype in ["ext2", "ext3"]:
> > + logger.warn(get_err_str(self))
> > + return
> > +
> > + ret, out = exec_native_cmd("dumpe2fs %s" % rootfs,
> > native_sysroot) +
> > + # if ext4 is affected by the Y2038 problem depends on the
> > inode size
> > + for line in out.splitlines():
> > + if line.startswith("Inode size:"):
> > + size = int(line.split(":")[1].strip())
> > + if size < 256:
> > + logger.warn("%s Inodes (of size %d) are too
> > small." %
> > + (get_err_str(self), size))
> > + break
> > +
> > diff --git a/scripts/lib/wic/plugins/imager/direct.py
> > b/scripts/lib/wic/plugins/imager/direct.py index 55db826..ea709e8
> > 100644 --- a/scripts/lib/wic/plugins/imager/direct.py
> > +++ b/scripts/lib/wic/plugins/imager/direct.py
> > @@ -54,15 +54,16 @@ class DirectPlugin(ImagerPlugin):
> > self.native_sysroot = native_sysroot
> > self.oe_builddir = oe_builddir
> > 
> > + self.debug = options.debug
> > self.outdir = options.outdir
> > self.compressor = options.compressor
> > self.bmap = options.bmap
> > self.no_fstab_update = options.no_fstab_update
> > - self.original_fstab = None
> > + self.updated_fstab_path = None
> > 
> > self.name = "%s-%s" %
> > (os.path.splitext(os.path.basename(wks_file))[0],
> > strftime("%Y%m%d%H%M"))
> > - self.workdir = tempfile.mkdtemp(dir=self.outdir,
> > prefix='tmp.wic.')
> > + self.workdir = self.setup_workdir(options.workdir)
> > self._image = None
> > self.ptable_format = self.ks.bootloader.ptable
> > self.parts = self.ks.partitions
> > @@ -78,6 +79,16 @@ class DirectPlugin(ImagerPlugin):
> > self._image = PartitionedImage(image_path,
> > self.ptable_format, self.parts, self.native_sysroot)
> > 
> > + def setup_workdir(self, workdir):
> > + if workdir:
> > + if os.path.exists(workdir):
> > + raise WicError("Internal workdir '%s' specified in
> > wic arguments already exists!" % (workdir)) +
> > + os.makedirs(workdir)
> > + return workdir
> > + else:
> > + return tempfile.mkdtemp(dir=self.outdir,
> > prefix='tmp.wic.') +
> > def do_create(self):
> > """
> > Plugin entry point.
> > @@ -90,11 +101,8 @@ class DirectPlugin(ImagerPlugin):
> > finally:
> > self.cleanup()
> > 
> > - def _write_fstab(self, image_rootfs):
> > - """overriden to generate fstab (temporarily) in rootfs. This
> > is called
> > - from _create, make sure it doesn't get called from
> > - BaseImage.create()
> > - """
> > + def update_fstab(self, image_rootfs):
> > + """Assume partition order same as in wks"""
> > if not image_rootfs:
> > return
> > 
> > @@ -104,18 +112,9 @@ class DirectPlugin(ImagerPlugin):
> > 
> > with open(fstab_path) as fstab:
> > fstab_lines = fstab.readlines()
> > - self.original_fstab = fstab_lines.copy()
> > -
> > - if self._update_fstab(fstab_lines, self.parts):
> > - with open(fstab_path, "w") as fstab:
> > - fstab.writelines(fstab_lines)
> > - else:
> > - self.original_fstab = None
> > 
> > - def _update_fstab(self, fstab_lines, parts):
> > - """Assume partition order same as in wks"""
> > updated = False
> > - for part in parts:
> > + for part in self.parts:
> > if not part.realnum or not part.mountpoint \
> > or part.mountpoint == "/":
> > continue
> > @@ -144,7 +143,10 @@ class DirectPlugin(ImagerPlugin):
> > fstab_lines.append(line)
> > updated = True
> > 
> > - return updated
> > + if updated:
> > + self.updated_fstab_path = os.path.join(self.workdir,
> > "fstab")
> > + with open(self.updated_fstab_path, "w") as f:
> > + f.writelines(fstab_lines)
> > 
> > def _full_path(self, path, name, extention):
> > """ Construct full file path to a file we generate. """
> > @@ -160,7 +162,7 @@ class DirectPlugin(ImagerPlugin):
> > a partitioned image.
> > """
> > if not self.no_fstab_update:
> > - self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
> > + self.update_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
> > 
> > for part in self.parts:
> > # get rootfs size from bitbake variable if it's not set
> > in .ks file @@ -273,14 +275,9 @@ class DirectPlugin(ImagerPlugin):
> > if os.path.isfile(path):
> > shutil.move(path, os.path.join(self.outdir, fname))
> > 
> > - #Restore original fstab
> > - if self.original_fstab:
> > - fstab_path = self.rootfs_dir.get("ROOTFS_DIR") +
> > "/etc/fstab"
> > - with open(fstab_path, "w") as fstab:
> > - fstab.writelines(self.original_fstab)
> > -
> > - # remove work directory
> > - shutil.rmtree(self.workdir, ignore_errors=True)
> > + # remove work directory when it is not in debugging mode
> > + if not self.debug:
> > + shutil.rmtree(self.workdir, ignore_errors=True)
> > 
> > # Overhead of the MBR partitioning scheme (just one sector)
> > MBR_OVERHEAD = 1
> > @@ -343,6 +340,13 @@ class PartitionedImage():
> > part.fsuuid = '0x' +
> > str(uuid.uuid4())[:8].upper() else:
> > part.fsuuid = str(uuid.uuid4())
> > + else:
> > + #make sure the fsuuid for vfat/msdos align with
> > format 0xYYYYYYYY
> > + if part.fstype == 'vfat' or part.fstype == 'msdos':
> > + if part.fsuuid.upper().startswith("0X"):
> > + part.fsuuid = '0x' +
> > part.fsuuid.upper()[2:].rjust(8,"0")
> > + else:
> > + part.fsuuid = '0x' +
> > part.fsuuid.upper().rjust(8,"0") 
> > def prepare(self, imager):
> > """Prepare an image. Call prepare method of all image
> > partitions.""" @@ -351,7 +355,8 @@ class PartitionedImage():
> > # sizes before we can add them and do the layout.
> > part.prepare(imager, imager.workdir, imager.oe_builddir,
> > imager.rootfs_dir, imager.bootimg_dir,
> > - imager.kernel_dir, imager.native_sysroot)
> > + imager.kernel_dir, imager.native_sysroot,
> > + imager.updated_fstab_path)
> > 
> > # Converting kB to sectors for parted
> > part.size_sec = part.disk_size * 1024 // self.sector_size
> > diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py
> > b/scripts/lib/wic/plugins/source/bootimg-efi.py index
> > 14c1723..cdc7254 100644 ---
> > a/scripts/lib/wic/plugins/source/bootimg-efi.py +++
> > b/scripts/lib/wic/plugins/source/bootimg-efi.py @@ -212,8 +212,8 @@
> > class BootimgEFIPlugin(SourcePlugin): except KeyError:
> > raise WicError("bootimg-efi requires a loader, none
> > specified") 
> > - if get_bitbake_var("IMAGE_BOOT_FILES") is None:
> > - logger.debug('No boot files defined in IMAGE_BOOT_FILES')
> > + if get_bitbake_var("IMAGE_EFI_BOOT_FILES") is None:
> > + logger.debug('No boot files defined in
> > IMAGE_EFI_BOOT_FILES') else:
> > boot_files = None
> > for (fmt, id) in (("_uuid-%s", part.uuid), ("_label-%s",
> > part.label), (None, None)): @@ -222,7 +222,7 @@ class
> > BootimgEFIPlugin(SourcePlugin): else:
> > var = ""
> > 
> > - boot_files = get_bitbake_var("IMAGE_BOOT_FILES" +
> > var)
> > + boot_files = get_bitbake_var("IMAGE_EFI_BOOT_FILES"
> > + var) if boot_files:
> > break
> > 
> > @@ -292,7 +292,7 @@ class BootimgEFIPlugin(SourcePlugin):
> > (staging_kernel_dir, kernel, hdddir, kernel)
> > exec_cmd(install_cmd)
> > 
> > - if get_bitbake_var("IMAGE_BOOT_FILES"):
> > + if get_bitbake_var("IMAGE_EFI_BOOT_FILES"):
> > for src_path, dst_path in cls.install_task:
> > install_cmd = "install -m 0644 -D %s %s" \
> > % (os.path.join(kernel_dir, src_path),
> > diff --git a/scripts/lib/wic/plugins/source/empty.py
> > b/scripts/lib/wic/plugins/source/empty.py new file mode 100644
> > index 0000000..041617d
> > --- /dev/null
> > +++ b/scripts/lib/wic/plugins/source/empty.py
> > @@ -0,0 +1,32 @@
> > +#
> > +# SPDX-License-Identifier: MIT
> > +#
> > +
> > +# The empty wic plugin is used to create unformatted empty
> > partitions for wic +# images.
> > +# To use it you must pass "empty" as argument for the "--source"
> > parameter in +# the wks file. For example:
> > +# part foo --source empty --ondisk sda --size="1024" --align 1024
> > +
> > +import logging
> > +
> > +from wic.pluginbase import SourcePlugin
> > +
> > +logger = logging.getLogger('wic')
> > +
> > +class EmptyPartitionPlugin(SourcePlugin):
> > + """
> > + Populate unformatted empty partition.
> > + """
> > +
> > + name = 'empty'
> > +
> > + @classmethod
> > + def do_prepare_partition(cls, part, source_params, cr,
> > cr_workdir,
> > + oe_builddir, bootimg_dir, kernel_dir,
> > + rootfs_dir, native_sysroot):
> > + """
> > + Called to do the actual content population for a partition
> > i.e. it
> > + 'prepares' the partition to be incorporated into the image.
> > + """
> > + return
> > diff --git a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
> > b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py index
> > 11326a2..afc9ea0 100644 ---
> > a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py +++
> > b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py @@ -216,6
> > +216,18 @@ class IsoImagePlugin(SourcePlugin): creator.name =
> > source_params['image_name'].strip() logger.debug("The name of the
> > image is: %s", creator.name) 
> > + @staticmethod
> > + def _install_payload(source_params, iso_dir):
> > + """
> > + Copies contents of payload directory (as specified in
> > 'payload_dir' param) into iso_dir
> > + """
> > +
> > + if source_params.get('payload_dir'):
> > + payload_dir = source_params['payload_dir']
> > +
> > + logger.debug("Payload directory: %s", payload_dir)
> > + shutil.copytree(payload_dir, iso_dir, symlinks=True,
> > dirs_exist_ok=True) +
> > @classmethod
> > def do_prepare_partition(cls, part, source_params, creator,
> > cr_workdir, oe_builddir, bootimg_dir, kernel_dir,
> > @@ -228,6 +240,8 @@ class IsoImagePlugin(SourcePlugin):
> > 
> > isodir = "%s/ISO" % cr_workdir
> > 
> > + cls._install_payload(source_params, isodir)
> > +
> > if part.rootfs_dir is None:
> > if not 'ROOTFS_DIR' in rootfs_dir:
> > raise WicError("Couldn't find --rootfs-dir,
> > exiting.") diff --git a/scripts/lib/wic/plugins/source/rootfs.py
> > b/scripts/lib/wic/plugins/source/rootfs.py index f1db83f..96d940a
> > 100644 --- a/scripts/lib/wic/plugins/source/rootfs.py
> > +++ b/scripts/lib/wic/plugins/source/rootfs.py
> > @@ -94,6 +94,7 @@ class RootfsPlugin(SourcePlugin):
> > "it is not a valid path, exiting" %
> > part.rootfs_dir) 
> > part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
> > + part.has_fstab =
> > os.path.exists(os.path.join(part.rootfs_dir, "etc/fstab")) pseudo_dir
> > = os.path.join(part.rootfs_dir, "../pseudo") if not
> > os.path.lexists(pseudo_dir): logger.warn("%s folder does not exist. "
> > @@ -103,9 +104,9 @@ class RootfsPlugin(SourcePlugin):
> > new_rootfs = None
> > new_pseudo = None
> > # Handle excluded paths.
> > - if part.exclude_path or part.include_path or
> > part.change_directory:
> > - # We need a new rootfs directory we can delete files
> > from. Copy to
> > - # workdir.
> > + if part.exclude_path or part.include_path or
> > part.change_directory or part.update_fstab_in_rootfs:
> > + # We need a new rootfs directory we can safely modify
> > without
> > + # interfering with other tasks. Copy to workdir.
> > new_rootfs = os.path.realpath(os.path.join(cr_workdir,
> > "rootfs%d" % part.lineno)) 
> > if os.path.lexists(new_rootfs):
> > @@ -199,17 +200,33 @@ class RootfsPlugin(SourcePlugin):
> > if not os.path.lexists(full_path):
> > continue
> > 
> > + if new_pseudo:
> > + pseudo = cls.__get_pseudo(native_sysroot,
> > new_rootfs, new_pseudo)
> > + else:
> > + pseudo = None
> > if path.endswith(os.sep):
> > # Delete content only.
> > for entry in os.listdir(full_path):
> > full_entry = os.path.join(full_path, entry)
> > - if os.path.isdir(full_entry) and not
> > os.path.islink(full_entry):
> > - shutil.rmtree(full_entry)
> > - else:
> > - os.remove(full_entry)
> > + rm_cmd = "rm -rf %s" % (full_entry)
> > + exec_native_cmd(rm_cmd, native_sysroot,
> > pseudo) else:
> > # Delete whole directory.
> > - shutil.rmtree(full_path)
> > + rm_cmd = "rm -rf %s" % (full_path)
> > + exec_native_cmd(rm_cmd, native_sysroot, pseudo)
> > +
> > + # Update part.has_fstab here as fstab may have been
> > added or
> > + # removed by the above modifications.
> > + part.has_fstab = os.path.exists(os.path.join(new_rootfs,
> > "etc/fstab"))
> > + if part.update_fstab_in_rootfs and part.has_fstab:
> > + fstab_path = os.path.join(new_rootfs, "etc/fstab")
> > + # Assume that fstab should always be owned by root
> > with fixed permissions
> > + install_cmd = "install -m 0644 %s %s" %
> > (part.updated_fstab_path, fstab_path)
> > + if new_pseudo:
> > + pseudo = cls.__get_pseudo(native_sysroot,
> > new_rootfs, new_pseudo)
> > + else:
> > + pseudo = None
> > + exec_native_cmd(install_cmd, native_sysroot, pseudo)
> > 
> > part.prepare_rootfs(cr_workdir, oe_builddir,
> > new_rootfs or part.rootfs_dir,
> > native_sysroot, diff --git a/scripts/wic b/scripts/wic
> > index 24700f3..a741aed 100755
> > --- a/scripts/wic
> > +++ b/scripts/wic
> > @@ -312,6 +312,8 @@ def wic_init_parser_create(subparser):
> > 
> > subparser.add_argument("-o", "--outdir", dest="outdir",
> > default='.', help="name of directory to create image in")
> > + subparser.add_argument("-w", "--workdir",
> > + help="temporary workdir to use for
> > intermediate files") subparser.add_argument("-e", "--image-name",
> > dest="image_name", help="name of the image to use the artifacts from "
> > "e.g. core-image-sato")
>
>

[-- Attachment #1.2: Type: text/html, Size: 37454 bytes --]

Re: [PATCH v2 2/3] wic: Update to the latest revision

[Henning Schild]

Am Fri, 13 Aug 2021 02:35:08 -0700 (PDT)
schrieb Uladzimir Bely <ubely@ilbers.de>:

> Original file (scripts/lib/wic/plugins/source/bootimg-efi.py) with
> latest update switched from IMAGE_BOOT_FILES usage to
> IMAGE_EFI_BOOT_FILES. None of these variables is used in forked
> plugin, so that it didn't get updates

The fork should follow the original and the diff should only be
isar-stuff, not lazy out-of-sync stuff mixed in as well.

In fact these forked files cause enough pain that we should look into
mainlining the changes. It would be about adding some variables to the
originals where isar would use different defaults.
The forking and renaming causes more forking ... just because the names
are different. i.e. wks files

Henning

> On Monday, August 9, 2021 at 4:53:43 PM UTC+3 Henning Schild wrote:
> 
> > Am Tue, 29 Jun 2021 11:22:24 +0200
> > schrieb Uladzimir Bely <ub...@ilbers.de>:
> >  
> > > From: Anton Mikanovich <ami...@ilbers.de>
> > > 
> > > Update to the latest wic from OE-core.
> > > 
> > > OE-core Revision: b85a09ea450a5e8f49418f4a930805fbb88dc83b
> > > 
> > > Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
> > > Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
> > > ---
> > > scripts/lib/wic/canned-wks/common.wks.inc | 2 +-
> > > scripts/lib/wic/canned-wks/directdisk-gpt.wks | 2 +-
> > > scripts/lib/wic/canned-wks/mkefidisk.wks | 2 +-
> > > scripts/lib/wic/ksparser.py | 17 ++++
> > > scripts/lib/wic/misc.py | 8 +-
> > > scripts/lib/wic/partition.py | 96
> > > ++++++++++++++++--- scripts/lib/wic/plugins/imager/direct.py |
> > > 61 ++++++------ scripts/lib/wic/plugins/source/bootimg-efi.py | 8
> > > +-  
> >
> > Our fork of that plugin now needs to sync up again to only differ in
> > the parts we need it to differ in.
> >
> > meta/scripts/wic/plugins/source/bootimg-efi-isar.py
> >
> > Henning
> >  
> > > scripts/lib/wic/plugins/source/empty.py | 32 +++++++
> > > .../wic/plugins/source/isoimage-isohybrid.py | 14 +++
> > > scripts/lib/wic/plugins/source/rootfs.py | 33 +++++--
> > > scripts/wic | 2 +
> > > 12 files changed, 217 insertions(+), 60 deletions(-)
> > > create mode 100644 scripts/lib/wic/plugins/source/empty.py
> > > 
> > > diff --git a/scripts/lib/wic/canned-wks/common.wks.inc
> > > b/scripts/lib/wic/canned-wks/common.wks.inc index 89880b4..4fd29fa
> > > 100644 --- a/scripts/lib/wic/canned-wks/common.wks.inc
> > > +++ b/scripts/lib/wic/canned-wks/common.wks.inc
> > > @@ -1,3 +1,3 @@
> > > # This file is included into 3 canned wks files from this
> > > directory part /boot --source bootimg-pcbios --ondisk sda --label
> > > boot --active --align 1024 -part / --source rootfs --use-uuid
> > > --fstype=ext4 --label platform --align 1024 +part / --source
> > > rootfs --use-uuid --fstype=ext4 --mkfs-extraopts "-T default"
> > > --label platform --align 1024 diff --git
> > > a/scripts/lib/wic/canned-wks/directdisk-gpt.wks
> > > b/scripts/lib/wic/canned-wks/directdisk-gpt.wks index
> > > 8d7d8de..cf16c0c 100644 ---
> > > a/scripts/lib/wic/canned-wks/directdisk-gpt.wks +++
> > > b/scripts/lib/wic/canned-wks/directdisk-gpt.wks @@ -4,7 +4,7 @@ 
> > > part /boot --source bootimg-pcbios --ondisk sda --label boot
> > > --active --align 1024 -part / --source rootfs --ondisk sda
> > > --fstype=ext4 --label platform --align 1024 --use-uuid +part /
> > > --source rootfs --ondisk sda --fstype=ext4 --mkfs-extraopts "-T
> > > default" --label platform --align 1024 --use-uuid bootloader
> > > --ptable gpt --timeout=0 --append="rootwait rootfstype=ext4
> > > video=vesafb vga=0x318 console=tty0 console=ttyS0,115200n8" diff
> > > --git a/scripts/lib/wic/canned-wks/mkefidisk.wks
> > > b/scripts/lib/wic/canned-wks/mkefidisk.wks index 9f534fe..d1878e2
> > > 100644 --- a/scripts/lib/wic/canned-wks/mkefidisk.wks +++
> > > b/scripts/lib/wic/canned-wks/mkefidisk.wks @@ -4,7 +4,7 @@
> > > 
> > > part /boot --source bootimg-efi --sourceparams="loader=grub-efi"
> > > --ondisk sda --label msdos --active --align 1024 
> > > -part / --source rootfs --ondisk sda --fstype=ext4 --label
> > > platform --align 1024 --use-uuid +part / --source rootfs --ondisk
> > > sda --fstype=ext4 --mkfs-extraopts "-T default" --label platform
> > > --align 1024 --use-uuid part swap --ondisk sda --size 44 --label
> > > swap1 --fstype=swap 
> > > diff --git a/scripts/lib/wic/ksparser.py
> > > b/scripts/lib/wic/ksparser.py index 913e328..3eb669d 100644
> > > --- a/scripts/lib/wic/ksparser.py
> > > +++ b/scripts/lib/wic/ksparser.py
> > > @@ -229,6 +229,23 @@ class KickStart():
> > > err = "%s:%d: SquashFS does not
> > > support LABEL" \ % (confpath, lineno)
> > > raise KickStartError(err)
> > > + if parsed.fstype == 'msdos' or parsed.fstype
> > > == 'vfat':
> > > + if parsed.fsuuid:
> > > + if
> > > parsed.fsuuid.upper().startswith('0X'):
> > > + if len(parsed.fsuuid) > 10:
> > > + err = "%s:%d: fsuuid %s
> > > given in wks kickstart file " \
> > > + "exceeds the length
> > > limit for %s filesystem. " \
> > > + "It should be in the
> > > form of a 32 bit hexadecimal" \
> > > + "number (for example,
> > > 0xABCD1234)." \
> > > + % (confpath, lineno,
> > > parsed.fsuuid, parsed.fstype)
> > > + raise KickStartError(err)
> > > + elif len(parsed.fsuuid) > 8:
> > > + err = "%s:%d: fsuuid %s given in
> > > wks kickstart file " \
> > > + "exceeds the length limit
> > > for %s filesystem. " \
> > > + "It should be in the form
> > > of a 32 bit hexadecimal" \
> > > + "number (for example,
> > > 0xABCD1234)." \
> > > + % (confpath, lineno,
> > > parsed.fsuuid, parsed.fstype)
> > > + raise KickStartError(err)
> > > if parsed.use_label and not parsed.label:
> > > err = "%s:%d: Must set the label with
> > > --label" \ % (confpath, lineno)
> > > diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py
> > > index 4b08d64..57c042c 100644
> > > --- a/scripts/lib/wic/misc.py
> > > +++ b/scripts/lib/wic/misc.py
> > > @@ -26,6 +26,7 @@ logger = logging.getLogger('wic')
> > > 
> > > # executable -> recipe pairs for exec_native_cmd
> > > NATIVE_RECIPES = {"bmaptool": "bmap-tools",
> > > + "dumpe2fs": "e2fsprogs",
> > > "grub-mkimage": "grub-efi",
> > > "isohybrid": "syslinux",
> > > "mcopy": "mtools",
> > > @@ -138,9 +139,12 @@ def exec_native_cmd(cmd_and_args,
> > > native_sysroot, pseudo=""): if pseudo:
> > > cmd_and_args = pseudo + cmd_and_args
> > > 
> > > - native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin" % \
> > > + hosttools_dir = get_bitbake_var("HOSTTOOLS_DIR")
> > > +
> > > + native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin:%s" % \
> > > (native_sysroot, native_sysroot,
> > > - native_sysroot, native_sysroot)
> > > + native_sysroot, native_sysroot,
> > > + hosttools_dir)
> > > 
> > > native_cmd_and_args = "export PATH=%s:$PATH;%s" % \
> > > (native_paths, cmd_and_args)
> > > diff --git a/scripts/lib/wic/partition.py
> > > b/scripts/lib/wic/partition.py index 85eb15c..76d144d 100644
> > > --- a/scripts/lib/wic/partition.py
> > > +++ b/scripts/lib/wic/partition.py
> > > @@ -54,6 +54,9 @@ class Partition():
> > > self.uuid = args.uuid
> > > self.fsuuid = args.fsuuid
> > > self.type = args.type
> > > + self.updated_fstab_path = None
> > > + self.has_fstab = False
> > > + self.update_fstab_in_rootfs = False
> > > 
> > > self.lineno = lineno
> > > self.source_file = ""
> > > @@ -118,11 +121,15 @@ class Partition():
> > > return self.fixed_size if self.fixed_size else self.size
> > > 
> > > def prepare(self, creator, cr_workdir, oe_builddir, rootfs_dir,
> > > - bootimg_dir, kernel_dir, native_sysroot):
> > > + bootimg_dir, kernel_dir, native_sysroot,
> > > updated_fstab_path): """
> > > Prepare content for individual partitions, depending on
> > > partition command parameters.
> > > """
> > > + self.updated_fstab_path = updated_fstab_path
> > > + if self.updated_fstab_path and not
> > > (self.fstype.startswith("ext") or self.fstype == "msdos"):
> > > + self.update_fstab_in_rootfs = True
> > > +
> > > if not self.source:
> > > if not self.size and not self.fixed_size:
> > > raise WicError("The %s partition has a size of zero.
> > > Please " @@ -199,21 +206,33 @@ class Partition():
> > > 
> > > Currently handles ext2/3/4, btrfs, vfat and squashfs.
> > > """
> > > +
> > > + rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
> > > + self.lineno, self.fstype)
> > > + if os.path.isfile(rootfs):
> > > + os.remove(rootfs)
> > > +
> > > p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" %
> > > native_sysroot) if (pseudo_dir):
> > > + # Canonicalize the ignore paths. This corresponds to
> > > + # calling oe.path.canonicalize(), which is used in
> > > bitbake.conf.
> > > + ignore_paths = [rootfs] +
> > > (get_bitbake_var("PSEUDO_IGNORE_PATHS") or "").split(",")
> > > + canonical_paths = []
> > > + for path in ignore_paths:
> > > + if "$" not in path:
> > > + trailing_slash = path.endswith("/") and "/" or ""
> > > + canonical_paths.append(os.path.realpath(path) +
> > > trailing_slash)
> > > + ignore_paths = ",".join(canonical_paths)
> > > +
> > > pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
> > > pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
> > > pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
> > > pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
> > > + pseudo += "export PSEUDO_IGNORE_PATHS=%s;" % ignore_paths
> > > pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
> > > else:
> > > pseudo = None
> > > 
> > > - rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
> > > - self.lineno, self.fstype)
> > > - if os.path.isfile(rootfs):
> > > - os.remove(rootfs)
> > > -
> > > if not self.size and real_rootfs:
> > > # The rootfs size is not set in .ks file so try to get it
> > > # from bitbake variable
> > > @@ -235,7 +254,7 @@ class Partition():
> > > 
> > > prefix = "ext" if self.fstype.startswith("ext") else
> > > self.fstype method = getattr(self, "prepare_rootfs_" + prefix)
> > > - method(rootfs, oe_builddir, rootfs_dir, native_sysroot,
> > > pseudo)
> > > + method(rootfs, cr_workdir, oe_builddir, rootfs_dir,
> > > native_sysroot, pseudo) self.source_file = rootfs
> > > 
> > > # get the rootfs size in the right units for kickstart (kB)
> > > @@ -243,7 +262,7 @@ class Partition():
> > > out = exec_cmd(du_cmd)
> > > self.size = int(out.split()[0])
> > > 
> > > - def prepare_rootfs_ext(self, rootfs, oe_builddir, rootfs_dir,
> > > + def prepare_rootfs_ext(self, rootfs, cr_workdir, oe_builddir,
> > > rootfs_dir, native_sysroot, pseudo):
> > > """
> > > Prepare content for an ext2/3/4 rootfs partition.
> > > @@ -267,10 +286,21 @@ class Partition():
> > > (self.fstype, extraopts, rootfs, label_str, self.fsuuid,
> > > rootfs_dir) exec_native_cmd(mkfs_cmd, native_sysroot,
> > > pseudo=pseudo)
> > > 
> > > + if self.updated_fstab_path and self.has_fstab:
> > > + debugfs_script_path = os.path.join(cr_workdir,
> > > "debugfs_script")
> > > + with open(debugfs_script_path, "w") as f:
> > > + f.write("cd etc\n")
> > > + f.write("rm fstab\n")
> > > + f.write("write %s fstab\n" %
> > > (self.updated_fstab_path))
> > > + debugfs_cmd = "debugfs -w -f %s %s" %
> > > (debugfs_script_path, rootfs)
> > > + exec_native_cmd(debugfs_cmd, native_sysroot)
> > > +
> > > mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs)
> > > exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
> > > 
> > > - def prepare_rootfs_btrfs(self, rootfs, oe_builddir, rootfs_dir,
> > > + self.check_for_Y2038_problem(rootfs, native_sysroot)
> > > +
> > > + def prepare_rootfs_btrfs(self, rootfs, cr_workdir, oe_builddir,
> > > rootfs_dir, native_sysroot, pseudo):
> > > """
> > > Prepare content for a btrfs rootfs partition.
> > > @@ -293,7 +323,7 @@ class Partition():
> > > self.mkfs_extraopts, self.fsuuid, rootfs)
> > > exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
> > > 
> > > - def prepare_rootfs_msdos(self, rootfs, oe_builddir, rootfs_dir,
> > > + def prepare_rootfs_msdos(self, rootfs, cr_workdir, oe_builddir,
> > > rootfs_dir, native_sysroot, pseudo):
> > > """
> > > Prepare content for a msdos/vfat rootfs partition.
> > > @@ -309,8 +339,6 @@ class Partition():
> > > label_str = "-n %s" % self.label
> > > 
> > > size_str = ""
> > > - if self.fstype == 'msdos':
> > > - size_str = "-F 16" # FAT 16
> > > 
> > > extraopts = self.mkfs_extraopts or '-S 512'
> > > 
> > > @@ -322,12 +350,16 @@ class Partition():
> > > mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir)
> > > exec_native_cmd(mcopy_cmd, native_sysroot)
> > > 
> > > + if self.updated_fstab_path and self.has_fstab:
> > > + mcopy_cmd = "mcopy -i %s %s ::/etc/fstab" % (rootfs,
> > > self.updated_fstab_path)
> > > + exec_native_cmd(mcopy_cmd, native_sysroot)
> > > +
> > > chmod_cmd = "chmod 644 %s" % rootfs
> > > exec_cmd(chmod_cmd)
> > > 
> > > prepare_rootfs_vfat = prepare_rootfs_msdos
> > > 
> > > - def prepare_rootfs_squashfs(self, rootfs, oe_builddir,
> > > rootfs_dir,
> > > + def prepare_rootfs_squashfs(self, rootfs, cr_workdir,
> > > oe_builddir, rootfs_dir, native_sysroot, pseudo):
> > > """
> > > Prepare content for a squashfs rootfs partition.
> > > @@ -356,6 +388,8 @@ class Partition():
> > > (self.fstype, extraopts, label_str, self.fsuuid, rootfs)
> > > exec_native_cmd(mkfs_cmd, native_sysroot)
> > > 
> > > + self.check_for_Y2038_problem(rootfs, native_sysroot)
> > > +
> > > def prepare_empty_partition_btrfs(self, rootfs, oe_builddir,
> > > native_sysroot):
> > > """
> > > @@ -386,8 +420,6 @@ class Partition():
> > > label_str = "-n %s" % self.label
> > > 
> > > size_str = ""
> > > - if self.fstype == 'msdos':
> > > - size_str = "-F 16" # FAT 16
> > > 
> > > extraopts = self.mkfs_extraopts or '-S 512'
> > > 
> > > @@ -417,3 +449,37 @@ class Partition():
> > > 
> > > mkswap_cmd = "mkswap %s -U %s %s" % (label_str, self.fsuuid,
> > > path) exec_native_cmd(mkswap_cmd, native_sysroot)
> > > +
> > > + def check_for_Y2038_problem(self, rootfs, native_sysroot):
> > > + """
> > > + Check if the filesystem is affected by the Y2038 problem
> > > + (Y2038 problem = 32 bit time_t overflow in January 2038)
> > > + """
> > > + def get_err_str(part):
> > > + err = "The {} filesystem {} has no Y2038 support."
> > > + if part.mountpoint:
> > > + args = [part.fstype, "mounted at %s" %
> > > part.mountpoint]
> > > + elif part.label:
> > > + args = [part.fstype, "labeled '%s'" % part.label]
> > > + elif part.part_name:
> > > + args = [part.fstype, "in partition '%s'" %
> > > part.part_name]
> > > + else:
> > > + args = [part.fstype, "in partition %s" % part.num]
> > > + return err.format(*args)
> > > +
> > > + # ext2 and ext3 are always affected by the Y2038 problem
> > > + if self.fstype in ["ext2", "ext3"]:
> > > + logger.warn(get_err_str(self))
> > > + return
> > > +
> > > + ret, out = exec_native_cmd("dumpe2fs %s" % rootfs,
> > > native_sysroot) +
> > > + # if ext4 is affected by the Y2038 problem depends on the
> > > inode size
> > > + for line in out.splitlines():
> > > + if line.startswith("Inode size:"):
> > > + size = int(line.split(":")[1].strip())
> > > + if size < 256:
> > > + logger.warn("%s Inodes (of size %d) are too
> > > small." %
> > > + (get_err_str(self), size))
> > > + break
> > > +
> > > diff --git a/scripts/lib/wic/plugins/imager/direct.py
> > > b/scripts/lib/wic/plugins/imager/direct.py index 55db826..ea709e8
> > > 100644 --- a/scripts/lib/wic/plugins/imager/direct.py
> > > +++ b/scripts/lib/wic/plugins/imager/direct.py
> > > @@ -54,15 +54,16 @@ class DirectPlugin(ImagerPlugin):
> > > self.native_sysroot = native_sysroot
> > > self.oe_builddir = oe_builddir
> > > 
> > > + self.debug = options.debug
> > > self.outdir = options.outdir
> > > self.compressor = options.compressor
> > > self.bmap = options.bmap
> > > self.no_fstab_update = options.no_fstab_update
> > > - self.original_fstab = None
> > > + self.updated_fstab_path = None
> > > 
> > > self.name = "%s-%s" %
> > > (os.path.splitext(os.path.basename(wks_file))[0],
> > > strftime("%Y%m%d%H%M"))
> > > - self.workdir = tempfile.mkdtemp(dir=self.outdir,
> > > prefix='tmp.wic.')
> > > + self.workdir = self.setup_workdir(options.workdir)
> > > self._image = None
> > > self.ptable_format = self.ks.bootloader.ptable
> > > self.parts = self.ks.partitions
> > > @@ -78,6 +79,16 @@ class DirectPlugin(ImagerPlugin):
> > > self._image = PartitionedImage(image_path,
> > > self.ptable_format, self.parts, self.native_sysroot)
> > > 
> > > + def setup_workdir(self, workdir):
> > > + if workdir:
> > > + if os.path.exists(workdir):
> > > + raise WicError("Internal workdir '%s' specified in
> > > wic arguments already exists!" % (workdir)) +
> > > + os.makedirs(workdir)
> > > + return workdir
> > > + else:
> > > + return tempfile.mkdtemp(dir=self.outdir,
> > > prefix='tmp.wic.') +
> > > def do_create(self):
> > > """
> > > Plugin entry point.
> > > @@ -90,11 +101,8 @@ class DirectPlugin(ImagerPlugin):
> > > finally:
> > > self.cleanup()
> > > 
> > > - def _write_fstab(self, image_rootfs):
> > > - """overriden to generate fstab (temporarily) in rootfs. This
> > > is called
> > > - from _create, make sure it doesn't get called from
> > > - BaseImage.create()
> > > - """
> > > + def update_fstab(self, image_rootfs):
> > > + """Assume partition order same as in wks"""
> > > if not image_rootfs:
> > > return
> > > 
> > > @@ -104,18 +112,9 @@ class DirectPlugin(ImagerPlugin):
> > > 
> > > with open(fstab_path) as fstab:
> > > fstab_lines = fstab.readlines()
> > > - self.original_fstab = fstab_lines.copy()
> > > -
> > > - if self._update_fstab(fstab_lines, self.parts):
> > > - with open(fstab_path, "w") as fstab:
> > > - fstab.writelines(fstab_lines)
> > > - else:
> > > - self.original_fstab = None
> > > 
> > > - def _update_fstab(self, fstab_lines, parts):
> > > - """Assume partition order same as in wks"""
> > > updated = False
> > > - for part in parts:
> > > + for part in self.parts:
> > > if not part.realnum or not part.mountpoint \
> > > or part.mountpoint == "/":
> > > continue
> > > @@ -144,7 +143,10 @@ class DirectPlugin(ImagerPlugin):
> > > fstab_lines.append(line)
> > > updated = True
> > > 
> > > - return updated
> > > + if updated:
> > > + self.updated_fstab_path = os.path.join(self.workdir,
> > > "fstab")
> > > + with open(self.updated_fstab_path, "w") as f:
> > > + f.writelines(fstab_lines)
> > > 
> > > def _full_path(self, path, name, extention):
> > > """ Construct full file path to a file we generate. """
> > > @@ -160,7 +162,7 @@ class DirectPlugin(ImagerPlugin):
> > > a partitioned image.
> > > """
> > > if not self.no_fstab_update:
> > > - self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
> > > + self.update_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
> > > 
> > > for part in self.parts:
> > > # get rootfs size from bitbake variable if it's not set
> > > in .ks file @@ -273,14 +275,9 @@ class DirectPlugin(ImagerPlugin):
> > > if os.path.isfile(path):
> > > shutil.move(path, os.path.join(self.outdir, fname))
> > > 
> > > - #Restore original fstab
> > > - if self.original_fstab:
> > > - fstab_path = self.rootfs_dir.get("ROOTFS_DIR") +
> > > "/etc/fstab"
> > > - with open(fstab_path, "w") as fstab:
> > > - fstab.writelines(self.original_fstab)
> > > -
> > > - # remove work directory
> > > - shutil.rmtree(self.workdir, ignore_errors=True)
> > > + # remove work directory when it is not in debugging mode
> > > + if not self.debug:
> > > + shutil.rmtree(self.workdir, ignore_errors=True)
> > > 
> > > # Overhead of the MBR partitioning scheme (just one sector)
> > > MBR_OVERHEAD = 1
> > > @@ -343,6 +340,13 @@ class PartitionedImage():
> > > part.fsuuid = '0x' +
> > > str(uuid.uuid4())[:8].upper() else:
> > > part.fsuuid = str(uuid.uuid4())
> > > + else:
> > > + #make sure the fsuuid for vfat/msdos align with
> > > format 0xYYYYYYYY
> > > + if part.fstype == 'vfat' or part.fstype == 'msdos':
> > > + if part.fsuuid.upper().startswith("0X"):
> > > + part.fsuuid = '0x' +
> > > part.fsuuid.upper()[2:].rjust(8,"0")
> > > + else:
> > > + part.fsuuid = '0x' +
> > > part.fsuuid.upper().rjust(8,"0") 
> > > def prepare(self, imager):
> > > """Prepare an image. Call prepare method of all image
> > > partitions.""" @@ -351,7 +355,8 @@ class PartitionedImage():
> > > # sizes before we can add them and do the layout.
> > > part.prepare(imager, imager.workdir, imager.oe_builddir,
> > > imager.rootfs_dir, imager.bootimg_dir,
> > > - imager.kernel_dir, imager.native_sysroot)
> > > + imager.kernel_dir, imager.native_sysroot,
> > > + imager.updated_fstab_path)
> > > 
> > > # Converting kB to sectors for parted
> > > part.size_sec = part.disk_size * 1024 // self.sector_size
> > > diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py
> > > b/scripts/lib/wic/plugins/source/bootimg-efi.py index
> > > 14c1723..cdc7254 100644 ---
> > > a/scripts/lib/wic/plugins/source/bootimg-efi.py +++
> > > b/scripts/lib/wic/plugins/source/bootimg-efi.py @@ -212,8 +212,8
> > > @@ class BootimgEFIPlugin(SourcePlugin): except KeyError:
> > > raise WicError("bootimg-efi requires a loader, none
> > > specified") 
> > > - if get_bitbake_var("IMAGE_BOOT_FILES") is None:
> > > - logger.debug('No boot files defined in IMAGE_BOOT_FILES')
> > > + if get_bitbake_var("IMAGE_EFI_BOOT_FILES") is None:
> > > + logger.debug('No boot files defined in
> > > IMAGE_EFI_BOOT_FILES') else:
> > > boot_files = None
> > > for (fmt, id) in (("_uuid-%s", part.uuid), ("_label-%s",
> > > part.label), (None, None)): @@ -222,7 +222,7 @@ class
> > > BootimgEFIPlugin(SourcePlugin): else:
> > > var = ""
> > > 
> > > - boot_files = get_bitbake_var("IMAGE_BOOT_FILES" +
> > > var)
> > > + boot_files = get_bitbake_var("IMAGE_EFI_BOOT_FILES"
> > > + var) if boot_files:
> > > break
> > > 
> > > @@ -292,7 +292,7 @@ class BootimgEFIPlugin(SourcePlugin):
> > > (staging_kernel_dir, kernel, hdddir, kernel)
> > > exec_cmd(install_cmd)
> > > 
> > > - if get_bitbake_var("IMAGE_BOOT_FILES"):
> > > + if get_bitbake_var("IMAGE_EFI_BOOT_FILES"):
> > > for src_path, dst_path in cls.install_task:
> > > install_cmd = "install -m 0644 -D %s %s" \
> > > % (os.path.join(kernel_dir, src_path),
> > > diff --git a/scripts/lib/wic/plugins/source/empty.py
> > > b/scripts/lib/wic/plugins/source/empty.py new file mode 100644
> > > index 0000000..041617d
> > > --- /dev/null
> > > +++ b/scripts/lib/wic/plugins/source/empty.py
> > > @@ -0,0 +1,32 @@
> > > +#
> > > +# SPDX-License-Identifier: MIT
> > > +#
> > > +
> > > +# The empty wic plugin is used to create unformatted empty
> > > partitions for wic +# images.
> > > +# To use it you must pass "empty" as argument for the "--source"
> > > parameter in +# the wks file. For example:
> > > +# part foo --source empty --ondisk sda --size="1024" --align 1024
> > > +
> > > +import logging
> > > +
> > > +from wic.pluginbase import SourcePlugin
> > > +
> > > +logger = logging.getLogger('wic')
> > > +
> > > +class EmptyPartitionPlugin(SourcePlugin):
> > > + """
> > > + Populate unformatted empty partition.
> > > + """
> > > +
> > > + name = 'empty'
> > > +
> > > + @classmethod
> > > + def do_prepare_partition(cls, part, source_params, cr,
> > > cr_workdir,
> > > + oe_builddir, bootimg_dir, kernel_dir,
> > > + rootfs_dir, native_sysroot):
> > > + """
> > > + Called to do the actual content population for a partition
> > > i.e. it
> > > + 'prepares' the partition to be incorporated into the image.
> > > + """
> > > + return
> > > diff --git a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
> > > b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py index
> > > 11326a2..afc9ea0 100644 ---
> > > a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py +++
> > > b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py @@ -216,6
> > > +216,18 @@ class IsoImagePlugin(SourcePlugin): creator.name =
> > > source_params['image_name'].strip() logger.debug("The name of the
> > > image is: %s", creator.name) 
> > > + @staticmethod
> > > + def _install_payload(source_params, iso_dir):
> > > + """
> > > + Copies contents of payload directory (as specified in
> > > 'payload_dir' param) into iso_dir
> > > + """
> > > +
> > > + if source_params.get('payload_dir'):
> > > + payload_dir = source_params['payload_dir']
> > > +
> > > + logger.debug("Payload directory: %s", payload_dir)
> > > + shutil.copytree(payload_dir, iso_dir, symlinks=True,
> > > dirs_exist_ok=True) +
> > > @classmethod
> > > def do_prepare_partition(cls, part, source_params, creator,
> > > cr_workdir, oe_builddir, bootimg_dir, kernel_dir,
> > > @@ -228,6 +240,8 @@ class IsoImagePlugin(SourcePlugin):
> > > 
> > > isodir = "%s/ISO" % cr_workdir
> > > 
> > > + cls._install_payload(source_params, isodir)
> > > +
> > > if part.rootfs_dir is None:
> > > if not 'ROOTFS_DIR' in rootfs_dir:
> > > raise WicError("Couldn't find --rootfs-dir,
> > > exiting.") diff --git a/scripts/lib/wic/plugins/source/rootfs.py
> > > b/scripts/lib/wic/plugins/source/rootfs.py index f1db83f..96d940a
> > > 100644 --- a/scripts/lib/wic/plugins/source/rootfs.py
> > > +++ b/scripts/lib/wic/plugins/source/rootfs.py
> > > @@ -94,6 +94,7 @@ class RootfsPlugin(SourcePlugin):
> > > "it is not a valid path, exiting" %
> > > part.rootfs_dir) 
> > > part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
> > > + part.has_fstab =
> > > os.path.exists(os.path.join(part.rootfs_dir, "etc/fstab"))
> > > pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo") if not
> > > os.path.lexists(pseudo_dir): logger.warn("%s folder does not
> > > exist. " @@ -103,9 +104,9 @@ class RootfsPlugin(SourcePlugin):
> > > new_rootfs = None
> > > new_pseudo = None
> > > # Handle excluded paths.
> > > - if part.exclude_path or part.include_path or
> > > part.change_directory:
> > > - # We need a new rootfs directory we can delete files
> > > from. Copy to
> > > - # workdir.
> > > + if part.exclude_path or part.include_path or
> > > part.change_directory or part.update_fstab_in_rootfs:
> > > + # We need a new rootfs directory we can safely modify
> > > without
> > > + # interfering with other tasks. Copy to workdir.
> > > new_rootfs = os.path.realpath(os.path.join(cr_workdir,
> > > "rootfs%d" % part.lineno)) 
> > > if os.path.lexists(new_rootfs):
> > > @@ -199,17 +200,33 @@ class RootfsPlugin(SourcePlugin):
> > > if not os.path.lexists(full_path):
> > > continue
> > > 
> > > + if new_pseudo:
> > > + pseudo = cls.__get_pseudo(native_sysroot,
> > > new_rootfs, new_pseudo)
> > > + else:
> > > + pseudo = None
> > > if path.endswith(os.sep):
> > > # Delete content only.
> > > for entry in os.listdir(full_path):
> > > full_entry = os.path.join(full_path, entry)
> > > - if os.path.isdir(full_entry) and not
> > > os.path.islink(full_entry):
> > > - shutil.rmtree(full_entry)
> > > - else:
> > > - os.remove(full_entry)
> > > + rm_cmd = "rm -rf %s" % (full_entry)
> > > + exec_native_cmd(rm_cmd, native_sysroot,
> > > pseudo) else:
> > > # Delete whole directory.
> > > - shutil.rmtree(full_path)
> > > + rm_cmd = "rm -rf %s" % (full_path)
> > > + exec_native_cmd(rm_cmd, native_sysroot, pseudo)
> > > +
> > > + # Update part.has_fstab here as fstab may have been
> > > added or
> > > + # removed by the above modifications.
> > > + part.has_fstab = os.path.exists(os.path.join(new_rootfs,
> > > "etc/fstab"))
> > > + if part.update_fstab_in_rootfs and part.has_fstab:
> > > + fstab_path = os.path.join(new_rootfs, "etc/fstab")
> > > + # Assume that fstab should always be owned by root
> > > with fixed permissions
> > > + install_cmd = "install -m 0644 %s %s" %
> > > (part.updated_fstab_path, fstab_path)
> > > + if new_pseudo:
> > > + pseudo = cls.__get_pseudo(native_sysroot,
> > > new_rootfs, new_pseudo)
> > > + else:
> > > + pseudo = None
> > > + exec_native_cmd(install_cmd, native_sysroot, pseudo)
> > > 
> > > part.prepare_rootfs(cr_workdir, oe_builddir,
> > > new_rootfs or part.rootfs_dir,
> > > native_sysroot, diff --git a/scripts/wic b/scripts/wic
> > > index 24700f3..a741aed 100755
> > > --- a/scripts/wic
> > > +++ b/scripts/wic
> > > @@ -312,6 +312,8 @@ def wic_init_parser_create(subparser):
> > > 
> > > subparser.add_argument("-o", "--outdir", dest="outdir",
> > > default='.', help="name of directory to create image in")
> > > + subparser.add_argument("-w", "--workdir",
> > > + help="temporary workdir to use for
> > > intermediate files") subparser.add_argument("-e", "--image-name",
> > > dest="image_name", help="name of the image to use the artifacts
> > > from " "e.g. core-image-sato")  
> >
> >  
> 

[PATCH v2 3/3] wic: Force mke2fs to use Y2038-compatible inode size

[Uladzimir Bely]

By using `--mkfs-extraopts "-T default"` we tell mke2fs not to
auto-detect the usage type based on the filesystem size but use
the default values for tuning parameters instead.

More info in corresponding commit in openembedded-core:
https://github.com/openembedded/openembedded-core/commit/eecbe62

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta-isar/scripts/lib/wic/canned-wks/bananapi.wks        | 2 +-
 meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc | 2 +-
 meta-isar/scripts/lib/wic/canned-wks/de0-nano-soc.wks.in | 2 +-
 meta-isar/scripts/lib/wic/canned-wks/hikey.wks           | 2 +-
 meta-isar/scripts/lib/wic/canned-wks/nanopi-neo.wks.in   | 2 +-
 meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks     | 2 +-
 meta-isar/scripts/lib/wic/canned-wks/sifive-fu540.wks    | 2 +-
 meta-isar/scripts/lib/wic/canned-wks/stm32mp15x.wks.in   | 2 +-
 8 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/meta-isar/scripts/lib/wic/canned-wks/bananapi.wks b/meta-isar/scripts/lib/wic/canned-wks/bananapi.wks
index dbec48e..1ba71b9 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/bananapi.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/bananapi.wks
@@ -1,5 +1,5 @@
 part u-boot --source rawcopy --sourceparams "file=/usr/lib/u-boot/Bananapi/u-boot-sunxi-with-spl.bin" --no-table --align 8
 
-part / --source rootfs-u-boot --ondisk mmcblk0 --fstype ext4 --label platform --align 1024
+part / --source rootfs-u-boot --ondisk mmcblk0 --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024
 
 bootloader --append "rootwait"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc b/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc
index cc38a01..a65e646 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc
+++ b/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc
@@ -1,3 +1,3 @@
 # This file is included into 3 canned wks files from this directory
 part /boot --source bootimg-pcbios-isar --ondisk sda --label boot --active --align 1024
-part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --exclude-path=boot
+part / --source rootfs --ondisk sda --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --exclude-path=boot
diff --git a/meta-isar/scripts/lib/wic/canned-wks/de0-nano-soc.wks.in b/meta-isar/scripts/lib/wic/canned-wks/de0-nano-soc.wks.in
index 05ce61b..0794a2f 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/de0-nano-soc.wks.in
+++ b/meta-isar/scripts/lib/wic/canned-wks/de0-nano-soc.wks.in
@@ -5,6 +5,6 @@
 
 part --source rawcopy --sourceparams "file=/usr/lib/u-boot/${MACHINE}/${U_BOOT_BIN}" --system-id 0xa2 --align 1
 
-part / --source rootfs-u-boot --ondisk mmcblk0 --fstype ext4 --label platform --align 1024 --active
+part / --source rootfs-u-boot --ondisk mmcblk0 --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --active
 
 bootloader --append "rw rootwait"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks
index 840858e..1b7a64e 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks
@@ -5,6 +5,6 @@
 
 part --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk mmcblk1 --label efi --part-type EF00 --align 1024
 
-part / --source rootfs --ondisk mmcblk1 --fstype ext4 --label platform --align 1024 --use-uuid
+part / --source rootfs --ondisk mmcblk1 --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid
 
 bootloader --ptable gpt --timeout=3
diff --git a/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo.wks.in b/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo.wks.in
index 2e98390..de1c92c 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo.wks.in
+++ b/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo.wks.in
@@ -5,6 +5,6 @@
 
 part u-boot --source rawcopy --sourceparams "file=/usr/lib/u-boot/nanopi_neo/u-boot-sunxi-with-spl.bin" --no-table --align 8
 
-part / --source rootfs-u-boot --ondisk mmcblk0 --fstype ext4 --label platform --align 1024 --active
+part / --source rootfs-u-boot --ondisk mmcblk0 --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --active
 
 bootloader --append "rw rootwait"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
index e0fa7a1..be46843 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
@@ -4,6 +4,6 @@
 
 part --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk sda --label efi --part-type EF00 --align 1024
 
-part / --source rootfs --ondisk sda --fstype ext4 --label platform --align 1024 --use-uuid
+part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid
 
 bootloader --ptable gpt --timeout 3 --append "rootwait console=ttyS0,115200 console=tty0"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/sifive-fu540.wks b/meta-isar/scripts/lib/wic/canned-wks/sifive-fu540.wks
index 445b9d0..9cd0e66 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/sifive-fu540.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/sifive-fu540.wks
@@ -5,6 +5,6 @@
 
 part --source rawcopy --sourceparams "file=/usr/lib/opensbi/sifive-fu540/fw_payload.bin" --ondisk mmcblk0 --fixed-size 8M --align 1 --part-type 2e54b353-1271-4842-806f-e436d6af6985
 
-part / --source rootfs-u-boot --ondisk mmcblk0 --fstype ext4 --label platform --active --align 1024 --sourceparams "no_initrd=yes,script_prepend=setenv fdtfile sifive/hifive-unleashed-a00.dtb"
+part / --source rootfs-u-boot --ondisk mmcblk0 --fstype ext4 --mkfs-extraopts "-T default" --label platform --active --align 1024 --sourceparams "no_initrd=yes,script_prepend=setenv fdtfile sifive/hifive-unleashed-a00.dtb"
 
 bootloader --ptable gpt --append "rootwait"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/stm32mp15x.wks.in b/meta-isar/scripts/lib/wic/canned-wks/stm32mp15x.wks.in
index 744330e..4ed4ee8 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/stm32mp15x.wks.in
+++ b/meta-isar/scripts/lib/wic/canned-wks/stm32mp15x.wks.in
@@ -11,6 +11,6 @@ part teeh --part-name teeh --source rawcopy --sourceparams "file=/usr/lib/optee-
 part teed --part-name teed --source rawcopy --sourceparams "file=/usr/lib/optee-os/${MACHINE}/tee-pageable_v2.stm32" --fstype=ext4 --fsoptions "noauto" --part-type 0x8301 --fixed-size 512K
 part teex --part-name teex --source rawcopy --sourceparams "file=/usr/lib/optee-os/${MACHINE}/tee-pager_v2.stm32" --fstype=ext4 --fsoptions "noauto" --part-type 0x8301 --fixed-size 256K
 
-part / --source rootfs-u-boot --fstype ext4 --label root --align 1024 --active --use-uuid
+part / --source rootfs-u-boot --fstype ext4 --mkfs-extraopts "-T default" --label root --align 1024 --active --use-uuid
 
 bootloader --ptable gpt --append="rootwait rw console=ttySTM0,115200"
-- 
2.20.1

Re: [PATCH v2 0/3] wic: Update to the latest revision

[Anton Mikanovich]

29.06.2021 12:22, Uladzimir Bely wrote:
> This patchset updates wic in isar to the OE-core revision b85a09e.
> This is the latest revision that is compatible with bitbake
> version in isar.
> Before updating to even newer wic we need first to update bitbake.

Applied to next, thanks.

-- 
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov