* [PATCH v2 00/10] WIC update
@ 2020-09-02 18:56 Vijai Kumar K
2020-09-02 18:56 ` [PATCH v2 01/10] wic: Update to the latest wic from openembedded core Vijai Kumar K
` (8 more replies)
0 siblings, 9 replies; 29+ messages in thread
From: Vijai Kumar K @ 2020-09-02 18:56 UTC (permalink / raw)
To: isar-users, Vijaikumar_Kanagarajan
Changes since RFC(v1):
- P9 commit is upstreamed and replaced with the version from OE-core.
- P10: Lot of downstream projects tend to use /boot mountpoints for bootloader/EFI
partitions. Added RECIPE-API-CHANGELOG entry to warn user about potential issues
that might arise when doing so with latest wic.
Henning Schild (1):
lib/oe/path: try hardlinking instead of guessing when it might fail
Paul Barker (1):
oe.path: Add copyhardlink() helper function
Vijai Kumar K (8):
wic: Update to the latest wic from openembedded core
wic/plugins: Fix wic plugins to work with the latest wic
wic-img: Satisfy the quirks of latest wic
wic_fakeroot: Handle standalone pseudo invocations
meta-isar/conf: Add provision to debug WIC
debian-common: Add tar as a dependency for wic
wic: misc: Add /bin to the list of searchpaths
meta-isar/canned-wks: Remove /boot mountpoint
RECIPE-API-CHANGELOG.md | 17 +
meta-isar/conf/local.conf.sample | 3 +
.../lib/wic/canned-wks/common-isar.wks.inc | 2 +-
.../scripts/lib/wic/canned-wks/hikey.wks | 2 +-
.../lib/wic/canned-wks/sdimage-efi.wks | 2 +-
meta/classes/wic-img.bbclass | 15 +-
meta/conf/distro/debian-common.conf | 3 +-
meta/lib/oe/path.py | 27 +-
.../wic/plugins/source/bootimg-efi-isar.py | 2 +-
.../wic/plugins/source/bootimg-pcbios-isar.py | 9 +-
.../lib/wic/plugins/source/rootfs-u-boot.py | 2 +-
scripts/lib/scriptpath.py | 32 ++
scripts/lib/wic/__init__.py | 14 +-
scripts/lib/wic/canned-wks/common.wks.inc | 2 +-
.../directdisk-bootloader-config.cfg | 8 +-
.../lib/wic/canned-wks/efi-bootdisk.wks.in | 3 +
scripts/lib/wic/canned-wks/mkhybridiso.wks | 2 +-
scripts/lib/wic/canned-wks/qemuriscv.wks | 3 +
.../lib/wic/canned-wks/qemux86-directdisk.wks | 2 +-
.../lib/wic/canned-wks/sdimage-bootpart.wks | 4 +-
.../lib/wic/canned-wks/systemd-bootdisk.wks | 4 +-
scripts/lib/wic/engine.py | 421 +++++++++++++++-
scripts/lib/wic/filemap.py | 170 ++++---
scripts/lib/wic/help.py | 401 ++++++++++++++--
scripts/lib/wic/ksparser.py | 115 +++--
scripts/lib/wic/{utils => }/misc.py | 100 ++--
scripts/lib/wic/partition.py | 234 ++++-----
scripts/lib/wic/pluginbase.py | 36 +-
scripts/lib/wic/plugins/imager/direct.py | 175 ++++---
.../wic/plugins/source/bootimg-biosplusefi.py | 213 +++++++++
scripts/lib/wic/plugins/source/bootimg-efi.py | 111 +++--
.../wic/plugins/source/bootimg-partition.py | 153 ++++--
.../lib/wic/plugins/source/bootimg-pcbios.py | 91 ++--
scripts/lib/wic/plugins/source/fsimage.py | 56 ---
.../wic/plugins/source/isoimage-isohybrid.py | 185 +++----
scripts/lib/wic/plugins/source/rawcopy.py | 44 +-
scripts/lib/wic/plugins/source/rootfs.py | 159 ++++--
scripts/lib/wic/utils/__init__.py | 0
scripts/lib/wic/utils/runner.py | 114 -----
scripts/wic | 452 +++++++++++++-----
scripts/wic_fakeroot | 5 +
41 files changed, 2357 insertions(+), 1036 deletions(-)
create mode 100644 scripts/lib/scriptpath.py
create mode 100644 scripts/lib/wic/canned-wks/efi-bootdisk.wks.in
create mode 100644 scripts/lib/wic/canned-wks/qemuriscv.wks
rename scripts/lib/wic/{utils => }/misc.py (70%)
create mode 100644 scripts/lib/wic/plugins/source/bootimg-biosplusefi.py
delete mode 100644 scripts/lib/wic/plugins/source/fsimage.py
delete mode 100644 scripts/lib/wic/utils/__init__.py
delete mode 100644 scripts/lib/wic/utils/runner.py
--
2.17.1
^ permalink raw reply [flat|nested] 29+ messages in thread* [PATCH v2 01/10] wic: Update to the latest wic from openembedded core 2020-09-02 18:56 [PATCH v2 00/10] WIC update Vijai Kumar K @ 2020-09-02 18:56 ` Vijai Kumar K 2020-09-05 9:00 ` Henning Schild 2020-09-02 18:56 ` [PATCH v2 02/10] wic/plugins: Fix wic plugins to work with the latest wic Vijai Kumar K ` (7 subsequent siblings) 8 siblings, 1 reply; 29+ messages in thread From: Vijai Kumar K @ 2020-09-02 18:56 UTC (permalink / raw) To: isar-users, Vijaikumar_Kanagarajan Update to the latest wic from OE-core. OE-core Revision: 532ae127c52c9f7b1d2e4ca5cbca91881d23a2ac Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> --- scripts/lib/scriptpath.py | 32 ++ scripts/lib/wic/__init__.py | 14 +- scripts/lib/wic/canned-wks/common.wks.inc | 2 +- .../directdisk-bootloader-config.cfg | 8 +- .../lib/wic/canned-wks/efi-bootdisk.wks.in | 3 + scripts/lib/wic/canned-wks/mkhybridiso.wks | 2 +- scripts/lib/wic/canned-wks/qemuriscv.wks | 3 + .../lib/wic/canned-wks/qemux86-directdisk.wks | 2 +- .../lib/wic/canned-wks/sdimage-bootpart.wks | 4 +- .../lib/wic/canned-wks/systemd-bootdisk.wks | 4 +- scripts/lib/wic/engine.py | 421 +++++++++++++++- scripts/lib/wic/filemap.py | 170 ++++--- scripts/lib/wic/help.py | 401 ++++++++++++++-- scripts/lib/wic/ksparser.py | 115 +++-- scripts/lib/wic/{utils => }/misc.py | 99 ++-- scripts/lib/wic/partition.py | 234 ++++----- scripts/lib/wic/pluginbase.py | 36 +- scripts/lib/wic/plugins/imager/direct.py | 175 ++++--- .../wic/plugins/source/bootimg-biosplusefi.py | 213 +++++++++ scripts/lib/wic/plugins/source/bootimg-efi.py | 111 +++-- .../wic/plugins/source/bootimg-partition.py | 153 ++++-- .../lib/wic/plugins/source/bootimg-pcbios.py | 91 ++-- scripts/lib/wic/plugins/source/fsimage.py | 56 --- .../wic/plugins/source/isoimage-isohybrid.py | 185 +++---- scripts/lib/wic/plugins/source/rawcopy.py | 44 +- scripts/lib/wic/plugins/source/rootfs.py | 159 ++++-- scripts/lib/wic/utils/__init__.py | 0 scripts/lib/wic/utils/runner.py | 114 ----- scripts/wic | 452 +++++++++++++----- 29 files changed, 2283 insertions(+), 1020 deletions(-) create mode 100644 scripts/lib/scriptpath.py create mode 100644 scripts/lib/wic/canned-wks/efi-bootdisk.wks.in create mode 100644 scripts/lib/wic/canned-wks/qemuriscv.wks rename scripts/lib/wic/{utils => }/misc.py (70%) create mode 100644 scripts/lib/wic/plugins/source/bootimg-biosplusefi.py delete mode 100644 scripts/lib/wic/plugins/source/fsimage.py delete mode 100644 scripts/lib/wic/utils/__init__.py delete mode 100644 scripts/lib/wic/utils/runner.py diff --git a/scripts/lib/scriptpath.py b/scripts/lib/scriptpath.py new file mode 100644 index 0000000..f32326d --- /dev/null +++ b/scripts/lib/scriptpath.py @@ -0,0 +1,32 @@ +# Path utility functions for OE python scripts +# +# Copyright (C) 2012-2014 Intel Corporation +# Copyright (C) 2011 Mentor Graphics Corporation +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import sys +import os +import os.path + +def add_oe_lib_path(): + basepath = os.path.abspath(os.path.dirname(__file__) + '/../..') + newpath = basepath + '/meta/lib' + sys.path.insert(0, newpath) + +def add_bitbake_lib_path(): + basepath = os.path.abspath(os.path.dirname(__file__) + '/../..') + bitbakepath = None + if os.path.exists(basepath + '/bitbake/lib/bb'): + bitbakepath = basepath + '/bitbake' + else: + # look for bitbake/bin dir in PATH + for pth in os.environ['PATH'].split(':'): + if os.path.exists(os.path.join(pth, '../lib/bb')): + bitbakepath = os.path.abspath(os.path.join(pth, '..')) + break + + if bitbakepath: + sys.path.insert(0, bitbakepath + '/lib') + return bitbakepath diff --git a/scripts/lib/wic/__init__.py b/scripts/lib/wic/__init__.py index 85876b1..8556793 100644 --- a/scripts/lib/wic/__init__.py +++ b/scripts/lib/wic/__init__.py @@ -1,20 +1,10 @@ -#!/usr/bin/env python -tt +#!/usr/bin/env python3 # # Copyright (c) 2007 Red Hat, Inc. # Copyright (c) 2011 Intel, Inc. # -# This program is free software; you can redistribute it and/or modify it -# under the terms of the GNU General Public License as published by the Free -# Software Foundation; version 2 of the License +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY -# or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License -# for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., 59 -# Temple Place - Suite 330, Boston, MA 02111-1307, USA. class WicError(Exception): pass diff --git a/scripts/lib/wic/canned-wks/common.wks.inc b/scripts/lib/wic/canned-wks/common.wks.inc index 5cf2fd1..89880b4 100644 --- a/scripts/lib/wic/canned-wks/common.wks.inc +++ b/scripts/lib/wic/canned-wks/common.wks.inc @@ -1,3 +1,3 @@ # This file is included into 3 canned wks files from this directory part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024 -part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 +part / --source rootfs --use-uuid --fstype=ext4 --label platform --align 1024 diff --git a/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg b/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg index d5a07d2..c58e74a 100644 --- a/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg +++ b/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg @@ -12,16 +12,16 @@ DEFAULT Graphics console boot LABEL Graphics console boot KERNEL /vmlinuz -APPEND label=boot root=/dev/sda2 rootwait +APPEND label=boot rootwait LABEL Serial console boot KERNEL /vmlinuz -APPEND label=boot root=/dev/sda2 rootwait console=ttyS0,115200 +APPEND label=boot rootwait console=ttyS0,115200 LABEL Graphics console install KERNEL /vmlinuz -APPEND label=install root=/dev/sda2 rootwait +APPEND label=install rootwait LABEL Serial console install KERNEL /vmlinuz -APPEND label=install root=/dev/sda2 rootwait console=ttyS0,115200 +APPEND label=install rootwait console=ttyS0,115200 diff --git a/scripts/lib/wic/canned-wks/efi-bootdisk.wks.in b/scripts/lib/wic/canned-wks/efi-bootdisk.wks.in new file mode 100644 index 0000000..7300e65 --- /dev/null +++ b/scripts/lib/wic/canned-wks/efi-bootdisk.wks.in @@ -0,0 +1,3 @@ +bootloader --ptable gpt +part /boot --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/boot --fstype=vfat --label boot --active --align 1024 --use-uuid --overhead-factor 1.0 +part / --source rootfs --fstype=ext4 --label root --align 1024 --exclude-path boot/ diff --git a/scripts/lib/wic/canned-wks/mkhybridiso.wks b/scripts/lib/wic/canned-wks/mkhybridiso.wks index 9d34e9b..48c5ac4 100644 --- a/scripts/lib/wic/canned-wks/mkhybridiso.wks +++ b/scripts/lib/wic/canned-wks/mkhybridiso.wks @@ -2,6 +2,6 @@ # long-description: Creates an EFI and legacy bootable hybrid ISO image # which can be used on optical media as well as USB media. -part /boot --source isoimage-isohybrid --sourceparams="loader=grub-efi,image_name=HYBRID_ISO_IMG" --ondisk cd --label HYBRIDISO --fstype=ext4 +part /boot --source isoimage-isohybrid --sourceparams="loader=grub-efi,image_name=HYBRID_ISO_IMG" --ondisk cd --label HYBRIDISO bootloader --timeout=15 --append="" diff --git a/scripts/lib/wic/canned-wks/qemuriscv.wks b/scripts/lib/wic/canned-wks/qemuriscv.wks new file mode 100644 index 0000000..12c68b7 --- /dev/null +++ b/scripts/lib/wic/canned-wks/qemuriscv.wks @@ -0,0 +1,3 @@ +# short-description: Create qcow2 image for RISC-V QEMU machines + +part / --source rootfs --fstype=ext4 --label root --align 4096 --size 5G diff --git a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks index a6518a0..22b4521 100644 --- a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks +++ b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks @@ -4,5 +4,5 @@ include common.wks.inc -bootloader --timeout=0 --append="vga=0 uvesafb.mode_option=640x480-32 root=/dev/vda2 rw mem=256M ip=192.168.7.2::192.168.7.1:255.255.255.0 oprofile.timer=1 rootfstype=ext4 " +bootloader --timeout=0 --append="rw oprofile.timer=1 rootfstype=ext4 " diff --git a/scripts/lib/wic/canned-wks/sdimage-bootpart.wks b/scripts/lib/wic/canned-wks/sdimage-bootpart.wks index 7ffd632..63bc4da 100644 --- a/scripts/lib/wic/canned-wks/sdimage-bootpart.wks +++ b/scripts/lib/wic/canned-wks/sdimage-bootpart.wks @@ -2,5 +2,5 @@ # long-description: Creates a partitioned SD card image. Boot files # are located in the first vfat partition. -part /boot --source bootimg-partition --ondisk mmcblk --fstype=vfat --label boot --active --align 4 --size 16 -part / --source rootfs --ondisk mmcblk --fstype=ext4 --label root --align 4 +part /boot --source bootimg-partition --ondisk mmcblk0 --fstype=vfat --label boot --active --align 4 --size 16 +part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --label root --align 4 diff --git a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks index 4bd9d6a..95d7b97 100644 --- a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks +++ b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks @@ -2,10 +2,10 @@ # long-description: Creates a partitioned EFI disk image that the user # can directly dd to boot media. The selected bootloader is systemd-boot. -part /boot --source bootimg-efi --sourceparams="loader=systemd-boot" --ondisk sda --label msdos --active --align 1024 +part /boot --source bootimg-efi --sourceparams="loader=systemd-boot" --ondisk sda --label msdos --active --align 1024 --use-uuid part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid -part swap --ondisk sda --size 44 --label swap1 --fstype=swap +part swap --ondisk sda --size 44 --label swap1 --fstype=swap --use-uuid bootloader --ptable gpt --timeout=5 --append="rootwait rootfstype=ext4 console=ttyS0,115200 console=tty0" diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py index f59821f..018815b 100644 --- a/scripts/lib/wic/engine.py +++ b/scripts/lib/wic/engine.py @@ -1,21 +1,7 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Copyright (c) 2013, Intel Corporation. -# All rights reserved. # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # DESCRIPTION @@ -30,10 +16,18 @@ import logging import os +import tempfile +import json +import subprocess +import re + +from collections import namedtuple, OrderedDict +from distutils.spawn import find_executable from wic import WicError +from wic.filemap import sparse_copy from wic.pluginbase import PluginMgr -from wic.utils.misc import get_bitbake_var +from wic.misc import get_bitbake_var, exec_cmd logger = logging.getLogger('wic') @@ -82,7 +76,8 @@ def find_canned_image(scripts_path, wks_file): for fname in files: if fname.endswith("~") or fname.endswith("#"): continue - if fname.endswith(".wks") and wks_file + ".wks" == fname: + if ((fname.endswith(".wks") and wks_file + ".wks" == fname) or \ + (fname.endswith(".wks.in") and wks_file + ".wks.in" == fname)): fullpath = os.path.join(canned_wks_dir, fname) return fullpath return None @@ -99,7 +94,7 @@ def list_canned_images(scripts_path): for fname in files: if fname.endswith("~") or fname.endswith("#"): continue - if fname.endswith(".wks"): + if fname.endswith(".wks") or fname.endswith(".wks.in"): fullpath = os.path.join(canned_wks_dir, fname) with open(fullpath) as wks: for line in wks: @@ -108,7 +103,7 @@ def list_canned_images(scripts_path): if idx != -1: desc = line[idx + len("short-description:"):].strip() break - basename = os.path.splitext(fname)[0] + basename = fname.split('.')[0] print(" %s\t\t%s" % (basename.ljust(30), desc)) @@ -184,7 +179,7 @@ def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir, if not os.path.exists(options.outdir): os.makedirs(options.outdir) - pname = 'direct' + pname = options.imager plugin_class = PluginMgr.get_plugins('imager').get(pname) if not plugin_class: raise WicError('Unknown plugin: %s' % pname) @@ -201,17 +196,18 @@ def wic_list(args, scripts_path): """ Print the list of images or source plugins. """ - if len(args) < 1: + if args.list_type is None: return False - if args == ["images"]: + if args.list_type == "images": + list_canned_images(scripts_path) return True - elif args == ["source-plugins"]: + elif args.list_type == "source-plugins": list_source_plugins() return True - elif len(args) == 2 and args[1] == "help": - wks_file = args[0] + elif len(args.help_for) == 1 and args.help_for[0] == 'help': + wks_file = args.list_type fullpath = find_canned_image(scripts_path, wks_file) if not fullpath: raise WicError("No image named %s found, exiting. " @@ -224,6 +220,381 @@ def wic_list(args, scripts_path): return False + +class Disk: + def __init__(self, imagepath, native_sysroot, fstypes=('fat', 'ext')): + self.imagepath = imagepath + self.native_sysroot = native_sysroot + self.fstypes = fstypes + self._partitions = None + self._partimages = {} + self._lsector_size = None + self._psector_size = None + self._ptable_format = None + + # find parted + # read paths from $PATH environment variable + # if it fails, use hardcoded paths + pathlist = "/bin:/usr/bin:/usr/sbin:/sbin/" + try: + self.paths = os.environ['PATH'] + ":" + pathlist + except KeyError: + self.paths = pathlist + + if native_sysroot: + for path in pathlist.split(':'): + self.paths = "%s%s:%s" % (native_sysroot, path, self.paths) + + self.parted = find_executable("parted", self.paths) + if not self.parted: + raise WicError("Can't find executable parted") + + self.partitions = self.get_partitions() + + def __del__(self): + for path in self._partimages.values(): + os.unlink(path) + + def get_partitions(self): + if self._partitions is None: + self._partitions = OrderedDict() + out = exec_cmd("%s -sm %s unit B print" % (self.parted, self.imagepath)) + parttype = namedtuple("Part", "pnum start end size fstype") + splitted = out.splitlines() + # skip over possible errors in exec_cmd output + try: + idx =splitted.index("BYT;") + except ValueError: + raise WicError("Error getting partition information from %s" % (self.parted)) + lsector_size, psector_size, self._ptable_format = splitted[idx + 1].split(":")[3:6] + self._lsector_size = int(lsector_size) + self._psector_size = int(psector_size) + for line in splitted[idx + 2:]: + pnum, start, end, size, fstype = line.split(':')[:5] + partition = parttype(int(pnum), int(start[:-1]), int(end[:-1]), + int(size[:-1]), fstype) + self._partitions[pnum] = partition + + return self._partitions + + def __getattr__(self, name): + """Get path to the executable in a lazy way.""" + if name in ("mdir", "mcopy", "mdel", "mdeltree", "sfdisk", "e2fsck", + "resize2fs", "mkswap", "mkdosfs", "debugfs","blkid"): + aname = "_%s" % name + if aname not in self.__dict__: + setattr(self, aname, find_executable(name, self.paths)) + if aname not in self.__dict__ or self.__dict__[aname] is None: + raise WicError("Can't find executable '{}'".format(name)) + return self.__dict__[aname] + return self.__dict__[name] + + def _get_part_image(self, pnum): + if pnum not in self.partitions: + raise WicError("Partition %s is not in the image" % pnum) + part = self.partitions[pnum] + # check if fstype is supported + for fstype in self.fstypes: + if part.fstype.startswith(fstype): + break + else: + raise WicError("Not supported fstype: {}".format(part.fstype)) + if pnum not in self._partimages: + tmpf = tempfile.NamedTemporaryFile(prefix="wic-part") + dst_fname = tmpf.name + tmpf.close() + sparse_copy(self.imagepath, dst_fname, skip=part.start, length=part.size) + self._partimages[pnum] = dst_fname + + return self._partimages[pnum] + + def _put_part_image(self, pnum): + """Put partition image into partitioned image.""" + sparse_copy(self._partimages[pnum], self.imagepath, + seek=self.partitions[pnum].start) + + def dir(self, pnum, path): + if pnum not in self.partitions: + raise WicError("Partition %s is not in the image" % pnum) + + if self.partitions[pnum].fstype.startswith('ext'): + return exec_cmd("{} {} -R 'ls -l {}'".format(self.debugfs, + self._get_part_image(pnum), + path), as_shell=True) + else: # fat + return exec_cmd("{} -i {} ::{}".format(self.mdir, + self._get_part_image(pnum), + path)) + + def copy(self, src, dest): + """Copy partition image into wic image.""" + pnum = dest.part if isinstance(src, str) else src.part + + if self.partitions[pnum].fstype.startswith('ext'): + if isinstance(src, str): + cmd = "printf 'cd {}\nwrite {} {}\n' | {} -w {}".\ + format(os.path.dirname(dest.path), src, os.path.basename(src), + self.debugfs, self._get_part_image(pnum)) + else: # copy from wic + # run both dump and rdump to support both files and directory + cmd = "printf 'cd {}\ndump /{} {}\nrdump /{} {}\n' | {} {}".\ + format(os.path.dirname(src.path), src.path, + dest, src.path, dest, self.debugfs, + self._get_part_image(pnum)) + else: # fat + if isinstance(src, str): + cmd = "{} -i {} -snop {} ::{}".format(self.mcopy, + self._get_part_image(pnum), + src, dest.path) + else: + cmd = "{} -i {} -snop ::{} {}".format(self.mcopy, + self._get_part_image(pnum), + src.path, dest) + + exec_cmd(cmd, as_shell=True) + self._put_part_image(pnum) + + def remove_ext(self, pnum, path, recursive): + """ + Remove files/dirs and their contents from the partition. + This only applies to ext* partition. + """ + abs_path = re.sub('\/\/+', '/', path) + cmd = "{} {} -wR 'rm \"{}\"'".format(self.debugfs, + self._get_part_image(pnum), + abs_path) + out = exec_cmd(cmd , as_shell=True) + for line in out.splitlines(): + if line.startswith("rm:"): + if "file is a directory" in line: + if recursive: + # loop through content and delete them one by one if + # flaged with -r + subdirs = iter(self.dir(pnum, abs_path).splitlines()) + next(subdirs) + for subdir in subdirs: + dir = subdir.split(':')[1].split(" ", 1)[1] + if not dir == "." and not dir == "..": + self.remove_ext(pnum, "%s/%s" % (abs_path, dir), recursive) + + rmdir_out = exec_cmd("{} {} -wR 'rmdir \"{}\"'".format(self.debugfs, + self._get_part_image(pnum), + abs_path.rstrip('/')) + , as_shell=True) + + for rmdir_line in rmdir_out.splitlines(): + if "directory not empty" in rmdir_line: + raise WicError("Could not complete operation: \n%s \n" + "use -r to remove non-empty directory" % rmdir_line) + if rmdir_line.startswith("rmdir:"): + raise WicError("Could not complete operation: \n%s " + "\n%s" % (str(line), rmdir_line)) + + else: + raise WicError("Could not complete operation: \n%s " + "\nUnable to remove %s" % (str(line), abs_path)) + + def remove(self, pnum, path, recursive): + """Remove files/dirs from the partition.""" + partimg = self._get_part_image(pnum) + if self.partitions[pnum].fstype.startswith('ext'): + self.remove_ext(pnum, path, recursive) + + else: # fat + cmd = "{} -i {} ::{}".format(self.mdel, partimg, path) + try: + exec_cmd(cmd) + except WicError as err: + if "not found" in str(err) or "non empty" in str(err): + # mdel outputs 'File ... not found' or 'directory .. non empty" + # try to use mdeltree as path could be a directory + cmd = "{} -i {} ::{}".format(self.mdeltree, + partimg, path) + exec_cmd(cmd) + else: + raise err + self._put_part_image(pnum) + + def write(self, target, expand): + """Write disk image to the media or file.""" + def write_sfdisk_script(outf, parts): + for key, val in parts['partitiontable'].items(): + if key in ("partitions", "device", "firstlba", "lastlba"): + continue + if key == "id": + key = "label-id" + outf.write("{}: {}\n".format(key, val)) + outf.write("\n") + for part in parts['partitiontable']['partitions']: + line = '' + for name in ('attrs', 'name', 'size', 'type', 'uuid'): + if name == 'size' and part['type'] == 'f': + # don't write size for extended partition + continue + val = part.get(name) + if val: + line += '{}={}, '.format(name, val) + if line: + line = line[:-2] # strip ', ' + if part.get('bootable'): + line += ' ,bootable' + outf.write("{}\n".format(line)) + outf.flush() + + def read_ptable(path): + out = exec_cmd("{} -J {}".format(self.sfdisk, path)) + return json.loads(out) + + def write_ptable(parts, target): + with tempfile.NamedTemporaryFile(prefix="wic-sfdisk-", mode='w') as outf: + write_sfdisk_script(outf, parts) + cmd = "{} --no-reread {} < {} ".format(self.sfdisk, target, outf.name) + exec_cmd(cmd, as_shell=True) + + if expand is None: + sparse_copy(self.imagepath, target) + else: + # copy first sectors that may contain bootloader + sparse_copy(self.imagepath, target, length=2048 * self._lsector_size) + + # copy source partition table to the target + parts = read_ptable(self.imagepath) + write_ptable(parts, target) + + # get size of unpartitioned space + free = None + for line in exec_cmd("{} -F {}".format(self.sfdisk, target)).splitlines(): + if line.startswith("Unpartitioned space ") and line.endswith("sectors"): + free = int(line.split()[-2]) + # Align free space to a 2048 sector boundary. YOCTO #12840. + free = free - (free % 2048) + if free is None: + raise WicError("Can't get size of unpartitioned space") + + # calculate expanded partitions sizes + sizes = {} + num_auto_resize = 0 + for num, part in enumerate(parts['partitiontable']['partitions'], 1): + if num in expand: + if expand[num] != 0: # don't resize partition if size is set to 0 + sectors = expand[num] // self._lsector_size + free -= sectors - part['size'] + part['size'] = sectors + sizes[num] = sectors + elif part['type'] != 'f': + sizes[num] = -1 + num_auto_resize += 1 + + for num, part in enumerate(parts['partitiontable']['partitions'], 1): + if sizes.get(num) == -1: + part['size'] += free // num_auto_resize + + # write resized partition table to the target + write_ptable(parts, target) + + # read resized partition table + parts = read_ptable(target) + + # copy partitions content + for num, part in enumerate(parts['partitiontable']['partitions'], 1): + pnum = str(num) + fstype = self.partitions[pnum].fstype + + # copy unchanged partition + if part['size'] == self.partitions[pnum].size // self._lsector_size: + logger.info("copying unchanged partition {}".format(pnum)) + sparse_copy(self._get_part_image(pnum), target, seek=part['start'] * self._lsector_size) + continue + + # resize or re-create partitions + if fstype.startswith('ext') or fstype.startswith('fat') or \ + fstype.startswith('linux-swap'): + + partfname = None + with tempfile.NamedTemporaryFile(prefix="wic-part{}-".format(pnum)) as partf: + partfname = partf.name + + if fstype.startswith('ext'): + logger.info("resizing ext partition {}".format(pnum)) + partimg = self._get_part_image(pnum) + sparse_copy(partimg, partfname) + exec_cmd("{} -pf {}".format(self.e2fsck, partfname)) + exec_cmd("{} {} {}s".format(\ + self.resize2fs, partfname, part['size'])) + elif fstype.startswith('fat'): + logger.info("copying content of the fat partition {}".format(pnum)) + with tempfile.TemporaryDirectory(prefix='wic-fatdir-') as tmpdir: + # copy content to the temporary directory + cmd = "{} -snompi {} :: {}".format(self.mcopy, + self._get_part_image(pnum), + tmpdir) + exec_cmd(cmd) + # create new msdos partition + label = part.get("name") + label_str = "-n {}".format(label) if label else '' + + cmd = "{} {} -C {} {}".format(self.mkdosfs, label_str, partfname, + part['size']) + exec_cmd(cmd) + # copy content from the temporary directory to the new partition + cmd = "{} -snompi {} {}/* ::".format(self.mcopy, partfname, tmpdir) + exec_cmd(cmd, as_shell=True) + elif fstype.startswith('linux-swap'): + logger.info("creating swap partition {}".format(pnum)) + label = part.get("name") + label_str = "-L {}".format(label) if label else '' + out = exec_cmd("{} --probe {}".format(self.blkid, self._get_part_image(pnum))) + uuid = out[out.index("UUID=\"")+6:out.index("UUID=\"")+42] + uuid_str = "-U {}".format(uuid) if uuid else '' + with open(partfname, 'w') as sparse: + os.ftruncate(sparse.fileno(), part['size'] * self._lsector_size) + exec_cmd("{} {} {} {}".format(self.mkswap, label_str, uuid_str, partfname)) + sparse_copy(partfname, target, seek=part['start'] * self._lsector_size) + os.unlink(partfname) + elif part['type'] != 'f': + logger.warning("skipping partition {}: unsupported fstype {}".format(pnum, fstype)) + +def wic_ls(args, native_sysroot): + """List contents of partitioned image or vfat partition.""" + disk = Disk(args.path.image, native_sysroot) + if not args.path.part: + if disk.partitions: + print('Num Start End Size Fstype') + for part in disk.partitions.values(): + print("{:2d} {:12d} {:12d} {:12d} {}".format(\ + part.pnum, part.start, part.end, + part.size, part.fstype)) + else: + path = args.path.path or '/' + print(disk.dir(args.path.part, path)) + +def wic_cp(args, native_sysroot): + """ + Copy file or directory to/from the vfat/ext partition of + partitioned image. + """ + if isinstance(args.dest, str): + disk = Disk(args.src.image, native_sysroot) + else: + disk = Disk(args.dest.image, native_sysroot) + disk.copy(args.src, args.dest) + + +def wic_rm(args, native_sysroot): + """ + Remove files or directories from the vfat partition of + partitioned image. + """ + disk = Disk(args.path.image, native_sysroot) + disk.remove(args.path.part, args.path.path, args.recursive_delete) + +def wic_write(args, native_sysroot): + """ + Write image to a target device. + """ + disk = Disk(args.image, native_sysroot, ('fat', 'ext', 'linux-swap')) + disk.write(args.target, args.expand) + def find_canned(scripts_path, file_name): """ Find a file either by its path or by name in the canned files dir. diff --git a/scripts/lib/wic/filemap.py b/scripts/lib/wic/filemap.py index 080668e..4d9da28 100644 --- a/scripts/lib/wic/filemap.py +++ b/scripts/lib/wic/filemap.py @@ -1,13 +1,8 @@ +# # Copyright (c) 2012 Intel, Inc. # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License, version 2, -# as published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. """ This module implements python implements a way to get file block. Two methods @@ -22,6 +17,7 @@ and returns an instance of the class. # * Too many instance attributes (R0902) # pylint: disable=R0902 +import errno import os import struct import array @@ -34,14 +30,23 @@ def get_block_size(file_obj): Returns block size for file object 'file_obj'. Errors are indicated by the 'IOError' exception. """ - - from fcntl import ioctl - import struct - # Get the block size of the host file-system for the image file by calling # the FIGETBSZ ioctl (number 2). - binary_data = ioctl(file_obj, 2, struct.pack('I', 0)) - return struct.unpack('I', binary_data)[0] + try: + binary_data = fcntl.ioctl(file_obj, 2, struct.pack('I', 0)) + bsize = struct.unpack('I', binary_data)[0] + except OSError: + bsize = None + + # If ioctl causes OSError or give bsize to zero failback to os.fstat + if not bsize: + import os + stat = os.fstat(file_obj.fileno()) + if hasattr(stat, 'st_blksize'): + bsize = stat.st_blksize + else: + raise IOError("Unable to determine block size") + return bsize class ErrorNotSupp(Exception): """ @@ -137,15 +142,6 @@ class _FilemapBase(object): raise Error("the method is not implemented") - def block_is_unmapped(self, block): # pylint: disable=W0613,R0201 - """ - This method has has to be implemented by child classes. It returns - 'True' if block number 'block' of the image file is not mapped (hole) - and 'False' otherwise. - """ - - raise Error("the method is not implemented") - def get_mapped_ranges(self, start, count): # pylint: disable=W0613,R0201 """ This method has has to be implemented by child classes. This is a @@ -159,15 +155,6 @@ class _FilemapBase(object): raise Error("the method is not implemented") - def get_unmapped_ranges(self, start, count): # pylint: disable=W0613,R0201 - """ - This method has has to be implemented by child classes. Just like - 'get_mapped_ranges()', but yields unmapped block ranges instead - (holes). - """ - - raise Error("the method is not implemented") - # The 'SEEK_HOLE' and 'SEEK_DATA' options of the file seek system call _SEEK_DATA = 3 @@ -185,9 +172,9 @@ def _lseek(file_obj, offset, whence): except OSError as err: # The 'lseek' system call returns the ENXIO if there is no data or # hole starting from the specified offset. - if err.errno == os.errno.ENXIO: + if err.errno == errno.ENXIO: return -1 - elif err.errno == os.errno.EINVAL: + elif err.errno == errno.EINVAL: raise ErrorNotSupp("the kernel or file-system does not support " "\"SEEK_HOLE\" and \"SEEK_DATA\"") else: @@ -228,7 +215,7 @@ class FilemapSeek(_FilemapBase): try: tmp_obj = tempfile.TemporaryFile("w+", dir=directory) except IOError as err: - raise ErrorNotSupp("cannot create a temporary in \"%s\": %s" + raise ErrorNotSupp("cannot create a temporary in \"%s\": %s" \ % (directory, err)) try: @@ -260,15 +247,10 @@ class FilemapSeek(_FilemapBase): % (block, result)) return result - def block_is_unmapped(self, block): - """Refer the '_FilemapBase' class for the documentation.""" - return not self.block_is_mapped(block) - def _get_ranges(self, start, count, whence1, whence2): """ - This function implements 'get_mapped_ranges()' and - 'get_unmapped_ranges()' depending on what is passed in the 'whence1' - and 'whence2' arguments. + This function implements 'get_mapped_ranges()' depending + on what is passed in the 'whence1' and 'whence2' arguments. """ assert whence1 != whence2 @@ -298,12 +280,6 @@ class FilemapSeek(_FilemapBase): % (start, count, start + count - 1)) return self._get_ranges(start, count, _SEEK_DATA, _SEEK_HOLE) - def get_unmapped_ranges(self, start, count): - """Refer the '_FilemapBase' class for the documentation.""" - self._log.debug("FilemapSeek: get_unmapped_ranges(%d, %d(%d))" - % (start, count, start + count - 1)) - return self._get_ranges(start, count, _SEEK_HOLE, _SEEK_DATA) - # Below goes the FIEMAP ioctl implementation, which is not very readable # because it deals with the rather complex FIEMAP ioctl. To understand the @@ -390,12 +366,12 @@ class FilemapFiemap(_FilemapBase): except IOError as err: # Note, the FIEMAP ioctl is supported by the Linux kernel starting # from version 2.6.28 (year 2008). - if err.errno == os.errno.EOPNOTSUPP: + if err.errno == errno.EOPNOTSUPP: errstr = "FilemapFiemap: the FIEMAP ioctl is not supported " \ "by the file-system" self._log.debug(errstr) raise ErrorNotSupp(errstr) - if err.errno == os.errno.ENOTTY: + if err.errno == errno.ENOTTY: errstr = "FilemapFiemap: the FIEMAP ioctl is not supported " \ "by the kernel" self._log.debug(errstr) @@ -417,10 +393,6 @@ class FilemapFiemap(_FilemapBase): % (block, result)) return result - def block_is_unmapped(self, block): - """Refer the '_FilemapBase' class for the documentation.""" - return not self.block_is_mapped(block) - def _unpack_fiemap_extent(self, index): """ Unpack a 'struct fiemap_extent' structure object number 'index' from @@ -497,23 +469,28 @@ class FilemapFiemap(_FilemapBase): % (first_prev, last_prev)) yield (first_prev, last_prev) - def get_unmapped_ranges(self, start, count): +class FilemapNobmap(_FilemapBase): + """ + This class is used when both the 'SEEK_DATA/HOLE' and FIEMAP are not + supported by the filesystem or kernel. + """ + + def __init__(self, image, log=None): """Refer the '_FilemapBase' class for the documentation.""" - self._log.debug("FilemapFiemap: get_unmapped_ranges(%d, %d(%d))" - % (start, count, start + count - 1)) - hole_first = start - for first, last in self._do_get_mapped_ranges(start, count): - if first > hole_first: - self._log.debug("FilemapFiemap: yielding range (%d, %d)" - % (hole_first, first - 1)) - yield (hole_first, first - 1) - hole_first = last + 1 + # Call the base class constructor first + _FilemapBase.__init__(self, image, log) + self._log.debug("FilemapNobmap: initializing") - if hole_first < start + count: - self._log.debug("FilemapFiemap: yielding range (%d, %d)" - % (hole_first, start + count - 1)) - yield (hole_first, start + count - 1) + def block_is_mapped(self, block): + """Refer the '_FilemapBase' class for the documentation.""" + return True + + def get_mapped_ranges(self, start, count): + """Refer the '_FilemapBase' class for the documentation.""" + self._log.debug("FilemapNobmap: get_mapped_ranges(%d, %d(%d))" + % (start, count, start + count - 1)) + yield (start, start + count -1) def filemap(image, log=None): """ @@ -528,26 +505,56 @@ def filemap(image, log=None): try: return FilemapFiemap(image, log) except ErrorNotSupp: - return FilemapSeek(image, log) + try: + return FilemapSeek(image, log) + except ErrorNotSupp: + return FilemapNobmap(image, log) -def sparse_copy(src_fname, dst_fname, offset=0, skip=0): - """Efficiently copy sparse file to or into another file.""" - fmap = filemap(src_fname) +def sparse_copy(src_fname, dst_fname, skip=0, seek=0, + length=0, api=None): + """ + Efficiently copy sparse file to or into another file. + + src_fname: path to source file + dst_fname: path to destination file + skip: skip N bytes at thestart of src + seek: seek N bytes from the start of dst + length: read N bytes from src and write them to dst + api: FilemapFiemap or FilemapSeek object + """ + if not api: + api = filemap + fmap = api(src_fname) try: dst_file = open(dst_fname, 'r+b') except IOError: dst_file = open(dst_fname, 'wb') - dst_file.truncate(os.path.getsize(src_fname)) + if length: + dst_size = length + seek + else: + dst_size = os.path.getsize(src_fname) + seek - skip + dst_file.truncate(dst_size) + written = 0 for first, last in fmap.get_mapped_ranges(0, fmap.blocks_cnt): start = first * fmap.block_size end = (last + 1) * fmap.block_size + if skip >= end: + continue + if start < skip < end: - fmap._f_image.seek(skip, os.SEEK_SET) - else: - fmap._f_image.seek(start, os.SEEK_SET) - dst_file.seek(offset + start, os.SEEK_SET) + start = skip + + fmap._f_image.seek(start, os.SEEK_SET) + + written += start - skip - written + if length and written >= length: + dst_file.seek(seek + length, os.SEEK_SET) + dst_file.close() + return + + dst_file.seek(seek + start - skip, os.SEEK_SET) chunk_size = 1024 * 1024 to_read = end - start @@ -556,7 +563,14 @@ def sparse_copy(src_fname, dst_fname, offset=0, skip=0): while read < to_read: if read + chunk_size > to_read: chunk_size = to_read - read - chunk = fmap._f_image.read(chunk_size) + size = chunk_size + if length and written + size > length: + size = length - written + chunk = fmap._f_image.read(size) dst_file.write(chunk) - read += chunk_size + read += size + written += size + if written == length: + dst_file.close() + return dst_file.close() diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py index 148da89..bd3a2b9 100644 --- a/scripts/lib/wic/help.py +++ b/scripts/lib/wic/help.py @@ -1,21 +1,6 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- -# # Copyright (c) 2013, Intel Corporation. -# All rights reserved. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # DESCRIPTION # This module implements some basic help invocation functions along @@ -56,7 +41,7 @@ def wic_help(args, usage_str, subcommands): """ Subcommand help dispatcher. """ - if len(args) == 1 or not display_help(args[1], subcommands): + if args.help_topic == None or not display_help(args.help_topic, subcommands): print(usage_str) @@ -82,19 +67,20 @@ def invoke_subcommand(args, parser, main_command_usage, subcommands): Dispatch to subcommand handler borrowed from combo-layer. Should use argparse, but has to work in 2.6. """ - if not args: + if not args.command: logger.error("No subcommand specified, exiting") parser.print_help() return 1 - elif args[0] == "help": + elif args.command == "help": wic_help(args, main_command_usage, subcommands) - elif args[0] not in subcommands: - logger.error("Unsupported subcommand %s, exiting\n", args[0]) + elif args.command not in subcommands: + logger.error("Unsupported subcommand %s, exiting\n", args.command) parser.print_help() return 1 else: - usage = subcommands.get(args[0], subcommand_error)[1] - subcommands.get(args[0], subcommand_error)[0](args[1:], usage) + subcmd = subcommands.get(args.command, subcommand_error) + usage = subcmd[1] + subcmd[0](args, usage) ## @@ -130,10 +116,10 @@ wic_create_usage = """ Create a new OpenEmbedded image usage: wic create <wks file or image name> [-o <DIRNAME> | --outdir <DIRNAME>] - [-i <JSON PROPERTY FILE> | --infile <JSON PROPERTY_FILE>] [-e | --image-name] [-s, --skip-build-check] [-D, --debug] [-r, --rootfs-dir] [-b, --bootimg-dir] [-k, --kernel-dir] [-n, --native-sysroot] [-f, --build-rootfs] + [-c, --compress-with] [-m, --bmap] This command creates an OpenEmbedded image based on the 'OE kickstart commands' found in the <wks file>. @@ -154,7 +140,7 @@ SYNOPSIS [-e | --image-name] [-s, --skip-build-check] [-D, --debug] [-r, --rootfs-dir] [-b, --bootimg-dir] [-k, --kernel-dir] [-n, --native-sysroot] [-f, --build-rootfs] - [-c, --compress-with] [-m, --bmap] + [-c, --compress-with] [-m, --bmap] [--no-fstab-update] DESCRIPTION This command creates an OpenEmbedded image based on the 'OE @@ -226,6 +212,11 @@ DESCRIPTION The -m option is used to produce .bmap file for the image. This file can be used to flash image using bmaptool utility. + + The --no-fstab-update option is used to doesn't change fstab file. When + using this option the final fstab file will be same that in rootfs and + wic doesn't update file, e.g adding a new mount point. User can control + the fstab file content in base-files recipe. """ wic_list_usage = """ @@ -283,6 +274,243 @@ DESCRIPTION details. """ +wic_ls_usage = """ + + List content of a partitioned image + + usage: wic ls <image>[:<partition>[<path>]] [--native-sysroot <path>] + + This command outputs either list of image partitions or directory contents + of vfat and ext* partitions. + + See 'wic help ls' for more detailed instructions. + +""" + +wic_ls_help = """ + +NAME + wic ls - List contents of partitioned image or partition + +SYNOPSIS + wic ls <image> + wic ls <image>:<vfat or ext* partition> + wic ls <image>:<vfat or ext* partition><path> + wic ls <image>:<vfat or ext* partition><path> --native-sysroot <path> + +DESCRIPTION + This command lists either partitions of the image or directory contents + of vfat or ext* partitions. + + The first form it lists partitions of the image. + For example: + $ wic ls tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic + Num Start End Size Fstype + 1 1048576 24438783 23390208 fat16 + 2 25165824 50315263 25149440 ext4 + + Second and third form list directory content of the partition: + $ wic ls tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 + Volume in drive : is boot + Volume Serial Number is 2DF2-5F02 + Directory for ::/ + + efi <DIR> 2017-05-11 10:54 + startup nsh 26 2017-05-11 10:54 + vmlinuz 6922288 2017-05-11 10:54 + 3 files 6 922 314 bytes + 15 818 752 bytes free + + + $ wic ls tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/EFI/boot/ + Volume in drive : is boot + Volume Serial Number is 2DF2-5F02 + Directory for ::/EFI/boot + + . <DIR> 2017-05-11 10:54 + .. <DIR> 2017-05-11 10:54 + grub cfg 679 2017-05-11 10:54 + bootx64 efi 571392 2017-05-11 10:54 + 4 files 572 071 bytes + 15 818 752 bytes free + + The -n option is used to specify the path to the native sysroot + containing the tools(parted and mtools) to use. + +""" + +wic_cp_usage = """ + + Copy files and directories to/from the vfat or ext* partition + + usage: wic cp <src> <dest> [--native-sysroot <path>] + + source/destination image in format <image>:<partition>[<path>] + + This command copies files or directories either + - from local to vfat or ext* partitions of partitioned image + - from vfat or ext* partitions of partitioned image to local + + See 'wic help cp' for more detailed instructions. + +""" + +wic_cp_help = """ + +NAME + wic cp - copy files and directories to/from the vfat or ext* partitions + +SYNOPSIS + wic cp <src> <dest>:<partition> + wic cp <src>:<partition> <dest> + wic cp <src> <dest-image>:<partition><path> + wic cp <src> <dest-image>:<partition><path> --native-sysroot <path> + +DESCRIPTION + This command copies files or directories either + - from local to vfat or ext* partitions of partitioned image + - from vfat or ext* partitions of partitioned image to local + + The first form of it copies file or directory to the root directory of + the partition: + $ wic cp test.wks tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 + $ wic ls tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 + Volume in drive : is boot + Volume Serial Number is DB4C-FD4C + Directory for ::/ + + efi <DIR> 2017-05-24 18:15 + loader <DIR> 2017-05-24 18:15 + startup nsh 26 2017-05-24 18:15 + vmlinuz 6926384 2017-05-24 18:15 + test wks 628 2017-05-24 21:22 + 5 files 6 927 038 bytes + 15 677 440 bytes free + + The second form of the command copies file or directory to the specified directory + on the partition: + $ wic cp test tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/efi/ + $ wic ls tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/efi/ + Volume in drive : is boot + Volume Serial Number is DB4C-FD4C + Directory for ::/efi + + . <DIR> 2017-05-24 18:15 + .. <DIR> 2017-05-24 18:15 + boot <DIR> 2017-05-24 18:15 + test <DIR> 2017-05-24 21:27 + 4 files 0 bytes + 15 675 392 bytes free + + The third form of the command copies file or directory from the specified directory + on the partition to local: + $ wic cp tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/vmlinuz test + + The -n option is used to specify the path to the native sysroot + containing the tools(parted and mtools) to use. +""" + +wic_rm_usage = """ + + Remove files or directories from the vfat or ext* partitions + + usage: wic rm <image>:<partition><path> [--native-sysroot <path>] + + This command removes files or directories from the vfat or ext* partitions of + the partitioned image. + + See 'wic help rm' for more detailed instructions. + +""" + +wic_rm_help = """ + +NAME + wic rm - remove files or directories from the vfat or ext* partitions + +SYNOPSIS + wic rm <src> <image>:<partition><path> + wic rm <src> <image>:<partition><path> --native-sysroot <path> + wic rm -r <image>:<partition><path> + +DESCRIPTION + This command removes files or directories from the vfat or ext* partition of the + partitioned image: + + $ wic ls ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 + Volume in drive : is boot + Volume Serial Number is 11D0-DE21 + Directory for ::/ + + libcom32 c32 186500 2017-06-02 15:15 + libutil c32 24148 2017-06-02 15:15 + syslinux cfg 209 2017-06-02 15:15 + vesamenu c32 27104 2017-06-02 15:15 + vmlinuz 6926384 2017-06-02 15:15 + 5 files 7 164 345 bytes + 16 582 656 bytes free + + $ wic rm ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/libutil.c32 + + $ wic ls ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 + Volume in drive : is boot + Volume Serial Number is 11D0-DE21 + Directory for ::/ + + libcom32 c32 186500 2017-06-02 15:15 + syslinux cfg 209 2017-06-02 15:15 + vesamenu c32 27104 2017-06-02 15:15 + vmlinuz 6926384 2017-06-02 15:15 + 4 files 7 140 197 bytes + 16 607 232 bytes free + + The -n option is used to specify the path to the native sysroot + containing the tools(parted and mtools) to use. + + The -r option is used to remove directories and their contents + recursively,this only applies to ext* partition. +""" + +wic_write_usage = """ + + Write image to a device + + usage: wic write <image> <target device> [--expand [rules]] [--native-sysroot <path>] + + This command writes partitioned image to a target device (USB stick, SD card etc). + + See 'wic help write' for more detailed instructions. + +""" + +wic_write_help = """ + +NAME + wic write - write an image to a device + +SYNOPSIS + wic write <image> <target> + wic write <image> <target> --expand auto + wic write <image> <target> --expand 1:100M,2:300M + wic write <image> <target> --native-sysroot <path> + +DESCRIPTION + This command writes an image to a target device (USB stick, SD card etc) + + $ wic write ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic /dev/sdb + + The --expand option is used to resize image partitions. + --expand auto expands partitions to occupy all free space available on the target device. + It's also possible to specify expansion rules in a format + <partition>:<size>[,<partition>:<size>...] for one or more partitions. + Specifying size 0 will keep partition unmodified. + Note: Resizing boot partition can result in non-bootable image for non-EFI images. It is + recommended to use size 0 for boot partition to keep image bootable. + + The --native-sysroot option is used to specify the path to the native sysroot + containing the tools(parted, resize2fs) to use. +""" + wic_plugins_help = """ NAME @@ -308,7 +536,8 @@ DESCRIPTION Source plugins can also be implemented and added by external layers - any plugins found in a scripts/lib/wic/plugins/source/ - directory in an external layer will also be made available. + or lib/wic/plugins/source/ directory in an external layer will + also be made available. When the wic implementation needs to invoke a partition-specific implementation, it looks for the plugin that has the same name as @@ -346,6 +575,10 @@ DESCRIPTION partition. In other words, it 'prepares' the final partition image which will be incorporated into the disk image. + do_post_partition() + Called after the partition is created. It is useful to add post + operations e.g. signing the partition. + do_configure_partition() Called before do_prepare_partition(), typically used to create custom configuration files for a partition, for @@ -632,8 +865,11 @@ DESCRIPTION Partitions with a <mountpoint> specified will be automatically mounted. This is achieved by wic adding entries to the fstab during image generation. In order for a valid fstab to be generated one of the - --ondrive, --ondisk or --use-uuid partition options must be used for - each partition that specifies a mountpoint. + --ondrive, --ondisk, --use-uuid or --use-label partition options must + be used for each partition that specifies a mountpoint. Note that with + --use-{uuid,label} and non-root <mountpoint>, including swap, the mount + program must understand the PARTUUID or LABEL syntax. This currently + excludes the busybox versions of these applications. The following are supported 'part' options: @@ -687,6 +923,8 @@ DESCRIPTION apply to partitions created using '--source rootfs' (see --source above). Valid values are: + vfat + msdos ext2 ext3 ext4 @@ -706,6 +944,14 @@ DESCRIPTION label is already in use by another filesystem, a new label is created for the partition. + --use-label: This option is specific to wic. It makes wic to use the + label in /etc/fstab to specify a partition. If the + --use-label and --use-uuid are used at the same time, + we prefer the uuid because it is less likely to cause + name confliction. We don't support using this parameter + on the root partition since it requires an initramfs to + parse this value and we do not currently support that. + --active: Marks the partition as active. --align (in KBytes): This option is specific to wic and says @@ -719,11 +965,31 @@ DESCRIPTION bootloaders. --exclude-path: This option is specific to wic. It excludes the given - absolute path from the resulting image. If the path + relative path from the resulting image. If the path ends with a slash, only the content of the directory is omitted, not the directory itself. This option only has an effect with the rootfs source plugin. + --include-path: This option is specific to wic. It adds the contents + of the given path or a rootfs to the resulting image. + The option contains two fields, the origin and the + destination. When the origin is a rootfs, it follows + the same logic as the rootfs-dir argument and the + permissions and owners are kept. When the origin is a + path, it is relative to the directory in which wic is + running not the rootfs itself so use of an absolute + path is recommended, and the owner and group is set to + root:root. If no destination is given it is + automatically set to the root of the rootfs. This + option only has an effect with the rootfs source + plugin. + + --change-directory: This option is specific to wic. It changes to the + given directory before copying the files. This + option is useful when we want to split a rootfs in + multiple partitions and we want to keep the right + permissions and usernames in all the partitions. + --extra-space: This option is specific to wic. It adds extra space after the space filled by the content of the partition. The final size can go @@ -738,6 +1004,8 @@ DESCRIPTION This option cannot be used with --fixed-size option. + --part-name: This option is specific to wic. It specifies name for GPT partitions. + --part-type: This option is specific to wic. It specifies partition type GUID for GPT partitions. List of partition type GUIDS can be found here: @@ -752,10 +1020,21 @@ DESCRIPTION in bootloader configuration before running wic. In this case .wks file can be generated or modified to set preconfigured parition UUID using this option. + --fsuuid: This option is specific to wic. It specifies filesystem UUID. + It's useful if preconfigured filesystem UUID is added to kernel command line + in bootloader configuration before running wic. In this case .wks file can + be generated or modified to set preconfigured filesystem UUID using this option. + --system-id: This option is specific to wic. It specifies partition system id. It's useful for the harware that requires non-default partition system ids. The parameter in one byte long hex number either with 0x prefix or without it. + --mkfs-extraopts: This option specifies extra options to pass to mkfs utility. + NOTE, that wic uses default options for some filesystems, for example + '-S 512' for mkfs.fat or '-F -i 8192' for mkfs.ext. Those options will + not take effect when --mkfs-extraopts is used. This should be taken into + account when using --mkfs-extraopts. + * bootloader This command allows the user to specify various bootloader @@ -793,3 +1072,67 @@ DESCRIPTION .wks files. """ + +wic_help_help = """ +NAME + wic help - display a help topic + +DESCRIPTION + Specify a help topic to display it. Topics are shown above. +""" + + +wic_help = """ +Creates a customized OpenEmbedded image. + +Usage: wic [--version] + wic help [COMMAND or TOPIC] + wic COMMAND [ARGS] + + usage 1: Returns the current version of Wic + usage 2: Returns detailed help for a COMMAND or TOPIC + usage 3: Executes COMMAND + + +COMMAND: + + list - List available canned images and source plugins + ls - List contents of partitioned image or partition + rm - Remove files or directories from the vfat or ext* partitions + help - Show help for a wic COMMAND or TOPIC + write - Write an image to a device + cp - Copy files and directories to the vfat or ext* partitions + create - Create a new OpenEmbedded image + + +TOPIC: + overview - Presents an overall overview of Wic + plugins - Presents an overview and API for Wic plugins + kickstart - Presents a Wic kicstart file reference + + +Examples: + + $ wic --version + + Returns the current version of Wic + + + $ wic help cp + + Returns the SYNOPSIS and DESCRIPTION for the Wic "cp" command. + + + $ wic list images + + Returns the list of canned images (i.e. *.wks files located in + the /scripts/lib/wic/canned-wks directory. + + + $ wic create mkefidisk -e core-image-minimal + + Creates an EFI disk image from artifacts used in a previous + core-image-minimal build in standard BitBake locations + (e.g. Cooked Mode). + +""" diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py index a039300..3453d9c 100644 --- a/scripts/lib/wic/ksparser.py +++ b/scripts/lib/wic/ksparser.py @@ -1,21 +1,8 @@ -#!/usr/bin/env python -tt -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- +#!/usr/bin/env python3 # # Copyright (c) 2016 Intel, Inc. # -# This program is free software; you can redistribute it and/or modify it -# under the terms of the GNU General Public License as published by the Free -# Software Foundation; version 2 of the License -# -# This program is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY -# or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License -# for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., 59 -# Temple Place - Suite 330, Boston, MA 02111-1307, USA. +# SPDX-License-Identifier: GPL-2.0-only # # DESCRIPTION # This module provides parser for kickstart format @@ -28,14 +15,30 @@ import os import shlex import logging +import re from argparse import ArgumentParser, ArgumentError, ArgumentTypeError from wic.engine import find_canned from wic.partition import Partition +from wic.misc import get_bitbake_var logger = logging.getLogger('wic') +__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}") + +def expand_line(line): + while True: + m = __expand_var_regexp__.search(line) + if not m: + return line + key = m.group()[2:-1] + val = get_bitbake_var(key) + if val is None: + logger.warning("cannot expand variable %s" % key) + return line + line = line[:m.start()] + val + line[m.end():] + class KickStartError(Exception): """Custom exception.""" pass @@ -48,26 +51,31 @@ class KickStartParser(ArgumentParser): def error(self, message): raise ArgumentError(None, message) -def sizetype(arg): - """ - Custom type for ArgumentParser - Converts size string in <num>[K|k|M|G] format into the integer value - """ - if arg.isdigit(): - return int(arg) * 1024 - - if not arg[:-1].isdigit(): - raise ArgumentTypeError("Invalid size: %r" % arg) +def sizetype(default): + def f(arg): + """ + Custom type for ArgumentParser + Converts size string in <num>[K|k|M|G] format into the integer value + """ + try: + suffix = default + size = int(arg) + except ValueError: + try: + suffix = arg[-1:] + size = int(arg[:-1]) + except ValueError: + raise ArgumentTypeError("Invalid size: %r" % arg) - size = int(arg[:-1]) - if arg.endswith("k") or arg.endswith("K"): - return size - if arg.endswith("M"): - return size * 1024 - if arg.endswith("G"): - return size * 1024 * 1024 + if suffix == "k" or suffix == "K": + return size + if suffix == "M": + return size * 1024 + if suffix == "G": + return size * 1024 * 1024 - raise ArgumentTypeError("Invalid size: %r" % arg) + raise ArgumentTypeError("Invalid size: %r" % arg) + return f def overheadtype(arg): """ @@ -114,7 +122,7 @@ def systemidtype(arg): return arg class KickStart(): - """"Kickstart parser implementation.""" + """Kickstart parser implementation.""" DEFAULT_EXTRA_SPACE = 10*1024 DEFAULT_OVERHEAD_FACTOR = 1.3 @@ -133,30 +141,41 @@ class KickStart(): part.add_argument('mountpoint', nargs='?') part.add_argument('--active', action='store_true') part.add_argument('--align', type=int) + part.add_argument('--offset', type=sizetype("K")) part.add_argument('--exclude-path', nargs='+') - part.add_argument("--extra-space", type=sizetype) + part.add_argument('--include-path', nargs='+', action='append') + part.add_argument('--change-directory') + part.add_argument("--extra-space", type=sizetype("M")) part.add_argument('--fsoptions', dest='fsopts') - part.add_argument('--fstype') + part.add_argument('--fstype', default='vfat', + choices=('ext2', 'ext3', 'ext4', 'btrfs', + 'squashfs', 'vfat', 'msdos', 'swap')) + part.add_argument('--mkfs-extraopts', default='') part.add_argument('--label') + part.add_argument('--use-label', action='store_true') part.add_argument('--no-table', action='store_true') part.add_argument('--ondisk', '--ondrive', dest='disk', default='sda') part.add_argument("--overhead-factor", type=overheadtype) + part.add_argument('--part-name') part.add_argument('--part-type') part.add_argument('--rootfs-dir') + part.add_argument('--type', default='primary', + choices = ('primary', 'logical')) # --size and --fixed-size cannot be specified together; options # ----extra-space and --overhead-factor should also raise a parser # --error, but since nesting mutually exclusive groups does not work, # ----extra-space/--overhead-factor are handled later sizeexcl = part.add_mutually_exclusive_group() - sizeexcl.add_argument('--size', type=sizetype, default=0) - sizeexcl.add_argument('--fixed-size', type=sizetype, default=0) + sizeexcl.add_argument('--size', type=sizetype("M"), default=0) + sizeexcl.add_argument('--fixed-size', type=sizetype("M"), default=0) part.add_argument('--source') part.add_argument('--sourceparams') part.add_argument('--system-id', type=systemidtype) part.add_argument('--use-uuid', action='store_true') part.add_argument('--uuid') + part.add_argument('--fsuuid') bootloader = subparsers.add_parser('bootloader') bootloader.add_argument('--append') @@ -184,6 +203,7 @@ class KickStart(): line = line.strip() lineno += 1 if line and line[0] != '#': + line = expand_line(line) try: line_args = shlex.split(line) parsed = parser.parse_args(line_args) @@ -191,6 +211,20 @@ class KickStart(): raise KickStartError('%s:%d: %s' % \ (confpath, lineno, err)) if line.startswith('part'): + # SquashFS does not support filesystem UUID + if parsed.fstype == 'squashfs': + if parsed.fsuuid: + err = "%s:%d: SquashFS does not support UUID" \ + % (confpath, lineno) + raise KickStartError(err) + if parsed.label: + err = "%s:%d: SquashFS does not support LABEL" \ + % (confpath, lineno) + raise KickStartError(err) + if parsed.use_label and not parsed.label: + err = "%s:%d: Must set the label with --label" \ + % (confpath, lineno) + raise KickStartError(err) # using ArgumentParser one cannot easily tell if option # was passed as argument, if said option has a default # value; --overhead-factor/--extra-space cannot be used @@ -219,6 +253,11 @@ class KickStart(): elif line.startswith('bootloader'): if not self.bootloader: self.bootloader = parsed + # Concatenate the strings set in APPEND + append_var = get_bitbake_var("APPEND") + if append_var: + self.bootloader.append = ' '.join(filter(None, \ + (self.bootloader.append, append_var))) else: err = "%s:%d: more than one bootloader specified" \ % (confpath, lineno) diff --git a/scripts/lib/wic/utils/misc.py b/scripts/lib/wic/misc.py similarity index 70% rename from scripts/lib/wic/utils/misc.py rename to scripts/lib/wic/misc.py index c941112..91975ba 100644 --- a/scripts/lib/wic/utils/misc.py +++ b/scripts/lib/wic/misc.py @@ -1,21 +1,7 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Copyright (c) 2013, Intel Corporation. -# All rights reserved. # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # DESCRIPTION # This module provides a place to collect various wic-related utils @@ -29,12 +15,12 @@ import logging import os import re +import subprocess from collections import defaultdict from distutils import spawn from wic import WicError -from wic.utils import runner logger = logging.getLogger('wic') @@ -43,6 +29,9 @@ NATIVE_RECIPES = {"bmaptool": "bmap-tools", "grub-mkimage": "grub-efi", "isohybrid": "syslinux", "mcopy": "mtools", + "mdel" : "mtools", + "mdeltree" : "mtools", + "mdir" : "mtools", "mkdosfs": "dosfstools", "mkisofs": "cdrtools", "mkfs.btrfs": "btrfs-tools", @@ -52,14 +41,48 @@ NATIVE_RECIPES = {"bmaptool": "bmap-tools", "mkfs.vfat": "dosfstools", "mksquashfs": "squashfs-tools", "mkswap": "util-linux", - "mmd": "syslinux", + "mmd": "mtools", "parted": "parted", "sfdisk": "util-linux", "sgdisk": "gptfdisk", - "syslinux": "syslinux" + "syslinux": "syslinux", + "tar": "tar" } -def _exec_cmd(cmd_and_args, as_shell=False, catch=3): +def runtool(cmdln_or_args): + """ wrapper for most of the subprocess calls + input: + cmdln_or_args: can be both args and cmdln str (shell=True) + return: + rc, output + """ + if isinstance(cmdln_or_args, list): + cmd = cmdln_or_args[0] + shell = False + else: + import shlex + cmd = shlex.split(cmdln_or_args)[0] + shell = True + + sout = subprocess.PIPE + serr = subprocess.STDOUT + + try: + process = subprocess.Popen(cmdln_or_args, stdout=sout, + stderr=serr, shell=shell) + sout, serr = process.communicate() + # combine stdout and stderr, filter None out and decode + out = ''.join([out.decode('utf-8') for out in [sout, serr] if out]) + except OSError as err: + if err.errno == 2: + # [Errno 2] No such file or directory + raise WicError('Cannot run command: %s, lost dependency?' % cmd) + else: + raise # relay + + return process.returncode, out + +def _exec_cmd(cmd_and_args, as_shell=False): """ Execute command, catching stderr, stdout @@ -70,9 +93,9 @@ def _exec_cmd(cmd_and_args, as_shell=False, catch=3): logger.debug(args) if as_shell: - ret, out = runner.runtool(cmd_and_args, catch) + ret, out = runtool(cmd_and_args) else: - ret, out = runner.runtool(args, catch) + ret, out = runtool(args) out = out.strip() if ret != 0: raise WicError("_exec_cmd: %s returned '%s' instead of 0\noutput: %s" % \ @@ -84,14 +107,23 @@ def _exec_cmd(cmd_and_args, as_shell=False, catch=3): return ret, out -def exec_cmd(cmd_and_args, as_shell=False, catch=3): +def exec_cmd(cmd_and_args, as_shell=False): """ Execute command, return output """ - return _exec_cmd(cmd_and_args, as_shell, catch)[1] + return _exec_cmd(cmd_and_args, as_shell)[1] + +def find_executable(cmd, paths): + recipe = cmd + if recipe in NATIVE_RECIPES: + recipe = NATIVE_RECIPES[recipe] + provided = get_bitbake_var("ASSUME_PROVIDED") + if provided and "%s-native" % recipe in provided: + return True + return spawn.find_executable(cmd, paths) -def exec_native_cmd(cmd_and_args, native_sysroot, catch=3, pseudo=""): +def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""): """ Execute native command, catching stderr, stdout @@ -106,19 +138,16 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3, pseudo=""): if pseudo: cmd_and_args = pseudo + cmd_and_args - wtools_sysroot = get_bitbake_var("RECIPE_SYSROOT_NATIVE", "wic-tools") + native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin" % \ + (native_sysroot, native_sysroot, native_sysroot) - native_paths = \ - "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/sbin:%s/usr/sbin:%s/usr/bin" % \ - (wtools_sysroot, wtools_sysroot, wtools_sysroot, - native_sysroot, native_sysroot, native_sysroot) native_cmd_and_args = "export PATH=%s:$PATH;%s" % \ - (native_paths, cmd_and_args) + (native_paths, cmd_and_args) logger.debug("exec_native_cmd: %s", native_cmd_and_args) # If the command isn't in the native sysroot say we failed. - if spawn.find_executable(args[0], native_paths): - ret, out = _exec_cmd(native_cmd_and_args, True, catch) + if find_executable(args[0], native_paths): + ret, out = _exec_cmd(native_cmd_and_args, True) else: ret = 127 out = "can't find native executable %s in %s" % (args[0], native_paths) @@ -131,8 +160,8 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3, pseudo=""): "was not found (see details above).\n\n" % prog recipe = NATIVE_RECIPES.get(prog) if recipe: - msg += "Please bake it with 'bitbake %s-native' "\ - "and try again.\n" % recipe + msg += "Please make sure wic-tools have %s-native in its DEPENDS, "\ + "build it with 'bitbake wic-tools' and try again.\n" % recipe else: msg += "Wic failed to find a recipe to build native %s. Please "\ "file a bug against wic.\n" % prog @@ -153,7 +182,7 @@ class BitbakeVars(defaultdict): self.default_image = None self.vars_dir = None - def _parse_line(self, line, image, matcher=re.compile(r"^(\w+)=(.+)")): + def _parse_line(self, line, image, matcher=re.compile(r"^([a-zA-Z0-9\-_+./~]+)=(.*)")): """ Parse one line from bitbake -e output or from .env file. Put result key-value pair into the storage. diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py index 8e32afc..85eb15c 100644 --- a/scripts/lib/wic/partition.py +++ b/scripts/lib/wic/partition.py @@ -1,21 +1,7 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Copyright (c) 2013-2016 Intel Corporation. -# All rights reserved. # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # DESCRIPTION # This module provides the OpenEmbedded partition object definitions. @@ -26,10 +12,10 @@ import logging import os -import tempfile +import uuid from wic import WicError -from wic.utils.misc import exec_cmd, exec_native_cmd, get_bitbake_var +from wic.misc import exec_cmd, exec_native_cmd, get_bitbake_var from wic.pluginbase import PluginMgr logger = logging.getLogger('wic') @@ -44,13 +30,19 @@ class Partition(): self.device = None self.extra_space = args.extra_space self.exclude_path = args.exclude_path + self.include_path = args.include_path + self.change_directory = args.change_directory self.fsopts = args.fsopts self.fstype = args.fstype self.label = args.label + self.use_label = args.use_label + self.mkfs_extraopts = args.mkfs_extraopts self.mountpoint = args.mountpoint self.no_table = args.no_table self.num = None + self.offset = args.offset self.overhead_factor = args.overhead_factor + self.part_name = args.part_name self.part_type = args.part_type self.rootfs_dir = args.rootfs_dir self.size = args.size @@ -60,10 +52,11 @@ class Partition(): self.system_id = args.system_id self.use_uuid = args.use_uuid self.uuid = args.uuid + self.fsuuid = args.fsuuid + self.type = args.type self.lineno = lineno self.source_file = "" - self.sourceparams_dict = {} def get_extra_block_count(self, current_blocks): """ @@ -136,22 +129,24 @@ class Partition(): "specify a non-zero --size/--fixed-size for that " "partition." % self.mountpoint) - if self.fstype and self.fstype == "swap": + if self.fstype == "swap": self.prepare_swap_partition(cr_workdir, oe_builddir, native_sysroot) self.source_file = "%s/fs.%s" % (cr_workdir, self.fstype) - elif self.fstype: + else: + if self.fstype == 'squashfs': + raise WicError("It's not possible to create empty squashfs " + "partition '%s'" % (self.mountpoint)) + rootfs = "%s/fs_%s.%s.%s" % (cr_workdir, self.label, self.lineno, self.fstype) if os.path.isfile(rootfs): os.remove(rootfs) - for prefix in ("ext", "btrfs", "vfat", "squashfs"): - if self.fstype.startswith(prefix): - method = getattr(self, - "prepare_empty_partition_" + prefix) - method(rootfs, oe_builddir, native_sysroot) - self.source_file = rootfs - break + + prefix = "ext" if self.fstype.startswith("ext") else self.fstype + method = getattr(self, "prepare_empty_partition_" + prefix) + method(rootfs, oe_builddir, native_sysroot) + self.source_file = rootfs return plugins = PluginMgr.get_plugins('source') @@ -168,7 +163,7 @@ class Partition(): # Split sourceparams string of the form key1=val1[,key2=val2,...] # into a dict. Also accepts valueless keys i.e. without = splitted = self.sourceparams.split(',') - srcparams_dict = dict(par.split('=') for par in splitted if par) + srcparams_dict = dict(par.split('=', 1) for par in splitted if par) plugin = PluginMgr.get_plugins('source')[self.source] plugin.do_configure_partition(self, srcparams_dict, creator, @@ -180,6 +175,9 @@ class Partition(): plugin.do_prepare_partition(self, srcparams_dict, creator, cr_workdir, oe_builddir, bootimg_dir, kernel_dir, rootfs_dir, native_sysroot) + plugin.do_post_partition(self, srcparams_dict, creator, + cr_workdir, oe_builddir, bootimg_dir, + kernel_dir, rootfs_dir, native_sysroot) # further processing required Partition.size to be an integer, make # sure that it is one @@ -193,74 +191,57 @@ class Partition(): "larger (%d kB) than its allowed size %d kB" % (self.mountpoint, self.size, self.fixed_size)) - def prepare_rootfs_from_fs_image(self, cr_workdir, oe_builddir, - rootfs_dir): - """ - Handle an already-created partition e.g. xxx.ext3 - """ - rootfs = oe_builddir - du_cmd = "du -Lbks %s" % rootfs - out = exec_cmd(du_cmd) - rootfs_size = out.split()[0] - - self.size = int(rootfs_size) - self.source_file = rootfs - def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir, - native_sysroot): + native_sysroot, real_rootfs = True, pseudo_dir = None): """ Prepare content for a rootfs partition i.e. create a partition and fill it from a /rootfs dir. - Currently handles ext2/3/4, btrfs and vfat. + Currently handles ext2/3/4, btrfs, vfat and squashfs. """ p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot) - p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", - "%s/../pseudo" % rootfs_dir) - p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir) - p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1") - pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix - pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir - pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd - pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp - pseudo += "%s " % get_bitbake_var("FAKEROOTCMD") + if (pseudo_dir): + pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix + pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir + pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" + pseudo += "%s " % get_bitbake_var("FAKEROOTCMD") + else: + pseudo = None rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label, self.lineno, self.fstype) if os.path.isfile(rootfs): os.remove(rootfs) - if not self.fstype: - raise WicError("File system for partition %s not specified in " - "kickstart, use --fstype option" % self.mountpoint) - - # Get rootfs size from bitbake variable if it's not set in .ks file - if not self.size: - # Bitbake variable ROOTFS_SIZE is calculated in - # Image._get_rootfs_size method from meta/lib/oe/image.py - # using IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT, - # IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE + if not self.size and real_rootfs: + # The rootfs size is not set in .ks file so try to get it + # from bitbake variable rsize_bb = get_bitbake_var('ROOTFS_SIZE') - if rsize_bb: - logger.warning('overhead-factor was specified, but size was not,' - ' so bitbake variables will be used for the size.' - ' In this case both IMAGE_OVERHEAD_FACTOR and ' - '--overhead-factor will be applied') + rdir = get_bitbake_var('IMAGE_ROOTFS') + if rsize_bb and rdir == rootfs_dir: + # Bitbake variable ROOTFS_SIZE is calculated in + # Image._get_rootfs_size method from meta/lib/oe/image.py + # using IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT, + # IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE self.size = int(round(float(rsize_bb))) - - for prefix in ("ext", "btrfs", "vfat", "squashfs"): - if self.fstype.startswith(prefix): - method = getattr(self, "prepare_rootfs_" + prefix) - method(rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo) - - self.source_file = rootfs - - # get the rootfs size in the right units for kickstart (kB) - du_cmd = "du -Lbks %s" % rootfs + else: + # Bitbake variable ROOTFS_SIZE is not defined so compute it + # from the rootfs_dir size using the same logic found in + # get_rootfs_size() from meta/classes/image.bbclass + du_cmd = "du -ks %s" % rootfs_dir out = exec_cmd(du_cmd) self.size = int(out.split()[0]) - break + prefix = "ext" if self.fstype.startswith("ext") else self.fstype + method = getattr(self, "prepare_rootfs_" + prefix) + method(rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo) + self.source_file = rootfs + + # get the rootfs size in the right units for kickstart (kB) + du_cmd = "du -Lbks %s" % rootfs + out = exec_cmd(du_cmd) + self.size = int(out.split()[0]) def prepare_rootfs_ext(self, rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo): @@ -276,25 +257,23 @@ class Partition(): with open(rootfs, 'w') as sparse: os.ftruncate(sparse.fileno(), rootfs_size * 1024) - extra_imagecmd = "-i 8192" + extraopts = self.mkfs_extraopts or "-F -i 8192" label_str = "" if self.label: label_str = "-L %s" % self.label - mkfs_cmd = "mkfs.%s -F %s %s %s -d %s" % \ - (self.fstype, extra_imagecmd, rootfs, label_str, rootfs_dir) + mkfs_cmd = "mkfs.%s %s %s %s -U %s -d %s" % \ + (self.fstype, extraopts, rootfs, label_str, self.fsuuid, rootfs_dir) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) - mkfs_cmd = "fsck.%s -fy %s" % (self.fstype, rootfs) + mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) def prepare_rootfs_btrfs(self, rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo): """ Prepare content for a btrfs rootfs partition. - - Currently handles ext2/3/4 and btrfs. """ du_cmd = "du -ks %s" % rootfs_dir out = exec_cmd(du_cmd) @@ -309,14 +288,15 @@ class Partition(): if self.label: label_str = "-L %s" % self.label - mkfs_cmd = "mkfs.%s -b %d -r %s %s %s" % \ - (self.fstype, rootfs_size * 1024, rootfs_dir, label_str, rootfs) + mkfs_cmd = "mkfs.%s -b %d -r %s %s %s -U %s %s" % \ + (self.fstype, rootfs_size * 1024, rootfs_dir, label_str, + self.mkfs_extraopts, self.fsuuid, rootfs) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) - def prepare_rootfs_vfat(self, rootfs, oe_builddir, rootfs_dir, - native_sysroot, pseudo): + def prepare_rootfs_msdos(self, rootfs, oe_builddir, rootfs_dir, + native_sysroot, pseudo): """ - Prepare content for a vfat rootfs partition. + Prepare content for a msdos/vfat rootfs partition. """ du_cmd = "du -bks %s" % rootfs_dir out = exec_cmd(du_cmd) @@ -328,7 +308,15 @@ class Partition(): if self.label: label_str = "-n %s" % self.label - dosfs_cmd = "mkdosfs %s -S 512 -C %s %d" % (label_str, rootfs, rootfs_size) + size_str = "" + if self.fstype == 'msdos': + size_str = "-F 16" # FAT 16 + + extraopts = self.mkfs_extraopts or '-S 512' + + dosfs_cmd = "mkdosfs %s -i %s %s %s -C %s %d" % \ + (label_str, self.fsuuid, size_str, extraopts, rootfs, + rootfs_size) exec_native_cmd(dosfs_cmd, native_sysroot) mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir) @@ -337,13 +325,16 @@ class Partition(): chmod_cmd = "chmod 644 %s" % rootfs exec_cmd(chmod_cmd) + prepare_rootfs_vfat = prepare_rootfs_msdos + def prepare_rootfs_squashfs(self, rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo): """ Prepare content for a squashfs rootfs partition. """ - squashfs_cmd = "mksquashfs %s %s -noappend" % \ - (rootfs_dir, rootfs) + extraopts = self.mkfs_extraopts or '-noappend' + squashfs_cmd = "mksquashfs %s %s %s" % \ + (rootfs_dir, rootfs, extraopts) exec_native_cmd(squashfs_cmd, native_sysroot, pseudo=pseudo) def prepare_empty_partition_ext(self, rootfs, oe_builddir, @@ -355,14 +346,14 @@ class Partition(): with open(rootfs, 'w') as sparse: os.ftruncate(sparse.fileno(), size * 1024) - extra_imagecmd = "-i 8192" + extraopts = self.mkfs_extraopts or "-i 8192" label_str = "" if self.label: label_str = "-L %s" % self.label - mkfs_cmd = "mkfs.%s -F %s %s %s" % \ - (self.fstype, extra_imagecmd, label_str, rootfs) + mkfs_cmd = "mkfs.%s -F %s %s -U %s %s" % \ + (self.fstype, extraopts, label_str, self.fsuuid, rootfs) exec_native_cmd(mkfs_cmd, native_sysroot) def prepare_empty_partition_btrfs(self, rootfs, oe_builddir, @@ -378,12 +369,13 @@ class Partition(): if self.label: label_str = "-L %s" % self.label - mkfs_cmd = "mkfs.%s -b %d %s %s" % \ - (self.fstype, self.size * 1024, label_str, rootfs) + mkfs_cmd = "mkfs.%s -b %d %s -U %s %s %s" % \ + (self.fstype, self.size * 1024, label_str, self.fsuuid, + self.mkfs_extraopts, rootfs) exec_native_cmd(mkfs_cmd, native_sysroot) - def prepare_empty_partition_vfat(self, rootfs, oe_builddir, - native_sysroot): + def prepare_empty_partition_msdos(self, rootfs, oe_builddir, + native_sysroot): """ Prepare an empty vfat partition. """ @@ -393,40 +385,22 @@ class Partition(): if self.label: label_str = "-n %s" % self.label - dosfs_cmd = "mkdosfs %s -S 512 -C %s %d" % (label_str, rootfs, blocks) - exec_native_cmd(dosfs_cmd, native_sysroot) + size_str = "" + if self.fstype == 'msdos': + size_str = "-F 16" # FAT 16 - chmod_cmd = "chmod 644 %s" % rootfs - exec_cmd(chmod_cmd) + extraopts = self.mkfs_extraopts or '-S 512' - def prepare_empty_partition_squashfs(self, cr_workdir, oe_builddir, - native_sysroot): - """ - Prepare an empty squashfs partition. - """ - logger.warning("Creating of an empty squashfs %s partition was attempted. " - "Proceeding as requested.", self.mountpoint) - - path = "%s/fs_%s.%s" % (cr_workdir, self.label, self.fstype) - if os.path.isfile(path): - os.remove(path) + dosfs_cmd = "mkdosfs %s -i %s %s %s -C %s %d" % \ + (label_str, self.fsuuid, extraopts, size_str, rootfs, + blocks) - # it is not possible to create a squashfs without source data, - # thus prepare an empty temp dir that is used as source - tmpdir = tempfile.mkdtemp() - - squashfs_cmd = "mksquashfs %s %s -noappend" % \ - (tmpdir, path) - exec_native_cmd(squashfs_cmd, native_sysroot) - - os.rmdir(tmpdir) + exec_native_cmd(dosfs_cmd, native_sysroot) - # get the rootfs size in the right units for kickstart (kB) - du_cmd = "du -Lbks %s" % path - out = exec_cmd(du_cmd) - fs_size = out.split()[0] + chmod_cmd = "chmod 644 %s" % rootfs + exec_cmd(chmod_cmd) - self.size = int(fs_size) + prepare_empty_partition_vfat = prepare_empty_partition_msdos def prepare_swap_partition(self, cr_workdir, oe_builddir, native_sysroot): """ @@ -437,9 +411,9 @@ class Partition(): with open(path, 'w') as sparse: os.ftruncate(sparse.fileno(), self.size * 1024) - import uuid label_str = "" if self.label: label_str = "-L %s" % self.label - mkswap_cmd = "mkswap %s -U %s %s" % (label_str, str(uuid.uuid1()), path) + + mkswap_cmd = "mkswap %s -U %s %s" % (label_str, self.fsuuid, path) exec_native_cmd(mkswap_cmd, native_sysroot) diff --git a/scripts/lib/wic/pluginbase.py b/scripts/lib/wic/pluginbase.py index fb3d179..d9b4e57 100644 --- a/scripts/lib/wic/pluginbase.py +++ b/scripts/lib/wic/pluginbase.py @@ -1,19 +1,9 @@ -#!/usr/bin/env python -tt +#!/usr/bin/env python3 # # Copyright (c) 2011 Intel, Inc. # -# This program is free software; you can redistribute it and/or modify it -# under the terms of the GNU General Public License as published by the Free -# Software Foundation; version 2 of the License +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY -# or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License -# for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., 59 -# Temple Place - Suite 330, Boston, MA 02111-1307, USA. __all__ = ['ImagerPlugin', 'SourcePlugin'] @@ -24,11 +14,11 @@ from collections import defaultdict from importlib.machinery import SourceFileLoader from wic import WicError -from wic.utils.misc import get_bitbake_var +from wic.misc import get_bitbake_var PLUGIN_TYPES = ["imager", "source"] -SCRIPTS_PLUGIN_DIR = "scripts/lib/wic/plugins" +SCRIPTS_PLUGIN_DIR = ["scripts/lib/wic/plugins", "lib/wic/plugins"] logger = logging.getLogger('wic') @@ -48,10 +38,11 @@ class PluginMgr: cls._plugin_dirs = [os.path.join(os.path.dirname(__file__), 'plugins')] layers = get_bitbake_var("BBLAYERS") or '' for layer_path in layers.split(): - path = os.path.join(layer_path, SCRIPTS_PLUGIN_DIR) - path = os.path.abspath(os.path.expanduser(path)) - if path not in cls._plugin_dirs and os.path.isdir(path): - cls._plugin_dirs.insert(0, path) + for script_plugin_dir in SCRIPTS_PLUGIN_DIR: + path = os.path.join(layer_path, script_plugin_dir) + path = os.path.abspath(os.path.expanduser(path)) + if path not in cls._plugin_dirs and os.path.isdir(path): + cls._plugin_dirs.insert(0, path) if ptype not in PLUGINS: # load all ptype plugins @@ -138,3 +129,12 @@ class SourcePlugin(metaclass=PluginMeta): """ logger.debug("SourcePlugin: do_prepare_partition: part: %s", part) + @classmethod + def do_post_partition(cls, part, source_params, creator, cr_workdir, + oe_builddir, bootimg_dir, kernel_dir, rootfs_dir, + native_sysroot): + """ + Called after the partition is created. It is useful to add post + operations e.g. security signing the partition. + """ + logger.debug("SourcePlugin: do_post_partition: part: %s", part) diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py index 7d38ab3..2f01999 100644 --- a/scripts/lib/wic/plugins/imager/direct.py +++ b/scripts/lib/wic/plugins/imager/direct.py @@ -1,21 +1,7 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Copyright (c) 2013, Intel Corporation. -# All rights reserved. # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # DESCRIPTION # This implements the 'direct' imager plugin class for 'wic' @@ -26,17 +12,20 @@ import logging import os +import random import shutil import tempfile import uuid from time import strftime +from oe.path import copyhardlinktree + from wic import WicError from wic.filemap import sparse_copy from wic.ksparser import KickStart, KickStartError from wic.pluginbase import PluginMgr, ImagerPlugin -from wic.utils.misc import get_bitbake_var, exec_cmd, exec_native_cmd +from wic.misc import get_bitbake_var, exec_cmd, exec_native_cmd logger = logging.getLogger('wic') @@ -68,6 +57,8 @@ class DirectPlugin(ImagerPlugin): self.outdir = options.outdir self.compressor = options.compressor self.bmap = options.bmap + self.no_fstab_update = options.no_fstab_update + self.original_fstab = None self.name = "%s-%s" % (os.path.splitext(os.path.basename(wks_file))[0], strftime("%Y%m%d%H%M")) @@ -113,26 +104,38 @@ class DirectPlugin(ImagerPlugin): with open(fstab_path) as fstab: fstab_lines = fstab.readlines() + self.original_fstab = fstab_lines.copy() if self._update_fstab(fstab_lines, self.parts): - shutil.copyfile(fstab_path, fstab_path + ".orig") - with open(fstab_path, "w") as fstab: fstab.writelines(fstab_lines) - - return fstab_path + else: + self.original_fstab = None def _update_fstab(self, fstab_lines, parts): """Assume partition order same as in wks""" updated = False for part in parts: if not part.realnum or not part.mountpoint \ - or part.mountpoint in ("/", "/boot"): + or part.mountpoint == "/": continue - # mmc device partitions are named mmcblk0p1, mmcblk0p2.. - prefix = 'p' if part.disk.startswith('mmcblk') else '' - device_name = "/dev/%s%s%d" % (part.disk, prefix, part.realnum) + if part.use_uuid: + if part.fsuuid: + # FAT UUID is different from others + if len(part.fsuuid) == 10: + device_name = "UUID=%s-%s" % \ + (part.fsuuid[2:6], part.fsuuid[6:]) + else: + device_name = "UUID=%s" % part.fsuuid + else: + device_name = "PARTUUID=%s" % part.uuid + elif part.use_label: + device_name = "LABEL=%s" % part.label + else: + # mmc device partitions are named mmcblk0p1, mmcblk0p2.. + prefix = 'p' if part.disk.startswith('mmcblk') else '' + device_name = "/dev/%s%s%d" % (part.disk, prefix, part.realnum) opts = part.fsopts if part.fsopts else "defaults" line = "\t".join([device_name, part.mountpoint, part.fstype, @@ -156,7 +159,8 @@ class DirectPlugin(ImagerPlugin): filesystems from the artifacts directly and combine them into a partitioned image. """ - fstab_path = self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR")) + if not self.no_fstab_update: + self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR")) for part in self.parts: # get rootfs size from bitbake variable if it's not set in .ks file @@ -173,10 +177,6 @@ class DirectPlugin(ImagerPlugin): part.size = int(round(float(rsize_bb))) self._image.prepare(self) - - if fstab_path: - shutil.move(fstab_path + ".orig", fstab_path) - self._image.layout_partitions() self._image.create() @@ -205,8 +205,10 @@ class DirectPlugin(ImagerPlugin): # Generate .bmap if self.bmap: logger.debug("Generating bmap file for %s", disk_name) - exec_native_cmd("bmaptool create %s -o %s.bmap" % (full_path, full_path), - self.native_sysroot) + python = os.path.join(self.native_sysroot, 'usr/bin/python3-native/python3') + bmaptool = os.path.join(self.native_sysroot, 'usr/bin/bmaptool') + exec_native_cmd("%s %s create %s -o %s.bmap" % \ + (python, bmaptool, full_path, full_path), self.native_sysroot) # Compress the image if self.compressor: logger.debug("Compressing disk %s with %s", disk_name, self.compressor) @@ -233,7 +235,8 @@ class DirectPlugin(ImagerPlugin): suffix = ':' else: suffix = '["%s"]:' % (part.mountpoint or part.label) - msg += ' ROOTFS_DIR%s%s\n' % (suffix.ljust(20), part.rootfs_dir) + rootdir = part.rootfs_dir + msg += ' ROOTFS_DIR%s%s\n' % (suffix.ljust(20), rootdir) msg += ' BOOTIMG_DIR: %s\n' % self.bootimg_dir msg += ' KERNEL_DIR: %s\n' % self.kernel_dir @@ -270,6 +273,12 @@ class DirectPlugin(ImagerPlugin): if os.path.isfile(path): shutil.move(path, os.path.join(self.outdir, fname)) + #Restore original fstab + if self.original_fstab: + fstab_path = self.rootfs_dir.get("ROOTFS_DIR") + "/etc/fstab" + with open(fstab_path, "w") as fstab: + fstab.writelines(self.original_fstab) + # remove work directory shutil.rmtree(self.workdir, ignore_errors=True) @@ -291,18 +300,23 @@ class PartitionedImage(): self.path = path # Path to the image file self.numpart = 0 # Number of allocated partitions self.realpart = 0 # Number of partitions in the partition table + self.primary_part_num = 0 # Number of primary partitions (msdos) + self.extendedpart = 0 # Create extended partition before this logical partition (msdos) + self.extended_size_sec = 0 # Size of exteded partition (msdos) + self.logical_part_cnt = 0 # Number of total logical paritions (msdos) self.offset = 0 # Offset of next partition (in sectors) self.min_size = 0 # Minimum required disk size to fit # all partitions (in bytes) self.ptable_format = ptable_format # Partition table format # Disk system identifier - self.identifier = int.from_bytes(os.urandom(4), 'little') + self.identifier = random.SystemRandom().randint(1, 0xffffffff) self.partitions = partitions self.partimages = [] # Size of a sector used in calculations self.sector_size = SECTOR_SIZE self.native_sysroot = native_sysroot + num_real_partitions = len([p for p in self.partitions if not p.no_table]) # calculate the real partition number, accounting for partitions not # in the partition table and logical partitions @@ -312,18 +326,23 @@ class PartitionedImage(): part.realnum = 0 else: realnum += 1 - if self.ptable_format == 'msdos' and realnum > 3: + if self.ptable_format == 'msdos' and realnum > 3 and num_real_partitions > 4: part.realnum = realnum + 1 continue part.realnum = realnum - # generate parition UUIDs + # generate parition and filesystem UUIDs for part in self.partitions: if not part.uuid and part.use_uuid: if self.ptable_format == 'gpt': part.uuid = str(uuid.uuid4()) else: # msdos partition table - part.uuid = '%0x-%02d' % (self.identifier, part.realnum) + part.uuid = '%08x-%02d' % (self.identifier, part.realnum) + if not part.fsuuid: + if part.fstype == 'vfat' or part.fstype == 'msdos': + part.fsuuid = '0x' + str(uuid.uuid4())[:8].upper() + else: + part.fsuuid = str(uuid.uuid4()) def prepare(self, imager): """Prepare an image. Call prepare method of all image partitions.""" @@ -352,6 +371,10 @@ class PartitionedImage(): for num in range(len(self.partitions)): part = self.partitions[num] + if self.ptable_format == 'msdos' and part.part_name: + raise WicError("setting custom partition name is not " \ + "implemented for msdos partitions") + if self.ptable_format == 'msdos' and part.part_type: # The --part-type can also be implemented for MBR partitions, # in which case it would map to the 1-byte "partition type" @@ -373,12 +396,16 @@ class PartitionedImage(): # Skip one sector required for the partitioning scheme overhead self.offset += overhead - if self.realpart > 3 and num_real_partitions > 4: + if self.ptable_format == "msdos": + if self.primary_part_num > 3 or \ + (self.extendedpart == 0 and self.primary_part_num >= 3 and num_real_partitions > 4): + part.type = 'logical' # Reserve a sector for EBR for every logical partition # before alignment is performed. - if self.ptable_format == "msdos": - self.offset += 1 + if part.type == 'logical': + self.offset += 2 + align_sectors = 0 if part.align: # If not first partition and we do have alignment set we need # to align the partition. @@ -401,21 +428,43 @@ class PartitionedImage(): # increase the offset so we actually start the partition on right alignment self.offset += align_sectors + if part.offset is not None: + offset = (part.offset * 1024) // self.sector_size + + if offset * self.sector_size != part.offset * 1024: + raise WicError("Could not place %s%s at offset %dK with sector size %d" % (part.disk, self.numpart, part.offset, self.sector_size)) + + delta = offset - self.offset + if delta < 0: + raise WicError("Could not place %s%s at offset %dK: next free sector is %d (delta: %d)" % (part.disk, self.numpart, part.offset, self.offset, delta)) + + logger.debug("Skipping %d sectors to place %s%s at offset %dK", + delta, part.disk, self.numpart, part.offset) + + self.offset = offset + part.start = self.offset self.offset += part.size_sec - part.type = 'primary' if not part.no_table: part.num = self.realpart else: part.num = 0 - if self.ptable_format == "msdos": - # only count the partitions that are in partition table - if num_real_partitions > 4: - if self.realpart > 3: - part.type = 'logical' - part.num = self.realpart + 1 + if self.ptable_format == "msdos" and not part.no_table: + if part.type == 'logical': + self.logical_part_cnt += 1 + part.num = self.logical_part_cnt + 4 + if self.extendedpart == 0: + # Create extended partition as a primary partition + self.primary_part_num += 1 + self.extendedpart = part.num + else: + self.extended_size_sec += align_sectors + self.extended_size_sec += part.size_sec + 2 + else: + self.primary_part_num += 1 + part.num = self.primary_part_num logger.debug("Assigned %s to %s%d, sectors range %d-%d size %d " "sectors (%d bytes).", part.mountpoint, part.disk, @@ -465,7 +514,7 @@ class PartitionedImage(): if part.num == 0: continue - if self.ptable_format == "msdos" and part.num == 5: + if self.ptable_format == "msdos" and part.num == self.extendedpart: # Create an extended partition (note: extended # partition is described in MBR and contains all # logical partitions). The logical partitions save a @@ -478,8 +527,8 @@ class PartitionedImage(): # add a sector at the back, so that there is enough # room for all logical partitions. self._create_partition(self.path, "extended", - None, part.start - 1, - self.offset - part.start + 1) + None, part.start - 2, + self.extended_size_sec) if part.fstype == "swap": parted_fs_type = "linux-swap" @@ -487,8 +536,8 @@ class PartitionedImage(): parted_fs_type = "fat32" elif part.fstype == "msdos": parted_fs_type = "fat16" - elif part.fstype == "ontrackdm6aux3": - parted_fs_type = "ontrackdm6aux3" + if not part.system_id: + part.system_id = '0x6' # FAT16 else: # Type for ext2/ext3/ext4/btrfs parted_fs_type = "ext2" @@ -505,6 +554,13 @@ class PartitionedImage(): self._create_partition(self.path, part.type, parted_fs_type, part.start, part.size_sec) + if part.part_name: + logger.debug("partition %d: set name to %s", + part.num, part.part_name) + exec_native_cmd("sgdisk --change-name=%d:%s %s" % \ + (part.num, part.part_name, + self.path), self.native_sysroot) + if part.part_type: logger.debug("partition %d: set type UID to %s", part.num, part.part_type) @@ -538,21 +594,8 @@ class PartitionedImage(): (self.path, part.num, part.system_id), self.native_sysroot) - # Parted defaults to enabling the lba flag for fat16 partitions, - # which causes compatibility issues with some firmware (and really - # isn't necessary). - if parted_fs_type == "fat16": - if self.ptable_format == 'msdos': - logger.debug("Disable 'lba' flag for partition '%s' on disk '%s'", - part.num, self.path) - exec_native_cmd("parted -s %s set %d lba off" % \ - (self.path, part.num), - self.native_sysroot) - def cleanup(self): - # remove partition images - for image in set(self.partimages): - os.remove(image) + pass def assemble(self): logger.debug("Installing partitions") @@ -561,7 +604,7 @@ class PartitionedImage(): source = part.source_file if source: # install source_file contents into a partition - sparse_copy(source, self.path, part.start * self.sector_size) + sparse_copy(source, self.path, seek=part.start * self.sector_size) logger.debug("Installed %s in partition %d, sectors %d-%d, " "size %d sectors", source, part.num, part.start, diff --git a/scripts/lib/wic/plugins/source/bootimg-biosplusefi.py b/scripts/lib/wic/plugins/source/bootimg-biosplusefi.py new file mode 100644 index 0000000..5bd7390 --- /dev/null +++ b/scripts/lib/wic/plugins/source/bootimg-biosplusefi.py @@ -0,0 +1,213 @@ +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# DESCRIPTION +# This implements the 'bootimg-biosplusefi' source plugin class for 'wic' +# +# AUTHORS +# William Bourque <wbourque [at) gmail.com> + +import types + +from wic.pluginbase import SourcePlugin +from importlib.machinery import SourceFileLoader + +class BootimgBiosPlusEFIPlugin(SourcePlugin): + """ + Create MBR + EFI boot partition + + This plugin creates a boot partition that contains both + legacy BIOS and EFI content. It will be able to boot from both. + This is useful when managing PC fleet with some older machines + without EFI support. + + Note it is possible to create an image that can boot from both + legacy BIOS and EFI by defining two partitions : one with arg + --source bootimg-efi and another one with --source bootimg-pcbios. + However, this method has the obvious downside that it requires TWO + partitions to be created on the storage device. + Both partitions will also be marked as "bootable" which does not work on + most BIOS, has BIOS often uses the "bootable" flag to determine + what to boot. If you have such a BIOS, you need to manually remove the + "bootable" flag from the EFI partition for the drive to be bootable. + Having two partitions also seems to confuse wic : the content of + the first partition will be duplicated into the second, even though it + will not be used at all. + + Also, unlike "isoimage-isohybrid" that also does BIOS and EFI, this plugin + allows you to have more than only a single rootfs partitions and does + not turn the rootfs into an initramfs RAM image. + + This plugin is made to put everything into a single /boot partition so it + does not have the limitations listed above. + + The plugin is made so it does tries not to reimplement what's already + been done in other plugins; as such it imports "bootimg-pcbios" + and "bootimg-efi". + Plugin "bootimg-pcbios" is used to generate legacy BIOS boot. + Plugin "bootimg-efi" is used to generate the UEFI boot. Note that it + requires a --sourceparams argument to know which loader to use; refer + to "bootimg-efi" code/documentation for the list of loader. + + Imports are handled with "SourceFileLoader" from importlib as it is + otherwise very difficult to import module that has hyphen "-" in their + filename. + The SourcePlugin() methods used in the plugins (do_install_disk, + do_configure_partition, do_prepare_partition) are then called on both, + beginning by "bootimg-efi". + + Plugin options, such as "--sourceparams" can still be passed to a + plugin, as long they does not cause issue in the other plugin. + + Example wic configuration: + part /boot --source bootimg-biosplusefi --sourceparams="loader=grub-efi"\\ + --ondisk sda --label os_boot --active --align 1024 --use-uuid + """ + + name = 'bootimg-biosplusefi' + + __PCBIOS_MODULE_NAME = "bootimg-pcbios" + __EFI_MODULE_NAME = "bootimg-efi" + + __imgEFIObj = None + __imgBiosObj = None + + @classmethod + def __init__(cls): + """ + Constructor (init) + """ + + # XXX + # For some reasons, __init__ constructor is never called. + # Something to do with how pluginbase works? + cls.__instanciateSubClasses() + + @classmethod + def __instanciateSubClasses(cls): + """ + + """ + + # Import bootimg-pcbios (class name "BootimgPcbiosPlugin") + modulePath = os.path.join(os.path.dirname(os.path.realpath(__file__)), + cls.__PCBIOS_MODULE_NAME + ".py") + loader = SourceFileLoader(cls.__PCBIOS_MODULE_NAME, modulePath) + mod = types.ModuleType(loader.name) + loader.exec_module(mod) + cls.__imgBiosObj = mod.BootimgPcbiosPlugin() + + # Import bootimg-efi (class name "BootimgEFIPlugin") + modulePath = os.path.join(os.path.dirname(os.path.realpath(__file__)), + cls.__EFI_MODULE_NAME + ".py") + loader = SourceFileLoader(cls.__EFI_MODULE_NAME, modulePath) + mod = types.ModuleType(loader.name) + loader.exec_module(mod) + cls.__imgEFIObj = mod.BootimgEFIPlugin() + + @classmethod + def do_install_disk(cls, disk, disk_name, creator, workdir, oe_builddir, + bootimg_dir, kernel_dir, native_sysroot): + """ + Called after all partitions have been prepared and assembled into a + disk image. + """ + + if ( (not cls.__imgEFIObj) or (not cls.__imgBiosObj) ): + cls.__instanciateSubClasses() + + cls.__imgEFIObj.do_install_disk( + disk, + disk_name, + creator, + workdir, + oe_builddir, + bootimg_dir, + kernel_dir, + native_sysroot) + + cls.__imgBiosObj.do_install_disk( + disk, + disk_name, + creator, + workdir, + oe_builddir, + bootimg_dir, + kernel_dir, + native_sysroot) + + @classmethod + def do_configure_partition(cls, part, source_params, creator, cr_workdir, + oe_builddir, bootimg_dir, kernel_dir, + native_sysroot): + """ + Called before do_prepare_partition() + """ + + if ( (not cls.__imgEFIObj) or (not cls.__imgBiosObj) ): + cls.__instanciateSubClasses() + + cls.__imgEFIObj.do_configure_partition( + part, + source_params, + creator, + cr_workdir, + oe_builddir, + bootimg_dir, + kernel_dir, + native_sysroot) + + cls.__imgBiosObj.do_configure_partition( + part, + source_params, + creator, + cr_workdir, + oe_builddir, + bootimg_dir, + kernel_dir, + native_sysroot) + + @classmethod + def do_prepare_partition(cls, part, source_params, creator, cr_workdir, + oe_builddir, bootimg_dir, kernel_dir, + rootfs_dir, native_sysroot): + """ + Called to do the actual content population for a partition i.e. it + 'prepares' the partition to be incorporated into the image. + """ + + if ( (not cls.__imgEFIObj) or (not cls.__imgBiosObj) ): + cls.__instanciateSubClasses() + + cls.__imgEFIObj.do_prepare_partition( + part, + source_params, + creator, + cr_workdir, + oe_builddir, + bootimg_dir, + kernel_dir, + rootfs_dir, + native_sysroot) + + cls.__imgBiosObj.do_prepare_partition( + part, + source_params, + creator, + cr_workdir, + oe_builddir, + bootimg_dir, + kernel_dir, + rootfs_dir, + native_sysroot) diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py index 9879cb9..2cfdc10 100644 --- a/scripts/lib/wic/plugins/source/bootimg-efi.py +++ b/scripts/lib/wic/plugins/source/bootimg-efi.py @@ -1,21 +1,7 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Copyright (c) 2014, Intel Corporation. -# All rights reserved. # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # DESCRIPTION # This implements the 'bootimg-efi' source plugin class for 'wic' @@ -31,8 +17,8 @@ import shutil from wic import WicError from wic.engine import get_custom_config from wic.pluginbase import SourcePlugin -from wic.utils.misc import (exec_cmd, exec_native_cmd, get_bitbake_var, - BOOTDD_EXTRA_SPACE) +from wic.misc import (exec_cmd, exec_native_cmd, + get_bitbake_var, BOOTDD_EXTRA_SPACE) logger = logging.getLogger('wic') @@ -45,7 +31,7 @@ class BootimgEFIPlugin(SourcePlugin): name = 'bootimg-efi' @classmethod - def do_configure_grubefi(cls, creator, cr_workdir): + def do_configure_grubefi(cls, hdddir, creator, cr_workdir, source_params): """ Create loader-specific (grub-efi) config """ @@ -62,20 +48,52 @@ class BootimgEFIPlugin(SourcePlugin): raise WicError("configfile is specified but failed to " "get it from %s." % configfile) + initrd = source_params.get('initrd') + + if initrd: + bootimg_dir = get_bitbake_var("DEPLOY_DIR_IMAGE") + if not bootimg_dir: + raise WicError("Couldn't find DEPLOY_DIR_IMAGE, exiting") + + initrds = initrd.split(';') + for rd in initrds: + cp_cmd = "cp %s/%s %s" % (bootimg_dir, rd, hdddir) + exec_cmd(cp_cmd, True) + else: + logger.debug("Ignoring missing initrd") + if not custom_cfg: # Create grub configuration using parameters from wks file bootloader = creator.ks.bootloader + title = source_params.get('title') grubefi_conf = "" grubefi_conf += "serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1\n" grubefi_conf += "default=boot\n" grubefi_conf += "timeout=%s\n" % bootloader.timeout - grubefi_conf += "menuentry 'boot'{\n" + grubefi_conf += "menuentry '%s'{\n" % (title if title else "boot") + + kernel = get_bitbake_var("KERNEL_IMAGETYPE") + if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1": + if get_bitbake_var("INITRAMFS_IMAGE"): + kernel = "%s-%s.bin" % \ + (get_bitbake_var("KERNEL_IMAGETYPE"), get_bitbake_var("INITRAMFS_LINK_NAME")) + + label = source_params.get('label') + label_conf = "root=%s" % creator.rootdev + if label: + label_conf = "LABEL=%s" % label - kernel = "/bzImage" + grubefi_conf += "linux /%s %s rootwait %s\n" \ + % (kernel, label_conf, bootloader.append) + + if initrd: + initrds = initrd.split(';') + grubefi_conf += "initrd" + for rd in initrds: + grubefi_conf += " /%s" % rd + grubefi_conf += "\n" - grubefi_conf += "linux %s root=%s rootwait %s\n" \ - % (kernel, creator.rootdev, bootloader.append) grubefi_conf += "}\n" logger.debug("Writing grubefi config %s/hdd/boot/EFI/BOOT/grub.cfg", @@ -109,8 +127,10 @@ class BootimgEFIPlugin(SourcePlugin): if not bootimg_dir: raise WicError("Couldn't find DEPLOY_DIR_IMAGE, exiting") - cp_cmd = "cp %s/%s %s" % (bootimg_dir, initrd, hdddir) - exec_cmd(cp_cmd, True) + initrds = initrd.split(';') + for rd in initrds: + cp_cmd = "cp %s/%s %s" % (bootimg_dir, rd, hdddir) + exec_cmd(cp_cmd, True) else: logger.debug("Ignoring missing initrd") @@ -135,16 +155,30 @@ class BootimgEFIPlugin(SourcePlugin): if not custom_cfg: # Create systemd-boot configuration using parameters from wks file - kernel = "/bzImage" + kernel = get_bitbake_var("KERNEL_IMAGETYPE") + if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1": + if get_bitbake_var("INITRAMFS_IMAGE"): + kernel = "%s-%s.bin" % \ + (get_bitbake_var("KERNEL_IMAGETYPE"), get_bitbake_var("INITRAMFS_LINK_NAME")) + + title = source_params.get('title') boot_conf = "" - boot_conf += "title boot\n" - boot_conf += "linux %s\n" % kernel - boot_conf += "options LABEL=Boot root=%s %s\n" % \ - (creator.rootdev, bootloader.append) + boot_conf += "title %s\n" % (title if title else "boot") + boot_conf += "linux /%s\n" % kernel + + label = source_params.get('label') + label_conf = "LABEL=Boot root=%s" % creator.rootdev + if label: + label_conf = "LABEL=%s" % label + + boot_conf += "options %s %s\n" % \ + (label_conf, bootloader.append) if initrd: - boot_conf += "initrd /%s\n" % initrd + initrds = initrd.split(';') + for rd in initrds: + boot_conf += "initrd /%s\n" % rd logger.debug("Writing systemd-boot config " "%s/hdd/boot/loader/entries/boot.conf", cr_workdir) @@ -167,7 +201,7 @@ class BootimgEFIPlugin(SourcePlugin): try: if source_params['loader'] == 'grub-efi': - cls.do_configure_grubefi(creator, cr_workdir) + cls.do_configure_grubefi(hdddir, creator, cr_workdir, source_params) elif source_params['loader'] == 'systemd-boot': cls.do_configure_systemdboot(hdddir, creator, cr_workdir, source_params) else: @@ -194,8 +228,14 @@ class BootimgEFIPlugin(SourcePlugin): hdddir = "%s/hdd/boot" % cr_workdir - install_cmd = "install -m 0644 %s/bzImage %s/bzImage" % \ - (staging_kernel_dir, hdddir) + kernel = get_bitbake_var("KERNEL_IMAGETYPE") + if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1": + if get_bitbake_var("INITRAMFS_IMAGE"): + kernel = "%s-%s.bin" % \ + (get_bitbake_var("KERNEL_IMAGETYPE"), get_bitbake_var("INITRAMFS_LINK_NAME")) + + install_cmd = "install -m 0644 %s/%s %s/%s" % \ + (staging_kernel_dir, kernel, hdddir, kernel) exec_cmd(install_cmd) @@ -240,7 +280,10 @@ class BootimgEFIPlugin(SourcePlugin): # dosfs image, created by mkdosfs bootimg = "%s/boot.img" % cr_workdir - dosfs_cmd = "mkdosfs -n efi -C %s %d" % (bootimg, blocks) + label = part.label if part.label else "ESP" + + dosfs_cmd = "mkdosfs -n %s -i %s -C %s %d" % \ + (label, part.fsuuid, bootimg, blocks) exec_native_cmd(dosfs_cmd, native_sysroot) mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir) diff --git a/scripts/lib/wic/plugins/source/bootimg-partition.py b/scripts/lib/wic/plugins/source/bootimg-partition.py index 13fddbd..138986a 100644 --- a/scripts/lib/wic/plugins/source/bootimg-partition.py +++ b/scripts/lib/wic/plugins/source/bootimg-partition.py @@ -1,18 +1,5 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # DESCRIPTION # This implements the 'bootimg-partition' source plugin class for @@ -30,8 +17,9 @@ import re from glob import glob from wic import WicError +from wic.engine import get_custom_config from wic.pluginbase import SourcePlugin -from wic.utils.misc import exec_cmd, get_bitbake_var +from wic.misc import exec_cmd, get_bitbake_var logger = logging.getLogger('wic') @@ -44,17 +32,13 @@ class BootimgPartitionPlugin(SourcePlugin): name = 'bootimg-partition' @classmethod - def do_prepare_partition(cls, part, source_params, cr, cr_workdir, + def do_configure_partition(cls, part, source_params, cr, cr_workdir, oe_builddir, bootimg_dir, kernel_dir, - rootfs_dir, native_sysroot): + native_sysroot): """ - Called to do the actual content population for a partition i.e. it - 'prepares' the partition to be incorporated into the image. - In this case, does the following: - - sets up a vfat partition - - copies all files listed in IMAGE_BOOT_FILES variable + Called before do_prepare_partition(), create u-boot specific boot config """ - hdddir = "%s/boot" % cr_workdir + hdddir = "%s/boot.%d" % (cr_workdir, part.lineno) install_cmd = "install -d %s" % hdddir exec_cmd(install_cmd) @@ -63,12 +47,19 @@ class BootimgPartitionPlugin(SourcePlugin): if not kernel_dir: raise WicError("Couldn't find DEPLOY_DIR_IMAGE, exiting") - logger.debug('Kernel dir: %s', bootimg_dir) + boot_files = None + for (fmt, id) in (("_uuid-%s", part.uuid), ("_label-%s", part.label), (None, None)): + if fmt: + var = fmt % id + else: + var = "" - boot_files = get_bitbake_var("IMAGE_BOOT_FILES") + boot_files = get_bitbake_var("IMAGE_BOOT_FILES" + var) + if boot_files is not None: + break - if not boot_files: - raise WicError('No boot files defined, IMAGE_BOOT_FILES unset') + if boot_files is None: + raise WicError('No boot files defined, IMAGE_BOOT_FILES unset for entry #%d' % part.lineno) logger.debug('Boot files: %s', boot_files) @@ -85,9 +76,9 @@ class BootimgPartitionPlugin(SourcePlugin): logger.debug('Destination entry: %r', dst_entry) deploy_files.append(dst_entry) + cls.install_task = []; for deploy_entry in deploy_files: src, dst = deploy_entry - install_task = [] if '*' in src: # by default install files under their basename entry_name_fn = os.path.basename @@ -102,22 +93,102 @@ class BootimgPartitionPlugin(SourcePlugin): logger.debug('Globbed sources: %s', ', '.join(srcs)) for entry in srcs: + src = os.path.relpath(entry, kernel_dir) entry_dst_name = entry_name_fn(entry) - install_task.append((entry, - os.path.join(hdddir, - entry_dst_name))) + cls.install_task.append((src, entry_dst_name)) else: - install_task = [(os.path.join(kernel_dir, src), - os.path.join(hdddir, dst))] + cls.install_task.append((src, dst)) + + if source_params.get('loader') != "u-boot": + return + + configfile = cr.ks.bootloader.configfile + custom_cfg = None + if configfile: + custom_cfg = get_custom_config(configfile) + if custom_cfg: + # Use a custom configuration for extlinux.conf + extlinux_conf = custom_cfg + logger.debug("Using custom configuration file " + "%s for extlinux.cfg", configfile) + else: + raise WicError("configfile is specified but failed to " + "get it from %s." % configfile) + + if not custom_cfg: + # The kernel types supported by the sysboot of u-boot + kernel_types = ["zImage", "Image", "fitImage", "uImage", "vmlinux"] + has_dtb = False + fdt_dir = '/' + kernel_name = None + + # Find the kernel image name, from the highest precedence to lowest + for image in kernel_types: + for task in cls.install_task: + src, dst = task + if re.match(image, src): + kernel_name = os.path.join('/', dst) + break + if kernel_name: + break + + for task in cls.install_task: + src, dst = task + # We suppose that all the dtb are in the same directory + if re.search(r'\.dtb', src) and fdt_dir == '/': + has_dtb = True + fdt_dir = os.path.join(fdt_dir, os.path.dirname(dst)) + break + + if not kernel_name: + raise WicError('No kernel file founded') + + # Compose the extlinux.conf + extlinux_conf = "default Yocto\n" + extlinux_conf += "label Yocto\n" + extlinux_conf += " kernel %s\n" % kernel_name + if has_dtb: + extlinux_conf += " fdtdir %s\n" % fdt_dir + bootloader = cr.ks.bootloader + extlinux_conf += "append root=%s rootwait %s\n" \ + % (cr.rootdev, bootloader.append if bootloader.append else '') + + install_cmd = "install -d %s/extlinux/" % hdddir + exec_cmd(install_cmd) + cfg = open("%s/extlinux/extlinux.conf" % hdddir, "w") + cfg.write(extlinux_conf) + cfg.close() + + + @classmethod + def do_prepare_partition(cls, part, source_params, cr, cr_workdir, + oe_builddir, bootimg_dir, kernel_dir, + rootfs_dir, native_sysroot): + """ + Called to do the actual content population for a partition i.e. it + 'prepares' the partition to be incorporated into the image. + In this case, does the following: + - sets up a vfat partition + - copies all files listed in IMAGE_BOOT_FILES variable + """ + hdddir = "%s/boot.%d" % (cr_workdir, part.lineno) + + if not kernel_dir: + kernel_dir = get_bitbake_var("DEPLOY_DIR_IMAGE") + if not kernel_dir: + raise WicError("Couldn't find DEPLOY_DIR_IMAGE, exiting") + + logger.debug('Kernel dir: %s', bootimg_dir) + - for task in install_task: - src_path, dst_path = task - logger.debug('Install %s as %s', - os.path.basename(src_path), dst_path) - install_cmd = "install -m 0644 -D %s %s" \ - % (src_path, dst_path) - exec_cmd(install_cmd) + for task in cls.install_task: + src_path, dst_path = task + logger.debug('Install %s as %s', src_path, dst_path) + install_cmd = "install -m 0644 -D %s %s" \ + % (os.path.join(kernel_dir, src_path), + os.path.join(hdddir, dst_path)) + exec_cmd(install_cmd) logger.debug('Prepare boot partition using rootfs in %s', hdddir) part.prepare_rootfs(cr_workdir, oe_builddir, hdddir, - native_sysroot) + native_sysroot, False) diff --git a/scripts/lib/wic/plugins/source/bootimg-pcbios.py b/scripts/lib/wic/plugins/source/bootimg-pcbios.py index 11db304..f2639e7 100644 --- a/scripts/lib/wic/plugins/source/bootimg-pcbios.py +++ b/scripts/lib/wic/plugins/source/bootimg-pcbios.py @@ -1,21 +1,7 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Copyright (c) 2014, Intel Corporation. -# All rights reserved. # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # DESCRIPTION # This implements the 'bootimg-pcbios' source plugin class for 'wic' @@ -26,13 +12,13 @@ import logging import os +import re from wic import WicError from wic.engine import get_custom_config -from wic.utils import runner from wic.pluginbase import SourcePlugin -from wic.utils.misc import (exec_cmd, exec_native_cmd, - get_bitbake_var, BOOTDD_EXTRA_SPACE) +from wic.misc import (exec_cmd, exec_native_cmd, + get_bitbake_var, BOOTDD_EXTRA_SPACE) logger = logging.getLogger('wic') @@ -44,19 +30,22 @@ class BootimgPcbiosPlugin(SourcePlugin): name = 'bootimg-pcbios' @classmethod - def _get_syslinux_dir(cls, bootimg_dir): + def _get_bootimg_dir(cls, bootimg_dir, dirname): """ - Get path to syslinux from either default bootimg_dir - or wic-tools STAGING_DIR. + Check if dirname exists in default bootimg_dir or in STAGING_DIR. """ - for path in (bootimg_dir, get_bitbake_var("STAGING_DATADIR", "wic-tools")): - if not path: - continue - syslinux_dir = os.path.join(path, 'syslinux') - if os.path.exists(syslinux_dir): - return syslinux_dir + staging_datadir = get_bitbake_var("STAGING_DATADIR") + for result in (bootimg_dir, staging_datadir): + if os.path.exists("%s/%s" % (result, dirname)): + return result + + # STAGING_DATADIR is expanded with MLPREFIX if multilib is enabled + # but dependency syslinux is still populated to original STAGING_DATADIR + nonarch_datadir = re.sub('/[^/]*recipe-sysroot', '/recipe-sysroot', staging_datadir) + if os.path.exists(os.path.join(nonarch_datadir, dirname)): + return nonarch_datadir - raise WicError("Couldn't find syslinux directory, exiting") + raise WicError("Couldn't find correct bootimg_dir, exiting") @classmethod def do_install_disk(cls, disk, disk_name, creator, workdir, oe_builddir, @@ -65,11 +54,12 @@ class BootimgPcbiosPlugin(SourcePlugin): Called after all partitions have been prepared and assembled into a disk image. In this case, we install the MBR. """ - syslinux_dir = cls._get_syslinux_dir(bootimg_dir) + bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux') + mbrfile = "%s/syslinux/" % bootimg_dir if creator.ptable_format == 'msdos': - mbrfile = os.path.join(syslinux_dir, "mbr.bin") + mbrfile += "mbr.bin" elif creator.ptable_format == 'gpt': - mbrfile = os.path.join(syslinux_dir, "gptmbr.bin") + mbrfile += "gptmbr.bin" else: raise WicError("Unsupported partition table: %s" % creator.ptable_format) @@ -83,10 +73,8 @@ class BootimgPcbiosPlugin(SourcePlugin): logger.debug("Installing MBR on disk %s as %s with size %s bytes", disk_name, full_path, disk.min_size) - rcode = runner.show(['dd', 'if=%s' % mbrfile, - 'of=%s' % full_path, 'conv=notrunc']) - if rcode != 0: - raise WicError("Unable to set MBR to %s" % full_path) + dd_cmd = "dd if=%s of=%s conv=notrunc" % (mbrfile, full_path) + exec_cmd(dd_cmd, native_sysroot) @classmethod def do_configure_partition(cls, part, source_params, creator, cr_workdir, @@ -155,22 +143,28 @@ class BootimgPcbiosPlugin(SourcePlugin): 'prepares' the partition to be incorporated into the image. In this case, prepare content for legacy bios boot partition. """ - syslinux_dir = cls._get_syslinux_dir(bootimg_dir) + bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux') staging_kernel_dir = kernel_dir hdddir = "%s/hdd/boot" % cr_workdir - cmds = ("install -m 0644 %s/bzImage %s/vmlinuz" % - (staging_kernel_dir, hdddir), - "install -m 444 %s/ldlinux.sys %s/ldlinux.sys" % - (syslinux_dir, hdddir), - "install -m 0644 %s/vesamenu.c32 %s/vesamenu.c32" % - (syslinux_dir, hdddir), - "install -m 444 %s/libcom32.c32 %s/libcom32.c32" % - (syslinux_dir, hdddir), - "install -m 444 %s/libutil.c32 %s/libutil.c32" % - (syslinux_dir, hdddir)) + kernel = get_bitbake_var("KERNEL_IMAGETYPE") + if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1": + if get_bitbake_var("INITRAMFS_IMAGE"): + kernel = "%s-%s.bin" % \ + (get_bitbake_var("KERNEL_IMAGETYPE"), get_bitbake_var("INITRAMFS_LINK_NAME")) + + cmds = ("install -m 0644 %s/%s %s/vmlinuz" % + (staging_kernel_dir, kernel, hdddir), + "install -m 444 %s/syslinux/ldlinux.sys %s/ldlinux.sys" % + (bootimg_dir, hdddir), + "install -m 0644 %s/syslinux/vesamenu.c32 %s/vesamenu.c32" % + (bootimg_dir, hdddir), + "install -m 444 %s/syslinux/libcom32.c32 %s/libcom32.c32" % + (bootimg_dir, hdddir), + "install -m 444 %s/syslinux/libutil.c32 %s/libutil.c32" % + (bootimg_dir, hdddir)) for install_cmd in cmds: exec_cmd(install_cmd) @@ -190,9 +184,10 @@ class BootimgPcbiosPlugin(SourcePlugin): extra_blocks, part.mountpoint, blocks) # dosfs image, created by mkdosfs - bootimg = "%s/boot.img" % cr_workdir + bootimg = "%s/boot%s.img" % (cr_workdir, part.lineno) - dosfs_cmd = "mkdosfs -n boot -S 512 -C %s %d" % (bootimg, blocks) + dosfs_cmd = "mkdosfs -n boot -i %s -S 512 -C %s %d" % \ + (part.fsuuid, bootimg, blocks) exec_native_cmd(dosfs_cmd, native_sysroot) mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir) diff --git a/scripts/lib/wic/plugins/source/fsimage.py b/scripts/lib/wic/plugins/source/fsimage.py deleted file mode 100644 index f781499..0000000 --- a/scripts/lib/wic/plugins/source/fsimage.py +++ /dev/null @@ -1,56 +0,0 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. -# - -import logging -import os - -from wic import WicError -from wic.pluginbase import SourcePlugin -from wic.utils.misc import get_bitbake_var - -logger = logging.getLogger('wic') - -class FSImagePlugin(SourcePlugin): - """ - Add an already existing filesystem image to the partition layout. - """ - - name = 'fsimage' - - @classmethod - def do_prepare_partition(cls, part, source_params, cr, cr_workdir, - oe_builddir, bootimg_dir, kernel_dir, - rootfs_dir, native_sysroot): - """ - Called to do the actual content population for a partition i.e. it - 'prepares' the partition to be incorporated into the image. - """ - if not bootimg_dir: - bootimg_dir = get_bitbake_var("DEPLOY_DIR_IMAGE") - if not bootimg_dir: - raise WicError("Couldn't find DEPLOY_DIR_IMAGE, exiting") - - logger.debug('Bootimg dir: %s', bootimg_dir) - - if 'file' not in source_params: - raise WicError("No file specified") - - src = os.path.join(bootimg_dir, source_params['file']) - - - logger.debug('Preparing partition using image %s', src) - part.prepare_rootfs_from_fs_image(cr_workdir, src, "") diff --git a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py index 1ceba62..11326a2 100644 --- a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py +++ b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py @@ -1,18 +1,5 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- - -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. # -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # DESCRIPTION # This implements the 'isoimage-isohybrid' source plugin class for 'wic' @@ -29,7 +16,7 @@ import shutil from wic import WicError from wic.engine import get_custom_config from wic.pluginbase import SourcePlugin -from wic.utils.misc import exec_cmd, exec_native_cmd, get_bitbake_var +from wic.misc import exec_cmd, exec_native_cmd, get_bitbake_var logger = logging.getLogger('wic') @@ -47,7 +34,7 @@ class IsoImagePlugin(SourcePlugin): Example kickstart file: part /boot --source isoimage-isohybrid --sourceparams="loader=grub-efi, \\ - image_name= IsoImage" --ondisk cd --label LIVECD --fstype=ext2 + image_name= IsoImage" --ondisk cd --label LIVECD bootloader --timeout=10 --append=" " In --sourceparams "loader" specifies the bootloader used for booting in EFI @@ -83,8 +70,13 @@ class IsoImagePlugin(SourcePlugin): syslinux_conf += "DEFAULT boot\n" syslinux_conf += "LABEL boot\n" - kernel = "/bzImage" - syslinux_conf += "KERNEL " + kernel + "\n" + kernel = get_bitbake_var("KERNEL_IMAGETYPE") + if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1": + if get_bitbake_var("INITRAMFS_IMAGE"): + kernel = "%s-%s.bin" % \ + (get_bitbake_var("KERNEL_IMAGETYPE"), get_bitbake_var("INITRAMFS_LINK_NAME")) + + syslinux_conf += "KERNEL /" + kernel + "\n" syslinux_conf += "APPEND initrd=/initrd LABEL=boot %s\n" \ % bootloader.append @@ -95,7 +87,7 @@ class IsoImagePlugin(SourcePlugin): cfg.write(syslinux_conf) @classmethod - def do_configure_grubefi(cls, part, creator, cr_workdir): + def do_configure_grubefi(cls, part, creator, target_dir): """ Create loader-specific (grub-efi) config """ @@ -109,7 +101,7 @@ class IsoImagePlugin(SourcePlugin): raise WicError("configfile is specified " "but failed to get it from %s", configfile) else: - splash = os.path.join(cr_workdir, "EFI/boot/splash.jpg") + splash = os.path.join(target_dir, "splash.jpg") if os.path.exists(splash): splashline = "menu background splash.jpg" else: @@ -127,9 +119,13 @@ class IsoImagePlugin(SourcePlugin): grubefi_conf += "\n" grubefi_conf += "menuentry 'boot'{\n" - kernel = "/bzImage" + kernel = get_bitbake_var("KERNEL_IMAGETYPE") + if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1": + if get_bitbake_var("INITRAMFS_IMAGE"): + kernel = "%s-%s.bin" % \ + (get_bitbake_var("KERNEL_IMAGETYPE"), get_bitbake_var("INITRAMFS_LINK_NAME")) - grubefi_conf += "linux %s rootwait %s\n" \ + grubefi_conf += "linux /%s rootwait %s\n" \ % (kernel, bootloader.append) grubefi_conf += "initrd /initrd \n" grubefi_conf += "}\n" @@ -137,9 +133,10 @@ class IsoImagePlugin(SourcePlugin): if splashline: grubefi_conf += "%s\n" % splashline - logger.debug("Writing grubefi config %s/EFI/BOOT/grub.cfg", cr_workdir) + cfg_path = os.path.join(target_dir, "grub.cfg") + logger.debug("Writing grubefi config %s", cfg_path) - with open("%s/EFI/BOOT/grub.cfg" % cr_workdir, "w") as cfg: + with open(cfg_path, "w") as cfg: cfg.write(grubefi_conf) @staticmethod @@ -162,13 +159,14 @@ class IsoImagePlugin(SourcePlugin): if not image_type: raise WicError("Couldn't find INITRAMFS_FSTYPES, exiting.") - target_arch = get_bitbake_var("TRANSLATED_TARGET_ARCH") - if not target_arch: - raise WicError("Couldn't find TRANSLATED_TARGET_ARCH, exiting.") + machine = os.path.basename(initrd_dir) - initrd = glob.glob('%s/%s*%s.%s' % (initrd_dir, image_name, target_arch, image_type))[0] + pattern = '%s/%s*%s.%s' % (initrd_dir, image_name, machine, image_type) + files = glob.glob(pattern) + if files: + initrd = files[0] - if not os.path.exists(initrd): + if not initrd or not os.path.exists(initrd): # Create initrd from rootfs directory initrd = "%s/initrd.cpio.gz" % cr_workdir initrd_dir = "%s/INITRD" % cr_workdir @@ -189,10 +187,9 @@ class IsoImagePlugin(SourcePlugin): else: raise WicError("Couldn't find or build initrd, exiting.") - exec_cmd("cd %s && find . | cpio -o -H newc -R +0:+0 >./initrd.cpio " \ - % initrd_dir, as_shell=True) - exec_cmd("gzip -f -9 -c %s/initrd.cpio > %s" \ - % (cr_workdir, initrd), as_shell=True) + exec_cmd("cd %s && find . | cpio -o -H newc -R root:root >%s/initrd.cpio " \ + % (initrd_dir, cr_workdir), as_shell=True) + exec_cmd("gzip -f -9 %s/initrd.cpio" % cr_workdir, as_shell=True) shutil.rmtree(initrd_dir) return initrd @@ -206,8 +203,8 @@ class IsoImagePlugin(SourcePlugin): """ isodir = "%s/ISO/" % cr_workdir - if os.path.exists(cr_workdir): - shutil.rmtree(cr_workdir) + if os.path.exists(isodir): + shutil.rmtree(isodir) install_cmd = "install -d %s " % isodir exec_cmd(install_cmd) @@ -251,33 +248,8 @@ class IsoImagePlugin(SourcePlugin): raise WicError("Couldn't find IMAGE_ROOTFS, exiting.") part.rootfs_dir = rootfs_dir - - # Prepare rootfs.img deploy_dir = get_bitbake_var("DEPLOY_DIR_IMAGE") img_iso_dir = get_bitbake_var("ISODIR") - rootfs_img = "%s/rootfs.img" % img_iso_dir - if not os.path.isfile(rootfs_img): - # check if rootfs.img is in deploydir - deploy_dir = get_bitbake_var("DEPLOY_DIR_IMAGE") - image_name = get_bitbake_var("IMAGE_LINK_NAME") - rootfs_img = "%s/%s.%s" \ - % (deploy_dir, image_name, part.fstype) - - if not os.path.isfile(rootfs_img): - # create image file with type specified by --fstype - # which contains rootfs - du_cmd = "du -bks %s" % rootfs_dir - out = exec_cmd(du_cmd) - part.size = int(out.split()[0]) - part.extra_space = 0 - part.overhead_factor = 1.2 - part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, \ - native_sysroot) - rootfs_img = part.source_file - - install_cmd = "install -m 0644 %s %s/rootfs.img" \ - % (rootfs_img, isodir) - exec_cmd(install_cmd) # Remove the temporary file created by part.prepare_rootfs() if os.path.isfile(part.source_file): @@ -305,27 +277,25 @@ class IsoImagePlugin(SourcePlugin): if os.path.isfile("%s/initrd.cpio.gz" % cr_workdir): os.remove("%s/initrd.cpio.gz" % cr_workdir) - # Install bzImage - install_cmd = "install -m 0644 %s/bzImage %s/bzImage" % \ - (kernel_dir, isodir) + kernel = get_bitbake_var("KERNEL_IMAGETYPE") + if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1": + if get_bitbake_var("INITRAMFS_IMAGE"): + kernel = "%s-%s.bin" % \ + (get_bitbake_var("KERNEL_IMAGETYPE"), get_bitbake_var("INITRAMFS_LINK_NAME")) + + install_cmd = "install -m 0644 %s/%s %s/%s" % \ + (kernel_dir, kernel, isodir, kernel) exec_cmd(install_cmd) #Create bootloader for efi boot try: - if source_params['loader'] == 'grub-efi': - # Builds grub.cfg if ISODIR didn't exist or - # didn't contains grub.cfg - bootimg_dir = img_iso_dir - if not os.path.exists("%s/EFI/BOOT" % bootimg_dir): - bootimg_dir = "%s/bootimg" % cr_workdir - if os.path.exists(bootimg_dir): - shutil.rmtree(bootimg_dir) - install_cmd = "install -d %s/EFI/BOOT" % bootimg_dir - exec_cmd(install_cmd) - - if not os.path.isfile("%s/EFI/BOOT/boot.cfg" % bootimg_dir): - cls.do_configure_grubefi(part, creator, bootimg_dir) + target_dir = "%s/EFI/BOOT" % isodir + if os.path.exists(target_dir): + shutil.rmtree(target_dir) + os.makedirs(target_dir) + + if source_params['loader'] == 'grub-efi': # Builds bootx64.efi/bootia32.efi if ISODIR didn't exist or # didn't contains it target_arch = get_bitbake_var("TARGET_SYS") @@ -333,37 +303,25 @@ class IsoImagePlugin(SourcePlugin): raise WicError("Coludn't find target architecture") if re.match("x86_64", target_arch): - grub_target = 'x86_64-efi' - grub_image = "bootx64.efi" + grub_src_image = "grub-efi-bootx64.efi" + grub_dest_image = "bootx64.efi" elif re.match('i.86', target_arch): - grub_target = 'i386-efi' - grub_image = "bootia32.efi" + grub_src_image = "grub-efi-bootia32.efi" + grub_dest_image = "bootia32.efi" else: raise WicError("grub-efi is incompatible with target %s" % target_arch) - if not os.path.isfile("%s/EFI/BOOT/%s" \ - % (bootimg_dir, grub_image)): - grub_path = get_bitbake_var("STAGING_LIBDIR", "wic-tools") - if not grub_path: - raise WicError("Couldn't find STAGING_LIBDIR, exiting.") - - grub_core = "%s/grub/%s" % (grub_path, grub_target) - if not os.path.exists(grub_core): - raise WicError("Please build grub-efi first") - - grub_cmd = "grub-mkimage -p '/EFI/BOOT' " - grub_cmd += "-d %s " % grub_core - grub_cmd += "-O %s -o %s/EFI/BOOT/%s " \ - % (grub_target, bootimg_dir, grub_image) - grub_cmd += "part_gpt part_msdos ntfs ntfscomp fat ext2 " - grub_cmd += "normal chain boot configfile linux multiboot " - grub_cmd += "search efi_gop efi_uga font gfxterm gfxmenu " - grub_cmd += "terminal minicmd test iorw loadenv echo help " - grub_cmd += "reboot serial terminfo iso9660 loopback tar " - grub_cmd += "memdisk ls search_fs_uuid udf btrfs xfs lvm " - grub_cmd += "reiserfs ata " - exec_native_cmd(grub_cmd, native_sysroot) + grub_target = os.path.join(target_dir, grub_dest_image) + if not os.path.isfile(grub_target): + grub_src = os.path.join(deploy_dir, grub_src_image) + if not os.path.exists(grub_src): + raise WicError("Grub loader %s is not found in %s. " + "Please build grub-efi first" % (grub_src_image, deploy_dir)) + shutil.copy(grub_src, grub_target) + + if not os.path.isfile(os.path.join(target_dir, "boot.cfg")): + cls.do_configure_grubefi(part, creator, target_dir) else: raise WicError("unrecognized bootimg-efi loader: %s" % @@ -371,15 +329,6 @@ class IsoImagePlugin(SourcePlugin): except KeyError: raise WicError("bootimg-efi requires a loader, none specified") - if os.path.exists("%s/EFI/BOOT" % isodir): - shutil.rmtree("%s/EFI/BOOT" % isodir) - - shutil.copytree(bootimg_dir+"/EFI/BOOT", isodir+"/EFI/BOOT") - - # If exists, remove cr_workdir/bootimg temporary folder - if os.path.exists("%s/bootimg" % cr_workdir): - shutil.rmtree("%s/bootimg" % cr_workdir) - # Create efi.img that contains bootloader files for EFI booting # if ISODIR didn't exist or didn't contains it if os.path.isfile("%s/efi.img" % img_iso_dir): @@ -387,19 +336,23 @@ class IsoImagePlugin(SourcePlugin): (img_iso_dir, isodir) exec_cmd(install_cmd) else: + # Default to 100 blocks of extra space for file system overhead + esp_extra_blocks = int(source_params.get('esp_extra_blocks', '100')) + du_cmd = "du -bks %s/EFI" % isodir out = exec_cmd(du_cmd) blocks = int(out.split()[0]) - # Add some extra space for file system overhead - blocks += 100 + blocks += esp_extra_blocks logger.debug("Added 100 extra blocks to %s to get to %d " "total blocks", part.mountpoint, blocks) # dosfs image for EFI boot bootimg = "%s/efi.img" % isodir - dosfs_cmd = 'mkfs.vfat -n "EFIimg" -S 512 -C %s %d' \ - % (bootimg, blocks) + esp_label = source_params.get('esp_label', 'EFIimg') + + dosfs_cmd = 'mkfs.vfat -n \'%s\' -S 512 -C %s %d' \ + % (esp_label, bootimg, blocks) exec_native_cmd(dosfs_cmd, native_sysroot) mmd_cmd = "mmd -i %s ::/EFI" % bootimg @@ -413,7 +366,7 @@ class IsoImagePlugin(SourcePlugin): exec_cmd(chmod_cmd) # Prepare files for legacy boot - syslinux_dir = get_bitbake_var("STAGING_DATADIR", "wic-tools") + syslinux_dir = get_bitbake_var("STAGING_DATADIR") if not syslinux_dir: raise WicError("Couldn't find STAGING_DATADIR, exiting.") diff --git a/scripts/lib/wic/plugins/source/rawcopy.py b/scripts/lib/wic/plugins/source/rawcopy.py index e1c4f5e..3c4997d 100644 --- a/scripts/lib/wic/plugins/source/rawcopy.py +++ b/scripts/lib/wic/plugins/source/rawcopy.py @@ -1,18 +1,5 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # import logging @@ -20,7 +7,7 @@ import os from wic import WicError from wic.pluginbase import SourcePlugin -from wic.utils.misc import exec_cmd, get_bitbake_var +from wic.misc import exec_cmd, get_bitbake_var from wic.filemap import sparse_copy logger = logging.getLogger('wic') @@ -32,6 +19,25 @@ class RawCopyPlugin(SourcePlugin): name = 'rawcopy' + @staticmethod + def do_image_label(fstype, dst, label): + if fstype.startswith('ext'): + cmd = 'tune2fs -L %s %s' % (label, dst) + elif fstype in ('msdos', 'vfat'): + cmd = 'dosfslabel %s %s' % (dst, label) + elif fstype == 'btrfs': + cmd = 'btrfs filesystem label %s %s' % (dst, label) + elif fstype == 'swap': + cmd = 'mkswap -L %s %s' % (label, dst) + elif fstype == 'squashfs': + raise WicError("It's not possible to update a squashfs " + "filesystem label '%s'" % (label)) + else: + raise WicError("Cannot update filesystem label: " + "Unknown fstype: '%s'" % (fstype)) + + exec_cmd(cmd) + @classmethod def do_prepare_partition(cls, part, source_params, cr, cr_workdir, oe_builddir, bootimg_dir, kernel_dir, @@ -51,7 +57,10 @@ class RawCopyPlugin(SourcePlugin): raise WicError("No file specified") src = os.path.join(kernel_dir, source_params['file']) - dst = os.path.join(cr_workdir, "%s.%s" % (source_params['file'], part.lineno)) + dst = os.path.join(cr_workdir, "%s.%s" % (os.path.basename(source_params['file']), part.lineno)) + + if not os.path.exists(os.path.dirname(dst)): + os.makedirs(os.path.dirname(dst)) if 'skip' in source_params: sparse_copy(src, dst, skip=int(source_params['skip'])) @@ -66,4 +75,7 @@ class RawCopyPlugin(SourcePlugin): if filesize > part.size: part.size = filesize + if part.label: + RawCopyPlugin.do_image_label(part.fstype, dst, part.label) + part.source_file = dst diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py index f2e2ca8..f1db83f 100644 --- a/scripts/lib/wic/plugins/source/rootfs.py +++ b/scripts/lib/wic/plugins/source/rootfs.py @@ -1,21 +1,7 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Copyright (c) 2014, Intel Corporation. -# All rights reserved. # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # DESCRIPTION # This implements the 'rootfs' source plugin class for 'wic' @@ -28,12 +14,14 @@ import logging import os import shutil +import sys from oe.path import copyhardlinktree +from pathlib import Path from wic import WicError from wic.pluginbase import SourcePlugin -from wic.utils.misc import get_bitbake_var, exec_cmd +from wic.misc import get_bitbake_var, exec_native_cmd logger = logging.getLogger('wic') @@ -44,10 +32,26 @@ class RootfsPlugin(SourcePlugin): name = 'rootfs' + @staticmethod + def __validate_path(cmd, rootfs_dir, path): + if os.path.isabs(path): + logger.error("%s: Must be relative: %s" % (cmd, orig_path)) + sys.exit(1) + + # Disallow climbing outside of parent directory using '..', + # because doing so could be quite disastrous (we will delete the + # directory, or modify a directory outside OpenEmbedded). + full_path = os.path.realpath(os.path.join(rootfs_dir, path)) + if not full_path.startswith(os.path.realpath(rootfs_dir)): + logger.error("%s: Must point inside the rootfs:" % (cmd, path)) + sys.exit(1) + + return full_path + @staticmethod def __get_rootfs_dir(rootfs_dir): if os.path.isdir(rootfs_dir): - return rootfs_dir + return os.path.realpath(rootfs_dir) image_rootfs_dir = get_bitbake_var("IMAGE_ROOTFS", rootfs_dir) if not os.path.isdir(image_rootfs_dir): @@ -55,7 +59,16 @@ class RootfsPlugin(SourcePlugin): "named %s has been found at %s, exiting." % (rootfs_dir, image_rootfs_dir)) - return image_rootfs_dir + return os.path.realpath(image_rootfs_dir) + + @staticmethod + def __get_pseudo(native_sysroot, rootfs, pseudo_dir): + pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot + pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir + pseudo += "export PSEUDO_PASSWD=%s;" % rootfs + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" + pseudo += "%s " % get_bitbake_var("FAKEROOTCMD") + return pseudo @classmethod def do_prepare_partition(cls, part, source_params, cr, cr_workdir, @@ -80,33 +93,111 @@ class RootfsPlugin(SourcePlugin): raise WicError("Couldn't find --rootfs-dir=%s connection or " "it is not a valid path, exiting" % part.rootfs_dir) - real_rootfs_dir = cls.__get_rootfs_dir(rootfs_dir) + part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir) + pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo") + if not os.path.lexists(pseudo_dir): + logger.warn("%s folder does not exist. " + "Usernames and permissions will be invalid " % pseudo_dir) + pseudo_dir = None + new_rootfs = None + new_pseudo = None # Handle excluded paths. - if part.exclude_path is not None: + if part.exclude_path or part.include_path or part.change_directory: # We need a new rootfs directory we can delete files from. Copy to # workdir. - new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs")) + new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno)) if os.path.lexists(new_rootfs): shutil.rmtree(os.path.join(new_rootfs)) - copyhardlinktree(real_rootfs_dir, new_rootfs) + if part.change_directory: + cd = part.change_directory + if cd[-1] == '/': + cd = cd[:-1] + orig_dir = cls.__validate_path("--change-directory", part.rootfs_dir, cd) + else: + orig_dir = part.rootfs_dir + copyhardlinktree(orig_dir, new_rootfs) + + # Convert the pseudo directory to its new location + if (pseudo_dir): + new_pseudo = os.path.realpath( + os.path.join(cr_workdir, "pseudo%d" % part.lineno)) + if os.path.lexists(new_pseudo): + shutil.rmtree(new_pseudo) + os.mkdir(new_pseudo) + shutil.copy(os.path.join(pseudo_dir, "files.db"), + os.path.join(new_pseudo, "files.db")) + + pseudo_cmd = "%s -B -m %s -M %s" % (cls.__get_pseudo(native_sysroot, + new_rootfs, + new_pseudo), + orig_dir, new_rootfs) + exec_native_cmd(pseudo_cmd, native_sysroot) + + for in_path in part.include_path or []: + #parse arguments + include_path = in_path[0] + if len(in_path) > 2: + logger.error("'Invalid number of arguments for include-path") + sys.exit(1) + if len(in_path) == 2: + path = in_path[1] + else: + path = None + + # Pack files to be included into a tar file. + # We need to create a tar file, because that way we can keep the + # permissions from the files even when they belong to different + # pseudo enviroments. + # If we simply copy files using copyhardlinktree/copytree... the + # copied files will belong to the user running wic. + tar_file = os.path.realpath( + os.path.join(cr_workdir, "include-path%d.tar" % part.lineno)) + if os.path.isfile(include_path): + parent = os.path.dirname(os.path.realpath(include_path)) + tar_cmd = "tar c --owner=root --group=root -f %s -C %s %s" % ( + tar_file, parent, os.path.relpath(include_path, parent)) + exec_native_cmd(tar_cmd, native_sysroot) + else: + if include_path in krootfs_dir: + include_path = krootfs_dir[include_path] + include_path = cls.__get_rootfs_dir(include_path) + include_pseudo = os.path.join(include_path, "../pseudo") + if os.path.lexists(include_pseudo): + pseudo = cls.__get_pseudo(native_sysroot, include_path, + include_pseudo) + tar_cmd = "tar cf %s -C %s ." % (tar_file, include_path) + else: + pseudo = None + tar_cmd = "tar c --owner=root --group=root -f %s -C %s ." % ( + tar_file, include_path) + exec_native_cmd(tar_cmd, native_sysroot, pseudo) + + #create destination + if path: + destination = cls.__validate_path("--include-path", new_rootfs, path) + Path(destination).mkdir(parents=True, exist_ok=True) + else: + destination = new_rootfs - real_rootfs_dir = new_rootfs + #extract destination + untar_cmd = "tar xf %s -C %s" % (tar_file, destination) + if new_pseudo: + pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo) + else: + pseudo = None + exec_native_cmd(untar_cmd, native_sysroot, pseudo) + os.remove(tar_file) - for orig_path in part.exclude_path: + for orig_path in part.exclude_path or []: path = orig_path - if os.path.isabs(path): - msger.error("Must be relative: --exclude-path=%s" % orig_path) - full_path = os.path.realpath(os.path.join(new_rootfs, path)) + full_path = cls.__validate_path("--exclude-path", new_rootfs, path) - # Disallow climbing outside of parent directory using '..', - # because doing so could be quite disastrous (we will delete the - # directory). - if not full_path.startswith(new_rootfs): - msger.error("'%s' points to a path outside the rootfs" % orig_path) + if not os.path.lexists(full_path): + continue if path.endswith(os.sep): # Delete content only. @@ -120,6 +211,6 @@ class RootfsPlugin(SourcePlugin): # Delete whole directory. shutil.rmtree(full_path) - part.rootfs_dir = real_rootfs_dir part.prepare_rootfs(cr_workdir, oe_builddir, - real_rootfs_dir, native_sysroot) + new_rootfs or part.rootfs_dir, native_sysroot, + pseudo_dir = new_pseudo or pseudo_dir) diff --git a/scripts/lib/wic/utils/__init__.py b/scripts/lib/wic/utils/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/scripts/lib/wic/utils/runner.py b/scripts/lib/wic/utils/runner.py deleted file mode 100644 index 56d7ea3..0000000 --- a/scripts/lib/wic/utils/runner.py +++ /dev/null @@ -1,114 +0,0 @@ -#!/usr/bin/env python -tt -# -# Copyright (c) 2011 Intel, Inc. -# -# This program is free software; you can redistribute it and/or modify it -# under the terms of the GNU General Public License as published by the Free -# Software Foundation; version 2 of the License -# -# This program is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY -# or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License -# for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., 59 -# Temple Place - Suite 330, Boston, MA 02111-1307, USA. - -import logging -import os -import subprocess - -from wic import WicError - -logger = logging.getLogger('wic') - -def runtool(cmdln_or_args, catch=1): - """ wrapper for most of the subprocess calls - input: - cmdln_or_args: can be both args and cmdln str (shell=True) - catch: 0, quitely run - 1, only STDOUT - 2, only STDERR - 3, both STDOUT and STDERR - return: - (rc, output) - if catch==0: the output will always None - """ - - if catch not in (0, 1, 2, 3): - # invalid catch selection, will cause exception, that's good - return None - - if isinstance(cmdln_or_args, list): - cmd = cmdln_or_args[0] - shell = False - else: - import shlex - cmd = shlex.split(cmdln_or_args)[0] - shell = True - - if catch != 3: - dev_null = os.open("/dev/null", os.O_WRONLY) - - if catch == 0: - sout = dev_null - serr = dev_null - elif catch == 1: - sout = subprocess.PIPE - serr = dev_null - elif catch == 2: - sout = dev_null - serr = subprocess.PIPE - elif catch == 3: - sout = subprocess.PIPE - serr = subprocess.STDOUT - - try: - process = subprocess.Popen(cmdln_or_args, stdout=sout, - stderr=serr, shell=shell) - (sout, serr) = process.communicate() - # combine stdout and stderr, filter None out and decode - out = ''.join([out.decode('utf-8') for out in [sout, serr] if out]) - except OSError as err: - if err.errno == 2: - # [Errno 2] No such file or directory - raise WicError('Cannot run command: %s, lost dependency?' % cmd) - else: - raise # relay - finally: - if catch != 3: - os.close(dev_null) - - return (process.returncode, out) - -def show(cmdln_or_args): - """Show all messages using logger.debug.""" - - rcode, out = runtool(cmdln_or_args, catch=3) - - if isinstance(cmdln_or_args, list): - cmd = ' '.join(cmdln_or_args) - else: - cmd = cmdln_or_args - - msg = 'running command: "%s"' % cmd - if out: - out = out.strip() - if out: - msg += ', with output::' - msg += '\n +----------------' - for line in out.splitlines(): - msg += '\n | %s' % line - msg += '\n +----------------' - - logger.debug(msg) - - return rcode - -def outs(cmdln_or_args, catch=1): - # get the outputs of tools - return runtool(cmdln_or_args, catch)[1].strip() - -def quiet(cmdln_or_args): - return runtool(cmdln_or_args, catch=0)[0] diff --git a/scripts/wic b/scripts/wic index a5f2dbf..24700f3 100755 --- a/scripts/wic +++ b/scripts/wic @@ -1,22 +1,8 @@ #!/usr/bin/env python3 -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Copyright (c) 2013, Intel Corporation. -# All rights reserved. # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # DESCRIPTION 'wic' is the OpenEmbedded Image Creator that users can # use to generate bootable images. Invoking it without any arguments @@ -33,28 +19,41 @@ __version__ = "0.2.0" # Python Standard Library modules import os import sys -import optparse +import argparse import logging +import subprocess + +from collections import namedtuple from distutils import spawn # External modules -scripts_path = os.path.abspath(os.path.dirname(__file__)) +scripts_path = os.path.dirname(os.path.realpath(__file__)) lib_path = scripts_path + '/lib' sys.path.insert(0, lib_path) -oe_lib_path = os.path.join(os.path.dirname(scripts_path), 'meta', 'lib') -sys.path.insert(0, oe_lib_path) +import scriptpath +scriptpath.add_oe_lib_path() + +# Check whether wic is running within eSDK environment +sdkroot = scripts_path +if os.environ.get('SDKTARGETSYSROOT'): + while sdkroot != '' and sdkroot != os.sep: + if os.path.exists(os.path.join(sdkroot, '.devtoolbase')): + # Set BUILDDIR for wic to work within eSDK + os.environ['BUILDDIR'] = sdkroot + # .devtoolbase only exists within eSDK + # If found, initialize bitbake path for eSDK environment and append to PATH + sdkroot = os.path.join(os.path.dirname(scripts_path), 'bitbake', 'bin') + os.environ['PATH'] += ":" + sdkroot + break + sdkroot = os.path.dirname(sdkroot) bitbake_exe = spawn.find_executable('bitbake') if bitbake_exe: - bitbake_path = os.path.join(os.path.dirname(bitbake_exe), '../lib') - sys.path.insert(0, bitbake_path) - from bb import cookerdata - from bb.main import bitbake_main, BitBakeConfigParameters -else: - bitbake_main = None + bitbake_path = scriptpath.add_bitbake_lib_path() + import bb from wic import WicError -from wic.utils.misc import get_bitbake_var, BB_VARS +from wic.misc import get_bitbake_var, BB_VARS from wic import engine from wic import help as hlp @@ -85,67 +84,31 @@ def rootfs_dir_to_args(krootfs_dir): rootfs_dir += '='.join([key, val]) return rootfs_dir.strip() -def callback_rootfs_dir(option, opt, value, parser): - """ - Build a dict using --rootfs_dir connection=dir - """ - if not type(parser.values.rootfs_dir) is dict: - parser.values.rootfs_dir = dict() - if '=' in value: - (key, rootfs_dir) = value.split('=') - else: - key = 'ROOTFS_DIR' - rootfs_dir = value +class RootfsArgAction(argparse.Action): + def __init__(self, **kwargs): + super().__init__(**kwargs) + + def __call__(self, parser, namespace, value, option_string=None): + if not "rootfs_dir" in vars(namespace) or \ + not type(namespace.__dict__['rootfs_dir']) is dict: + namespace.__dict__['rootfs_dir'] = {} - parser.values.rootfs_dir[key] = rootfs_dir + if '=' in value: + (key, rootfs_dir) = value.split('=') + else: + key = 'ROOTFS_DIR' + rootfs_dir = value -def wic_create_subcommand(args, usage_str): + namespace.__dict__['rootfs_dir'][key] = rootfs_dir + + +def wic_create_subcommand(options, usage_str): """ Command-line handling for image creation. The real work is done by image.engine.wic_create() """ - parser = optparse.OptionParser(usage=usage_str) - - parser.add_option("-o", "--outdir", dest="outdir", default='.', - help="name of directory to create image in") - parser.add_option("-e", "--image-name", dest="image_name", - help="name of the image to use the artifacts from " - "e.g. core-image-sato") - parser.add_option("-r", "--rootfs-dir", dest="rootfs_dir", type="string", - action="callback", callback=callback_rootfs_dir, - help="path to the /rootfs dir to use as the " - ".wks rootfs source") - parser.add_option("-b", "--bootimg-dir", dest="bootimg_dir", - help="path to the dir containing the boot artifacts " - "(e.g. /EFI or /syslinux dirs) to use as the " - ".wks bootimg source") - parser.add_option("-k", "--kernel-dir", dest="kernel_dir", - help="path to the dir containing the kernel to use " - "in the .wks bootimg") - parser.add_option("-n", "--native-sysroot", dest="native_sysroot", - help="path to the native sysroot containing the tools " - "to use to build the image") - parser.add_option("-s", "--skip-build-check", dest="build_check", - action="store_false", default=True, help="skip the build check") - parser.add_option("-f", "--build-rootfs", action="store_true", help="build rootfs") - parser.add_option("-c", "--compress-with", choices=("gzip", "bzip2", "xz"), - dest='compressor', - help="compress image with specified compressor") - parser.add_option("-m", "--bmap", action="store_true", help="generate .bmap") - parser.add_option("-v", "--vars", dest='vars_dir', - help="directory with <image>.env files that store " - "bitbake variables") - parser.add_option("-D", "--debug", dest="debug", action="store_true", - default=False, help="output debug information") - - (options, args) = parser.parse_args(args) - - if len(args) != 1: - parser.print_help() - raise WicError("Wrong number of arguments, exiting") - - if options.build_rootfs and not bitbake_main: + if options.build_rootfs and not bitbake_exe: raise WicError("Can't build rootfs as bitbake is not in the $PATH") if not options.image_name: @@ -181,39 +144,37 @@ def wic_create_subcommand(args, usage_str): argv.append("--debug") logger.info("Building rootfs...\n") - if bitbake_main(BitBakeConfigParameters(argv), - cookerdata.CookerConfiguration()): - raise WicError("bitbake exited with error") + subprocess.check_call(argv) rootfs_dir = get_bitbake_var("IMAGE_ROOTFS", options.image_name) kernel_dir = get_bitbake_var("DEPLOY_DIR_IMAGE", options.image_name) bootimg_dir = get_bitbake_var("STAGING_DATADIR", options.image_name) - native_sysroot = get_bitbake_var("RECIPE_SYSROOT_NATIVE", - options.image_name) #, cache=False) + + native_sysroot = options.native_sysroot + if options.vars_dir and not native_sysroot: + native_sysroot = get_bitbake_var("RECIPE_SYSROOT_NATIVE", options.image_name) else: if options.build_rootfs: raise WicError("Image name is not specified, exiting. " "(Use -e/--image-name to specify it)") native_sysroot = options.native_sysroot - if not native_sysroot or not os.path.isdir(native_sysroot): + if not options.vars_dir and (not native_sysroot or not os.path.isdir(native_sysroot)): logger.info("Building wic-tools...\n") - if bitbake_main(BitBakeConfigParameters("bitbake wic-tools".split()), - cookerdata.CookerConfiguration()): - raise WicError("bitbake wic-tools failed") + subprocess.check_call(["bitbake", "wic-tools"]) native_sysroot = get_bitbake_var("RECIPE_SYSROOT_NATIVE", "wic-tools") - if not native_sysroot: - raise WicError("Unable to find the location of the native " - "tools sysroot to use") - wks_file = args[0] + if not native_sysroot: + raise WicError("Unable to find the location of the native tools sysroot") + + wks_file = options.wks_file if not wks_file.endswith(".wks"): wks_file = engine.find_canned_image(scripts_path, wks_file) if not wks_file: raise WicError("No image named %s found, exiting. (Use 'wic list images' " "to list available images, or specify a fully-qualified OE " - "kickstart (.wks) filename)" % args[0]) + "kickstart (.wks) filename)" % options.wks_file) if not options.image_name: rootfs_dir = '' @@ -264,59 +225,312 @@ def wic_list_subcommand(args, usage_str): Command-line handling for listing available images. The real work is done by image.engine.wic_list() """ - parser = optparse.OptionParser(usage=usage_str) - args = parser.parse_args(args)[1] - if not engine.wic_list(args, scripts_path): - parser.print_help() raise WicError("Bad list arguments, exiting") -def wic_help_topic_subcommand(args, usage_str): +def wic_ls_subcommand(args, usage_str): + """ + Command-line handling for list content of images. + The real work is done by engine.wic_ls() + """ + engine.wic_ls(args, args.native_sysroot) + +def wic_cp_subcommand(args, usage_str): + """ + Command-line handling for copying files/dirs to images. + The real work is done by engine.wic_cp() + """ + engine.wic_cp(args, args.native_sysroot) + +def wic_rm_subcommand(args, usage_str): """ - Command-line handling for help-only 'subcommands'. This is - essentially a dummy command that doesn nothing but allow users to - use the existing subcommand infrastructure to display help on a - particular topic not attached to any particular subcommand. + Command-line handling for removing files/dirs from images. + The real work is done by engine.wic_rm() + """ + engine.wic_rm(args, args.native_sysroot) + +def wic_write_subcommand(args, usage_str): + """ + Command-line handling for writing images. + The real work is done by engine.wic_write() + """ + engine.wic_write(args, args.native_sysroot) + +def wic_help_subcommand(args, usage_str): + """ + Command-line handling for help subcommand to keep the current + structure of the function definitions. """ pass +def wic_help_topic_subcommand(usage_str, help_str): + """ + Display function for help 'sub-subcommands'. + """ + print(help_str) + return + + wic_help_topic_usage = """ """ -subcommands = { - "create": [wic_create_subcommand, - hlp.wic_create_usage, - hlp.wic_create_help], - "list": [wic_list_subcommand, - hlp.wic_list_usage, - hlp.wic_list_help], +helptopics = { "plugins": [wic_help_topic_subcommand, wic_help_topic_usage, - hlp.get_wic_plugins_help], + hlp.wic_plugins_help], "overview": [wic_help_topic_subcommand, wic_help_topic_usage, hlp.wic_overview_help], "kickstart": [wic_help_topic_subcommand, wic_help_topic_usage, hlp.wic_kickstart_help], + "create": [wic_help_topic_subcommand, + wic_help_topic_usage, + hlp.wic_create_help], + "ls": [wic_help_topic_subcommand, + wic_help_topic_usage, + hlp.wic_ls_help], + "cp": [wic_help_topic_subcommand, + wic_help_topic_usage, + hlp.wic_cp_help], + "rm": [wic_help_topic_subcommand, + wic_help_topic_usage, + hlp.wic_rm_help], + "write": [wic_help_topic_subcommand, + wic_help_topic_usage, + hlp.wic_write_help], + "list": [wic_help_topic_subcommand, + wic_help_topic_usage, + hlp.wic_list_help] } +def wic_init_parser_create(subparser): + subparser.add_argument("wks_file") + + subparser.add_argument("-o", "--outdir", dest="outdir", default='.', + help="name of directory to create image in") + subparser.add_argument("-e", "--image-name", dest="image_name", + help="name of the image to use the artifacts from " + "e.g. core-image-sato") + subparser.add_argument("-r", "--rootfs-dir", action=RootfsArgAction, + help="path to the /rootfs dir to use as the " + ".wks rootfs source") + subparser.add_argument("-b", "--bootimg-dir", dest="bootimg_dir", + help="path to the dir containing the boot artifacts " + "(e.g. /EFI or /syslinux dirs) to use as the " + ".wks bootimg source") + subparser.add_argument("-k", "--kernel-dir", dest="kernel_dir", + help="path to the dir containing the kernel to use " + "in the .wks bootimg") + subparser.add_argument("-n", "--native-sysroot", dest="native_sysroot", + help="path to the native sysroot containing the tools " + "to use to build the image") + subparser.add_argument("-s", "--skip-build-check", dest="build_check", + action="store_false", default=True, help="skip the build check") + subparser.add_argument("-f", "--build-rootfs", action="store_true", help="build rootfs") + subparser.add_argument("-c", "--compress-with", choices=("gzip", "bzip2", "xz"), + dest='compressor', + help="compress image with specified compressor") + subparser.add_argument("-m", "--bmap", action="store_true", help="generate .bmap") + subparser.add_argument("--no-fstab-update" ,action="store_true", + help="Do not change fstab file.") + subparser.add_argument("-v", "--vars", dest='vars_dir', + help="directory with <image>.env files that store " + "bitbake variables") + subparser.add_argument("-D", "--debug", dest="debug", action="store_true", + default=False, help="output debug information") + subparser.add_argument("-i", "--imager", dest="imager", + default="direct", help="the wic imager plugin") + return + + +def wic_init_parser_list(subparser): + subparser.add_argument("list_type", + help="can be 'images' or 'source-plugins' " + "to obtain a list. " + "If value is a valid .wks image file") + subparser.add_argument("help_for", default=[], nargs='*', + help="If 'list_type' is a valid .wks image file " + "this value can be 'help' to show the help information " + "defined inside the .wks file") + return + +def imgtype(arg): + """ + Custom type for ArgumentParser + Converts path spec to named tuple: (image, partition, path) + """ + image = arg + part = path = None + if ':' in image: + image, part = image.split(':') + if '/' in part: + part, path = part.split('/', 1) + if not path: + path = '/' + + if not os.path.isfile(image): + err = "%s is not a regular file or symlink" % image + raise argparse.ArgumentTypeError(err) + + return namedtuple('ImgType', 'image part path')(image, part, path) + +def wic_init_parser_ls(subparser): + subparser.add_argument("path", type=imgtype, + help="image spec: <image>[:<vfat partition>[<path>]]") + subparser.add_argument("-n", "--native-sysroot", + help="path to the native sysroot containing the tools") + +def imgpathtype(arg): + img = imgtype(arg) + if img.part is None: + raise argparse.ArgumentTypeError("partition number is not specified") + return img + +def wic_init_parser_cp(subparser): + subparser.add_argument("src", + help="image spec: <image>:<vfat partition>[<path>] or <file>") + subparser.add_argument("dest", + help="image spec: <image>:<vfat partition>[<path>] or <file>") + subparser.add_argument("-n", "--native-sysroot", + help="path to the native sysroot containing the tools") + +def wic_init_parser_rm(subparser): + subparser.add_argument("path", type=imgpathtype, + help="path: <image>:<vfat partition><path>") + subparser.add_argument("-n", "--native-sysroot", + help="path to the native sysroot containing the tools") + subparser.add_argument("-r", dest="recursive_delete", action="store_true", default=False, + help="remove directories and their contents recursively, " + " this only applies to ext* partition") + +def expandtype(rules): + """ + Custom type for ArgumentParser + Converts expand rules to the dictionary {<partition>: size} + """ + if rules == 'auto': + return {} + result = {} + for rule in rules.split(','): + try: + part, size = rule.split(':') + except ValueError: + raise argparse.ArgumentTypeError("Incorrect rule format: %s" % rule) + + if not part.isdigit(): + raise argparse.ArgumentTypeError("Rule '%s': partition number must be integer" % rule) + + # validate size + multiplier = 1 + for suffix, mult in [('K', 1024), ('M', 1024 * 1024), ('G', 1024 * 1024 * 1024)]: + if size.upper().endswith(suffix): + multiplier = mult + size = size[:-1] + break + if not size.isdigit(): + raise argparse.ArgumentTypeError("Rule '%s': size must be integer" % rule) + + result[int(part)] = int(size) * multiplier + + return result + +def wic_init_parser_write(subparser): + subparser.add_argument("image", + help="path to the wic image") + subparser.add_argument("target", + help="target file or device") + subparser.add_argument("-e", "--expand", type=expandtype, + help="expand rules: auto or <partition>:<size>[,<partition>:<size>]") + subparser.add_argument("-n", "--native-sysroot", + help="path to the native sysroot containing the tools") + +def wic_init_parser_help(subparser): + helpparsers = subparser.add_subparsers(dest='help_topic', help=hlp.wic_usage) + for helptopic in helptopics: + helpparsers.add_parser(helptopic, help=helptopics[helptopic][2]) + return + + +subcommands = { + "create": [wic_create_subcommand, + hlp.wic_create_usage, + hlp.wic_create_help, + wic_init_parser_create], + "list": [wic_list_subcommand, + hlp.wic_list_usage, + hlp.wic_list_help, + wic_init_parser_list], + "ls": [wic_ls_subcommand, + hlp.wic_ls_usage, + hlp.wic_ls_help, + wic_init_parser_ls], + "cp": [wic_cp_subcommand, + hlp.wic_cp_usage, + hlp.wic_cp_help, + wic_init_parser_cp], + "rm": [wic_rm_subcommand, + hlp.wic_rm_usage, + hlp.wic_rm_help, + wic_init_parser_rm], + "write": [wic_write_subcommand, + hlp.wic_write_usage, + hlp.wic_write_help, + wic_init_parser_write], + "help": [wic_help_subcommand, + wic_help_topic_usage, + hlp.wic_help_help, + wic_init_parser_help] +} + + +def init_parser(parser): + parser.add_argument("--version", action="version", + version="%(prog)s {version}".format(version=__version__)) + parser.add_argument("-D", "--debug", dest="debug", action="store_true", + default=False, help="output debug information") + + subparsers = parser.add_subparsers(dest='command', help=hlp.wic_usage) + for subcmd in subcommands: + subparser = subparsers.add_parser(subcmd, help=subcommands[subcmd][2]) + subcommands[subcmd][3](subparser) + +class WicArgumentParser(argparse.ArgumentParser): + def format_help(self): + return hlp.wic_help + def main(argv): - parser = optparse.OptionParser(version="wic version %s" % __version__, - usage=hlp.wic_usage) + parser = WicArgumentParser( + description="wic version %s" % __version__) + + init_parser(parser) - parser.disable_interspersed_args() + args = parser.parse_args(argv) - args = parser.parse_args(argv)[1] + if args.debug: + logger.setLevel(logging.DEBUG) - if len(args): - if args[0] == "help": - if len(args) == 1: + if "command" in vars(args): + if args.command == "help": + if args.help_topic is None: parser.print_help() - raise WicError("help command requires parameter") + elif args.help_topic in helptopics: + hlpt = helptopics[args.help_topic] + hlpt[0](hlpt[1], hlpt[2]) + return 0 + + # validate wic cp src and dest parameter to identify which one of it is + # image and cast it into imgtype + if args.command == "cp": + if ":" in args.dest: + args.dest = imgtype(args.dest) + elif ":" in args.src: + args.src = imgtype(args.src) + else: + raise argparse.ArgumentTypeError("no image or partition number specified.") return hlp.invoke_subcommand(args, parser, hlp.wic_help_usage, subcommands) -- 2.17.1 ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 01/10] wic: Update to the latest wic from openembedded core 2020-09-02 18:56 ` [PATCH v2 01/10] wic: Update to the latest wic from openembedded core Vijai Kumar K @ 2020-09-05 9:00 ` Henning Schild 2020-09-05 9:21 ` vijai kumar 0 siblings, 1 reply; 29+ messages in thread From: Henning Schild @ 2020-09-05 9:00 UTC (permalink / raw) To: Vijai Kumar K; +Cc: isar-users I assume if i took that commit and copied it over i would end up with 100% the same patch, right? On Thu, 3 Sep 2020 00:26:15 +0530 Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> wrote: > Update to the latest wic from OE-core. > > OE-core Revision: 532ae127c52c9f7b1d2e4ca5cbca91881d23a2ac > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > --- > scripts/lib/scriptpath.py | 32 ++ > scripts/lib/wic/__init__.py | 14 +- > scripts/lib/wic/canned-wks/common.wks.inc | 2 +- > .../directdisk-bootloader-config.cfg | 8 +- > .../lib/wic/canned-wks/efi-bootdisk.wks.in | 3 + > scripts/lib/wic/canned-wks/mkhybridiso.wks | 2 +- > scripts/lib/wic/canned-wks/qemuriscv.wks | 3 + > .../lib/wic/canned-wks/qemux86-directdisk.wks | 2 +- > .../lib/wic/canned-wks/sdimage-bootpart.wks | 4 +- > .../lib/wic/canned-wks/systemd-bootdisk.wks | 4 +- > scripts/lib/wic/engine.py | 421 +++++++++++++++- > scripts/lib/wic/filemap.py | 170 ++++--- > scripts/lib/wic/help.py | 401 ++++++++++++++-- > scripts/lib/wic/ksparser.py | 115 +++-- > scripts/lib/wic/{utils => }/misc.py | 99 ++-- > scripts/lib/wic/partition.py | 234 ++++----- > scripts/lib/wic/pluginbase.py | 36 +- > scripts/lib/wic/plugins/imager/direct.py | 175 ++++--- > .../wic/plugins/source/bootimg-biosplusefi.py | 213 +++++++++ > scripts/lib/wic/plugins/source/bootimg-efi.py | 111 +++-- > .../wic/plugins/source/bootimg-partition.py | 153 ++++-- > .../lib/wic/plugins/source/bootimg-pcbios.py | 91 ++-- > scripts/lib/wic/plugins/source/fsimage.py | 56 --- > .../wic/plugins/source/isoimage-isohybrid.py | 185 +++---- > scripts/lib/wic/plugins/source/rawcopy.py | 44 +- > scripts/lib/wic/plugins/source/rootfs.py | 159 ++++-- > scripts/lib/wic/utils/__init__.py | 0 > scripts/lib/wic/utils/runner.py | 114 ----- > scripts/wic | 452 > +++++++++++++----- 29 files changed, 2283 insertions(+), 1020 > deletions(-) create mode 100644 scripts/lib/scriptpath.py > create mode 100644 scripts/lib/wic/canned-wks/efi-bootdisk.wks.in > create mode 100644 scripts/lib/wic/canned-wks/qemuriscv.wks > rename scripts/lib/wic/{utils => }/misc.py (70%) > create mode 100644 > scripts/lib/wic/plugins/source/bootimg-biosplusefi.py delete mode > 100644 scripts/lib/wic/plugins/source/fsimage.py delete mode 100644 > scripts/lib/wic/utils/__init__.py delete mode 100644 > scripts/lib/wic/utils/runner.py > > diff --git a/scripts/lib/scriptpath.py b/scripts/lib/scriptpath.py > new file mode 100644 > index 0000000..f32326d > --- /dev/null > +++ b/scripts/lib/scriptpath.py > @@ -0,0 +1,32 @@ > +# Path utility functions for OE python scripts > +# > +# Copyright (C) 2012-2014 Intel Corporation > +# Copyright (C) 2011 Mentor Graphics Corporation > +# > +# SPDX-License-Identifier: GPL-2.0-only > +# > + > +import sys > +import os > +import os.path > + > +def add_oe_lib_path(): > + basepath = os.path.abspath(os.path.dirname(__file__) + '/../..') > + newpath = basepath + '/meta/lib' > + sys.path.insert(0, newpath) > + > +def add_bitbake_lib_path(): > + basepath = os.path.abspath(os.path.dirname(__file__) + '/../..') > + bitbakepath = None > + if os.path.exists(basepath + '/bitbake/lib/bb'): > + bitbakepath = basepath + '/bitbake' > + else: > + # look for bitbake/bin dir in PATH > + for pth in os.environ['PATH'].split(':'): > + if os.path.exists(os.path.join(pth, '../lib/bb')): > + bitbakepath = os.path.abspath(os.path.join(pth, > '..')) > + break > + > + if bitbakepath: > + sys.path.insert(0, bitbakepath + '/lib') > + return bitbakepath > diff --git a/scripts/lib/wic/__init__.py b/scripts/lib/wic/__init__.py > index 85876b1..8556793 100644 > --- a/scripts/lib/wic/__init__.py > +++ b/scripts/lib/wic/__init__.py > @@ -1,20 +1,10 @@ > -#!/usr/bin/env python -tt > +#!/usr/bin/env python3 > # > # Copyright (c) 2007 Red Hat, Inc. > # Copyright (c) 2011 Intel, Inc. > # > -# This program is free software; you can redistribute it and/or > modify it -# under the terms of the GNU General Public License as > published by the Free -# Software Foundation; version 2 of the License > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, but > -# WITHOUT ANY WARRANTY; without even the implied warranty of > MERCHANTABILITY -# or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > General Public License -# for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., 59 -# Temple Place - Suite 330, Boston, MA > 02111-1307, USA. > class WicError(Exception): > pass > diff --git a/scripts/lib/wic/canned-wks/common.wks.inc > b/scripts/lib/wic/canned-wks/common.wks.inc index 5cf2fd1..89880b4 > 100644 --- a/scripts/lib/wic/canned-wks/common.wks.inc > +++ b/scripts/lib/wic/canned-wks/common.wks.inc > @@ -1,3 +1,3 @@ > # This file is included into 3 canned wks files from this directory > part /boot --source bootimg-pcbios --ondisk sda --label boot > --active --align 1024 -part / --source rootfs --ondisk sda > --fstype=ext4 --label platform --align 1024 +part / --source rootfs > --use-uuid --fstype=ext4 --label platform --align 1024 diff --git > a/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg > b/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg index > d5a07d2..c58e74a 100644 --- > a/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg +++ > b/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg @@ > -12,16 +12,16 @@ DEFAULT Graphics console boot LABEL Graphics console > boot KERNEL /vmlinuz > -APPEND label=boot root=/dev/sda2 rootwait > +APPEND label=boot rootwait > > LABEL Serial console boot > KERNEL /vmlinuz > -APPEND label=boot root=/dev/sda2 rootwait console=ttyS0,115200 > +APPEND label=boot rootwait console=ttyS0,115200 > > LABEL Graphics console install > KERNEL /vmlinuz > -APPEND label=install root=/dev/sda2 rootwait > +APPEND label=install rootwait > > LABEL Serial console install > KERNEL /vmlinuz > -APPEND label=install root=/dev/sda2 rootwait console=ttyS0,115200 > +APPEND label=install rootwait console=ttyS0,115200 > diff --git a/scripts/lib/wic/canned-wks/efi-bootdisk.wks.in > b/scripts/lib/wic/canned-wks/efi-bootdisk.wks.in new file mode 100644 > index 0000000..7300e65 > --- /dev/null > +++ b/scripts/lib/wic/canned-wks/efi-bootdisk.wks.in > @@ -0,0 +1,3 @@ > +bootloader --ptable gpt > +part /boot --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/boot > --fstype=vfat --label boot --active --align 1024 --use-uuid > --overhead-factor 1.0 +part / --source rootfs --fstype=ext4 --label > root --align 1024 --exclude-path boot/ diff --git > a/scripts/lib/wic/canned-wks/mkhybridiso.wks > b/scripts/lib/wic/canned-wks/mkhybridiso.wks index 9d34e9b..48c5ac4 > 100644 --- a/scripts/lib/wic/canned-wks/mkhybridiso.wks +++ > b/scripts/lib/wic/canned-wks/mkhybridiso.wks @@ -2,6 +2,6 @@ # > long-description: Creates an EFI and legacy bootable hybrid ISO image > # which can be used on optical media as well as USB media. > -part /boot --source isoimage-isohybrid > --sourceparams="loader=grub-efi,image_name=HYBRID_ISO_IMG" --ondisk > cd --label HYBRIDISO --fstype=ext4 +part /boot --source > isoimage-isohybrid > --sourceparams="loader=grub-efi,image_name=HYBRID_ISO_IMG" --ondisk > cd --label HYBRIDISO bootloader --timeout=15 --append="" diff --git > a/scripts/lib/wic/canned-wks/qemuriscv.wks > b/scripts/lib/wic/canned-wks/qemuriscv.wks new file mode 100644 index > 0000000..12c68b7 --- /dev/null +++ > b/scripts/lib/wic/canned-wks/qemuriscv.wks @@ -0,0 +1,3 @@ > +# short-description: Create qcow2 image for RISC-V QEMU machines > + > +part / --source rootfs --fstype=ext4 --label root --align 4096 > --size 5G diff --git > a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks > b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks index > a6518a0..22b4521 100644 --- > a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks +++ > b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks @@ -4,5 +4,5 @@ > include common.wks.inc > > -bootloader --timeout=0 --append="vga=0 > uvesafb.mode_option=640x480-32 root=/dev/vda2 rw mem=256M > ip=192.168.7.2::192.168.7.1:255.255.255.0 oprofile.timer=1 > rootfstype=ext4 " +bootloader --timeout=0 --append="rw > oprofile.timer=1 rootfstype=ext4 " diff --git > a/scripts/lib/wic/canned-wks/sdimage-bootpart.wks > b/scripts/lib/wic/canned-wks/sdimage-bootpart.wks index > 7ffd632..63bc4da 100644 --- > a/scripts/lib/wic/canned-wks/sdimage-bootpart.wks +++ > b/scripts/lib/wic/canned-wks/sdimage-bootpart.wks @@ -2,5 +2,5 @@ # > long-description: Creates a partitioned SD card image. Boot files # > are located in the first vfat partition. -part /boot --source > bootimg-partition --ondisk mmcblk --fstype=vfat --label boot --active > --align 4 --size 16 -part / --source rootfs --ondisk mmcblk > --fstype=ext4 --label root --align 4 +part /boot --source > bootimg-partition --ondisk mmcblk0 --fstype=vfat --label boot > --active --align 4 --size 16 +part / --source rootfs --ondisk mmcblk0 > --fstype=ext4 --label root --align 4 diff --git > a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks > b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks index > 4bd9d6a..95d7b97 100644 --- > a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks +++ > b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks @@ -2,10 +2,10 @@ # > long-description: Creates a partitioned EFI disk image that the user > # can directly dd to boot media. The selected bootloader is > systemd-boot. -part /boot --source bootimg-efi > --sourceparams="loader=systemd-boot" --ondisk sda --label msdos > --active --align 1024 +part /boot --source bootimg-efi > --sourceparams="loader=systemd-boot" --ondisk sda --label msdos > --active --align 1024 --use-uuid part / --source rootfs --ondisk sda > --fstype=ext4 --label platform --align 1024 --use-uuid -part swap > --ondisk sda --size 44 --label swap1 --fstype=swap +part swap > --ondisk sda --size 44 --label swap1 --fstype=swap --use-uuid > bootloader --ptable gpt --timeout=5 --append="rootwait > rootfstype=ext4 console=ttyS0,115200 console=tty0" diff --git > a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py index > f59821f..018815b 100644 --- a/scripts/lib/wic/engine.py +++ > b/scripts/lib/wic/engine.py @@ -1,21 +1,7 @@ -# ex:ts=4:sw=4:sts=4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Copyright (c) 2013, Intel Corporation. > -# All rights reserved. > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # DESCRIPTION > > @@ -30,10 +16,18 @@ > > import logging > import os > +import tempfile > +import json > +import subprocess > +import re > + > +from collections import namedtuple, OrderedDict > +from distutils.spawn import find_executable > > from wic import WicError > +from wic.filemap import sparse_copy > from wic.pluginbase import PluginMgr > -from wic.utils.misc import get_bitbake_var > +from wic.misc import get_bitbake_var, exec_cmd > > logger = logging.getLogger('wic') > > @@ -82,7 +76,8 @@ def find_canned_image(scripts_path, wks_file): > for fname in files: > if fname.endswith("~") or fname.endswith("#"): > continue > - if fname.endswith(".wks") and wks_file + ".wks" == > fname: > + if ((fname.endswith(".wks") and wks_file + ".wks" == > fname) or \ > + (fname.endswith(".wks.in") and wks_file + > ".wks.in" == fname)): fullpath = os.path.join(canned_wks_dir, fname) > return fullpath > return None > @@ -99,7 +94,7 @@ def list_canned_images(scripts_path): > for fname in files: > if fname.endswith("~") or fname.endswith("#"): > continue > - if fname.endswith(".wks"): > + if fname.endswith(".wks") or > fname.endswith(".wks.in"): fullpath = os.path.join(canned_wks_dir, > fname) with open(fullpath) as wks: > for line in wks: > @@ -108,7 +103,7 @@ def list_canned_images(scripts_path): > if idx != -1: > desc = line[idx + > len("short-description:"):].strip() break > - basename = os.path.splitext(fname)[0] > + basename = fname.split('.')[0] > print(" %s\t\t%s" % (basename.ljust(30), desc)) > > > @@ -184,7 +179,7 @@ def wic_create(wks_file, rootfs_dir, bootimg_dir, > kernel_dir, if not os.path.exists(options.outdir): > os.makedirs(options.outdir) > > - pname = 'direct' > + pname = options.imager > plugin_class = PluginMgr.get_plugins('imager').get(pname) > if not plugin_class: > raise WicError('Unknown plugin: %s' % pname) > @@ -201,17 +196,18 @@ def wic_list(args, scripts_path): > """ > Print the list of images or source plugins. > """ > - if len(args) < 1: > + if args.list_type is None: > return False > > - if args == ["images"]: > + if args.list_type == "images": > + > list_canned_images(scripts_path) > return True > - elif args == ["source-plugins"]: > + elif args.list_type == "source-plugins": > list_source_plugins() > return True > - elif len(args) == 2 and args[1] == "help": > - wks_file = args[0] > + elif len(args.help_for) == 1 and args.help_for[0] == 'help': > + wks_file = args.list_type > fullpath = find_canned_image(scripts_path, wks_file) > if not fullpath: > raise WicError("No image named %s found, exiting. " > @@ -224,6 +220,381 @@ def wic_list(args, scripts_path): > > return False > > + > +class Disk: > + def __init__(self, imagepath, native_sysroot, fstypes=('fat', > 'ext')): > + self.imagepath = imagepath > + self.native_sysroot = native_sysroot > + self.fstypes = fstypes > + self._partitions = None > + self._partimages = {} > + self._lsector_size = None > + self._psector_size = None > + self._ptable_format = None > + > + # find parted > + # read paths from $PATH environment variable > + # if it fails, use hardcoded paths > + pathlist = "/bin:/usr/bin:/usr/sbin:/sbin/" > + try: > + self.paths = os.environ['PATH'] + ":" + pathlist > + except KeyError: > + self.paths = pathlist > + > + if native_sysroot: > + for path in pathlist.split(':'): > + self.paths = "%s%s:%s" % (native_sysroot, path, > self.paths) + > + self.parted = find_executable("parted", self.paths) > + if not self.parted: > + raise WicError("Can't find executable parted") > + > + self.partitions = self.get_partitions() > + > + def __del__(self): > + for path in self._partimages.values(): > + os.unlink(path) > + > + def get_partitions(self): > + if self._partitions is None: > + self._partitions = OrderedDict() > + out = exec_cmd("%s -sm %s unit B print" % (self.parted, > self.imagepath)) > + parttype = namedtuple("Part", "pnum start end size > fstype") > + splitted = out.splitlines() > + # skip over possible errors in exec_cmd output > + try: > + idx =splitted.index("BYT;") > + except ValueError: > + raise WicError("Error getting partition information > from %s" % (self.parted)) > + lsector_size, psector_size, self._ptable_format = > splitted[idx + 1].split(":")[3:6] > + self._lsector_size = int(lsector_size) > + self._psector_size = int(psector_size) > + for line in splitted[idx + 2:]: > + pnum, start, end, size, fstype = line.split(':')[:5] > + partition = parttype(int(pnum), int(start[:-1]), > int(end[:-1]), > + int(size[:-1]), fstype) > + self._partitions[pnum] = partition > + > + return self._partitions > + > + def __getattr__(self, name): > + """Get path to the executable in a lazy way.""" > + if name in ("mdir", "mcopy", "mdel", "mdeltree", "sfdisk", > "e2fsck", > + "resize2fs", "mkswap", "mkdosfs", > "debugfs","blkid"): > + aname = "_%s" % name > + if aname not in self.__dict__: > + setattr(self, aname, find_executable(name, > self.paths)) > + if aname not in self.__dict__ or > self.__dict__[aname] is None: > + raise WicError("Can't find executable > '{}'".format(name)) > + return self.__dict__[aname] > + return self.__dict__[name] > + > + def _get_part_image(self, pnum): > + if pnum not in self.partitions: > + raise WicError("Partition %s is not in the image" % pnum) > + part = self.partitions[pnum] > + # check if fstype is supported > + for fstype in self.fstypes: > + if part.fstype.startswith(fstype): > + break > + else: > + raise WicError("Not supported fstype: > {}".format(part.fstype)) > + if pnum not in self._partimages: > + tmpf = tempfile.NamedTemporaryFile(prefix="wic-part") > + dst_fname = tmpf.name > + tmpf.close() > + sparse_copy(self.imagepath, dst_fname, skip=part.start, > length=part.size) > + self._partimages[pnum] = dst_fname > + > + return self._partimages[pnum] > + > + def _put_part_image(self, pnum): > + """Put partition image into partitioned image.""" > + sparse_copy(self._partimages[pnum], self.imagepath, > + seek=self.partitions[pnum].start) > + > + def dir(self, pnum, path): > + if pnum not in self.partitions: > + raise WicError("Partition %s is not in the image" % pnum) > + > + if self.partitions[pnum].fstype.startswith('ext'): > + return exec_cmd("{} {} -R 'ls -l > {}'".format(self.debugfs, > + > self._get_part_image(pnum), > + path), > as_shell=True) > + else: # fat > + return exec_cmd("{} -i {} ::{}".format(self.mdir, > + > self._get_part_image(pnum), > + path)) > + > + def copy(self, src, dest): > + """Copy partition image into wic image.""" > + pnum = dest.part if isinstance(src, str) else src.part > + > + if self.partitions[pnum].fstype.startswith('ext'): > + if isinstance(src, str): > + cmd = "printf 'cd {}\nwrite {} {}\n' | {} -w {}".\ > + format(os.path.dirname(dest.path), src, > os.path.basename(src), > + self.debugfs, > self._get_part_image(pnum)) > + else: # copy from wic > + # run both dump and rdump to support both files and > directory > + cmd = "printf 'cd {}\ndump /{} {}\nrdump /{} {}\n' | > {} {}".\ > + format(os.path.dirname(src.path), src.path, > + dest, src.path, dest, self.debugfs, > + self._get_part_image(pnum)) > + else: # fat > + if isinstance(src, str): > + cmd = "{} -i {} -snop {} ::{}".format(self.mcopy, > + > self._get_part_image(pnum), > + src, dest.path) > + else: > + cmd = "{} -i {} -snop ::{} {}".format(self.mcopy, > + > self._get_part_image(pnum), > + src.path, dest) > + > + exec_cmd(cmd, as_shell=True) > + self._put_part_image(pnum) > + > + def remove_ext(self, pnum, path, recursive): > + """ > + Remove files/dirs and their contents from the partition. > + This only applies to ext* partition. > + """ > + abs_path = re.sub('\/\/+', '/', path) > + cmd = "{} {} -wR 'rm \"{}\"'".format(self.debugfs, > + > self._get_part_image(pnum), > + abs_path) > + out = exec_cmd(cmd , as_shell=True) > + for line in out.splitlines(): > + if line.startswith("rm:"): > + if "file is a directory" in line: > + if recursive: > + # loop through content and delete them one > by one if > + # flaged with -r > + subdirs = iter(self.dir(pnum, > abs_path).splitlines()) > + next(subdirs) > + for subdir in subdirs: > + dir = subdir.split(':')[1].split(" ", > 1)[1] > + if not dir == "." and not dir == "..": > + self.remove_ext(pnum, "%s/%s" % > (abs_path, dir), recursive) + > + rmdir_out = exec_cmd("{} {} -wR 'rmdir > \"{}\"'".format(self.debugfs, > + > self._get_part_image(pnum), > + > abs_path.rstrip('/')) > + , as_shell=True) > + > + for rmdir_line in rmdir_out.splitlines(): > + if "directory not empty" in rmdir_line: > + raise WicError("Could not complete > operation: \n%s \n" > + "use -r to remove > non-empty directory" % rmdir_line) > + if rmdir_line.startswith("rmdir:"): > + raise WicError("Could not complete > operation: \n%s " > + "\n%s" % (str(line), > rmdir_line)) + > + else: > + raise WicError("Could not complete operation: > \n%s " > + "\nUnable to remove %s" % > (str(line), abs_path)) + > + def remove(self, pnum, path, recursive): > + """Remove files/dirs from the partition.""" > + partimg = self._get_part_image(pnum) > + if self.partitions[pnum].fstype.startswith('ext'): > + self.remove_ext(pnum, path, recursive) > + > + else: # fat > + cmd = "{} -i {} ::{}".format(self.mdel, partimg, path) > + try: > + exec_cmd(cmd) > + except WicError as err: > + if "not found" in str(err) or "non empty" in > str(err): > + # mdel outputs 'File ... not found' or > 'directory .. non empty" > + # try to use mdeltree as path could be a > directory > + cmd = "{} -i {} ::{}".format(self.mdeltree, > + partimg, path) > + exec_cmd(cmd) > + else: > + raise err > + self._put_part_image(pnum) > + > + def write(self, target, expand): > + """Write disk image to the media or file.""" > + def write_sfdisk_script(outf, parts): > + for key, val in parts['partitiontable'].items(): > + if key in ("partitions", "device", "firstlba", > "lastlba"): > + continue > + if key == "id": > + key = "label-id" > + outf.write("{}: {}\n".format(key, val)) > + outf.write("\n") > + for part in parts['partitiontable']['partitions']: > + line = '' > + for name in ('attrs', 'name', 'size', 'type', > 'uuid'): > + if name == 'size' and part['type'] == 'f': > + # don't write size for extended partition > + continue > + val = part.get(name) > + if val: > + line += '{}={}, '.format(name, val) > + if line: > + line = line[:-2] # strip ', ' > + if part.get('bootable'): > + line += ' ,bootable' > + outf.write("{}\n".format(line)) > + outf.flush() > + > + def read_ptable(path): > + out = exec_cmd("{} -J {}".format(self.sfdisk, path)) > + return json.loads(out) > + > + def write_ptable(parts, target): > + with tempfile.NamedTemporaryFile(prefix="wic-sfdisk-", > mode='w') as outf: > + write_sfdisk_script(outf, parts) > + cmd = "{} --no-reread {} < {} ".format(self.sfdisk, > target, outf.name) > + exec_cmd(cmd, as_shell=True) > + > + if expand is None: > + sparse_copy(self.imagepath, target) > + else: > + # copy first sectors that may contain bootloader > + sparse_copy(self.imagepath, target, length=2048 * > self._lsector_size) + > + # copy source partition table to the target > + parts = read_ptable(self.imagepath) > + write_ptable(parts, target) > + > + # get size of unpartitioned space > + free = None > + for line in exec_cmd("{} -F {}".format(self.sfdisk, > target)).splitlines(): > + if line.startswith("Unpartitioned space ") and > line.endswith("sectors"): > + free = int(line.split()[-2]) > + # Align free space to a 2048 sector boundary. > YOCTO #12840. > + free = free - (free % 2048) > + if free is None: > + raise WicError("Can't get size of unpartitioned > space") + > + # calculate expanded partitions sizes > + sizes = {} > + num_auto_resize = 0 > + for num, part in > enumerate(parts['partitiontable']['partitions'], 1): > + if num in expand: > + if expand[num] != 0: # don't resize partition if > size is set to 0 > + sectors = expand[num] // self._lsector_size > + free -= sectors - part['size'] > + part['size'] = sectors > + sizes[num] = sectors > + elif part['type'] != 'f': > + sizes[num] = -1 > + num_auto_resize += 1 > + > + for num, part in > enumerate(parts['partitiontable']['partitions'], 1): > + if sizes.get(num) == -1: > + part['size'] += free // num_auto_resize > + > + # write resized partition table to the target > + write_ptable(parts, target) > + > + # read resized partition table > + parts = read_ptable(target) > + > + # copy partitions content > + for num, part in > enumerate(parts['partitiontable']['partitions'], 1): > + pnum = str(num) > + fstype = self.partitions[pnum].fstype > + > + # copy unchanged partition > + if part['size'] == self.partitions[pnum].size // > self._lsector_size: > + logger.info("copying unchanged partition > {}".format(pnum)) > + sparse_copy(self._get_part_image(pnum), target, > seek=part['start'] * self._lsector_size) > + continue > + > + # resize or re-create partitions > + if fstype.startswith('ext') or > fstype.startswith('fat') or \ > + fstype.startswith('linux-swap'): > + > + partfname = None > + with > tempfile.NamedTemporaryFile(prefix="wic-part{}-".format(pnum)) as > partf: > + partfname = partf.name > + > + if fstype.startswith('ext'): > + logger.info("resizing ext partition > {}".format(pnum)) > + partimg = self._get_part_image(pnum) > + sparse_copy(partimg, partfname) > + exec_cmd("{} -pf {}".format(self.e2fsck, > partfname)) > + exec_cmd("{} {} {}s".format(\ > + self.resize2fs, partfname, > part['size'])) > + elif fstype.startswith('fat'): > + logger.info("copying content of the fat > partition {}".format(pnum)) > + with > tempfile.TemporaryDirectory(prefix='wic-fatdir-') as tmpdir: > + # copy content to the temporary directory > + cmd = "{} -snompi {} :: > {}".format(self.mcopy, > + > self._get_part_image(pnum), > + > tmpdir) > + exec_cmd(cmd) > + # create new msdos partition > + label = part.get("name") > + label_str = "-n {}".format(label) if > label else '' + > + cmd = "{} {} -C {} > {}".format(self.mkdosfs, label_str, partfname, > + > part['size']) > + exec_cmd(cmd) > + # copy content from the temporary > directory to the new partition > + cmd = "{} -snompi {} {}/* > ::".format(self.mcopy, partfname, tmpdir) > + exec_cmd(cmd, as_shell=True) > + elif fstype.startswith('linux-swap'): > + logger.info("creating swap partition > {}".format(pnum)) > + label = part.get("name") > + label_str = "-L {}".format(label) if label > else '' > + out = exec_cmd("{} --probe > {}".format(self.blkid, self._get_part_image(pnum))) > + uuid = > out[out.index("UUID=\"")+6:out.index("UUID=\"")+42] > + uuid_str = "-U {}".format(uuid) if uuid else > '' > + with open(partfname, 'w') as sparse: > + os.ftruncate(sparse.fileno(), > part['size'] * self._lsector_size) > + exec_cmd("{} {} {} {}".format(self.mkswap, > label_str, uuid_str, partfname)) > + sparse_copy(partfname, target, > seek=part['start'] * self._lsector_size) > + os.unlink(partfname) > + elif part['type'] != 'f': > + logger.warning("skipping partition {}: > unsupported fstype {}".format(pnum, fstype)) + > +def wic_ls(args, native_sysroot): > + """List contents of partitioned image or vfat partition.""" > + disk = Disk(args.path.image, native_sysroot) > + if not args.path.part: > + if disk.partitions: > + print('Num Start End Size > Fstype') > + for part in disk.partitions.values(): > + print("{:2d} {:12d} {:12d} {:12d} {}".format(\ > + part.pnum, part.start, part.end, > + part.size, part.fstype)) > + else: > + path = args.path.path or '/' > + print(disk.dir(args.path.part, path)) > + > +def wic_cp(args, native_sysroot): > + """ > + Copy file or directory to/from the vfat/ext partition of > + partitioned image. > + """ > + if isinstance(args.dest, str): > + disk = Disk(args.src.image, native_sysroot) > + else: > + disk = Disk(args.dest.image, native_sysroot) > + disk.copy(args.src, args.dest) > + > + > +def wic_rm(args, native_sysroot): > + """ > + Remove files or directories from the vfat partition of > + partitioned image. > + """ > + disk = Disk(args.path.image, native_sysroot) > + disk.remove(args.path.part, args.path.path, > args.recursive_delete) + > +def wic_write(args, native_sysroot): > + """ > + Write image to a target device. > + """ > + disk = Disk(args.image, native_sysroot, ('fat', 'ext', > 'linux-swap')) > + disk.write(args.target, args.expand) > + > def find_canned(scripts_path, file_name): > """ > Find a file either by its path or by name in the canned files > dir. diff --git a/scripts/lib/wic/filemap.py > b/scripts/lib/wic/filemap.py index 080668e..4d9da28 100644 > --- a/scripts/lib/wic/filemap.py > +++ b/scripts/lib/wic/filemap.py > @@ -1,13 +1,8 @@ > +# > # Copyright (c) 2012 Intel, Inc. > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License, > version 2, -# as published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, but > -# WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > -# General Public License for more details. > > """ > This module implements python implements a way to get file block. > Two methods @@ -22,6 +17,7 @@ and returns an instance of the class. > # * Too many instance attributes (R0902) > # pylint: disable=R0902 > > +import errno > import os > import struct > import array > @@ -34,14 +30,23 @@ def get_block_size(file_obj): > Returns block size for file object 'file_obj'. Errors are > indicated by the 'IOError' exception. > """ > - > - from fcntl import ioctl > - import struct > - > # Get the block size of the host file-system for the image file > by calling # the FIGETBSZ ioctl (number 2). > - binary_data = ioctl(file_obj, 2, struct.pack('I', 0)) > - return struct.unpack('I', binary_data)[0] > + try: > + binary_data = fcntl.ioctl(file_obj, 2, struct.pack('I', 0)) > + bsize = struct.unpack('I', binary_data)[0] > + except OSError: > + bsize = None > + > + # If ioctl causes OSError or give bsize to zero failback to > os.fstat > + if not bsize: > + import os > + stat = os.fstat(file_obj.fileno()) > + if hasattr(stat, 'st_blksize'): > + bsize = stat.st_blksize > + else: > + raise IOError("Unable to determine block size") > + return bsize > > class ErrorNotSupp(Exception): > """ > @@ -137,15 +142,6 @@ class _FilemapBase(object): > > raise Error("the method is not implemented") > > - def block_is_unmapped(self, block): # pylint: disable=W0613,R0201 > - """ > - This method has has to be implemented by child classes. It > returns > - 'True' if block number 'block' of the image file is not > mapped (hole) > - and 'False' otherwise. > - """ > - > - raise Error("the method is not implemented") > - > def get_mapped_ranges(self, start, count): # pylint: > disable=W0613,R0201 """ > This method has has to be implemented by child classes. This > is a @@ -159,15 +155,6 @@ class _FilemapBase(object): > > raise Error("the method is not implemented") > > - def get_unmapped_ranges(self, start, count): # pylint: > disable=W0613,R0201 > - """ > - This method has has to be implemented by child classes. Just > like > - 'get_mapped_ranges()', but yields unmapped block ranges > instead > - (holes). > - """ > - > - raise Error("the method is not implemented") > - > > # The 'SEEK_HOLE' and 'SEEK_DATA' options of the file seek system > call _SEEK_DATA = 3 > @@ -185,9 +172,9 @@ def _lseek(file_obj, offset, whence): > except OSError as err: > # The 'lseek' system call returns the ENXIO if there is no > data or # hole starting from the specified offset. > - if err.errno == os.errno.ENXIO: > + if err.errno == errno.ENXIO: > return -1 > - elif err.errno == os.errno.EINVAL: > + elif err.errno == errno.EINVAL: > raise ErrorNotSupp("the kernel or file-system does not > support " "\"SEEK_HOLE\" and \"SEEK_DATA\"") > else: > @@ -228,7 +215,7 @@ class FilemapSeek(_FilemapBase): > try: > tmp_obj = tempfile.TemporaryFile("w+", dir=directory) > except IOError as err: > - raise ErrorNotSupp("cannot create a temporary in \"%s\": > %s" > + raise ErrorNotSupp("cannot create a temporary in \"%s\": > %s" \ % (directory, err)) > > try: > @@ -260,15 +247,10 @@ class FilemapSeek(_FilemapBase): > % (block, result)) > return result > > - def block_is_unmapped(self, block): > - """Refer the '_FilemapBase' class for the documentation.""" > - return not self.block_is_mapped(block) > - > def _get_ranges(self, start, count, whence1, whence2): > """ > - This function implements 'get_mapped_ranges()' and > - 'get_unmapped_ranges()' depending on what is passed in the > 'whence1' > - and 'whence2' arguments. > + This function implements 'get_mapped_ranges()' depending > + on what is passed in the 'whence1' and 'whence2' arguments. > """ > > assert whence1 != whence2 > @@ -298,12 +280,6 @@ class FilemapSeek(_FilemapBase): > % (start, count, start + count - 1)) > return self._get_ranges(start, count, _SEEK_DATA, _SEEK_HOLE) > > - def get_unmapped_ranges(self, start, count): > - """Refer the '_FilemapBase' class for the documentation.""" > - self._log.debug("FilemapSeek: get_unmapped_ranges(%d, > %d(%d))" > - % (start, count, start + count - 1)) > - return self._get_ranges(start, count, _SEEK_HOLE, _SEEK_DATA) > - > > # Below goes the FIEMAP ioctl implementation, which is not very > readable # because it deals with the rather complex FIEMAP ioctl. To > understand the @@ -390,12 +366,12 @@ class > FilemapFiemap(_FilemapBase): except IOError as err: > # Note, the FIEMAP ioctl is supported by the Linux > kernel starting # from version 2.6.28 (year 2008). > - if err.errno == os.errno.EOPNOTSUPP: > + if err.errno == errno.EOPNOTSUPP: > errstr = "FilemapFiemap: the FIEMAP ioctl is not > supported " \ "by the file-system" > self._log.debug(errstr) > raise ErrorNotSupp(errstr) > - if err.errno == os.errno.ENOTTY: > + if err.errno == errno.ENOTTY: > errstr = "FilemapFiemap: the FIEMAP ioctl is not > supported " \ "by the kernel" > self._log.debug(errstr) > @@ -417,10 +393,6 @@ class FilemapFiemap(_FilemapBase): > % (block, result)) > return result > > - def block_is_unmapped(self, block): > - """Refer the '_FilemapBase' class for the documentation.""" > - return not self.block_is_mapped(block) > - > def _unpack_fiemap_extent(self, index): > """ > Unpack a 'struct fiemap_extent' structure object number > 'index' from @@ -497,23 +469,28 @@ class FilemapFiemap(_FilemapBase): > % (first_prev, last_prev)) > yield (first_prev, last_prev) > > - def get_unmapped_ranges(self, start, count): > +class FilemapNobmap(_FilemapBase): > + """ > + This class is used when both the 'SEEK_DATA/HOLE' and FIEMAP are > not > + supported by the filesystem or kernel. > + """ > + > + def __init__(self, image, log=None): > """Refer the '_FilemapBase' class for the documentation.""" > - self._log.debug("FilemapFiemap: get_unmapped_ranges(%d, > %d(%d))" > - % (start, count, start + count - 1)) > - hole_first = start > - for first, last in self._do_get_mapped_ranges(start, count): > - if first > hole_first: > - self._log.debug("FilemapFiemap: yielding range (%d, > %d)" > - % (hole_first, first - 1)) > - yield (hole_first, first - 1) > > - hole_first = last + 1 > + # Call the base class constructor first > + _FilemapBase.__init__(self, image, log) > + self._log.debug("FilemapNobmap: initializing") > > - if hole_first < start + count: > - self._log.debug("FilemapFiemap: yielding range (%d, %d)" > - % (hole_first, start + count - 1)) > - yield (hole_first, start + count - 1) > + def block_is_mapped(self, block): > + """Refer the '_FilemapBase' class for the documentation.""" > + return True > + > + def get_mapped_ranges(self, start, count): > + """Refer the '_FilemapBase' class for the documentation.""" > + self._log.debug("FilemapNobmap: get_mapped_ranges(%d, > %d(%d))" > + % (start, count, start + count - 1)) > + yield (start, start + count -1) > > def filemap(image, log=None): > """ > @@ -528,26 +505,56 @@ def filemap(image, log=None): > try: > return FilemapFiemap(image, log) > except ErrorNotSupp: > - return FilemapSeek(image, log) > + try: > + return FilemapSeek(image, log) > + except ErrorNotSupp: > + return FilemapNobmap(image, log) > > -def sparse_copy(src_fname, dst_fname, offset=0, skip=0): > - """Efficiently copy sparse file to or into another file.""" > - fmap = filemap(src_fname) > +def sparse_copy(src_fname, dst_fname, skip=0, seek=0, > + length=0, api=None): > + """ > + Efficiently copy sparse file to or into another file. > + > + src_fname: path to source file > + dst_fname: path to destination file > + skip: skip N bytes at thestart of src > + seek: seek N bytes from the start of dst > + length: read N bytes from src and write them to dst > + api: FilemapFiemap or FilemapSeek object > + """ > + if not api: > + api = filemap > + fmap = api(src_fname) > try: > dst_file = open(dst_fname, 'r+b') > except IOError: > dst_file = open(dst_fname, 'wb') > - dst_file.truncate(os.path.getsize(src_fname)) > + if length: > + dst_size = length + seek > + else: > + dst_size = os.path.getsize(src_fname) + seek - skip > + dst_file.truncate(dst_size) > > + written = 0 > for first, last in fmap.get_mapped_ranges(0, fmap.blocks_cnt): > start = first * fmap.block_size > end = (last + 1) * fmap.block_size > > + if skip >= end: > + continue > + > if start < skip < end: > - fmap._f_image.seek(skip, os.SEEK_SET) > - else: > - fmap._f_image.seek(start, os.SEEK_SET) > - dst_file.seek(offset + start, os.SEEK_SET) > + start = skip > + > + fmap._f_image.seek(start, os.SEEK_SET) > + > + written += start - skip - written > + if length and written >= length: > + dst_file.seek(seek + length, os.SEEK_SET) > + dst_file.close() > + return > + > + dst_file.seek(seek + start - skip, os.SEEK_SET) > > chunk_size = 1024 * 1024 > to_read = end - start > @@ -556,7 +563,14 @@ def sparse_copy(src_fname, dst_fname, offset=0, > skip=0): while read < to_read: > if read + chunk_size > to_read: > chunk_size = to_read - read > - chunk = fmap._f_image.read(chunk_size) > + size = chunk_size > + if length and written + size > length: > + size = length - written > + chunk = fmap._f_image.read(size) > dst_file.write(chunk) > - read += chunk_size > + read += size > + written += size > + if written == length: > + dst_file.close() > + return > dst_file.close() > diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py > index 148da89..bd3a2b9 100644 > --- a/scripts/lib/wic/help.py > +++ b/scripts/lib/wic/help.py > @@ -1,21 +1,6 @@ > -# ex:ts=4:sw=4:sts=4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > -# > # Copyright (c) 2013, Intel Corporation. > -# All rights reserved. > -# > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # DESCRIPTION > # This module implements some basic help invocation functions along > @@ -56,7 +41,7 @@ def wic_help(args, usage_str, subcommands): > """ > Subcommand help dispatcher. > """ > - if len(args) == 1 or not display_help(args[1], subcommands): > + if args.help_topic == None or not display_help(args.help_topic, > subcommands): print(usage_str) > > > @@ -82,19 +67,20 @@ def invoke_subcommand(args, parser, > main_command_usage, subcommands): Dispatch to subcommand handler > borrowed from combo-layer. Should use argparse, but has to work in > 2.6. """ > - if not args: > + if not args.command: > logger.error("No subcommand specified, exiting") > parser.print_help() > return 1 > - elif args[0] == "help": > + elif args.command == "help": > wic_help(args, main_command_usage, subcommands) > - elif args[0] not in subcommands: > - logger.error("Unsupported subcommand %s, exiting\n", args[0]) > + elif args.command not in subcommands: > + logger.error("Unsupported subcommand %s, exiting\n", > args.command) parser.print_help() > return 1 > else: > - usage = subcommands.get(args[0], subcommand_error)[1] > - subcommands.get(args[0], subcommand_error)[0](args[1:], > usage) > + subcmd = subcommands.get(args.command, subcommand_error) > + usage = subcmd[1] > + subcmd[0](args, usage) > > > ## > @@ -130,10 +116,10 @@ wic_create_usage = """ > Create a new OpenEmbedded image > > usage: wic create <wks file or image name> [-o <DIRNAME> | --outdir > <DIRNAME>] > - [-i <JSON PROPERTY FILE> | --infile <JSON PROPERTY_FILE>] > [-e | --image-name] [-s, --skip-build-check] [-D, > --debug] [-r, --rootfs-dir] [-b, --bootimg-dir] > [-k, --kernel-dir] [-n, --native-sysroot] [-f, > --build-rootfs] > + [-c, --compress-with] [-m, --bmap] > > This command creates an OpenEmbedded image based on the 'OE > kickstart commands' found in the <wks file>. > @@ -154,7 +140,7 @@ SYNOPSIS > [-e | --image-name] [-s, --skip-build-check] [-D, --debug] > [-r, --rootfs-dir] [-b, --bootimg-dir] > [-k, --kernel-dir] [-n, --native-sysroot] [-f, > --build-rootfs] > - [-c, --compress-with] [-m, --bmap] > + [-c, --compress-with] [-m, --bmap] [--no-fstab-update] > > DESCRIPTION > This command creates an OpenEmbedded image based on the 'OE > @@ -226,6 +212,11 @@ DESCRIPTION > > The -m option is used to produce .bmap file for the image. This > file can be used to flash image using bmaptool utility. > + > + The --no-fstab-update option is used to doesn't change fstab > file. When > + using this option the final fstab file will be same that in > rootfs and > + wic doesn't update file, e.g adding a new mount point. User can > control > + the fstab file content in base-files recipe. > """ > > wic_list_usage = """ > @@ -283,6 +274,243 @@ DESCRIPTION > details. > """ > > +wic_ls_usage = """ > + > + List content of a partitioned image > + > + usage: wic ls <image>[:<partition>[<path>]] [--native-sysroot > <path>] + > + This command outputs either list of image partitions or directory > contents > + of vfat and ext* partitions. > + > + See 'wic help ls' for more detailed instructions. > + > +""" > + > +wic_ls_help = """ > + > +NAME > + wic ls - List contents of partitioned image or partition > + > +SYNOPSIS > + wic ls <image> > + wic ls <image>:<vfat or ext* partition> > + wic ls <image>:<vfat or ext* partition><path> > + wic ls <image>:<vfat or ext* partition><path> --native-sysroot > <path> + > +DESCRIPTION > + This command lists either partitions of the image or directory > contents > + of vfat or ext* partitions. > + > + The first form it lists partitions of the image. > + For example: > + $ wic ls > tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic > + Num Start End Size Fstype > + 1 1048576 24438783 23390208 fat16 > + 2 25165824 50315263 25149440 ext4 > + > + Second and third form list directory content of the partition: > + $ wic ls > tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 > + Volume in drive : is boot > + Volume Serial Number is 2DF2-5F02 > + Directory for ::/ > + > + efi <DIR> 2017-05-11 10:54 > + startup nsh 26 2017-05-11 10:54 > + vmlinuz 6922288 2017-05-11 10:54 > + 3 files 6 922 314 bytes > + 15 818 752 bytes free > + > + > + $ wic ls > tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/EFI/boot/ > + Volume in drive : is boot > + Volume Serial Number is 2DF2-5F02 > + Directory for ::/EFI/boot > + > + . <DIR> 2017-05-11 10:54 > + .. <DIR> 2017-05-11 10:54 > + grub cfg 679 2017-05-11 10:54 > + bootx64 efi 571392 2017-05-11 10:54 > + 4 files 572 071 bytes > + 15 818 752 bytes free > + > + The -n option is used to specify the path to the native sysroot > + containing the tools(parted and mtools) to use. > + > +""" > + > +wic_cp_usage = """ > + > + Copy files and directories to/from the vfat or ext* partition > + > + usage: wic cp <src> <dest> [--native-sysroot <path>] > + > + source/destination image in format <image>:<partition>[<path>] > + > + This command copies files or directories either > + - from local to vfat or ext* partitions of partitioned image > + - from vfat or ext* partitions of partitioned image to local > + > + See 'wic help cp' for more detailed instructions. > + > +""" > + > +wic_cp_help = """ > + > +NAME > + wic cp - copy files and directories to/from the vfat or ext* > partitions + > +SYNOPSIS > + wic cp <src> <dest>:<partition> > + wic cp <src>:<partition> <dest> > + wic cp <src> <dest-image>:<partition><path> > + wic cp <src> <dest-image>:<partition><path> --native-sysroot > <path> + > +DESCRIPTION > + This command copies files or directories either > + - from local to vfat or ext* partitions of partitioned image > + - from vfat or ext* partitions of partitioned image to local > + > + The first form of it copies file or directory to the root > directory of > + the partition: > + $ wic cp test.wks > tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 > + $ wic ls > tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 > + Volume in drive : is boot > + Volume Serial Number is DB4C-FD4C > + Directory for ::/ > + > + efi <DIR> 2017-05-24 18:15 > + loader <DIR> 2017-05-24 18:15 > + startup nsh 26 2017-05-24 18:15 > + vmlinuz 6926384 2017-05-24 18:15 > + test wks 628 2017-05-24 21:22 > + 5 files 6 927 038 bytes > + 15 677 440 bytes free > + > + The second form of the command copies file or directory to the > specified directory > + on the partition: > + $ wic cp test > tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/efi/ > + $ wic ls > tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/efi/ > + Volume in drive : is boot > + Volume Serial Number is DB4C-FD4C > + Directory for ::/efi > + > + . <DIR> 2017-05-24 18:15 > + .. <DIR> 2017-05-24 18:15 > + boot <DIR> 2017-05-24 18:15 > + test <DIR> 2017-05-24 21:27 > + 4 files 0 bytes > + 15 675 392 bytes free > + > + The third form of the command copies file or directory from the > specified directory > + on the partition to local: > + $ wic cp > tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/vmlinuz > test + > + The -n option is used to specify the path to the native sysroot > + containing the tools(parted and mtools) to use. > +""" > + > +wic_rm_usage = """ > + > + Remove files or directories from the vfat or ext* partitions > + > + usage: wic rm <image>:<partition><path> [--native-sysroot <path>] > + > + This command removes files or directories from the vfat or ext* > partitions of > + the partitioned image. > + > + See 'wic help rm' for more detailed instructions. > + > +""" > + > +wic_rm_help = """ > + > +NAME > + wic rm - remove files or directories from the vfat or ext* > partitions + > +SYNOPSIS > + wic rm <src> <image>:<partition><path> > + wic rm <src> <image>:<partition><path> --native-sysroot <path> > + wic rm -r <image>:<partition><path> > + > +DESCRIPTION > + This command removes files or directories from the vfat or ext* > partition of the > + partitioned image: > + > + $ wic ls > ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 > + Volume in drive : is boot > + Volume Serial Number is 11D0-DE21 > + Directory for ::/ > + > + libcom32 c32 186500 2017-06-02 15:15 > + libutil c32 24148 2017-06-02 15:15 > + syslinux cfg 209 2017-06-02 15:15 > + vesamenu c32 27104 2017-06-02 15:15 > + vmlinuz 6926384 2017-06-02 15:15 > + 5 files 7 164 345 bytes > + 16 582 656 bytes free > + > + $ wic rm > ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/libutil.c32 > + > + $ wic ls > ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 > + Volume in drive : is boot > + Volume Serial Number is 11D0-DE21 > + Directory for ::/ > + > + libcom32 c32 186500 2017-06-02 15:15 > + syslinux cfg 209 2017-06-02 15:15 > + vesamenu c32 27104 2017-06-02 15:15 > + vmlinuz 6926384 2017-06-02 15:15 > + 4 files 7 140 197 bytes > + 16 607 232 bytes free > + > + The -n option is used to specify the path to the native sysroot > + containing the tools(parted and mtools) to use. > + > + The -r option is used to remove directories and their contents > + recursively,this only applies to ext* partition. > +""" > + > +wic_write_usage = """ > + > + Write image to a device > + > + usage: wic write <image> <target device> [--expand [rules]] > [--native-sysroot <path>] + > + This command writes partitioned image to a target device (USB > stick, SD card etc). + > + See 'wic help write' for more detailed instructions. > + > +""" > + > +wic_write_help = """ > + > +NAME > + wic write - write an image to a device > + > +SYNOPSIS > + wic write <image> <target> > + wic write <image> <target> --expand auto > + wic write <image> <target> --expand 1:100M,2:300M > + wic write <image> <target> --native-sysroot <path> > + > +DESCRIPTION > + This command writes an image to a target device (USB stick, SD > card etc) + > + $ wic write > ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic > /dev/sdb + > + The --expand option is used to resize image partitions. > + --expand auto expands partitions to occupy all free space > available on the target device. > + It's also possible to specify expansion rules in a format > + <partition>:<size>[,<partition>:<size>...] for one or more > partitions. > + Specifying size 0 will keep partition unmodified. > + Note: Resizing boot partition can result in non-bootable image > for non-EFI images. It is > + recommended to use size 0 for boot partition to keep image > bootable. + > + The --native-sysroot option is used to specify the path to the > native sysroot > + containing the tools(parted, resize2fs) to use. > +""" > + > wic_plugins_help = """ > > NAME > @@ -308,7 +536,8 @@ DESCRIPTION > > Source plugins can also be implemented and added by external > layers - any plugins found in a scripts/lib/wic/plugins/source/ > - directory in an external layer will also be made available. > + or lib/wic/plugins/source/ directory in an external layer will > + also be made available. > > When the wic implementation needs to invoke a partition-specific > implementation, it looks for the plugin that has the same name as > @@ -346,6 +575,10 @@ DESCRIPTION > partition. In other words, it 'prepares' the final > partition image which will be incorporated into the disk image. > > + do_post_partition() > + Called after the partition is created. It is useful to add > post > + operations e.g. signing the partition. > + > do_configure_partition() > Called before do_prepare_partition(), typically used to > create custom configuration files for a partition, for > @@ -632,8 +865,11 @@ DESCRIPTION > Partitions with a <mountpoint> specified will be > automatically mounted. This is achieved by wic adding entries to the > fstab during image generation. In order for a valid fstab to be > generated one of the > - --ondrive, --ondisk or --use-uuid partition options must be > used for > - each partition that specifies a mountpoint. > + --ondrive, --ondisk, --use-uuid or --use-label partition > options must > + be used for each partition that specifies a mountpoint. Note > that with > + --use-{uuid,label} and non-root <mountpoint>, including swap, > the mount > + program must understand the PARTUUID or LABEL syntax. This > currently > + excludes the busybox versions of these applications. > > > The following are supported 'part' options: > @@ -687,6 +923,8 @@ DESCRIPTION > apply to partitions created using '--source rootfs' (see > --source above). Valid values are: > > + vfat > + msdos > ext2 > ext3 > ext4 > @@ -706,6 +944,14 @@ DESCRIPTION > label is already in use by another > filesystem, a new label is created for the partition. > > + --use-label: This option is specific to wic. It makes wic > to use the > + label in /etc/fstab to specify a partition. If > the > + --use-label and --use-uuid are used at the > same time, > + we prefer the uuid because it is less likely > to cause > + name confliction. We don't support using this > parameter > + on the root partition since it requires an > initramfs to > + parse this value and we do not currently > support that. + > --active: Marks the partition as active. > > --align (in KBytes): This option is specific to wic and says > @@ -719,11 +965,31 @@ DESCRIPTION > bootloaders. > > --exclude-path: This option is specific to wic. It excludes > the given > - absolute path from the resulting image. If > the path > + relative path from the resulting image. If > the path ends with a slash, only the content of the directory > is omitted, not the directory itself. This > option only has an effect with the rootfs source plugin. > > + --include-path: This option is specific to wic. It adds the > contents > + of the given path or a rootfs to the > resulting image. > + The option contains two fields, the origin > and the > + destination. When the origin is a rootfs, > it follows > + the same logic as the rootfs-dir argument > and the > + permissions and owners are kept. When the > origin is a > + path, it is relative to the directory in > which wic is > + running not the rootfs itself so use of an > absolute > + path is recommended, and the owner and > group is set to > + root:root. If no destination is given it is > + automatically set to the root of the > rootfs. This > + option only has an effect with the rootfs > source > + plugin. > + > + --change-directory: This option is specific to wic. It > changes to the > + given directory before copying the > files. This > + option is useful when we want to split > a rootfs in > + multiple partitions and we want to keep > the right > + permissions and usernames in all the > partitions. + > --extra-space: This option is specific to wic. It adds extra > space after the space filled by the content > of the partition. The final size can go > @@ -738,6 +1004,8 @@ DESCRIPTION > This option cannot be used with > --fixed-size option. > > + --part-name: This option is specific to wic. It specifies > name for GPT partitions. + > --part-type: This option is specific to wic. It specifies > partition type GUID for GPT partitions. > List of partition type GUIDS can be found here: > @@ -752,10 +1020,21 @@ DESCRIPTION > in bootloader configuration before running wic. In > this case .wks file can be generated or modified to set preconfigured > parition UUID using this option. > + --fsuuid: This option is specific to wic. It specifies > filesystem UUID. > + It's useful if preconfigured filesystem UUID is > added to kernel command line > + in bootloader configuration before running wic. > In this case .wks file can > + be generated or modified to set preconfigured > filesystem UUID using this option. + > --system-id: This option is specific to wic. It specifies > partition system id. It's useful for the harware that requires > non-default partition system ids. The parameter in one byte long hex > number either with 0x prefix or without it. > + --mkfs-extraopts: This option specifies extra options to > pass to mkfs utility. > + NOTE, that wic uses default options for > some filesystems, for example > + '-S 512' for mkfs.fat or '-F -i 8192' for > mkfs.ext. Those options will > + not take effect when --mkfs-extraopts is > used. This should be taken into > + account when using --mkfs-extraopts. > + > * bootloader > > This command allows the user to specify various bootloader > @@ -793,3 +1072,67 @@ DESCRIPTION > .wks files. > > """ > + > +wic_help_help = """ > +NAME > + wic help - display a help topic > + > +DESCRIPTION > + Specify a help topic to display it. Topics are shown above. > +""" > + > + > +wic_help = """ > +Creates a customized OpenEmbedded image. > + > +Usage: wic [--version] > + wic help [COMMAND or TOPIC] > + wic COMMAND [ARGS] > + > + usage 1: Returns the current version of Wic > + usage 2: Returns detailed help for a COMMAND or TOPIC > + usage 3: Executes COMMAND > + > + > +COMMAND: > + > + list - List available canned images and source plugins > + ls - List contents of partitioned image or partition > + rm - Remove files or directories from the vfat or ext* > partitions > + help - Show help for a wic COMMAND or TOPIC > + write - Write an image to a device > + cp - Copy files and directories to the vfat or ext* > partitions > + create - Create a new OpenEmbedded image > + > + > +TOPIC: > + overview - Presents an overall overview of Wic > + plugins - Presents an overview and API for Wic plugins > + kickstart - Presents a Wic kicstart file reference > + > + > +Examples: > + > + $ wic --version > + > + Returns the current version of Wic > + > + > + $ wic help cp > + > + Returns the SYNOPSIS and DESCRIPTION for the Wic "cp" command. > + > + > + $ wic list images > + > + Returns the list of canned images (i.e. *.wks files located in > + the /scripts/lib/wic/canned-wks directory. > + > + > + $ wic create mkefidisk -e core-image-minimal > + > + Creates an EFI disk image from artifacts used in a previous > + core-image-minimal build in standard BitBake locations > + (e.g. Cooked Mode). > + > +""" > diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py > index a039300..3453d9c 100644 > --- a/scripts/lib/wic/ksparser.py > +++ b/scripts/lib/wic/ksparser.py > @@ -1,21 +1,8 @@ > -#!/usr/bin/env python -tt > -# ex:ts=4:sw=4:sts=4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > +#!/usr/bin/env python3 > # > # Copyright (c) 2016 Intel, Inc. > # > -# This program is free software; you can redistribute it and/or > modify it -# under the terms of the GNU General Public License as > published by the Free -# Software Foundation; version 2 of the License > -# > -# This program is distributed in the hope that it will be useful, but > -# WITHOUT ANY WARRANTY; without even the implied warranty of > MERCHANTABILITY -# or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > General Public License -# for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., 59 -# Temple Place - Suite 330, Boston, MA > 02111-1307, USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # DESCRIPTION > # This module provides parser for kickstart format > @@ -28,14 +15,30 @@ > import os > import shlex > import logging > +import re > > from argparse import ArgumentParser, ArgumentError, ArgumentTypeError > > from wic.engine import find_canned > from wic.partition import Partition > +from wic.misc import get_bitbake_var > > logger = logging.getLogger('wic') > > +__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}") > + > +def expand_line(line): > + while True: > + m = __expand_var_regexp__.search(line) > + if not m: > + return line > + key = m.group()[2:-1] > + val = get_bitbake_var(key) > + if val is None: > + logger.warning("cannot expand variable %s" % key) > + return line > + line = line[:m.start()] + val + line[m.end():] > + > class KickStartError(Exception): > """Custom exception.""" > pass > @@ -48,26 +51,31 @@ class KickStartParser(ArgumentParser): > def error(self, message): > raise ArgumentError(None, message) > > -def sizetype(arg): > - """ > - Custom type for ArgumentParser > - Converts size string in <num>[K|k|M|G] format into the integer > value > - """ > - if arg.isdigit(): > - return int(arg) * 1024 > - > - if not arg[:-1].isdigit(): > - raise ArgumentTypeError("Invalid size: %r" % arg) > +def sizetype(default): > + def f(arg): > + """ > + Custom type for ArgumentParser > + Converts size string in <num>[K|k|M|G] format into the > integer value > + """ > + try: > + suffix = default > + size = int(arg) > + except ValueError: > + try: > + suffix = arg[-1:] > + size = int(arg[:-1]) > + except ValueError: > + raise ArgumentTypeError("Invalid size: %r" % arg) > > - size = int(arg[:-1]) > - if arg.endswith("k") or arg.endswith("K"): > - return size > - if arg.endswith("M"): > - return size * 1024 > - if arg.endswith("G"): > - return size * 1024 * 1024 > + if suffix == "k" or suffix == "K": > + return size > + if suffix == "M": > + return size * 1024 > + if suffix == "G": > + return size * 1024 * 1024 > > - raise ArgumentTypeError("Invalid size: %r" % arg) > + raise ArgumentTypeError("Invalid size: %r" % arg) > + return f > > def overheadtype(arg): > """ > @@ -114,7 +122,7 @@ def systemidtype(arg): > return arg > > class KickStart(): > - """"Kickstart parser implementation.""" > + """Kickstart parser implementation.""" > > DEFAULT_EXTRA_SPACE = 10*1024 > DEFAULT_OVERHEAD_FACTOR = 1.3 > @@ -133,30 +141,41 @@ class KickStart(): > part.add_argument('mountpoint', nargs='?') > part.add_argument('--active', action='store_true') > part.add_argument('--align', type=int) > + part.add_argument('--offset', type=sizetype("K")) > part.add_argument('--exclude-path', nargs='+') > - part.add_argument("--extra-space", type=sizetype) > + part.add_argument('--include-path', nargs='+', > action='append') > + part.add_argument('--change-directory') > + part.add_argument("--extra-space", type=sizetype("M")) > part.add_argument('--fsoptions', dest='fsopts') > - part.add_argument('--fstype') > + part.add_argument('--fstype', default='vfat', > + choices=('ext2', 'ext3', 'ext4', 'btrfs', > + 'squashfs', 'vfat', 'msdos', > 'swap')) > + part.add_argument('--mkfs-extraopts', default='') > part.add_argument('--label') > + part.add_argument('--use-label', action='store_true') > part.add_argument('--no-table', action='store_true') > part.add_argument('--ondisk', '--ondrive', dest='disk', > default='sda') part.add_argument("--overhead-factor", > type=overheadtype) > + part.add_argument('--part-name') > part.add_argument('--part-type') > part.add_argument('--rootfs-dir') > + part.add_argument('--type', default='primary', > + choices = ('primary', 'logical')) > > # --size and --fixed-size cannot be specified together; > options # ----extra-space and --overhead-factor should also raise a > parser # --error, but since nesting mutually exclusive groups does > not work, # ----extra-space/--overhead-factor are handled later > sizeexcl = part.add_mutually_exclusive_group() > - sizeexcl.add_argument('--size', type=sizetype, default=0) > - sizeexcl.add_argument('--fixed-size', type=sizetype, > default=0) > + sizeexcl.add_argument('--size', type=sizetype("M"), > default=0) > + sizeexcl.add_argument('--fixed-size', type=sizetype("M"), > default=0) > part.add_argument('--source') > part.add_argument('--sourceparams') > part.add_argument('--system-id', type=systemidtype) > part.add_argument('--use-uuid', action='store_true') > part.add_argument('--uuid') > + part.add_argument('--fsuuid') > > bootloader = subparsers.add_parser('bootloader') > bootloader.add_argument('--append') > @@ -184,6 +203,7 @@ class KickStart(): > line = line.strip() > lineno += 1 > if line and line[0] != '#': > + line = expand_line(line) > try: > line_args = shlex.split(line) > parsed = parser.parse_args(line_args) > @@ -191,6 +211,20 @@ class KickStart(): > raise KickStartError('%s:%d: %s' % \ > (confpath, lineno, err)) > if line.startswith('part'): > + # SquashFS does not support filesystem UUID > + if parsed.fstype == 'squashfs': > + if parsed.fsuuid: > + err = "%s:%d: SquashFS does not > support UUID" \ > + % (confpath, lineno) > + raise KickStartError(err) > + if parsed.label: > + err = "%s:%d: SquashFS does not > support LABEL" \ > + % (confpath, lineno) > + raise KickStartError(err) > + if parsed.use_label and not parsed.label: > + err = "%s:%d: Must set the label with > --label" \ > + % (confpath, lineno) > + raise KickStartError(err) > # using ArgumentParser one cannot easily > tell if option # was passed as argument, if said option has a default > # value; --overhead-factor/--extra-space > cannot be used @@ -219,6 +253,11 @@ class KickStart(): > elif line.startswith('bootloader'): > if not self.bootloader: > self.bootloader = parsed > + # Concatenate the strings set in APPEND > + append_var = get_bitbake_var("APPEND") > + if append_var: > + self.bootloader.append = ' > '.join(filter(None, \ > + > (self.bootloader.append, append_var))) else: > err = "%s:%d: more than one bootloader > specified" \ % (confpath, lineno) > diff --git a/scripts/lib/wic/utils/misc.py b/scripts/lib/wic/misc.py > similarity index 70% > rename from scripts/lib/wic/utils/misc.py > rename to scripts/lib/wic/misc.py > index c941112..91975ba 100644 > --- a/scripts/lib/wic/utils/misc.py > +++ b/scripts/lib/wic/misc.py > @@ -1,21 +1,7 @@ > -# ex:ts=4:sw=4:sts=4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Copyright (c) 2013, Intel Corporation. > -# All rights reserved. > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # DESCRIPTION > # This module provides a place to collect various wic-related utils > @@ -29,12 +15,12 @@ > import logging > import os > import re > +import subprocess > > from collections import defaultdict > from distutils import spawn > > from wic import WicError > -from wic.utils import runner > > logger = logging.getLogger('wic') > > @@ -43,6 +29,9 @@ NATIVE_RECIPES = {"bmaptool": "bmap-tools", > "grub-mkimage": "grub-efi", > "isohybrid": "syslinux", > "mcopy": "mtools", > + "mdel" : "mtools", > + "mdeltree" : "mtools", > + "mdir" : "mtools", > "mkdosfs": "dosfstools", > "mkisofs": "cdrtools", > "mkfs.btrfs": "btrfs-tools", > @@ -52,14 +41,48 @@ NATIVE_RECIPES = {"bmaptool": "bmap-tools", > "mkfs.vfat": "dosfstools", > "mksquashfs": "squashfs-tools", > "mkswap": "util-linux", > - "mmd": "syslinux", > + "mmd": "mtools", > "parted": "parted", > "sfdisk": "util-linux", > "sgdisk": "gptfdisk", > - "syslinux": "syslinux" > + "syslinux": "syslinux", > + "tar": "tar" > } > > -def _exec_cmd(cmd_and_args, as_shell=False, catch=3): > +def runtool(cmdln_or_args): > + """ wrapper for most of the subprocess calls > + input: > + cmdln_or_args: can be both args and cmdln str (shell=True) > + return: > + rc, output > + """ > + if isinstance(cmdln_or_args, list): > + cmd = cmdln_or_args[0] > + shell = False > + else: > + import shlex > + cmd = shlex.split(cmdln_or_args)[0] > + shell = True > + > + sout = subprocess.PIPE > + serr = subprocess.STDOUT > + > + try: > + process = subprocess.Popen(cmdln_or_args, stdout=sout, > + stderr=serr, shell=shell) > + sout, serr = process.communicate() > + # combine stdout and stderr, filter None out and decode > + out = ''.join([out.decode('utf-8') for out in [sout, serr] > if out]) > + except OSError as err: > + if err.errno == 2: > + # [Errno 2] No such file or directory > + raise WicError('Cannot run command: %s, lost > dependency?' % cmd) > + else: > + raise # relay > + > + return process.returncode, out > + > +def _exec_cmd(cmd_and_args, as_shell=False): > """ > Execute command, catching stderr, stdout > > @@ -70,9 +93,9 @@ def _exec_cmd(cmd_and_args, as_shell=False, > catch=3): logger.debug(args) > > if as_shell: > - ret, out = runner.runtool(cmd_and_args, catch) > + ret, out = runtool(cmd_and_args) > else: > - ret, out = runner.runtool(args, catch) > + ret, out = runtool(args) > out = out.strip() > if ret != 0: > raise WicError("_exec_cmd: %s returned '%s' instead of > 0\noutput: %s" % \ @@ -84,14 +107,23 @@ def _exec_cmd(cmd_and_args, > as_shell=False, catch=3): return ret, out > > > -def exec_cmd(cmd_and_args, as_shell=False, catch=3): > +def exec_cmd(cmd_and_args, as_shell=False): > """ > Execute command, return output > """ > - return _exec_cmd(cmd_and_args, as_shell, catch)[1] > + return _exec_cmd(cmd_and_args, as_shell)[1] > + > +def find_executable(cmd, paths): > + recipe = cmd > + if recipe in NATIVE_RECIPES: > + recipe = NATIVE_RECIPES[recipe] > + provided = get_bitbake_var("ASSUME_PROVIDED") > + if provided and "%s-native" % recipe in provided: > + return True > > + return spawn.find_executable(cmd, paths) > > -def exec_native_cmd(cmd_and_args, native_sysroot, catch=3, > pseudo=""): +def exec_native_cmd(cmd_and_args, native_sysroot, > pseudo=""): """ > Execute native command, catching stderr, stdout > > @@ -106,19 +138,16 @@ def exec_native_cmd(cmd_and_args, > native_sysroot, catch=3, pseudo=""): if pseudo: > cmd_and_args = pseudo + cmd_and_args > > - wtools_sysroot = get_bitbake_var("RECIPE_SYSROOT_NATIVE", > "wic-tools") > + native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin" % \ > + (native_sysroot, native_sysroot, native_sysroot) > > - native_paths = \ > - > "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/sbin:%s/usr/sbin:%s/usr/bin" % \ > - (wtools_sysroot, wtools_sysroot, wtools_sysroot, > - native_sysroot, native_sysroot, native_sysroot) > native_cmd_and_args = "export PATH=%s:$PATH;%s" % \ > - (native_paths, cmd_and_args) > + (native_paths, cmd_and_args) > logger.debug("exec_native_cmd: %s", native_cmd_and_args) > > # If the command isn't in the native sysroot say we failed. > - if spawn.find_executable(args[0], native_paths): > - ret, out = _exec_cmd(native_cmd_and_args, True, catch) > + if find_executable(args[0], native_paths): > + ret, out = _exec_cmd(native_cmd_and_args, True) > else: > ret = 127 > out = "can't find native executable %s in %s" % (args[0], > native_paths) @@ -131,8 +160,8 @@ def exec_native_cmd(cmd_and_args, > native_sysroot, catch=3, pseudo=""): "was not found (see details > above).\n\n" % prog recipe = NATIVE_RECIPES.get(prog) > if recipe: > - msg += "Please bake it with 'bitbake %s-native' "\ > - "and try again.\n" % recipe > + msg += "Please make sure wic-tools have %s-native in its > DEPENDS, "\ > + "build it with 'bitbake wic-tools' and try > again.\n" % recipe else: > msg += "Wic failed to find a recipe to build native %s. > Please "\ "file a bug against wic.\n" % prog > @@ -153,7 +182,7 @@ class BitbakeVars(defaultdict): > self.default_image = None > self.vars_dir = None > > - def _parse_line(self, line, image, > matcher=re.compile(r"^(\w+)=(.+)")): > + def _parse_line(self, line, image, > matcher=re.compile(r"^([a-zA-Z0-9\-_+./~]+)=(.*)")): """ > Parse one line from bitbake -e output or from .env file. > Put result key-value pair into the storage. > diff --git a/scripts/lib/wic/partition.py > b/scripts/lib/wic/partition.py index 8e32afc..85eb15c 100644 > --- a/scripts/lib/wic/partition.py > +++ b/scripts/lib/wic/partition.py > @@ -1,21 +1,7 @@ > -# ex:ts=4:sw=4:sts=4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Copyright (c) 2013-2016 Intel Corporation. > -# All rights reserved. > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # DESCRIPTION > # This module provides the OpenEmbedded partition object definitions. > @@ -26,10 +12,10 @@ > > import logging > import os > -import tempfile > +import uuid > > from wic import WicError > -from wic.utils.misc import exec_cmd, exec_native_cmd, get_bitbake_var > +from wic.misc import exec_cmd, exec_native_cmd, get_bitbake_var > from wic.pluginbase import PluginMgr > > logger = logging.getLogger('wic') > @@ -44,13 +30,19 @@ class Partition(): > self.device = None > self.extra_space = args.extra_space > self.exclude_path = args.exclude_path > + self.include_path = args.include_path > + self.change_directory = args.change_directory > self.fsopts = args.fsopts > self.fstype = args.fstype > self.label = args.label > + self.use_label = args.use_label > + self.mkfs_extraopts = args.mkfs_extraopts > self.mountpoint = args.mountpoint > self.no_table = args.no_table > self.num = None > + self.offset = args.offset > self.overhead_factor = args.overhead_factor > + self.part_name = args.part_name > self.part_type = args.part_type > self.rootfs_dir = args.rootfs_dir > self.size = args.size > @@ -60,10 +52,11 @@ class Partition(): > self.system_id = args.system_id > self.use_uuid = args.use_uuid > self.uuid = args.uuid > + self.fsuuid = args.fsuuid > + self.type = args.type > > self.lineno = lineno > self.source_file = "" > - self.sourceparams_dict = {} > > def get_extra_block_count(self, current_blocks): > """ > @@ -136,22 +129,24 @@ class Partition(): > "specify a non-zero > --size/--fixed-size for that " "partition." % self.mountpoint) > > - if self.fstype and self.fstype == "swap": > + if self.fstype == "swap": > self.prepare_swap_partition(cr_workdir, oe_builddir, > native_sysroot) > self.source_file = "%s/fs.%s" % (cr_workdir, > self.fstype) > - elif self.fstype: > + else: > + if self.fstype == 'squashfs': > + raise WicError("It's not possible to create > empty squashfs " > + "partition '%s'" % > (self.mountpoint)) + > rootfs = "%s/fs_%s.%s.%s" % (cr_workdir, self.label, > self.lineno, > self.fstype) if os.path.isfile(rootfs): > os.remove(rootfs) > - for prefix in ("ext", "btrfs", "vfat", "squashfs"): > - if self.fstype.startswith(prefix): > - method = getattr(self, > - "prepare_empty_partition_" > + prefix) > - method(rootfs, oe_builddir, native_sysroot) > - self.source_file = rootfs > - break > + > + prefix = "ext" if self.fstype.startswith("ext") else > self.fstype > + method = getattr(self, "prepare_empty_partition_" + > prefix) > + method(rootfs, oe_builddir, native_sysroot) > + self.source_file = rootfs > return > > plugins = PluginMgr.get_plugins('source') > @@ -168,7 +163,7 @@ class Partition(): > # Split sourceparams string of the form > key1=val1[,key2=val2,...] # into a dict. Also accepts valueless keys > i.e. without = splitted = self.sourceparams.split(',') > - srcparams_dict = dict(par.split('=') for par in splitted > if par) > + srcparams_dict = dict(par.split('=', 1) for par in > splitted if par) > plugin = PluginMgr.get_plugins('source')[self.source] > plugin.do_configure_partition(self, srcparams_dict, creator, > @@ -180,6 +175,9 @@ class Partition(): > plugin.do_prepare_partition(self, srcparams_dict, creator, > cr_workdir, oe_builddir, > bootimg_dir, kernel_dir, rootfs_dir, native_sysroot) > + plugin.do_post_partition(self, srcparams_dict, creator, > + cr_workdir, oe_builddir, > bootimg_dir, > + kernel_dir, rootfs_dir, > native_sysroot) > # further processing required Partition.size to be an > integer, make # sure that it is one > @@ -193,74 +191,57 @@ class Partition(): > "larger (%d kB) than its allowed size %d > kB" % (self.mountpoint, self.size, self.fixed_size)) > > - def prepare_rootfs_from_fs_image(self, cr_workdir, oe_builddir, > - rootfs_dir): > - """ > - Handle an already-created partition e.g. xxx.ext3 > - """ > - rootfs = oe_builddir > - du_cmd = "du -Lbks %s" % rootfs > - out = exec_cmd(du_cmd) > - rootfs_size = out.split()[0] > - > - self.size = int(rootfs_size) > - self.source_file = rootfs > - > def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir, > - native_sysroot): > + native_sysroot, real_rootfs = True, > pseudo_dir = None): """ > Prepare content for a rootfs partition i.e. create a > partition and fill it from a /rootfs dir. > > - Currently handles ext2/3/4, btrfs and vfat. > + Currently handles ext2/3/4, btrfs, vfat and squashfs. > """ > p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % > native_sysroot) > - p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", > - "%s/../pseudo" % rootfs_dir) > - p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir) > - p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1") > - pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix > - pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir > - pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd > - pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp > - pseudo += "%s " % get_bitbake_var("FAKEROOTCMD") > + if (pseudo_dir): > + pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix > + pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir > + pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir > + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" > + pseudo += "%s " % get_bitbake_var("FAKEROOTCMD") > + else: > + pseudo = None > > rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label, > self.lineno, self.fstype) > if os.path.isfile(rootfs): > os.remove(rootfs) > > - if not self.fstype: > - raise WicError("File system for partition %s not > specified in " > - "kickstart, use --fstype option" % > self.mountpoint) - > - # Get rootfs size from bitbake variable if it's not set in > .ks file > - if not self.size: > - # Bitbake variable ROOTFS_SIZE is calculated in > - # Image._get_rootfs_size method from meta/lib/oe/image.py > - # using IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT, > - # IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE > + if not self.size and real_rootfs: > + # The rootfs size is not set in .ks file so try to get it > + # from bitbake variable > rsize_bb = get_bitbake_var('ROOTFS_SIZE') > - if rsize_bb: > - logger.warning('overhead-factor was specified, but > size was not,' > - ' so bitbake variables will be used > for the size.' > - ' In this case both > IMAGE_OVERHEAD_FACTOR and ' > - '--overhead-factor will be applied') > + rdir = get_bitbake_var('IMAGE_ROOTFS') > + if rsize_bb and rdir == rootfs_dir: > + # Bitbake variable ROOTFS_SIZE is calculated in > + # Image._get_rootfs_size method from > meta/lib/oe/image.py > + # using IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT, > + # IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE > self.size = int(round(float(rsize_bb))) > - > - for prefix in ("ext", "btrfs", "vfat", "squashfs"): > - if self.fstype.startswith(prefix): > - method = getattr(self, "prepare_rootfs_" + prefix) > - method(rootfs, oe_builddir, rootfs_dir, > native_sysroot, pseudo) - > - self.source_file = rootfs > - > - # get the rootfs size in the right units for > kickstart (kB) > - du_cmd = "du -Lbks %s" % rootfs > + else: > + # Bitbake variable ROOTFS_SIZE is not defined so > compute it > + # from the rootfs_dir size using the same logic > found in > + # get_rootfs_size() from meta/classes/image.bbclass > + du_cmd = "du -ks %s" % rootfs_dir > out = exec_cmd(du_cmd) > self.size = int(out.split()[0]) > > - break > + prefix = "ext" if self.fstype.startswith("ext") else > self.fstype > + method = getattr(self, "prepare_rootfs_" + prefix) > + method(rootfs, oe_builddir, rootfs_dir, native_sysroot, > pseudo) > + self.source_file = rootfs > + > + # get the rootfs size in the right units for kickstart (kB) > + du_cmd = "du -Lbks %s" % rootfs > + out = exec_cmd(du_cmd) > + self.size = int(out.split()[0]) > > def prepare_rootfs_ext(self, rootfs, oe_builddir, rootfs_dir, > native_sysroot, pseudo): > @@ -276,25 +257,23 @@ class Partition(): > with open(rootfs, 'w') as sparse: > os.ftruncate(sparse.fileno(), rootfs_size * 1024) > > - extra_imagecmd = "-i 8192" > + extraopts = self.mkfs_extraopts or "-F -i 8192" > > label_str = "" > if self.label: > label_str = "-L %s" % self.label > > - mkfs_cmd = "mkfs.%s -F %s %s %s -d %s" % \ > - (self.fstype, extra_imagecmd, rootfs, label_str, > rootfs_dir) > + mkfs_cmd = "mkfs.%s %s %s %s -U %s -d %s" % \ > + (self.fstype, extraopts, rootfs, label_str, self.fsuuid, > rootfs_dir) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) > > - mkfs_cmd = "fsck.%s -fy %s" % (self.fstype, rootfs) > + mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs) > exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) > > def prepare_rootfs_btrfs(self, rootfs, oe_builddir, rootfs_dir, > native_sysroot, pseudo): > """ > Prepare content for a btrfs rootfs partition. > - > - Currently handles ext2/3/4 and btrfs. > """ > du_cmd = "du -ks %s" % rootfs_dir > out = exec_cmd(du_cmd) > @@ -309,14 +288,15 @@ class Partition(): > if self.label: > label_str = "-L %s" % self.label > > - mkfs_cmd = "mkfs.%s -b %d -r %s %s %s" % \ > - (self.fstype, rootfs_size * 1024, rootfs_dir, label_str, > rootfs) > + mkfs_cmd = "mkfs.%s -b %d -r %s %s %s -U %s %s" % \ > + (self.fstype, rootfs_size * 1024, rootfs_dir, label_str, > + self.mkfs_extraopts, self.fsuuid, rootfs) > exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) > > - def prepare_rootfs_vfat(self, rootfs, oe_builddir, rootfs_dir, > - native_sysroot, pseudo): > + def prepare_rootfs_msdos(self, rootfs, oe_builddir, rootfs_dir, > + native_sysroot, pseudo): > """ > - Prepare content for a vfat rootfs partition. > + Prepare content for a msdos/vfat rootfs partition. > """ > du_cmd = "du -bks %s" % rootfs_dir > out = exec_cmd(du_cmd) > @@ -328,7 +308,15 @@ class Partition(): > if self.label: > label_str = "-n %s" % self.label > > - dosfs_cmd = "mkdosfs %s -S 512 -C %s %d" % (label_str, > rootfs, rootfs_size) > + size_str = "" > + if self.fstype == 'msdos': > + size_str = "-F 16" # FAT 16 > + > + extraopts = self.mkfs_extraopts or '-S 512' > + > + dosfs_cmd = "mkdosfs %s -i %s %s %s -C %s %d" % \ > + (label_str, self.fsuuid, size_str, extraopts, > rootfs, > + rootfs_size) > exec_native_cmd(dosfs_cmd, native_sysroot) > > mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir) > @@ -337,13 +325,16 @@ class Partition(): > chmod_cmd = "chmod 644 %s" % rootfs > exec_cmd(chmod_cmd) > > + prepare_rootfs_vfat = prepare_rootfs_msdos > + > def prepare_rootfs_squashfs(self, rootfs, oe_builddir, > rootfs_dir, native_sysroot, pseudo): > """ > Prepare content for a squashfs rootfs partition. > """ > - squashfs_cmd = "mksquashfs %s %s -noappend" % \ > - (rootfs_dir, rootfs) > + extraopts = self.mkfs_extraopts or '-noappend' > + squashfs_cmd = "mksquashfs %s %s %s" % \ > + (rootfs_dir, rootfs, extraopts) > exec_native_cmd(squashfs_cmd, native_sysroot, pseudo=pseudo) > > def prepare_empty_partition_ext(self, rootfs, oe_builddir, > @@ -355,14 +346,14 @@ class Partition(): > with open(rootfs, 'w') as sparse: > os.ftruncate(sparse.fileno(), size * 1024) > > - extra_imagecmd = "-i 8192" > + extraopts = self.mkfs_extraopts or "-i 8192" > > label_str = "" > if self.label: > label_str = "-L %s" % self.label > > - mkfs_cmd = "mkfs.%s -F %s %s %s" % \ > - (self.fstype, extra_imagecmd, label_str, rootfs) > + mkfs_cmd = "mkfs.%s -F %s %s -U %s %s" % \ > + (self.fstype, extraopts, label_str, self.fsuuid, rootfs) > exec_native_cmd(mkfs_cmd, native_sysroot) > > def prepare_empty_partition_btrfs(self, rootfs, oe_builddir, > @@ -378,12 +369,13 @@ class Partition(): > if self.label: > label_str = "-L %s" % self.label > > - mkfs_cmd = "mkfs.%s -b %d %s %s" % \ > - (self.fstype, self.size * 1024, label_str, rootfs) > + mkfs_cmd = "mkfs.%s -b %d %s -U %s %s %s" % \ > + (self.fstype, self.size * 1024, label_str, > self.fsuuid, > + self.mkfs_extraopts, rootfs) > exec_native_cmd(mkfs_cmd, native_sysroot) > > - def prepare_empty_partition_vfat(self, rootfs, oe_builddir, > - native_sysroot): > + def prepare_empty_partition_msdos(self, rootfs, oe_builddir, > + native_sysroot): > """ > Prepare an empty vfat partition. > """ > @@ -393,40 +385,22 @@ class Partition(): > if self.label: > label_str = "-n %s" % self.label > > - dosfs_cmd = "mkdosfs %s -S 512 -C %s %d" % (label_str, > rootfs, blocks) > - exec_native_cmd(dosfs_cmd, native_sysroot) > + size_str = "" > + if self.fstype == 'msdos': > + size_str = "-F 16" # FAT 16 > > - chmod_cmd = "chmod 644 %s" % rootfs > - exec_cmd(chmod_cmd) > + extraopts = self.mkfs_extraopts or '-S 512' > > - def prepare_empty_partition_squashfs(self, cr_workdir, > oe_builddir, > - native_sysroot): > - """ > - Prepare an empty squashfs partition. > - """ > - logger.warning("Creating of an empty squashfs %s partition > was attempted. " > - "Proceeding as requested.", self.mountpoint) > - > - path = "%s/fs_%s.%s" % (cr_workdir, self.label, self.fstype) > - if os.path.isfile(path): > - os.remove(path) > + dosfs_cmd = "mkdosfs %s -i %s %s %s -C %s %d" % \ > + (label_str, self.fsuuid, extraopts, size_str, > rootfs, > + blocks) > > - # it is not possible to create a squashfs without source > data, > - # thus prepare an empty temp dir that is used as source > - tmpdir = tempfile.mkdtemp() > - > - squashfs_cmd = "mksquashfs %s %s -noappend" % \ > - (tmpdir, path) > - exec_native_cmd(squashfs_cmd, native_sysroot) > - > - os.rmdir(tmpdir) > + exec_native_cmd(dosfs_cmd, native_sysroot) > > - # get the rootfs size in the right units for kickstart (kB) > - du_cmd = "du -Lbks %s" % path > - out = exec_cmd(du_cmd) > - fs_size = out.split()[0] > + chmod_cmd = "chmod 644 %s" % rootfs > + exec_cmd(chmod_cmd) > > - self.size = int(fs_size) > + prepare_empty_partition_vfat = prepare_empty_partition_msdos > > def prepare_swap_partition(self, cr_workdir, oe_builddir, > native_sysroot): """ > @@ -437,9 +411,9 @@ class Partition(): > with open(path, 'w') as sparse: > os.ftruncate(sparse.fileno(), self.size * 1024) > > - import uuid > label_str = "" > if self.label: > label_str = "-L %s" % self.label > - mkswap_cmd = "mkswap %s -U %s %s" % (label_str, > str(uuid.uuid1()), path) + > + mkswap_cmd = "mkswap %s -U %s %s" % (label_str, self.fsuuid, > path) exec_native_cmd(mkswap_cmd, native_sysroot) > diff --git a/scripts/lib/wic/pluginbase.py > b/scripts/lib/wic/pluginbase.py index fb3d179..d9b4e57 100644 > --- a/scripts/lib/wic/pluginbase.py > +++ b/scripts/lib/wic/pluginbase.py > @@ -1,19 +1,9 @@ > -#!/usr/bin/env python -tt > +#!/usr/bin/env python3 > # > # Copyright (c) 2011 Intel, Inc. > # > -# This program is free software; you can redistribute it and/or > modify it -# under the terms of the GNU General Public License as > published by the Free -# Software Foundation; version 2 of the License > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, but > -# WITHOUT ANY WARRANTY; without even the implied warranty of > MERCHANTABILITY -# or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > General Public License -# for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., 59 -# Temple Place - Suite 330, Boston, MA > 02111-1307, USA. > __all__ = ['ImagerPlugin', 'SourcePlugin'] > > @@ -24,11 +14,11 @@ from collections import defaultdict > from importlib.machinery import SourceFileLoader > > from wic import WicError > -from wic.utils.misc import get_bitbake_var > +from wic.misc import get_bitbake_var > > PLUGIN_TYPES = ["imager", "source"] > > -SCRIPTS_PLUGIN_DIR = "scripts/lib/wic/plugins" > +SCRIPTS_PLUGIN_DIR = ["scripts/lib/wic/plugins", "lib/wic/plugins"] > > logger = logging.getLogger('wic') > > @@ -48,10 +38,11 @@ class PluginMgr: > cls._plugin_dirs = > [os.path.join(os.path.dirname(__file__), 'plugins')] layers = > get_bitbake_var("BBLAYERS") or '' for layer_path in layers.split(): > - path = os.path.join(layer_path, SCRIPTS_PLUGIN_DIR) > - path = os.path.abspath(os.path.expanduser(path)) > - if path not in cls._plugin_dirs and > os.path.isdir(path): > - cls._plugin_dirs.insert(0, path) > + for script_plugin_dir in SCRIPTS_PLUGIN_DIR: > + path = os.path.join(layer_path, > script_plugin_dir) > + path = os.path.abspath(os.path.expanduser(path)) > + if path not in cls._plugin_dirs and > os.path.isdir(path): > + cls._plugin_dirs.insert(0, path) > > if ptype not in PLUGINS: > # load all ptype plugins > @@ -138,3 +129,12 @@ class SourcePlugin(metaclass=PluginMeta): > """ > logger.debug("SourcePlugin: do_prepare_partition: part: %s", > part) > + @classmethod > + def do_post_partition(cls, part, source_params, creator, > cr_workdir, > + oe_builddir, bootimg_dir, kernel_dir, > rootfs_dir, > + native_sysroot): > + """ > + Called after the partition is created. It is useful to add > post > + operations e.g. security signing the partition. > + """ > + logger.debug("SourcePlugin: do_post_partition: part: %s", > part) diff --git a/scripts/lib/wic/plugins/imager/direct.py > b/scripts/lib/wic/plugins/imager/direct.py index 7d38ab3..2f01999 > 100644 --- a/scripts/lib/wic/plugins/imager/direct.py > +++ b/scripts/lib/wic/plugins/imager/direct.py > @@ -1,21 +1,7 @@ > -# ex:ts=4:sw=4:sts=4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Copyright (c) 2013, Intel Corporation. > -# All rights reserved. > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # DESCRIPTION > # This implements the 'direct' imager plugin class for 'wic' > @@ -26,17 +12,20 @@ > > import logging > import os > +import random > import shutil > import tempfile > import uuid > > from time import strftime > > +from oe.path import copyhardlinktree > + > from wic import WicError > from wic.filemap import sparse_copy > from wic.ksparser import KickStart, KickStartError > from wic.pluginbase import PluginMgr, ImagerPlugin > -from wic.utils.misc import get_bitbake_var, exec_cmd, exec_native_cmd > +from wic.misc import get_bitbake_var, exec_cmd, exec_native_cmd > > logger = logging.getLogger('wic') > > @@ -68,6 +57,8 @@ class DirectPlugin(ImagerPlugin): > self.outdir = options.outdir > self.compressor = options.compressor > self.bmap = options.bmap > + self.no_fstab_update = options.no_fstab_update > + self.original_fstab = None > > self.name = "%s-%s" % > (os.path.splitext(os.path.basename(wks_file))[0], > strftime("%Y%m%d%H%M")) @@ -113,26 +104,38 @@ class > DirectPlugin(ImagerPlugin): > with open(fstab_path) as fstab: > fstab_lines = fstab.readlines() > + self.original_fstab = fstab_lines.copy() > > if self._update_fstab(fstab_lines, self.parts): > - shutil.copyfile(fstab_path, fstab_path + ".orig") > - > with open(fstab_path, "w") as fstab: > fstab.writelines(fstab_lines) > - > - return fstab_path > + else: > + self.original_fstab = None > > def _update_fstab(self, fstab_lines, parts): > """Assume partition order same as in wks""" > updated = False > for part in parts: > if not part.realnum or not part.mountpoint \ > - or part.mountpoint in ("/", "/boot"): > + or part.mountpoint == "/": > continue > > - # mmc device partitions are named mmcblk0p1, mmcblk0p2.. > - prefix = 'p' if part.disk.startswith('mmcblk') else '' > - device_name = "/dev/%s%s%d" % (part.disk, prefix, > part.realnum) > + if part.use_uuid: > + if part.fsuuid: > + # FAT UUID is different from others > + if len(part.fsuuid) == 10: > + device_name = "UUID=%s-%s" % \ > + (part.fsuuid[2:6], > part.fsuuid[6:]) > + else: > + device_name = "UUID=%s" % part.fsuuid > + else: > + device_name = "PARTUUID=%s" % part.uuid > + elif part.use_label: > + device_name = "LABEL=%s" % part.label > + else: > + # mmc device partitions are named mmcblk0p1, > mmcblk0p2.. > + prefix = 'p' if part.disk.startswith('mmcblk') else > '' > + device_name = "/dev/%s%s%d" % (part.disk, prefix, > part.realnum) > opts = part.fsopts if part.fsopts else "defaults" > line = "\t".join([device_name, part.mountpoint, > part.fstype, @@ -156,7 +159,8 @@ class DirectPlugin(ImagerPlugin): > filesystems from the artifacts directly and combine them into > a partitioned image. > """ > - fstab_path = > self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR")) > + if not self.no_fstab_update: > + self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR")) > > for part in self.parts: > # get rootfs size from bitbake variable if it's not set > in .ks file @@ -173,10 +177,6 @@ class DirectPlugin(ImagerPlugin): > part.size = int(round(float(rsize_bb))) > > self._image.prepare(self) > - > - if fstab_path: > - shutil.move(fstab_path + ".orig", fstab_path) > - > self._image.layout_partitions() > self._image.create() > > @@ -205,8 +205,10 @@ class DirectPlugin(ImagerPlugin): > # Generate .bmap > if self.bmap: > logger.debug("Generating bmap file for %s", disk_name) > - exec_native_cmd("bmaptool create %s -o %s.bmap" % > (full_path, full_path), > - self.native_sysroot) > + python = os.path.join(self.native_sysroot, > 'usr/bin/python3-native/python3') > + bmaptool = os.path.join(self.native_sysroot, > 'usr/bin/bmaptool') > + exec_native_cmd("%s %s create %s -o %s.bmap" % \ > + (python, bmaptool, full_path, > full_path), self.native_sysroot) # Compress the image > if self.compressor: > logger.debug("Compressing disk %s with %s", disk_name, > self.compressor) @@ -233,7 +235,8 @@ class DirectPlugin(ImagerPlugin): > suffix = ':' > else: > suffix = '["%s"]:' % (part.mountpoint or part.label) > - msg += ' ROOTFS_DIR%s%s\n' % (suffix.ljust(20), > part.rootfs_dir) > + rootdir = part.rootfs_dir > + msg += ' ROOTFS_DIR%s%s\n' % (suffix.ljust(20), rootdir) > > msg += ' BOOTIMG_DIR: %s\n' % > self.bootimg_dir msg += ' KERNEL_DIR: %s\n' % > self.kernel_dir @@ -270,6 +273,12 @@ class DirectPlugin(ImagerPlugin): > if os.path.isfile(path): > shutil.move(path, os.path.join(self.outdir, fname)) > > + #Restore original fstab > + if self.original_fstab: > + fstab_path = self.rootfs_dir.get("ROOTFS_DIR") + > "/etc/fstab" > + with open(fstab_path, "w") as fstab: > + fstab.writelines(self.original_fstab) > + > # remove work directory > shutil.rmtree(self.workdir, ignore_errors=True) > > @@ -291,18 +300,23 @@ class PartitionedImage(): > self.path = path # Path to the image file > self.numpart = 0 # Number of allocated partitions > self.realpart = 0 # Number of partitions in the partition > table > + self.primary_part_num = 0 # Number of primary partitions > (msdos) > + self.extendedpart = 0 # Create extended partition > before this logical partition (msdos) > + self.extended_size_sec = 0 # Size of exteded partition > (msdos) > + self.logical_part_cnt = 0 # Number of total logical > paritions (msdos) self.offset = 0 # Offset of next partition (in > sectors) self.min_size = 0 # Minimum required disk size to fit > # all partitions (in bytes) > self.ptable_format = ptable_format # Partition table format > # Disk system identifier > - self.identifier = int.from_bytes(os.urandom(4), 'little') > + self.identifier = random.SystemRandom().randint(1, > 0xffffffff) > self.partitions = partitions > self.partimages = [] > # Size of a sector used in calculations > self.sector_size = SECTOR_SIZE > self.native_sysroot = native_sysroot > + num_real_partitions = len([p for p in self.partitions if not > p.no_table]) > # calculate the real partition number, accounting for > partitions not # in the partition table and logical partitions > @@ -312,18 +326,23 @@ class PartitionedImage(): > part.realnum = 0 > else: > realnum += 1 > - if self.ptable_format == 'msdos' and realnum > 3: > + if self.ptable_format == 'msdos' and realnum > 3 and > num_real_partitions > 4: part.realnum = realnum + 1 > continue > part.realnum = realnum > > - # generate parition UUIDs > + # generate parition and filesystem UUIDs > for part in self.partitions: > if not part.uuid and part.use_uuid: > if self.ptable_format == 'gpt': > part.uuid = str(uuid.uuid4()) > else: # msdos partition table > - part.uuid = '%0x-%02d' % (self.identifier, > part.realnum) > + part.uuid = '%08x-%02d' % (self.identifier, > part.realnum) > + if not part.fsuuid: > + if part.fstype == 'vfat' or part.fstype == 'msdos': > + part.fsuuid = '0x' + > str(uuid.uuid4())[:8].upper() > + else: > + part.fsuuid = str(uuid.uuid4()) > > def prepare(self, imager): > """Prepare an image. Call prepare method of all image > partitions.""" @@ -352,6 +371,10 @@ class PartitionedImage(): > for num in range(len(self.partitions)): > part = self.partitions[num] > > + if self.ptable_format == 'msdos' and part.part_name: > + raise WicError("setting custom partition name is not > " \ > + "implemented for msdos partitions") > + > if self.ptable_format == 'msdos' and part.part_type: > # The --part-type can also be implemented for MBR > partitions, # in which case it would map to the 1-byte "partition > type" @@ -373,12 +396,16 @@ class PartitionedImage(): > # Skip one sector required for the partitioning > scheme overhead self.offset += overhead > > - if self.realpart > 3 and num_real_partitions > 4: > + if self.ptable_format == "msdos": > + if self.primary_part_num > 3 or \ > + (self.extendedpart == 0 and self.primary_part_num > >= 3 and num_real_partitions > 4): > + part.type = 'logical' > # Reserve a sector for EBR for every logical > partition # before alignment is performed. > - if self.ptable_format == "msdos": > - self.offset += 1 > + if part.type == 'logical': > + self.offset += 2 > > + align_sectors = 0 > if part.align: > # If not first partition and we do have alignment > set we need # to align the partition. > @@ -401,21 +428,43 @@ class PartitionedImage(): > # increase the offset so we actually start the > partition on right alignment self.offset += align_sectors > > + if part.offset is not None: > + offset = (part.offset * 1024) // self.sector_size > + > + if offset * self.sector_size != part.offset * 1024: > + raise WicError("Could not place %s%s at offset > %dK with sector size %d" % (part.disk, self.numpart, part.offset, > self.sector_size)) + > + delta = offset - self.offset > + if delta < 0: > + raise WicError("Could not place %s%s at offset > %dK: next free sector is %d (delta: %d)" % (part.disk, self.numpart, > part.offset, self.offset, delta)) + > + logger.debug("Skipping %d sectors to place %s%s at > offset %dK", > + delta, part.disk, self.numpart, > part.offset) + > + self.offset = offset > + > part.start = self.offset > self.offset += part.size_sec > > - part.type = 'primary' > if not part.no_table: > part.num = self.realpart > else: > part.num = 0 > > - if self.ptable_format == "msdos": > - # only count the partitions that are in partition > table > - if num_real_partitions > 4: > - if self.realpart > 3: > - part.type = 'logical' > - part.num = self.realpart + 1 > + if self.ptable_format == "msdos" and not part.no_table: > + if part.type == 'logical': > + self.logical_part_cnt += 1 > + part.num = self.logical_part_cnt + 4 > + if self.extendedpart == 0: > + # Create extended partition as a primary > partition > + self.primary_part_num += 1 > + self.extendedpart = part.num > + else: > + self.extended_size_sec += align_sectors > + self.extended_size_sec += part.size_sec + 2 > + else: > + self.primary_part_num += 1 > + part.num = self.primary_part_num > > logger.debug("Assigned %s to %s%d, sectors range %d-%d > size %d " "sectors (%d bytes).", part.mountpoint, part.disk, > @@ -465,7 +514,7 @@ class PartitionedImage(): > if part.num == 0: > continue > > - if self.ptable_format == "msdos" and part.num == 5: > + if self.ptable_format == "msdos" and part.num == > self.extendedpart: # Create an extended partition (note: extended > # partition is described in MBR and contains all > # logical partitions). The logical partitions save a > @@ -478,8 +527,8 @@ class PartitionedImage(): > # add a sector at the back, so that there is enough > # room for all logical partitions. > self._create_partition(self.path, "extended", > - None, part.start - 1, > - self.offset - part.start + 1) > + None, part.start - 2, > + self.extended_size_sec) > > if part.fstype == "swap": > parted_fs_type = "linux-swap" > @@ -487,8 +536,8 @@ class PartitionedImage(): > parted_fs_type = "fat32" > elif part.fstype == "msdos": > parted_fs_type = "fat16" > - elif part.fstype == "ontrackdm6aux3": > - parted_fs_type = "ontrackdm6aux3" > + if not part.system_id: > + part.system_id = '0x6' # FAT16 > else: > # Type for ext2/ext3/ext4/btrfs > parted_fs_type = "ext2" > @@ -505,6 +554,13 @@ class PartitionedImage(): > self._create_partition(self.path, part.type, > parted_fs_type, part.start, > part.size_sec) > + if part.part_name: > + logger.debug("partition %d: set name to %s", > + part.num, part.part_name) > + exec_native_cmd("sgdisk --change-name=%d:%s %s" % \ > + (part.num, part.part_name, > + self.path), > self.native_sysroot) + > if part.part_type: > logger.debug("partition %d: set type UID to %s", > part.num, part.part_type) > @@ -538,21 +594,8 @@ class PartitionedImage(): > (self.path, part.num, > part.system_id), self.native_sysroot) > > - # Parted defaults to enabling the lba flag for fat16 > partitions, > - # which causes compatibility issues with some firmware > (and really > - # isn't necessary). > - if parted_fs_type == "fat16": > - if self.ptable_format == 'msdos': > - logger.debug("Disable 'lba' flag for partition > '%s' on disk '%s'", > - part.num, self.path) > - exec_native_cmd("parted -s %s set %d lba off" % \ > - (self.path, part.num), > - self.native_sysroot) > - > def cleanup(self): > - # remove partition images > - for image in set(self.partimages): > - os.remove(image) > + pass > > def assemble(self): > logger.debug("Installing partitions") > @@ -561,7 +604,7 @@ class PartitionedImage(): > source = part.source_file > if source: > # install source_file contents into a partition > - sparse_copy(source, self.path, part.start * > self.sector_size) > + sparse_copy(source, self.path, seek=part.start * > self.sector_size) > logger.debug("Installed %s in partition %d, sectors > %d-%d, " "size %d sectors", source, part.num, part.start, > diff --git a/scripts/lib/wic/plugins/source/bootimg-biosplusefi.py > b/scripts/lib/wic/plugins/source/bootimg-biosplusefi.py new file mode > 100644 index 0000000..5bd7390 > --- /dev/null > +++ b/scripts/lib/wic/plugins/source/bootimg-biosplusefi.py > @@ -0,0 +1,213 @@ > +# > +# This program is free software; you can redistribute it and/or > modify +# it under the terms of the GNU General Public License > version 2 as +# published by the Free Software Foundation. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > along +# with this program; if not, write to the Free Software > Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# > +# DESCRIPTION > +# This implements the 'bootimg-biosplusefi' source plugin class for > 'wic' +# > +# AUTHORS > +# William Bourque <wbourque [at) gmail.com> > + > +import types > + > +from wic.pluginbase import SourcePlugin > +from importlib.machinery import SourceFileLoader > + > +class BootimgBiosPlusEFIPlugin(SourcePlugin): > + """ > + Create MBR + EFI boot partition > + > + This plugin creates a boot partition that contains both > + legacy BIOS and EFI content. It will be able to boot from both. > + This is useful when managing PC fleet with some older machines > + without EFI support. > + > + Note it is possible to create an image that can boot from both > + legacy BIOS and EFI by defining two partitions : one with arg > + --source bootimg-efi and another one with --source > bootimg-pcbios. > + However, this method has the obvious downside that it requires > TWO > + partitions to be created on the storage device. > + Both partitions will also be marked as "bootable" which does not > work on > + most BIOS, has BIOS often uses the "bootable" flag to determine > + what to boot. If you have such a BIOS, you need to manually > remove the > + "bootable" flag from the EFI partition for the drive to be > bootable. > + Having two partitions also seems to confuse wic : the content of > + the first partition will be duplicated into the second, even > though it > + will not be used at all. > + > + Also, unlike "isoimage-isohybrid" that also does BIOS and EFI, > this plugin > + allows you to have more than only a single rootfs partitions and > does > + not turn the rootfs into an initramfs RAM image. > + > + This plugin is made to put everything into a single /boot > partition so it > + does not have the limitations listed above. > + > + The plugin is made so it does tries not to reimplement what's > already > + been done in other plugins; as such it imports "bootimg-pcbios" > + and "bootimg-efi". > + Plugin "bootimg-pcbios" is used to generate legacy BIOS boot. > + Plugin "bootimg-efi" is used to generate the UEFI boot. Note > that it > + requires a --sourceparams argument to know which loader to use; > refer > + to "bootimg-efi" code/documentation for the list of loader. > + > + Imports are handled with "SourceFileLoader" from importlib as it > is > + otherwise very difficult to import module that has hyphen "-" in > their > + filename. > + The SourcePlugin() methods used in the plugins (do_install_disk, > + do_configure_partition, do_prepare_partition) are then called on > both, > + beginning by "bootimg-efi". > + > + Plugin options, such as "--sourceparams" can still be passed to a > + plugin, as long they does not cause issue in the other plugin. > + > + Example wic configuration: > + part /boot --source bootimg-biosplusefi > --sourceparams="loader=grub-efi"\\ > + --ondisk sda --label os_boot --active --align 1024 > --use-uuid > + """ > + > + name = 'bootimg-biosplusefi' > + > + __PCBIOS_MODULE_NAME = "bootimg-pcbios" > + __EFI_MODULE_NAME = "bootimg-efi" > + > + __imgEFIObj = None > + __imgBiosObj = None > + > + @classmethod > + def __init__(cls): > + """ > + Constructor (init) > + """ > + > + # XXX > + # For some reasons, __init__ constructor is never called. > + # Something to do with how pluginbase works? > + cls.__instanciateSubClasses() > + > + @classmethod > + def __instanciateSubClasses(cls): > + """ > + > + """ > + > + # Import bootimg-pcbios (class name "BootimgPcbiosPlugin") > + modulePath = > os.path.join(os.path.dirname(os.path.realpath(__file__)), > + cls.__PCBIOS_MODULE_NAME + ".py") > + loader = SourceFileLoader(cls.__PCBIOS_MODULE_NAME, > modulePath) > + mod = types.ModuleType(loader.name) > + loader.exec_module(mod) > + cls.__imgBiosObj = mod.BootimgPcbiosPlugin() > + > + # Import bootimg-efi (class name "BootimgEFIPlugin") > + modulePath = > os.path.join(os.path.dirname(os.path.realpath(__file__)), > + cls.__EFI_MODULE_NAME + ".py") > + loader = SourceFileLoader(cls.__EFI_MODULE_NAME, modulePath) > + mod = types.ModuleType(loader.name) > + loader.exec_module(mod) > + cls.__imgEFIObj = mod.BootimgEFIPlugin() > + > + @classmethod > + def do_install_disk(cls, disk, disk_name, creator, workdir, > oe_builddir, > + bootimg_dir, kernel_dir, native_sysroot): > + """ > + Called after all partitions have been prepared and assembled > into a > + disk image. > + """ > + > + if ( (not cls.__imgEFIObj) or (not cls.__imgBiosObj) ): > + cls.__instanciateSubClasses() > + > + cls.__imgEFIObj.do_install_disk( > + disk, > + disk_name, > + creator, > + workdir, > + oe_builddir, > + bootimg_dir, > + kernel_dir, > + native_sysroot) > + > + cls.__imgBiosObj.do_install_disk( > + disk, > + disk_name, > + creator, > + workdir, > + oe_builddir, > + bootimg_dir, > + kernel_dir, > + native_sysroot) > + > + @classmethod > + def do_configure_partition(cls, part, source_params, creator, > cr_workdir, > + oe_builddir, bootimg_dir, kernel_dir, > + native_sysroot): > + """ > + Called before do_prepare_partition() > + """ > + > + if ( (not cls.__imgEFIObj) or (not cls.__imgBiosObj) ): > + cls.__instanciateSubClasses() > + > + cls.__imgEFIObj.do_configure_partition( > + part, > + source_params, > + creator, > + cr_workdir, > + oe_builddir, > + bootimg_dir, > + kernel_dir, > + native_sysroot) > + > + cls.__imgBiosObj.do_configure_partition( > + part, > + source_params, > + creator, > + cr_workdir, > + oe_builddir, > + bootimg_dir, > + kernel_dir, > + native_sysroot) > + > + @classmethod > + def do_prepare_partition(cls, part, source_params, creator, > cr_workdir, > + oe_builddir, bootimg_dir, kernel_dir, > + rootfs_dir, native_sysroot): > + """ > + Called to do the actual content population for a partition > i.e. it > + 'prepares' the partition to be incorporated into the image. > + """ > + > + if ( (not cls.__imgEFIObj) or (not cls.__imgBiosObj) ): > + cls.__instanciateSubClasses() > + > + cls.__imgEFIObj.do_prepare_partition( > + part, > + source_params, > + creator, > + cr_workdir, > + oe_builddir, > + bootimg_dir, > + kernel_dir, > + rootfs_dir, > + native_sysroot) > + > + cls.__imgBiosObj.do_prepare_partition( > + part, > + source_params, > + creator, > + cr_workdir, > + oe_builddir, > + bootimg_dir, > + kernel_dir, > + rootfs_dir, > + native_sysroot) > diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py > b/scripts/lib/wic/plugins/source/bootimg-efi.py index > 9879cb9..2cfdc10 100644 --- > a/scripts/lib/wic/plugins/source/bootimg-efi.py +++ > b/scripts/lib/wic/plugins/source/bootimg-efi.py @@ -1,21 +1,7 @@ > -# ex:ts=4:sw=4:sts=4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Copyright (c) 2014, Intel Corporation. > -# All rights reserved. > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # DESCRIPTION > # This implements the 'bootimg-efi' source plugin class for 'wic' > @@ -31,8 +17,8 @@ import shutil > from wic import WicError > from wic.engine import get_custom_config > from wic.pluginbase import SourcePlugin > -from wic.utils.misc import (exec_cmd, exec_native_cmd, > get_bitbake_var, > - BOOTDD_EXTRA_SPACE) > +from wic.misc import (exec_cmd, exec_native_cmd, > + get_bitbake_var, BOOTDD_EXTRA_SPACE) > > logger = logging.getLogger('wic') > > @@ -45,7 +31,7 @@ class BootimgEFIPlugin(SourcePlugin): > name = 'bootimg-efi' > > @classmethod > - def do_configure_grubefi(cls, creator, cr_workdir): > + def do_configure_grubefi(cls, hdddir, creator, cr_workdir, > source_params): """ > Create loader-specific (grub-efi) config > """ > @@ -62,20 +48,52 @@ class BootimgEFIPlugin(SourcePlugin): > raise WicError("configfile is specified but failed > to " "get it from %s." % configfile) > > + initrd = source_params.get('initrd') > + > + if initrd: > + bootimg_dir = get_bitbake_var("DEPLOY_DIR_IMAGE") > + if not bootimg_dir: > + raise WicError("Couldn't find DEPLOY_DIR_IMAGE, > exiting") + > + initrds = initrd.split(';') > + for rd in initrds: > + cp_cmd = "cp %s/%s %s" % (bootimg_dir, rd, hdddir) > + exec_cmd(cp_cmd, True) > + else: > + logger.debug("Ignoring missing initrd") > + > if not custom_cfg: > # Create grub configuration using parameters from wks > file bootloader = creator.ks.bootloader > + title = source_params.get('title') > > grubefi_conf = "" > grubefi_conf += "serial --unit=0 --speed=115200 --word=8 > --parity=no --stop=1\n" grubefi_conf += "default=boot\n" > grubefi_conf += "timeout=%s\n" % bootloader.timeout > - grubefi_conf += "menuentry 'boot'{\n" > + grubefi_conf += "menuentry '%s'{\n" % (title if title > else "boot") + > + kernel = get_bitbake_var("KERNEL_IMAGETYPE") > + if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1": > + if get_bitbake_var("INITRAMFS_IMAGE"): > + kernel = "%s-%s.bin" % \ > + (get_bitbake_var("KERNEL_IMAGETYPE"), > get_bitbake_var("INITRAMFS_LINK_NAME")) + > + label = source_params.get('label') > + label_conf = "root=%s" % creator.rootdev > + if label: > + label_conf = "LABEL=%s" % label > > - kernel = "/bzImage" > + grubefi_conf += "linux /%s %s rootwait %s\n" \ > + % (kernel, label_conf, bootloader.append) > + > + if initrd: > + initrds = initrd.split(';') > + grubefi_conf += "initrd" > + for rd in initrds: > + grubefi_conf += " /%s" % rd > + grubefi_conf += "\n" > > - grubefi_conf += "linux %s root=%s rootwait %s\n" \ > - % (kernel, creator.rootdev, bootloader.append) > grubefi_conf += "}\n" > > logger.debug("Writing grubefi config > %s/hdd/boot/EFI/BOOT/grub.cfg", @@ -109,8 +127,10 @@ class > BootimgEFIPlugin(SourcePlugin): if not bootimg_dir: > raise WicError("Couldn't find DEPLOY_DIR_IMAGE, > exiting") > - cp_cmd = "cp %s/%s %s" % (bootimg_dir, initrd, hdddir) > - exec_cmd(cp_cmd, True) > + initrds = initrd.split(';') > + for rd in initrds: > + cp_cmd = "cp %s/%s %s" % (bootimg_dir, rd, hdddir) > + exec_cmd(cp_cmd, True) > else: > logger.debug("Ignoring missing initrd") > > @@ -135,16 +155,30 @@ class BootimgEFIPlugin(SourcePlugin): > > if not custom_cfg: > # Create systemd-boot configuration using parameters > from wks file > - kernel = "/bzImage" > + kernel = get_bitbake_var("KERNEL_IMAGETYPE") > + if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1": > + if get_bitbake_var("INITRAMFS_IMAGE"): > + kernel = "%s-%s.bin" % \ > + (get_bitbake_var("KERNEL_IMAGETYPE"), > get_bitbake_var("INITRAMFS_LINK_NAME")) + > + title = source_params.get('title') > > boot_conf = "" > - boot_conf += "title boot\n" > - boot_conf += "linux %s\n" % kernel > - boot_conf += "options LABEL=Boot root=%s %s\n" % \ > - (creator.rootdev, bootloader.append) > + boot_conf += "title %s\n" % (title if title else "boot") > + boot_conf += "linux /%s\n" % kernel > + > + label = source_params.get('label') > + label_conf = "LABEL=Boot root=%s" % creator.rootdev > + if label: > + label_conf = "LABEL=%s" % label > + > + boot_conf += "options %s %s\n" % \ > + (label_conf, bootloader.append) > > if initrd: > - boot_conf += "initrd /%s\n" % initrd > + initrds = initrd.split(';') > + for rd in initrds: > + boot_conf += "initrd /%s\n" % rd > > logger.debug("Writing systemd-boot config " > "%s/hdd/boot/loader/entries/boot.conf", > cr_workdir) @@ -167,7 +201,7 @@ class BootimgEFIPlugin(SourcePlugin): > > try: > if source_params['loader'] == 'grub-efi': > - cls.do_configure_grubefi(creator, cr_workdir) > + cls.do_configure_grubefi(hdddir, creator, > cr_workdir, source_params) elif source_params['loader'] == > 'systemd-boot': cls.do_configure_systemdboot(hdddir, creator, > cr_workdir, source_params) else: > @@ -194,8 +228,14 @@ class BootimgEFIPlugin(SourcePlugin): > > hdddir = "%s/hdd/boot" % cr_workdir > > - install_cmd = "install -m 0644 %s/bzImage %s/bzImage" % \ > - (staging_kernel_dir, hdddir) > + kernel = get_bitbake_var("KERNEL_IMAGETYPE") > + if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1": > + if get_bitbake_var("INITRAMFS_IMAGE"): > + kernel = "%s-%s.bin" % \ > + (get_bitbake_var("KERNEL_IMAGETYPE"), > get_bitbake_var("INITRAMFS_LINK_NAME")) + > + install_cmd = "install -m 0644 %s/%s %s/%s" % \ > + (staging_kernel_dir, kernel, hdddir, kernel) > exec_cmd(install_cmd) > > > @@ -240,7 +280,10 @@ class BootimgEFIPlugin(SourcePlugin): > # dosfs image, created by mkdosfs > bootimg = "%s/boot.img" % cr_workdir > > - dosfs_cmd = "mkdosfs -n efi -C %s %d" % (bootimg, blocks) > + label = part.label if part.label else "ESP" > + > + dosfs_cmd = "mkdosfs -n %s -i %s -C %s %d" % \ > + (label, part.fsuuid, bootimg, blocks) > exec_native_cmd(dosfs_cmd, native_sysroot) > > mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir) > diff --git a/scripts/lib/wic/plugins/source/bootimg-partition.py > b/scripts/lib/wic/plugins/source/bootimg-partition.py index > 13fddbd..138986a 100644 --- > a/scripts/lib/wic/plugins/source/bootimg-partition.py +++ > b/scripts/lib/wic/plugins/source/bootimg-partition.py @@ -1,18 +1,5 @@ > -# ex:ts=4:sw=4:sts=4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # DESCRIPTION > # This implements the 'bootimg-partition' source plugin class for > @@ -30,8 +17,9 @@ import re > from glob import glob > > from wic import WicError > +from wic.engine import get_custom_config > from wic.pluginbase import SourcePlugin > -from wic.utils.misc import exec_cmd, get_bitbake_var > +from wic.misc import exec_cmd, get_bitbake_var > > logger = logging.getLogger('wic') > > @@ -44,17 +32,13 @@ class BootimgPartitionPlugin(SourcePlugin): > name = 'bootimg-partition' > > @classmethod > - def do_prepare_partition(cls, part, source_params, cr, > cr_workdir, > + def do_configure_partition(cls, part, source_params, cr, > cr_workdir, oe_builddir, bootimg_dir, kernel_dir, > - rootfs_dir, native_sysroot): > + native_sysroot): > """ > - Called to do the actual content population for a partition > i.e. it > - 'prepares' the partition to be incorporated into the image. > - In this case, does the following: > - - sets up a vfat partition > - - copies all files listed in IMAGE_BOOT_FILES variable > + Called before do_prepare_partition(), create u-boot specific > boot config """ > - hdddir = "%s/boot" % cr_workdir > + hdddir = "%s/boot.%d" % (cr_workdir, part.lineno) > install_cmd = "install -d %s" % hdddir > exec_cmd(install_cmd) > > @@ -63,12 +47,19 @@ class BootimgPartitionPlugin(SourcePlugin): > if not kernel_dir: > raise WicError("Couldn't find DEPLOY_DIR_IMAGE, > exiting") > - logger.debug('Kernel dir: %s', bootimg_dir) > + boot_files = None > + for (fmt, id) in (("_uuid-%s", part.uuid), ("_label-%s", > part.label), (None, None)): > + if fmt: > + var = fmt % id > + else: > + var = "" > > - boot_files = get_bitbake_var("IMAGE_BOOT_FILES") > + boot_files = get_bitbake_var("IMAGE_BOOT_FILES" + var) > + if boot_files is not None: > + break > > - if not boot_files: > - raise WicError('No boot files defined, IMAGE_BOOT_FILES > unset') > + if boot_files is None: > + raise WicError('No boot files defined, IMAGE_BOOT_FILES > unset for entry #%d' % part.lineno) > logger.debug('Boot files: %s', boot_files) > > @@ -85,9 +76,9 @@ class BootimgPartitionPlugin(SourcePlugin): > logger.debug('Destination entry: %r', dst_entry) > deploy_files.append(dst_entry) > > + cls.install_task = []; > for deploy_entry in deploy_files: > src, dst = deploy_entry > - install_task = [] > if '*' in src: > # by default install files under their basename > entry_name_fn = os.path.basename > @@ -102,22 +93,102 @@ class BootimgPartitionPlugin(SourcePlugin): > > logger.debug('Globbed sources: %s', ', '.join(srcs)) > for entry in srcs: > + src = os.path.relpath(entry, kernel_dir) > entry_dst_name = entry_name_fn(entry) > - install_task.append((entry, > - os.path.join(hdddir, > - > entry_dst_name))) > + cls.install_task.append((src, entry_dst_name)) > else: > - install_task = [(os.path.join(kernel_dir, src), > - os.path.join(hdddir, dst))] > + cls.install_task.append((src, dst)) > + > + if source_params.get('loader') != "u-boot": > + return > + > + configfile = cr.ks.bootloader.configfile > + custom_cfg = None > + if configfile: > + custom_cfg = get_custom_config(configfile) > + if custom_cfg: > + # Use a custom configuration for extlinux.conf > + extlinux_conf = custom_cfg > + logger.debug("Using custom configuration file " > + "%s for extlinux.cfg", configfile) > + else: > + raise WicError("configfile is specified but failed > to " > + "get it from %s." % configfile) > + > + if not custom_cfg: > + # The kernel types supported by the sysboot of u-boot > + kernel_types = ["zImage", "Image", "fitImage", "uImage", > "vmlinux"] > + has_dtb = False > + fdt_dir = '/' > + kernel_name = None > + > + # Find the kernel image name, from the highest > precedence to lowest > + for image in kernel_types: > + for task in cls.install_task: > + src, dst = task > + if re.match(image, src): > + kernel_name = os.path.join('/', dst) > + break > + if kernel_name: > + break > + > + for task in cls.install_task: > + src, dst = task > + # We suppose that all the dtb are in the same > directory > + if re.search(r'\.dtb', src) and fdt_dir == '/': > + has_dtb = True > + fdt_dir = os.path.join(fdt_dir, > os.path.dirname(dst)) > + break > + > + if not kernel_name: > + raise WicError('No kernel file founded') > + > + # Compose the extlinux.conf > + extlinux_conf = "default Yocto\n" > + extlinux_conf += "label Yocto\n" > + extlinux_conf += " kernel %s\n" % kernel_name > + if has_dtb: > + extlinux_conf += " fdtdir %s\n" % fdt_dir > + bootloader = cr.ks.bootloader > + extlinux_conf += "append root=%s rootwait %s\n" \ > + % (cr.rootdev, bootloader.append if > bootloader.append else '') + > + install_cmd = "install -d %s/extlinux/" % hdddir > + exec_cmd(install_cmd) > + cfg = open("%s/extlinux/extlinux.conf" % hdddir, "w") > + cfg.write(extlinux_conf) > + cfg.close() > + > + > + @classmethod > + def do_prepare_partition(cls, part, source_params, cr, > cr_workdir, > + oe_builddir, bootimg_dir, kernel_dir, > + rootfs_dir, native_sysroot): > + """ > + Called to do the actual content population for a partition > i.e. it > + 'prepares' the partition to be incorporated into the image. > + In this case, does the following: > + - sets up a vfat partition > + - copies all files listed in IMAGE_BOOT_FILES variable > + """ > + hdddir = "%s/boot.%d" % (cr_workdir, part.lineno) > + > + if not kernel_dir: > + kernel_dir = get_bitbake_var("DEPLOY_DIR_IMAGE") > + if not kernel_dir: > + raise WicError("Couldn't find DEPLOY_DIR_IMAGE, > exiting") + > + logger.debug('Kernel dir: %s', bootimg_dir) > + > > - for task in install_task: > - src_path, dst_path = task > - logger.debug('Install %s as %s', > - os.path.basename(src_path), dst_path) > - install_cmd = "install -m 0644 -D %s %s" \ > - % (src_path, dst_path) > - exec_cmd(install_cmd) > + for task in cls.install_task: > + src_path, dst_path = task > + logger.debug('Install %s as %s', src_path, dst_path) > + install_cmd = "install -m 0644 -D %s %s" \ > + % (os.path.join(kernel_dir, src_path), > + os.path.join(hdddir, dst_path)) > + exec_cmd(install_cmd) > > logger.debug('Prepare boot partition using rootfs in %s', > hdddir) part.prepare_rootfs(cr_workdir, oe_builddir, hdddir, > - native_sysroot) > + native_sysroot, False) > diff --git a/scripts/lib/wic/plugins/source/bootimg-pcbios.py > b/scripts/lib/wic/plugins/source/bootimg-pcbios.py index > 11db304..f2639e7 100644 --- > a/scripts/lib/wic/plugins/source/bootimg-pcbios.py +++ > b/scripts/lib/wic/plugins/source/bootimg-pcbios.py @@ -1,21 +1,7 @@ > -# ex:ts=4:sw=4:sts=4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Copyright (c) 2014, Intel Corporation. > -# All rights reserved. > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # DESCRIPTION > # This implements the 'bootimg-pcbios' source plugin class for 'wic' > @@ -26,13 +12,13 @@ > > import logging > import os > +import re > > from wic import WicError > from wic.engine import get_custom_config > -from wic.utils import runner > from wic.pluginbase import SourcePlugin > -from wic.utils.misc import (exec_cmd, exec_native_cmd, > - get_bitbake_var, BOOTDD_EXTRA_SPACE) > +from wic.misc import (exec_cmd, exec_native_cmd, > + get_bitbake_var, BOOTDD_EXTRA_SPACE) > > logger = logging.getLogger('wic') > > @@ -44,19 +30,22 @@ class BootimgPcbiosPlugin(SourcePlugin): > name = 'bootimg-pcbios' > > @classmethod > - def _get_syslinux_dir(cls, bootimg_dir): > + def _get_bootimg_dir(cls, bootimg_dir, dirname): > """ > - Get path to syslinux from either default bootimg_dir > - or wic-tools STAGING_DIR. > + Check if dirname exists in default bootimg_dir or in > STAGING_DIR. """ > - for path in (bootimg_dir, get_bitbake_var("STAGING_DATADIR", > "wic-tools")): > - if not path: > - continue > - syslinux_dir = os.path.join(path, 'syslinux') > - if os.path.exists(syslinux_dir): > - return syslinux_dir > + staging_datadir = get_bitbake_var("STAGING_DATADIR") > + for result in (bootimg_dir, staging_datadir): > + if os.path.exists("%s/%s" % (result, dirname)): > + return result > + > + # STAGING_DATADIR is expanded with MLPREFIX if multilib is > enabled > + # but dependency syslinux is still populated to original > STAGING_DATADIR > + nonarch_datadir = re.sub('/[^/]*recipe-sysroot', > '/recipe-sysroot', staging_datadir) > + if os.path.exists(os.path.join(nonarch_datadir, dirname)): > + return nonarch_datadir > > - raise WicError("Couldn't find syslinux directory, exiting") > + raise WicError("Couldn't find correct bootimg_dir, exiting") > > @classmethod > def do_install_disk(cls, disk, disk_name, creator, workdir, > oe_builddir, @@ -65,11 +54,12 @@ class > BootimgPcbiosPlugin(SourcePlugin): Called after all partitions have > been prepared and assembled into a disk image. In this case, we > install the MBR. """ > - syslinux_dir = cls._get_syslinux_dir(bootimg_dir) > + bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux') > + mbrfile = "%s/syslinux/" % bootimg_dir > if creator.ptable_format == 'msdos': > - mbrfile = os.path.join(syslinux_dir, "mbr.bin") > + mbrfile += "mbr.bin" > elif creator.ptable_format == 'gpt': > - mbrfile = os.path.join(syslinux_dir, "gptmbr.bin") > + mbrfile += "gptmbr.bin" > else: > raise WicError("Unsupported partition table: %s" % > creator.ptable_format) > @@ -83,10 +73,8 @@ class BootimgPcbiosPlugin(SourcePlugin): > logger.debug("Installing MBR on disk %s as %s with size %s > bytes", disk_name, full_path, disk.min_size) > > - rcode = runner.show(['dd', 'if=%s' % mbrfile, > - 'of=%s' % full_path, 'conv=notrunc']) > - if rcode != 0: > - raise WicError("Unable to set MBR to %s" % full_path) > + dd_cmd = "dd if=%s of=%s conv=notrunc" % (mbrfile, full_path) > + exec_cmd(dd_cmd, native_sysroot) > > @classmethod > def do_configure_partition(cls, part, source_params, creator, > cr_workdir, @@ -155,22 +143,28 @@ class > BootimgPcbiosPlugin(SourcePlugin): 'prepares' the partition to be > incorporated into the image. In this case, prepare content for legacy > bios boot partition. """ > - syslinux_dir = cls._get_syslinux_dir(bootimg_dir) > + bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux') > > staging_kernel_dir = kernel_dir > > hdddir = "%s/hdd/boot" % cr_workdir > > - cmds = ("install -m 0644 %s/bzImage %s/vmlinuz" % > - (staging_kernel_dir, hdddir), > - "install -m 444 %s/ldlinux.sys %s/ldlinux.sys" % > - (syslinux_dir, hdddir), > - "install -m 0644 %s/vesamenu.c32 %s/vesamenu.c32" % > - (syslinux_dir, hdddir), > - "install -m 444 %s/libcom32.c32 %s/libcom32.c32" % > - (syslinux_dir, hdddir), > - "install -m 444 %s/libutil.c32 %s/libutil.c32" % > - (syslinux_dir, hdddir)) > + kernel = get_bitbake_var("KERNEL_IMAGETYPE") > + if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1": > + if get_bitbake_var("INITRAMFS_IMAGE"): > + kernel = "%s-%s.bin" % \ > + (get_bitbake_var("KERNEL_IMAGETYPE"), > get_bitbake_var("INITRAMFS_LINK_NAME")) + > + cmds = ("install -m 0644 %s/%s %s/vmlinuz" % > + (staging_kernel_dir, kernel, hdddir), > + "install -m 444 %s/syslinux/ldlinux.sys > %s/ldlinux.sys" % > + (bootimg_dir, hdddir), > + "install -m 0644 %s/syslinux/vesamenu.c32 > %s/vesamenu.c32" % > + (bootimg_dir, hdddir), > + "install -m 444 %s/syslinux/libcom32.c32 > %s/libcom32.c32" % > + (bootimg_dir, hdddir), > + "install -m 444 %s/syslinux/libutil.c32 > %s/libutil.c32" % > + (bootimg_dir, hdddir)) > > for install_cmd in cmds: > exec_cmd(install_cmd) > @@ -190,9 +184,10 @@ class BootimgPcbiosPlugin(SourcePlugin): > extra_blocks, part.mountpoint, blocks) > > # dosfs image, created by mkdosfs > - bootimg = "%s/boot.img" % cr_workdir > + bootimg = "%s/boot%s.img" % (cr_workdir, part.lineno) > > - dosfs_cmd = "mkdosfs -n boot -S 512 -C %s %d" % (bootimg, > blocks) > + dosfs_cmd = "mkdosfs -n boot -i %s -S 512 -C %s %d" % \ > + (part.fsuuid, bootimg, blocks) > exec_native_cmd(dosfs_cmd, native_sysroot) > > mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir) > diff --git a/scripts/lib/wic/plugins/source/fsimage.py > b/scripts/lib/wic/plugins/source/fsimage.py deleted file mode 100644 > index f781499..0000000 > --- a/scripts/lib/wic/plugins/source/fsimage.py > +++ /dev/null > @@ -1,56 +0,0 @@ > -# ex:ts=4:sw=4:sts=4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > -# > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. -# > - > -import logging > -import os > - > -from wic import WicError > -from wic.pluginbase import SourcePlugin > -from wic.utils.misc import get_bitbake_var > - > -logger = logging.getLogger('wic') > - > -class FSImagePlugin(SourcePlugin): > - """ > - Add an already existing filesystem image to the partition layout. > - """ > - > - name = 'fsimage' > - > - @classmethod > - def do_prepare_partition(cls, part, source_params, cr, > cr_workdir, > - oe_builddir, bootimg_dir, kernel_dir, > - rootfs_dir, native_sysroot): > - """ > - Called to do the actual content population for a partition > i.e. it > - 'prepares' the partition to be incorporated into the image. > - """ > - if not bootimg_dir: > - bootimg_dir = get_bitbake_var("DEPLOY_DIR_IMAGE") > - if not bootimg_dir: > - raise WicError("Couldn't find DEPLOY_DIR_IMAGE, > exiting") - > - logger.debug('Bootimg dir: %s', bootimg_dir) > - > - if 'file' not in source_params: > - raise WicError("No file specified") > - > - src = os.path.join(bootimg_dir, source_params['file']) > - > - > - logger.debug('Preparing partition using image %s', src) > - part.prepare_rootfs_from_fs_image(cr_workdir, src, "") > diff --git a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py > b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py index > 1ceba62..11326a2 100644 --- > a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py +++ > b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py @@ -1,18 +1,5 > @@ -# ex:ts=4:sw=4:sts=4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > - > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > # > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # DESCRIPTION > # This implements the 'isoimage-isohybrid' source plugin class for > 'wic' @@ -29,7 +16,7 @@ import shutil > from wic import WicError > from wic.engine import get_custom_config > from wic.pluginbase import SourcePlugin > -from wic.utils.misc import exec_cmd, exec_native_cmd, get_bitbake_var > +from wic.misc import exec_cmd, exec_native_cmd, get_bitbake_var > > logger = logging.getLogger('wic') > > @@ -47,7 +34,7 @@ class IsoImagePlugin(SourcePlugin): > > Example kickstart file: > part /boot --source isoimage-isohybrid > --sourceparams="loader=grub-efi, \\ > - image_name= IsoImage" --ondisk cd --label LIVECD --fstype=ext2 > + image_name= IsoImage" --ondisk cd --label LIVECD > bootloader --timeout=10 --append=" " > > In --sourceparams "loader" specifies the bootloader used for > booting in EFI @@ -83,8 +70,13 @@ class IsoImagePlugin(SourcePlugin): > syslinux_conf += "DEFAULT boot\n" > syslinux_conf += "LABEL boot\n" > > - kernel = "/bzImage" > - syslinux_conf += "KERNEL " + kernel + "\n" > + kernel = get_bitbake_var("KERNEL_IMAGETYPE") > + if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1": > + if get_bitbake_var("INITRAMFS_IMAGE"): > + kernel = "%s-%s.bin" % \ > + (get_bitbake_var("KERNEL_IMAGETYPE"), > get_bitbake_var("INITRAMFS_LINK_NAME")) + > + syslinux_conf += "KERNEL /" + kernel + "\n" > syslinux_conf += "APPEND initrd=/initrd LABEL=boot %s\n" \ > % bootloader.append > > @@ -95,7 +87,7 @@ class IsoImagePlugin(SourcePlugin): > cfg.write(syslinux_conf) > > @classmethod > - def do_configure_grubefi(cls, part, creator, cr_workdir): > + def do_configure_grubefi(cls, part, creator, target_dir): > """ > Create loader-specific (grub-efi) config > """ > @@ -109,7 +101,7 @@ class IsoImagePlugin(SourcePlugin): > raise WicError("configfile is specified " > "but failed to get it from %s", > configfile) else: > - splash = os.path.join(cr_workdir, "EFI/boot/splash.jpg") > + splash = os.path.join(target_dir, "splash.jpg") > if os.path.exists(splash): > splashline = "menu background splash.jpg" > else: > @@ -127,9 +119,13 @@ class IsoImagePlugin(SourcePlugin): > grubefi_conf += "\n" > grubefi_conf += "menuentry 'boot'{\n" > > - kernel = "/bzImage" > + kernel = get_bitbake_var("KERNEL_IMAGETYPE") > + if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1": > + if get_bitbake_var("INITRAMFS_IMAGE"): > + kernel = "%s-%s.bin" % \ > + (get_bitbake_var("KERNEL_IMAGETYPE"), > get_bitbake_var("INITRAMFS_LINK_NAME")) > - grubefi_conf += "linux %s rootwait %s\n" \ > + grubefi_conf += "linux /%s rootwait %s\n" \ > % (kernel, bootloader.append) > grubefi_conf += "initrd /initrd \n" > grubefi_conf += "}\n" > @@ -137,9 +133,10 @@ class IsoImagePlugin(SourcePlugin): > if splashline: > grubefi_conf += "%s\n" % splashline > > - logger.debug("Writing grubefi config %s/EFI/BOOT/grub.cfg", > cr_workdir) > + cfg_path = os.path.join(target_dir, "grub.cfg") > + logger.debug("Writing grubefi config %s", cfg_path) > > - with open("%s/EFI/BOOT/grub.cfg" % cr_workdir, "w") as cfg: > + with open(cfg_path, "w") as cfg: > cfg.write(grubefi_conf) > > @staticmethod > @@ -162,13 +159,14 @@ class IsoImagePlugin(SourcePlugin): > if not image_type: > raise WicError("Couldn't find INITRAMFS_FSTYPES, > exiting.") > - target_arch = get_bitbake_var("TRANSLATED_TARGET_ARCH") > - if not target_arch: > - raise WicError("Couldn't find > TRANSLATED_TARGET_ARCH, exiting.") > + machine = os.path.basename(initrd_dir) > > - initrd = glob.glob('%s/%s*%s.%s' % (initrd_dir, > image_name, target_arch, image_type))[0] > + pattern = '%s/%s*%s.%s' % (initrd_dir, image_name, > machine, image_type) > + files = glob.glob(pattern) > + if files: > + initrd = files[0] > > - if not os.path.exists(initrd): > + if not initrd or not os.path.exists(initrd): > # Create initrd from rootfs directory > initrd = "%s/initrd.cpio.gz" % cr_workdir > initrd_dir = "%s/INITRD" % cr_workdir > @@ -189,10 +187,9 @@ class IsoImagePlugin(SourcePlugin): > else: > raise WicError("Couldn't find or build initrd, > exiting.") > - exec_cmd("cd %s && find . | cpio -o -H newc -R +0:+0 > >./initrd.cpio " \ > - % initrd_dir, as_shell=True) > - exec_cmd("gzip -f -9 -c %s/initrd.cpio > %s" \ > - % (cr_workdir, initrd), as_shell=True) > + exec_cmd("cd %s && find . | cpio -o -H newc -R root:root > >%s/initrd.cpio " \ > + % (initrd_dir, cr_workdir), as_shell=True) > + exec_cmd("gzip -f -9 %s/initrd.cpio" % cr_workdir, > as_shell=True) shutil.rmtree(initrd_dir) > > return initrd > @@ -206,8 +203,8 @@ class IsoImagePlugin(SourcePlugin): > """ > isodir = "%s/ISO/" % cr_workdir > > - if os.path.exists(cr_workdir): > - shutil.rmtree(cr_workdir) > + if os.path.exists(isodir): > + shutil.rmtree(isodir) > > install_cmd = "install -d %s " % isodir > exec_cmd(install_cmd) > @@ -251,33 +248,8 @@ class IsoImagePlugin(SourcePlugin): > raise WicError("Couldn't find IMAGE_ROOTFS, exiting.") > > part.rootfs_dir = rootfs_dir > - > - # Prepare rootfs.img > deploy_dir = get_bitbake_var("DEPLOY_DIR_IMAGE") > img_iso_dir = get_bitbake_var("ISODIR") > - rootfs_img = "%s/rootfs.img" % img_iso_dir > - if not os.path.isfile(rootfs_img): > - # check if rootfs.img is in deploydir > - deploy_dir = get_bitbake_var("DEPLOY_DIR_IMAGE") > - image_name = get_bitbake_var("IMAGE_LINK_NAME") > - rootfs_img = "%s/%s.%s" \ > - % (deploy_dir, image_name, part.fstype) > - > - if not os.path.isfile(rootfs_img): > - # create image file with type specified by --fstype > - # which contains rootfs > - du_cmd = "du -bks %s" % rootfs_dir > - out = exec_cmd(du_cmd) > - part.size = int(out.split()[0]) > - part.extra_space = 0 > - part.overhead_factor = 1.2 > - part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, > \ > - native_sysroot) > - rootfs_img = part.source_file > - > - install_cmd = "install -m 0644 %s %s/rootfs.img" \ > - % (rootfs_img, isodir) > - exec_cmd(install_cmd) > > # Remove the temporary file created by part.prepare_rootfs() > if os.path.isfile(part.source_file): > @@ -305,27 +277,25 @@ class IsoImagePlugin(SourcePlugin): > if os.path.isfile("%s/initrd.cpio.gz" % cr_workdir): > os.remove("%s/initrd.cpio.gz" % cr_workdir) > > - # Install bzImage > - install_cmd = "install -m 0644 %s/bzImage %s/bzImage" % \ > - (kernel_dir, isodir) > + kernel = get_bitbake_var("KERNEL_IMAGETYPE") > + if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1": > + if get_bitbake_var("INITRAMFS_IMAGE"): > + kernel = "%s-%s.bin" % \ > + (get_bitbake_var("KERNEL_IMAGETYPE"), > get_bitbake_var("INITRAMFS_LINK_NAME")) + > + install_cmd = "install -m 0644 %s/%s %s/%s" % \ > + (kernel_dir, kernel, isodir, kernel) > exec_cmd(install_cmd) > > #Create bootloader for efi boot > try: > - if source_params['loader'] == 'grub-efi': > - # Builds grub.cfg if ISODIR didn't exist or > - # didn't contains grub.cfg > - bootimg_dir = img_iso_dir > - if not os.path.exists("%s/EFI/BOOT" % bootimg_dir): > - bootimg_dir = "%s/bootimg" % cr_workdir > - if os.path.exists(bootimg_dir): > - shutil.rmtree(bootimg_dir) > - install_cmd = "install -d %s/EFI/BOOT" % > bootimg_dir > - exec_cmd(install_cmd) > - > - if not os.path.isfile("%s/EFI/BOOT/boot.cfg" % > bootimg_dir): > - cls.do_configure_grubefi(part, creator, > bootimg_dir) > + target_dir = "%s/EFI/BOOT" % isodir > + if os.path.exists(target_dir): > + shutil.rmtree(target_dir) > > + os.makedirs(target_dir) > + > + if source_params['loader'] == 'grub-efi': > # Builds bootx64.efi/bootia32.efi if ISODIR didn't > exist or # didn't contains it > target_arch = get_bitbake_var("TARGET_SYS") > @@ -333,37 +303,25 @@ class IsoImagePlugin(SourcePlugin): > raise WicError("Coludn't find target > architecture") > if re.match("x86_64", target_arch): > - grub_target = 'x86_64-efi' > - grub_image = "bootx64.efi" > + grub_src_image = "grub-efi-bootx64.efi" > + grub_dest_image = "bootx64.efi" > elif re.match('i.86', target_arch): > - grub_target = 'i386-efi' > - grub_image = "bootia32.efi" > + grub_src_image = "grub-efi-bootia32.efi" > + grub_dest_image = "bootia32.efi" > else: > raise WicError("grub-efi is incompatible with > target %s" % target_arch) > > - if not os.path.isfile("%s/EFI/BOOT/%s" \ > - % (bootimg_dir, grub_image)): > - grub_path = get_bitbake_var("STAGING_LIBDIR", > "wic-tools") > - if not grub_path: > - raise WicError("Couldn't find > STAGING_LIBDIR, exiting.") - > - grub_core = "%s/grub/%s" % (grub_path, > grub_target) > - if not os.path.exists(grub_core): > - raise WicError("Please build grub-efi first") > - > - grub_cmd = "grub-mkimage -p '/EFI/BOOT' " > - grub_cmd += "-d %s " % grub_core > - grub_cmd += "-O %s -o %s/EFI/BOOT/%s " \ > - % (grub_target, bootimg_dir, > grub_image) > - grub_cmd += "part_gpt part_msdos ntfs ntfscomp > fat ext2 " > - grub_cmd += "normal chain boot configfile linux > multiboot " > - grub_cmd += "search efi_gop efi_uga font gfxterm > gfxmenu " > - grub_cmd += "terminal minicmd test iorw loadenv > echo help " > - grub_cmd += "reboot serial terminfo iso9660 > loopback tar " > - grub_cmd += "memdisk ls search_fs_uuid udf btrfs > xfs lvm " > - grub_cmd += "reiserfs ata " > - exec_native_cmd(grub_cmd, native_sysroot) > + grub_target = os.path.join(target_dir, > grub_dest_image) > + if not os.path.isfile(grub_target): > + grub_src = os.path.join(deploy_dir, > grub_src_image) > + if not os.path.exists(grub_src): > + raise WicError("Grub loader %s is not found > in %s. " > + "Please build grub-efi first" > % (grub_src_image, deploy_dir)) > + shutil.copy(grub_src, grub_target) > + > + if not os.path.isfile(os.path.join(target_dir, > "boot.cfg")): > + cls.do_configure_grubefi(part, creator, > target_dir) > else: > raise WicError("unrecognized bootimg-efi loader: %s" > % @@ -371,15 +329,6 @@ class IsoImagePlugin(SourcePlugin): > except KeyError: > raise WicError("bootimg-efi requires a loader, none > specified") > - if os.path.exists("%s/EFI/BOOT" % isodir): > - shutil.rmtree("%s/EFI/BOOT" % isodir) > - > - shutil.copytree(bootimg_dir+"/EFI/BOOT", isodir+"/EFI/BOOT") > - > - # If exists, remove cr_workdir/bootimg temporary folder > - if os.path.exists("%s/bootimg" % cr_workdir): > - shutil.rmtree("%s/bootimg" % cr_workdir) > - > # Create efi.img that contains bootloader files for EFI > booting # if ISODIR didn't exist or didn't contains it > if os.path.isfile("%s/efi.img" % img_iso_dir): > @@ -387,19 +336,23 @@ class IsoImagePlugin(SourcePlugin): > (img_iso_dir, isodir) > exec_cmd(install_cmd) > else: > + # Default to 100 blocks of extra space for file system > overhead > + esp_extra_blocks = > int(source_params.get('esp_extra_blocks', '100')) + > du_cmd = "du -bks %s/EFI" % isodir > out = exec_cmd(du_cmd) > blocks = int(out.split()[0]) > - # Add some extra space for file system overhead > - blocks += 100 > + blocks += esp_extra_blocks > logger.debug("Added 100 extra blocks to %s to get to %d " > "total blocks", part.mountpoint, blocks) > > # dosfs image for EFI boot > bootimg = "%s/efi.img" % isodir > > - dosfs_cmd = 'mkfs.vfat -n "EFIimg" -S 512 -C %s %d' \ > - % (bootimg, blocks) > + esp_label = source_params.get('esp_label', 'EFIimg') > + > + dosfs_cmd = 'mkfs.vfat -n \'%s\' -S 512 -C %s %d' \ > + % (esp_label, bootimg, blocks) > exec_native_cmd(dosfs_cmd, native_sysroot) > > mmd_cmd = "mmd -i %s ::/EFI" % bootimg > @@ -413,7 +366,7 @@ class IsoImagePlugin(SourcePlugin): > exec_cmd(chmod_cmd) > > # Prepare files for legacy boot > - syslinux_dir = get_bitbake_var("STAGING_DATADIR", > "wic-tools") > + syslinux_dir = get_bitbake_var("STAGING_DATADIR") > if not syslinux_dir: > raise WicError("Couldn't find STAGING_DATADIR, exiting.") > > diff --git a/scripts/lib/wic/plugins/source/rawcopy.py > b/scripts/lib/wic/plugins/source/rawcopy.py index e1c4f5e..3c4997d > 100644 --- a/scripts/lib/wic/plugins/source/rawcopy.py > +++ b/scripts/lib/wic/plugins/source/rawcopy.py > @@ -1,18 +1,5 @@ > -# ex:ts=4:sw=4:sts=4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > > import logging > @@ -20,7 +7,7 @@ import os > > from wic import WicError > from wic.pluginbase import SourcePlugin > -from wic.utils.misc import exec_cmd, get_bitbake_var > +from wic.misc import exec_cmd, get_bitbake_var > from wic.filemap import sparse_copy > > logger = logging.getLogger('wic') > @@ -32,6 +19,25 @@ class RawCopyPlugin(SourcePlugin): > > name = 'rawcopy' > > + @staticmethod > + def do_image_label(fstype, dst, label): > + if fstype.startswith('ext'): > + cmd = 'tune2fs -L %s %s' % (label, dst) > + elif fstype in ('msdos', 'vfat'): > + cmd = 'dosfslabel %s %s' % (dst, label) > + elif fstype == 'btrfs': > + cmd = 'btrfs filesystem label %s %s' % (dst, label) > + elif fstype == 'swap': > + cmd = 'mkswap -L %s %s' % (label, dst) > + elif fstype == 'squashfs': > + raise WicError("It's not possible to update a squashfs " > + "filesystem label '%s'" % (label)) > + else: > + raise WicError("Cannot update filesystem label: " > + "Unknown fstype: '%s'" % (fstype)) > + > + exec_cmd(cmd) > + > @classmethod > def do_prepare_partition(cls, part, source_params, cr, > cr_workdir, oe_builddir, bootimg_dir, kernel_dir, > @@ -51,7 +57,10 @@ class RawCopyPlugin(SourcePlugin): > raise WicError("No file specified") > > src = os.path.join(kernel_dir, source_params['file']) > - dst = os.path.join(cr_workdir, "%s.%s" % > (source_params['file'], part.lineno)) > + dst = os.path.join(cr_workdir, "%s.%s" % > (os.path.basename(source_params['file']), part.lineno)) + > + if not os.path.exists(os.path.dirname(dst)): > + os.makedirs(os.path.dirname(dst)) > > if 'skip' in source_params: > sparse_copy(src, dst, skip=int(source_params['skip'])) > @@ -66,4 +75,7 @@ class RawCopyPlugin(SourcePlugin): > if filesize > part.size: > part.size = filesize > > + if part.label: > + RawCopyPlugin.do_image_label(part.fstype, dst, > part.label) + > part.source_file = dst > diff --git a/scripts/lib/wic/plugins/source/rootfs.py > b/scripts/lib/wic/plugins/source/rootfs.py index f2e2ca8..f1db83f > 100644 --- a/scripts/lib/wic/plugins/source/rootfs.py > +++ b/scripts/lib/wic/plugins/source/rootfs.py > @@ -1,21 +1,7 @@ > -# ex:ts=4:sw=4:sts=4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Copyright (c) 2014, Intel Corporation. > -# All rights reserved. > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # DESCRIPTION > # This implements the 'rootfs' source plugin class for 'wic' > @@ -28,12 +14,14 @@ > import logging > import os > import shutil > +import sys > > from oe.path import copyhardlinktree > +from pathlib import Path > > from wic import WicError > from wic.pluginbase import SourcePlugin > -from wic.utils.misc import get_bitbake_var, exec_cmd > +from wic.misc import get_bitbake_var, exec_native_cmd > > logger = logging.getLogger('wic') > > @@ -44,10 +32,26 @@ class RootfsPlugin(SourcePlugin): > > name = 'rootfs' > > + @staticmethod > + def __validate_path(cmd, rootfs_dir, path): > + if os.path.isabs(path): > + logger.error("%s: Must be relative: %s" % (cmd, > orig_path)) > + sys.exit(1) > + > + # Disallow climbing outside of parent directory using '..', > + # because doing so could be quite disastrous (we will delete > the > + # directory, or modify a directory outside OpenEmbedded). > + full_path = os.path.realpath(os.path.join(rootfs_dir, path)) > + if not full_path.startswith(os.path.realpath(rootfs_dir)): > + logger.error("%s: Must point inside the rootfs:" % (cmd, > path)) > + sys.exit(1) > + > + return full_path > + > @staticmethod > def __get_rootfs_dir(rootfs_dir): > if os.path.isdir(rootfs_dir): > - return rootfs_dir > + return os.path.realpath(rootfs_dir) > > image_rootfs_dir = get_bitbake_var("IMAGE_ROOTFS", > rootfs_dir) if not os.path.isdir(image_rootfs_dir): > @@ -55,7 +59,16 @@ class RootfsPlugin(SourcePlugin): > "named %s has been found at %s, exiting." > % (rootfs_dir, image_rootfs_dir)) > > - return image_rootfs_dir > + return os.path.realpath(image_rootfs_dir) > + > + @staticmethod > + def __get_pseudo(native_sysroot, rootfs, pseudo_dir): > + pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot > + pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir > + pseudo += "export PSEUDO_PASSWD=%s;" % rootfs > + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" > + pseudo += "%s " % get_bitbake_var("FAKEROOTCMD") > + return pseudo > > @classmethod > def do_prepare_partition(cls, part, source_params, cr, > cr_workdir, @@ -80,33 +93,111 @@ class RootfsPlugin(SourcePlugin): > raise WicError("Couldn't find --rootfs-dir=%s > connection or " "it is not a valid path, exiting" % part.rootfs_dir) > > - real_rootfs_dir = cls.__get_rootfs_dir(rootfs_dir) > + part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir) > + pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo") > + if not os.path.lexists(pseudo_dir): > + logger.warn("%s folder does not exist. " > + "Usernames and permissions will be invalid " > % pseudo_dir) > + pseudo_dir = None > > + new_rootfs = None > + new_pseudo = None > # Handle excluded paths. > - if part.exclude_path is not None: > + if part.exclude_path or part.include_path or > part.change_directory: # We need a new rootfs directory we can delete > files from. Copy to # workdir. > - new_rootfs = os.path.realpath(os.path.join(cr_workdir, > "rootfs")) > + new_rootfs = os.path.realpath(os.path.join(cr_workdir, > "rootfs%d" % part.lineno)) > if os.path.lexists(new_rootfs): > shutil.rmtree(os.path.join(new_rootfs)) > > - copyhardlinktree(real_rootfs_dir, new_rootfs) > + if part.change_directory: > + cd = part.change_directory > + if cd[-1] == '/': > + cd = cd[:-1] > + orig_dir = cls.__validate_path("--change-directory", > part.rootfs_dir, cd) > + else: > + orig_dir = part.rootfs_dir > + copyhardlinktree(orig_dir, new_rootfs) > + > + # Convert the pseudo directory to its new location > + if (pseudo_dir): > + new_pseudo = os.path.realpath( > + os.path.join(cr_workdir, "pseudo%d" % > part.lineno)) > + if os.path.lexists(new_pseudo): > + shutil.rmtree(new_pseudo) > + os.mkdir(new_pseudo) > + shutil.copy(os.path.join(pseudo_dir, "files.db"), > + os.path.join(new_pseudo, "files.db")) > + > + pseudo_cmd = "%s -B -m %s -M %s" % > (cls.__get_pseudo(native_sysroot, > + > new_rootfs, > + > new_pseudo), > + orig_dir, > new_rootfs) > + exec_native_cmd(pseudo_cmd, native_sysroot) > + > + for in_path in part.include_path or []: > + #parse arguments > + include_path = in_path[0] > + if len(in_path) > 2: > + logger.error("'Invalid number of arguments for > include-path") > + sys.exit(1) > + if len(in_path) == 2: > + path = in_path[1] > + else: > + path = None > + > + # Pack files to be included into a tar file. > + # We need to create a tar file, because that way we > can keep the > + # permissions from the files even when they belong > to different > + # pseudo enviroments. > + # If we simply copy files using > copyhardlinktree/copytree... the > + # copied files will belong to the user running wic. > + tar_file = os.path.realpath( > + os.path.join(cr_workdir, > "include-path%d.tar" % part.lineno)) > + if os.path.isfile(include_path): > + parent = > os.path.dirname(os.path.realpath(include_path)) > + tar_cmd = "tar c --owner=root --group=root -f %s > -C %s %s" % ( > + tar_file, parent, > os.path.relpath(include_path, parent)) > + exec_native_cmd(tar_cmd, native_sysroot) > + else: > + if include_path in krootfs_dir: > + include_path = krootfs_dir[include_path] > + include_path = cls.__get_rootfs_dir(include_path) > + include_pseudo = os.path.join(include_path, > "../pseudo") > + if os.path.lexists(include_pseudo): > + pseudo = cls.__get_pseudo(native_sysroot, > include_path, > + include_pseudo) > + tar_cmd = "tar cf %s -C %s ." % (tar_file, > include_path) > + else: > + pseudo = None > + tar_cmd = "tar c --owner=root --group=root > -f %s -C %s ." % ( > + tar_file, include_path) > + exec_native_cmd(tar_cmd, native_sysroot, pseudo) > + > + #create destination > + if path: > + destination = > cls.__validate_path("--include-path", new_rootfs, path) > + Path(destination).mkdir(parents=True, > exist_ok=True) > + else: > + destination = new_rootfs > > - real_rootfs_dir = new_rootfs > + #extract destination > + untar_cmd = "tar xf %s -C %s" % (tar_file, > destination) > + if new_pseudo: > + pseudo = cls.__get_pseudo(native_sysroot, > new_rootfs, new_pseudo) > + else: > + pseudo = None > + exec_native_cmd(untar_cmd, native_sysroot, pseudo) > + os.remove(tar_file) > > - for orig_path in part.exclude_path: > + for orig_path in part.exclude_path or []: > path = orig_path > - if os.path.isabs(path): > - msger.error("Must be relative: > --exclude-path=%s" % orig_path) > - full_path = > os.path.realpath(os.path.join(new_rootfs, path)) > + full_path = cls.__validate_path("--exclude-path", > new_rootfs, path) > - # Disallow climbing outside of parent directory > using '..', > - # because doing so could be quite disastrous (we > will delete the > - # directory). > - if not full_path.startswith(new_rootfs): > - msger.error("'%s' points to a path outside the > rootfs" % orig_path) > + if not os.path.lexists(full_path): > + continue > > if path.endswith(os.sep): > # Delete content only. > @@ -120,6 +211,6 @@ class RootfsPlugin(SourcePlugin): > # Delete whole directory. > shutil.rmtree(full_path) > > - part.rootfs_dir = real_rootfs_dir > part.prepare_rootfs(cr_workdir, oe_builddir, > - real_rootfs_dir, native_sysroot) > + new_rootfs or part.rootfs_dir, > native_sysroot, > + pseudo_dir = new_pseudo or pseudo_dir) > diff --git a/scripts/lib/wic/utils/__init__.py > b/scripts/lib/wic/utils/__init__.py deleted file mode 100644 > index e69de29..0000000 > diff --git a/scripts/lib/wic/utils/runner.py > b/scripts/lib/wic/utils/runner.py deleted file mode 100644 > index 56d7ea3..0000000 > --- a/scripts/lib/wic/utils/runner.py > +++ /dev/null > @@ -1,114 +0,0 @@ > -#!/usr/bin/env python -tt > -# > -# Copyright (c) 2011 Intel, Inc. > -# > -# This program is free software; you can redistribute it and/or > modify it -# under the terms of the GNU General Public License as > published by the Free -# Software Foundation; version 2 of the License > -# > -# This program is distributed in the hope that it will be useful, but > -# WITHOUT ANY WARRANTY; without even the implied warranty of > MERCHANTABILITY -# or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > General Public License -# for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., 59 -# Temple Place - Suite 330, Boston, MA > 02111-1307, USA. - > -import logging > -import os > -import subprocess > - > -from wic import WicError > - > -logger = logging.getLogger('wic') > - > -def runtool(cmdln_or_args, catch=1): > - """ wrapper for most of the subprocess calls > - input: > - cmdln_or_args: can be both args and cmdln str (shell=True) > - catch: 0, quitely run > - 1, only STDOUT > - 2, only STDERR > - 3, both STDOUT and STDERR > - return: > - (rc, output) > - if catch==0: the output will always None > - """ > - > - if catch not in (0, 1, 2, 3): > - # invalid catch selection, will cause exception, that's good > - return None > - > - if isinstance(cmdln_or_args, list): > - cmd = cmdln_or_args[0] > - shell = False > - else: > - import shlex > - cmd = shlex.split(cmdln_or_args)[0] > - shell = True > - > - if catch != 3: > - dev_null = os.open("/dev/null", os.O_WRONLY) > - > - if catch == 0: > - sout = dev_null > - serr = dev_null > - elif catch == 1: > - sout = subprocess.PIPE > - serr = dev_null > - elif catch == 2: > - sout = dev_null > - serr = subprocess.PIPE > - elif catch == 3: > - sout = subprocess.PIPE > - serr = subprocess.STDOUT > - > - try: > - process = subprocess.Popen(cmdln_or_args, stdout=sout, > - stderr=serr, shell=shell) > - (sout, serr) = process.communicate() > - # combine stdout and stderr, filter None out and decode > - out = ''.join([out.decode('utf-8') for out in [sout, serr] > if out]) > - except OSError as err: > - if err.errno == 2: > - # [Errno 2] No such file or directory > - raise WicError('Cannot run command: %s, lost > dependency?' % cmd) > - else: > - raise # relay > - finally: > - if catch != 3: > - os.close(dev_null) > - > - return (process.returncode, out) > - > -def show(cmdln_or_args): > - """Show all messages using logger.debug.""" > - > - rcode, out = runtool(cmdln_or_args, catch=3) > - > - if isinstance(cmdln_or_args, list): > - cmd = ' '.join(cmdln_or_args) > - else: > - cmd = cmdln_or_args > - > - msg = 'running command: "%s"' % cmd > - if out: > - out = out.strip() > - if out: > - msg += ', with output::' > - msg += '\n +----------------' > - for line in out.splitlines(): > - msg += '\n | %s' % line > - msg += '\n +----------------' > - > - logger.debug(msg) > - > - return rcode > - > -def outs(cmdln_or_args, catch=1): > - # get the outputs of tools > - return runtool(cmdln_or_args, catch)[1].strip() > - > -def quiet(cmdln_or_args): > - return runtool(cmdln_or_args, catch=0)[0] > diff --git a/scripts/wic b/scripts/wic > index a5f2dbf..24700f3 100755 > --- a/scripts/wic > +++ b/scripts/wic > @@ -1,22 +1,8 @@ > #!/usr/bin/env python3 > -# ex:ts=4:sw=4:sts=4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Copyright (c) 2013, Intel Corporation. > -# All rights reserved. > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # DESCRIPTION 'wic' is the OpenEmbedded Image Creator that users can > # use to generate bootable images. Invoking it without any arguments > @@ -33,28 +19,41 @@ __version__ = "0.2.0" > # Python Standard Library modules > import os > import sys > -import optparse > +import argparse > import logging > +import subprocess > + > +from collections import namedtuple > from distutils import spawn > > # External modules > -scripts_path = os.path.abspath(os.path.dirname(__file__)) > +scripts_path = os.path.dirname(os.path.realpath(__file__)) > lib_path = scripts_path + '/lib' > sys.path.insert(0, lib_path) > -oe_lib_path = os.path.join(os.path.dirname(scripts_path), 'meta', > 'lib') -sys.path.insert(0, oe_lib_path) > +import scriptpath > +scriptpath.add_oe_lib_path() > + > +# Check whether wic is running within eSDK environment > +sdkroot = scripts_path > +if os.environ.get('SDKTARGETSYSROOT'): > + while sdkroot != '' and sdkroot != os.sep: > + if os.path.exists(os.path.join(sdkroot, '.devtoolbase')): > + # Set BUILDDIR for wic to work within eSDK > + os.environ['BUILDDIR'] = sdkroot > + # .devtoolbase only exists within eSDK > + # If found, initialize bitbake path for eSDK environment > and append to PATH > + sdkroot = os.path.join(os.path.dirname(scripts_path), > 'bitbake', 'bin') > + os.environ['PATH'] += ":" + sdkroot > + break > + sdkroot = os.path.dirname(sdkroot) > > bitbake_exe = spawn.find_executable('bitbake') > if bitbake_exe: > - bitbake_path = os.path.join(os.path.dirname(bitbake_exe), > '../lib') > - sys.path.insert(0, bitbake_path) > - from bb import cookerdata > - from bb.main import bitbake_main, BitBakeConfigParameters > -else: > - bitbake_main = None > + bitbake_path = scriptpath.add_bitbake_lib_path() > + import bb > > from wic import WicError > -from wic.utils.misc import get_bitbake_var, BB_VARS > +from wic.misc import get_bitbake_var, BB_VARS > from wic import engine > from wic import help as hlp > > @@ -85,67 +84,31 @@ def rootfs_dir_to_args(krootfs_dir): > rootfs_dir += '='.join([key, val]) > return rootfs_dir.strip() > > -def callback_rootfs_dir(option, opt, value, parser): > - """ > - Build a dict using --rootfs_dir connection=dir > - """ > - if not type(parser.values.rootfs_dir) is dict: > - parser.values.rootfs_dir = dict() > > - if '=' in value: > - (key, rootfs_dir) = value.split('=') > - else: > - key = 'ROOTFS_DIR' > - rootfs_dir = value > +class RootfsArgAction(argparse.Action): > + def __init__(self, **kwargs): > + super().__init__(**kwargs) > + > + def __call__(self, parser, namespace, value, option_string=None): > + if not "rootfs_dir" in vars(namespace) or \ > + not type(namespace.__dict__['rootfs_dir']) is dict: > + namespace.__dict__['rootfs_dir'] = {} > > - parser.values.rootfs_dir[key] = rootfs_dir > + if '=' in value: > + (key, rootfs_dir) = value.split('=') > + else: > + key = 'ROOTFS_DIR' > + rootfs_dir = value > > -def wic_create_subcommand(args, usage_str): > + namespace.__dict__['rootfs_dir'][key] = rootfs_dir > + > + > +def wic_create_subcommand(options, usage_str): > """ > Command-line handling for image creation. The real work is done > by image.engine.wic_create() > """ > - parser = optparse.OptionParser(usage=usage_str) > - > - parser.add_option("-o", "--outdir", dest="outdir", default='.', > - help="name of directory to create image in") > - parser.add_option("-e", "--image-name", dest="image_name", > - help="name of the image to use the artifacts > from " > - "e.g. core-image-sato") > - parser.add_option("-r", "--rootfs-dir", dest="rootfs_dir", > type="string", > - action="callback", > callback=callback_rootfs_dir, > - help="path to the /rootfs dir to use as the " > - ".wks rootfs source") > - parser.add_option("-b", "--bootimg-dir", dest="bootimg_dir", > - help="path to the dir containing the boot > artifacts " > - "(e.g. /EFI or /syslinux dirs) to use as > the " > - ".wks bootimg source") > - parser.add_option("-k", "--kernel-dir", dest="kernel_dir", > - help="path to the dir containing the kernel to > use " > - "in the .wks bootimg") > - parser.add_option("-n", "--native-sysroot", > dest="native_sysroot", > - help="path to the native sysroot containing > the tools " > - "to use to build the image") > - parser.add_option("-s", "--skip-build-check", dest="build_check", > - action="store_false", default=True, help="skip > the build check") > - parser.add_option("-f", "--build-rootfs", action="store_true", > help="build rootfs") > - parser.add_option("-c", "--compress-with", choices=("gzip", > "bzip2", "xz"), > - dest='compressor', > - help="compress image with specified > compressor") > - parser.add_option("-m", "--bmap", action="store_true", > help="generate .bmap") > - parser.add_option("-v", "--vars", dest='vars_dir', > - help="directory with <image>.env files that > store " > - "bitbake variables") > - parser.add_option("-D", "--debug", dest="debug", > action="store_true", > - default=False, help="output debug information") > - > - (options, args) = parser.parse_args(args) > - > - if len(args) != 1: > - parser.print_help() > - raise WicError("Wrong number of arguments, exiting") > - > - if options.build_rootfs and not bitbake_main: > + if options.build_rootfs and not bitbake_exe: > raise WicError("Can't build rootfs as bitbake is not in the > $PATH") > if not options.image_name: > @@ -181,39 +144,37 @@ def wic_create_subcommand(args, usage_str): > argv.append("--debug") > > logger.info("Building rootfs...\n") > - if bitbake_main(BitBakeConfigParameters(argv), > - cookerdata.CookerConfiguration()): > - raise WicError("bitbake exited with error") > + subprocess.check_call(argv) > > rootfs_dir = get_bitbake_var("IMAGE_ROOTFS", > options.image_name) kernel_dir = get_bitbake_var("DEPLOY_DIR_IMAGE", > options.image_name) bootimg_dir = get_bitbake_var("STAGING_DATADIR", > options.image_name) > - native_sysroot = get_bitbake_var("RECIPE_SYSROOT_NATIVE", > - options.image_name) #, > cache=False) + > + native_sysroot = options.native_sysroot > + if options.vars_dir and not native_sysroot: > + native_sysroot = > get_bitbake_var("RECIPE_SYSROOT_NATIVE", options.image_name) else: > if options.build_rootfs: > raise WicError("Image name is not specified, exiting. " > "(Use -e/--image-name to specify it)") > native_sysroot = options.native_sysroot > > - if not native_sysroot or not os.path.isdir(native_sysroot): > + if not options.vars_dir and (not native_sysroot or not > os.path.isdir(native_sysroot)): logger.info("Building wic-tools...\n") > - if bitbake_main(BitBakeConfigParameters("bitbake > wic-tools".split()), > - cookerdata.CookerConfiguration()): > - raise WicError("bitbake wic-tools failed") > + subprocess.check_call(["bitbake", "wic-tools"]) > native_sysroot = get_bitbake_var("RECIPE_SYSROOT_NATIVE", > "wic-tools") > - if not native_sysroot: > - raise WicError("Unable to find the location of the > native " > - "tools sysroot to use") > > - wks_file = args[0] > + if not native_sysroot: > + raise WicError("Unable to find the location of the native > tools sysroot") + > + wks_file = options.wks_file > > if not wks_file.endswith(".wks"): > wks_file = engine.find_canned_image(scripts_path, wks_file) > if not wks_file: > raise WicError("No image named %s found, exiting. (Use > 'wic list images' " "to list available images, or specify a > fully-qualified OE " > - "kickstart (.wks) filename)" % args[0]) > + "kickstart (.wks) filename)" % > options.wks_file) > if not options.image_name: > rootfs_dir = '' > @@ -264,59 +225,312 @@ def wic_list_subcommand(args, usage_str): > Command-line handling for listing available images. > The real work is done by image.engine.wic_list() > """ > - parser = optparse.OptionParser(usage=usage_str) > - args = parser.parse_args(args)[1] > - > if not engine.wic_list(args, scripts_path): > - parser.print_help() > raise WicError("Bad list arguments, exiting") > > > -def wic_help_topic_subcommand(args, usage_str): > +def wic_ls_subcommand(args, usage_str): > + """ > + Command-line handling for list content of images. > + The real work is done by engine.wic_ls() > + """ > + engine.wic_ls(args, args.native_sysroot) > + > +def wic_cp_subcommand(args, usage_str): > + """ > + Command-line handling for copying files/dirs to images. > + The real work is done by engine.wic_cp() > + """ > + engine.wic_cp(args, args.native_sysroot) > + > +def wic_rm_subcommand(args, usage_str): > """ > - Command-line handling for help-only 'subcommands'. This is > - essentially a dummy command that doesn nothing but allow users to > - use the existing subcommand infrastructure to display help on a > - particular topic not attached to any particular subcommand. > + Command-line handling for removing files/dirs from images. > + The real work is done by engine.wic_rm() > + """ > + engine.wic_rm(args, args.native_sysroot) > + > +def wic_write_subcommand(args, usage_str): > + """ > + Command-line handling for writing images. > + The real work is done by engine.wic_write() > + """ > + engine.wic_write(args, args.native_sysroot) > + > +def wic_help_subcommand(args, usage_str): > + """ > + Command-line handling for help subcommand to keep the current > + structure of the function definitions. > """ > pass > > > +def wic_help_topic_subcommand(usage_str, help_str): > + """ > + Display function for help 'sub-subcommands'. > + """ > + print(help_str) > + return > + > + > wic_help_topic_usage = """ > """ > > -subcommands = { > - "create": [wic_create_subcommand, > - hlp.wic_create_usage, > - hlp.wic_create_help], > - "list": [wic_list_subcommand, > - hlp.wic_list_usage, > - hlp.wic_list_help], > +helptopics = { > "plugins": [wic_help_topic_subcommand, > wic_help_topic_usage, > - hlp.get_wic_plugins_help], > + hlp.wic_plugins_help], > "overview": [wic_help_topic_subcommand, > wic_help_topic_usage, > hlp.wic_overview_help], > "kickstart": [wic_help_topic_subcommand, > wic_help_topic_usage, > hlp.wic_kickstart_help], > + "create": [wic_help_topic_subcommand, > + wic_help_topic_usage, > + hlp.wic_create_help], > + "ls": [wic_help_topic_subcommand, > + wic_help_topic_usage, > + hlp.wic_ls_help], > + "cp": [wic_help_topic_subcommand, > + wic_help_topic_usage, > + hlp.wic_cp_help], > + "rm": [wic_help_topic_subcommand, > + wic_help_topic_usage, > + hlp.wic_rm_help], > + "write": [wic_help_topic_subcommand, > + wic_help_topic_usage, > + hlp.wic_write_help], > + "list": [wic_help_topic_subcommand, > + wic_help_topic_usage, > + hlp.wic_list_help] > } > > > +def wic_init_parser_create(subparser): > + subparser.add_argument("wks_file") > + > + subparser.add_argument("-o", "--outdir", dest="outdir", > default='.', > + help="name of directory to create image in") > + subparser.add_argument("-e", "--image-name", dest="image_name", > + help="name of the image to use the artifacts > from " > + "e.g. core-image-sato") > + subparser.add_argument("-r", "--rootfs-dir", > action=RootfsArgAction, > + help="path to the /rootfs dir to use as the " > + ".wks rootfs source") > + subparser.add_argument("-b", "--bootimg-dir", dest="bootimg_dir", > + help="path to the dir containing the boot > artifacts " > + "(e.g. /EFI or /syslinux dirs) to use as > the " > + ".wks bootimg source") > + subparser.add_argument("-k", "--kernel-dir", dest="kernel_dir", > + help="path to the dir containing the kernel to > use " > + "in the .wks bootimg") > + subparser.add_argument("-n", "--native-sysroot", > dest="native_sysroot", > + help="path to the native sysroot containing > the tools " > + "to use to build the image") > + subparser.add_argument("-s", "--skip-build-check", > dest="build_check", > + action="store_false", default=True, help="skip > the build check") > + subparser.add_argument("-f", "--build-rootfs", > action="store_true", help="build rootfs") > + subparser.add_argument("-c", "--compress-with", choices=("gzip", > "bzip2", "xz"), > + dest='compressor', > + help="compress image with specified > compressor") > + subparser.add_argument("-m", "--bmap", action="store_true", > help="generate .bmap") > + subparser.add_argument("--no-fstab-update" ,action="store_true", > + help="Do not change fstab file.") > + subparser.add_argument("-v", "--vars", dest='vars_dir', > + help="directory with <image>.env files that > store " > + "bitbake variables") > + subparser.add_argument("-D", "--debug", dest="debug", > action="store_true", > + default=False, help="output debug information") > + subparser.add_argument("-i", "--imager", dest="imager", > + default="direct", help="the wic imager plugin") > + return > + > + > +def wic_init_parser_list(subparser): > + subparser.add_argument("list_type", > + help="can be 'images' or 'source-plugins' " > + "to obtain a list. " > + "If value is a valid .wks image file") > + subparser.add_argument("help_for", default=[], nargs='*', > + help="If 'list_type' is a valid .wks image > file " > + "this value can be 'help' to show the > help information " > + "defined inside the .wks file") > + return > + > +def imgtype(arg): > + """ > + Custom type for ArgumentParser > + Converts path spec to named tuple: (image, partition, path) > + """ > + image = arg > + part = path = None > + if ':' in image: > + image, part = image.split(':') > + if '/' in part: > + part, path = part.split('/', 1) > + if not path: > + path = '/' > + > + if not os.path.isfile(image): > + err = "%s is not a regular file or symlink" % image > + raise argparse.ArgumentTypeError(err) > + > + return namedtuple('ImgType', 'image part path')(image, part, > path) + > +def wic_init_parser_ls(subparser): > + subparser.add_argument("path", type=imgtype, > + help="image spec: <image>[:<vfat > partition>[<path>]]") > + subparser.add_argument("-n", "--native-sysroot", > + help="path to the native sysroot containing > the tools") + > +def imgpathtype(arg): > + img = imgtype(arg) > + if img.part is None: > + raise argparse.ArgumentTypeError("partition number is not > specified") > + return img > + > +def wic_init_parser_cp(subparser): > + subparser.add_argument("src", > + help="image spec: <image>:<vfat > partition>[<path>] or <file>") > + subparser.add_argument("dest", > + help="image spec: <image>:<vfat > partition>[<path>] or <file>") > + subparser.add_argument("-n", "--native-sysroot", > + help="path to the native sysroot containing > the tools") + > +def wic_init_parser_rm(subparser): > + subparser.add_argument("path", type=imgpathtype, > + help="path: <image>:<vfat partition><path>") > + subparser.add_argument("-n", "--native-sysroot", > + help="path to the native sysroot containing > the tools") > + subparser.add_argument("-r", dest="recursive_delete", > action="store_true", default=False, > + help="remove directories and their contents > recursively, " > + " this only applies to ext* partition") > + > +def expandtype(rules): > + """ > + Custom type for ArgumentParser > + Converts expand rules to the dictionary {<partition>: size} > + """ > + if rules == 'auto': > + return {} > + result = {} > + for rule in rules.split(','): > + try: > + part, size = rule.split(':') > + except ValueError: > + raise argparse.ArgumentTypeError("Incorrect rule format: > %s" % rule) + > + if not part.isdigit(): > + raise argparse.ArgumentTypeError("Rule '%s': partition > number must be integer" % rule) + > + # validate size > + multiplier = 1 > + for suffix, mult in [('K', 1024), ('M', 1024 * 1024), ('G', > 1024 * 1024 * 1024)]: > + if size.upper().endswith(suffix): > + multiplier = mult > + size = size[:-1] > + break > + if not size.isdigit(): > + raise argparse.ArgumentTypeError("Rule '%s': size must > be integer" % rule) + > + result[int(part)] = int(size) * multiplier > + > + return result > + > +def wic_init_parser_write(subparser): > + subparser.add_argument("image", > + help="path to the wic image") > + subparser.add_argument("target", > + help="target file or device") > + subparser.add_argument("-e", "--expand", type=expandtype, > + help="expand rules: auto or > <partition>:<size>[,<partition>:<size>]") > + subparser.add_argument("-n", "--native-sysroot", > + help="path to the native sysroot containing > the tools") + > +def wic_init_parser_help(subparser): > + helpparsers = subparser.add_subparsers(dest='help_topic', > help=hlp.wic_usage) > + for helptopic in helptopics: > + helpparsers.add_parser(helptopic, > help=helptopics[helptopic][2]) > + return > + > + > +subcommands = { > + "create": [wic_create_subcommand, > + hlp.wic_create_usage, > + hlp.wic_create_help, > + wic_init_parser_create], > + "list": [wic_list_subcommand, > + hlp.wic_list_usage, > + hlp.wic_list_help, > + wic_init_parser_list], > + "ls": [wic_ls_subcommand, > + hlp.wic_ls_usage, > + hlp.wic_ls_help, > + wic_init_parser_ls], > + "cp": [wic_cp_subcommand, > + hlp.wic_cp_usage, > + hlp.wic_cp_help, > + wic_init_parser_cp], > + "rm": [wic_rm_subcommand, > + hlp.wic_rm_usage, > + hlp.wic_rm_help, > + wic_init_parser_rm], > + "write": [wic_write_subcommand, > + hlp.wic_write_usage, > + hlp.wic_write_help, > + wic_init_parser_write], > + "help": [wic_help_subcommand, > + wic_help_topic_usage, > + hlp.wic_help_help, > + wic_init_parser_help] > +} > + > + > +def init_parser(parser): > + parser.add_argument("--version", action="version", > + version="%(prog)s {version}".format(version=__version__)) > + parser.add_argument("-D", "--debug", dest="debug", > action="store_true", > + default=False, help="output debug information") > + > + subparsers = parser.add_subparsers(dest='command', > help=hlp.wic_usage) > + for subcmd in subcommands: > + subparser = subparsers.add_parser(subcmd, > help=subcommands[subcmd][2]) > + subcommands[subcmd][3](subparser) > + > +class WicArgumentParser(argparse.ArgumentParser): > + def format_help(self): > + return hlp.wic_help > + > def main(argv): > - parser = optparse.OptionParser(version="wic version %s" % > __version__, > - usage=hlp.wic_usage) > + parser = WicArgumentParser( > + description="wic version %s" % __version__) > + > + init_parser(parser) > > - parser.disable_interspersed_args() > + args = parser.parse_args(argv) > > - args = parser.parse_args(argv)[1] > + if args.debug: > + logger.setLevel(logging.DEBUG) > > - if len(args): > - if args[0] == "help": > - if len(args) == 1: > + if "command" in vars(args): > + if args.command == "help": > + if args.help_topic is None: > parser.print_help() > - raise WicError("help command requires parameter") > + elif args.help_topic in helptopics: > + hlpt = helptopics[args.help_topic] > + hlpt[0](hlpt[1], hlpt[2]) > + return 0 > + > + # validate wic cp src and dest parameter to identify which one > of it is > + # image and cast it into imgtype > + if args.command == "cp": > + if ":" in args.dest: > + args.dest = imgtype(args.dest) > + elif ":" in args.src: > + args.src = imgtype(args.src) > + else: > + raise argparse.ArgumentTypeError("no image or partition > number specified.") > return hlp.invoke_subcommand(args, parser, hlp.wic_help_usage, > subcommands) ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 01/10] wic: Update to the latest wic from openembedded core 2020-09-05 9:00 ` Henning Schild @ 2020-09-05 9:21 ` vijai kumar 0 siblings, 0 replies; 29+ messages in thread From: vijai kumar @ 2020-09-05 9:21 UTC (permalink / raw) To: Henning Schild; +Cc: Vijai Kumar K, isar-users [-- Attachment #1: Type: text/plain, Size: 85712 bytes --] On Sat, 5 Sep 2020 at 2:30 PM, Henning Schild <henning.schild@siemens.com> wrote: > I assume if i took that commit and copied it over i would end up with > > 100% the same patch, right? Yes. This patch is just copying the wic from the oe core. Although, we have one other separate patch (P9)which is cherry picked from oe core over the top of this copy based commit. Thanks, Vijai Kumar K > > > > > On Thu, 3 Sep 2020 00:26:15 +0530 > > Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> wrote: > > > > > Update to the latest wic from OE-core. > > > > > > OE-core Revision: 532ae127c52c9f7b1d2e4ca5cbca91881d23a2ac > > > > > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > > > --- > > > scripts/lib/scriptpath.py | 32 ++ > > > scripts/lib/wic/__init__.py | 14 +- > > > scripts/lib/wic/canned-wks/common.wks.inc | 2 +- > > > .../directdisk-bootloader-config.cfg | 8 +- > > > .../lib/wic/canned-wks/efi-bootdisk.wks.in | 3 + > > > scripts/lib/wic/canned-wks/mkhybridiso.wks | 2 +- > > > scripts/lib/wic/canned-wks/qemuriscv.wks | 3 + > > > .../lib/wic/canned-wks/qemux86-directdisk.wks | 2 +- > > > .../lib/wic/canned-wks/sdimage-bootpart.wks | 4 +- > > > .../lib/wic/canned-wks/systemd-bootdisk.wks | 4 +- > > > scripts/lib/wic/engine.py | 421 +++++++++++++++- > > > scripts/lib/wic/filemap.py | 170 ++++--- > > > scripts/lib/wic/help.py | 401 ++++++++++++++-- > > > scripts/lib/wic/ksparser.py | 115 +++-- > > > scripts/lib/wic/{utils => }/misc.py | 99 ++-- > > > scripts/lib/wic/partition.py | 234 ++++----- > > > scripts/lib/wic/pluginbase.py | 36 +- > > > scripts/lib/wic/plugins/imager/direct.py | 175 ++++--- > > > .../wic/plugins/source/bootimg-biosplusefi.py | 213 +++++++++ > > > scripts/lib/wic/plugins/source/bootimg-efi.py | 111 +++-- > > > .../wic/plugins/source/bootimg-partition.py | 153 ++++-- > > > .../lib/wic/plugins/source/bootimg-pcbios.py | 91 ++-- > > > scripts/lib/wic/plugins/source/fsimage.py | 56 --- > > > .../wic/plugins/source/isoimage-isohybrid.py | 185 +++---- > > > scripts/lib/wic/plugins/source/rawcopy.py | 44 +- > > > scripts/lib/wic/plugins/source/rootfs.py | 159 ++++-- > > > scripts/lib/wic/utils/__init__.py | 0 > > > scripts/lib/wic/utils/runner.py | 114 ----- > > > scripts/wic | 452 > > > +++++++++++++----- 29 files changed, 2283 insertions(+), 1020 > > > deletions(-) create mode 100644 scripts/lib/scriptpath.py > > > create mode 100644 scripts/lib/wic/canned-wks/efi-bootdisk.wks.in > > > create mode 100644 scripts/lib/wic/canned-wks/qemuriscv.wks > > > rename scripts/lib/wic/{utils => }/misc.py (70%) > > > create mode 100644 > > > scripts/lib/wic/plugins/source/bootimg-biosplusefi.py delete mode > > > 100644 scripts/lib/wic/plugins/source/fsimage.py delete mode 100644 > > > scripts/lib/wic/utils/__init__.py delete mode 100644 > > > scripts/lib/wic/utils/runner.py > > > > > > diff --git a/scripts/lib/scriptpath.py b/scripts/lib/scriptpath.py > > > new file mode 100644 > > > index 0000000..f32326d > > > --- /dev/null > > > +++ b/scripts/lib/scriptpath.py > > > @@ -0,0 +1,32 @@ > > > +# Path utility functions for OE python scripts > > > +# > > > +# Copyright (C) 2012-2014 Intel Corporation > > > +# Copyright (C) 2011 Mentor Graphics Corporation > > > +# > > > +# SPDX-License-Identifier: GPL-2.0-only > > > +# > > > + > > > +import sys > > > +import os > > > +import os.path > > > + > > > +def add_oe_lib_path(): > > > + basepath = os.path.abspath(os.path.dirname(__file__) + '/../..') > > > + newpath = basepath + '/meta/lib' > > > + sys.path.insert(0, newpath) > > > + > > > +def add_bitbake_lib_path(): > > > + basepath = os.path.abspath(os.path.dirname(__file__) + '/../..') > > > + bitbakepath = None > > > + if os.path.exists(basepath + '/bitbake/lib/bb'): > > > + bitbakepath = basepath + '/bitbake' > > > + else: > > > + # look for bitbake/bin dir in PATH > > > + for pth in os.environ['PATH'].split(':'): > > > + if os.path.exists(os.path.join(pth, '../lib/bb')): > > > + bitbakepath = os.path.abspath(os.path.join(pth, > > > '..')) > > > + break > > > + > > > + if bitbakepath: > > > + sys.path.insert(0, bitbakepath + '/lib') > > > + return bitbakepath > > > diff --git a/scripts/lib/wic/__init__.py b/scripts/lib/wic/__init__.py > > > index 85876b1..8556793 100644 > > > --- a/scripts/lib/wic/__init__.py > > > +++ b/scripts/lib/wic/__init__.py > > > @@ -1,20 +1,10 @@ > > > -#!/usr/bin/env python -tt > > > +#!/usr/bin/env python3 > > > # > > > # Copyright (c) 2007 Red Hat, Inc. > > > # Copyright (c) 2011 Intel, Inc. > > > # > > > -# This program is free software; you can redistribute it and/or > > > modify it -# under the terms of the GNU General Public License as > > > published by the Free -# Software Foundation; version 2 of the License > > > +# SPDX-License-Identifier: GPL-2.0-only > > > # > > > -# This program is distributed in the hope that it will be useful, but > > > -# WITHOUT ANY WARRANTY; without even the implied warranty of > > > MERCHANTABILITY -# or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > > > General Public License -# for more details. > > > -# > > > -# You should have received a copy of the GNU General Public License > > > along -# with this program; if not, write to the Free Software > > > Foundation, Inc., 59 -# Temple Place - Suite 330, Boston, MA > > > 02111-1307, USA. > > > class WicError(Exception): > > > pass > > > diff --git a/scripts/lib/wic/canned-wks/common.wks.inc > > > b/scripts/lib/wic/canned-wks/common.wks.inc index 5cf2fd1..89880b4 > > > 100644 --- a/scripts/lib/wic/canned-wks/common.wks.inc > > > +++ b/scripts/lib/wic/canned-wks/common.wks.inc > > > @@ -1,3 +1,3 @@ > > > # This file is included into 3 canned wks files from this directory > > > part /boot --source bootimg-pcbios --ondisk sda --label boot > > > --active --align 1024 -part / --source rootfs --ondisk sda > > > --fstype=ext4 --label platform --align 1024 +part / --source rootfs > > > --use-uuid --fstype=ext4 --label platform --align 1024 diff --git > > > a/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg > > > b/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg index > > > d5a07d2..c58e74a 100644 --- > > > a/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg +++ > > > b/scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg @@ > > > -12,16 +12,16 @@ DEFAULT Graphics console boot LABEL Graphics console > > > boot KERNEL /vmlinuz > > > -APPEND label=boot root=/dev/sda2 rootwait > > > +APPEND label=boot rootwait > > > > > > LABEL Serial console boot > > > KERNEL /vmlinuz > > > -APPEND label=boot root=/dev/sda2 rootwait console=ttyS0,115200 > > > +APPEND label=boot rootwait console=ttyS0,115200 > > > > > > LABEL Graphics console install > > > KERNEL /vmlinuz > > > -APPEND label=install root=/dev/sda2 rootwait > > > +APPEND label=install rootwait > > > > > > LABEL Serial console install > > > KERNEL /vmlinuz > > > -APPEND label=install root=/dev/sda2 rootwait console=ttyS0,115200 > > > +APPEND label=install rootwait console=ttyS0,115200 > > > diff --git a/scripts/lib/wic/canned-wks/efi-bootdisk.wks.in > > > b/scripts/lib/wic/canned-wks/efi-bootdisk.wks.in new file mode 100644 > > > index 0000000..7300e65 > > > --- /dev/null > > > +++ b/scripts/lib/wic/canned-wks/efi-bootdisk.wks.in > > > @@ -0,0 +1,3 @@ > > > +bootloader --ptable gpt > > > +part /boot --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/boot > > > --fstype=vfat --label boot --active --align 1024 --use-uuid > > > --overhead-factor 1.0 +part / --source rootfs --fstype=ext4 --label > > > root --align 1024 --exclude-path boot/ diff --git > > > a/scripts/lib/wic/canned-wks/mkhybridiso.wks > > > b/scripts/lib/wic/canned-wks/mkhybridiso.wks index 9d34e9b..48c5ac4 > > > 100644 --- a/scripts/lib/wic/canned-wks/mkhybridiso.wks +++ > > > b/scripts/lib/wic/canned-wks/mkhybridiso.wks @@ -2,6 +2,6 @@ # > > > long-description: Creates an EFI and legacy bootable hybrid ISO image > > > # which can be used on optical media as well as USB media. > > > -part /boot --source isoimage-isohybrid > > > --sourceparams="loader=grub-efi,image_name=HYBRID_ISO_IMG" --ondisk > > > cd --label HYBRIDISO --fstype=ext4 +part /boot --source > > > isoimage-isohybrid > > > --sourceparams="loader=grub-efi,image_name=HYBRID_ISO_IMG" --ondisk > > > cd --label HYBRIDISO bootloader --timeout=15 --append="" diff --git > > > a/scripts/lib/wic/canned-wks/qemuriscv.wks > > > b/scripts/lib/wic/canned-wks/qemuriscv.wks new file mode 100644 index > > > 0000000..12c68b7 --- /dev/null +++ > > > b/scripts/lib/wic/canned-wks/qemuriscv.wks @@ -0,0 +1,3 @@ > > > +# short-description: Create qcow2 image for RISC-V QEMU machines > > > + > > > +part / --source rootfs --fstype=ext4 --label root --align 4096 > > > --size 5G diff --git > > > a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks > > > b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks index > > > a6518a0..22b4521 100644 --- > > > a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks +++ > > > b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks @@ -4,5 +4,5 @@ > > > include common.wks.inc > > > > > > -bootloader --timeout=0 --append="vga=0 > > > uvesafb.mode_option=640x480-32 root=/dev/vda2 rw mem=256M > > > ip=192.168.7.2::192.168.7.1:255.255.255.0 oprofile.timer=1 > > > rootfstype=ext4 " +bootloader --timeout=0 --append="rw > > > oprofile.timer=1 rootfstype=ext4 " diff --git > > > a/scripts/lib/wic/canned-wks/sdimage-bootpart.wks > > > b/scripts/lib/wic/canned-wks/sdimage-bootpart.wks index > > > 7ffd632..63bc4da 100644 --- > > > a/scripts/lib/wic/canned-wks/sdimage-bootpart.wks +++ > > > b/scripts/lib/wic/canned-wks/sdimage-bootpart.wks @@ -2,5 +2,5 @@ # > > > long-description: Creates a partitioned SD card image. Boot files # > > > are located in the first vfat partition. -part /boot --source > > > bootimg-partition --ondisk mmcblk --fstype=vfat --label boot --active > > > --align 4 --size 16 -part / --source rootfs --ondisk mmcblk > > > --fstype=ext4 --label root --align 4 +part /boot --source > > > bootimg-partition --ondisk mmcblk0 --fstype=vfat --label boot > > > --active --align 4 --size 16 +part / --source rootfs --ondisk mmcblk0 > > > --fstype=ext4 --label root --align 4 diff --git > > > a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks > > > b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks index > > > 4bd9d6a..95d7b97 100644 --- > > > a/scripts/lib/wic/canned-wks/systemd-bootdisk.wks +++ > > > b/scripts/lib/wic/canned-wks/systemd-bootdisk.wks @@ -2,10 +2,10 @@ # > > > long-description: Creates a partitioned EFI disk image that the user > > > # can directly dd to boot media. The selected bootloader is > > > systemd-boot. -part /boot --source bootimg-efi > > > --sourceparams="loader=systemd-boot" --ondisk sda --label msdos > > > --active --align 1024 +part /boot --source bootimg-efi > > > --sourceparams="loader=systemd-boot" --ondisk sda --label msdos > > > --active --align 1024 --use-uuid part / --source rootfs --ondisk sda > > > --fstype=ext4 --label platform --align 1024 --use-uuid -part swap > > > --ondisk sda --size 44 --label swap1 --fstype=swap +part swap > > > --ondisk sda --size 44 --label swap1 --fstype=swap --use-uuid > > > bootloader --ptable gpt --timeout=5 --append="rootwait > > > rootfstype=ext4 console=ttyS0,115200 console=tty0" diff --git > > > a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py index > > > f59821f..018815b 100644 --- a/scripts/lib/wic/engine.py +++ > > > b/scripts/lib/wic/engine.py @@ -1,21 +1,7 @@ -# ex:ts=4:sw=4:sts=4:et > > > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > > > # > > > # Copyright (c) 2013, Intel Corporation. > > > -# All rights reserved. > > > # > > > -# This program is free software; you can redistribute it and/or > > > modify -# it under the terms of the G > <https://www.google.com/maps/search/the+terms+of+the+G?entry=gmail&source=g>NU > General Public License > > > version 2 as -# published by the Free Software Foundation. > > > -# > > > -# This program is distributed in the hope that it will be useful, > > > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > > > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > > -# GNU General Public License for more details. > > > -# > > > -# You should have received a copy of the GNU General Public License > > > along -# with this program; if not, write to the Free Software > > > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > > > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > > > # > > > # DESCRIPTION > > > > > > @@ -30,10 +16,18 @@ > > > > > > import logging > > > import os > > > +import tempfile > > > +import json > > > +import subprocess > > > +import re > > > + > > > +from collections import namedtuple, OrderedDict > > > +from distutils.spawn import find_executable > > > > > > from wic import WicError > > > +from wic.filemap import sparse_copy > > > from wic.pluginbase import PluginMgr > > > -from wic.utils.misc import get_bitbake_var > > > +from wic.misc import get_bitbake_var, exec_cmd > > > > > > logger = logging.getLogger('wic') > > > > > > @@ -82,7 +76,8 @@ def find_canned_image(scripts_path, wks_file): > > > for fname in files: > > > if fname.endswith("~") or fname.endswith("#"): > > > continue > > > - if fname.endswith(".wks") and wks_file + ".wks" == > > > fname: > > > + if ((fname.endswith(".wks") and wks_file + ".wks" == > > > fname) or \ > > > + (fname.endswith(".wks.in") and wks_file + > > > ".wks.in" == fname)): fullpath = os.path.join(canned_wks_dir, fname) > > > return fullpath > > > return None > > > @@ -99,7 +94,7 @@ def list_canned_images(scripts_path): > > > for fname in files: > > > if fname.endswith("~") or fname.endswith("#"): > > > continue > > > - if fname.endswith(".wks"): > > > + if fname.endswith(".wks") or > > > fname.endswith(".wks.in"): fullpath = os.path.join(canned_wks_dir, > > > fname) with open(fullpath) as wks: > > > for line in wks: > > > @@ -108,7 +103,7 @@ def list_canned_images(scripts_path): > > > if idx != -1: > > > desc = line[idx + > > > len("short-description:"):].strip() break > > > - basename = os.path.splitext(fname)[0] > > > + basename = fname.split('.')[0] > > > print(" %s\t\t%s" % (basename.ljust(30), desc)) > > > > > > > > > @@ -184,7 +179,7 @@ def wic_create(wks_file, rootfs_dir, bootimg_dir, > > > kernel_dir, if not os.path.exists(options.outdir): > > > os.makedirs(options.outdir) > > > > > > - pname = 'direct' > > > + pname = options.imager > > > plugin_class = PluginMgr.get_plugins('imager').get(pname) > > > if not plugin_class: > > > raise WicError('Unknown plugin: %s' % pname) > > > @@ -201,17 +196,18 @@ def wic_list(args, scripts_path): > > > """ > > > Print the list of images or source plugins. > > > """ > > > - if len(args) < 1: > > > + if args.list_type is None: > > > return False > > > > > > - if args == ["images"]: > > > + if args.list_type == "images": > > > + > > > list_canned_images(scripts_path) > > > return True > > > - elif args == ["source-plugins"]: > > > + elif args.list_type == "source-plugins": > > > list_source_plugins() > > > return True > > > - elif len(args) == 2 and args[1] == "help": > > > - wks_file = args[0] > > > + elif len(args.help_for) == 1 and args.help_for[0] == 'help': > > > + wks_file = args.list_type > > > fullpath = find_canned_image(scripts_path, wks_file) > > > if not fullpath: > > > raise WicError("No image named %s found, exiting. " > > > @@ -224,6 +220,381 @@ def wic_list(args, scripts_path): > > > > > > return False > > > > > > + > > > +class Disk: > > > + def __init__(self, imagepath, native_sysroot, fstypes=('fat', > > > 'ext')): > > > + self.imagepath = imagepath > > > + self.native_sysroot = native_sysroot > > > + self.fstypes = fstypes > > > + self._partitions = None > > > + self._partimages = {} > > > + self._lsector_size = None > > > + self._psector_size = None > > > + self._ptable_format = None > > > + > > > + # find parted > > > + # read paths from $PATH environment variable > > > + # if it fails, use hardcoded paths > > > + pathlist = "/bin:/usr/bin:/usr/sbin:/sbin/" > > > + try: > > > + self.paths = os.environ['PATH'] + ":" + pathlist > > > + except KeyError: > > > + self.paths = pathlist > > > + > > > + if native_sysroot: > > > + for path in pathlist.split(':'): > > > + self.paths = "%s%s:%s" % (native_sysroot, path, > > > self.paths) + > > > + self.parted = find_executable("parted", self.paths) > > > + if not self.parted: > > > + raise WicError("Can't find executable parted") > > > + > > > + self.partitions = self.get_partitions() > > > + > > > + def __del__(self): > > > + for path in self._partimages.values(): > > > + os.unlink(path) > > > + > > > + def get_partitions(self): > > > + if self._partitions is None: > > > + self._partitions = OrderedDict() > > > + out = exec_cmd("%s -sm %s unit B print" % (self.parted, > > > self.imagepath)) > > > + parttype = namedtuple("Part", "pnum start end size > > > fstype") > > > + splitted = out.splitlines() > > > + # skip over possible errors in exec_cmd output > > > + try: > > > + idx =splitted.index("BYT;") > > > + except ValueError: > > > + raise WicError("Error getting partition information > > > from %s" % (self.parted)) > > > + lsector_size, psector_size, self._ptable_format = > > > splitted[idx + 1].split(":")[3:6] > > > + self._lsector_size = int(lsector_size) > > > + self._psector_size = int(psector_size) > > > + for line in splitted[idx + 2:]: > > > + pnum, start, end, size, fstype = line.split(':')[:5] > > > + partition = parttype(int(pnum), int(start[:-1]), > > > int(end[:-1]), > > > + int(size[:-1]), fstype) > > > + self._partitions[pnum] = partition > > > + > > > + return self._partitions > > > + > > > + def __getattr__(self, name): > > > + """Get path to the executable in a lazy way.""" > > > + if name in ("mdir", "mcopy", "mdel", "mdeltree", "sfdisk", > > > "e2fsck", > > > + "resize2fs", "mkswap", "mkdosfs", > > > "debugfs","blkid"): > > > + aname = "_%s" % name > > > + if aname not in self.__dict__: > > > + setattr(self, aname, find_executable(name, > > > self.paths)) > > > + if aname not in self.__dict__ or > > > self.__dict__[aname] is None: > > > + raise WicError("Can't find executable > > > '{}'".format(name)) > > > + return self.__dict__[aname] > > > + return self.__dict__[name] > > > + > > > + def _get_part_image(self, pnum): > > > + if pnum not in self.partitions: > > > + raise WicError("Partition %s is not in the image" % pnum) > > > + part = self.partitions[pnum] > > > + # check if fstype is supported > > > + for fstype in self.fstypes: > > > + if part.fstype.startswith(fstype): > > > + break > > > + else: > > > + raise WicError("Not supported fstype: > > > {}".format(part.fstype)) > > > + if pnum not in self._partimages: > > > + tmpf = tempfile.NamedTemporaryFile(prefix="wic-part") > > > + dst_fname = tmpf.name > > > + tmpf.close() > > > + sparse_copy(self.imagepath, dst_fname, skip=part.start, > > > length=part.size) > > > + self._partimages[pnum] = dst_fname > > > + > > > + return self._partimages[pnum] > > > + > > > + def _put_part_image(self, pnum): > > > + """Put partition image into partitioned image.""" > > > + sparse_copy(self._partimages[pnum], self.imagepath, > > > + seek=self.partitions[pnum].start) > > > + > > > + def dir(self, pnum, path): > > > + if pnum not in self.partitions: > > > + raise WicError("Partition %s is not in the image" % pnum) > > > + > > > + if self.partitions[pnum].fstype.startswith('ext'): > > > + return exec_cmd("{} {} -R 'ls -l > > > {}'".format(self.debugfs, > > > + > > > self._get_part_image(pnum), > > > + path), > > > as_shell=True) > > > + else: # fat > > > + return exec_cmd("{} -i {} ::{}".format(self.mdir, > > > + > > > self._get_part_image(pnum), > > > + path)) > > > + > > > + def copy(self, src, dest): > > > + """Copy partition image into wic image.""" > > > + pnum = dest.part if isinstance(src, str) else src.part > > > + > > > + if self.partitions[pnum].fstype.startswith('ext'): > > > + if isinstance(src, str): > > > + cmd = "printf 'cd {}\nwrite {} {}\n' | {} -w {}".\ > > > + format(os.path.dirname(dest.path), src, > > > os.path.basename(src), > > > + self.debugfs, > > > self._get_part_image(pnum)) > > > + else: # copy from wic > > > + # run both dump and rdump to support both files and > > > directory > > > + cmd = "printf 'cd {}\ndump /{} {}\nrdump /{} {}\n' | > > > {} {}".\ > > > + format(os.path.dirname(src.path), src.path, > > > + dest, src.path, dest, self.debugfs, > > > + self._get_part_image(pnum)) > > > + else: # fat > > > + if isinstance(src, str): > > > + cmd = "{} -i {} -snop {} ::{}".format(self.mcopy, > > > + > > > self._get_part_image(pnum), > > > + src, dest.path) > > > + else: > > > + cmd = "{} -i {} -snop ::{} {}".format(self.mcopy, > > > + > > > self._get_part_image(pnum), > > > + src.path, dest) > > > + > > > + exec_cmd(cmd, as_shell=True) > > > + self._put_part_image(pnum) > > > + > > > + def remove_ext(self, pnum, path, recursive): > > > + """ > > > + Remove files/dirs and their contents from the partition. > > > + This only applies to ext* partition. > > > + """ > > > + abs_path = re.sub('\/\/+', '/', path) > > > + cmd = "{} {} -wR 'rm \"{}\"'".format(self.debugfs, > > > + > > > self._get_part_image(pnum), > > > + abs_path) > > > + out = exec_cmd(cmd , as_shell=True) > > > + for line in out.splitlines(): > > > + if line.startswith("rm:"): > > > + if "file is a directory" in line: > > > + if recursive: > > > + # loop through content and delete them one > > > by one if > > > + # flaged with -r > > > + subdirs = iter(self.dir(pnum, > > > abs_path).splitlines()) > > > + next(subdirs) > > > + for subdir in subdirs: > > > + dir = subdir.split(':')[1].split(" ", > > > 1)[1] > > > + if not dir == "." and not dir == "..": > > > + self.remove_ext(pnum, "%s/%s" % > > > (abs_path, dir), recursive) + > > > + rmdir_out = exec_cmd("{} {} -wR 'rmdir > > > \"{}\"'".format(self.debugfs, > > > + > > > self._get_part_image(pnum), > > > + > > > abs_path.rstrip('/')) > > > + , as_shell=True) > > > + > > > + for rmdir_line in rmdir_out.splitlines(): > > > + if "directory not empty" in rmdir_line: > > > + raise WicError("Could not complete > > > operation: \n%s \n" > > > + "use -r to remove > > > non-empty directory" % rmdir_line) > > > + if rmdir_line.startswith("rmdir:"): > > > + raise WicError("Could not complete > > > operation: \n%s " > > > + "\n%s" % (str(line), > > > rmdir_line)) + > > > + else: > > > + raise WicError("Could not complete operation: > > > \n%s " > > > + "\nUnable to remove %s" % > > > (str(line), abs_path)) + > > > + def remove(self, pnum, path, recursive): > > > + """Remove files/dirs from the partition.""" > > > + partimg = self._get_part_image(pnum) > > > + if self.partitions[pnum].fstype.startswith('ext'): > > > + self.remove_ext(pnum, path, recursive) > > > + > > > + else: # fat > > > + cmd = "{} -i {} ::{}".format(self.mdel, partimg, path) > > > + try: > > > + exec_cmd(cmd) > > > + except WicError as err: > > > + if "not found" in str(err) or "non empty" in > > > str(err): > > > + # mdel outputs 'File ... not found' or > > > 'directory .. non empty" > > > + # try to use mdeltree as path could be a > > > directory > > > + cmd = "{} -i {} ::{}".format(self.mdeltree, > > > + partimg, path) > > > + exec_cmd(cmd) > > > + else: > > > + raise err > > > + self._put_part_image(pnum) > > > + > > > + def write(self, target, expand): > > > + """Write disk image to the media or file.""" > > > + def write_sfdisk_script(outf, parts): > > > + for key, val in parts['partitiontable'].items(): > > > + if key in ("partitions", "device", "firstlba", > > > "lastlba"): > > > + continue > > > + if key == "id": > > > + key = "label-id" > > > + outf.write("{}: {}\n".format(key, val)) > > > + outf.write("\n") > > > + for part in parts['partitiontable']['partitions']: > > > + line = '' > > > + for name in ('attrs', 'name', 'size', 'type', > > > 'uuid'): > > > + if name == 'size' and part['type'] == 'f': > > > + # don't write size for extended partition > > > + continue > > > + val = part.get(name) > > > + if val: > > > + line += '{}={}, '.format(name, val) > > > + if line: > > > + line = line[:-2] # strip ', ' > > > + if part.get('bootable'): > > > + line += ' ,bootable' > > > + outf.write("{}\n".format(line)) > > > + outf.flush() > > > + > > > + def read_ptable(path): > > > + out = exec_cmd("{} -J {}".format(self.sfdisk, path)) > > > + return json.loads(out) > > > + > > > + def write_ptable(parts, target): > > > + with tempfile.NamedTemporaryFile(prefix="wic-sfdisk-", > > > mode='w') as outf: > > > + write_sfdisk_script(outf, parts) > > > + cmd = "{} --no-reread {} < {} ".format(self.sfdisk, > > > target, outf.name) > > > + exec_cmd(cmd, as_shell=True) > > > + > > > + if expand is None: > > > + sparse_copy(self.imagepath, target) > > > + else: > > > + # copy first sectors that may contain bootloader > > > + sparse_copy(self.imagepath, target, length=2048 * > > > self._lsector_size) + > > > + # copy source partition table to the target > > > + parts = read_ptable(self.imagepath) > > > + write_ptable(parts, target) > > > + > > > + # get size of unpartitioned space > > > + free = None > > > + for line in exec_cmd("{} -F {}".format(self.sfdisk, > > > target)).splitlines(): > > > + if line.startswith("Unpartitioned space ") and > > > line.endswith("sectors"): > > > + free = int(line.split()[-2]) > > > + # Align free space to a 2048 sector boundary. > > > YOCTO #12840. > > > + free = free - (free % 2048) > > > + if free is None: > > > + raise WicError("Can't get size of unpartitioned > > > space") + > > > + # calculate expanded partitions sizes > > > + sizes = {} > > > + num_auto_resize = 0 > > > + for num, part in > > > enumerate(parts['partitiontable']['partitions'], 1): > > > + if num in expand: > > > + if expand[num] != 0: # don't resize partition if > > > size is set to 0 > > > + sectors = expand[num] // self._lsector_size > > > + free -= sectors - part['size'] > > > + part['size'] = sectors > > > + sizes[num] = sectors > > > + elif part['type'] != 'f': > > > + sizes[num] = -1 > > > + num_auto_resize += 1 > > > + > > > + for num, part in > > > enumerate(parts['partitiontable']['partitions'], 1): > > > + if sizes.get(num) == -1: > > > + part['size'] += free // num_auto_resize > > > + > > > + # write resized partition table to the target > > > + write_ptable(parts, target) > > > + > > > + # read resized partition table > > > + parts = read_ptable(target) > > > + > > > + # copy partitions content > > > + for num, part in > > > enumerate(parts['partitiontable']['partitions'], 1): > > > + pnum = str(num) > > > + fstype = self.partitions[pnum].fstype > > > + > > > + # copy unchanged partition > > > + if part['size'] == self.partitions[pnum].size // > > > self._lsector_size: > > > + logger.info("copying unchanged partition > > > {}".format(pnum)) > > > + sparse_copy(self._get_part_image(pnum), target, > > > seek=part['start'] * self._lsector_size) > > > + continue > > > + > > > + # resize or re-create partitions > > > + if fstype.startswith('ext') or > > > fstype.startswith('fat') or \ > > > + fstype.startswith('linux-swap'): > > > + > > > + partfname = None > > > + with > > > tempfile.NamedTemporaryFile(prefix="wic-part{}-".format(pnum)) as > > > partf: > > > + partfname = partf.name > > > + > > > + if fstype.startswith('ext'): > > > + logger.info("resizing ext partition > > > {}".format(pnum)) > > > + partimg = self._get_part_image(pnum) > > > + sparse_copy(partimg, partfname) > > > + exec_cmd("{} -pf {}".format(self.e2fsck, > > > partfname)) > > > + exec_cmd("{} {} {}s".format(\ > > > + self.resize2fs, partfname, > > > part['size'])) > > > + elif fstype.startswith('fat'): > > > + logger.info("copying content of the fat > > > partition {}".format(pnum)) > > > + with > > > tempfile.TemporaryDirectory(prefix='wic-fatdir-') as tmpdir: > > > + # copy content to the temporary directory > > > + cmd = "{} -snompi {} :: > > > {}".format(self.mcopy, > > > + > > > self._get_part_image(pnum), > > > + > > > tmpdir) > > > + exec_cmd(cmd) > > > + # create new msdos partition > > > + label = part.get("name") > > > + label_str = "-n {}".format(label) if > > > label else '' + > > > + cmd = "{} {} -C {} > > > {}".format(self.mkdosfs, label_str, partfname, > > > + > > > part['size']) > > > + exec_cmd(cmd) > > > + # copy content from the temporary > > > directory to the new partition > > > + cmd = "{} -snompi {} {}/* > > > ::".format(self.mcopy, partfname, tmpdir) > > > + exec_cmd(cmd, as_shell=True) > > > + elif fstype.startswith('linux-swap'): > > > + logger.info("creating swap partition > > > {}".format(pnum)) > > > + label = part.get("name") > > > + label_str = "-L {}".format(label) if label > > > else '' > > > + out = exec_cmd("{} --probe > > > {}".format(self.blkid, self._get_part_image(pnum))) > > > + uuid = > > > out[out.index("UUID=\"")+6:out.index("UUID=\"")+42] > > > + uuid_str = "-U {}".format(uuid) if uuid else > > > '' > > > + with open(partfname, 'w') as sparse: > > > + os.ftruncate(sparse.fileno(), > > > part['size'] * self._lsector_size) > > > + exec_cmd("{} {} {} {}".format(self.mkswap, > > > label_str, uuid_str, partfname)) > > > + sparse_copy(partfname, target, > > > seek=part['start'] * self._lsector_size) > > > + os.unlink(partfname) > > > + elif part['type'] != 'f': > > > + logger.warning("skipping partition {}: > > > unsupported fstype {}".format(pnum, fstype)) + > > > +def wic_ls(args, native_sysroot): > > > + """List contents of partitioned image or vfat partition.""" > > > + disk = Disk(args.path.image, native_sysroot) > > > + if not args.path.part: > > > + if disk.partitions: > > > + print('Num Start End Size > > > Fstype') > > > + for part in disk.partitions.values(): > > > + print("{:2d} {:12d} {:12d} {:12d} {}".format(\ > > > + part.pnum, part.start, part.end, > > > + part.size, part.fstype)) > > > + else: > > > + path = args.path.path or '/' > > > + print(disk.dir(args.path.part, path)) > > > + > > > +def wic_cp(args, native_sysroot): > > > + """ > > > + Copy file or directory to/from the vfat/ext partition of > > > + partitioned image. > > > + """ > > > + if isinstance(args.dest, str): > > > + disk = Disk(args.src.image, native_sysroot) > > > + else: > > > + disk = Disk(args.dest.image, native_sysroot) > > > + disk.copy(args.src, args.dest) > > > + > > > + > > > +def wic_rm(args, native_sysroot): > > > + """ > > > + Remove files or directories from the vfat partition of > > > + partitioned image. > > > + """ > > > + disk = Disk(args.path.image, native_sysroot) > > > + disk.remove(args.path.part, args.path.path, > > > args.recursive_delete) + > > > +def wic_write(args, native_sysroot): > > > + """ > > > + Write image to a target device. > > > + """ > > > + disk = Disk(args.image, native_sysroot, ('fat', 'ext', > > > 'linux-swap')) > > > + disk.write(args.target, args.expand) > > > + > > > def find_canned(scripts_path, file_name): > > > """ > > > Find a file either by its path or by name in the canned files > > > dir. diff --git a/scripts/lib/wic/filemap.py > > > b/scripts/lib/wic/filemap.py index 080668e..4d9da28 100644 > > > --- a/scripts/lib/wic/filemap.py > > > +++ b/scripts/lib/wic/filemap.py > > > @@ -1,13 +1,8 @@ > > > +# > > > # Copyright (c) 2012 Intel, Inc. > > > # > > > -# This program is free software; you can redistribute it and/or > > > modify -# it under the terms of the GNU General Public License, > > > version 2, -# as published by the Free Software Foundation. > > > +# SPDX-License-Identifier: GPL-2.0-only > > > # > > > -# This program is distributed in the hope that it will be useful, but > > > -# WITHOUT ANY WARRANTY; without even the implied warranty of > > > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > > > -# General Public License for more details. > > > > > > """ > > > This module implements python implements a way to get file block. > > > Two methods @@ -22,6 +17,7 @@ and returns an instance of the class. > > > # * Too many instance attributes (R0902) > > > # pylint: disable=R0902 > > > > > > +import errno > > > import os > > > import struct > > > import array > > > @@ -34,14 +30,23 @@ def get_block_size(file_obj): > > > Returns block size for file object 'file_obj'. Errors are > > > indicated by the 'IOError' exception. > > > """ > > > - > > > - from fcntl import ioctl > > > - import struct > > > - > > > # Get the block size of the host file-system for the image file > > > by calling # the FIGETBSZ ioctl (number 2). > > > - binary_data = ioctl(file_obj, 2, struct.pack('I', 0)) > > > - return struct.unpack('I', binary_data)[0] > > > + try: > > > + binary_data = fcntl.ioctl(file_obj, 2, struct.pack('I', 0)) > > > + bsize = struct.unpack('I', binary_data)[0] > > > + except OSError: > > > + bsize = None > > > + > > > + # If ioctl causes OSError or give bsize to zero failback to > > > os.fstat > > > + if not bsize: > > > + import os > > > + stat = os.fstat(file_obj.fileno()) > > > + if hasattr(stat, 'st_blksize'): > > > + bsize = stat.st_blksize > > > + else: > > > + raise IOError("Unable to determine block size") > > > + return bsize > > > > > > class ErrorNotSupp(Exception): > > > """ > > > @@ -137,15 +142,6 @@ class _FilemapBase(object): > > > > > > raise Error("the method is not implemented") > > > > > > - def block_is_unmapped(self, block): # pylint: disable=W0613,R0201 > > > - """ > > > - This method has has to be implemented by child classes. It > > > returns > > > - 'True' if block number 'block' of the image file is not > > > mapped (hole) > > > - and 'False' otherwise. > > > - """ > > > - > > > - raise Error("the method is not implemented") > > > - > > > def get_mapped_ranges(self, start, count): # pylint: > > > disable=W0613,R0201 """ > > > This method has has to be implemented by child classes. This > > > is a @@ -159,15 +155,6 @@ class _FilemapBase(object): > > > > > > raise Error("the method is not implemented") > > > > > > - def get_unmapped_ranges(self, start, count): # pylint: > > > disable=W0613,R0201 > > > - """ > > > - This method has has to be implemented by child classes. Just > > > like > > > - 'get_mapped_ranges()', but yields unmapped block ranges > > > instead > > > - (holes). > > > - """ > > > - > > > - raise Error("the method is not implemented") > > > - > > > > > > # The 'SEEK_HOLE' and 'SEEK_DATA' options of the file seek system > > > call _SEEK_DATA = 3 > > > @@ -185,9 +172,9 @@ def _lseek(file_obj, offset, whence): > > > except OSError as err: > > > # The 'lseek' system call returns the ENXIO if there is no > > > data or # hole starting from the specified offset. > > > - if err.errno == os.errno.ENXIO: > > > + if err.errno == errno.ENXIO: > > > return -1 > > > - elif err.errno == os.errno.EINVAL: > > > + elif err.errno == errno.EINVAL: > > > raise ErrorNotSupp("the kernel or file-system does not > > > support " "\"SEEK_HOLE\" and \"SEEK_DATA\"") > > > else: > > > @@ -228,7 +215,7 @@ class FilemapSeek(_FilemapBase): > > > try: > > > tmp_obj = tempfile.TemporaryFile("w+", dir=directory) > > > except IOError as err: > > > - raise ErrorNotSupp("cannot create a temporary in \"%s\": > > > %s" > > > + raise ErrorNotSupp("cannot create a temporary in \"%s\": > > > %s" \ % (directory, err)) > > > > > > try: > > > @@ -260,15 +247,10 @@ class FilemapSeek(_FilemapBase): > > > % (block, result)) > > > return result > > > > > > - def block_is_unmapped(self, block): > > > - """Refer the '_FilemapBase' class for the documentation.""" > > > - return not self.block_is_mapped(block) > > > - > > > def _get_ranges(self, start, count, whence1, whence2): > > > """ > > > - This function implements 'get_mapped_ranges()' and > > > - 'get_unmapped_ranges()' depending on what is passed in the > > > 'whence1' > > > - and 'whence2' arguments. > > > + This function implements 'get_mapped_ranges()' depending > > > + on what is passed in the 'whence1' and 'whence2' arguments. > > > """ > > > > > > assert whence1 != whence2 > > > @@ -298,12 +280,6 @@ class FilemapSeek(_FilemapBase): > > > % (start, count, start + count - 1)) > > > return self._get_ranges(start, count, _SEEK_DATA, _SEEK_HOLE) > > > > > > - def get_unmapped_ranges(self, start, count): > > > - """Refer the '_FilemapBase' class for the documentation.""" > > > - self._log.debug("FilemapSeek: get_unmapped_ranges(%d, > > > %d(%d))" > > > - % (start, count, start + count - 1)) > > > - return self._get_ranges(start, count, _SEEK_HOLE, _SEEK_DATA) > > > - > > > > > > # Below goes the FIEMAP ioctl implementation, which is not very > > > readable # because it deals with the rather complex FIEMAP ioctl. To > > > understand the @@ -390,12 +366,12 @@ class > > > FilemapFiemap(_FilemapBase): except IOError as err: > > > # Note, the FIEMAP ioctl is supported by the Linux > > > kernel starting # from version 2.6.28 (year 2008). > > > - if err.errno == os.errno.EOPNOTSUPP: > > > + if err.errno == errno.EOPNOTSUPP: > > > errstr = "FilemapFiemap: the FIEMAP ioctl is not > > > supported " \ "by the file-system" > > > self._log.debug(errstr) > > > raise ErrorNotSupp(errstr) > > > - if err.errno == os.errno.ENOTTY: > > > + if err.errno == errno.ENOTTY: > > > errstr = "FilemapFiemap: the FIEMAP ioctl is not > > > supported " \ "by the kernel" > > > self._log.debug(errstr) > > > @@ -417,10 +393,6 @@ class FilemapFiemap(_FilemapBase): > > > % (block, result)) > > > return result > > > > > > - def block_is_unmapped(self, block): > > > - """Refer the '_FilemapBase' class for the documentation.""" > > > - return not self.block_is_mapped(block) > > > - > > > def _unpack_fiemap_extent(self, index): > > > """ > > > Unpack a 'struct fiemap_extent' structure object number > > > 'index' from @@ -497,23 +469,28 @@ class FilemapFiemap(_FilemapBase): > > > % (first_prev, last_prev)) > > > yield (first_prev, last_prev) > > > > > > - def get_unmapped_ranges(self, start, count): > > > +class FilemapNobmap(_FilemapBase): > > > + """ > > > + This class is used when both the 'SEEK_DATA/HOLE' and FIEMAP are > > > not > > > + supported by the filesystem or kernel. > > > + """ > > > + > > > + def __init__(self, image, log=None): > > > """Refer the '_FilemapBase' class for the documentation.""" > > > - self._log.debug("FilemapFiemap: get_unmapped_ranges(%d, > > > %d(%d))" > > > - % (start, count, start + count - 1)) > > > - hole_first = start > > > - for first, last in self._do_get_mapped_ranges(start, count): > > > - if first > hole_first: > > > - self._log.debug("FilemapFiemap: yielding range (%d, > > > %d)" > > > - % (hole_first, first - 1)) > > > - yield (hole_first, first - 1) > > > > > > - hole_first = last + 1 > > > + # Call the base class constructor first > > > + _FilemapBase.__init__(self, image, log) > > > + self._log.debug("FilemapNobmap: initializing") > > > > > > - if hole_first < start + count: > > > - self._log.debug("FilemapFiemap: yielding range (%d, %d)" > > > - % (hole_first, start + count - 1)) > > > - yield (hole_first, start + count - 1) > > > + def block_is_mapped(self, block): > > > + """Refer the '_FilemapBase' class for the documentation.""" > > > + return True > > > + > > > + def get_mapped_ranges(self, start, count): > > > + """Refer the '_FilemapBase' class for the documentation.""" > > > + self._log.debug("FilemapNobmap: get_mapped_ranges(%d, > > > %d(%d))" > > > + % (start, count, start + count - 1)) > > > + yield (start, start + count -1) > > > > > > def filemap(image, log=None): > > > """ > > > @@ -528,26 +505,56 @@ def filemap(image, log=None): > > > try: > > > return FilemapFiemap(image, log) > > > except ErrorNotSupp: > > > - return FilemapSeek(image, log) > > > + try: > > > + return FilemapSeek(image, log) > > > + except ErrorNotSupp: > > > + return FilemapNobmap(image, log) > > > > > > -def sparse_copy(src_fname, dst_fname, offset=0, skip=0): > > > - """Efficiently copy sparse file to or into another file.""" > > > - fmap = filemap(src_fname) > > > +def sparse_copy(src_fname, dst_fname, skip=0, seek=0, > > > + length=0, api=None): > > > + """ > > > + Efficiently copy sparse file to or into another file. > > > + > > > + src_fname: path to source file > > > + dst_fname: path > <https://www.google.com/maps/search/dst_fname:+path?entry=gmail&source=g> > to destination file > > > + skip: skip N bytes at thestart of src > > > + seek: seek N bytes from the start of dst > > > + length: read N bytes from src and write them to dst > > > + api: FilemapFiemap or FilemapSeek object > > > + """ > > > + if not api: > > > + api = filemap > > > + fmap = api(src_fname) > > > try: > > > dst_file = open(dst_fname, 'r+b') > > > except IOError: > > > dst_file = open(dst_fname, 'wb') > > > - dst_file.truncate(os.path.getsize(src_fname)) > > > + if length: > > > + dst_size = length + seek > > > + else: > > > + dst_size = os.path.getsize(src_fname) + seek - skip > > > + dst_file.truncate(dst_size) > > > > > > + written = 0 > > > for first, last in fmap.get_mapped_ranges(0, fmap.blocks_cnt): > > > start = first * fmap.block_size > > > end = (last + 1) * fmap.block_size > > > > > > + if skip >= end: > > > + continue > > > + > > > if start < skip < end: > > > - fmap._f_image.seek(skip, os.SEEK_SET) > > > - else: > > > - fmap._f_image.seek(start, os.SEEK_SET) > > > - dst_file.seek(offset + start, os.SEEK_SET) > > > + start = skip > > > + > > > + fmap._f_image.seek(start, os.SEEK_SET) > > > + > > > + written += start - skip - written > > > + if length and written >= length: > > > + dst_file.seek(seek + length, os.SEEK_SET) > > > + dst_file.close() > > > + return > > > + > > > + dst_file.seek(seek + start - skip, os.SEEK_SET) > > > > > > chunk_size = 1024 * 1024 > > > to_read = end - start > > > @@ -556,7 +563,14 @@ def sparse_copy(src_fname, dst_fname, offset=0, > > > skip=0): while read < to_read: > > > if read + chunk_size > to_read: > > > chunk_size = to_read - read > > > - chunk = fmap._f_image.read(chunk_size) > > > + size = chunk_size > > > + if length and written + size > length: > > > + size = length - written > > > + chunk = fmap._f_image.read(size) > > > dst_file.write(chunk) > > > - read += chunk_size > > > + read += size > > > + written += size > > > + if written == length: > > > + dst_file.close() > > > + return > > > dst_file.close() > > > diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py > > > index 148da89..bd3a2b9 100644 > > > --- a/scripts/lib/wic/help.py > > > +++ b/scripts/lib/wic/help.py > > > @@ -1,21 +1,6 @@ > > > -# ex:ts=4:sw=4:sts=4:et > > > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > > > -# > > > # Copyright (c) 2013, Intel Corporation. > > > -# All rights reserved. > > > -# > > > -# This program is free software; you can redistribute it and/or > > > modify -# it under the terms of the GNU General Public License > > > version 2 as -# published by the Free Software Foundation. > > > # > > > -# This program is distributed in the hope that it will be useful, > > > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > > > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > > -# GNU General Public License for more details. > > > -# > > > -# You should have received a copy of the GNU General Public License > > > along -# with this program; if not, write to the Free Software > > > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > > > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > > > # > > > # DESCRIPTION > > > # This module implements some basic help invocation functions along > > > @@ -56,7 +41,7 @@ def wic_help(args, usage_str, subcommands): > > > """ > > > Subcommand help dispatcher. > > > """ > > > - if len(args) == 1 or not display_help(args[1], subcommands): > > > + if args.help_topic == None or not display_help(args.help_topic, > > > subcommands): print(usage_str) > > > > > > > > > @@ -82,19 +67,20 @@ def invoke_subcommand(args, parser, > > > main_command_usage, subcommands): Dispatch to subcommand handler > > > borrowed from combo-layer. Should use argparse, but has to work in > > > 2.6. """ > > > - if not args: > > > + if not args.command: > > > logger.error("No subcommand specified, exiting") > > > parser.print_help() > > > return 1 > > > - elif args[0] == "help": > > > + elif args.command == "help": > > > wic_help(args, main_command_usage, subcommands) > > > - elif args[0] not in subcommands: > > > - logger.error("Unsupported subcommand %s, exiting\n", args[0]) > > > + elif args.command not in subcommands: > > > + logger.error("Unsupported subcommand %s, exiting\n", > > > args.command) parser.print_help() > > > return 1 > > > else: > > > - usage = subcommands.get(args[0], subcommand_error)[1] > > > - subcommands.get(args[0], subcommand_error)[0](args[1:], > > > usage) > > > + subcmd = subcommands.get(args.command, subcommand_error) > > > + usage = subcmd[1] > > > + subcmd[0](args, usage) > > > > > > > > > ## > > > @@ -130,10 +116,10 @@ wic_create_usage = """ > > > Create a new OpenEmbedded image > > > > > > usage: wic create <wks file or image name> [-o <DIRNAME> | --outdir > > > <DIRNAME>] > > > - [-i <JSON PROPERTY FILE> | --infile <JSON PROPERTY_FILE>] > > > [-e | --image-name] [-s, --skip-build-check] [-D, > > > --debug] [-r, --rootfs-dir] [-b, --bootimg-dir] > > > [-k, --kernel-dir] [-n, --native-sysroot] [-f, > > > --build-rootfs] > > > + [-c, --compress-with] [-m, --bmap] > > > > > > This command creates an OpenEmbedded image based on the 'OE > > > kickstart commands' found in the <wks file>. > > > @@ -154,7 +140,7 @@ SYNOPSIS > > > [-e | --image-name] [-s, --skip-build-check] [-D, --debug] > > > [-r, --rootfs-dir] [-b, --bootimg-dir] > > > [-k, --kernel-dir] [-n, --native-sysroot] [-f, > > > --build-rootfs] > > > - [-c, --compress-with] [-m, --bmap] > > > + [-c, --compress-with] [-m, --bmap] [--no-fstab-update] > > > > > > DESCRIPTION > > > This command creates an OpenEmbedded image based on the 'OE > > > @@ -226,6 +212,11 @@ DESCRIPTION > > > > > > The -m option is used to produce .bmap file for the image. This > > > file can be used to flash image using bmaptool utility. > > > + > > > + The --no-fstab-update option is used to doesn't change fstab > > > file. When > > > + using this option the final fstab file will be same that in > > > rootfs and > > > + wic doesn't update file, e.g adding a new mount point. User can > > > control > > > + the fstab file content in base-files recipe. > > > """ > > > > > > wic_list_usage = """ > > > @@ -283,6 +274,243 @@ DESCRIPTION > > > details. > > > """ > > > > > > +wic_ls_usage = """ > > > + > > > + List content of a partitioned image > > > + > > > + usage: wic ls <image>[:<partition>[<path>]] [--native-sysroot > > > <path>] + > > > + This command outputs either list of image partitions or directory > > > contents > > > + of vfat and ext* partitions. > > > + > > > + See 'wic help ls' for more detailed instructions. > > > + > > > +""" > > > + > > > +wic_ls_help = """ > > > + > > > +NAME > > > + wic ls - List contents of partitioned image or partition > > > + > > > +SYNOPSIS > > > + wic ls <image> > > > + wic ls <image>:<vfat or ext* partition> > > > + wic ls <image>:<vfat or ext* partition><path> > > > + wic ls <image>:<vfat or ext* partition><path> --native-sysroot > > > <path> + > > > +DESCRIPTION > > > + This command lists either partitions of the image or directory > > > contents > > > + of vfat or ext* partitions. > > > + > > > + The first form it lists partitions of the image. > > > + For example: > > > + $ wic ls > > > tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic > > > + Num Start End Size Fstype > > > + 1 1048576 24438783 23390208 fat16 > > > + 2 25165824 50315263 25149440 ext4 > > > + > > > + Second and third form list directory content of the partition: > > > + $ wic ls > > > tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 > > > + Volume in drive : is boot > > > + Volume Serial Number is 2DF2-5F02 > > > + Directory for ::/ > > > + > > > + efi <DIR> 2017-05-11 10:54 > > > + startup nsh 26 2017-05-11 10:54 > > > + vmlinuz 6922288 2017-05-11 10:54 > > > + 3 files 6 922 314 bytes > > > + 15 818 752 bytes free > > > + > > > + > > > + $ wic ls > > > > tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/EFI/boot/ > > > + Volume in drive : is boot > > > + Volume Serial Number is 2DF2-5F02 > > > + Directory for ::/EFI/boot > > > + > > > + . <DIR> 2017-05-11 10:54 > > > + .. <DIR> 2017-05-11 10:54 > > > + grub cfg 679 2017-05-11 10:54 > > > + bootx64 efi 571392 2017-05-11 10:54 > > > + 4 files 572 071 bytes > > > + 15 818 752 bytes free > > > + > > > + The -n option is used to specify the path to the native sysroot > > > + containing the tools(parted and mtools) to use. > > > + > > > +""" > > > + > > > +wic_cp_usage = """ > > > + > > > + Copy files and directories to/from the vfat or ext* partition > > > + > > > + usage: wic cp <src> <dest> [--native-sysroot <path>] > > > + > > > + source/destination image in format <image>:<partition>[<path>] > > > + > > > + This command copies files or directories either > > > + - from local to vfat or ext* partitions of partitioned image > > > + - from vfat or ext* partitions of partitioned image to local > > > + > > > + See 'wic help cp' for more detailed instructions. > > > + > > > +""" > > > + > > > +wic_cp_help = """ > > > + > > > +NAME > > > + wic cp - copy files and directories to/from the vfat or ext* > > > partitions + > > > +SYNOPSIS > > > + wic cp <src> <dest>:<partition> > > > + wic cp <src>:<partition> <dest> > > > + wic cp <src> <dest-image>:<partition><path> > > > + wic cp <src> <dest-image>:<partition><path> --native-sysroot > > > <path> + > > > +DESCRIPTION > > > + This command copies files or directories either > > > + - from local to vfat or ext* partitions of partitioned image > > > + - from vfat or ext* partitions of partitioned image to local > > > + > > > + The first form of it copies file or directory to the root > > > directory of > > > + the partition: > > > + $ wic cp test.wks > > > tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 > > > + $ wic ls > > > tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 > > > + Volume in drive : is boot > > > + Volume Serial Number is DB4C-FD4C > > > + Directory for ::/ > > > + > > > + efi <DIR> 2017-05-24 18:15 > > > + loader <DIR> 2017-05-24 18:15 > > > + startup nsh 26 2017-05-24 18:15 > > > + vmlinuz 6926384 2017-05-24 18:15 > > > + test wks 628 2017-05-24 21:22 > > > + 5 files 6 927 038 bytes > > > + 15 677 440 bytes free > > > + > > > + The second form of the command copies file or directory to the > > > specified directory > > > + on the partition: > > > + $ wic cp test > > > tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/efi/ > > > + $ wic ls > > > tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/efi/ > > > + Volume in drive : is boot > > > + Volume Serial Number is DB4C-FD4C > > > + Directory for ::/efi > > > + > > > + . <DIR> 2017-05-24 18:15 > > > + .. <DIR> 2017-05-24 18:15 > > > + boot <DIR> 2017-05-24 18:15 > > > + test <DIR> 2017-05-24 21:27 > > > + 4 files 0 bytes > > > + 15 675 392 bytes free > > > + > > > + The third form of the command copies file or directory from the > > > specified directory > > > + on the partition to local: > > > + $ wic cp > > > tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/vmlinuz > > > test + > > > + The -n option is used to specify the path to the native sysroot > > > + containing the tools(parted and mtools) to use. > > > +""" > > > + > > > +wic_rm_usage = """ > > > + > > > + Remove files or directories from the vfat or ext* partitions > > > + > > > + usage: wic rm <image>:<partition><path> [--native-sysroot <path>] > > > + > > > + This command removes files or directories from the vfat or ext* > > > partitions of > > > + the partitioned image. > > > + > > > + See 'wic help rm' for more detailed instructions. > > > + > > > +""" > > > + > > > +wic_rm_help = """ > > > + > > > +NAME > > > + wic rm - remove files or directories from the vfat or ext* > > > partitions + > > > +SYNOPSIS > > > + wic rm <src> <image>:<partition><path> > > > + wic rm <src> <image>:<partition><path> --native-sysroot <path> > > > + wic rm -r <image>:<partition><path> > > > + > > > +DESCRIPTION > > > + This command removes files or directories from the vfat or ext* > > > partition of the > > > + partitioned image: > > > + > > > + $ wic ls > > > ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 > > > + Volume in drive : is boot > > > + Volume Serial Number is 11D0-DE21 > > > + Directory for ::/ > > > + > > > + libcom32 c32 186500 2017-06-02 15:15 > > > + libutil c32 24148 2017-06-02 15:15 > > > + syslinux cfg 209 2017-06-02 15:15 > > > + vesamenu c32 27104 2017-06-02 15:15 > > > + vmlinuz 6926384 2017-06-02 15:15 > > > + 5 files 7 164 345 bytes > > > + 16 582 656 bytes free > > > + > > > + $ wic rm > > > > ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/libutil.c32 > > > + > > > + $ wic ls > > > ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 > > > + Volume in drive : is boot > > > + Volume Serial Number is 11D0-DE21 > > > + Directory for ::/ > > > + > > > + libcom32 c32 186500 2017-06-02 15:15 > > > + syslinux cfg 209 2017-06-02 15:15 > > > + vesamenu c32 27104 2017-06-02 15:15 > > > + vmlinuz 6926384 2017-06-02 15:15 > > > + 4 files 7 140 197 bytes > > > + 16 607 232 bytes free > > > + > > > + The -n option is used to specify the path to the native sysroot > > > + containing the tools(parted and mtools) to use. > > > + > > > + The -r option is used to remove directories and their contents > > > + recursively,this only applies to ext* partition. > > > +""" > > > + > > > +wic_write_usage = """ > > > + > > > + Write image to a device > > > + > > > + usage: wic write <image> <target device> [--expand [rules]] > > > [--native-sysroot <path>] + > > > + This command writes partitioned image to a target device (USB > > > stick, SD card etc). + > > > + See 'wic help write' for more detailed instructions. > > > + > > > +""" > > > + > > > +wic_write_help = """ > > > + > > > +NAME > > > + wic write - write an image to a device > > > + > > > +SYNOPSIS > > > + wic write <image> <target> > > > + wic write <image> <target> --expand auto > > > + wic write <image> <target> --expand 1:100M,2:300M > > > + wic write <image> <target> --native-sysroot <path> > > > + > > > +DESCRIPTION > > > + This command writes an image to a target device (USB stick, SD > > > card etc) + > > > + $ wic write > > > ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic > > > /dev/sdb + > > > + The --expand option is used to resize image partitions. > > > + --expand auto expands partitions to occupy all free space > > > available on the target device. > > > + It's also possible to specify expansion rules in a format > > > + <partition>:<size>[,<partition>:<size>...] for one or more > > > partitions. > > > + Specifying size 0 will keep partition unmodified. > > > + Note: Resizing boot partition can result in non-bootable image > > > for non-EFI images. It is > > > + recommended to use size 0 for boot partition to keep image > > > bootable. + > > > + The --native-sysroot option is used to specify the path to the > > > native sysroot > > > + containing the tools(parted, resize2fs) to use. > > > +""" > > > + > > > wic_plugins_help = """ > > > > > > NAME > > > @@ -308,7 +536,8 @@ DESCRIPTION > > > > > > Source plugins can also be implemented and added by external > > > layers - any plugins found in a scripts/lib/wic/plugins/source/ > > > - directory in an external layer will also be made available. > > > + or lib/wic/plugins/source/ directory in an external layer will > > > + also be made available. > > > > > > When the wic implementation needs to invoke a partition-specific > > > implementation, it looks for the plugin that has the same name as > > > @@ -346,6 +575,10 @@ DESCRIPTION > > > partition. In other words, it 'prepares' the final > > > partition image which will be incorporated into the disk image. > > > > > > + do_post_partition() > > > + Called after the partition is created. It is useful to add > > > post > > > + operations e.g. signing the partition. > > > + > > > do_configure_partition() > > > Called before do_prepare_partition(), typically used to > > > create custom configuration files for a partition, for > > > @@ -632,8 +865,11 @@ DESCRIPTION > > > Partitions with a <mountpoint> specified will be > > > automatically mounted. This is achieved by wic adding entries to the > > > fstab during image generation. In order for a valid fstab to be > > > generated one of the > > > - --ondrive, --ondisk or --use-uuid partition options must be > > > used for > > > - each partition that specifies a mountpoint. > > > + --ondrive, --ondisk, --use-uuid or --use-label partition > > > options must > > > + be used for each partition that specifies a mountpoint. Note > > > that with > > > + --use-{uuid,label} and non-root <mountpoint>, including swap, > > > the mount > > > + program must understand the PARTUUID or LABEL syntax. This > > > currently > > > + excludes the busybox versions of these applications. > > > > > > > > > The following are supported 'part' options: > > > @@ -687,6 +923,8 @@ DESCRIPTION > > > apply to partitions created using '--source rootfs' (see > > > --source above). Valid values are: > > > > > > + vfat > > > + msdos > > > ext2 > > > ext3 > > > ext4 > > > @@ -706,6 +944,14 @@ DESCRIPTION > > > label is already in use by another > > > filesystem, a new label is created for the partition. > > > > > > + --use-label: This option is specific to wic. It makes wic > > > to use the > > > + label in /etc/fstab to specify a partition. If > > > the > > > + --use-label and --use-uuid are used at the > > > same time, > > > + we prefer the uuid because it is less likely > > > to cause > > > + name confliction. We don't support using this > > > parameter > > > + on the root partition since it requires an > > > initramfs to > > > + parse this value and we do not currently > > > support that. + > > > --active: Marks the partition as active. > > > > > > --align (in KBytes): This option is specific to wic and says > > > @@ -719,11 +965,31 @@ DESCRIPTION > > > bootloaders. > > > > > > --exclude-path: This option is specific to wic. It excludes > > > the given > > > - absolute path from the resulting image. If > > > the path > > > + relative path from the resulting image. If > > > the path ends with a slash, only the content of the directory > > > is omitted, not the directory itself. This > > > option only has an effect with the rootfs source plugin. > > > > > > + --include-path: This option is specific to wic. It adds the > > > contents > > > + of the given path or a rootfs to the > > > resulting image. > > > + The option contains two fields, the origin > > > and the > > > + destination. When the origin is a rootfs, > > > it follows > > > + the same logic as the rootfs-dir argument > > > and the > > > + permissions and owners are kept. When the > > > origin is a > > > + path, it is relative to the directory in > > > which wic is > > > + running not the rootfs itself so use of an > > > absolute > > > + path is recommended, and the owner and > > > group is set to > > > + root:root. If no destination is given it is > > > + automatically set to the root of the > > > rootfs. This > > > + option only has an effect with the rootfs > > > source > > > + plugin. > > > + > > > + --change-directory: This option is specific to wic. It > > > changes to the > > > + given directory before copying the > > > files. This > > > + option is useful when we want to split > > > a rootfs in > > > + multiple partitions and we want to keep > > > the right > > > + permissions and usernames in all the > > > partitions. + > > > --extra-space: This option is specific to wic. It adds extra > > > space after the space filled by the content > > > of the partition. The final size can go > > > @@ -738,6 +1004,8 @@ DESCRIPTION > > > This option cannot be used with > > > --fixed-size option. > > > > > > + --part-name: This option is specific to wic. It specifies > > > name for GPT partitions. + > > > --part-type: This option is specific to wic. It specifies > > > partition type GUID for GPT partitions. > > > List of partition type GUIDS can be found here: > > > @@ -752,10 +1020,21 @@ DESCRIPTION > > > in bootloader configuration before running wic. In > > > this case .wks file can be generated or modified to set preconfigured > > > parition UUID using this option. > > > + --fsuuid: This option is specific to wic. It specifies > > > filesystem UUID. > > > + It's useful if preconfigured filesystem UUID is > > > added to kernel command line > > > + in bootloader configuration before running wic. > > > In this case .wks file can > > > + be generated or modified to set preconfigured > > > filesystem UUID using this option. + > > > --system-id: This option is specific to wic. It specifies > > > partition system id. It's useful for the harware that requires > > > non-default partition system ids. The parameter in one byte long hex > > > number either with 0x prefix or without it. > > > + --mkfs-extraopts: This option specifies extra options to > > > pass to mkfs utility. > > > + NOTE, that wic uses default options for > > > some filesystems, for example > > > + '-S 512' for mkfs.fat or '-F -i 8192' for > > > mkfs.ext. Those options will > > > + not take effect when --mkfs-extraopts is > > > used. This should be taken into > > > + account when using --mkfs-extraopts. > > > + > > > * bootloader > > > > > > This command allows the user to specify various bootloader > > > @@ -793,3 +1072,67 @@ DESCRIPTION > > > .wks files. > > > > > > """ > > > + > > > +wic_help_help = """ > > > +NAME > > > + wic help - display a help topic > > > + > > > +DESCRIPTION > > > + Specify a help topic to display it. Topics are shown above. > > > +""" > > > + > > > + > > > +wic_help = """ > > > +Creates a customized OpenEmbedded image. > > > + > > > +Usage: wic [--version] > > > + wic help [COMMAND or TOPIC] > > > + wic COMMAND [ARGS] > > > + > > > + usage 1: Returns the current version of Wic > > > + usage 2: Returns detailed help for a COMMAND or TOPIC > > > + usage 3: Executes COMMAND > > > + > > > + > > > +COMMAND: > > > + > > > + list - List available canned images and source plugins > > > + ls - List contents of partitioned image or partition > > > + rm - Remove files or directories from the vfat or ext* > > > partitions > > > + help - Show help for a wic COMMAND or TOPIC > > > + write - Write an image to a device > > > + cp - Copy files and directories to the vfat or ext* > > > partitions > > > + create - Create a new OpenEmbedded image > > > + > > > + > > > +TOPIC: > > > + overview - Presents an overall overview of Wic > > > + plugins - Presents an overview and API for Wic plugins > > > + kickstart - Presents a Wic kicstart file reference > > > + > > > + > > > +Examples: > > > + > > > + $ wic --version > > > + > > > + Returns the current version of Wic > > > + > > > + > > > + $ wic help cp > > > + > > > + Returns the SYNOPSIS and DESCRIPTION for the Wic "cp" command. > > > + > > > + > > > + $ wic list images > > > + > > > + Returns the list of canned images (i.e. *.wks files located in > > > + the /scripts/lib/wic/canned-wks directory. > > > + > > > + > > > + $ wic create mkefidisk -e core-image-minimal > > > + > > > + Creates an EFI disk image from artifacts used in a previous > > > + core-image-minimal build in standard BitBake locations > > > + (e.g. Cooked Mode). > > > + > > > +""" > > > diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py > > > index a039300..3453d9c 100644 > > > --- a/scripts/lib/wic/ksparser.py > > > +++ b/scripts/lib/wic/ksparser.py > > > @@ -1,21 +1,8 @@ > > > -#!/usr/bin/env python -tt > > > -# ex:ts=4:sw=4:sts=4:et > > > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > > > +#!/usr/bin/env python3 > > > # > > > # Copyright (c) 2016 Intel, Inc. > > > # > > > -# This program is free software; you can redistribute it and/or > > > modify it -# under the terms of the GNU General Public License as > > > published by the Free -# Software Foundation; version 2 of the License > > > -# > > > -# This program is distributed in the hope that it will be useful, but > > > -# WITHOUT ANY WARRANTY; without even the implied warranty of > > > MERCHANTABILITY -# or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > > > General Public License -# for more details. > > > -# > > > -# You should have received a copy of the GNU General Public License > > > along -# with this program; if not, write to the Free Software > > > Foundation, Inc., 59 -# Temple Place - Suite 330, Boston, MA > > > 02111-1307, USA. +# SPDX-License-Identifier: GPL-2.0-only > > > # > > > # DESCRIPTION > > > # This module provides parser for kickstart format > > > @@ -28,14 +15,30 @@ > > > import os > > > import shlex > > > import logging > > > +import re > > > > > > from argparse import ArgumentParser, ArgumentError, ArgumentTypeError > > > > > > from wic.engine import find_canned > > > from wic.partition import Partition > > > +from wic.misc import get_bitbake_var > > > > > > logger = logging.getLogger('wic') > > > > > > +__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}") > > > + > > > +def expand_line(line): > > > + while True: > > > + m = __expand_var_regexp__.search(line) > > > + if not m: > > > + return line > > > + key = m.group()[2:-1] > > > + val = get_bitbake_var(key) > > > + if val is None: > > > + logger.warning("cannot expand variable %s" % key) > > > + return line > > > + line = line[:m.start()] + val + line[m.end():] > > > + > > > class KickStartError(Exception): > > > """Custom exception.""" > > > pass > > > @@ -48,26 +51,31 @@ class KickStartParser(ArgumentParser): > > > def error(self, message): > > > raise ArgumentError(None, message) > > > > > > -def sizetype(arg): > > > - """ > > > - Custom type for ArgumentParser > > > - Converts size string in <num>[K|k|M|G] format into the integer > > > value > > > - """ > > > - if arg.isdigit(): > > > - return int(arg) * 1024 > > > - > > > - if not arg[:-1].isdigit(): > > > - raise ArgumentTypeError("Invalid size: %r" % arg) > > > +def sizetype(default): > > > + def f(arg): > > > + """ > > > + Custom type for ArgumentParser > > > + Converts size string in <num>[K|k|M|G] format into the > > > integer value > > > + """ > > > + try: > > > + suffix = default > > > + size = int(arg) > > > + except ValueError: > > > + try: > > > + suffix = arg[-1:] > > > + size = int(arg[:-1]) > > > + except ValueError: > > > + raise ArgumentTypeError("Invalid size: %r" % arg) > > > > > > - size = int(arg[:-1]) > > > - if arg.endswith("k") or arg.endswith("K"): > > > - return size > > > - if arg.endswith("M"): > > > - return size * 1024 > > > - if arg.endswith("G"): > > > - return size * 1024 * 1024 > > > + if suffix == "k" or suffix == "K": > > > + return size > > > + if suffix == "M": > > > + return size * 1024 > > > + if suffix == "G": > > > + return size * 1024 * 1024 > > > > > > - raise ArgumentTypeError("Invalid size: %r" % arg) > > > + raise ArgumentTypeError("Invalid size: %r" % arg) > > > + return f > > > > > > def overheadtype(arg): > > > """ > > > @@ -114,7 +122,7 @@ def systemidtype(arg): > > > return arg > > > > > > class KickStart(): > > > - """"Kickstart parser implementation.""" > > > + """Kickstart parser implementation.""" > > > > > > DEFAULT_EXTRA_SPACE = 10*1024 > > > DEFAULT_OVERHEAD_FACTOR > <https://www.google.com/maps/search/LT_OVERHEAD_FACTOR?entry=gmail&source=g> > = 1.3 > > > @@ -133,30 +141,41 @@ class KickStart(): > > > part.add_argument('mountpoint', nargs='?') > > > part.add_argument('--active', action='store_true') > > > part.add_argument('--align', type=int) > > > + part.add_argument('--offset', type=sizetype("K")) > > > part.add_argument('--exclude-path', nargs='+') > > > - part.add_argument("--extra-space", type=sizetype) > > > + part.add_argument('--include-path', nargs='+', > > > action='append') > > > + part.add_argument('--change-directory') > > > + part.add_argument("--extra-space", type=sizetype("M")) > > > part.add_argument('--fsoptions', dest='fsopts') > > > - part.add_argument('--fstype') > > > + part.add_argument('--fstype', default='vfat', > > > + choices=('ext2', 'ext3', 'ext4', 'btrfs', > > > + 'squashfs', 'vfat', 'msdos', > > > 'swap')) > > > + part.add_argument('--mkfs-extraopts', default='') > > > part.add_argument('--label') > > > + part.add_argument('--use-label', action='store_true') > > > part.add_argument('--no-table', action='store_true') > > > part.add_argument('--ondisk', '--ondrive', dest='disk', > > > default='sda') part.add_argument("--overhead-factor", > > > type=overheadtype) > > > + part.add_argument('--part-name') > > > part.add_argument('--part-type') > > > part.add_argument('--rootfs-dir') > > > + part.add_argument('--type', default='primary', > > > + choices = ('primary', 'logical')) > > > > > > # --size and --fixed-size cannot be specified together; > > > options # ----extra-space and --overhead-factor should also raise a > > > parser # --error, but since nesting mutually exclusive groups does > > > not work, # ----extra-space/--overhead-factor are handled later > > > sizeexcl = part.add_mutually_exclusive_group() > > > - sizeexcl.add_argument('--size', type=sizetype, default=0) > > > - sizeexcl.add_argument('--fixed-size', type=sizetype, > > > default=0) > > > + sizeexcl.add_argument('--size', type=sizetype("M"), > > > default=0) > > > + sizeexcl.add_argument('--fixed-size', type=sizetype("M"), > > > default=0) > > > part.add_argument('--source') > > > part.add_argument('--sourceparams') > > > part.add_argument('--system-id', type=systemidtype) > > > part.add_argument('--use-uuid', action='store_true') > > > part.add_argument('--uuid') > > > + part.add_argument('--fsuuid') > > > > > > bootloader = subparsers.add_parser('bootloader') > > > bootloader.add_argument('--append') > > > @@ -184,6 +203,7 @@ class KickStart(): > > > line = line.strip() > > > lineno += 1 > > > if line and line[0] != '#': > > > + line = expand_line(line) > > > try: > > > line_args = shlex.split(line) > > > parsed = parser.parse_args(line_args) > > > @@ -191,6 +211,20 @@ class KickStart(): > > > raise KickStartError('%s:%d: %s' % \ > > > (confpath, lineno, err)) > > > if line.startswith('part'): > > > + # SquashFS does not support filesystem UUID > > > + if parsed.fstype == 'squashfs': > > > + if parsed.fsuuid: > > > + err = "%s:%d: SquashFS does not > > > support UUID" \ > > > + % (confpath, lineno) > > > + raise KickStartError(err) > > > + if parsed.label: > > > + err = "%s:%d: SquashFS does not > > > support LABEL" \ > > > + % (confpath, lineno) > > > + raise KickStartError(err) > > > + if parsed.use_label and not parsed.label: > > > + err = "%s:%d: Must set the label with > > > --label" \ > > > + % (confpath, lineno) > > > + raise KickStartError(err) > > > # using ArgumentParser one cannot easily > > > tell if option # was passed as argument, if said option has a default > > > # value; --overhead-factor/--extra-space > > > cannot be used @@ -219,6 +253,11 @@ class KickStart(): > > > elif line.startswith('bootloader'): > > > if not self.bootloader: > > > self.bootloader = parsed > > > + # Concatenate the strings set in APPEND > > > + append_var = get_bitbake_var("APPEND") > > > + if append_var: > > > + self.bootloader.append = ' > > > '.join(filter(None, \ > > > + > > > (self.bootloader.append, append_var))) else: > > > err = "%s:%d: more than one bootloader > > > specified" \ % (confpath, lineno) > > > diff --git a/scripts/lib/wic/utils/misc.py b/scripts/lib/wic/misc.py > > > similarity index 70% > > > rename from scripts/lib/wic/utils/misc.py > > > rename to scripts/lib/wic/misc.py > > > index c941112..91975ba 100644 > > > --- a/scripts/lib/wic/utils/misc.py > > > +++ b/scripts/lib/wic/misc.py > > > @@ -1,21 +1,7 @@ > > > -# ex:ts=4:sw=4:sts=4:et > > > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > > > # > > > # Copyright (c) 2013, Intel Corporation. > > > -# All rights reserved. > > > # > > > -# This program is free software; you can redistribute it and/or > > > modify -# it under the terms of the GNU General Public License > > > version 2 as -# published by the Free Software Foundation. > > > -# > > > -# This program is distributed in the hope that it will be useful, > > > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > > > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > > -# GNU General Public License for more details. > > > -# > > > -# You should have received a copy of the GNU General Public License > > > along -# with this program; if not, write to the Free Software > > > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > > > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > > > # > > > # DESCRIPTION > > > # This module provides a place to collect various wic-related utils > > > @@ -29,12 +15,12 @@ > > > import logging > > > import os > > > import re > > > +import subprocess > > > > > > from collections import defaultdict > > > from distutils import spawn > > > > > > from wic import WicError > > > -from wic.utils import runner > > > > > > logger = logging.getLogger('wic') > > > > > > @@ -43,6 +29,9 @@ NATIVE_RECIPES = {"bmaptool": "bmap-tools", > > > "grub-mkimage": "grub-efi", > > > "isohybrid": "syslinux", > > > "mcopy" > <https://www.google.com/maps/search/%22mcopy%22?entry=gmail&source=g>: > "mtools", > > > + "mdel" : "mtools", > > > + "mdeltree" : [-- Attachment #2: Type: text/html, Size: 111657 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v2 02/10] wic/plugins: Fix wic plugins to work with the latest wic 2020-09-02 18:56 [PATCH v2 00/10] WIC update Vijai Kumar K 2020-09-02 18:56 ` [PATCH v2 01/10] wic: Update to the latest wic from openembedded core Vijai Kumar K @ 2020-09-02 18:56 ` Vijai Kumar K 2020-09-02 18:56 ` [PATCH v2 03/10] wic-img: Satisfy the quirks of " Vijai Kumar K ` (6 subsequent siblings) 8 siblings, 0 replies; 29+ messages in thread From: Vijai Kumar K @ 2020-09-02 18:56 UTC (permalink / raw) To: isar-users, Vijaikumar_Kanagarajan In the latest wic, wic.utils is removed and part of the implmentation is moved to wic.misc. Fix the wic imports accordingly. Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> --- meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py | 2 +- .../lib/wic/plugins/source/bootimg-pcbios-isar.py | 9 +++------ meta/scripts/lib/wic/plugins/source/rootfs-u-boot.py | 2 +- 3 files changed, 5 insertions(+), 8 deletions(-) diff --git a/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py b/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py index 25b12ca..2285d2e 100644 --- a/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py +++ b/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py @@ -31,7 +31,7 @@ import shutil from wic import WicError from wic.engine import get_custom_config from wic.pluginbase import SourcePlugin -from wic.utils.misc import (exec_cmd, get_bitbake_var, BOOTDD_EXTRA_SPACE) +from wic.misc import (exec_cmd, get_bitbake_var, BOOTDD_EXTRA_SPACE) logger = logging.getLogger('wic') diff --git a/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py b/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py index 0423862..4d916d6 100644 --- a/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py +++ b/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py @@ -29,9 +29,8 @@ import os from wic import WicError from wic.engine import get_custom_config -from wic.utils import runner from wic.pluginbase import SourcePlugin -from wic.utils.misc import (exec_cmd, exec_native_cmd, +from wic.misc import (exec_cmd, exec_native_cmd, get_bitbake_var, BOOTDD_EXTRA_SPACE) logger = logging.getLogger('wic') @@ -83,10 +82,8 @@ class BootimgPcbiosIsarPlugin(SourcePlugin): logger.debug("Installing MBR on disk %s as %s with size %s bytes", disk_name, full_path, disk.min_size) - rcode = runner.show(['dd', 'if=%s' % mbrfile, - 'of=%s' % full_path, 'conv=notrunc']) - if rcode != 0: - raise WicError("Unable to set MBR to %s" % full_path) + dd_cmd = "dd if=%s of=%s conv=notrunc" % (mbrfile, full_path) + exec_cmd(dd_cmd, native_sysroot) @classmethod def do_configure_partition(cls, part, source_params, creator, cr_workdir, diff --git a/meta/scripts/lib/wic/plugins/source/rootfs-u-boot.py b/meta/scripts/lib/wic/plugins/source/rootfs-u-boot.py index f60afb8..3ac0aa0 100644 --- a/meta/scripts/lib/wic/plugins/source/rootfs-u-boot.py +++ b/meta/scripts/lib/wic/plugins/source/rootfs-u-boot.py @@ -18,7 +18,7 @@ import os from wic import WicError from wic.plugins.source.rootfs import RootfsPlugin -from wic.utils.misc import exec_cmd +from wic.misc import exec_cmd logger = logging.getLogger('wic') -- 2.17.1 ^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v2 03/10] wic-img: Satisfy the quirks of latest wic 2020-09-02 18:56 [PATCH v2 00/10] WIC update Vijai Kumar K 2020-09-02 18:56 ` [PATCH v2 01/10] wic: Update to the latest wic from openembedded core Vijai Kumar K 2020-09-02 18:56 ` [PATCH v2 02/10] wic/plugins: Fix wic plugins to work with the latest wic Vijai Kumar K @ 2020-09-02 18:56 ` Vijai Kumar K 2020-09-05 8:30 ` Henning Schild 2020-09-02 18:56 ` [PATCH v2 04/10] oe.path: Add copyhardlink() helper function Vijai Kumar K ` (5 subsequent siblings) 8 siblings, 1 reply; 29+ messages in thread From: Vijai Kumar K @ 2020-09-02 18:56 UTC (permalink / raw) To: isar-users, Vijaikumar_Kanagarajan The latest WIC doesnot use fakeroot if pseudo_dir(IMAGE_ROOTFS/../pseudo) is non-existent. This results in wic_fakeroot not being used. Create a dummy pseudo_dir to satisfy wic. WIC hardcodes the python(/usr/bin/python3-native/python3) to be used while running bmaptool. The above path is OE specific and hence breaks build in ISAR. Create a symbolic link in the above location to satisfy wic. Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> --- meta/classes/wic-img.bbclass | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass index afab394..a2c9627 100644 --- a/meta/classes/wic-img.bbclass +++ b/meta/classes/wic-img.bbclass @@ -143,6 +143,7 @@ EOSUDO export FAKEROOTCMD=${FAKEROOTCMD} export BUILDDIR=${BUILDDIR} export MTOOLS_SKIP_CHECK=1 + mkdir -p ${IMAGE_ROOTFS}/../pseudo # create the temp dir in the buildchroot to ensure uniqueness WICTMP=$(cd ${BUILDCHROOT_DIR}; mktemp -d -p tmp) @@ -157,22 +158,31 @@ EOSUDO IMAGE_FULLNAME="$6" IMAGE_BASENAME="$7" shift 7 - + # The python path is hard-coded as /usr/bin/python3-native/python3 in wic. Handle that. + mkdir -p /usr/bin/python3-native/ + if [ $(head -1 $(which bmaptool) | grep python3) ];then + ln -s /usr/bin/python3 /usr/bin/python3-native/python3 + else + ln -s /usr/bin/python2 /usr/bin/python3-native/python3 + fi export PATH="$ISARROOT/bitbake/bin:$PATH" "$ISARROOT"/scripts/wic create "$WKS_FULL_PATH" \ --vars "$STAGING_DIR/$MACHINE/imgdata/" \ -o "/$WICTMP/${IMAGE_FULLNAME}.wic/" \ --bmap \ - -e "$IMAGE_BASENAME" $@' \ + -e "$IMAGE_BASENAME" $@ + rm -rf /usr/bin/python3-native' \ my_script "${ISARROOT}" "${WKS_FULL_PATH}" "${STAGING_DIR}" \ "${MACHINE}" "${WICTMP}" "${IMAGE_FULLNAME}" "${IMAGE_BASENAME}" \ ${WIC_CREATE_EXTRA_ARGS} + sudo chown -R $(stat -c "%U" ${ISARROOT}) ${ISARROOT}/meta ${ISARROOT}/meta-isar ${ISARROOT}/scripts || true WIC_DIRECT=$(ls -t -1 ${BUILDCHROOT_DIR}/$WICTMP/${IMAGE_FULLNAME}.wic/*.direct | head -1) sudo chown -R $(id -u):$(id -g) ${BUILDCHROOT_DIR}/${WICTMP} mv -f ${WIC_DIRECT} ${WIC_IMAGE_FILE} mv -f ${WIC_DIRECT}.bmap ${WIC_IMAGE_FILE}.bmap rm -rf ${BUILDCHROOT_DIR}/${WICTMP} + rm -rf ${IMAGE_ROOTFS}/../pseudo } do_wic_image[file-checksums] += "${WKS_FILE_CHECKSUM}" -- 2.17.1 ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 03/10] wic-img: Satisfy the quirks of latest wic 2020-09-02 18:56 ` [PATCH v2 03/10] wic-img: Satisfy the quirks of " Vijai Kumar K @ 2020-09-05 8:30 ` Henning Schild 2020-09-05 16:24 ` vijaikumar....@gmail.com 0 siblings, 1 reply; 29+ messages in thread From: Henning Schild @ 2020-09-05 8:30 UTC (permalink / raw) To: Vijai Kumar K; +Cc: isar-users On Thu, 3 Sep 2020 00:26:17 +0530 Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> wrote: > The latest WIC doesnot use fakeroot if > pseudo_dir(IMAGE_ROOTFS/../pseudo) is non-existent. This results in > wic_fakeroot not being used. Create a dummy pseudo_dir to satisfy wic. > > WIC hardcodes the python(/usr/bin/python3-native/python3) to be used > while running bmaptool. The above path is OE specific and hence breaks > build in ISAR. Create a symbolic link in the above location to satisfy > wic. > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > --- > meta/classes/wic-img.bbclass | 14 ++++++++++++-- > 1 file changed, 12 insertions(+), 2 deletions(-) > > diff --git a/meta/classes/wic-img.bbclass > b/meta/classes/wic-img.bbclass index afab394..a2c9627 100644 > --- a/meta/classes/wic-img.bbclass > +++ b/meta/classes/wic-img.bbclass > @@ -143,6 +143,7 @@ EOSUDO > export FAKEROOTCMD=${FAKEROOTCMD} > export BUILDDIR=${BUILDDIR} > export MTOOLS_SKIP_CHECK=1 > + mkdir -p ${IMAGE_ROOTFS}/../pseudo > > # create the temp dir in the buildchroot to ensure uniqueness > WICTMP=$(cd ${BUILDCHROOT_DIR}; mktemp -d -p tmp) > @@ -157,22 +158,31 @@ EOSUDO > IMAGE_FULLNAME="$6" > IMAGE_BASENAME="$7" > shift 7 > - > + # The python path is hard-coded as > /usr/bin/python3-native/python3 in wic. Handle that. > + mkdir -p /usr/bin/python3-native/ > + if [ $(head -1 $(which bmaptool) | grep python3) ];then > + ln -s /usr/bin/python3 /usr/bin/python3-native/python3 > + else > + ln -s /usr/bin/python2 /usr/bin/python3-native/python3 > + fi Can this be moved to wic_fakeroot? That script was designed to hold all the weird hacks ... I guess you need to "ln -sf" to make that whole thing executable multiple times. Can you explain why we need the case where python2 is linked as python3? > export PATH="$ISARROOT/bitbake/bin:$PATH" > "$ISARROOT"/scripts/wic create "$WKS_FULL_PATH" \ > --vars "$STAGING_DIR/$MACHINE/imgdata/" \ > -o "/$WICTMP/${IMAGE_FULLNAME}.wic/" \ > --bmap \ > - -e "$IMAGE_BASENAME" $@' \ > + -e "$IMAGE_BASENAME" $@ > + rm -rf /usr/bin/python3-native' \ This looks like the rm is in the middle of wic, should probably come after. > my_script "${ISARROOT}" "${WKS_FULL_PATH}" > "${STAGING_DIR}" \ "${MACHINE}" "${WICTMP}" "${IMAGE_FULLNAME}" > "${IMAGE_BASENAME}" \ ${WIC_CREATE_EXTRA_ARGS} > + > sudo chown -R $(stat -c "%U" ${ISARROOT}) ${ISARROOT}/meta > ${ISARROOT}/meta-isar ${ISARROOT}/scripts || true WIC_DIRECT=$(ls -t > -1 ${BUILDCHROOT_DIR}/$WICTMP/${IMAGE_FULLNAME}.wic/*.direct | head > -1) sudo chown -R $(id -u):$(id -g) ${BUILDCHROOT_DIR}/${WICTMP} mv > -f ${WIC_DIRECT} ${WIC_IMAGE_FILE} mv -f ${WIC_DIRECT}.bmap > ${WIC_IMAGE_FILE}.bmap rm -rf ${BUILDCHROOT_DIR}/${WICTMP} > + rm -rf ${IMAGE_ROOTFS}/../pseudo > } > > do_wic_image[file-checksums] += "${WKS_FILE_CHECKSUM}" ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 03/10] wic-img: Satisfy the quirks of latest wic 2020-09-05 8:30 ` Henning Schild @ 2020-09-05 16:24 ` vijaikumar....@gmail.com 2020-09-09 14:52 ` Henning Schild 0 siblings, 1 reply; 29+ messages in thread From: vijaikumar....@gmail.com @ 2020-09-05 16:24 UTC (permalink / raw) To: isar-users [-- Attachment #1.1: Type: text/plain, Size: 4362 bytes --] On Saturday, September 5, 2020 at 2:00:59 PM UTC+5:30 Henning Schild wrote: > On Thu, 3 Sep 2020 00:26:17 +0530 > Vijai Kumar K <Vijaikumar_...@mentor.com> wrote: > > > The latest WIC doesnot use fakeroot if > > pseudo_dir(IMAGE_ROOTFS/../pseudo) is non-existent. This results in > > wic_fakeroot not being used. Create a dummy pseudo_dir to satisfy wic. > > > > WIC hardcodes the python(/usr/bin/python3-native/python3) to be used > > while running bmaptool. The above path is OE specific and hence breaks > > build in ISAR. Create a symbolic link in the above location to satisfy > > wic. > > > > Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com> > > --- > > meta/classes/wic-img.bbclass | 14 ++++++++++++-- > > 1 file changed, 12 insertions(+), 2 deletions(-) > > > > diff --git a/meta/classes/wic-img.bbclass > > b/meta/classes/wic-img.bbclass index afab394..a2c9627 100644 > > --- a/meta/classes/wic-img.bbclass > > +++ b/meta/classes/wic-img.bbclass > > @@ -143,6 +143,7 @@ EOSUDO > > export FAKEROOTCMD=${FAKEROOTCMD} > > export BUILDDIR=${BUILDDIR} > > export MTOOLS_SKIP_CHECK=1 > > + mkdir -p ${IMAGE_ROOTFS}/../pseudo > > > > # create the temp dir in the buildchroot to ensure uniqueness > > WICTMP=$(cd ${BUILDCHROOT_DIR}; mktemp -d -p tmp) > > @@ -157,22 +158,31 @@ EOSUDO > > IMAGE_FULLNAME="$6" > > IMAGE_BASENAME="$7" > > shift 7 > > - > > + # The python path is hard-coded as > > /usr/bin/python3-native/python3 in wic. Handle that. > > + mkdir -p /usr/bin/python3-native/ > > + if [ $(head -1 $(which bmaptool) | grep python3) ];then > > + ln -s /usr/bin/python3 /usr/bin/python3-native/python3 > > + else > > + ln -s /usr/bin/python2 /usr/bin/python3-native/python3 > > + fi > > Can this be moved to wic_fakeroot? That script was designed to hold all > the weird hacks ... > Unfortunately no. This invocation doesnot use fakeroot at all. Please see [1] & [2]. pseudo argument not used for exec_native_cmd which will in-turn not use fakeroot. [1] https://github.com/openembedded/openembedded-core/blob/404292b570a78895a1c7900eeb319e36e31dec20/scripts/lib/wic/plugins/imager/direct.py#L210 [2] https://github.com/openembedded/openembedded-core/blob/404292b570a78895a1c7900eeb319e36e31dec20/scripts/lib/wic/misc.py#L138 > I guess you need to "ln -sf" to make that whole thing executable > multiple times. > Sorry I did not get this completely. -f could be good to have to overwrite. We are remove this after wic runs so i didnot use -f here. > > Can you explain why we need the case where python2 is linked as python3? > I thought I captured it in commit message but looks like I have missed it. Because of stretch. bmaptools in stretch is python2 based and in buster and subsequent releases it is python3 based. OE-core in [1] has hardcoded to python3 since they have moved bmaptools to the latest one which uses python3. We would still need to workaround [1] for stretch too. Will capture this missed detail in commit message in V3. [1] https://github.com/openembedded/openembedded-core/blob/404292b570a78895a1c7900eeb319e36e31dec20/scripts/lib/wic/plugins/imager/direct.py#L210 > > export PATH="$ISARROOT/bitbake/bin:$PATH" > > "$ISARROOT"/scripts/wic create "$WKS_FULL_PATH" \ > > --vars "$STAGING_DIR/$MACHINE/imgdata/" \ > > -o "/$WICTMP/${IMAGE_FULLNAME}.wic/" \ > > --bmap \ > > - -e "$IMAGE_BASENAME" $@' \ > > + -e "$IMAGE_BASENAME" $@ > > + rm -rf /usr/bin/python3-native' \ > > This looks like the rm is in the middle of wic, should probably come > after. > It is at the end of my_script. Should be okay but bit difficult to read though. > > my_script "${ISARROOT}" "${WKS_FULL_PATH}" > > "${STAGING_DIR}" \ "${MACHINE}" "${WICTMP}" "${IMAGE_FULLNAME}" > > "${IMAGE_BASENAME}" \ ${WIC_CREATE_EXTRA_ARGS} > > + > > sudo chown -R $(stat -c "%U" ${ISARROOT}) ${ISARROOT}/meta > > ${ISARROOT}/meta-isar ${ISARROOT}/scripts || true WIC_DIRECT=$(ls -t > > -1 ${BUILDCHROOT_DIR}/$WICTMP/${IMAGE_FULLNAME}.wic/*.direct | head > > -1) sudo chown -R $(id -u):$(id -g) ${BUILDCHROOT_DIR}/${WICTMP} mv > > -f ${WIC_DIRECT} ${WIC_IMAGE_FILE} mv -f ${WIC_DIRECT}.bmap > > ${WIC_IMAGE_FILE}.bmap rm -rf ${BUILDCHROOT_DIR}/${WICTMP} > > + rm -rf ${IMAGE_ROOTFS}/../pseudo > > } > > > > do_wic_image[file-checksums] += "${WKS_FILE_CHECKSUM}" > > [-- Attachment #1.2: Type: text/html, Size: 6088 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 03/10] wic-img: Satisfy the quirks of latest wic 2020-09-05 16:24 ` vijaikumar....@gmail.com @ 2020-09-09 14:52 ` Henning Schild 0 siblings, 0 replies; 29+ messages in thread From: Henning Schild @ 2020-09-09 14:52 UTC (permalink / raw) To: vijaikumar....@gmail.com; +Cc: isar-users On Sat, 5 Sep 2020 09:24:08 -0700 (PDT) "vijaikumar....@gmail.com" <vijaikumar.kanagarajan@gmail.com> wrote: > On Saturday, September 5, 2020 at 2:00:59 PM UTC+5:30 Henning Schild > wrote: > > > On Thu, 3 Sep 2020 00:26:17 +0530 > > Vijai Kumar K <Vijaikumar_...@mentor.com> wrote: > > > > > The latest WIC doesnot use fakeroot if > > > pseudo_dir(IMAGE_ROOTFS/../pseudo) is non-existent. This results > > > in wic_fakeroot not being used. Create a dummy pseudo_dir to > > > satisfy wic. > > > > > > WIC hardcodes the python(/usr/bin/python3-native/python3) to be > > > used while running bmaptool. The above path is OE specific and > > > hence breaks build in ISAR. Create a symbolic link in the above > > > location to satisfy wic. > > > > > > Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com> > > > --- > > > meta/classes/wic-img.bbclass | 14 ++++++++++++-- > > > 1 file changed, 12 insertions(+), 2 deletions(-) > > > > > > diff --git a/meta/classes/wic-img.bbclass > > > b/meta/classes/wic-img.bbclass index afab394..a2c9627 100644 > > > --- a/meta/classes/wic-img.bbclass > > > +++ b/meta/classes/wic-img.bbclass > > > @@ -143,6 +143,7 @@ EOSUDO > > > export FAKEROOTCMD=${FAKEROOTCMD} > > > export BUILDDIR=${BUILDDIR} > > > export MTOOLS_SKIP_CHECK=1 > > > + mkdir -p ${IMAGE_ROOTFS}/../pseudo > > > > > > # create the temp dir in the buildchroot to ensure uniqueness > > > WICTMP=$(cd ${BUILDCHROOT_DIR}; mktemp -d -p tmp) > > > @@ -157,22 +158,31 @@ EOSUDO > > > IMAGE_FULLNAME="$6" > > > IMAGE_BASENAME="$7" > > > shift 7 > > > - > > > + # The python path is hard-coded as > > > /usr/bin/python3-native/python3 in wic. Handle that. > > > + mkdir -p /usr/bin/python3-native/ > > > + if [ $(head -1 $(which bmaptool) | grep python3) ];then > > > + ln -s /usr/bin/python3 /usr/bin/python3-native/python3 > > > + else > > > + ln -s /usr/bin/python2 /usr/bin/python3-native/python3 > > > + fi > > > > Can this be moved to wic_fakeroot? That script was designed to hold > > all the weird hacks ... > > > > Unfortunately no. This invocation doesnot use fakeroot at all. Please > see [1] & [2]. pseudo argument not used for exec_native_cmd > which will in-turn not use fakeroot. > > [1] > https://github.com/openembedded/openembedded-core/blob/404292b570a78895a1c7900eeb319e36e31dec20/scripts/lib/wic/plugins/imager/direct.py#L210 > [2] > https://github.com/openembedded/openembedded-core/blob/404292b570a78895a1c7900eeb319e36e31dec20/scripts/lib/wic/misc.py#L138 > > > > > I guess you need to "ln -sf" to make that whole thing executable > > multiple times. > > > > Sorry I did not get this completely. -f could be good to have to > overwrite. We are remove this > after wic runs so i didnot use -f here. if wic fails for whatever reason ... i.e. missing or broken wks file ... a partial rebuild after the bugfix would run "ln -s" again while "rm" has never been done. Did not try, but it is in general a good idea to write any task so that it can be partially re-executed. you could also look into do_wic_image[cleandirs] In fact those symlinks could probably be placed there using a package-recipe and installed as a wic imager_depends. Just an idea, might be overkill. Henning > > > > > Can you explain why we need the case where python2 is linked as > > python3? > > I thought I captured it in commit message but looks like I have > missed it. Because of stretch. bmaptools in stretch is python2 based > and in buster and subsequent releases it is python3 based. > OE-core in [1] has hardcoded to python3 since they have moved > bmaptools to the latest one > which uses python3. We would still need to workaround [1] for stretch > too. > > Will capture this missed detail in commit message in V3. > > [1] > https://github.com/openembedded/openembedded-core/blob/404292b570a78895a1c7900eeb319e36e31dec20/scripts/lib/wic/plugins/imager/direct.py#L210 > > > > > export PATH="$ISARROOT/bitbake/bin:$PATH" > > > "$ISARROOT"/scripts/wic create "$WKS_FULL_PATH" \ > > > --vars "$STAGING_DIR/$MACHINE/imgdata/" \ > > > -o "/$WICTMP/${IMAGE_FULLNAME}.wic/" \ > > > --bmap \ > > > - -e "$IMAGE_BASENAME" $@' \ > > > + -e "$IMAGE_BASENAME" $@ > > > + rm -rf /usr/bin/python3-native' \ > > > > This looks like the rm is in the middle of wic, should probably > > come after. > > > > It is at the end of my_script. Should be okay but bit difficult to > read though. > > > > > my_script "${ISARROOT}" "${WKS_FULL_PATH}" > > > "${STAGING_DIR}" \ "${MACHINE}" "${WICTMP}" "${IMAGE_FULLNAME}" > > > "${IMAGE_BASENAME}" \ ${WIC_CREATE_EXTRA_ARGS} > > > + > > > sudo chown -R $(stat -c "%U" ${ISARROOT}) ${ISARROOT}/meta > > > ${ISARROOT}/meta-isar ${ISARROOT}/scripts || true WIC_DIRECT=$(ls > > > -t -1 ${BUILDCHROOT_DIR}/$WICTMP/${IMAGE_FULLNAME}.wic/*.direct | > > > head -1) sudo chown -R $(id -u):$(id -g) > > > ${BUILDCHROOT_DIR}/${WICTMP} mv -f ${WIC_DIRECT} > > > ${WIC_IMAGE_FILE} mv -f ${WIC_DIRECT}.bmap ${WIC_IMAGE_FILE}.bmap > > > rm -rf ${BUILDCHROOT_DIR}/${WICTMP} > > > + rm -rf ${IMAGE_ROOTFS}/../pseudo > > > } > > > > > > do_wic_image[file-checksums] += "${WKS_FILE_CHECKSUM}" > > > > > ^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v2 04/10] oe.path: Add copyhardlink() helper function 2020-09-02 18:56 [PATCH v2 00/10] WIC update Vijai Kumar K ` (2 preceding siblings ...) 2020-09-02 18:56 ` [PATCH v2 03/10] wic-img: Satisfy the quirks of " Vijai Kumar K @ 2020-09-02 18:56 ` Vijai Kumar K 2020-09-02 18:56 ` [PATCH v2 05/10] lib/oe/path: try hardlinking instead of guessing when it might fail Vijai Kumar K ` (4 subsequent siblings) 8 siblings, 0 replies; 29+ messages in thread From: Vijai Kumar K @ 2020-09-02 18:56 UTC (permalink / raw) To: isar-users, Vijaikumar_Kanagarajan From: Paul Barker <paul@betafive.co.uk> This function creates hard links if possible, falling back to copying the file if the destination is on a different volume to the source. The docstring for copyhardlinktree() is also updated to make the difference between the two functions a little clearer. Signed-off-by: Paul Barker <paul@betafive.co.uk> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> [Vijai: Imported from OE-core 5437efa16f9bec914e417c6c939a39c247084f52] Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> --- meta/lib/oe/path.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py index 90c87f5..c6bb604 100644 --- a/meta/lib/oe/path.py +++ b/meta/lib/oe/path.py @@ -94,7 +94,7 @@ def copytree(src, dst): subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) def copyhardlinktree(src, dst): - """ Make the hard link when possible, otherwise copy. """ + """Make a tree of hard links when possible, otherwise copy.""" bb.utils.mkdirhier(dst) if os.path.isdir(src) and not len(os.listdir(src)): return @@ -118,6 +118,17 @@ def copyhardlinktree(src, dst): else: copytree(src, dst) +def copyhardlink(src, dst): + """Make a hard link when possible, otherwise copy.""" + + # We need to stat the destination directory as the destination file probably + # doesn't exist yet. + dstdir = os.path.dirname(dst) + if os.stat(src).st_dev == os.stat(dstdir).st_dev: + os.link(src, dst) + else: + shutil.copy(src, dst) + def remove(path, recurse=True): """ Equivalent to rm -f or rm -rf -- 2.17.1 ^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v2 05/10] lib/oe/path: try hardlinking instead of guessing when it might fail 2020-09-02 18:56 [PATCH v2 00/10] WIC update Vijai Kumar K ` (3 preceding siblings ...) 2020-09-02 18:56 ` [PATCH v2 04/10] oe.path: Add copyhardlink() helper function Vijai Kumar K @ 2020-09-02 18:56 ` Vijai Kumar K 2020-09-02 18:56 ` [PATCH v2 06/10] wic_fakeroot: Handle standalone pseudo invocations Vijai Kumar K ` (3 subsequent siblings) 8 siblings, 0 replies; 29+ messages in thread From: Vijai Kumar K @ 2020-09-02 18:56 UTC (permalink / raw) To: isar-users, Vijaikumar_Kanagarajan From: Henning Schild <henning.schild@siemens.com> The comparison of the stat st_dev is not enough to judge whether hardlinking will work. One example would be where you try and hardlink across two bind-mounts of a directory. The st_dev will be the same and the operation will still fail. Instead of implementing a check to try and figure out hardlink support just try hardlinking and fall back to a copy when running into an exception. Signed-off-by: Henning Schild <henning.schild@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> [Vijai: Imported from OE core f5571bda8327f927feb23b167ab4594b7d0c95bc] Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> --- meta/lib/oe/path.py | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py index c6bb604..3506e2c 100644 --- a/meta/lib/oe/path.py +++ b/meta/lib/oe/path.py @@ -99,7 +99,22 @@ def copyhardlinktree(src, dst): if os.path.isdir(src) and not len(os.listdir(src)): return - if (os.stat(src).st_dev == os.stat(dst).st_dev): + canhard = False + testfile = None + for root, dirs, files in os.walk(src): + if len(files): + testfile = os.path.join(root, files[0]) + break + + if testfile is not None: + try: + os.link(testfile, os.path.join(dst, 'testfile')) + os.unlink(os.path.join(dst, 'testfile')) + canhard = True + except Exception as e: + bb.debug(2, "Hardlink test failed with " + str(e)) + + if (canhard): # Need to copy directories only with tar first since cp will error if two # writers try and create a directory at the same time cmd = "cd %s; find . -type d -print | tar --xattrs --xattrs-include='*' -cf - -S -C %s -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' -xhf - -C %s" % (src, src, dst) @@ -121,12 +136,9 @@ def copyhardlinktree(src, dst): def copyhardlink(src, dst): """Make a hard link when possible, otherwise copy.""" - # We need to stat the destination directory as the destination file probably - # doesn't exist yet. - dstdir = os.path.dirname(dst) - if os.stat(src).st_dev == os.stat(dstdir).st_dev: + try: os.link(src, dst) - else: + except OSError: shutil.copy(src, dst) def remove(path, recurse=True): -- 2.17.1 ^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v2 06/10] wic_fakeroot: Handle standalone pseudo invocations 2020-09-02 18:56 [PATCH v2 00/10] WIC update Vijai Kumar K ` (4 preceding siblings ...) 2020-09-02 18:56 ` [PATCH v2 05/10] lib/oe/path: try hardlinking instead of guessing when it might fail Vijai Kumar K @ 2020-09-02 18:56 ` Vijai Kumar K 2020-09-05 8:38 ` Henning Schild 2020-09-05 9:19 ` Henning Schild 2020-09-02 18:56 ` [PATCH v2 07/10] meta-isar/conf: Add provision to debug WIC Vijai Kumar K ` (2 subsequent siblings) 8 siblings, 2 replies; 29+ messages in thread From: Vijai Kumar K @ 2020-09-02 18:56 UTC (permalink / raw) To: isar-users, Vijaikumar_Kanagarajan When using --exclude-path option wic copies the rootfs to a new location and invokes pseudo as a standalone command to rebuild the database in the new rootfs. This is not applicable when using wic_fakeroot. Return 0 for such standalone invocations in wic_fakeroot. It also looks for files.db inside the pseudo directory and throws an exception if it is not found. Handle that too. Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> --- meta/classes/wic-img.bbclass | 1 + scripts/wic_fakeroot | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass index a2c9627..b1a7259 100644 --- a/meta/classes/wic-img.bbclass +++ b/meta/classes/wic-img.bbclass @@ -144,6 +144,7 @@ EOSUDO export BUILDDIR=${BUILDDIR} export MTOOLS_SKIP_CHECK=1 mkdir -p ${IMAGE_ROOTFS}/../pseudo + touch ${IMAGE_ROOTFS}/../pseudo/files.db # create the temp dir in the buildchroot to ensure uniqueness WICTMP=$(cd ${BUILDCHROOT_DIR}; mktemp -d -p tmp) diff --git a/scripts/wic_fakeroot b/scripts/wic_fakeroot index 88a03fa..16b011e 100755 --- a/scripts/wic_fakeroot +++ b/scripts/wic_fakeroot @@ -25,6 +25,11 @@ cmd = args[0] # rootfs/root ... assert os.geteuid() == 0, "wic_fakeroot must be run as root!" +# Check if we are calling the pseudo command itself. Return 1 +# for standalone pseudo operations. +if cmd.startswith('-'): + sys.exit(0) + # e2fsck <= 1.43.5 returns 1 on non-errors (stretch and before affected) # treat 1 as safe ... the filesystem was successfully repaired and is OK if cmd.startswith('fsck.'): -- 2.17.1 ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 06/10] wic_fakeroot: Handle standalone pseudo invocations 2020-09-02 18:56 ` [PATCH v2 06/10] wic_fakeroot: Handle standalone pseudo invocations Vijai Kumar K @ 2020-09-05 8:38 ` Henning Schild 2020-09-05 15:19 ` vijaikumar....@gmail.com 2020-09-05 9:19 ` Henning Schild 1 sibling, 1 reply; 29+ messages in thread From: Henning Schild @ 2020-09-05 8:38 UTC (permalink / raw) To: Vijai Kumar K; +Cc: isar-users On Thu, 3 Sep 2020 00:26:20 +0530 Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> wrote: > When using --exclude-path option wic copies the rootfs to a new > location and invokes pseudo as a standalone command to rebuild the > database in the new rootfs. > > This is not applicable when using wic_fakeroot. Return 0 for such > standalone invocations in wic_fakeroot. > > It also looks for files.db inside the pseudo directory and throws an > exception if it is not found. Handle that too. > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > --- > meta/classes/wic-img.bbclass | 1 + > scripts/wic_fakeroot | 5 +++++ > 2 files changed, 6 insertions(+) > > diff --git a/meta/classes/wic-img.bbclass > b/meta/classes/wic-img.bbclass index a2c9627..b1a7259 100644 > --- a/meta/classes/wic-img.bbclass > +++ b/meta/classes/wic-img.bbclass > @@ -144,6 +144,7 @@ EOSUDO > export BUILDDIR=${BUILDDIR} > export MTOOLS_SKIP_CHECK=1 > mkdir -p ${IMAGE_ROOTFS}/../pseudo > + touch ${IMAGE_ROOTFS}/../pseudo/files.db Where is this coming from? It is not mentioned in the commit message and not used in the code. > > # create the temp dir in the buildchroot to ensure uniqueness > WICTMP=$(cd ${BUILDCHROOT_DIR}; mktemp -d -p tmp) > diff --git a/scripts/wic_fakeroot b/scripts/wic_fakeroot > index 88a03fa..16b011e 100755 > --- a/scripts/wic_fakeroot > +++ b/scripts/wic_fakeroot > @@ -25,6 +25,11 @@ cmd = args[0] > # rootfs/root ... > assert os.geteuid() == 0, "wic_fakeroot must be run as root!" > > +# Check if we are calling the pseudo command itself. Return 1 > +# for standalone pseudo operations. > +if cmd.startswith('-'): > + sys.exit(0) I find it hard to match the comment to the code i see. "-" means its not a cmd but an arg to wic_fakeroot? And what about the 0 vs 1. Henning > # e2fsck <= 1.43.5 returns 1 on non-errors (stretch and before > affected) # treat 1 as safe ... the filesystem was successfully > repaired and is OK if cmd.startswith('fsck.'): ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 06/10] wic_fakeroot: Handle standalone pseudo invocations 2020-09-05 8:38 ` Henning Schild @ 2020-09-05 15:19 ` vijaikumar....@gmail.com 0 siblings, 0 replies; 29+ messages in thread From: vijaikumar....@gmail.com @ 2020-09-05 15:19 UTC (permalink / raw) To: isar-users [-- Attachment #1.1: Type: text/plain, Size: 3384 bytes --] On Saturday, September 5, 2020 at 2:08:07 PM UTC+5:30 Henning Schild wrote: > On Thu, 3 Sep 2020 00:26:20 +0530 > Vijai Kumar K <Vijaikumar_...@mentor.com> wrote: > > > When using --exclude-path option wic copies the rootfs to a new > > location and invokes pseudo as a standalone command to rebuild the > > database in the new rootfs. > > > > This is not applicable when using wic_fakeroot. Return 0 for such > > standalone invocations in wic_fakeroot. > > > > It also looks for files.db inside the pseudo directory and throws an > > exception if it is not found. Handle that too. > > > > Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com> > > --- > > meta/classes/wic-img.bbclass | 1 + > > scripts/wic_fakeroot | 5 +++++ > > 2 files changed, 6 insertions(+) > > > > diff --git a/meta/classes/wic-img.bbclass > > b/meta/classes/wic-img.bbclass index a2c9627..b1a7259 100644 > > --- a/meta/classes/wic-img.bbclass > > +++ b/meta/classes/wic-img.bbclass > > @@ -144,6 +144,7 @@ EOSUDO > > export BUILDDIR=${BUILDDIR} > > export MTOOLS_SKIP_CHECK=1 > > mkdir -p ${IMAGE_ROOTFS}/../pseudo > > + touch ${IMAGE_ROOTFS}/../pseudo/files.db > > Where is this coming from? It is not mentioned in the commit message > and not used in the code. > This is to handle [2]. These kind of workarounds come because we use fakeroot. And our fakeroot was just to handle the fsck issue in stretch. That issue is still there in stretch package. The other approach is to drop the wic_fakeroot and these subsequent quirks handling and carry one patch on top of wic just for the fsck support in stretch. Anyway I proceeded with wic_fakeroot assuming that it might be useful when facing such package compatibility issues. But I see that has become an overhead. If only we can carry one patch on top of wic this all touch pseudo/files.db, startswith(-) quirks are not needed. [2]https://github.com/openembedded/openembedded-core/blob/404292b570a78895a1c7900eeb319e36e31dec20/scripts/lib/wic/plugins/source/rootfs.py#L130 > > > > # create the temp dir in the buildchroot to ensure uniqueness > > WICTMP=$(cd ${BUILDCHROOT_DIR}; mktemp -d -p tmp) > > diff --git a/scripts/wic_fakeroot b/scripts/wic_fakeroot > > index 88a03fa..16b011e 100755 > > --- a/scripts/wic_fakeroot > > +++ b/scripts/wic_fakeroot > > @@ -25,6 +25,11 @@ cmd = args[0] > > # rootfs/root ... > > assert os.geteuid() == 0, "wic_fakeroot must be run as root!" > > > > +# Check if we are calling the pseudo command itself. Return 1 > > +# for standalone pseudo operations. > > +if cmd.startswith('-'): > > + sys.exit(0) > > I find it hard to match the comment to the code i see. "-" means its > not a cmd but an arg to wic_fakeroot? And what about the 0 vs 1. > Yes. There is an instance[1] where FAKEROOT, in case of oe the pseudo, is called as a standalone command with options. We are checking whether the first option is an argument starting with '-' returning 0 on such calls. Good catch. I will fix the comment. [1] https://github.com/openembedded/openembedded-core/blob/404292b570a78895a1c7900eeb319e36e31dec20/scripts/lib/wic/plugins/source/rootfs.py#L133 > Henning > > > # e2fsck <= 1.43.5 returns 1 on non-errors (stretch and before > > affected) # treat 1 as safe ... the filesystem was successfully > > repaired and is OK if cmd.startswith('fsck.'): > > [-- Attachment #1.2: Type: text/html, Size: 4547 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 06/10] wic_fakeroot: Handle standalone pseudo invocations 2020-09-02 18:56 ` [PATCH v2 06/10] wic_fakeroot: Handle standalone pseudo invocations Vijai Kumar K 2020-09-05 8:38 ` Henning Schild @ 2020-09-05 9:19 ` Henning Schild 2020-09-05 15:22 ` vijaikumar....@gmail.com 1 sibling, 1 reply; 29+ messages in thread From: Henning Schild @ 2020-09-05 9:19 UTC (permalink / raw) To: Vijai Kumar K; +Cc: isar-users wic_fakeroot says that it can eventually be replaced with "true", which would make this patch obsolete. And probably the python symlinking from the previous patch. The stretch debian package has gotten a few updates since, and jessie is not support by isar anymore. So maybe worth checking if the fsck hack is still needed, and therefore the whole script. Also what wic calls might have changed, or it might itself "ignore" the problem that my code seems to work around there. Same goes for "export MTOOLS_SKIP_CHECK=1", that might not be required anymore. and should be double-checked. Henning On Thu, 3 Sep 2020 00:26:20 +0530 Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> wrote: > When using --exclude-path option wic copies the rootfs to a new > location and invokes pseudo as a standalone command to rebuild the > database in the new rootfs. > > This is not applicable when using wic_fakeroot. Return 0 for such > standalone invocations in wic_fakeroot. > > It also looks for files.db inside the pseudo directory and throws an > exception if it is not found. Handle that too. > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > --- > meta/classes/wic-img.bbclass | 1 + > scripts/wic_fakeroot | 5 +++++ > 2 files changed, 6 insertions(+) > > diff --git a/meta/classes/wic-img.bbclass > b/meta/classes/wic-img.bbclass index a2c9627..b1a7259 100644 > --- a/meta/classes/wic-img.bbclass > +++ b/meta/classes/wic-img.bbclass > @@ -144,6 +144,7 @@ EOSUDO > export BUILDDIR=${BUILDDIR} > export MTOOLS_SKIP_CHECK=1 > mkdir -p ${IMAGE_ROOTFS}/../pseudo > + touch ${IMAGE_ROOTFS}/../pseudo/files.db > > # create the temp dir in the buildchroot to ensure uniqueness > WICTMP=$(cd ${BUILDCHROOT_DIR}; mktemp -d -p tmp) > diff --git a/scripts/wic_fakeroot b/scripts/wic_fakeroot > index 88a03fa..16b011e 100755 > --- a/scripts/wic_fakeroot > +++ b/scripts/wic_fakeroot > @@ -25,6 +25,11 @@ cmd = args[0] > # rootfs/root ... > assert os.geteuid() == 0, "wic_fakeroot must be run as root!" > > +# Check if we are calling the pseudo command itself. Return 1 > +# for standalone pseudo operations. > +if cmd.startswith('-'): > + sys.exit(0) > + > # e2fsck <= 1.43.5 returns 1 on non-errors (stretch and before > affected) # treat 1 as safe ... the filesystem was successfully > repaired and is OK if cmd.startswith('fsck.'): ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 06/10] wic_fakeroot: Handle standalone pseudo invocations 2020-09-05 9:19 ` Henning Schild @ 2020-09-05 15:22 ` vijaikumar....@gmail.com 0 siblings, 0 replies; 29+ messages in thread From: vijaikumar....@gmail.com @ 2020-09-05 15:22 UTC (permalink / raw) To: isar-users [-- Attachment #1.1: Type: text/plain, Size: 2821 bytes --] On Saturday, September 5, 2020 at 2:49:26 PM UTC+5:30 Henning Schild wrote: > wic_fakeroot says that it can eventually be replaced with "true", which > would make this patch obsolete. And probably the python symlinking from > the previous patch. > > The stretch debian package has gotten a few updates since, and jessie > is not support by isar anymore. So maybe worth checking if the fsck > hack is still needed, and therefore the whole script. Also what wic > calls might have changed, or it might itself "ignore" the problem that > my code seems to work around there. > When I tested, this was not the case, stretch would still need this fsck hack to work. > > Same goes for "export MTOOLS_SKIP_CHECK=1", that might not be required > anymore. and should be double-checked. > Not either. We would still need this at least for stretch I think. I will check again though just to verify my observation back then. > > Henning > > > On Thu, 3 Sep 2020 00:26:20 +0530 > Vijai Kumar K <Vijaikumar_...@mentor.com> wrote: > > > When using --exclude-path option wic copies the rootfs to a new > > location and invokes pseudo as a standalone command to rebuild the > > database in the new rootfs. > > > > This is not applicable when using wic_fakeroot. Return 0 for such > > standalone invocations in wic_fakeroot. > > > > It also looks for files.db inside the pseudo directory and throws an > > exception if it is not found. Handle that too. > > > > Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com> > > --- > > meta/classes/wic-img.bbclass | 1 + > > scripts/wic_fakeroot | 5 +++++ > > 2 files changed, 6 insertions(+) > > > > diff --git a/meta/classes/wic-img.bbclass > > b/meta/classes/wic-img.bbclass index a2c9627..b1a7259 100644 > > --- a/meta/classes/wic-img.bbclass > > +++ b/meta/classes/wic-img.bbclass > > @@ -144,6 +144,7 @@ EOSUDO > > export BUILDDIR=${BUILDDIR} > > export MTOOLS_SKIP_CHECK=1 > > mkdir -p ${IMAGE_ROOTFS}/../pseudo > > + touch ${IMAGE_ROOTFS}/../pseudo/files.db > > > > # create the temp dir in the buildchroot to ensure uniqueness > > WICTMP=$(cd ${BUILDCHROOT_DIR}; mktemp -d -p tmp) > > diff --git a/scripts/wic_fakeroot b/scripts/wic_fakeroot > > index 88a03fa..16b011e 100755 > > --- a/scripts/wic_fakeroot > > +++ b/scripts/wic_fakeroot > > @@ -25,6 +25,11 @@ cmd = args[0] > > # rootfs/root ... > > assert os.geteuid() == 0, "wic_fakeroot must be run as root!" > > > > +# Check if we are calling the pseudo command itself. Return 1 > > +# for standalone pseudo operations. > > +if cmd.startswith('-'): > > + sys.exit(0) > > + > > # e2fsck <= 1.43.5 returns 1 on non-errors (stretch and before > > affected) # treat 1 as safe ... the filesystem was successfully > > repaired and is OK if cmd.startswith('fsck.'): > > [-- Attachment #1.2: Type: text/html, Size: 3781 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v2 07/10] meta-isar/conf: Add provision to debug WIC 2020-09-02 18:56 [PATCH v2 00/10] WIC update Vijai Kumar K ` (5 preceding siblings ...) 2020-09-02 18:56 ` [PATCH v2 06/10] wic_fakeroot: Handle standalone pseudo invocations Vijai Kumar K @ 2020-09-02 18:56 ` Vijai Kumar K 2020-09-02 18:56 ` [PATCH v2 08/10] debian-common: Add tar as a dependency for wic Vijai Kumar K 2020-09-03 5:46 ` [PATCH v2 00/10] WIC update vijaikumar....@gmail.com 8 siblings, 0 replies; 29+ messages in thread From: Vijai Kumar K @ 2020-09-02 18:56 UTC (permalink / raw) To: isar-users, Vijaikumar_Kanagarajan Add provision in local conf to enable wic debug flag. It is helpful in case things doesnot work as expected. Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> --- meta-isar/conf/local.conf.sample | 3 +++ 1 file changed, 3 insertions(+) diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample index 274cdae..25a63f1 100644 --- a/meta-isar/conf/local.conf.sample +++ b/meta-isar/conf/local.conf.sample @@ -199,3 +199,6 @@ USER_isar[gid] = "isar" USER_isar[home] = "/var/lib/isar" USER_isar[comment] = "My isar user" USER_isar[flags] = "system create-home" + +# Uncomment the below line to debug WIC. +# WIC_CREATE_EXTRA_ARGS += "-D" -- 2.17.1 ^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v2 08/10] debian-common: Add tar as a dependency for wic 2020-09-02 18:56 [PATCH v2 00/10] WIC update Vijai Kumar K ` (6 preceding siblings ...) 2020-09-02 18:56 ` [PATCH v2 07/10] meta-isar/conf: Add provision to debug WIC Vijai Kumar K @ 2020-09-02 18:56 ` Vijai Kumar K 2020-09-02 19:02 ` [PATCH v2 09/10] wic: misc: Add /bin to the list of searchpaths Vijai Kumar K 2020-09-03 5:46 ` [PATCH v2 00/10] WIC update vijaikumar....@gmail.com 8 siblings, 1 reply; 29+ messages in thread From: Vijai Kumar K @ 2020-09-02 18:56 UTC (permalink / raw) To: isar-users, Vijaikumar_Kanagarajan For --include-path option, wic requires tar. Add that as a dependency. Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> --- meta/conf/distro/debian-common.conf | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf index 7d681c4..3e3e7d2 100644 --- a/meta/conf/distro/debian-common.conf +++ b/meta/conf/distro/debian-common.conf @@ -14,7 +14,8 @@ WIC_IMAGER_INSTALL = "parted \ mtools \ e2fsprogs \ python3 \ - bmap-tools" + bmap-tools \ + tar" GRUB_BOOTLOADER_INSTALL_amd64 = "grub-efi-amd64-bin" GRUB_BOOTLOADER_INSTALL_i386 = "grub-efi-ia32-bin" -- 2.17.1 ^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v2 09/10] wic: misc: Add /bin to the list of searchpaths 2020-09-02 18:56 ` [PATCH v2 08/10] debian-common: Add tar as a dependency for wic Vijai Kumar K @ 2020-09-02 19:02 ` Vijai Kumar K 2020-09-02 19:02 ` [PATCH v2 10/10] meta-isar/canned-wks: Remove /boot mountpoint Vijai Kumar K 2020-09-05 8:45 ` [PATCH v2 09/10] wic: misc: Add /bin to the list of searchpaths Henning Schild 0 siblings, 2 replies; 29+ messages in thread From: Vijai Kumar K @ 2020-09-02 19:02 UTC (permalink / raw) To: isar-users, Vijaikumar_Kanagarajan From: Vijai Kumar K <vijaikumar.kanagarajan@gmail.com> /bin is also a valid path where one can find executables. Add that to the search path. Signed-off-by: Vijai Kumar K <vijaikumar.kanagarajan@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> [cherry-picked from OE-core ca0a6025351cb2135e87cecf828633cf12aa34c6] Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> --- scripts/lib/wic/misc.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py index 91975ba..4b08d64 100644 --- a/scripts/lib/wic/misc.py +++ b/scripts/lib/wic/misc.py @@ -138,8 +138,9 @@ def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""): if pseudo: cmd_and_args = pseudo + cmd_and_args - native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin" % \ - (native_sysroot, native_sysroot, native_sysroot) + native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin" % \ + (native_sysroot, native_sysroot, + native_sysroot, native_sysroot) native_cmd_and_args = "export PATH=%s:$PATH;%s" % \ (native_paths, cmd_and_args) -- 2.17.1 ^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v2 10/10] meta-isar/canned-wks: Remove /boot mountpoint 2020-09-02 19:02 ` [PATCH v2 09/10] wic: misc: Add /bin to the list of searchpaths Vijai Kumar K @ 2020-09-02 19:02 ` Vijai Kumar K 2020-09-05 8:58 ` Henning Schild 2020-09-05 8:45 ` [PATCH v2 09/10] wic: misc: Add /bin to the list of searchpaths Henning Schild 1 sibling, 1 reply; 29+ messages in thread From: Vijai Kumar K @ 2020-09-02 19:02 UTC (permalink / raw) To: isar-users, Vijaikumar_Kanagarajan In the older version of wic, the fstab entry for /boot mountpoint was skipped. However in the latest wic this is not the case. Adding a /boot mountpoint in part results in creating a fstab entry for automount. Remove it to avoid the unwanted mount which makes original contents unavailable in debian rootfilesystem. Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> --- RECIPE-API-CHANGELOG.md | 17 +++++++++++++++++ .../lib/wic/canned-wks/common-isar.wks.inc | 2 +- meta-isar/scripts/lib/wic/canned-wks/hikey.wks | 2 +- .../scripts/lib/wic/canned-wks/sdimage-efi.wks | 2 +- 4 files changed, 20 insertions(+), 3 deletions(-) diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index 1839770..070cde3 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -246,3 +246,20 @@ by setting DEBIAN_BUILD_DEPENDS. ${S} can now be used for checking out sources without being linked implicitly with ${D} which needs to be filled explicitly in do_install as before. + +### Wic adds /boot mountpoint to fstab + +With the latest wic, /boot mount point, if any, is added to /etc/fstab. Debian +uses /boot to store the kernel and initrd images. If an image's wks file +specifies /boot mountpoint for boot partitions like EFI, the image may fail to +boot or might make the original contents of /boot unavailable after boot. +All unwanted /boot mountpoints should be removed. + +Below is an example wks entry that might cause an issue. +``` +part /boot --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk sda --label efi --part-type EF00 --align 1024 +``` +Drop the /boot entry to avoid issues. +``` +part --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk sda --label efi --part-type EF00 --align 1024 +``` diff --git a/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc b/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc index c8ea4c2..e7dabd4 100644 --- a/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc +++ b/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc @@ -1,3 +1,3 @@ # This file is included into 3 canned wks files from this directory -part /boot --source bootimg-pcbios-isar --ondisk sda --label boot --active --align 1024 +part --source bootimg-pcbios-isar --ondisk sda --label boot --active --align 1024 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 diff --git a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks index 1e82f8c..840858e 100644 --- a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks +++ b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks @@ -3,7 +3,7 @@ # # SPDX-License-Identifier: MIT -part /boot --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk mmcblk1 --label efi --part-type EF00 --align 1024 +part --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk mmcblk1 --label efi --part-type EF00 --align 1024 part / --source rootfs --ondisk mmcblk1 --fstype ext4 --label platform --align 1024 --use-uuid diff --git a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks index 2afa016..e0fa7a1 100644 --- a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks +++ b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks @@ -2,7 +2,7 @@ # long-description: Creates a partitioned EFI disk image without any swap that # the user can directly dd to boot media. -part /boot --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk sda --label efi --part-type EF00 --align 1024 +part --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk sda --label efi --part-type EF00 --align 1024 part / --source rootfs --ondisk sda --fstype ext4 --label platform --align 1024 --use-uuid -- 2.17.1 ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 10/10] meta-isar/canned-wks: Remove /boot mountpoint 2020-09-02 19:02 ` [PATCH v2 10/10] meta-isar/canned-wks: Remove /boot mountpoint Vijai Kumar K @ 2020-09-05 8:58 ` Henning Schild 2020-09-05 16:06 ` vijaikumar....@gmail.com 0 siblings, 1 reply; 29+ messages in thread From: Henning Schild @ 2020-09-05 8:58 UTC (permalink / raw) To: Vijai Kumar K; +Cc: isar-users Hi, i do not like this one at all. Suggesting people to hide the boot partition from debian as hard as they can? If i understand it correctly. In fact the hiding is an issue that should be solved, debian should know about that partition and use it correctly. On EFI it only contains the bootloader, so such isar images will be able to receive kernel updates with apt-get during their lifetime. They will never be able to update the bootloader, but that might be ok. On BIOS the story is different. These systems have the kernel in the roots/boot only that kernel is never used for booting and the machines will never be able to update the kernel they boot. I always wanted to solve that but it was not too easy. The first idea was just bootloader in the boot partition, like efi. But syslinux can not boot files from other partitions, so we need the kernel and initrd there. But now with the exclude feature it might be easier. Place the bootloader and config into rootfs/boot and point to the kernel initrd symlinks in the config. (Done in our custom BIOS plugin) And now take rootfs/boot as boot, excluding it from rootfs. Henning On Thu, 3 Sep 2020 00:32:47 +0530 Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> wrote: > In the older version of wic, the fstab entry for /boot mountpoint > was skipped. > However in the latest wic this is not the case. Adding a /boot > mountpoint in part results in creating a fstab entry for automount. > Remove it to avoid the unwanted mount which makes original contents > unavailable in debian rootfilesystem. > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > --- > RECIPE-API-CHANGELOG.md | 17 > +++++++++++++++++ .../lib/wic/canned-wks/common-isar.wks.inc | > 2 +- meta-isar/scripts/lib/wic/canned-wks/hikey.wks | 2 +- > .../scripts/lib/wic/canned-wks/sdimage-efi.wks | 2 +- > 4 files changed, 20 insertions(+), 3 deletions(-) > > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md > index 1839770..070cde3 100644 > --- a/RECIPE-API-CHANGELOG.md > +++ b/RECIPE-API-CHANGELOG.md > @@ -246,3 +246,20 @@ by setting DEBIAN_BUILD_DEPENDS. > > ${S} can now be used for checking out sources without being linked > implicitly with ${D} which needs to be filled explicitly in > do_install as before. + > +### Wic adds /boot mountpoint to fstab > + > +With the latest wic, /boot mount point, if any, is added to > /etc/fstab. Debian +uses /boot to store the kernel and initrd images. > If an image's wks file +specifies /boot mountpoint for boot > partitions like EFI, the image may fail to +boot or might make the > original contents of /boot unavailable after boot. +All unwanted > /boot mountpoints should be removed. + > +Below is an example wks entry that might cause an issue. > +``` > +part /boot --source bootimg-efi-isar --sourceparams > "loader=grub-efi" --ondisk sda --label efi --part-type EF00 --align > 1024 +``` +Drop the /boot entry to avoid issues. > +``` > +part --source bootimg-efi-isar --sourceparams "loader=grub-efi" > --ondisk sda --label efi --part-type EF00 --align 1024 +``` > diff --git a/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc > b/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc index > c8ea4c2..e7dabd4 100644 --- > a/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc +++ > b/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc @@ -1,3 > +1,3 @@ # This file is included into 3 canned wks files from this > directory -part /boot --source bootimg-pcbios-isar --ondisk sda > --label boot --active --align 1024 +part --source bootimg-pcbios-isar > --ondisk sda --label boot --active --align 1024 part / --source > rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 diff > --git a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks > b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks index > 1e82f8c..840858e 100644 --- > a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks +++ > b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks @@ -3,7 +3,7 @@ # > # SPDX-License-Identifier: MIT > > -part /boot --source bootimg-efi-isar --sourceparams > "loader=grub-efi" --ondisk mmcblk1 --label efi --part-type EF00 > --align 1024 +part --source bootimg-efi-isar --sourceparams > "loader=grub-efi" --ondisk mmcblk1 --label efi --part-type EF00 > --align 1024 part / --source rootfs --ondisk mmcblk1 --fstype ext4 > --label platform --align 1024 --use-uuid diff --git > a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks > b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks index > 2afa016..e0fa7a1 100644 --- > a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks +++ > b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks @@ -2,7 +2,7 > @@ # long-description: Creates a partitioned EFI disk image without > any swap that # the user can directly dd to boot media. > -part /boot --source bootimg-efi-isar --sourceparams > "loader=grub-efi" --ondisk sda --label efi --part-type EF00 --align > 1024 +part --source bootimg-efi-isar --sourceparams "loader=grub-efi" > --ondisk sda --label efi --part-type EF00 --align 1024 part / > --source rootfs --ondisk sda --fstype ext4 --label platform --align > 1024 --use-uuid ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 10/10] meta-isar/canned-wks: Remove /boot mountpoint 2020-09-05 8:58 ` Henning Schild @ 2020-09-05 16:06 ` vijaikumar....@gmail.com 2020-09-09 15:09 ` Henning Schild 0 siblings, 1 reply; 29+ messages in thread From: vijaikumar....@gmail.com @ 2020-09-05 16:06 UTC (permalink / raw) To: isar-users [-- Attachment #1.1: Type: text/plain, Size: 7072 bytes --] On Saturday, September 5, 2020 at 2:28:17 PM UTC+5:30 Henning Schild wrote: > Hi, > > i do not like this one at all. Suggesting people to hide the boot > partition from debian as hard as they can? If i understand it correctly. > AFAIU , /boot mount-point[1] is wrong and is not needed without --exclude-path of /boot in rootfs partition. This optional mount-point serves only one purpose, i.e, to have an entry added in fstab or not by wic. We were okay with saying this EFI/bootloader partition mount-point as /boot, since older wic was not adding the /boot mount-point to fstab. With this commit [2] in OE this is no longer the case. /boot is added to fstab and at boot the system would try to mount to this efi partiton to /boot directory which, if successful, will make the kernel, initrd and config files which were already available in /boot of rootfs unavailable when the system is booted. I donot see --exclude-path for /boot in the wks file, so this would just be wrong to have a mount-point like this. [1] https://github.com/ilbers/isar/blob/a0d6de4ac7dfb926b38cfcd44f2bc8852f2e43e3/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks#L5 [2] https://github.com/openembedded/openembedded-core/commit/2376b05512ddb8c4ec3aaf1df11071f536a76bd9 > In fact the hiding is an issue that should be solved, debian should > know about that partition and use it correctly. > > On EFI it only contains the bootloader, so such isar images will be > able to receive kernel updates with apt-get during their lifetime. They > will never be able to update the bootloader, but that might be ok. > > On BIOS the story is different. These systems have the kernel in the > roots/boot only that kernel is never used for booting and the machines > will never be able to update the kernel they boot. > > I always wanted to solve that but it was not too easy. The first idea > was just bootloader in the boot partition, like efi. But syslinux can > not boot files from other partitions, so we need the kernel and initrd > there. > But now with the exclude feature it might be easier. Place the > bootloader and config into rootfs/boot and point to the kernel initrd > symlinks in the config. (Done in our custom BIOS plugin) > And now take rootfs/boot as boot, excluding it from rootfs. Can you point me to some code where these kind of scenarios are? I assume these are all downstream? For downstream you could customize the wks with neccessary /boot as mountpoint for EFI and "--exclude-path /boot" to achieve what you want. I donot see the issue in removing it here. > > Henning > > On Thu, 3 Sep 2020 00:32:47 +0530 > Vijai Kumar K <Vijaikumar_...@mentor.com> wrote: > > > In the older version of wic, the fstab entry for /boot mountpoint > > was skipped. > > However in the latest wic this is not the case. Adding a /boot > > mountpoint in part results in creating a fstab entry for automount. > > Remove it to avoid the unwanted mount which makes original contents > > unavailable in debian rootfilesystem. > > > > Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com> > > --- > > RECIPE-API-CHANGELOG.md | 17 > > +++++++++++++++++ .../lib/wic/canned-wks/common-isar.wks.inc | > > 2 +- meta-isar/scripts/lib/wic/canned-wks/hikey.wks | 2 +- > > .../scripts/lib/wic/canned-wks/sdimage-efi.wks | 2 +- > > 4 files changed, 20 insertions(+), 3 deletions(-) > > > > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md > > index 1839770..070cde3 100644 > > --- a/RECIPE-API-CHANGELOG.md > > +++ b/RECIPE-API-CHANGELOG.md > > @@ -246,3 +246,20 @@ by setting DEBIAN_BUILD_DEPENDS. > > > > ${S} can now be used for checking out sources without being linked > > implicitly with ${D} which needs to be filled explicitly in > > do_install as before. + > > +### Wic adds /boot mountpoint to fstab > > + > > +With the latest wic, /boot mount point, if any, is added to > > /etc/fstab. Debian +uses /boot to store the kernel and initrd images. > > If an image's wks file +specifies /boot mountpoint for boot > > partitions like EFI, the image may fail to +boot or might make the > > original contents of /boot unavailable after boot. +All unwanted > > /boot mountpoints should be removed. + > Maybe I can add a bit more to this sentence to make it clear, something like unless you know what you are doing. I could add the rest of the wks entry below to highlight that /boot is not excluded in the first place. > > +Below is an example wks entry that might cause an issue. > +``` > > +part /boot --source bootimg-efi-isar --sourceparams > > "loader=grub-efi" --ondisk sda --label efi --part-type EF00 --align > > 1024 +``` +Drop the /boot entry to avoid issues. > > +``` > > +part --source bootimg-efi-isar --sourceparams "loader=grub-efi" > > --ondisk sda --label efi --part-type EF00 --align 1024 +``` > > diff --git a/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc > > b/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc index > > c8ea4c2..e7dabd4 100644 --- > > a/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc +++ > > b/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc @@ -1,3 > > +1,3 @@ # This file is included into 3 canned wks files from this > > directory -part /boot --source bootimg-pcbios-isar --ondisk sda > > --label boot --active --align 1024 +part --source bootimg-pcbios-isar > > --ondisk sda --label boot --active --align 1024 part / --source > > rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 diff > > --git a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks > > b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks index > > 1e82f8c..840858e 100644 --- > > a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks +++ > > b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks @@ -3,7 +3,7 @@ # > > # SPDX-License-Identifier: MIT > > > > -part /boot --source bootimg-efi-isar --sourceparams > > "loader=grub-efi" --ondisk mmcblk1 --label efi --part-type EF00 > > --align 1024 +part --source bootimg-efi-isar --sourceparams > > "loader=grub-efi" --ondisk mmcblk1 --label efi --part-type EF00 > > --align 1024 part / --source rootfs --ondisk mmcblk1 --fstype ext4 > > --label platform --align 1024 --use-uuid diff --git > > a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks > > b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks index > > 2afa016..e0fa7a1 100644 --- > > a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks +++ > > b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks @@ -2,7 +2,7 > > @@ # long-description: Creates a partitioned EFI disk image without > > any swap that # the user can directly dd to boot media. > > -part /boot --source bootimg-efi-isar --sourceparams > > "loader=grub-efi" --ondisk sda --label efi --part-type EF00 --align > > 1024 +part --source bootimg-efi-isar --sourceparams "loader=grub-efi" > > --ondisk sda --label efi --part-type EF00 --align 1024 part / > > --source rootfs --ondisk sda --fstype ext4 --label platform --align > > 1024 --use-uuid > > [-- Attachment #1.2: Type: text/html, Size: 8745 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 10/10] meta-isar/canned-wks: Remove /boot mountpoint 2020-09-05 16:06 ` vijaikumar....@gmail.com @ 2020-09-09 15:09 ` Henning Schild 2020-09-14 5:32 ` vijaikumar....@gmail.com 0 siblings, 1 reply; 29+ messages in thread From: Henning Schild @ 2020-09-09 15:09 UTC (permalink / raw) To: vijaikumar....@gmail.com; +Cc: isar-users On Sat, 5 Sep 2020 09:06:08 -0700 (PDT) "vijaikumar....@gmail.com" <vijaikumar.kanagarajan@gmail.com> wrote: > On Saturday, September 5, 2020 at 2:28:17 PM UTC+5:30 Henning Schild > wrote: > > > Hi, > > > > i do not like this one at all. Suggesting people to hide the boot > > partition from debian as hard as they can? If i understand it > > correctly. > > AFAIU , /boot mount-point[1] is wrong and is not needed without > --exclude-path of /boot in rootfs partition. > This optional mount-point serves only one purpose, i.e, to have an > entry added in fstab or not by wic. > We were okay with saying this EFI/bootloader partition mount-point as > /boot, since older wic > was not adding the /boot mount-point to fstab. > With this commit [2] in OE this is no longer the case. > /boot is added to fstab and at boot the system would try to mount to > this efi partiton to /boot > directory which, if successful, will make the kernel, initrd and > config files which were already available in /boot > of rootfs unavailable when the system is booted. I donot see > --exclude-path for /boot in the wks file, > so this would just be wrong to have a mount-point like this. Not sure i get all that, but to me it seems we need a wks file without boot or we need one with boot and the exclude. The exclude might not be needed on OE, on debian it is ... so adding the exclude could be a better default/example than removing the boot partition. > [1] > https://github.com/ilbers/isar/blob/a0d6de4ac7dfb926b38cfcd44f2bc8852f2e43e3/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks#L5 > [2] > https://github.com/openembedded/openembedded-core/commit/2376b05512ddb8c4ec3aaf1df11071f536a76bd9 > > > > In fact the hiding is an issue that should be solved, debian should > > know about that partition and use it correctly. > > > > On EFI it only contains the bootloader, so such isar images will be > > able to receive kernel updates with apt-get during their lifetime. > > They will never be able to update the bootloader, but that might be > > ok. > > > > On BIOS the story is different. These systems have the kernel in > > the roots/boot only that kernel is never used for booting and the > > machines will never be able to update the kernel they boot. > > > > I always wanted to solve that but it was not too easy. The first > > idea was just bootloader in the boot partition, like efi. But > > syslinux can not boot files from other partitions, so we need the > > kernel and initrd there. > > But now with the exclude feature it might be easier. Place the > > bootloader and config into rootfs/boot and point to the kernel > > initrd symlinks in the config. (Done in our custom BIOS plugin) > > And now take rootfs/boot as boot, excluding it from rootfs. > > > Can you point me to some code where these kind of scenarios are? I > assume these are all downstream? For downstream you could customize > the wks with neccessary /boot as mountpoint for EFI and > "--exclude-path /boot" to achieve > what you want. I donot see the issue in removing it here. There is no code to point to. What i described is the current state of Isar and a possible direction it could take to fix the fact that the boot partition is never visible in the running system. An issue especially problematic for BIOS boot, since kernel updating is broken. Just build a BIOS booting qemu (conf/machine/qemui386.conf), and install i.e. linux-image-rt in the running system with apt-get I am not saying this should not be merged. But this is a good time to look at the "debians do not know about that boot partition" issue again, instead of possibly making it worse and including bad examples into meta-isar ... which will later end up in other layers. Henning > > > > > Henning > > > > On Thu, 3 Sep 2020 00:32:47 +0530 > > Vijai Kumar K <Vijaikumar_...@mentor.com> wrote: > > > > > In the older version of wic, the fstab entry for /boot mountpoint > > > was skipped. > > > However in the latest wic this is not the case. Adding a /boot > > > mountpoint in part results in creating a fstab entry for > > > automount. Remove it to avoid the unwanted mount which makes > > > original contents unavailable in debian rootfilesystem. > > > > > > Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com> > > > --- > > > RECIPE-API-CHANGELOG.md | 17 > > > +++++++++++++++++ .../lib/wic/canned-wks/common-isar.wks.inc | > > > 2 +- meta-isar/scripts/lib/wic/canned-wks/hikey.wks | 2 +- > > > .../scripts/lib/wic/canned-wks/sdimage-efi.wks | 2 +- > > > 4 files changed, 20 insertions(+), 3 deletions(-) > > > > > > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md > > > index 1839770..070cde3 100644 > > > --- a/RECIPE-API-CHANGELOG.md > > > +++ b/RECIPE-API-CHANGELOG.md > > > @@ -246,3 +246,20 @@ by setting DEBIAN_BUILD_DEPENDS. > > > > > > ${S} can now be used for checking out sources without being > > > linked implicitly with ${D} which needs to be filled explicitly > > > in do_install as before. + > > > +### Wic adds /boot mountpoint to fstab > > > + > > > +With the latest wic, /boot mount point, if any, is added to > > > /etc/fstab. Debian +uses /boot to store the kernel and initrd > > > images. If an image's wks file +specifies /boot mountpoint for > > > boot partitions like EFI, the image may fail to +boot or might > > > make the original contents of /boot unavailable after boot. +All > > > unwanted /boot mountpoints should be removed. + > > > > Maybe I can add a bit more to > this sentence to make it clear, something like unless you know what > you are doing. > I could add the rest of the wks entry below to highlight that /boot > is not excluded in the first place. > > > > > +Below is an example wks entry that might cause an issue. > > > +``` > > > +part /boot --source bootimg-efi-isar --sourceparams > > > "loader=grub-efi" --ondisk sda --label efi --part-type EF00 > > > --align 1024 +``` +Drop the /boot entry to avoid issues. > > > +``` > > > +part --source bootimg-efi-isar --sourceparams "loader=grub-efi" > > > --ondisk sda --label efi --part-type EF00 --align 1024 +``` > > > diff --git > > > a/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc > > > b/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc index > > > c8ea4c2..e7dabd4 100644 --- > > > a/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc +++ > > > b/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc @@ > > > -1,3 +1,3 @@ # This file is included into 3 canned wks files from > > > this directory -part /boot --source bootimg-pcbios-isar --ondisk > > > sda --label boot --active --align 1024 +part --source > > > bootimg-pcbios-isar --ondisk sda --label boot --active --align > > > 1024 part / --source rootfs --ondisk sda --fstype=ext4 --label > > > platform --align 1024 diff --git > > > a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks > > > b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks index > > > 1e82f8c..840858e 100644 --- > > > a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks +++ > > > b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks @@ -3,7 +3,7 @@ > > > # # SPDX-License-Identifier: MIT > > > > > > -part /boot --source bootimg-efi-isar --sourceparams > > > "loader=grub-efi" --ondisk mmcblk1 --label efi --part-type EF00 > > > --align 1024 +part --source bootimg-efi-isar --sourceparams > > > "loader=grub-efi" --ondisk mmcblk1 --label efi --part-type EF00 > > > --align 1024 part / --source rootfs --ondisk mmcblk1 --fstype > > > ext4 --label platform --align 1024 --use-uuid diff --git > > > a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks > > > b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks index > > > 2afa016..e0fa7a1 100644 --- > > > a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks +++ > > > b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks @@ -2,7 > > > +2,7 @@ # long-description: Creates a partitioned EFI disk image > > > without any swap that # the user can directly dd to boot media. > > > -part /boot --source bootimg-efi-isar --sourceparams > > > "loader=grub-efi" --ondisk sda --label efi --part-type EF00 > > > --align 1024 +part --source bootimg-efi-isar --sourceparams > > > "loader=grub-efi" --ondisk sda --label efi --part-type EF00 > > > --align 1024 part / --source rootfs --ondisk sda --fstype ext4 > > > --label platform --align 1024 --use-uuid > > > > > ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 10/10] meta-isar/canned-wks: Remove /boot mountpoint 2020-09-09 15:09 ` Henning Schild @ 2020-09-14 5:32 ` vijaikumar....@gmail.com 0 siblings, 0 replies; 29+ messages in thread From: vijaikumar....@gmail.com @ 2020-09-14 5:32 UTC (permalink / raw) To: isar-users [-- Attachment #1.1: Type: text/plain, Size: 9486 bytes --] On Wednesday, September 9, 2020 at 8:39:27 PM UTC+5:30 Henning Schild wrote: > On Sat, 5 Sep 2020 09:06:08 -0700 (PDT) > "vijaikumar....@gmail.com" <vijaikumar....@gmail.com> wrote: > > > On Saturday, September 5, 2020 at 2:28:17 PM UTC+5:30 Henning Schild > > wrote: > > > > > Hi, > > > > > > i do not like this one at all. Suggesting people to hide the boot > > > partition from debian as hard as they can? If i understand it > > > correctly. > > > > AFAIU , /boot mount-point[1] is wrong and is not needed without > > --exclude-path of /boot in rootfs partition. > > This optional mount-point serves only one purpose, i.e, to have an > > entry added in fstab or not by wic. > > We were okay with saying this EFI/bootloader partition mount-point as > > /boot, since older wic > > was not adding the /boot mount-point to fstab. > > With this commit [2] in OE this is no longer the case. > > /boot is added to fstab and at boot the system would try to mount to > > this efi partiton to /boot > > directory which, if successful, will make the kernel, initrd and > > config files which were already available in /boot > > of rootfs unavailable when the system is booted. I donot see > > --exclude-path for /boot in the wks file, > > so this would just be wrong to have a mount-point like this. > > Not sure i get all that, but to me it seems we need a wks file without > boot or we need one with boot and the exclude. The exclude might not be > needed on OE, on debian it is ... so adding the exclude could be a > better default/example than removing the boot partition. > > > [1] > > > https://github.com/ilbers/isar/blob/a0d6de4ac7dfb926b38cfcd44f2bc8852f2e43e3/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks#L5 > > [2] > > > https://github.com/openembedded/openembedded-core/commit/2376b05512ddb8c4ec3aaf1df11071f536a76bd9 > > > > > > > In fact the hiding is an issue that should be solved, debian should > > > know about that partition and use it correctly. > > > > > > On EFI it only contains the bootloader, so such isar images will be > > > able to receive kernel updates with apt-get during their lifetime. > > > They will never be able to update the bootloader, but that might be > > > ok. > > > > > > On BIOS the story is different. These systems have the kernel in > > > the roots/boot only that kernel is never used for booting and the > > > machines will never be able to update the kernel they boot. > > > > > > I always wanted to solve that but it was not too easy. The first > > > idea was just bootloader in the boot partition, like efi. But > > > syslinux can not boot files from other partitions, so we need the > > > kernel and initrd there. > > > But now with the exclude feature it might be easier. Place the > > > bootloader and config into rootfs/boot and point to the kernel > > > initrd symlinks in the config. (Done in our custom BIOS plugin) > > > And now take rootfs/boot as boot, excluding it from rootfs. > > > > > > Can you point me to some code where these kind of scenarios are? I > > assume these are all downstream? For downstream you could customize > > the wks with neccessary /boot as mountpoint for EFI and > > "--exclude-path /boot" to achieve > > what you want. I donot see the issue in removing it here. > > There is no code to point to. What i described is the current state of > Isar and a possible direction it could take to fix the fact that the > boot partition is never visible in the running system. An issue > especially problematic for BIOS boot, since kernel updating is broken. > I see, this is only with the the legacy bios based boot. In that case, since we have the /boot mount automatically, we can modify the wks file to exclude /boot and mount the syslinux boot partition in the rootfs. we might also need to copy over the config and system map files to the syslinux partition for completion. I will add an additional patch for legacy bios and send in v3. For rest of the images, not specifying /boot is the right way, since those fetch the kernel and initrd directly from the root partition. Thanks, Vijai Kumar K > > Just build a BIOS booting qemu (conf/machine/qemui386.conf), and install > i.e. linux-image-rt in the running system with apt-get > > I am not saying this should not be merged. But this is a good time to > look at the "debians do not know about that boot partition" issue > again, instead of possibly making it worse and including bad examples > into meta-isar ... which will later end up in other layers. > > Henning > > > > > > > > > Henning > > > > > > On Thu, 3 Sep 2020 00:32:47 +0530 > > > Vijai Kumar K <Vijaikumar_...@mentor.com> wrote: > > > > > > > In the older version of wic, the fstab entry for /boot mountpoint > > > > was skipped. > > > > However in the latest wic this is not the case. Adding a /boot > > > > mountpoint in part results in creating a fstab entry for > > > > automount. Remove it to avoid the unwanted mount which makes > > > > original contents unavailable in debian rootfilesystem. > > > > > > > > Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com> > > > > --- > > > > RECIPE-API-CHANGELOG.md | 17 > > > > +++++++++++++++++ .../lib/wic/canned-wks/common-isar.wks.inc | > > > > 2 +- meta-isar/scripts/lib/wic/canned-wks/hikey.wks | 2 +- > > > > .../scripts/lib/wic/canned-wks/sdimage-efi.wks | 2 +- > > > > 4 files changed, 20 insertions(+), 3 deletions(-) > > > > > > > > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md > > > > index 1839770..070cde3 100644 > > > > --- a/RECIPE-API-CHANGELOG.md > > > > +++ b/RECIPE-API-CHANGELOG.md > > > > @@ -246,3 +246,20 @@ by setting DEBIAN_BUILD_DEPENDS. > > > > > > > > ${S} can now be used for checking out sources without being > > > > linked implicitly with ${D} which needs to be filled explicitly > > > > in do_install as before. + > > > > +### Wic adds /boot mountpoint to fstab > > > > + > > > > +With the latest wic, /boot mount point, if any, is added to > > > > /etc/fstab. Debian +uses /boot to store the kernel and initrd > > > > images. If an image's wks file +specifies /boot mountpoint for > > > > boot partitions like EFI, the image may fail to +boot or might > > > > make the original contents of /boot unavailable after boot. +All > > > > unwanted /boot mountpoints should be removed. + > > > > > > > Maybe I can add a bit more to > > this sentence to make it clear, something like unless you know what > > you are doing. > > I could add the rest of the wks entry below to highlight that /boot > > is not excluded in the first place. > > > > > > > > +Below is an example wks entry that might cause an issue. > > > > > +``` > > > > +part /boot --source bootimg-efi-isar --sourceparams > > > > "loader=grub-efi" --ondisk sda --label efi --part-type EF00 > > > > --align 1024 +``` +Drop the /boot entry to avoid issues. > > > > +``` > > > > +part --source bootimg-efi-isar --sourceparams "loader=grub-efi" > > > > --ondisk sda --label efi --part-type EF00 --align 1024 +``` > > > > diff --git > > > > a/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc > > > > b/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc index > > > > c8ea4c2..e7dabd4 100644 --- > > > > a/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc +++ > > > > b/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc @@ > > > > -1,3 +1,3 @@ # This file is included into 3 canned wks files from > > > > this directory -part /boot --source bootimg-pcbios-isar --ondisk > > > > sda --label boot --active --align 1024 +part --source > > > > bootimg-pcbios-isar --ondisk sda --label boot --active --align > > > > 1024 part / --source rootfs --ondisk sda --fstype=ext4 --label > > > > platform --align 1024 diff --git > > > > a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks > > > > b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks index > > > > 1e82f8c..840858e 100644 --- > > > > a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks +++ > > > > b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks @@ -3,7 +3,7 @@ > > > > # # SPDX-License-Identifier: MIT > > > > > > > > -part /boot --source bootimg-efi-isar --sourceparams > > > > "loader=grub-efi" --ondisk mmcblk1 --label efi --part-type EF00 > > > > --align 1024 +part --source bootimg-efi-isar --sourceparams > > > > "loader=grub-efi" --ondisk mmcblk1 --label efi --part-type EF00 > > > > --align 1024 part / --source rootfs --ondisk mmcblk1 --fstype > > > > ext4 --label platform --align 1024 --use-uuid diff --git > > > > a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks > > > > b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks index > > > > 2afa016..e0fa7a1 100644 --- > > > > a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks +++ > > > > b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks @@ -2,7 > > > > +2,7 @@ # long-description: Creates a partitioned EFI disk image > > > > without any swap that # the user can directly dd to boot media. > > > > -part /boot --source bootimg-efi-isar --sourceparams > > > > "loader=grub-efi" --ondisk sda --label efi --part-type EF00 > > > > --align 1024 +part --source bootimg-efi-isar --sourceparams > > > > "loader=grub-efi" --ondisk sda --label efi --part-type EF00 > > > > --align 1024 part / --source rootfs --ondisk sda --fstype ext4 > > > > --label platform --align 1024 --use-uuid > > > > > > > > > > [-- Attachment #1.2: Type: text/html, Size: 12390 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 09/10] wic: misc: Add /bin to the list of searchpaths 2020-09-02 19:02 ` [PATCH v2 09/10] wic: misc: Add /bin to the list of searchpaths Vijai Kumar K 2020-09-02 19:02 ` [PATCH v2 10/10] meta-isar/canned-wks: Remove /boot mountpoint Vijai Kumar K @ 2020-09-05 8:45 ` Henning Schild 2020-09-05 9:33 ` vijai kumar 1 sibling, 1 reply; 29+ messages in thread From: Henning Schild @ 2020-09-05 8:45 UTC (permalink / raw) To: Vijai Kumar K; +Cc: isar-users We are patching wic here ... If that is upstream simply take another version in p1. Wic still is "clean" and purely taken from upstream, which probably made your version bump much easier ... it probably was not easy. Dont ruin that on the last meters. Maybe will will reach that clean state for bitbake as well, again. Henning On Thu, 3 Sep 2020 00:32:46 +0530 Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> wrote: > From: Vijai Kumar K <vijaikumar.kanagarajan@gmail.com> > > /bin is also a valid path where one can find executables. Add > that to the search path. > > Signed-off-by: Vijai Kumar K <vijaikumar.kanagarajan@gmail.com> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > [cherry-picked from OE-core ca0a6025351cb2135e87cecf828633cf12aa34c6] > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > --- > scripts/lib/wic/misc.py | 5 +++-- > 1 file changed, 3 insertions(+), 2 deletions(-) > > diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py > index 91975ba..4b08d64 100644 > --- a/scripts/lib/wic/misc.py > +++ b/scripts/lib/wic/misc.py > @@ -138,8 +138,9 @@ def exec_native_cmd(cmd_and_args, native_sysroot, > pseudo=""): if pseudo: > cmd_and_args = pseudo + cmd_and_args > > - native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin" % \ > - (native_sysroot, native_sysroot, native_sysroot) > + native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin" % \ > + (native_sysroot, native_sysroot, > + native_sysroot, native_sysroot) > > native_cmd_and_args = "export PATH=%s:$PATH;%s" % \ > (native_paths, cmd_and_args) ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 09/10] wic: misc: Add /bin to the list of searchpaths 2020-09-05 8:45 ` [PATCH v2 09/10] wic: misc: Add /bin to the list of searchpaths Henning Schild @ 2020-09-05 9:33 ` vijai kumar 0 siblings, 0 replies; 29+ messages in thread From: vijai kumar @ 2020-09-05 9:33 UTC (permalink / raw) To: Henning Schild; +Cc: Vijai Kumar K, isar-users [-- Attachment #1: Type: text/plain, Size: 2364 bytes --] On Sat, 5 Sep 2020 at 2:15 PM, Henning Schild <henning.schild@siemens.com> wrote: > We are patching wic here ... If that is upstream simply take another > > version in p1. > > Wic still is "clean" and purely taken from upstream, which probably > > made your version bump much easier ... it probably was not easy. Dont > > ruin that on the last meters. > > > > Maybe will will reach that clean state for bitbake as well, again. Ok. I will try to update to a version that has P9. > > > > Henning > > > > On Thu, 3 Sep 2020 00:32:46 +0530 > > Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> wrote: > > > > > From: Vijai Kumar K <vijaikumar.kanagarajan@gmail.com> > > > > > > /bin is also a valid path where one can find executables. Add > > > that to the search path. > > > > > > Signed-off-by: Vijai Kumar K <vijaikumar.kanagarajan@gmail.com> > > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > > > [cherry-picked from OE-core ca0a6025351cb2135e87cecf828633cf12aa34c6] > > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > > > --- > > > scripts/lib/wic/misc.py | 5 +++-- > > > 1 file changed, 3 insertions(+), 2 deletions(-) > > > > > > diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py > > > index 91975ba..4b08d64 100644 > > > --- a/scripts/lib/wic/misc.py > > > +++ b/scripts/lib/wic/misc.py > > > @@ -138,8 +138,9 @@ def exec_native_cmd(cmd_and_args, native_sysroot, > > > pseudo=""): if pseudo: > > > cmd_and_args = pseudo + cmd_and_args > > > > > > - native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin" % \ > > > - (native_sysroot, native_sysroot, native_sysroot) > > > + native_paths = "%s/sbin:%s/usr/sbin:%s/usr/bin:%s/bin" % \ > > > + (native_sysroot, native_sysroot, > > > + native_sysroot, native_sysroot) > > > > > > native_cmd_and_args = "export PATH=%s:$PATH;%s" % \ > > > (native_paths, cmd_and_args) > > > > -- > > You received this message because you are subscribed to the Google Groups > "isar-users" group. > > To unsubscribe from this group and stop receiving emails from it, send an > email to isar-users+unsubscribe@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/isar-users/20200905104504.11f7b0a9%40md1za8fc.ad001.siemens.net > . > > [-- Attachment #2: Type: text/html, Size: 3777 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 00/10] WIC update 2020-09-02 18:56 [PATCH v2 00/10] WIC update Vijai Kumar K ` (7 preceding siblings ...) 2020-09-02 18:56 ` [PATCH v2 08/10] debian-common: Add tar as a dependency for wic Vijai Kumar K @ 2020-09-03 5:46 ` vijaikumar....@gmail.com 2020-09-05 9:04 ` Henning Schild 8 siblings, 1 reply; 29+ messages in thread From: vijaikumar....@gmail.com @ 2020-09-03 5:46 UTC (permalink / raw) To: isar-users [-- Attachment #1.1: Type: text/plain, Size: 3684 bytes --] If there are no review comments, can this get merged to next? Thanks, Vijai Kumar K On Thursday, September 3, 2020 at 12:26:57 AM UTC+5:30 vijaikumar_...@mentor.com wrote: > Changes since RFC(v1): > > - P9 commit is upstreamed and replaced with the version from OE-core. > - P10: Lot of downstream projects tend to use /boot mountpoints for > bootloader/EFI > partitions. Added RECIPE-API-CHANGELOG entry to warn user about potential > issues > that might arise when doing so with latest wic. > > Henning Schild (1): > lib/oe/path: try hardlinking instead of guessing when it might fail > > Paul Barker (1): > oe.path: Add copyhardlink() helper function > > Vijai Kumar K (8): > wic: Update to the latest wic from openembedded core > wic/plugins: Fix wic plugins to work with the latest wic > wic-img: Satisfy the quirks of latest wic > wic_fakeroot: Handle standalone pseudo invocations > meta-isar/conf: Add provision to debug WIC > debian-common: Add tar as a dependency for wic > wic: misc: Add /bin to the list of searchpaths > meta-isar/canned-wks: Remove /boot mountpoint > > RECIPE-API-CHANGELOG.md | 17 + > meta-isar/conf/local.conf.sample | 3 + > .../lib/wic/canned-wks/common-isar.wks.inc | 2 +- > .../scripts/lib/wic/canned-wks/hikey.wks | 2 +- > .../lib/wic/canned-wks/sdimage-efi.wks | 2 +- > meta/classes/wic-img.bbclass | 15 +- > meta/conf/distro/debian-common.conf | 3 +- > meta/lib/oe/path.py | 27 +- > .../wic/plugins/source/bootimg-efi-isar.py | 2 +- > .../wic/plugins/source/bootimg-pcbios-isar.py | 9 +- > .../lib/wic/plugins/source/rootfs-u-boot.py | 2 +- > scripts/lib/scriptpath.py | 32 ++ > scripts/lib/wic/__init__.py | 14 +- > scripts/lib/wic/canned-wks/common.wks.inc | 2 +- > .../directdisk-bootloader-config.cfg | 8 +- > .../lib/wic/canned-wks/efi-bootdisk.wks.in | 3 + > scripts/lib/wic/canned-wks/mkhybridiso.wks | 2 +- > scripts/lib/wic/canned-wks/qemuriscv.wks | 3 + > .../lib/wic/canned-wks/qemux86-directdisk.wks | 2 +- > .../lib/wic/canned-wks/sdimage-bootpart.wks | 4 +- > .../lib/wic/canned-wks/systemd-bootdisk.wks | 4 +- > scripts/lib/wic/engine.py | 421 +++++++++++++++- > scripts/lib/wic/filemap.py | 170 ++++--- > scripts/lib/wic/help.py | 401 ++++++++++++++-- > scripts/lib/wic/ksparser.py | 115 +++-- > scripts/lib/wic/{utils => }/misc.py | 100 ++-- > scripts/lib/wic/partition.py | 234 ++++----- > scripts/lib/wic/pluginbase.py | 36 +- > scripts/lib/wic/plugins/imager/direct.py | 175 ++++--- > .../wic/plugins/source/bootimg-biosplusefi.py | 213 +++++++++ > scripts/lib/wic/plugins/source/bootimg-efi.py | 111 +++-- > .../wic/plugins/source/bootimg-partition.py | 153 ++++-- > .../lib/wic/plugins/source/bootimg-pcbios.py | 91 ++-- > scripts/lib/wic/plugins/source/fsimage.py | 56 --- > .../wic/plugins/source/isoimage-isohybrid.py | 185 +++---- > scripts/lib/wic/plugins/source/rawcopy.py | 44 +- > scripts/lib/wic/plugins/source/rootfs.py | 159 ++++-- > scripts/lib/wic/utils/__init__.py | 0 > scripts/lib/wic/utils/runner.py | 114 ----- > scripts/wic | 452 +++++++++++++----- > scripts/wic_fakeroot | 5 + > 41 files changed, 2357 insertions(+), 1036 deletions(-) > create mode 100644 scripts/lib/scriptpath.py > create mode 100644 scripts/lib/wic/canned-wks/efi-bootdisk.wks.in > create mode 100644 scripts/lib/wic/canned-wks/qemuriscv.wks > rename scripts/lib/wic/{utils => }/misc.py (70%) > create mode 100644 scripts/lib/wic/plugins/source/bootimg-biosplusefi.py > delete mode 100644 scripts/lib/wic/plugins/source/fsimage.py > delete mode 100644 scripts/lib/wic/utils/__init__.py > delete mode 100644 scripts/lib/wic/utils/runner.py > > -- > 2.17.1 > > [-- Attachment #1.2: Type: text/html, Size: 5034 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 00/10] WIC update 2020-09-03 5:46 ` [PATCH v2 00/10] WIC update vijaikumar....@gmail.com @ 2020-09-05 9:04 ` Henning Schild 2020-09-05 9:25 ` vijai kumar 0 siblings, 1 reply; 29+ messages in thread From: Henning Schild @ 2020-09-05 9:04 UTC (permalink / raw) To: vijaikumar....@gmail.com; +Cc: isar-users This looks good at a first glance, left some comments. I assume the isar tests work with it. We already heard from Jan that he tested it on one of his layers. Did you happen to test it on one of you mentor layers, maybe a board with a "complex" wic setup. Henning On Wed, 2 Sep 2020 22:46:18 -0700 (PDT) "vijaikumar....@gmail.com" <vijaikumar.kanagarajan@gmail.com> wrote: > If there are no review comments, can this get merged to next? > > Thanks, > Vijai Kumar K > > On Thursday, September 3, 2020 at 12:26:57 AM UTC+5:30 > vijaikumar_...@mentor.com wrote: > > > Changes since RFC(v1): > > > > - P9 commit is upstreamed and replaced with the version from > > OE-core. > > - P10: Lot of downstream projects tend to use /boot mountpoints for > > bootloader/EFI > > partitions. Added RECIPE-API-CHANGELOG entry to warn user about > > potential issues > > that might arise when doing so with latest wic. > > > > Henning Schild (1): > > lib/oe/path: try hardlinking instead of guessing when it might fail > > > > Paul Barker (1): > > oe.path: Add copyhardlink() helper function > > > > Vijai Kumar K (8): > > wic: Update to the latest wic from openembedded core > > wic/plugins: Fix wic plugins to work with the latest wic > > wic-img: Satisfy the quirks of latest wic > > wic_fakeroot: Handle standalone pseudo invocations > > meta-isar/conf: Add provision to debug WIC > > debian-common: Add tar as a dependency for wic > > wic: misc: Add /bin to the list of searchpaths > > meta-isar/canned-wks: Remove /boot mountpoint > > > > RECIPE-API-CHANGELOG.md | 17 + > > meta-isar/conf/local.conf.sample | 3 + > > .../lib/wic/canned-wks/common-isar.wks.inc | 2 +- > > .../scripts/lib/wic/canned-wks/hikey.wks | 2 +- > > .../lib/wic/canned-wks/sdimage-efi.wks | 2 +- > > meta/classes/wic-img.bbclass | 15 +- > > meta/conf/distro/debian-common.conf | 3 +- > > meta/lib/oe/path.py | 27 +- > > .../wic/plugins/source/bootimg-efi-isar.py | 2 +- > > .../wic/plugins/source/bootimg-pcbios-isar.py | 9 +- > > .../lib/wic/plugins/source/rootfs-u-boot.py | 2 +- > > scripts/lib/scriptpath.py | 32 ++ > > scripts/lib/wic/__init__.py | 14 +- > > scripts/lib/wic/canned-wks/common.wks.inc | 2 +- > > .../directdisk-bootloader-config.cfg | 8 +- > > .../lib/wic/canned-wks/efi-bootdisk.wks.in | 3 + > > scripts/lib/wic/canned-wks/mkhybridiso.wks | 2 +- > > scripts/lib/wic/canned-wks/qemuriscv.wks | 3 + > > .../lib/wic/canned-wks/qemux86-directdisk.wks | 2 +- > > .../lib/wic/canned-wks/sdimage-bootpart.wks | 4 +- > > .../lib/wic/canned-wks/systemd-bootdisk.wks | 4 +- > > scripts/lib/wic/engine.py | 421 +++++++++++++++- > > scripts/lib/wic/filemap.py | 170 ++++--- > > scripts/lib/wic/help.py | 401 ++++++++++++++-- > > scripts/lib/wic/ksparser.py | 115 +++-- > > scripts/lib/wic/{utils => }/misc.py | 100 ++-- > > scripts/lib/wic/partition.py | 234 ++++----- > > scripts/lib/wic/pluginbase.py | 36 +- > > scripts/lib/wic/plugins/imager/direct.py | 175 ++++--- > > .../wic/plugins/source/bootimg-biosplusefi.py | 213 +++++++++ > > scripts/lib/wic/plugins/source/bootimg-efi.py | 111 +++-- > > .../wic/plugins/source/bootimg-partition.py | 153 ++++-- > > .../lib/wic/plugins/source/bootimg-pcbios.py | 91 ++-- > > scripts/lib/wic/plugins/source/fsimage.py | 56 --- > > .../wic/plugins/source/isoimage-isohybrid.py | 185 +++---- > > scripts/lib/wic/plugins/source/rawcopy.py | 44 +- > > scripts/lib/wic/plugins/source/rootfs.py | 159 ++++-- > > scripts/lib/wic/utils/__init__.py | 0 > > scripts/lib/wic/utils/runner.py | 114 ----- > > scripts/wic | 452 +++++++++++++----- > > scripts/wic_fakeroot | 5 + > > 41 files changed, 2357 insertions(+), 1036 deletions(-) > > create mode 100644 scripts/lib/scriptpath.py > > create mode 100644 scripts/lib/wic/canned-wks/efi-bootdisk.wks.in > > create mode 100644 scripts/lib/wic/canned-wks/qemuriscv.wks > > rename scripts/lib/wic/{utils => }/misc.py (70%) > > create mode 100644 > > scripts/lib/wic/plugins/source/bootimg-biosplusefi.py delete mode > > 100644 scripts/lib/wic/plugins/source/fsimage.py delete mode 100644 > > scripts/lib/wic/utils/__init__.py delete mode 100644 > > scripts/lib/wic/utils/runner.py > > > > -- > > 2.17.1 > > > > > ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v2 00/10] WIC update 2020-09-05 9:04 ` Henning Schild @ 2020-09-05 9:25 ` vijai kumar 0 siblings, 0 replies; 29+ messages in thread From: vijai kumar @ 2020-09-05 9:25 UTC (permalink / raw) To: Henning Schild; +Cc: isar-users [-- Attachment #1: Type: text/plain, Size: 4863 bytes --] On Sat, 5 Sep 2020 at 2:34 PM, Henning Schild <henning.schild@siemens.com> wrote: > This looks good at a first glance, left some comments. > > > > I assume the isar tests work with it. We already heard from Jan that he > > tested it on one of his layers. Did you happen to test it on one of you > > mentor layers, maybe a board with a "complex" wic setup. > We tested with isar siemens and jailhouse images project. I didnt test with our mentor layers though. It is there in pipeline. Maybe in coming weeks I would give it a try. Thanks, Vijai Kumar K > > > Henning > > > > On Wed, 2 Sep 2020 22:46:18 -0700 (PDT) > > "vijaikumar....@gmail.com" <vijaikumar.kanagarajan@gmail.com> wrote: > > > > > If there are no review comments, can this get merged to next? > > > > > > Thanks, > > > Vijai Kumar K > > > > > > On Thursday, September 3, 2020 at 12:26:57 AM UTC+5:30 > > > vijaikumar_...@mentor.com wrote: > > > > > > > Changes since RFC(v1): > > > > > > > > - P9 commit is upstreamed and replaced with the version from > > > > OE-core. > > > > - P10: Lot of downstream projects tend to use /boot mountpoints for > > > > bootloader/EFI > > > > partitions. Added RECIPE-API-CHANGELOG entry to warn user about > > > > potential issues > > > > that might arise when doing so with latest wic. > > > > > > > > Henning Schild (1): > > > > lib/oe/path: try hardlinking instead of guessing when it might fail > > > > > > > > Paul Barker (1): > > > > oe.path: Add copyhardlink() helper function > > > > > > > > Vijai Kumar K (8): > > > > wic: Update to the latest wic from openembedded core > > > > wic/plugins: Fix wic plugins to work with the latest wic > > > > wic-img: Satisfy the quirks of latest wic > > > > wic_fakeroot: Handle standalone pseudo invocations > > > > meta-isar/conf: Add provision to debug WIC > > > > debian-common: Add tar as a dependency for wic > > > > wic: misc: Add /bin to the list of searchpaths > > > > meta-isar/canned-wks: Remove /boot mountpoint > > > > > > > > RECIPE-API-CHANGELOG.md | 17 + > > > > meta-isar/conf/local.conf.sample | 3 + > > > > .../lib/wic/canned-wks/common-isar.wks.inc | 2 +- > > > > .../scripts/lib/wic/canned-wks/hikey.wks | 2 +- > > > > .../lib/wic/canned-wks/sdimage-efi.wks | 2 +- > > > > meta/classes/wic-img.bbclass | 15 +- > > > > meta/conf/distro/debian-common.conf | 3 +- > > > > meta/lib/oe/path.py | 27 +- > > > > .../wic/plugins/source/bootimg-efi-isar.py | 2 +- > > > > .../wic/plugins/source/bootimg-pcbios-isar.py | 9 +- > > > > .../lib/wic/plugins/source/rootfs-u-boot.py | 2 +- > > > > scripts/lib/scriptpath.py | 32 ++ > > > > scripts/lib/wic/__init__.py | 14 +- > > > > scripts/lib/wic/canned-wks/common.wks.inc | 2 +- > > > > .../directdisk-bootloader-config.cfg | 8 +- > > > > .../lib/wic/canned-wks/efi-bootdisk.wks.in | 3 + > > > > scripts/lib/wic/canned-wks/mkhybridiso.wks | 2 +- > > > > scripts/lib/wic/canned-wks/qemuriscv.wks | 3 + > > > > .../lib/wic/canned-wks/qemux86-directdisk.wks | 2 +- > > > > .../lib/wic/canned-wks/sdimage-bootpart.wks | 4 +- > > > > .../lib/wic/canned-wks/systemd-bootdisk.wks | 4 +- > > > > scripts/lib/wic/engine.py | 421 +++++++++++++++- > > > > scripts/lib/wic/filemap.py | 170 ++++--- > > > > scripts/lib/wic/help.py | 401 ++++++++++++++-- > > > > scripts/lib/wic/ksparser.py | 115 +++-- > > > > scripts/lib/wic/{utils => }/misc.py | 100 ++-- > > > > scripts/lib/wic/partition.py | 234 ++++----- > > > > scripts/lib/wic/pluginbase.py | 36 +- > > > > scripts/lib/wic/plugins/imager/direct.py | 175 ++++--- > > > > .../wic/plugins/source/bootimg-biosplusefi.py | 213 +++++++++ > > > > scripts/lib/wic/plugins/source/bootimg-efi.py | 111 +++-- > > > > .../wic/plugins/source/bootimg-partition.py | 153 ++++-- > > > > .../lib/wic/plugins/source/bootimg-pcbios.py | 91 ++-- > > > > scripts/lib/wic/plugins/source/fsimage.py | 56 --- > > > > .../wic/plugins/source/isoimage-isohybrid.py | 185 +++---- > > > > scripts/lib/wic/plugins/source/rawcopy.py | 44 +- > > > > scripts/lib/wic/plugins/source/rootfs.py | 159 ++++-- > > > > scripts/lib/wic/utils/__init__.py | 0 > > > > scripts/lib/wic/utils/runner.py | 114 ----- > > > > scripts/wic | 452 +++++++++++++----- > > > > scripts/wic_fakeroot | 5 + > > > > 41 files changed, 2357 insertions(+), 1036 deletions(-) > > > > create mode 100644 scripts/lib/scriptpath.py > > > > create mode 100644 scripts/lib/wic/canned-wks/efi-bootdisk.wks.in > > > > create mode 100644 scripts/lib/wic/canned-wks/qemuriscv.wks > > > > rename scripts/lib/wic/{utils => }/misc.py (70%) > > > > create mode 100644 > > > > scripts/lib/wic/plugins/source/bootimg-biosplusefi.py delete mode > > > > 100644 scripts/lib/wic/plugins/source/fsimage.py delete mode 100644 > > > > scripts/lib/wic/utils/__init__.py delete mode 100644 > > > > scripts/lib/wic/utils/runner.py > > > > > > > > -- > > > > 2.17.1 > > > > > > > > > > > > > > > [-- Attachment #2: Type: text/html, Size: 6671 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
end of thread, other threads:[~2020-09-14 5:32 UTC | newest] Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-09-02 18:56 [PATCH v2 00/10] WIC update Vijai Kumar K 2020-09-02 18:56 ` [PATCH v2 01/10] wic: Update to the latest wic from openembedded core Vijai Kumar K 2020-09-05 9:00 ` Henning Schild 2020-09-05 9:21 ` vijai kumar 2020-09-02 18:56 ` [PATCH v2 02/10] wic/plugins: Fix wic plugins to work with the latest wic Vijai Kumar K 2020-09-02 18:56 ` [PATCH v2 03/10] wic-img: Satisfy the quirks of " Vijai Kumar K 2020-09-05 8:30 ` Henning Schild 2020-09-05 16:24 ` vijaikumar....@gmail.com 2020-09-09 14:52 ` Henning Schild 2020-09-02 18:56 ` [PATCH v2 04/10] oe.path: Add copyhardlink() helper function Vijai Kumar K 2020-09-02 18:56 ` [PATCH v2 05/10] lib/oe/path: try hardlinking instead of guessing when it might fail Vijai Kumar K 2020-09-02 18:56 ` [PATCH v2 06/10] wic_fakeroot: Handle standalone pseudo invocations Vijai Kumar K 2020-09-05 8:38 ` Henning Schild 2020-09-05 15:19 ` vijaikumar....@gmail.com 2020-09-05 9:19 ` Henning Schild 2020-09-05 15:22 ` vijaikumar....@gmail.com 2020-09-02 18:56 ` [PATCH v2 07/10] meta-isar/conf: Add provision to debug WIC Vijai Kumar K 2020-09-02 18:56 ` [PATCH v2 08/10] debian-common: Add tar as a dependency for wic Vijai Kumar K 2020-09-02 19:02 ` [PATCH v2 09/10] wic: misc: Add /bin to the list of searchpaths Vijai Kumar K 2020-09-02 19:02 ` [PATCH v2 10/10] meta-isar/canned-wks: Remove /boot mountpoint Vijai Kumar K 2020-09-05 8:58 ` Henning Schild 2020-09-05 16:06 ` vijaikumar....@gmail.com 2020-09-09 15:09 ` Henning Schild 2020-09-14 5:32 ` vijaikumar....@gmail.com 2020-09-05 8:45 ` [PATCH v2 09/10] wic: misc: Add /bin to the list of searchpaths Henning Schild 2020-09-05 9:33 ` vijai kumar 2020-09-03 5:46 ` [PATCH v2 00/10] WIC update vijaikumar....@gmail.com 2020-09-05 9:04 ` Henning Schild 2020-09-05 9:25 ` vijai kumar
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox