From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6543937367387930624 X-Received: by 10.28.225.68 with SMTP id y65mr537616wmg.30.1523629149914; Fri, 13 Apr 2018 07:19:09 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.209.78 with SMTP id i75ls467523wmg.9.canary-gmail; Fri, 13 Apr 2018 07:19:09 -0700 (PDT) X-Google-Smtp-Source: AIpwx484wNCDWEBPiEGep227lRkGoJa5yUvR/xkEMJj0L51c8gA+R08QzHiT1k1pKV4Q2g+gvLtQ X-Received: by 10.28.14.65 with SMTP id 62mr503931wmo.31.1523629149300; Fri, 13 Apr 2018 07:19:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1523629149; cv=none; d=google.com; s=arc-20160816; b=sRVbXMDXyMi0CF/xN7WFoPKkIyc4rpHasP3UExi6pmHtj9+pdnkwLyfiD/JRWHvbAi hG0MLGn184bIjKsB0yToUHPAltL0Gbf+AxS7jBpqZ9YN23mOc1gkYAo/CSz2wCIx6C90 iQBZaM4EIf9VYzwb/MmUWtzeduSffKHDVF3sUAVcDSGRu9F/ltvjqLmU1/BuJi63RMus 9jZdIpGzP3yjV8e9kZYJvYE7YGwEyA7VPrc9p5NL1k47s0ZYelECFW5xbvcwYfCoB7KX lwla2o7BnknCcmms8OAJV87OW8VdoPZB4E1laXbrokoQGVw9+FBbsjB9uwSqo0XjMU6W nQGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=xunByo57ne2qdLRe+2OEoTKc++V1glZqPwH37lent3s=; b=DiJnqAADewpI2rdGW/eN+qvQwBzZWoqDbwauh4f6OmDih/Zoy7ec9D3qVO9Cc1itVJ I+4EfQNshM+tITTOB8QmQsZ+I2QSFTFIcQJeJ/bXxZ/WuvqpUCWK2vWTQhrCOIgNdUaX Z4/92U4+vl/5jUFXY67oFhFnA57ShbNUsH7A1Rdgq1oB+sb7yT2GTjzDxZelwEaQdFNV eGLequgDqO0HD/NwmC98gDKIAYQiDC30rZn8vBcWpKBDDU6u8wOHpU5n01dKa2iuEGT/ koL9SJwFxtmA71wzv87i+WjH+aLYjtAvZEITrrTj5XjIUClc5wJjxcBQ6VJCVUE9Z0Uq HDJg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id k9si182337wri.5.2018.04.13.07.19.09 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Apr 2018 07:19:09 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Received: from mail1.siemens.de (mail1.siemens.de [139.23.33.14]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id w3DEJ8ko005517 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 13 Apr 2018 16:19:08 +0200 Received: from md1pvb1c.ad001.siemens.net (md1pvb1c.ad001.siemens.net [139.25.68.40] (may be forged)) by mail1.siemens.de (8.15.2/8.15.2) with ESMTP id w3DEJ8Vw011006; Fri, 13 Apr 2018 16:19:08 +0200 From: Henning Schild To: isar-users@googlegroups.com Cc: Cedric Hombourger , Andreas Reichel , Henning Schild Subject: [PATCH v2 06/17] wic: now truly go for the wic version we claim to have Date: Fri, 13 Apr 2018 16:18:55 +0200 Message-Id: X-Mailer: git-send-email 2.16.1 In-Reply-To: References: In-Reply-To: References: X-TUID: sEbsbBo4IGwu 2b164b18fd639c9 claims to introduce wic hash 131629ca6238ea05 This commit really carries that version of wic. Issue: - the wic version in Isar was modified - that causes: - confusion, maintainability and updateability issues - potential quality issues Impact: This patch and the previous reverts get wic back to a state where all these Issues are solved. We could now just update our wic without having to worry about local patches. In case of a wic-update Isar and layers on top would still have to review their plugins. Signed-off-by: Henning Schild --- scripts/lib/wic/canned-wks/qemux86-directdisk.wks | 2 +- scripts/lib/wic/filemap.py | 6 +- scripts/lib/wic/help.py | 2 - scripts/lib/wic/ksparser.py | 4 +- scripts/lib/wic/partition.py | 114 ++++++++++++++-------- scripts/lib/wic/plugins/imager/direct.py | 17 +++- scripts/lib/wic/plugins/source/bootimg-pcbios.py | 48 ++++----- scripts/lib/wic/plugins/source/fsimage.py | 56 +++++++++++ scripts/lib/wic/utils/misc.py | 16 +-- scripts/lib/wic/utils/runner.py | 74 ++++++++++++-- 10 files changed, 250 insertions(+), 89 deletions(-) create mode 100644 scripts/lib/wic/plugins/source/fsimage.py diff --git a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks index db30bbc..a6518a0 100644 --- a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks +++ b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks @@ -4,5 +4,5 @@ include common.wks.inc -bootloader --timeout=0 --append="vga=0 uvesafb.mode_option=640x480-32 root=/dev/sda2 rw mem=256M ip=192.168.7.2::192.168.7.1:255.255.255.0 oprofile.timer=1 rootfstype=ext4 " +bootloader --timeout=0 --append="vga=0 uvesafb.mode_option=640x480-32 root=/dev/vda2 rw mem=256M ip=192.168.7.2::192.168.7.1:255.255.255.0 oprofile.timer=1 rootfstype=ext4 " diff --git a/scripts/lib/wic/filemap.py b/scripts/lib/wic/filemap.py index 1f1aacc..080668e 100644 --- a/scripts/lib/wic/filemap.py +++ b/scripts/lib/wic/filemap.py @@ -530,11 +530,9 @@ def filemap(image, log=None): except ErrorNotSupp: return FilemapSeek(image, log) -def sparse_copy(src_fname, dst_fname, offset=0, skip=0, api=None): +def sparse_copy(src_fname, dst_fname, offset=0, skip=0): """Efficiently copy sparse file to or into another file.""" - if not api: - api = filemap - fmap = api(src_fname) + fmap = filemap(src_fname) try: dst_file = open(dst_fname, 'r+b') except IOError: diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py index aee2451..148da89 100644 --- a/scripts/lib/wic/help.py +++ b/scripts/lib/wic/help.py @@ -687,8 +687,6 @@ DESCRIPTION apply to partitions created using '--source rootfs' (see --source above). Valid values are: - vfat - msdos ext2 ext3 ext4 diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py index d026caa..a039300 100644 --- a/scripts/lib/wic/ksparser.py +++ b/scripts/lib/wic/ksparser.py @@ -136,9 +136,7 @@ class KickStart(): part.add_argument('--exclude-path', nargs='+') part.add_argument("--extra-space", type=sizetype) part.add_argument('--fsoptions', dest='fsopts') - part.add_argument('--fstype', default='vfat', - choices=('ext2', 'ext3', 'ext4', 'btrfs', - 'squashfs', 'vfat', 'msdos', 'swap')) + part.add_argument('--fstype') part.add_argument('--label') part.add_argument('--no-table', action='store_true') part.add_argument('--ondisk', '--ondrive', dest='disk', default='sda') diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py index 939e667..8e32afc 100644 --- a/scripts/lib/wic/partition.py +++ b/scripts/lib/wic/partition.py @@ -136,24 +136,22 @@ class Partition(): "specify a non-zero --size/--fixed-size for that " "partition." % self.mountpoint) - if self.fstype == "swap": + if self.fstype and self.fstype == "swap": self.prepare_swap_partition(cr_workdir, oe_builddir, native_sysroot) self.source_file = "%s/fs.%s" % (cr_workdir, self.fstype) - else: - if self.fstype == 'squashfs': - raise WicError("It's not possible to create empty squashfs " - "partition '%s'" % (self.mountpoint)) - + elif self.fstype: rootfs = "%s/fs_%s.%s.%s" % (cr_workdir, self.label, self.lineno, self.fstype) if os.path.isfile(rootfs): os.remove(rootfs) - - prefix = "ext" if self.fstype.startswith("ext") else self.fstype - method = getattr(self, "prepare_empty_partition_" + prefix) - method(rootfs, oe_builddir, native_sysroot) - self.source_file = rootfs + for prefix in ("ext", "btrfs", "vfat", "squashfs"): + if self.fstype.startswith(prefix): + method = getattr(self, + "prepare_empty_partition_" + prefix) + method(rootfs, oe_builddir, native_sysroot) + self.source_file = rootfs + break return plugins = PluginMgr.get_plugins('source') @@ -195,6 +193,19 @@ class Partition(): "larger (%d kB) than its allowed size %d kB" % (self.mountpoint, self.size, self.fixed_size)) + def prepare_rootfs_from_fs_image(self, cr_workdir, oe_builddir, + rootfs_dir): + """ + Handle an already-created partition e.g. xxx.ext3 + """ + rootfs = oe_builddir + du_cmd = "du -Lbks %s" % rootfs + out = exec_cmd(du_cmd) + rootfs_size = out.split()[0] + + self.size = int(rootfs_size) + self.source_file = rootfs + def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir, native_sysroot): """ @@ -219,6 +230,10 @@ class Partition(): if os.path.isfile(rootfs): os.remove(rootfs) + if not self.fstype: + raise WicError("File system for partition %s not specified in " + "kickstart, use --fstype option" % self.mountpoint) + # Get rootfs size from bitbake variable if it's not set in .ks file if not self.size: # Bitbake variable ROOTFS_SIZE is calculated in @@ -233,15 +248,19 @@ class Partition(): '--overhead-factor will be applied') self.size = int(round(float(rsize_bb))) - prefix = "ext" if self.fstype.startswith("ext") else self.fstype - method = getattr(self, "prepare_rootfs_" + prefix) - method(rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo) - self.source_file = rootfs + for prefix in ("ext", "btrfs", "vfat", "squashfs"): + if self.fstype.startswith(prefix): + method = getattr(self, "prepare_rootfs_" + prefix) + method(rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo) - # get the rootfs size in the right units for kickstart (kB) - du_cmd = "du -Lbks %s" % rootfs - out = exec_cmd(du_cmd) - self.size = int(out.split()[0]) + self.source_file = rootfs + + # get the rootfs size in the right units for kickstart (kB) + du_cmd = "du -Lbks %s" % rootfs + out = exec_cmd(du_cmd) + self.size = int(out.split()[0]) + + break def prepare_rootfs_ext(self, rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo): @@ -267,7 +286,7 @@ class Partition(): (self.fstype, extra_imagecmd, rootfs, label_str, rootfs_dir) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) - mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs) + mkfs_cmd = "fsck.%s -fy %s" % (self.fstype, rootfs) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) def prepare_rootfs_btrfs(self, rootfs, oe_builddir, rootfs_dir, @@ -294,10 +313,10 @@ class Partition(): (self.fstype, rootfs_size * 1024, rootfs_dir, label_str, rootfs) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) - def prepare_rootfs_msdos(self, rootfs, oe_builddir, rootfs_dir, - native_sysroot, pseudo): + def prepare_rootfs_vfat(self, rootfs, oe_builddir, rootfs_dir, + native_sysroot, pseudo): """ - Prepare content for a msdos/vfat rootfs partition. + Prepare content for a vfat rootfs partition. """ du_cmd = "du -bks %s" % rootfs_dir out = exec_cmd(du_cmd) @@ -309,12 +328,7 @@ class Partition(): if self.label: label_str = "-n %s" % self.label - size_str = "" - if self.fstype == 'msdos': - size_str = "-F 16" # FAT 16 - - dosfs_cmd = "mkdosfs %s -S 512 %s -C %s %d" % (label_str, size_str, - rootfs, rootfs_size) + dosfs_cmd = "mkdosfs %s -S 512 -C %s %d" % (label_str, rootfs, rootfs_size) exec_native_cmd(dosfs_cmd, native_sysroot) mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir) @@ -323,8 +337,6 @@ class Partition(): chmod_cmd = "chmod 644 %s" % rootfs exec_cmd(chmod_cmd) - prepare_rootfs_vfat = prepare_rootfs_msdos - def prepare_rootfs_squashfs(self, rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo): """ @@ -370,8 +382,8 @@ class Partition(): (self.fstype, self.size * 1024, label_str, rootfs) exec_native_cmd(mkfs_cmd, native_sysroot) - def prepare_empty_partition_msdos(self, rootfs, oe_builddir, - native_sysroot): + def prepare_empty_partition_vfat(self, rootfs, oe_builddir, + native_sysroot): """ Prepare an empty vfat partition. """ @@ -381,18 +393,40 @@ class Partition(): if self.label: label_str = "-n %s" % self.label - size_str = "" - if self.fstype == 'msdos': - size_str = "-F 16" # FAT 16 - - dosfs_cmd = "mkdosfs %s -S 512 %s -C %s %d" % (label_str, size_str, - rootfs, blocks) + dosfs_cmd = "mkdosfs %s -S 512 -C %s %d" % (label_str, rootfs, blocks) exec_native_cmd(dosfs_cmd, native_sysroot) chmod_cmd = "chmod 644 %s" % rootfs exec_cmd(chmod_cmd) - prepare_empty_partition_vfat = prepare_empty_partition_msdos + def prepare_empty_partition_squashfs(self, cr_workdir, oe_builddir, + native_sysroot): + """ + Prepare an empty squashfs partition. + """ + logger.warning("Creating of an empty squashfs %s partition was attempted. " + "Proceeding as requested.", self.mountpoint) + + path = "%s/fs_%s.%s" % (cr_workdir, self.label, self.fstype) + if os.path.isfile(path): + os.remove(path) + + # it is not possible to create a squashfs without source data, + # thus prepare an empty temp dir that is used as source + tmpdir = tempfile.mkdtemp() + + squashfs_cmd = "mksquashfs %s %s -noappend" % \ + (tmpdir, path) + exec_native_cmd(squashfs_cmd, native_sysroot) + + os.rmdir(tmpdir) + + # get the rootfs size in the right units for kickstart (kB) + du_cmd = "du -Lbks %s" % path + out = exec_cmd(du_cmd) + fs_size = out.split()[0] + + self.size = int(fs_size) def prepare_swap_partition(self, cr_workdir, oe_builddir, native_sysroot): """ diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py index f2e6127..7d38ab3 100644 --- a/scripts/lib/wic/plugins/imager/direct.py +++ b/scripts/lib/wic/plugins/imager/direct.py @@ -323,7 +323,7 @@ class PartitionedImage(): if self.ptable_format == 'gpt': part.uuid = str(uuid.uuid4()) else: # msdos partition table - part.uuid = '%08x-%02d' % (self.identifier, part.realnum) + part.uuid = '%0x-%02d' % (self.identifier, part.realnum) def prepare(self, imager): """Prepare an image. Call prepare method of all image partitions.""" @@ -487,8 +487,8 @@ class PartitionedImage(): parted_fs_type = "fat32" elif part.fstype == "msdos": parted_fs_type = "fat16" - if not part.system_id: - part.system_id = '0x6' # FAT16 + elif part.fstype == "ontrackdm6aux3": + parted_fs_type = "ontrackdm6aux3" else: # Type for ext2/ext3/ext4/btrfs parted_fs_type = "ext2" @@ -538,6 +538,17 @@ class PartitionedImage(): (self.path, part.num, part.system_id), self.native_sysroot) + # Parted defaults to enabling the lba flag for fat16 partitions, + # which causes compatibility issues with some firmware (and really + # isn't necessary). + if parted_fs_type == "fat16": + if self.ptable_format == 'msdos': + logger.debug("Disable 'lba' flag for partition '%s' on disk '%s'", + part.num, self.path) + exec_native_cmd("parted -s %s set %d lba off" % \ + (self.path, part.num), + self.native_sysroot) + def cleanup(self): # remove partition images for image in set(self.partimages): diff --git a/scripts/lib/wic/plugins/source/bootimg-pcbios.py b/scripts/lib/wic/plugins/source/bootimg-pcbios.py index 5890c12..11db304 100644 --- a/scripts/lib/wic/plugins/source/bootimg-pcbios.py +++ b/scripts/lib/wic/plugins/source/bootimg-pcbios.py @@ -44,16 +44,19 @@ class BootimgPcbiosPlugin(SourcePlugin): name = 'bootimg-pcbios' @classmethod - def _get_bootimg_dir(cls, bootimg_dir, dirname): + def _get_syslinux_dir(cls, bootimg_dir): """ - Check if dirname exists in default bootimg_dir or - in wic-tools STAGING_DIR. + Get path to syslinux from either default bootimg_dir + or wic-tools STAGING_DIR. """ - for result in (bootimg_dir, get_bitbake_var("STAGING_DATADIR", "wic-tools")): - if os.path.exists("%s/%s" % (result, dirname)): - return result + for path in (bootimg_dir, get_bitbake_var("STAGING_DATADIR", "wic-tools")): + if not path: + continue + syslinux_dir = os.path.join(path, 'syslinux') + if os.path.exists(syslinux_dir): + return syslinux_dir - raise WicError("Couldn't find correct bootimg_dir, exiting") + raise WicError("Couldn't find syslinux directory, exiting") @classmethod def do_install_disk(cls, disk, disk_name, creator, workdir, oe_builddir, @@ -62,12 +65,11 @@ class BootimgPcbiosPlugin(SourcePlugin): Called after all partitions have been prepared and assembled into a disk image. In this case, we install the MBR. """ - bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux') - mbrfile = "%s/syslinux/" % bootimg_dir + syslinux_dir = cls._get_syslinux_dir(bootimg_dir) if creator.ptable_format == 'msdos': - mbrfile += "mbr.bin" + mbrfile = os.path.join(syslinux_dir, "mbr.bin") elif creator.ptable_format == 'gpt': - mbrfile += "gptmbr.bin" + mbrfile = os.path.join(syslinux_dir, "gptmbr.bin") else: raise WicError("Unsupported partition table: %s" % creator.ptable_format) @@ -81,8 +83,10 @@ class BootimgPcbiosPlugin(SourcePlugin): logger.debug("Installing MBR on disk %s as %s with size %s bytes", disk_name, full_path, disk.min_size) - dd_cmd = "dd if=%s of=%s conv=notrunc" % (mbrfile, full_path) - exec_cmd(dd_cmd, native_sysroot) + rcode = runner.show(['dd', 'if=%s' % mbrfile, + 'of=%s' % full_path, 'conv=notrunc']) + if rcode != 0: + raise WicError("Unable to set MBR to %s" % full_path) @classmethod def do_configure_partition(cls, part, source_params, creator, cr_workdir, @@ -151,7 +155,7 @@ class BootimgPcbiosPlugin(SourcePlugin): 'prepares' the partition to be incorporated into the image. In this case, prepare content for legacy bios boot partition. """ - bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux') + syslinux_dir = cls._get_syslinux_dir(bootimg_dir) staging_kernel_dir = kernel_dir @@ -159,14 +163,14 @@ class BootimgPcbiosPlugin(SourcePlugin): cmds = ("install -m 0644 %s/bzImage %s/vmlinuz" % (staging_kernel_dir, hdddir), - "install -m 444 %s/syslinux/ldlinux.sys %s/ldlinux.sys" % - (bootimg_dir, hdddir), - "install -m 0644 %s/syslinux/vesamenu.c32 %s/vesamenu.c32" % - (bootimg_dir, hdddir), - "install -m 444 %s/syslinux/libcom32.c32 %s/libcom32.c32" % - (bootimg_dir, hdddir), - "install -m 444 %s/syslinux/libutil.c32 %s/libutil.c32" % - (bootimg_dir, hdddir)) + "install -m 444 %s/ldlinux.sys %s/ldlinux.sys" % + (syslinux_dir, hdddir), + "install -m 0644 %s/vesamenu.c32 %s/vesamenu.c32" % + (syslinux_dir, hdddir), + "install -m 444 %s/libcom32.c32 %s/libcom32.c32" % + (syslinux_dir, hdddir), + "install -m 444 %s/libutil.c32 %s/libutil.c32" % + (syslinux_dir, hdddir)) for install_cmd in cmds: exec_cmd(install_cmd) diff --git a/scripts/lib/wic/plugins/source/fsimage.py b/scripts/lib/wic/plugins/source/fsimage.py new file mode 100644 index 0000000..f781499 --- /dev/null +++ b/scripts/lib/wic/plugins/source/fsimage.py @@ -0,0 +1,56 @@ +# ex:ts=4:sw=4:sts=4:et +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +import logging +import os + +from wic import WicError +from wic.pluginbase import SourcePlugin +from wic.utils.misc import get_bitbake_var + +logger = logging.getLogger('wic') + +class FSImagePlugin(SourcePlugin): + """ + Add an already existing filesystem image to the partition layout. + """ + + name = 'fsimage' + + @classmethod + def do_prepare_partition(cls, part, source_params, cr, cr_workdir, + oe_builddir, bootimg_dir, kernel_dir, + rootfs_dir, native_sysroot): + """ + Called to do the actual content population for a partition i.e. it + 'prepares' the partition to be incorporated into the image. + """ + if not bootimg_dir: + bootimg_dir = get_bitbake_var("DEPLOY_DIR_IMAGE") + if not bootimg_dir: + raise WicError("Couldn't find DEPLOY_DIR_IMAGE, exiting") + + logger.debug('Bootimg dir: %s', bootimg_dir) + + if 'file' not in source_params: + raise WicError("No file specified") + + src = os.path.join(bootimg_dir, source_params['file']) + + + logger.debug('Preparing partition using image %s', src) + part.prepare_rootfs_from_fs_image(cr_workdir, src, "") diff --git a/scripts/lib/wic/utils/misc.py b/scripts/lib/wic/utils/misc.py index 4609984..c941112 100644 --- a/scripts/lib/wic/utils/misc.py +++ b/scripts/lib/wic/utils/misc.py @@ -59,7 +59,7 @@ NATIVE_RECIPES = {"bmaptool": "bmap-tools", "syslinux": "syslinux" } -def _exec_cmd(cmd_and_args, as_shell=False): +def _exec_cmd(cmd_and_args, as_shell=False, catch=3): """ Execute command, catching stderr, stdout @@ -70,9 +70,9 @@ def _exec_cmd(cmd_and_args, as_shell=False): logger.debug(args) if as_shell: - ret, out = runner.runtool(cmd_and_args) + ret, out = runner.runtool(cmd_and_args, catch) else: - ret, out = runner.runtool(args) + ret, out = runner.runtool(args, catch) out = out.strip() if ret != 0: raise WicError("_exec_cmd: %s returned '%s' instead of 0\noutput: %s" % \ @@ -84,14 +84,14 @@ def _exec_cmd(cmd_and_args, as_shell=False): return ret, out -def exec_cmd(cmd_and_args, as_shell=False): +def exec_cmd(cmd_and_args, as_shell=False, catch=3): """ Execute command, return output """ - return _exec_cmd(cmd_and_args, as_shell)[1] + return _exec_cmd(cmd_and_args, as_shell, catch)[1] -def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""): +def exec_native_cmd(cmd_and_args, native_sysroot, catch=3, pseudo=""): """ Execute native command, catching stderr, stdout @@ -118,7 +118,7 @@ def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""): # If the command isn't in the native sysroot say we failed. if spawn.find_executable(args[0], native_paths): - ret, out = _exec_cmd(native_cmd_and_args, True) + ret, out = _exec_cmd(native_cmd_and_args, True, catch) else: ret = 127 out = "can't find native executable %s in %s" % (args[0], native_paths) @@ -131,7 +131,7 @@ def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""): "was not found (see details above).\n\n" % prog recipe = NATIVE_RECIPES.get(prog) if recipe: - msg += "Please make sure wic-tools have %s-native in its DEPENDS, bake it with 'bitbake wic-tools' "\ + msg += "Please bake it with 'bitbake %s-native' "\ "and try again.\n" % recipe else: msg += "Wic failed to find a recipe to build native %s. Please "\ diff --git a/scripts/lib/wic/utils/runner.py b/scripts/lib/wic/utils/runner.py index 4aa00fb..56d7ea3 100644 --- a/scripts/lib/wic/utils/runner.py +++ b/scripts/lib/wic/utils/runner.py @@ -14,17 +14,32 @@ # You should have received a copy of the GNU General Public License along # with this program; if not, write to the Free Software Foundation, Inc., 59 # Temple Place - Suite 330, Boston, MA 02111-1307, USA. + +import logging +import os import subprocess from wic import WicError -def runtool(cmdln_or_args): +logger = logging.getLogger('wic') + +def runtool(cmdln_or_args, catch=1): """ wrapper for most of the subprocess calls input: cmdln_or_args: can be both args and cmdln str (shell=True) + catch: 0, quitely run + 1, only STDOUT + 2, only STDERR + 3, both STDOUT and STDERR return: - rc, output + (rc, output) + if catch==0: the output will always None """ + + if catch not in (0, 1, 2, 3): + # invalid catch selection, will cause exception, that's good + return None + if isinstance(cmdln_or_args, list): cmd = cmdln_or_args[0] shell = False @@ -33,13 +48,26 @@ def runtool(cmdln_or_args): cmd = shlex.split(cmdln_or_args)[0] shell = True - sout = subprocess.PIPE - serr = subprocess.STDOUT + if catch != 3: + dev_null = os.open("/dev/null", os.O_WRONLY) + + if catch == 0: + sout = dev_null + serr = dev_null + elif catch == 1: + sout = subprocess.PIPE + serr = dev_null + elif catch == 2: + sout = dev_null + serr = subprocess.PIPE + elif catch == 3: + sout = subprocess.PIPE + serr = subprocess.STDOUT try: process = subprocess.Popen(cmdln_or_args, stdout=sout, stderr=serr, shell=shell) - sout, serr = process.communicate() + (sout, serr) = process.communicate() # combine stdout and stderr, filter None out and decode out = ''.join([out.decode('utf-8') for out in [sout, serr] if out]) except OSError as err: @@ -48,5 +76,39 @@ def runtool(cmdln_or_args): raise WicError('Cannot run command: %s, lost dependency?' % cmd) else: raise # relay + finally: + if catch != 3: + os.close(dev_null) + + return (process.returncode, out) + +def show(cmdln_or_args): + """Show all messages using logger.debug.""" + + rcode, out = runtool(cmdln_or_args, catch=3) + + if isinstance(cmdln_or_args, list): + cmd = ' '.join(cmdln_or_args) + else: + cmd = cmdln_or_args + + msg = 'running command: "%s"' % cmd + if out: + out = out.strip() + if out: + msg += ', with output::' + msg += '\n +----------------' + for line in out.splitlines(): + msg += '\n | %s' % line + msg += '\n +----------------' + + logger.debug(msg) + + return rcode + +def outs(cmdln_or_args, catch=1): + # get the outputs of tools + return runtool(cmdln_or_args, catch)[1].strip() - return process.returncode, out +def quiet(cmdln_or_args): + return runtool(cmdln_or_args, catch=0)[0] -- 2.16.1