* [PATCH 1/9] classes: image: introduce size measuring function, for before do_*_image
2018-01-31 9:41 [PATCH 0/9] first wic integration Henning Schild
@ 2018-01-31 9:41 ` Henning Schild
2018-01-31 9:41 ` [PATCH 2/9] images: new class wic-img for wic intregration Henning Schild
` (10 subsequent siblings)
11 siblings, 0 replies; 51+ messages in thread
From: Henning Schild @ 2018-01-31 9:41 UTC (permalink / raw)
To: isar-users; +Cc: Henning Schild
Measure the size and export it to the variable ROOTFS_SIZE just before
the imaging step. The concept is inspired by OE.
Issue:
The size of the rootfs is required for all sort of images, at the moment
Isar only supports one "real" image. Later patches will introduce
another image. And this patch prepares for that by moving the size
measurement into our image base class.
Impact:
- Issue is solved
- nothing changes for users
Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
meta/classes/ext4-img.bbclass | 7 ++-----
meta/classes/image.bbclass | 19 +++++++++++++++++++
2 files changed, 21 insertions(+), 5 deletions(-)
diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass
index 83cb137..78036a2 100644
--- a/meta/classes/ext4-img.bbclass
+++ b/meta/classes/ext4-img.bbclass
@@ -1,9 +1,6 @@
# This software is a part of ISAR.
# Copyright (C) 2015-2017 ilbers GmbH
-# Extra space for rootfs in MB
-ROOTFS_EXTRA ?= "64"
-
EXT4_IMAGE_FILE = "${DEPLOY_DIR_IMAGE}/${PN}-${DISTRO}-${MACHINE}.ext4.img"
do_ext4_image[stamp-extra-info] = "${DISTRO}-${MACHINE}"
@@ -12,8 +9,7 @@ do_ext4_image[stamp-extra-info] = "${DISTRO}-${MACHINE}"
do_ext4_image() {
rm -f ${EXT4_IMAGE_FILE}
- ROOTFS_SIZE=`sudo du -sm ${IMAGE_ROOTFS} | awk '{print $1 + ${ROOTFS_EXTRA};}'`
- dd if=/dev/zero of=${EXT4_IMAGE_FILE} bs=1M count=${ROOTFS_SIZE}
+ dd if=/dev/zero of=${EXT4_IMAGE_FILE} bs=1k count=${ROOTFS_SIZE}
sudo mkfs.ext4 -F ${EXT4_IMAGE_FILE}
@@ -31,3 +27,4 @@ do_ext4_image() {
}
addtask ext4_image before do_build after do_copy_boot_files
+do_ext4_image[prefuncs] = 'set_image_size'
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index e2cb01b..47fa553 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -5,6 +5,9 @@ IMAGE_INSTALL ?= ""
IMAGE_TYPE ?= "ext4-img"
IMAGE_ROOTFS = "${WORKDIR}/rootfs"
+# Extra space for rootfs in MB
+ROOTFS_EXTRA ?= "64"
+
def get_image_name(d, name_link):
S = d.getVar("IMAGE_ROOTFS", True)
path_link = os.path.join(S, name_link)
@@ -12,6 +15,22 @@ def get_image_name(d, name_link):
return os.path.basename(os.path.realpath(path_link))
return ""
+def get_rootfs_size(d):
+ import subprocess
+ rootfs_extra = int(d.getVar("ROOTFS_EXTRA", True))
+
+ output = subprocess.check_output(['sudo', 'du', '-s', '--block-size=1k',
+ d.getVar("IMAGE_ROOTFS", True)])
+ base_size = int(output.split()[0])
+
+ return base_size + rootfs_extra * 1024
+
+python set_image_size () {
+ rootfs_size = get_rootfs_size(d)
+ d.setVar('ROOTFS_SIZE', str(rootfs_size))
+ d.setVarFlag('ROOTFS_SIZE', 'export', '1')
+}
+
# These variables are used by wic and start_vm
KERNEL_IMAGE ?= "${@get_image_name(d, 'vmlinuz')}"
INITRD_IMAGE ?= "${@get_image_name(d, 'initrd.img')}"
--
2.13.6
^ permalink raw reply [flat|nested] 51+ messages in thread* [PATCH 2/9] images: new class wic-img for wic intregration
2018-01-31 9:41 [PATCH 0/9] first wic integration Henning Schild
2018-01-31 9:41 ` [PATCH 1/9] classes: image: introduce size measuring function, for before do_*_image Henning Schild
@ 2018-01-31 9:41 ` Henning Schild
2018-02-13 14:44 ` Alexander Smirnov
2018-01-31 9:41 ` [PATCH 3/9] wic: add a bootimg-efi-isar plugin outside the wic tree Henning Schild
` (9 subsequent siblings)
11 siblings, 1 reply; 51+ messages in thread
From: Henning Schild @ 2018-01-31 9:41 UTC (permalink / raw)
To: isar-users; +Cc: Henning Schild
This patch integrates wic into the bitbake workflow, wic will be used
for the imaging step, no need to call it manually.
The target of this patch is to work with an unmodified wic, that is why
it contains a few wic-specifics, mostly taken from OE.
There is one major difference to OE, we assume that wic is run as root.
And we smuggle in a wrapper that patches calls to an fsck that does not
behave like the one on OE.
Issues:
- wic was never integrated
- you always had to build an ext4-img to create a wic image later
- there was never a way to control the size of wic disks/partition,
only directly in the wks
Impact:
The patch solves the Issues without changing the default behaviour of
Isar.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
meta/classes/wic-img.bbclass | 64 ++++++++++++++++++++++++++++++++++++++++++++
scripts/wic_fakeroot | 37 +++++++++++++++++++++++++
2 files changed, 101 insertions(+)
create mode 100644 meta/classes/wic-img.bbclass
create mode 100755 scripts/wic_fakeroot
diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass
new file mode 100644
index 0000000..e8d2678
--- /dev/null
+++ b/meta/classes/wic-img.bbclass
@@ -0,0 +1,64 @@
+# This software is a part of ISAR.
+# Copyright (C) 2018 Siemens AG
+#
+# this class is heavily inspired by OEs ./meta/classes/image_types_wic.bbclass
+#
+
+WKS_FILE ?= "sdimage-efi"
+
+# TODO here we leak buildenv into the image
+# this needs to come from buildchroot
+# and the isar efi plugin has the same problem
+# syslinux in debian has different folder structure, need to for those plugins
+STAGING_DATADIR ?= "/usr/share/"
+STAGING_LIBDIR ?= "/usr/lib/"
+STAGING_DIR ?= "${TMPDIR}"
+IMAGE_BASENAME ?= "multiconfig:${MACHINE}-${DISTRO}:${PN}"
+FAKEROOTCMD ?= "wic_fakeroot"
+RECIPE_SYSROOT_NATIVE ?= "/"
+
+do_wic_image[stamp-extra-info] = "${DISTRO}-${MACHINE}"
+
+WIC_CREATE_EXTRA_ARGS ?= ""
+
+WICVARS ?= "\
+ BBLAYERS IMGDEPLOYDIR DEPLOY_DIR_IMAGE FAKEROOTCMD IMAGE_BASENAME IMAGE_BOOT_FILES \
+ IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD INITRD_LIVE ISODIR RECIPE_SYSROOT_NATIVE \
+ ROOTFS_SIZE STAGING_DATADIR STAGING_DIR STAGING_LIBDIR TARGET_SYS TRANSLATED_TARGET_ARCH"
+
+# Isar specific vars used in our plugins
+WICVARS += "KERNEL_IMAGE INITRD_IMAGE DISTRO_ARCH"
+
+python do_rootfs_wicenv () {
+ wicvars = d.getVar('WICVARS', True)
+ if not wicvars:
+ return
+
+ stdir = d.getVar('STAGING_DIR', True)
+ outdir = os.path.join(stdir, d.getVar('MACHINE', True), 'imgdata')
+ bb.utils.mkdirhier(outdir)
+ basename = d.getVar('IMAGE_BASENAME', True)
+ with open(os.path.join(outdir, basename) + '.env', 'w') as envf:
+ for var in wicvars.split():
+ value = d.getVar(var, True)
+ if value:
+ envf.write('%s="%s"\n' % (var, value.strip()))
+
+ # this part is stolen from OE ./meta/recipes-core/meta/wic-tools.bb
+ with open(os.path.join(outdir, "wic-tools.env"), 'w') as envf:
+ for var in ('RECIPE_SYSROOT_NATIVE', 'STAGING_DATADIR', 'STAGING_LIBDIR'):
+ envf.write('%s="%s"\n' % (var, d.getVar(var, True).strip()))
+
+}
+
+addtask do_rootfs_wicenv after do_copy_boot_files before do_wic_image
+do_rootfs_wicenv[vardeps] += "${WICVARS}"
+do_rootfs_wicenv[prefuncs] = 'set_image_size'
+
+do_wic_image() {
+ export BUILDDIR="${BUILDDIR}"
+
+ sudo -E PATH="$PATH:/builder/isar/bitbake/bin:/builder/isar/scripts" /builder/isar/scripts/wic create ${WKS_FILE} --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -o ${DEPLOY_DIR_IMAGE} -e ${IMAGE_BASENAME} ${WIC_CREATE_EXTRA_ARGS}
+}
+
+addtask wic_image before do_build after do_copy_boot_files
diff --git a/scripts/wic_fakeroot b/scripts/wic_fakeroot
new file mode 100755
index 0000000..9e01c38
--- /dev/null
+++ b/scripts/wic_fakeroot
@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+#
+# wic needs a FAKEROOT cmd to run, the default is pseudo. In Isar we do/can not
+# use pseudo. And we call wic as root to begin with, so this script could be a
+# dummy doing nothing. It is almost a dummy ...
+#
+# If the fsck hack ever becomes obsolete, FAKEROOTCMD ?= "true;" can be used
+#
+# This software is a part of Isar.
+# Copyright (C) 2018 Siemens AG
+#
+import os
+import sys
+import shutil
+import subprocess
+
+args = sys.argv
+args.pop(0)
+cmd = args[0]
+
+# expect to be running as root
+# we could loosen that and execv(sudo, args) but even some early
+# "du"s fail, which do not use the fakeroot-wrapper
+# i.e. in wics partition.py the "du -ks" fails on
+# var/cache/apt/archives/partial
+# rootfs/root ...
+assert 'root' == os.environ["USER"]
+
+# e2fsck <= 1.43.5 returns 1 on non-errors (stretch and before affected)
+# treat 1 as safe ... the filesystem was successfully repaired and is OK
+if cmd.startswith('fsck.'):
+ ret = subprocess.call(args)
+ if ret == 0 or ret == 1:
+ sys.exit(0)
+ sys.exit(ret)
+
+os.execv(shutil.which(cmd), args)
--
2.13.6
^ permalink raw reply [flat|nested] 51+ messages in thread* Re: [PATCH 2/9] images: new class wic-img for wic intregration
2018-01-31 9:41 ` [PATCH 2/9] images: new class wic-img for wic intregration Henning Schild
@ 2018-02-13 14:44 ` Alexander Smirnov
2018-02-13 16:06 ` Henning Schild
0 siblings, 1 reply; 51+ messages in thread
From: Alexander Smirnov @ 2018-02-13 14:44 UTC (permalink / raw)
To: Henning Schild, isar-users
On 01/31/2018 12:41 PM, Henning Schild wrote:
> This patch integrates wic into the bitbake workflow, wic will be used
> for the imaging step, no need to call it manually.
>
> The target of this patch is to work with an unmodified wic, that is why
> it contains a few wic-specifics, mostly taken from OE.
>
> There is one major difference to OE, we assume that wic is run as root.
> And we smuggle in a wrapper that patches calls to an fsck that does not
> behave like the one on OE.
>
> Issues:
> - wic was never integrated
> - you always had to build an ext4-img to create a wic image later
> - there was never a way to control the size of wic disks/partition,
> only directly in the wks
>
> Impact:
> The patch solves the Issues without changing the default behaviour of
> Isar.
>
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
> meta/classes/wic-img.bbclass | 64 ++++++++++++++++++++++++++++++++++++++++++++
> scripts/wic_fakeroot | 37 +++++++++++++++++++++++++
> 2 files changed, 101 insertions(+)
> create mode 100644 meta/classes/wic-img.bbclass
> create mode 100755 scripts/wic_fakeroot
>
> diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass
> new file mode 100644
> index 0000000..e8d2678
> --- /dev/null
> +++ b/meta/classes/wic-img.bbclass
> @@ -0,0 +1,64 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2018 Siemens AG
> +#
> +# this class is heavily inspired by OEs ./meta/classes/image_types_wic.bbclass
> +#
> +
> +WKS_FILE ?= "sdimage-efi"
> +
> +# TODO here we leak buildenv into the image
> +# this needs to come from buildchroot
> +# and the isar efi plugin has the same problem
> +# syslinux in debian has different folder structure, need to for those plugins
> +STAGING_DATADIR ?= "/usr/share/"
> +STAGING_LIBDIR ?= "/usr/lib/"
> +STAGING_DIR ?= "${TMPDIR}"
> +IMAGE_BASENAME ?= "multiconfig:${MACHINE}-${DISTRO}:${PN}"
> +FAKEROOTCMD ?= "wic_fakeroot"
> +RECIPE_SYSROOT_NATIVE ?= "/"
> +
> +do_wic_image[stamp-extra-info] = "${DISTRO}-${MACHINE}"
> +
> +WIC_CREATE_EXTRA_ARGS ?= ""
> +
> +WICVARS ?= "\
> + BBLAYERS IMGDEPLOYDIR DEPLOY_DIR_IMAGE FAKEROOTCMD IMAGE_BASENAME IMAGE_BOOT_FILES \
> + IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD INITRD_LIVE ISODIR RECIPE_SYSROOT_NATIVE \
> + ROOTFS_SIZE STAGING_DATADIR STAGING_DIR STAGING_LIBDIR TARGET_SYS TRANSLATED_TARGET_ARCH"
> +
> +# Isar specific vars used in our plugins
> +WICVARS += "KERNEL_IMAGE INITRD_IMAGE DISTRO_ARCH"
> +
> +python do_rootfs_wicenv () {
> + wicvars = d.getVar('WICVARS', True)
> + if not wicvars:
> + return
> +
> + stdir = d.getVar('STAGING_DIR', True)
> + outdir = os.path.join(stdir, d.getVar('MACHINE', True), 'imgdata')
> + bb.utils.mkdirhier(outdir)
> + basename = d.getVar('IMAGE_BASENAME', True)
> + with open(os.path.join(outdir, basename) + '.env', 'w') as envf:
> + for var in wicvars.split():
> + value = d.getVar(var, True)
> + if value:
> + envf.write('%s="%s"\n' % (var, value.strip()))
> +
> + # this part is stolen from OE ./meta/recipes-core/meta/wic-tools.bb
> + with open(os.path.join(outdir, "wic-tools.env"), 'w') as envf:
> + for var in ('RECIPE_SYSROOT_NATIVE', 'STAGING_DATADIR', 'STAGING_LIBDIR'):
> + envf.write('%s="%s"\n' % (var, d.getVar(var, True).strip()))
> +
> +}
> +
> +addtask do_rootfs_wicenv after do_copy_boot_files before do_wic_image
> +do_rootfs_wicenv[vardeps] += "${WICVARS}"
> +do_rootfs_wicenv[prefuncs] = 'set_image_size'
> +
> +do_wic_image() {
> + export BUILDDIR="${BUILDDIR}"
> +
> + sudo -E PATH="$PATH:/builder/isar/bitbake/bin:/builder/isar/scripts" /builder/isar/scripts/wic create ${WKS_FILE} --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -o ${DEPLOY_DIR_IMAGE} -e ${IMAGE_BASENAME} ${WIC_CREATE_EXTRA_ARGS}
> +}
How this hardcoded path intended to work?
Alex
> +
> +addtask wic_image before do_build after do_copy_boot_files
> diff --git a/scripts/wic_fakeroot b/scripts/wic_fakeroot
> new file mode 100755
> index 0000000..9e01c38
> --- /dev/null
> +++ b/scripts/wic_fakeroot
> @@ -0,0 +1,37 @@
> +#!/usr/bin/env python3
> +#
> +# wic needs a FAKEROOT cmd to run, the default is pseudo. In Isar we do/can not
> +# use pseudo. And we call wic as root to begin with, so this script could be a
> +# dummy doing nothing. It is almost a dummy ...
> +#
> +# If the fsck hack ever becomes obsolete, FAKEROOTCMD ?= "true;" can be used
> +#
> +# This software is a part of Isar.
> +# Copyright (C) 2018 Siemens AG
> +#
> +import os
> +import sys
> +import shutil
> +import subprocess
> +
> +args = sys.argv
> +args.pop(0)
> +cmd = args[0]
> +
> +# expect to be running as root
> +# we could loosen that and execv(sudo, args) but even some early
> +# "du"s fail, which do not use the fakeroot-wrapper
> +# i.e. in wics partition.py the "du -ks" fails on
> +# var/cache/apt/archives/partial
> +# rootfs/root ...
> +assert 'root' == os.environ["USER"]
> +
> +# e2fsck <= 1.43.5 returns 1 on non-errors (stretch and before affected)
> +# treat 1 as safe ... the filesystem was successfully repaired and is OK
> +if cmd.startswith('fsck.'):
> + ret = subprocess.call(args)
> + if ret == 0 or ret == 1:
> + sys.exit(0)
> + sys.exit(ret)
> +
> +os.execv(shutil.which(cmd), args)
>
^ permalink raw reply [flat|nested] 51+ messages in thread* Re: [PATCH 2/9] images: new class wic-img for wic intregration
2018-02-13 14:44 ` Alexander Smirnov
@ 2018-02-13 16:06 ` Henning Schild
0 siblings, 0 replies; 51+ messages in thread
From: Henning Schild @ 2018-02-13 16:06 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users
Am Tue, 13 Feb 2018 17:44:24 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> On 01/31/2018 12:41 PM, Henning Schild wrote:
> > This patch integrates wic into the bitbake workflow, wic will be
> > used for the imaging step, no need to call it manually.
> >
> > The target of this patch is to work with an unmodified wic, that is
> > why it contains a few wic-specifics, mostly taken from OE.
> >
> > There is one major difference to OE, we assume that wic is run as
> > root. And we smuggle in a wrapper that patches calls to an fsck
> > that does not behave like the one on OE.
> >
> > Issues:
> > - wic was never integrated
> > - you always had to build an ext4-img to create a wic image later
> > - there was never a way to control the size of wic
> > disks/partition, only directly in the wks
> >
> > Impact:
> > The patch solves the Issues without changing the default
> > behaviour of Isar.
> >
> > Signed-off-by: Henning Schild <henning.schild@siemens.com>
> > ---
> > meta/classes/wic-img.bbclass | 64
> > ++++++++++++++++++++++++++++++++++++++++++++
> > scripts/wic_fakeroot | 37 +++++++++++++++++++++++++ 2 files
> > changed, 101 insertions(+) create mode 100644
> > meta/classes/wic-img.bbclass create mode 100755 scripts/wic_fakeroot
> >
> > diff --git a/meta/classes/wic-img.bbclass
> > b/meta/classes/wic-img.bbclass new file mode 100644
> > index 0000000..e8d2678
> > --- /dev/null
> > +++ b/meta/classes/wic-img.bbclass
> > @@ -0,0 +1,64 @@
> > +# This software is a part of ISAR.
> > +# Copyright (C) 2018 Siemens AG
> > +#
> > +# this class is heavily inspired by
> > OEs ./meta/classes/image_types_wic.bbclass +#
> > +
> > +WKS_FILE ?= "sdimage-efi"
> > +
> > +# TODO here we leak buildenv into the image
> > +# this needs to come from buildchroot
> > +# and the isar efi plugin has the same problem
> > +# syslinux in debian has different folder structure, need to for
> > those plugins +STAGING_DATADIR ?= "/usr/share/"
> > +STAGING_LIBDIR ?= "/usr/lib/"
> > +STAGING_DIR ?= "${TMPDIR}"
> > +IMAGE_BASENAME ?= "multiconfig:${MACHINE}-${DISTRO}:${PN}"
> > +FAKEROOTCMD ?= "wic_fakeroot"
> > +RECIPE_SYSROOT_NATIVE ?= "/"
> > +
> > +do_wic_image[stamp-extra-info] = "${DISTRO}-${MACHINE}"
> > +
> > +WIC_CREATE_EXTRA_ARGS ?= ""
> > +
> > +WICVARS ?= "\
> > + BBLAYERS IMGDEPLOYDIR DEPLOY_DIR_IMAGE FAKEROOTCMD
> > IMAGE_BASENAME IMAGE_BOOT_FILES \
> > + IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD
> > INITRD_LIVE ISODIR RECIPE_SYSROOT_NATIVE \
> > + ROOTFS_SIZE STAGING_DATADIR STAGING_DIR STAGING_LIBDIR
> > TARGET_SYS TRANSLATED_TARGET_ARCH" +
> > +# Isar specific vars used in our plugins
> > +WICVARS += "KERNEL_IMAGE INITRD_IMAGE DISTRO_ARCH"
> > +
> > +python do_rootfs_wicenv () {
> > + wicvars = d.getVar('WICVARS', True)
> > + if not wicvars:
> > + return
> > +
> > + stdir = d.getVar('STAGING_DIR', True)
> > + outdir = os.path.join(stdir, d.getVar('MACHINE', True),
> > 'imgdata')
> > + bb.utils.mkdirhier(outdir)
> > + basename = d.getVar('IMAGE_BASENAME', True)
> > + with open(os.path.join(outdir, basename) + '.env', 'w') as
> > envf:
> > + for var in wicvars.split():
> > + value = d.getVar(var, True)
> > + if value:
> > + envf.write('%s="%s"\n' % (var, value.strip()))
> > +
> > + # this part is stolen from
> > OE ./meta/recipes-core/meta/wic-tools.bb
> > + with open(os.path.join(outdir, "wic-tools.env"), 'w') as envf:
> > + for var in ('RECIPE_SYSROOT_NATIVE', 'STAGING_DATADIR',
> > 'STAGING_LIBDIR'):
> > + envf.write('%s="%s"\n' % (var, d.getVar(var,
> > True).strip())) +
> > +}
> > +
> > +addtask do_rootfs_wicenv after do_copy_boot_files before
> > do_wic_image +do_rootfs_wicenv[vardeps] += "${WICVARS}"
> > +do_rootfs_wicenv[prefuncs] = 'set_image_size'
> > +
> > +do_wic_image() {
> > + export BUILDDIR="${BUILDDIR}"
> > +
> > + sudo -E
> > PATH="$PATH:/builder/isar/bitbake/bin:/builder/isar/scripts" /builder/isar/scripts/wic
> > create ${WKS_FILE} --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -o
> > ${DEPLOY_DIR_IMAGE} -e ${IMAGE_BASENAME} ${WIC_CREATE_EXTRA_ARGS}
> > +}
>
> How this hardcoded path intended to work?
You are right, that is the result of us building in docker where you
can assume the path to be fixed. Will make it variable.
Henning
> Alex
>
> > +
> > +addtask wic_image before do_build after do_copy_boot_files
> > diff --git a/scripts/wic_fakeroot b/scripts/wic_fakeroot
> > new file mode 100755
> > index 0000000..9e01c38
> > --- /dev/null
> > +++ b/scripts/wic_fakeroot
> > @@ -0,0 +1,37 @@
> > +#!/usr/bin/env python3
> > +#
> > +# wic needs a FAKEROOT cmd to run, the default is pseudo. In Isar
> > we do/can not +# use pseudo. And we call wic as root to begin with,
> > so this script could be a +# dummy doing nothing. It is almost a
> > dummy ... +#
> > +# If the fsck hack ever becomes obsolete, FAKEROOTCMD ?= "true;"
> > can be used +#
> > +# This software is a part of Isar.
> > +# Copyright (C) 2018 Siemens AG
> > +#
> > +import os
> > +import sys
> > +import shutil
> > +import subprocess
> > +
> > +args = sys.argv
> > +args.pop(0)
> > +cmd = args[0]
> > +
> > +# expect to be running as root
> > +# we could loosen that and execv(sudo, args) but even some early
> > +# "du"s fail, which do not use the fakeroot-wrapper
> > +# i.e. in wics partition.py the "du -ks" fails on
> > +# var/cache/apt/archives/partial
> > +# rootfs/root ...
> > +assert 'root' == os.environ["USER"]
> > +
> > +# e2fsck <= 1.43.5 returns 1 on non-errors (stretch and before
> > affected) +# treat 1 as safe ... the filesystem was successfully
> > repaired and is OK +if cmd.startswith('fsck.'):
> > + ret = subprocess.call(args)
> > + if ret == 0 or ret == 1:
> > + sys.exit(0)
> > + sys.exit(ret)
> > +
> > +os.execv(shutil.which(cmd), args)
> >
^ permalink raw reply [flat|nested] 51+ messages in thread
* [PATCH 3/9] wic: add a bootimg-efi-isar plugin outside the wic tree
2018-01-31 9:41 [PATCH 0/9] first wic integration Henning Schild
2018-01-31 9:41 ` [PATCH 1/9] classes: image: introduce size measuring function, for before do_*_image Henning Schild
2018-01-31 9:41 ` [PATCH 2/9] images: new class wic-img for wic intregration Henning Schild
@ 2018-01-31 9:41 ` Henning Schild
2018-02-12 17:48 ` Jan Kiszka
2018-01-31 9:41 ` [PATCH 4/9] Revert "wic: Make the bootimg-efi plugin generate usable images" Henning Schild
` (8 subsequent siblings)
11 siblings, 1 reply; 51+ messages in thread
From: Henning Schild @ 2018-01-31 9:41 UTC (permalink / raw)
To: isar-users; +Cc: Henning Schild
The added plugin is a copy of the modified version in our wic. Move it
out and rename it so we can get back to a clean wic.
Issues:
- wic in Isar has been messed with
Impact:
This patch prepares for the Issue beeing solved, because now the plugin
is outside the wic-tree. Nothing changes for users.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
.../scripts/lib/wic/canned-wks/sdimage-efi.wks | 2 +-
.../lib/wic/plugins/source/bootimg-efi-isar.py | 307 +++++++++++++++++++++
2 files changed, 308 insertions(+), 1 deletion(-)
create mode 100644 meta-isar/scripts/lib/wic/plugins/source/bootimg-efi-isar.py
diff --git a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
index 1171a26..580ad21 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
@@ -2,7 +2,7 @@
# long-description: Creates a partitioned EFI disk image without any swap that
# the user can directly dd to boot media.
-part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
+part /boot --source bootimg-efi-isar --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
diff --git a/meta-isar/scripts/lib/wic/plugins/source/bootimg-efi-isar.py b/meta-isar/scripts/lib/wic/plugins/source/bootimg-efi-isar.py
new file mode 100644
index 0000000..68ff67b
--- /dev/null
+++ b/meta-isar/scripts/lib/wic/plugins/source/bootimg-efi-isar.py
@@ -0,0 +1,307 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Copyright (c) 2014, Intel Corporation.
+# All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# DESCRIPTION
+# This implements the 'bootimg-efi-isar' source plugin class for 'wic'
+#
+# AUTHORS
+# Tom Zanussi <tom.zanussi (at] linux.intel.com>
+#
+
+import logging
+import os
+import shutil
+
+from wic import WicError
+from wic.engine import get_custom_config
+from wic.pluginbase import SourcePlugin
+from wic.utils.misc import (exec_cmd, get_bitbake_var, BOOTDD_EXTRA_SPACE)
+
+logger = logging.getLogger('wic')
+
+class BootimgEFIPlugin(SourcePlugin):
+ """
+ Create EFI boot partition.
+ This plugin supports GRUB 2 and systemd-boot bootloaders.
+ """
+
+ name = 'bootimg-efi-isar'
+
+ @classmethod
+ def do_configure_grubefi(cls, creator, cr_workdir):
+ """
+ Create loader-specific (grub-efi) config
+ """
+ configfile = creator.ks.bootloader.configfile
+ custom_cfg = None
+ if configfile:
+ custom_cfg = get_custom_config(configfile)
+ if custom_cfg:
+ # Use a custom configuration for grub
+ grubefi_conf = custom_cfg
+ logger.debug("Using custom configuration file "
+ "%s for grub.cfg", configfile)
+ else:
+ raise WicError("configfile is specified but failed to "
+ "get it from %s." % configfile)
+
+ if not custom_cfg:
+ # Create grub configuration using parameters from wks file
+ bootloader = creator.ks.bootloader
+
+ grubefi_conf = ""
+ grubefi_conf += "serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1\n"
+ grubefi_conf += "default=boot\n"
+ grubefi_conf += "timeout=%s\n" % bootloader.timeout
+ grubefi_conf += "set root='hd0,gpt2'\n"
+ grubefi_conf += "menuentry 'boot'{\n"
+
+ kernel_image = get_bitbake_var("KERNEL_IMAGE")
+ kernel = "/boot/%s" % kernel_image
+
+ grubefi_conf += "linux %s root=/dev/sda2 rootwait %s\n" \
+ % (kernel, bootloader.append)
+
+ initrd_image = get_bitbake_var("INITRD_IMAGE")
+ initrd = "/boot/%s" % initrd_image
+
+ grubefi_conf += "initrd %s\n" % initrd
+
+ grubefi_conf += "}\n"
+
+ logger.debug("Writing grubefi config %s/hdd/boot/EFI/BOOT/grub.cfg",
+ cr_workdir)
+ cfg = open("%s/hdd/boot/EFI/BOOT/grub.cfg" % cr_workdir, "w")
+ cfg.write(grubefi_conf)
+ cfg.close()
+
+ cfg = open("%s/hdd/boot/EFI/BOOT/grub-mkimage.cfg" % cr_workdir, "w")
+ mkimage_conf = "set root='hd0,gpt1'\n"
+ mkimage_conf += "set prefix=($root)/EFI/BOOT\n"
+ cfg.write(mkimage_conf)
+ cfg.close()
+
+ @classmethod
+ def do_configure_systemdboot(cls, hdddir, creator, cr_workdir, source_params):
+ """
+ Create loader-specific systemd-boot/gummiboot config
+ """
+ install_cmd = "install -d %s/loader" % hdddir
+ exec_cmd(install_cmd)
+
+ install_cmd = "install -d %s/loader/entries" % hdddir
+ exec_cmd(install_cmd)
+
+ bootloader = creator.ks.bootloader
+
+ loader_conf = ""
+ loader_conf += "default boot\n"
+ loader_conf += "timeout %d\n" % bootloader.timeout
+
+ initrd = source_params.get('initrd')
+
+ if initrd:
+ # obviously we need to have a common common deploy var
+ bootimg_dir = get_bitbake_var("DEPLOY_DIR_IMAGE")
+ if not bootimg_dir:
+ raise WicError("Couldn't find DEPLOY_DIR_IMAGE, exiting")
+
+ cp_cmd = "cp %s/%s %s" % (bootimg_dir, initrd, hdddir)
+ exec_cmd(cp_cmd, True)
+ else:
+ logger.debug("Ignoring missing initrd")
+
+ logger.debug("Writing systemd-boot config "
+ "%s/hdd/boot/loader/loader.conf", cr_workdir)
+ cfg = open("%s/hdd/boot/loader/loader.conf" % cr_workdir, "w")
+ cfg.write(loader_conf)
+ cfg.close()
+
+ configfile = creator.ks.bootloader.configfile
+ custom_cfg = None
+ if configfile:
+ custom_cfg = get_custom_config(configfile)
+ if custom_cfg:
+ # Use a custom configuration for systemd-boot
+ boot_conf = custom_cfg
+ logger.debug("Using custom configuration file "
+ "%s for systemd-boots's boot.conf", configfile)
+ else:
+ raise WicError("configfile is specified but failed to "
+ "get it from %s.", configfile)
+
+ if not custom_cfg:
+ # Create systemd-boot configuration using parameters from wks file
+ kernel_name = get_bitbake_var("KERNEL_IMAGE")
+ kernel = "/%s" % kernel_name
+
+ boot_conf = ""
+ boot_conf += "title boot\n"
+ boot_conf += "linux %s\n" % kernel
+ boot_conf += "options LABEL=Boot root=%s %s\n" % \
+ (creator.rootdev, bootloader.append)
+
+ if initrd:
+ boot_conf += "initrd /%s\n" % initrd
+
+ logger.debug("Writing systemd-boot config "
+ "%s/hdd/boot/loader/entries/boot.conf", cr_workdir)
+ cfg = open("%s/hdd/boot/loader/entries/boot.conf" % cr_workdir, "w")
+ cfg.write(boot_conf)
+ cfg.close()
+
+
+ @classmethod
+ def do_configure_partition(cls, part, source_params, creator, cr_workdir,
+ oe_builddir, bootimg_dir, kernel_dir,
+ native_sysroot):
+ """
+ Called before do_prepare_partition(), creates loader-specific config
+ """
+ hdddir = "%s/hdd/boot" % cr_workdir
+
+ install_cmd = "install -d %s/EFI/BOOT" % hdddir
+ exec_cmd(install_cmd)
+
+ try:
+ if source_params['loader'] == 'grub-efi':
+ cls.do_configure_grubefi(creator, cr_workdir)
+ elif source_params['loader'] == 'systemd-boot':
+ cls.do_configure_systemdboot(hdddir, creator, cr_workdir, source_params)
+ else:
+ raise WicError("unrecognized bootimg-efi-isar loader: %s" % source_params['loader'])
+ except KeyError:
+ raise WicError("bootimg-efi-isar requires a loader, none specified")
+
+
+ @classmethod
+ def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
+ oe_builddir, bootimg_dir, kernel_dir,
+ rootfs_dir, native_sysroot):
+ """
+ Called to do the actual content population for a partition i.e. it
+ 'prepares' the partition to be incorporated into the image.
+ In this case, prepare content for an EFI (grub) boot partition.
+ """
+ if not kernel_dir:
+ kernel_dir = get_bitbake_var("DEPLOY_DIR_IMAGE")
+ if not kernel_dir:
+ raise WicError("Couldn't find DEPLOY_DIR_IMAGE, exiting")
+
+ staging_kernel_dir = kernel_dir
+
+ hdddir = "%s/hdd/boot" % cr_workdir
+
+ kernel_name = get_bitbake_var("KERNEL_IMAGE")
+ install_cmd = "install -m 0644 %s/%s %s/%s" % \
+ (staging_kernel_dir, kernel_name, hdddir, kernel_name)
+ exec_cmd(install_cmd)
+
+
+ try:
+ if source_params['loader'] == 'grub-efi':
+ shutil.copyfile("%s/hdd/boot/EFI/BOOT/grub.cfg" % cr_workdir,
+ "%s/grub.cfg" % cr_workdir)
+ shutil.copyfile("%s/hdd/boot/EFI/BOOT/grub-mkimage.cfg" % cr_workdir,
+ "%s/grub-mkimage.cfg" % cr_workdir)
+ for mod in [x for x in os.listdir(kernel_dir) if x.startswith("grub-efi-")]:
+ cp_cmd = "cp %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, hdddir, mod[9:])
+ exec_cmd(cp_cmd, True)
+ shutil.move("%s/grub.cfg" % cr_workdir,
+ "%s/hdd/boot/EFI/BOOT/grub.cfg" % cr_workdir)
+
+ distro_arch = get_bitbake_var("DISTRO_ARCH")
+ if not distro_arch:
+ raise WicError("Couldn't find target architecture")
+
+ if distro_arch == "amd64":
+ grub_target = 'x86_64-efi'
+ grub_image = "bootx64.efi"
+ elif distro_arch == "i386":
+ grub_target = 'i386-efi'
+ grub_image = "bootia32.efi"
+ else:
+ raise WicError("grub-efi is incompatible with target %s" %
+ distro_arch)
+
+ bootimg_dir = "%s/hdd/boot" % cr_workdir
+ if not os.path.isfile("%s/EFI/BOOT/%s" \
+ % (bootimg_dir, grub_image)):
+
+ # TODO: check that grub-mkimage is available
+ grub_cmd = "grub-mkimage -p /EFI/BOOT "
+ grub_cmd += "-c %s/grub-mkimage.cfg " % cr_workdir
+ grub_cmd += "-O %s -o %s/EFI/BOOT/%s " \
+ % (grub_target, bootimg_dir, grub_image)
+ grub_cmd += "part_gpt part_msdos ntfs ntfscomp fat ext2 "
+ grub_cmd += "normal chain boot configfile linux multiboot "
+ grub_cmd += "search efi_gop efi_uga font gfxterm gfxmenu "
+ grub_cmd += "terminal minicmd test iorw loadenv echo help "
+ grub_cmd += "reboot serial terminfo iso9660 loopback tar "
+ grub_cmd += "memdisk ls search_fs_uuid udf btrfs xfs lvm "
+ grub_cmd += "reiserfs ata "
+ exec_cmd(grub_cmd)
+ elif source_params['loader'] == 'systemd-boot':
+ for mod in [x for x in os.listdir(kernel_dir) if x.startswith("systemd-")]:
+ cp_cmd = "cp %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, hdddir, mod[8:])
+ exec_cmd(cp_cmd, True)
+ else:
+ raise WicError("unrecognized bootimg-efi-isar loader: %s" %
+ source_params['loader'])
+ except KeyError:
+ raise WicError("bootimg-efi-isar requires a loader, none specified")
+
+ startup = os.path.join(kernel_dir, "startup.nsh")
+ if os.path.exists(startup):
+ cp_cmd = "cp %s %s/" % (startup, hdddir)
+ exec_cmd(cp_cmd, True)
+
+ du_cmd = "du -bks %s" % hdddir
+ out = exec_cmd(du_cmd)
+ blocks = int(out.split()[0])
+
+ extra_blocks = part.get_extra_block_count(blocks)
+
+ if extra_blocks < BOOTDD_EXTRA_SPACE:
+ extra_blocks = BOOTDD_EXTRA_SPACE
+
+ blocks += extra_blocks
+
+ logger.debug("Added %d extra blocks to %s to get to %d total blocks",
+ extra_blocks, part.mountpoint, blocks)
+
+ # dosfs image, created by mkdosfs
+ bootimg = "%s/boot.img" % cr_workdir
+
+ dosfs_cmd = "mkdosfs -n efi -C %s %d" % (bootimg, blocks)
+ exec_cmd(dosfs_cmd)
+
+ mcopy_cmd = "env MTOOLS_SKIP_CHECK=1 mcopy -i %s -s %s/* ::/" % \
+ (bootimg, hdddir)
+ exec_cmd(mcopy_cmd, True)
+
+ chmod_cmd = "chmod 644 %s" % bootimg
+ exec_cmd(chmod_cmd)
+
+ du_cmd = "du -Lbks %s" % bootimg
+ out = exec_cmd(du_cmd)
+ bootimg_size = out.split()[0]
+
+ part.size = int(bootimg_size)
+ part.source_file = bootimg
--
2.13.6
^ permalink raw reply [flat|nested] 51+ messages in thread* Re: [PATCH 3/9] wic: add a bootimg-efi-isar plugin outside the wic tree
2018-01-31 9:41 ` [PATCH 3/9] wic: add a bootimg-efi-isar plugin outside the wic tree Henning Schild
@ 2018-02-12 17:48 ` Jan Kiszka
0 siblings, 0 replies; 51+ messages in thread
From: Jan Kiszka @ 2018-02-12 17:48 UTC (permalink / raw)
To: [ext] Henning Schild, isar-users
On 2018-01-31 10:41, [ext] Henning Schild wrote:
> The added plugin is a copy of the modified version in our wic. Move it
> out and rename it so we can get back to a clean wic.
>
> Issues:
> - wic in Isar has been messed with
>
> Impact:
> This patch prepares for the Issue beeing solved, because now the plugin
> is outside the wic-tree. Nothing changes for users.
>
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
> .../scripts/lib/wic/canned-wks/sdimage-efi.wks | 2 +-
> .../lib/wic/plugins/source/bootimg-efi-isar.py | 307 +++++++++++++++++++++
> 2 files changed, 308 insertions(+), 1 deletion(-)
> create mode 100644 meta-isar/scripts/lib/wic/plugins/source/bootimg-efi-isar.py
>
> diff --git a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
> index 1171a26..580ad21 100644
> --- a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
> +++ b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
> @@ -2,7 +2,7 @@
> # long-description: Creates a partitioned EFI disk image without any swap that
> # the user can directly dd to boot media.
>
> -part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
> +part /boot --source bootimg-efi-isar --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
>
> part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
>
A downside of this renaming is that we need to adjust wks files that
might be shared with OE setups.
An alternative way of overloading the original plugin does not exist,
does it?
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 51+ messages in thread
* [PATCH 4/9] Revert "wic: Make the bootimg-efi plugin generate usable images"
2018-01-31 9:41 [PATCH 0/9] first wic integration Henning Schild
` (2 preceding siblings ...)
2018-01-31 9:41 ` [PATCH 3/9] wic: add a bootimg-efi-isar plugin outside the wic tree Henning Schild
@ 2018-01-31 9:41 ` Henning Schild
2018-01-31 9:41 ` [PATCH 5/9] Revert "wic: Introduce the `WicExecError` exception class" Henning Schild
` (7 subsequent siblings)
11 siblings, 0 replies; 51+ messages in thread
From: Henning Schild @ 2018-01-31 9:41 UTC (permalink / raw)
To: isar-users; +Cc: Henning Schild
This reverts commit c8f10e3f541643f6ed6bb3c971bf36267519fba8.
---
scripts/lib/wic/plugins/source/bootimg-efi.py | 62 +++------------------------
1 file changed, 6 insertions(+), 56 deletions(-)
diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
index 45635f3..f631e3f 100644
--- a/scripts/lib/wic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
@@ -69,20 +69,12 @@ class BootimgEFIPlugin(SourcePlugin):
grubefi_conf += "serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1\n"
grubefi_conf += "default=boot\n"
grubefi_conf += "timeout=%s\n" % bootloader.timeout
- grubefi_conf += "set root='hd0,gpt2'\n"
grubefi_conf += "menuentry 'boot'{\n"
- kernel_image = get_bitbake_var("KERNEL_IMAGE")
- kernel = "/boot/%s" % kernel_image
-
- grubefi_conf += "linux %s root=/dev/sda2 rootwait %s\n" \
- % (kernel, bootloader.append)
-
- initrd_image = get_bitbake_var("INITRD_IMAGE")
- initrd = "/boot/%s" % initrd_image
-
- grubefi_conf += "initrd %s\n" % initrd
+ kernel = "/bzImage"
+ grubefi_conf += "linux %s root=%s rootwait %s\n" \
+ % (kernel, creator.rootdev, bootloader.append)
grubefi_conf += "}\n"
logger.debug("Writing grubefi config %s/hdd/boot/EFI/BOOT/grub.cfg",
@@ -91,12 +83,6 @@ class BootimgEFIPlugin(SourcePlugin):
cfg.write(grubefi_conf)
cfg.close()
- cfg = open("%s/hdd/boot/EFI/BOOT/grub-mkimage.cfg" % cr_workdir, "w")
- mkimage_conf = "set root='hd0,gpt1'\n"
- mkimage_conf += "set prefix=($root)/EFI/BOOT\n"
- cfg.write(mkimage_conf)
- cfg.close()
-
@classmethod
def do_configure_systemdboot(cls, hdddir, creator, cr_workdir, source_params):
"""
@@ -148,8 +134,7 @@ class BootimgEFIPlugin(SourcePlugin):
if not custom_cfg:
# Create systemd-boot configuration using parameters from wks file
- kernel_name = get_bitbake_var("KERNEL_IMAGE")
- kernel = "/%s" % kernel_name
+ kernel = "/bzImage"
boot_conf = ""
boot_conf += "title boot\n"
@@ -206,9 +191,8 @@ class BootimgEFIPlugin(SourcePlugin):
hdddir = "%s/hdd/boot" % cr_workdir
- kernel_name = get_bitbake_var("KERNEL_IMAGE")
- install_cmd = "install -m 0644 %s/%s %s/%s" % \
- (staging_kernel_dir, kernel_name, hdddir, kernel_name)
+ install_cmd = "install -m 0644 %s/bzImage %s/bzImage" % \
+ (staging_kernel_dir, hdddir)
exec_cmd(install_cmd)
@@ -216,45 +200,11 @@ class BootimgEFIPlugin(SourcePlugin):
if source_params['loader'] == 'grub-efi':
shutil.copyfile("%s/hdd/boot/EFI/BOOT/grub.cfg" % cr_workdir,
"%s/grub.cfg" % cr_workdir)
- shutil.copyfile("%s/hdd/boot/EFI/BOOT/grub-mkimage.cfg" % cr_workdir,
- "%s/grub-mkimage.cfg" % cr_workdir)
for mod in [x for x in os.listdir(kernel_dir) if x.startswith("grub-efi-")]:
cp_cmd = "cp %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, hdddir, mod[9:])
exec_cmd(cp_cmd, True)
shutil.move("%s/grub.cfg" % cr_workdir,
"%s/hdd/boot/EFI/BOOT/grub.cfg" % cr_workdir)
-
- distro_arch = get_bitbake_var("DISTRO_ARCH")
- if not distro_arch:
- raise WicError("Couldn't find target architecture")
-
- if distro_arch == "amd64":
- grub_target = 'x86_64-efi'
- grub_image = "bootx64.efi"
- elif distro_arch == "i386":
- grub_target = 'i386-efi'
- grub_image = "bootia32.efi"
- else:
- raise WicError("grub-efi is incompatible with target %s" %
- distro_arch)
-
- bootimg_dir = "%s/hdd/boot" % cr_workdir
- if not os.path.isfile("%s/EFI/BOOT/%s" \
- % (bootimg_dir, grub_image)):
-
- # TODO: check that grub-mkimage is available
- grub_cmd = "grub-mkimage -p /EFI/BOOT "
- grub_cmd += "-c %s/grub-mkimage.cfg " % cr_workdir
- grub_cmd += "-O %s -o %s/EFI/BOOT/%s " \
- % (grub_target, bootimg_dir, grub_image)
- grub_cmd += "part_gpt part_msdos ntfs ntfscomp fat ext2 "
- grub_cmd += "normal chain boot configfile linux multiboot "
- grub_cmd += "search efi_gop efi_uga font gfxterm gfxmenu "
- grub_cmd += "terminal minicmd test iorw loadenv echo help "
- grub_cmd += "reboot serial terminfo iso9660 loopback tar "
- grub_cmd += "memdisk ls search_fs_uuid udf btrfs xfs lvm "
- grub_cmd += "reiserfs ata "
- exec_cmd(grub_cmd)
elif source_params['loader'] == 'systemd-boot':
for mod in [x for x in os.listdir(kernel_dir) if x.startswith("systemd-")]:
cp_cmd = "cp %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, hdddir, mod[8:])
--
2.13.6
^ permalink raw reply [flat|nested] 51+ messages in thread* [PATCH 5/9] Revert "wic: Introduce the `WicExecError` exception class"
2018-01-31 9:41 [PATCH 0/9] first wic integration Henning Schild
` (3 preceding siblings ...)
2018-01-31 9:41 ` [PATCH 4/9] Revert "wic: Make the bootimg-efi plugin generate usable images" Henning Schild
@ 2018-01-31 9:41 ` Henning Schild
2018-01-31 9:41 ` [PATCH 6/9] Revert "wic: Work around mcopy error" Henning Schild
` (6 subsequent siblings)
11 siblings, 0 replies; 51+ messages in thread
From: Henning Schild @ 2018-01-31 9:41 UTC (permalink / raw)
To: isar-users; +Cc: Henning Schild
This reverts commit 17f9196e0f2cb7e1952369fb79f156fc8a3c678d.
---
scripts/lib/wic/partition.py | 10 +++-------
scripts/lib/wic/utils/misc.py | 17 +++--------------
2 files changed, 6 insertions(+), 21 deletions(-)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index c137a17..26be958 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -29,7 +29,7 @@ import os
import tempfile
from wic import WicError
-from wic.utils.misc import WicExecError, exec_cmd, get_bitbake_var
+from wic.utils.misc import exec_cmd, get_bitbake_var
from wic.pluginbase import PluginMgr
logger = logging.getLogger('wic')
@@ -253,12 +253,8 @@ class Partition():
(self.fstype, extra_imagecmd, rootfs, label_str, rootfs_dir)
exec_cmd(mkfs_cmd)
- try:
- mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs)
- exec_cmd(mkfs_cmd)
- except WicExecError as e:
- if e.returncode != 1:
- raise e
+ mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs)
+ exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
def prepare_rootfs_btrfs(self, rootfs, oe_builddir, rootfs_dir):
"""
diff --git a/scripts/lib/wic/utils/misc.py b/scripts/lib/wic/utils/misc.py
index b3a619d..4609984 100644
--- a/scripts/lib/wic/utils/misc.py
+++ b/scripts/lib/wic/utils/misc.py
@@ -59,16 +59,6 @@ NATIVE_RECIPES = {"bmaptool": "bmap-tools",
"syslinux": "syslinux"
}
-class WicExecError(WicError):
- def __init__(self, command, output, returncode):
- self.command = command
- self.output = output
- self.returncode = returncode
-
- def __str__(self):
- return "_exec_cmd: %s returned '%s' instead of 0\noutput: %s" % \
- (self.command, self.returncode, self.output)
-
def _exec_cmd(cmd_and_args, as_shell=False):
"""
Execute command, catching stderr, stdout
@@ -79,18 +69,17 @@ def _exec_cmd(cmd_and_args, as_shell=False):
args = cmd_and_args.split()
logger.debug(args)
- cmd = args
if as_shell:
- cmd = cmd_and_args
ret, out = runner.runtool(cmd_and_args)
else:
ret, out = runner.runtool(args)
out = out.strip()
if ret != 0:
- raise WicExecError(cmd, out, ret)
+ raise WicError("_exec_cmd: %s returned '%s' instead of 0\noutput: %s" % \
+ (cmd_and_args, ret, out))
logger.debug("_exec_cmd: output for %s (rc = %d): %s",
- cmd_and_args, out, ret)
+ cmd_and_args, ret, out)
return ret, out
--
2.13.6
^ permalink raw reply [flat|nested] 51+ messages in thread* [PATCH 6/9] Revert "wic: Work around mcopy error"
2018-01-31 9:41 [PATCH 0/9] first wic integration Henning Schild
` (4 preceding siblings ...)
2018-01-31 9:41 ` [PATCH 5/9] Revert "wic: Introduce the `WicExecError` exception class" Henning Schild
@ 2018-01-31 9:41 ` Henning Schild
2018-01-31 9:41 ` [PATCH 7/9] Revert "wic: Use sudo instead of pseudo" Henning Schild
` (5 subsequent siblings)
11 siblings, 0 replies; 51+ messages in thread
From: Henning Schild @ 2018-01-31 9:41 UTC (permalink / raw)
To: isar-users; +Cc: Henning Schild
This reverts commit b1511ec6f7bbbb3b872274cdb7320c7d6b868e2c.
---
scripts/lib/wic/partition.py | 5 ++---
scripts/lib/wic/plugins/source/bootimg-efi.py | 5 ++---
scripts/lib/wic/plugins/source/bootimg-pcbios.py | 5 ++---
scripts/lib/wic/plugins/source/isoimage-isohybrid.py | 5 ++---
4 files changed, 8 insertions(+), 12 deletions(-)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 26be958..34130d8 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -301,9 +301,8 @@ class Partition():
rootfs, rootfs_size)
exec_cmd(dosfs_cmd)
- mcopy_cmd = "env MTOOLS_SKIP_CHECK=1 mcopy -i %s -s %s/* ::/" % \
- (rootfs, rootfs_dir)
- exec_cmd(mcopy_cmd, True)
+ mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir)
+ exec_native_cmd(mcopy_cmd, native_sysroot)
chmod_cmd = "chmod 644 %s" % rootfs
exec_cmd(chmod_cmd)
diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
index f631e3f..823a536 100644
--- a/scripts/lib/wic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
@@ -240,9 +240,8 @@ class BootimgEFIPlugin(SourcePlugin):
dosfs_cmd = "mkdosfs -n efi -C %s %d" % (bootimg, blocks)
exec_cmd(dosfs_cmd)
- mcopy_cmd = "env MTOOLS_SKIP_CHECK=1 mcopy -i %s -s %s/* ::/" % \
- (bootimg, hdddir)
- exec_cmd(mcopy_cmd, True)
+ mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir)
+ exec_cmd(mcopy_cmd)
chmod_cmd = "chmod 644 %s" % bootimg
exec_cmd(chmod_cmd)
diff --git a/scripts/lib/wic/plugins/source/bootimg-pcbios.py b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
index 61c3837..b029c2f 100644
--- a/scripts/lib/wic/plugins/source/bootimg-pcbios.py
+++ b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
@@ -188,9 +188,8 @@ class BootimgPcbiosPlugin(SourcePlugin):
dosfs_cmd = "mkdosfs -n boot -S 512 -C %s %d" % (bootimg, blocks)
exec_cmd(dosfs_cmd)
- mcopy_cmd = "env MTOOLS_SKIP_CHECK=1 mcopy -i %s -s %s/* ::/" % \
- (bootimg, hdddir)
- exec_cmd(mcopy_cmd, True)
+ mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir)
+ exec_cmd(mcopy_cmd)
syslinux_cmd = "syslinux %s" % bootimg
exec_cmd(syslinux_cmd)
diff --git a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
index 68d2402..dcb6434 100644
--- a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
+++ b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
@@ -402,10 +402,9 @@ class IsoImagePlugin(SourcePlugin):
mmd_cmd = "mmd -i %s ::/EFI" % bootimg
exec_cmd(mmd_cmd)
- mcopy_cmd = "env MTOOLS_SKIP_CHECK=1 " \
- "mcopy -i %s -s %s/EFI/* ::/EFI/" \
+ mcopy_cmd = "mcopy -i %s -s %s/EFI/* ::/EFI/" \
% (bootimg, isodir)
- exec_cmd(mcopy_cmd, True)
+ exec_cmd(mcopy_cmd)
chmod_cmd = "chmod 644 %s" % bootimg
exec_cmd(chmod_cmd)
--
2.13.6
^ permalink raw reply [flat|nested] 51+ messages in thread* [PATCH 7/9] Revert "wic: Use sudo instead of pseudo"
2018-01-31 9:41 [PATCH 0/9] first wic integration Henning Schild
` (5 preceding siblings ...)
2018-01-31 9:41 ` [PATCH 6/9] Revert "wic: Work around mcopy error" Henning Schild
@ 2018-01-31 9:41 ` Henning Schild
2018-01-31 9:41 ` [PATCH 8/9] Revert "wic: Remove sysroot support" Henning Schild
` (4 subsequent siblings)
11 siblings, 0 replies; 51+ messages in thread
From: Henning Schild @ 2018-01-31 9:41 UTC (permalink / raw)
To: isar-users; +Cc: Henning Schild
This reverts commit bf873d3b089474d9061df4c1f0985abfac038ff6.
---
scripts/lib/wic/partition.py | 27 +++++++++++++++++++--------
1 file changed, 19 insertions(+), 8 deletions(-)
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 34130d8..20bc4a2 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -201,6 +201,17 @@ class Partition():
Currently handles ext2/3/4, btrfs and vfat.
"""
+ p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
+ p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
+ "%s/../pseudo" % rootfs_dir)
+ p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
+ p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
+ pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
+ pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir
+ pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd
+ pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp
+ pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+
rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
self.lineno, self.fstype)
if os.path.isfile(rootfs):
@@ -226,7 +237,7 @@ class Partition():
self.source_file = rootfs
# get the rootfs size in the right units for kickstart (kB)
- du_cmd = "sudo du -Lbks %s" % rootfs
+ du_cmd = "du -Lbks %s" % rootfs
out = exec_cmd(du_cmd)
self.size = int(out.split()[0])
@@ -234,7 +245,7 @@ class Partition():
"""
Prepare content for an ext2/3/4 rootfs partition.
"""
- du_cmd = "sudo du -ks %s" % rootfs_dir
+ du_cmd = "du -ks %s" % rootfs_dir
out = exec_cmd(du_cmd)
actual_rootfs_size = int(out.split()[0])
@@ -249,7 +260,7 @@ class Partition():
if self.label:
label_str = "-L %s" % self.label
- mkfs_cmd = "sudo mkfs.%s -F %s %s %s -d %s" % \
+ mkfs_cmd = "mkfs.%s -F %s %s %s -d %s" % \
(self.fstype, extra_imagecmd, rootfs, label_str, rootfs_dir)
exec_cmd(mkfs_cmd)
@@ -262,7 +273,7 @@ class Partition():
Currently handles ext2/3/4 and btrfs.
"""
- du_cmd = "sudo du -ks %s" % rootfs_dir
+ du_cmd = "du -ks %s" % rootfs_dir
out = exec_cmd(du_cmd)
actual_rootfs_size = int(out.split()[0])
@@ -275,7 +286,7 @@ class Partition():
if self.label:
label_str = "-L %s" % self.label
- mkfs_cmd = "sudo mkfs.%s -b %d -r %s %s %s" % \
+ mkfs_cmd = "mkfs.%s -b %d -r %s %s %s" % \
(self.fstype, rootfs_size * 1024, rootfs_dir, label_str, rootfs)
exec_cmd(mkfs_cmd)
@@ -283,7 +294,7 @@ class Partition():
"""
Prepare content for a msdos/vfat rootfs partition.
"""
- du_cmd = "sudo du -bks %s" % rootfs_dir
+ du_cmd = "du -bks %s" % rootfs_dir
out = exec_cmd(du_cmd)
blocks = int(out.split()[0])
@@ -331,7 +342,7 @@ class Partition():
if self.label:
label_str = "-L %s" % self.label
- mkfs_cmd = "sudo mkfs.%s -F %s %s %s" % \
+ mkfs_cmd = "mkfs.%s -F %s %s %s" % \
(self.fstype, extra_imagecmd, label_str, rootfs)
exec_cmd(mkfs_cmd)
@@ -347,7 +358,7 @@ class Partition():
if self.label:
label_str = "-L %s" % self.label
- mkfs_cmd = "sudo mkfs.%s -b %d %s %s" % \
+ mkfs_cmd = "mkfs.%s -b %d %s %s" % \
(self.fstype, self.size * 1024, label_str, rootfs)
exec_cmd(mkfs_cmd)
--
2.13.6
^ permalink raw reply [flat|nested] 51+ messages in thread* [PATCH 8/9] Revert "wic: Remove sysroot support"
2018-01-31 9:41 [PATCH 0/9] first wic integration Henning Schild
` (6 preceding siblings ...)
2018-01-31 9:41 ` [PATCH 7/9] Revert "wic: Use sudo instead of pseudo" Henning Schild
@ 2018-01-31 9:41 ` Henning Schild
2018-01-31 9:42 ` [PATCH 9/9] wic: now truly go for the wic version we claim to have Henning Schild
` (3 subsequent siblings)
11 siblings, 0 replies; 51+ messages in thread
From: Henning Schild @ 2018-01-31 9:41 UTC (permalink / raw)
To: isar-users; +Cc: Henning Schild
This reverts commit dd109de433577a55f26e8a76f04dfb39e6ed95a5.
---
scripts/lib/wic/engine.py | 7 ++-
scripts/lib/wic/partition.py | 59 +++++++++++++---------
scripts/lib/wic/pluginbase.py | 11 ++--
scripts/lib/wic/plugins/imager/direct.py | 49 ++++++++++--------
scripts/lib/wic/plugins/source/bootimg-efi.py | 13 +++--
.../lib/wic/plugins/source/bootimg-partition.py | 6 ++-
scripts/lib/wic/plugins/source/bootimg-pcbios.py | 19 ++++---
.../lib/wic/plugins/source/isoimage-isohybrid.py | 25 +++++----
scripts/lib/wic/plugins/source/rawcopy.py | 2 +-
scripts/lib/wic/plugins/source/rootfs.py | 4 +-
scripts/wic | 27 +++++++++-
11 files changed, 140 insertions(+), 82 deletions(-)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 15826f2..f59821f 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -145,7 +145,8 @@ def list_source_plugins():
print(" %s" % plugin)
-def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir, options):
+def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
+ native_sysroot, options):
"""
Create image
@@ -153,6 +154,7 @@ def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir, options):
rootfs_dir - absolute path to the build's /rootfs dir
bootimg_dir - absolute path to the build's boot artifacts directory
kernel_dir - absolute path to the build's kernel directory
+ native_sysroot - absolute path to the build's native sysroots dir
image_output_dir - dirname to create for image
options - wic command line options (debug, bmap, etc)
@@ -164,6 +166,7 @@ def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir, options):
rootfs_dir: IMAGE_ROOTFS
kernel_dir: DEPLOY_DIR_IMAGE
+ native_sysroot: STAGING_DIR_NATIVE
In the above case, bootimg_dir remains unset and the
plugin-specific image creation code is responsible for finding the
@@ -187,7 +190,7 @@ def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir, options):
raise WicError('Unknown plugin: %s' % pname)
plugin = plugin_class(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
- oe_builddir, options)
+ native_sysroot, oe_builddir, options)
plugin.do_create()
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 20bc4a2..939e667 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -29,7 +29,7 @@ import os
import tempfile
from wic import WicError
-from wic.utils.misc import exec_cmd, get_bitbake_var
+from wic.utils.misc import exec_cmd, exec_native_cmd, get_bitbake_var
from wic.pluginbase import PluginMgr
logger = logging.getLogger('wic')
@@ -125,7 +125,7 @@ class Partition():
return self.fixed_size if self.fixed_size else self.size
def prepare(self, creator, cr_workdir, oe_builddir, rootfs_dir,
- bootimg_dir, kernel_dir):
+ bootimg_dir, kernel_dir, native_sysroot):
"""
Prepare content for individual partitions, depending on
partition command parameters.
@@ -137,7 +137,8 @@ class Partition():
"partition." % self.mountpoint)
if self.fstype == "swap":
- self.prepare_swap_partition(cr_workdir, oe_builddir)
+ self.prepare_swap_partition(cr_workdir, oe_builddir,
+ native_sysroot)
self.source_file = "%s/fs.%s" % (cr_workdir, self.fstype)
else:
if self.fstype == 'squashfs':
@@ -151,7 +152,7 @@ class Partition():
prefix = "ext" if self.fstype.startswith("ext") else self.fstype
method = getattr(self, "prepare_empty_partition_" + prefix)
- method(rootfs, oe_builddir)
+ method(rootfs, oe_builddir, native_sysroot)
self.source_file = rootfs
return
@@ -174,13 +175,13 @@ class Partition():
plugin = PluginMgr.get_plugins('source')[self.source]
plugin.do_configure_partition(self, srcparams_dict, creator,
cr_workdir, oe_builddir, bootimg_dir,
- kernel_dir)
+ kernel_dir, native_sysroot)
plugin.do_stage_partition(self, srcparams_dict, creator,
cr_workdir, oe_builddir, bootimg_dir,
- kernel_dir)
+ kernel_dir, native_sysroot)
plugin.do_prepare_partition(self, srcparams_dict, creator,
cr_workdir, oe_builddir, bootimg_dir,
- kernel_dir, rootfs_dir)
+ kernel_dir, rootfs_dir, native_sysroot)
# further processing required Partition.size to be an integer, make
# sure that it is one
@@ -194,7 +195,8 @@ class Partition():
"larger (%d kB) than its allowed size %d kB" %
(self.mountpoint, self.size, self.fixed_size))
- def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir):
+ def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
+ native_sysroot):
"""
Prepare content for a rootfs partition i.e. create a partition
and fill it from a /rootfs dir.
@@ -233,7 +235,7 @@ class Partition():
prefix = "ext" if self.fstype.startswith("ext") else self.fstype
method = getattr(self, "prepare_rootfs_" + prefix)
- method(rootfs, oe_builddir, rootfs_dir)
+ method(rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo)
self.source_file = rootfs
# get the rootfs size in the right units for kickstart (kB)
@@ -241,7 +243,8 @@ class Partition():
out = exec_cmd(du_cmd)
self.size = int(out.split()[0])
- def prepare_rootfs_ext(self, rootfs, oe_builddir, rootfs_dir):
+ def prepare_rootfs_ext(self, rootfs, oe_builddir, rootfs_dir,
+ native_sysroot, pseudo):
"""
Prepare content for an ext2/3/4 rootfs partition.
"""
@@ -262,12 +265,13 @@ class Partition():
mkfs_cmd = "mkfs.%s -F %s %s %s -d %s" % \
(self.fstype, extra_imagecmd, rootfs, label_str, rootfs_dir)
- exec_cmd(mkfs_cmd)
+ exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs)
exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
- def prepare_rootfs_btrfs(self, rootfs, oe_builddir, rootfs_dir):
+ def prepare_rootfs_btrfs(self, rootfs, oe_builddir, rootfs_dir,
+ native_sysroot, pseudo):
"""
Prepare content for a btrfs rootfs partition.
@@ -288,9 +292,10 @@ class Partition():
mkfs_cmd = "mkfs.%s -b %d -r %s %s %s" % \
(self.fstype, rootfs_size * 1024, rootfs_dir, label_str, rootfs)
- exec_cmd(mkfs_cmd)
+ exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
- def prepare_rootfs_msdos(self, rootfs, oe_builddir, rootfs_dir):
+ def prepare_rootfs_msdos(self, rootfs, oe_builddir, rootfs_dir,
+ native_sysroot, pseudo):
"""
Prepare content for a msdos/vfat rootfs partition.
"""
@@ -310,7 +315,7 @@ class Partition():
dosfs_cmd = "mkdosfs %s -S 512 %s -C %s %d" % (label_str, size_str,
rootfs, rootfs_size)
- exec_cmd(dosfs_cmd)
+ exec_native_cmd(dosfs_cmd, native_sysroot)
mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir)
exec_native_cmd(mcopy_cmd, native_sysroot)
@@ -320,15 +325,17 @@ class Partition():
prepare_rootfs_vfat = prepare_rootfs_msdos
- def prepare_rootfs_squashfs(self, rootfs, oe_builddir, rootfs_dir):
+ def prepare_rootfs_squashfs(self, rootfs, oe_builddir, rootfs_dir,
+ native_sysroot, pseudo):
"""
Prepare content for a squashfs rootfs partition.
"""
squashfs_cmd = "mksquashfs %s %s -noappend" % \
(rootfs_dir, rootfs)
- exec_cmd(squashfs_cmd)
+ exec_native_cmd(squashfs_cmd, native_sysroot, pseudo=pseudo)
- def prepare_empty_partition_ext(self, rootfs, oe_builddir):
+ def prepare_empty_partition_ext(self, rootfs, oe_builddir,
+ native_sysroot):
"""
Prepare an empty ext2/3/4 partition.
"""
@@ -344,9 +351,10 @@ class Partition():
mkfs_cmd = "mkfs.%s -F %s %s %s" % \
(self.fstype, extra_imagecmd, label_str, rootfs)
- exec_cmd(mkfs_cmd)
+ exec_native_cmd(mkfs_cmd, native_sysroot)
- def prepare_empty_partition_btrfs(self, rootfs, oe_builddir):
+ def prepare_empty_partition_btrfs(self, rootfs, oe_builddir,
+ native_sysroot):
"""
Prepare an empty btrfs partition.
"""
@@ -360,9 +368,10 @@ class Partition():
mkfs_cmd = "mkfs.%s -b %d %s %s" % \
(self.fstype, self.size * 1024, label_str, rootfs)
- exec_cmd(mkfs_cmd)
+ exec_native_cmd(mkfs_cmd, native_sysroot)
- def prepare_empty_partition_msdos(self, rootfs, oe_builddir):
+ def prepare_empty_partition_msdos(self, rootfs, oe_builddir,
+ native_sysroot):
"""
Prepare an empty vfat partition.
"""
@@ -378,14 +387,14 @@ class Partition():
dosfs_cmd = "mkdosfs %s -S 512 %s -C %s %d" % (label_str, size_str,
rootfs, blocks)
- exec_cmd(dosfs_cmd)
+ exec_native_cmd(dosfs_cmd, native_sysroot)
chmod_cmd = "chmod 644 %s" % rootfs
exec_cmd(chmod_cmd)
prepare_empty_partition_vfat = prepare_empty_partition_msdos
- def prepare_swap_partition(self, cr_workdir, oe_builddir):
+ def prepare_swap_partition(self, cr_workdir, oe_builddir, native_sysroot):
"""
Prepare a swap partition.
"""
@@ -399,4 +408,4 @@ class Partition():
if self.label:
label_str = "-L %s" % self.label
mkswap_cmd = "mkswap %s -U %s %s" % (label_str, str(uuid.uuid1()), path)
- exec_cmd(mkswap_cmd)
+ exec_native_cmd(mkswap_cmd, native_sysroot)
diff --git a/scripts/lib/wic/pluginbase.py b/scripts/lib/wic/pluginbase.py
index 86d4a6d..fb3d179 100644
--- a/scripts/lib/wic/pluginbase.py
+++ b/scripts/lib/wic/pluginbase.py
@@ -92,7 +92,7 @@ class SourcePlugin(metaclass=PluginMeta):
@classmethod
def do_install_disk(cls, disk, disk_name, creator, workdir, oe_builddir,
- bootimg_dir, kernel_dir):
+ bootimg_dir, kernel_dir, native_sysroot):
"""
Called after all partitions have been prepared and assembled into a
disk image. This provides a hook to allow finalization of a
@@ -102,7 +102,8 @@ class SourcePlugin(metaclass=PluginMeta):
@classmethod
def do_stage_partition(cls, part, source_params, creator, cr_workdir,
- oe_builddir, bootimg_dir, kernel_dir):
+ oe_builddir, bootimg_dir, kernel_dir,
+ native_sysroot):
"""
Special content staging hook called before do_prepare_partition(),
normally empty.
@@ -118,7 +119,8 @@ class SourcePlugin(metaclass=PluginMeta):
@classmethod
def do_configure_partition(cls, part, source_params, creator, cr_workdir,
- oe_builddir, bootimg_dir, kernel_dir):
+ oe_builddir, bootimg_dir, kernel_dir,
+ native_sysroot):
"""
Called before do_prepare_partition(), typically used to create
custom configuration files for a partition, for example
@@ -128,7 +130,8 @@ class SourcePlugin(metaclass=PluginMeta):
@classmethod
def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
- oe_builddir, bootimg_dir, kernel_dir, rootfs_dir):
+ oe_builddir, bootimg_dir, kernel_dir, rootfs_dir,
+ native_sysroot):
"""
Called to do the actual content population for a partition i.e. it
'prepares' the partition to be incorporated into the image.
diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index 2e587b5..f2e6127 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -36,7 +36,7 @@ from wic import WicError
from wic.filemap import sparse_copy
from wic.ksparser import KickStart, KickStartError
from wic.pluginbase import PluginMgr, ImagerPlugin
-from wic.utils.misc import get_bitbake_var, exec_cmd
+from wic.utils.misc import get_bitbake_var, exec_cmd, exec_native_cmd
logger = logging.getLogger('wic')
@@ -52,7 +52,7 @@ class DirectPlugin(ImagerPlugin):
name = 'direct'
def __init__(self, wks_file, rootfs_dir, bootimg_dir, kernel_dir,
- oe_builddir, options):
+ native_sysroot, oe_builddir, options):
try:
self.ks = KickStart(wks_file)
except KickStartError as err:
@@ -62,6 +62,7 @@ class DirectPlugin(ImagerPlugin):
self.rootfs_dir = dict(rdir.split('=') for rdir in rootfs_dir.split(' '))
self.bootimg_dir = bootimg_dir
self.kernel_dir = kernel_dir
+ self.native_sysroot = native_sysroot
self.oe_builddir = oe_builddir
self.outdir = options.outdir
@@ -84,7 +85,7 @@ class DirectPlugin(ImagerPlugin):
image_path = self._full_path(self.workdir, self.parts[0].disk, "direct")
self._image = PartitionedImage(image_path, self.ptable_format,
- self.parts)
+ self.parts, self.native_sysroot)
def do_create(self):
"""
@@ -198,13 +199,14 @@ class DirectPlugin(ImagerPlugin):
plugin = PluginMgr.get_plugins('source')[source_plugin]
plugin.do_install_disk(self._image, disk_name, self, self.workdir,
self.oe_builddir, self.bootimg_dir,
- self.kernel_dir)
+ self.kernel_dir, self.native_sysroot)
full_path = self._image.path
# Generate .bmap
if self.bmap:
logger.debug("Generating bmap file for %s", disk_name)
- exec_cmd("bmaptool create %s -o %s.bmap" % (full_path, full_path))
+ exec_native_cmd("bmaptool create %s -o %s.bmap" % (full_path, full_path),
+ self.native_sysroot)
# Compress the image
if self.compressor:
logger.debug("Compressing disk %s with %s", disk_name, self.compressor)
@@ -235,6 +237,7 @@ class DirectPlugin(ImagerPlugin):
msg += ' BOOTIMG_DIR: %s\n' % self.bootimg_dir
msg += ' KERNEL_DIR: %s\n' % self.kernel_dir
+ msg += ' NATIVE_SYSROOT: %s\n' % self.native_sysroot
logger.info(msg)
@@ -284,7 +287,7 @@ class PartitionedImage():
Partitioned image in a file.
"""
- def __init__(self, path, ptable_format, partitions):
+ def __init__(self, path, ptable_format, partitions, native_sysroot=None):
self.path = path # Path to the image file
self.numpart = 0 # Number of allocated partitions
self.realpart = 0 # Number of partitions in the partition table
@@ -299,6 +302,7 @@ class PartitionedImage():
self.partimages = []
# Size of a sector used in calculations
self.sector_size = SECTOR_SIZE
+ self.native_sysroot = native_sysroot
# calculate the real partition number, accounting for partitions not
# in the partition table and logical partitions
@@ -328,7 +332,7 @@ class PartitionedImage():
# sizes before we can add them and do the layout.
part.prepare(imager, imager.workdir, imager.oe_builddir,
imager.rootfs_dir, imager.bootimg_dir,
- imager.kernel_dir)
+ imager.kernel_dir, imager.native_sysroot)
# Converting kB to sectors for parted
part.size_sec = part.disk_size * 1024 // self.sector_size
@@ -439,7 +443,7 @@ class PartitionedImage():
cmd += " %s" % fstype
cmd += " %d %d" % (start, end)
- return exec_cmd(cmd)
+ return exec_native_cmd(cmd, self.native_sysroot)
def create(self):
logger.debug("Creating sparse file %s", self.path)
@@ -447,8 +451,8 @@ class PartitionedImage():
os.ftruncate(sparse.fileno(), self.min_size)
logger.debug("Initializing partition table for %s", self.path)
- exec_cmd("parted -s %s mklabel %s" %
- (self.path, self.ptable_format))
+ exec_native_cmd("parted -s %s mklabel %s" %
+ (self.path, self.ptable_format), self.native_sysroot)
logger.debug("Set disk identifier %x", self.identifier)
with open(self.path, 'r+b') as img:
@@ -504,30 +508,35 @@ class PartitionedImage():
if part.part_type:
logger.debug("partition %d: set type UID to %s",
part.num, part.part_type)
- exec_cmd("sgdisk --typecode=%d:%s %s" % \
- (part.num, part.part_type, self.path))
+ exec_native_cmd("sgdisk --typecode=%d:%s %s" % \
+ (part.num, part.part_type,
+ self.path), self.native_sysroot)
if part.uuid and self.ptable_format == "gpt":
logger.debug("partition %d: set UUID to %s",
part.num, part.uuid)
- exec_cmd("sgdisk --partition-guid=%d:%s %s" % \
- (part.num, part.uuid, self.path))
+ exec_native_cmd("sgdisk --partition-guid=%d:%s %s" % \
+ (part.num, part.uuid, self.path),
+ self.native_sysroot)
if part.label and self.ptable_format == "gpt":
logger.debug("partition %d: set name to %s",
part.num, part.label)
- exec_cmd("parted -s %s name %d %s" % \
- (self.path, part.num, part.label))
+ exec_native_cmd("parted -s %s name %d %s" % \
+ (self.path, part.num, part.label),
+ self.native_sysroot)
if part.active:
flag_name = "legacy_boot" if self.ptable_format == 'gpt' else "boot"
logger.debug("Set '%s' flag for partition '%s' on disk '%s'",
flag_name, part.num, self.path)
- exec_cmd("parted -s %s set %d %s on" % \
- (self.path, part.num, flag_name))
+ exec_native_cmd("parted -s %s set %d %s on" % \
+ (self.path, part.num, flag_name),
+ self.native_sysroot)
if part.system_id:
- exec_cmd("sfdisk --part-type %s %s %s" % \
- (self.path, part.num, part.system_id))
+ exec_native_cmd("sfdisk --part-type %s %s %s" % \
+ (self.path, part.num, part.system_id),
+ self.native_sysroot)
def cleanup(self):
# remove partition images
diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
index 823a536..9879cb9 100644
--- a/scripts/lib/wic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
@@ -31,7 +31,8 @@ import shutil
from wic import WicError
from wic.engine import get_custom_config
from wic.pluginbase import SourcePlugin
-from wic.utils.misc import (exec_cmd, get_bitbake_var, BOOTDD_EXTRA_SPACE)
+from wic.utils.misc import (exec_cmd, exec_native_cmd, get_bitbake_var,
+ BOOTDD_EXTRA_SPACE)
logger = logging.getLogger('wic')
@@ -154,7 +155,8 @@ class BootimgEFIPlugin(SourcePlugin):
@classmethod
def do_configure_partition(cls, part, source_params, creator, cr_workdir,
- oe_builddir, bootimg_dir, kernel_dir):
+ oe_builddir, bootimg_dir, kernel_dir,
+ native_sysroot):
"""
Called before do_prepare_partition(), creates loader-specific config
"""
@@ -176,7 +178,8 @@ class BootimgEFIPlugin(SourcePlugin):
@classmethod
def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
- oe_builddir, bootimg_dir, kernel_dir, rootfs_dir):
+ oe_builddir, bootimg_dir, kernel_dir,
+ rootfs_dir, native_sysroot):
"""
Called to do the actual content population for a partition i.e. it
'prepares' the partition to be incorporated into the image.
@@ -238,10 +241,10 @@ class BootimgEFIPlugin(SourcePlugin):
bootimg = "%s/boot.img" % cr_workdir
dosfs_cmd = "mkdosfs -n efi -C %s %d" % (bootimg, blocks)
- exec_cmd(dosfs_cmd)
+ exec_native_cmd(dosfs_cmd, native_sysroot)
mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir)
- exec_cmd(mcopy_cmd)
+ exec_native_cmd(mcopy_cmd, native_sysroot)
chmod_cmd = "chmod 644 %s" % bootimg
exec_cmd(chmod_cmd)
diff --git a/scripts/lib/wic/plugins/source/bootimg-partition.py b/scripts/lib/wic/plugins/source/bootimg-partition.py
index a00dc44..13fddbd 100644
--- a/scripts/lib/wic/plugins/source/bootimg-partition.py
+++ b/scripts/lib/wic/plugins/source/bootimg-partition.py
@@ -45,7 +45,8 @@ class BootimgPartitionPlugin(SourcePlugin):
@classmethod
def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
- oe_builddir, bootimg_dir, kernel_dir, rootfs_dir):
+ oe_builddir, bootimg_dir, kernel_dir,
+ rootfs_dir, native_sysroot):
"""
Called to do the actual content population for a partition i.e. it
'prepares' the partition to be incorporated into the image.
@@ -118,4 +119,5 @@ class BootimgPartitionPlugin(SourcePlugin):
exec_cmd(install_cmd)
logger.debug('Prepare boot partition using rootfs in %s', hdddir)
- part.prepare_rootfs(cr_workdir, oe_builddir, hdddir)
+ part.prepare_rootfs(cr_workdir, oe_builddir, hdddir,
+ native_sysroot)
diff --git a/scripts/lib/wic/plugins/source/bootimg-pcbios.py b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
index b029c2f..5890c12 100644
--- a/scripts/lib/wic/plugins/source/bootimg-pcbios.py
+++ b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
@@ -31,7 +31,8 @@ from wic import WicError
from wic.engine import get_custom_config
from wic.utils import runner
from wic.pluginbase import SourcePlugin
-from wic.utils.misc import (exec_cmd, get_bitbake_var, BOOTDD_EXTRA_SPACE)
+from wic.utils.misc import (exec_cmd, exec_native_cmd,
+ get_bitbake_var, BOOTDD_EXTRA_SPACE)
logger = logging.getLogger('wic')
@@ -56,7 +57,7 @@ class BootimgPcbiosPlugin(SourcePlugin):
@classmethod
def do_install_disk(cls, disk, disk_name, creator, workdir, oe_builddir,
- bootimg_dir, kernel_dir):
+ bootimg_dir, kernel_dir, native_sysroot):
"""
Called after all partitions have been prepared and assembled into a
disk image. In this case, we install the MBR.
@@ -81,11 +82,12 @@ class BootimgPcbiosPlugin(SourcePlugin):
disk_name, full_path, disk.min_size)
dd_cmd = "dd if=%s of=%s conv=notrunc" % (mbrfile, full_path)
- exec_cmd(dd_cmd)
+ exec_cmd(dd_cmd, native_sysroot)
@classmethod
def do_configure_partition(cls, part, source_params, creator, cr_workdir,
- oe_builddir, bootimg_dir, kernel_dir):
+ oe_builddir, bootimg_dir, kernel_dir,
+ native_sysroot):
"""
Called before do_prepare_partition(), creates syslinux config
"""
@@ -142,7 +144,8 @@ class BootimgPcbiosPlugin(SourcePlugin):
@classmethod
def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
- oe_builddir, bootimg_dir, kernel_dir, rootfs_dir):
+ oe_builddir, bootimg_dir, kernel_dir,
+ rootfs_dir, native_sysroot):
"""
Called to do the actual content population for a partition i.e. it
'prepares' the partition to be incorporated into the image.
@@ -186,13 +189,13 @@ class BootimgPcbiosPlugin(SourcePlugin):
bootimg = "%s/boot.img" % cr_workdir
dosfs_cmd = "mkdosfs -n boot -S 512 -C %s %d" % (bootimg, blocks)
- exec_cmd(dosfs_cmd)
+ exec_native_cmd(dosfs_cmd, native_sysroot)
mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir)
- exec_cmd(mcopy_cmd)
+ exec_native_cmd(mcopy_cmd, native_sysroot)
syslinux_cmd = "syslinux %s" % bootimg
- exec_cmd(syslinux_cmd)
+ exec_native_cmd(syslinux_cmd, native_sysroot)
chmod_cmd = "chmod 644 %s" % bootimg
exec_cmd(chmod_cmd)
diff --git a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
index dcb6434..1ceba62 100644
--- a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
+++ b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
@@ -29,7 +29,7 @@ import shutil
from wic import WicError
from wic.engine import get_custom_config
from wic.pluginbase import SourcePlugin
-from wic.utils.misc import exec_cmd, get_bitbake_var
+from wic.utils.misc import exec_cmd, exec_native_cmd, get_bitbake_var
logger = logging.getLogger('wic')
@@ -199,7 +199,8 @@ class IsoImagePlugin(SourcePlugin):
@classmethod
def do_configure_partition(cls, part, source_params, creator, cr_workdir,
- oe_builddir, bootimg_dir, kernel_dir):
+ oe_builddir, bootimg_dir, kernel_dir,
+ native_sysroot):
"""
Called before do_prepare_partition(), creates loader-specific config
"""
@@ -220,7 +221,8 @@ class IsoImagePlugin(SourcePlugin):
@classmethod
def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
- oe_builddir, bootimg_dir, kernel_dir, rootfs_dir):
+ oe_builddir, bootimg_dir, kernel_dir,
+ rootfs_dir, native_sysroot):
"""
Called to do the actual content population for a partition i.e. it
'prepares' the partition to be incorporated into the image.
@@ -269,7 +271,8 @@ class IsoImagePlugin(SourcePlugin):
part.size = int(out.split()[0])
part.extra_space = 0
part.overhead_factor = 1.2
- part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir)
+ part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, \
+ native_sysroot)
rootfs_img = part.source_file
install_cmd = "install -m 0644 %s %s/rootfs.img" \
@@ -360,7 +363,7 @@ class IsoImagePlugin(SourcePlugin):
grub_cmd += "reboot serial terminfo iso9660 loopback tar "
grub_cmd += "memdisk ls search_fs_uuid udf btrfs xfs lvm "
grub_cmd += "reiserfs ata "
- exec_cmd(grub_cmd)
+ exec_native_cmd(grub_cmd, native_sysroot)
else:
raise WicError("unrecognized bootimg-efi loader: %s" %
@@ -397,14 +400,14 @@ class IsoImagePlugin(SourcePlugin):
dosfs_cmd = 'mkfs.vfat -n "EFIimg" -S 512 -C %s %d' \
% (bootimg, blocks)
- exec_cmd(dosfs_cmd)
+ exec_native_cmd(dosfs_cmd, native_sysroot)
mmd_cmd = "mmd -i %s ::/EFI" % bootimg
- exec_cmd(mmd_cmd)
+ exec_native_cmd(mmd_cmd, native_sysroot)
mcopy_cmd = "mcopy -i %s -s %s/EFI/* ::/EFI/" \
% (bootimg, isodir)
- exec_cmd(mcopy_cmd)
+ exec_native_cmd(mcopy_cmd, native_sysroot)
chmod_cmd = "chmod 644 %s" % bootimg
exec_cmd(chmod_cmd)
@@ -453,7 +456,7 @@ class IsoImagePlugin(SourcePlugin):
mkisofs_cmd += "-no-emul-boot %s " % isodir
logger.debug("running command: %s", mkisofs_cmd)
- exec_cmd(mkisofs_cmd)
+ exec_native_cmd(mkisofs_cmd, native_sysroot)
shutil.rmtree(isodir)
@@ -466,7 +469,7 @@ class IsoImagePlugin(SourcePlugin):
@classmethod
def do_install_disk(cls, disk, disk_name, creator, workdir, oe_builddir,
- bootimg_dir, kernel_dir):
+ bootimg_dir, kernel_dir, native_sysroot):
"""
Called after all partitions have been prepared and assembled into a
disk image. In this case, we insert/modify the MBR using isohybrid
@@ -479,7 +482,7 @@ class IsoImagePlugin(SourcePlugin):
isohybrid_cmd = "isohybrid -u %s" % iso_img
logger.debug("running command: %s", isohybrid_cmd)
- exec_cmd(isohybrid_cmd)
+ exec_native_cmd(isohybrid_cmd, native_sysroot)
# Replace the image created by direct plugin with the one created by
# mkisofs command. This is necessary because the iso image created by
diff --git a/scripts/lib/wic/plugins/source/rawcopy.py b/scripts/lib/wic/plugins/source/rawcopy.py
index 392ae7e..e1c4f5e 100644
--- a/scripts/lib/wic/plugins/source/rawcopy.py
+++ b/scripts/lib/wic/plugins/source/rawcopy.py
@@ -35,7 +35,7 @@ class RawCopyPlugin(SourcePlugin):
@classmethod
def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
oe_builddir, bootimg_dir, kernel_dir,
- rootfs_dir):
+ rootfs_dir, native_sysroot):
"""
Called to do the actual content population for a partition i.e. it
'prepares' the partition to be incorporated into the image.
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index e6cdc88..f2e2ca8 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -60,7 +60,7 @@ class RootfsPlugin(SourcePlugin):
@classmethod
def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
oe_builddir, bootimg_dir, kernel_dir,
- krootfs_dir):
+ krootfs_dir, native_sysroot):
"""
Called to do the actual content population for a partition i.e. it
'prepares' the partition to be incorporated into the image.
@@ -122,4 +122,4 @@ class RootfsPlugin(SourcePlugin):
part.rootfs_dir = real_rootfs_dir
part.prepare_rootfs(cr_workdir, oe_builddir,
- real_rootfs_dir)
+ real_rootfs_dir, native_sysroot)
diff --git a/scripts/wic b/scripts/wic
index 641dd2e..a5f2dbf 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -123,6 +123,9 @@ def wic_create_subcommand(args, usage_str):
parser.add_option("-k", "--kernel-dir", dest="kernel_dir",
help="path to the dir containing the kernel to use "
"in the .wks bootimg")
+ parser.add_option("-n", "--native-sysroot", dest="native_sysroot",
+ help="path to the native sysroot containing the tools "
+ "to use to build the image")
parser.add_option("-s", "--skip-build-check", dest="build_check",
action="store_false", default=True, help="skip the build check")
parser.add_option("-f", "--build-rootfs", action="store_true", help="build rootfs")
@@ -149,7 +152,8 @@ def wic_create_subcommand(args, usage_str):
missed = []
for val, opt in [(options.rootfs_dir, 'rootfs-dir'),
(options.bootimg_dir, 'bootimg-dir'),
- (options.kernel_dir, 'kernel-dir')]:
+ (options.kernel_dir, 'kernel-dir'),
+ (options.native_sysroot, 'native-sysroot')]:
if not val:
missed.append(opt)
if missed:
@@ -184,10 +188,23 @@ def wic_create_subcommand(args, usage_str):
rootfs_dir = get_bitbake_var("IMAGE_ROOTFS", options.image_name)
kernel_dir = get_bitbake_var("DEPLOY_DIR_IMAGE", options.image_name)
bootimg_dir = get_bitbake_var("STAGING_DATADIR", options.image_name)
+ native_sysroot = get_bitbake_var("RECIPE_SYSROOT_NATIVE",
+ options.image_name) #, cache=False)
else:
if options.build_rootfs:
raise WicError("Image name is not specified, exiting. "
"(Use -e/--image-name to specify it)")
+ native_sysroot = options.native_sysroot
+
+ if not native_sysroot or not os.path.isdir(native_sysroot):
+ logger.info("Building wic-tools...\n")
+ if bitbake_main(BitBakeConfigParameters("bitbake wic-tools".split()),
+ cookerdata.CookerConfiguration()):
+ raise WicError("bitbake wic-tools failed")
+ native_sysroot = get_bitbake_var("RECIPE_SYSROOT_NATIVE", "wic-tools")
+ if not native_sysroot:
+ raise WicError("Unable to find the location of the native "
+ "tools sysroot to use")
wks_file = args[0]
@@ -204,18 +221,23 @@ def wic_create_subcommand(args, usage_str):
rootfs_dir = options.rootfs_dir['ROOTFS_DIR']
bootimg_dir = options.bootimg_dir
kernel_dir = options.kernel_dir
+ native_sysroot = options.native_sysroot
if rootfs_dir and not os.path.isdir(rootfs_dir):
raise WicError("--rootfs-dir (-r) not found, exiting")
if not os.path.isdir(bootimg_dir):
raise WicError("--bootimg-dir (-b) not found, exiting")
if not os.path.isdir(kernel_dir):
raise WicError("--kernel-dir (-k) not found, exiting")
+ if not os.path.isdir(native_sysroot):
+ raise WicError("--native-sysroot (-n) not found, exiting")
else:
not_found = not_found_dir = ""
if not os.path.isdir(rootfs_dir):
(not_found, not_found_dir) = ("rootfs-dir", rootfs_dir)
elif not os.path.isdir(kernel_dir):
(not_found, not_found_dir) = ("kernel-dir", kernel_dir)
+ elif not os.path.isdir(native_sysroot):
+ (not_found, not_found_dir) = ("native-sysroot", native_sysroot)
if not_found:
if not not_found_dir:
not_found_dir = "Completely missing artifact - wrong image (.wks) used?"
@@ -233,7 +255,8 @@ def wic_create_subcommand(args, usage_str):
rootfs_dir = rootfs_dir_to_args(krootfs_dir)
logger.info("Creating image(s)...\n")
- engine.wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir, options)
+ engine.wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
+ native_sysroot, options)
def wic_list_subcommand(args, usage_str):
--
2.13.6
^ permalink raw reply [flat|nested] 51+ messages in thread* [PATCH 9/9] wic: now truly go for the wic version we claim to have
2018-01-31 9:41 [PATCH 0/9] first wic integration Henning Schild
` (7 preceding siblings ...)
2018-01-31 9:41 ` [PATCH 8/9] Revert "wic: Remove sysroot support" Henning Schild
@ 2018-01-31 9:42 ` Henning Schild
2018-01-31 10:11 ` Alexander Smirnov
2018-01-31 10:02 ` [PATCH 0/9] first wic integration Alexander Smirnov
` (2 subsequent siblings)
11 siblings, 1 reply; 51+ messages in thread
From: Henning Schild @ 2018-01-31 9:42 UTC (permalink / raw)
To: isar-users; +Cc: Henning Schild
2b164b18fd639c9 claims to introduce wic hash 131629ca6238ea05
This commit really carries that version of wic.
Issue:
- the wic version in Isar was modified
- that causes:
- confusion, maintainability and updateability issues
- potential quality issues
Impact:
This patch and the previous reverts get wic back to a state where all
these Issues are solved. We could now just update our wic without having
to worry about local patches. In case of a wic-update Isar and layers
on top would still have to review their plugins.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
scripts/lib/wic/canned-wks/qemux86-directdisk.wks | 2 +-
scripts/lib/wic/filemap.py | 6 +-
scripts/lib/wic/help.py | 2 -
scripts/lib/wic/ksparser.py | 4 +-
scripts/lib/wic/partition.py | 114 ++++++++++++++--------
scripts/lib/wic/plugins/imager/direct.py | 17 +++-
scripts/lib/wic/plugins/source/bootimg-pcbios.py | 48 ++++-----
scripts/lib/wic/plugins/source/fsimage.py | 56 +++++++++++
scripts/lib/wic/utils/misc.py | 16 +--
scripts/lib/wic/utils/runner.py | 74 ++++++++++++--
10 files changed, 250 insertions(+), 89 deletions(-)
create mode 100644 scripts/lib/wic/plugins/source/fsimage.py
diff --git a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks
index db30bbc..a6518a0 100644
--- a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks
+++ b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks
@@ -4,5 +4,5 @@
include common.wks.inc
-bootloader --timeout=0 --append="vga=0 uvesafb.mode_option=640x480-32 root=/dev/sda2 rw mem=256M ip=192.168.7.2::192.168.7.1:255.255.255.0 oprofile.timer=1 rootfstype=ext4 "
+bootloader --timeout=0 --append="vga=0 uvesafb.mode_option=640x480-32 root=/dev/vda2 rw mem=256M ip=192.168.7.2::192.168.7.1:255.255.255.0 oprofile.timer=1 rootfstype=ext4 "
diff --git a/scripts/lib/wic/filemap.py b/scripts/lib/wic/filemap.py
index 1f1aacc..080668e 100644
--- a/scripts/lib/wic/filemap.py
+++ b/scripts/lib/wic/filemap.py
@@ -530,11 +530,9 @@ def filemap(image, log=None):
except ErrorNotSupp:
return FilemapSeek(image, log)
-def sparse_copy(src_fname, dst_fname, offset=0, skip=0, api=None):
+def sparse_copy(src_fname, dst_fname, offset=0, skip=0):
"""Efficiently copy sparse file to or into another file."""
- if not api:
- api = filemap
- fmap = api(src_fname)
+ fmap = filemap(src_fname)
try:
dst_file = open(dst_fname, 'r+b')
except IOError:
diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index aee2451..148da89 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -687,8 +687,6 @@ DESCRIPTION
apply to partitions created using '--source rootfs' (see
--source above). Valid values are:
- vfat
- msdos
ext2
ext3
ext4
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index d026caa..a039300 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -136,9 +136,7 @@ class KickStart():
part.add_argument('--exclude-path', nargs='+')
part.add_argument("--extra-space", type=sizetype)
part.add_argument('--fsoptions', dest='fsopts')
- part.add_argument('--fstype', default='vfat',
- choices=('ext2', 'ext3', 'ext4', 'btrfs',
- 'squashfs', 'vfat', 'msdos', 'swap'))
+ part.add_argument('--fstype')
part.add_argument('--label')
part.add_argument('--no-table', action='store_true')
part.add_argument('--ondisk', '--ondrive', dest='disk', default='sda')
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 939e667..8e32afc 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -136,24 +136,22 @@ class Partition():
"specify a non-zero --size/--fixed-size for that "
"partition." % self.mountpoint)
- if self.fstype == "swap":
+ if self.fstype and self.fstype == "swap":
self.prepare_swap_partition(cr_workdir, oe_builddir,
native_sysroot)
self.source_file = "%s/fs.%s" % (cr_workdir, self.fstype)
- else:
- if self.fstype == 'squashfs':
- raise WicError("It's not possible to create empty squashfs "
- "partition '%s'" % (self.mountpoint))
-
+ elif self.fstype:
rootfs = "%s/fs_%s.%s.%s" % (cr_workdir, self.label,
self.lineno, self.fstype)
if os.path.isfile(rootfs):
os.remove(rootfs)
-
- prefix = "ext" if self.fstype.startswith("ext") else self.fstype
- method = getattr(self, "prepare_empty_partition_" + prefix)
- method(rootfs, oe_builddir, native_sysroot)
- self.source_file = rootfs
+ for prefix in ("ext", "btrfs", "vfat", "squashfs"):
+ if self.fstype.startswith(prefix):
+ method = getattr(self,
+ "prepare_empty_partition_" + prefix)
+ method(rootfs, oe_builddir, native_sysroot)
+ self.source_file = rootfs
+ break
return
plugins = PluginMgr.get_plugins('source')
@@ -195,6 +193,19 @@ class Partition():
"larger (%d kB) than its allowed size %d kB" %
(self.mountpoint, self.size, self.fixed_size))
+ def prepare_rootfs_from_fs_image(self, cr_workdir, oe_builddir,
+ rootfs_dir):
+ """
+ Handle an already-created partition e.g. xxx.ext3
+ """
+ rootfs = oe_builddir
+ du_cmd = "du -Lbks %s" % rootfs
+ out = exec_cmd(du_cmd)
+ rootfs_size = out.split()[0]
+
+ self.size = int(rootfs_size)
+ self.source_file = rootfs
+
def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
native_sysroot):
"""
@@ -219,6 +230,10 @@ class Partition():
if os.path.isfile(rootfs):
os.remove(rootfs)
+ if not self.fstype:
+ raise WicError("File system for partition %s not specified in "
+ "kickstart, use --fstype option" % self.mountpoint)
+
# Get rootfs size from bitbake variable if it's not set in .ks file
if not self.size:
# Bitbake variable ROOTFS_SIZE is calculated in
@@ -233,15 +248,19 @@ class Partition():
'--overhead-factor will be applied')
self.size = int(round(float(rsize_bb)))
- prefix = "ext" if self.fstype.startswith("ext") else self.fstype
- method = getattr(self, "prepare_rootfs_" + prefix)
- method(rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo)
- self.source_file = rootfs
+ for prefix in ("ext", "btrfs", "vfat", "squashfs"):
+ if self.fstype.startswith(prefix):
+ method = getattr(self, "prepare_rootfs_" + prefix)
+ method(rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo)
- # get the rootfs size in the right units for kickstart (kB)
- du_cmd = "du -Lbks %s" % rootfs
- out = exec_cmd(du_cmd)
- self.size = int(out.split()[0])
+ self.source_file = rootfs
+
+ # get the rootfs size in the right units for kickstart (kB)
+ du_cmd = "du -Lbks %s" % rootfs
+ out = exec_cmd(du_cmd)
+ self.size = int(out.split()[0])
+
+ break
def prepare_rootfs_ext(self, rootfs, oe_builddir, rootfs_dir,
native_sysroot, pseudo):
@@ -267,7 +286,7 @@ class Partition():
(self.fstype, extra_imagecmd, rootfs, label_str, rootfs_dir)
exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
- mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs)
+ mkfs_cmd = "fsck.%s -fy %s" % (self.fstype, rootfs)
exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
def prepare_rootfs_btrfs(self, rootfs, oe_builddir, rootfs_dir,
@@ -294,10 +313,10 @@ class Partition():
(self.fstype, rootfs_size * 1024, rootfs_dir, label_str, rootfs)
exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
- def prepare_rootfs_msdos(self, rootfs, oe_builddir, rootfs_dir,
- native_sysroot, pseudo):
+ def prepare_rootfs_vfat(self, rootfs, oe_builddir, rootfs_dir,
+ native_sysroot, pseudo):
"""
- Prepare content for a msdos/vfat rootfs partition.
+ Prepare content for a vfat rootfs partition.
"""
du_cmd = "du -bks %s" % rootfs_dir
out = exec_cmd(du_cmd)
@@ -309,12 +328,7 @@ class Partition():
if self.label:
label_str = "-n %s" % self.label
- size_str = ""
- if self.fstype == 'msdos':
- size_str = "-F 16" # FAT 16
-
- dosfs_cmd = "mkdosfs %s -S 512 %s -C %s %d" % (label_str, size_str,
- rootfs, rootfs_size)
+ dosfs_cmd = "mkdosfs %s -S 512 -C %s %d" % (label_str, rootfs, rootfs_size)
exec_native_cmd(dosfs_cmd, native_sysroot)
mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir)
@@ -323,8 +337,6 @@ class Partition():
chmod_cmd = "chmod 644 %s" % rootfs
exec_cmd(chmod_cmd)
- prepare_rootfs_vfat = prepare_rootfs_msdos
-
def prepare_rootfs_squashfs(self, rootfs, oe_builddir, rootfs_dir,
native_sysroot, pseudo):
"""
@@ -370,8 +382,8 @@ class Partition():
(self.fstype, self.size * 1024, label_str, rootfs)
exec_native_cmd(mkfs_cmd, native_sysroot)
- def prepare_empty_partition_msdos(self, rootfs, oe_builddir,
- native_sysroot):
+ def prepare_empty_partition_vfat(self, rootfs, oe_builddir,
+ native_sysroot):
"""
Prepare an empty vfat partition.
"""
@@ -381,18 +393,40 @@ class Partition():
if self.label:
label_str = "-n %s" % self.label
- size_str = ""
- if self.fstype == 'msdos':
- size_str = "-F 16" # FAT 16
-
- dosfs_cmd = "mkdosfs %s -S 512 %s -C %s %d" % (label_str, size_str,
- rootfs, blocks)
+ dosfs_cmd = "mkdosfs %s -S 512 -C %s %d" % (label_str, rootfs, blocks)
exec_native_cmd(dosfs_cmd, native_sysroot)
chmod_cmd = "chmod 644 %s" % rootfs
exec_cmd(chmod_cmd)
- prepare_empty_partition_vfat = prepare_empty_partition_msdos
+ def prepare_empty_partition_squashfs(self, cr_workdir, oe_builddir,
+ native_sysroot):
+ """
+ Prepare an empty squashfs partition.
+ """
+ logger.warning("Creating of an empty squashfs %s partition was attempted. "
+ "Proceeding as requested.", self.mountpoint)
+
+ path = "%s/fs_%s.%s" % (cr_workdir, self.label, self.fstype)
+ if os.path.isfile(path):
+ os.remove(path)
+
+ # it is not possible to create a squashfs without source data,
+ # thus prepare an empty temp dir that is used as source
+ tmpdir = tempfile.mkdtemp()
+
+ squashfs_cmd = "mksquashfs %s %s -noappend" % \
+ (tmpdir, path)
+ exec_native_cmd(squashfs_cmd, native_sysroot)
+
+ os.rmdir(tmpdir)
+
+ # get the rootfs size in the right units for kickstart (kB)
+ du_cmd = "du -Lbks %s" % path
+ out = exec_cmd(du_cmd)
+ fs_size = out.split()[0]
+
+ self.size = int(fs_size)
def prepare_swap_partition(self, cr_workdir, oe_builddir, native_sysroot):
"""
diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index f2e6127..7d38ab3 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -323,7 +323,7 @@ class PartitionedImage():
if self.ptable_format == 'gpt':
part.uuid = str(uuid.uuid4())
else: # msdos partition table
- part.uuid = '%08x-%02d' % (self.identifier, part.realnum)
+ part.uuid = '%0x-%02d' % (self.identifier, part.realnum)
def prepare(self, imager):
"""Prepare an image. Call prepare method of all image partitions."""
@@ -487,8 +487,8 @@ class PartitionedImage():
parted_fs_type = "fat32"
elif part.fstype == "msdos":
parted_fs_type = "fat16"
- if not part.system_id:
- part.system_id = '0x6' # FAT16
+ elif part.fstype == "ontrackdm6aux3":
+ parted_fs_type = "ontrackdm6aux3"
else:
# Type for ext2/ext3/ext4/btrfs
parted_fs_type = "ext2"
@@ -538,6 +538,17 @@ class PartitionedImage():
(self.path, part.num, part.system_id),
self.native_sysroot)
+ # Parted defaults to enabling the lba flag for fat16 partitions,
+ # which causes compatibility issues with some firmware (and really
+ # isn't necessary).
+ if parted_fs_type == "fat16":
+ if self.ptable_format == 'msdos':
+ logger.debug("Disable 'lba' flag for partition '%s' on disk '%s'",
+ part.num, self.path)
+ exec_native_cmd("parted -s %s set %d lba off" % \
+ (self.path, part.num),
+ self.native_sysroot)
+
def cleanup(self):
# remove partition images
for image in set(self.partimages):
diff --git a/scripts/lib/wic/plugins/source/bootimg-pcbios.py b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
index 5890c12..11db304 100644
--- a/scripts/lib/wic/plugins/source/bootimg-pcbios.py
+++ b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
@@ -44,16 +44,19 @@ class BootimgPcbiosPlugin(SourcePlugin):
name = 'bootimg-pcbios'
@classmethod
- def _get_bootimg_dir(cls, bootimg_dir, dirname):
+ def _get_syslinux_dir(cls, bootimg_dir):
"""
- Check if dirname exists in default bootimg_dir or
- in wic-tools STAGING_DIR.
+ Get path to syslinux from either default bootimg_dir
+ or wic-tools STAGING_DIR.
"""
- for result in (bootimg_dir, get_bitbake_var("STAGING_DATADIR", "wic-tools")):
- if os.path.exists("%s/%s" % (result, dirname)):
- return result
+ for path in (bootimg_dir, get_bitbake_var("STAGING_DATADIR", "wic-tools")):
+ if not path:
+ continue
+ syslinux_dir = os.path.join(path, 'syslinux')
+ if os.path.exists(syslinux_dir):
+ return syslinux_dir
- raise WicError("Couldn't find correct bootimg_dir, exiting")
+ raise WicError("Couldn't find syslinux directory, exiting")
@classmethod
def do_install_disk(cls, disk, disk_name, creator, workdir, oe_builddir,
@@ -62,12 +65,11 @@ class BootimgPcbiosPlugin(SourcePlugin):
Called after all partitions have been prepared and assembled into a
disk image. In this case, we install the MBR.
"""
- bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux')
- mbrfile = "%s/syslinux/" % bootimg_dir
+ syslinux_dir = cls._get_syslinux_dir(bootimg_dir)
if creator.ptable_format == 'msdos':
- mbrfile += "mbr.bin"
+ mbrfile = os.path.join(syslinux_dir, "mbr.bin")
elif creator.ptable_format == 'gpt':
- mbrfile += "gptmbr.bin"
+ mbrfile = os.path.join(syslinux_dir, "gptmbr.bin")
else:
raise WicError("Unsupported partition table: %s" %
creator.ptable_format)
@@ -81,8 +83,10 @@ class BootimgPcbiosPlugin(SourcePlugin):
logger.debug("Installing MBR on disk %s as %s with size %s bytes",
disk_name, full_path, disk.min_size)
- dd_cmd = "dd if=%s of=%s conv=notrunc" % (mbrfile, full_path)
- exec_cmd(dd_cmd, native_sysroot)
+ rcode = runner.show(['dd', 'if=%s' % mbrfile,
+ 'of=%s' % full_path, 'conv=notrunc'])
+ if rcode != 0:
+ raise WicError("Unable to set MBR to %s" % full_path)
@classmethod
def do_configure_partition(cls, part, source_params, creator, cr_workdir,
@@ -151,7 +155,7 @@ class BootimgPcbiosPlugin(SourcePlugin):
'prepares' the partition to be incorporated into the image.
In this case, prepare content for legacy bios boot partition.
"""
- bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux')
+ syslinux_dir = cls._get_syslinux_dir(bootimg_dir)
staging_kernel_dir = kernel_dir
@@ -159,14 +163,14 @@ class BootimgPcbiosPlugin(SourcePlugin):
cmds = ("install -m 0644 %s/bzImage %s/vmlinuz" %
(staging_kernel_dir, hdddir),
- "install -m 444 %s/syslinux/ldlinux.sys %s/ldlinux.sys" %
- (bootimg_dir, hdddir),
- "install -m 0644 %s/syslinux/vesamenu.c32 %s/vesamenu.c32" %
- (bootimg_dir, hdddir),
- "install -m 444 %s/syslinux/libcom32.c32 %s/libcom32.c32" %
- (bootimg_dir, hdddir),
- "install -m 444 %s/syslinux/libutil.c32 %s/libutil.c32" %
- (bootimg_dir, hdddir))
+ "install -m 444 %s/ldlinux.sys %s/ldlinux.sys" %
+ (syslinux_dir, hdddir),
+ "install -m 0644 %s/vesamenu.c32 %s/vesamenu.c32" %
+ (syslinux_dir, hdddir),
+ "install -m 444 %s/libcom32.c32 %s/libcom32.c32" %
+ (syslinux_dir, hdddir),
+ "install -m 444 %s/libutil.c32 %s/libutil.c32" %
+ (syslinux_dir, hdddir))
for install_cmd in cmds:
exec_cmd(install_cmd)
diff --git a/scripts/lib/wic/plugins/source/fsimage.py b/scripts/lib/wic/plugins/source/fsimage.py
new file mode 100644
index 0000000..f781499
--- /dev/null
+++ b/scripts/lib/wic/plugins/source/fsimage.py
@@ -0,0 +1,56 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+
+import logging
+import os
+
+from wic import WicError
+from wic.pluginbase import SourcePlugin
+from wic.utils.misc import get_bitbake_var
+
+logger = logging.getLogger('wic')
+
+class FSImagePlugin(SourcePlugin):
+ """
+ Add an already existing filesystem image to the partition layout.
+ """
+
+ name = 'fsimage'
+
+ @classmethod
+ def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
+ oe_builddir, bootimg_dir, kernel_dir,
+ rootfs_dir, native_sysroot):
+ """
+ Called to do the actual content population for a partition i.e. it
+ 'prepares' the partition to be incorporated into the image.
+ """
+ if not bootimg_dir:
+ bootimg_dir = get_bitbake_var("DEPLOY_DIR_IMAGE")
+ if not bootimg_dir:
+ raise WicError("Couldn't find DEPLOY_DIR_IMAGE, exiting")
+
+ logger.debug('Bootimg dir: %s', bootimg_dir)
+
+ if 'file' not in source_params:
+ raise WicError("No file specified")
+
+ src = os.path.join(bootimg_dir, source_params['file'])
+
+
+ logger.debug('Preparing partition using image %s', src)
+ part.prepare_rootfs_from_fs_image(cr_workdir, src, "")
diff --git a/scripts/lib/wic/utils/misc.py b/scripts/lib/wic/utils/misc.py
index 4609984..c941112 100644
--- a/scripts/lib/wic/utils/misc.py
+++ b/scripts/lib/wic/utils/misc.py
@@ -59,7 +59,7 @@ NATIVE_RECIPES = {"bmaptool": "bmap-tools",
"syslinux": "syslinux"
}
-def _exec_cmd(cmd_and_args, as_shell=False):
+def _exec_cmd(cmd_and_args, as_shell=False, catch=3):
"""
Execute command, catching stderr, stdout
@@ -70,9 +70,9 @@ def _exec_cmd(cmd_and_args, as_shell=False):
logger.debug(args)
if as_shell:
- ret, out = runner.runtool(cmd_and_args)
+ ret, out = runner.runtool(cmd_and_args, catch)
else:
- ret, out = runner.runtool(args)
+ ret, out = runner.runtool(args, catch)
out = out.strip()
if ret != 0:
raise WicError("_exec_cmd: %s returned '%s' instead of 0\noutput: %s" % \
@@ -84,14 +84,14 @@ def _exec_cmd(cmd_and_args, as_shell=False):
return ret, out
-def exec_cmd(cmd_and_args, as_shell=False):
+def exec_cmd(cmd_and_args, as_shell=False, catch=3):
"""
Execute command, return output
"""
- return _exec_cmd(cmd_and_args, as_shell)[1]
+ return _exec_cmd(cmd_and_args, as_shell, catch)[1]
-def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
+def exec_native_cmd(cmd_and_args, native_sysroot, catch=3, pseudo=""):
"""
Execute native command, catching stderr, stdout
@@ -118,7 +118,7 @@ def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
# If the command isn't in the native sysroot say we failed.
if spawn.find_executable(args[0], native_paths):
- ret, out = _exec_cmd(native_cmd_and_args, True)
+ ret, out = _exec_cmd(native_cmd_and_args, True, catch)
else:
ret = 127
out = "can't find native executable %s in %s" % (args[0], native_paths)
@@ -131,7 +131,7 @@ def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
"was not found (see details above).\n\n" % prog
recipe = NATIVE_RECIPES.get(prog)
if recipe:
- msg += "Please make sure wic-tools have %s-native in its DEPENDS, bake it with 'bitbake wic-tools' "\
+ msg += "Please bake it with 'bitbake %s-native' "\
"and try again.\n" % recipe
else:
msg += "Wic failed to find a recipe to build native %s. Please "\
diff --git a/scripts/lib/wic/utils/runner.py b/scripts/lib/wic/utils/runner.py
index 4aa00fb..56d7ea3 100644
--- a/scripts/lib/wic/utils/runner.py
+++ b/scripts/lib/wic/utils/runner.py
@@ -14,17 +14,32 @@
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc., 59
# Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+
+import logging
+import os
import subprocess
from wic import WicError
-def runtool(cmdln_or_args):
+logger = logging.getLogger('wic')
+
+def runtool(cmdln_or_args, catch=1):
""" wrapper for most of the subprocess calls
input:
cmdln_or_args: can be both args and cmdln str (shell=True)
+ catch: 0, quitely run
+ 1, only STDOUT
+ 2, only STDERR
+ 3, both STDOUT and STDERR
return:
- rc, output
+ (rc, output)
+ if catch==0: the output will always None
"""
+
+ if catch not in (0, 1, 2, 3):
+ # invalid catch selection, will cause exception, that's good
+ return None
+
if isinstance(cmdln_or_args, list):
cmd = cmdln_or_args[0]
shell = False
@@ -33,13 +48,26 @@ def runtool(cmdln_or_args):
cmd = shlex.split(cmdln_or_args)[0]
shell = True
- sout = subprocess.PIPE
- serr = subprocess.STDOUT
+ if catch != 3:
+ dev_null = os.open("/dev/null", os.O_WRONLY)
+
+ if catch == 0:
+ sout = dev_null
+ serr = dev_null
+ elif catch == 1:
+ sout = subprocess.PIPE
+ serr = dev_null
+ elif catch == 2:
+ sout = dev_null
+ serr = subprocess.PIPE
+ elif catch == 3:
+ sout = subprocess.PIPE
+ serr = subprocess.STDOUT
try:
process = subprocess.Popen(cmdln_or_args, stdout=sout,
stderr=serr, shell=shell)
- sout, serr = process.communicate()
+ (sout, serr) = process.communicate()
# combine stdout and stderr, filter None out and decode
out = ''.join([out.decode('utf-8') for out in [sout, serr] if out])
except OSError as err:
@@ -48,5 +76,39 @@ def runtool(cmdln_or_args):
raise WicError('Cannot run command: %s, lost dependency?' % cmd)
else:
raise # relay
+ finally:
+ if catch != 3:
+ os.close(dev_null)
+
+ return (process.returncode, out)
+
+def show(cmdln_or_args):
+ """Show all messages using logger.debug."""
+
+ rcode, out = runtool(cmdln_or_args, catch=3)
+
+ if isinstance(cmdln_or_args, list):
+ cmd = ' '.join(cmdln_or_args)
+ else:
+ cmd = cmdln_or_args
+
+ msg = 'running command: "%s"' % cmd
+ if out:
+ out = out.strip()
+ if out:
+ msg += ', with output::'
+ msg += '\n +----------------'
+ for line in out.splitlines():
+ msg += '\n | %s' % line
+ msg += '\n +----------------'
+
+ logger.debug(msg)
+
+ return rcode
+
+def outs(cmdln_or_args, catch=1):
+ # get the outputs of tools
+ return runtool(cmdln_or_args, catch)[1].strip()
- return process.returncode, out
+def quiet(cmdln_or_args):
+ return runtool(cmdln_or_args, catch=0)[0]
--
2.13.6
^ permalink raw reply [flat|nested] 51+ messages in thread* Re: [PATCH 9/9] wic: now truly go for the wic version we claim to have
2018-01-31 9:42 ` [PATCH 9/9] wic: now truly go for the wic version we claim to have Henning Schild
@ 2018-01-31 10:11 ` Alexander Smirnov
2018-01-31 10:55 ` Jan Kiszka
0 siblings, 1 reply; 51+ messages in thread
From: Alexander Smirnov @ 2018-01-31 10:11 UTC (permalink / raw)
To: Henning Schild, isar-users
On 01/31/2018 12:42 PM, Henning Schild wrote:
> 2b164b18fd639c9 claims to introduce wic hash 131629ca6238ea05
> This commit really carries that version of wic.
>
> Issue:
> - the wic version in Isar was modified
> - that causes:
> - confusion, maintainability and updateability issues
> - potential quality issues
>
> Impact:
> This patch and the previous reverts get wic back to a state where all
> these Issues are solved. We could now just update our wic without having
> to worry about local patches. In case of a wic-update Isar and layers
> on top would still have to review their plugins.
Does upstream wic script rely on the content from lib/plugins?
For example some recent commit updates both:
https://github.com/openembedded/openembedded-core/commit/00420ec42140c1b752132bda190dede85756d157#diff-df4a70bc146d3159891d7a410f2521a1
So due to customized plugins below we should worry and keep this in mind.
Alex
>
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
> scripts/lib/wic/canned-wks/qemux86-directdisk.wks | 2 +-
> scripts/lib/wic/filemap.py | 6 +-
> scripts/lib/wic/help.py | 2 -
> scripts/lib/wic/ksparser.py | 4 +-
> scripts/lib/wic/partition.py | 114 ++++++++++++++--------
> scripts/lib/wic/plugins/imager/direct.py | 17 +++-
> scripts/lib/wic/plugins/source/bootimg-pcbios.py | 48 ++++-----
> scripts/lib/wic/plugins/source/fsimage.py | 56 +++++++++++
> scripts/lib/wic/utils/misc.py | 16 +--
> scripts/lib/wic/utils/runner.py | 74 ++++++++++++--
> 10 files changed, 250 insertions(+), 89 deletions(-)
> create mode 100644 scripts/lib/wic/plugins/source/fsimage.py
>
> diff --git a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks
> index db30bbc..a6518a0 100644
> --- a/scripts/lib/wic/canned-wks/qemux86-directdisk.wks
> +++ b/scripts/lib/wic/canned-wks/qemux86-directdisk.wks
> @@ -4,5 +4,5 @@
>
> include common.wks.inc
>
> -bootloader --timeout=0 --append="vga=0 uvesafb.mode_option=640x480-32 root=/dev/sda2 rw mem=256M ip=192.168.7.2::192.168.7.1:255.255.255.0 oprofile.timer=1 rootfstype=ext4 "
> +bootloader --timeout=0 --append="vga=0 uvesafb.mode_option=640x480-32 root=/dev/vda2 rw mem=256M ip=192.168.7.2::192.168.7.1:255.255.255.0 oprofile.timer=1 rootfstype=ext4 "
>
> diff --git a/scripts/lib/wic/filemap.py b/scripts/lib/wic/filemap.py
> index 1f1aacc..080668e 100644
> --- a/scripts/lib/wic/filemap.py
> +++ b/scripts/lib/wic/filemap.py
> @@ -530,11 +530,9 @@ def filemap(image, log=None):
> except ErrorNotSupp:
> return FilemapSeek(image, log)
>
> -def sparse_copy(src_fname, dst_fname, offset=0, skip=0, api=None):
> +def sparse_copy(src_fname, dst_fname, offset=0, skip=0):
> """Efficiently copy sparse file to or into another file."""
> - if not api:
> - api = filemap
> - fmap = api(src_fname)
> + fmap = filemap(src_fname)
> try:
> dst_file = open(dst_fname, 'r+b')
> except IOError:
> diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
> index aee2451..148da89 100644
> --- a/scripts/lib/wic/help.py
> +++ b/scripts/lib/wic/help.py
> @@ -687,8 +687,6 @@ DESCRIPTION
> apply to partitions created using '--source rootfs' (see
> --source above). Valid values are:
>
> - vfat
> - msdos
> ext2
> ext3
> ext4
> diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> index d026caa..a039300 100644
> --- a/scripts/lib/wic/ksparser.py
> +++ b/scripts/lib/wic/ksparser.py
> @@ -136,9 +136,7 @@ class KickStart():
> part.add_argument('--exclude-path', nargs='+')
> part.add_argument("--extra-space", type=sizetype)
> part.add_argument('--fsoptions', dest='fsopts')
> - part.add_argument('--fstype', default='vfat',
> - choices=('ext2', 'ext3', 'ext4', 'btrfs',
> - 'squashfs', 'vfat', 'msdos', 'swap'))
> + part.add_argument('--fstype')
> part.add_argument('--label')
> part.add_argument('--no-table', action='store_true')
> part.add_argument('--ondisk', '--ondrive', dest='disk', default='sda')
> diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
> index 939e667..8e32afc 100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -136,24 +136,22 @@ class Partition():
> "specify a non-zero --size/--fixed-size for that "
> "partition." % self.mountpoint)
>
> - if self.fstype == "swap":
> + if self.fstype and self.fstype == "swap":
> self.prepare_swap_partition(cr_workdir, oe_builddir,
> native_sysroot)
> self.source_file = "%s/fs.%s" % (cr_workdir, self.fstype)
> - else:
> - if self.fstype == 'squashfs':
> - raise WicError("It's not possible to create empty squashfs "
> - "partition '%s'" % (self.mountpoint))
> -
> + elif self.fstype:
> rootfs = "%s/fs_%s.%s.%s" % (cr_workdir, self.label,
> self.lineno, self.fstype)
> if os.path.isfile(rootfs):
> os.remove(rootfs)
> -
> - prefix = "ext" if self.fstype.startswith("ext") else self.fstype
> - method = getattr(self, "prepare_empty_partition_" + prefix)
> - method(rootfs, oe_builddir, native_sysroot)
> - self.source_file = rootfs
> + for prefix in ("ext", "btrfs", "vfat", "squashfs"):
> + if self.fstype.startswith(prefix):
> + method = getattr(self,
> + "prepare_empty_partition_" + prefix)
> + method(rootfs, oe_builddir, native_sysroot)
> + self.source_file = rootfs
> + break
> return
>
> plugins = PluginMgr.get_plugins('source')
> @@ -195,6 +193,19 @@ class Partition():
> "larger (%d kB) than its allowed size %d kB" %
> (self.mountpoint, self.size, self.fixed_size))
>
> + def prepare_rootfs_from_fs_image(self, cr_workdir, oe_builddir,
> + rootfs_dir):
> + """
> + Handle an already-created partition e.g. xxx.ext3
> + """
> + rootfs = oe_builddir
> + du_cmd = "du -Lbks %s" % rootfs
> + out = exec_cmd(du_cmd)
> + rootfs_size = out.split()[0]
> +
> + self.size = int(rootfs_size)
> + self.source_file = rootfs
> +
> def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
> native_sysroot):
> """
> @@ -219,6 +230,10 @@ class Partition():
> if os.path.isfile(rootfs):
> os.remove(rootfs)
>
> + if not self.fstype:
> + raise WicError("File system for partition %s not specified in "
> + "kickstart, use --fstype option" % self.mountpoint)
> +
> # Get rootfs size from bitbake variable if it's not set in .ks file
> if not self.size:
> # Bitbake variable ROOTFS_SIZE is calculated in
> @@ -233,15 +248,19 @@ class Partition():
> '--overhead-factor will be applied')
> self.size = int(round(float(rsize_bb)))
>
> - prefix = "ext" if self.fstype.startswith("ext") else self.fstype
> - method = getattr(self, "prepare_rootfs_" + prefix)
> - method(rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo)
> - self.source_file = rootfs
> + for prefix in ("ext", "btrfs", "vfat", "squashfs"):
> + if self.fstype.startswith(prefix):
> + method = getattr(self, "prepare_rootfs_" + prefix)
> + method(rootfs, oe_builddir, rootfs_dir, native_sysroot, pseudo)
>
> - # get the rootfs size in the right units for kickstart (kB)
> - du_cmd = "du -Lbks %s" % rootfs
> - out = exec_cmd(du_cmd)
> - self.size = int(out.split()[0])
> + self.source_file = rootfs
> +
> + # get the rootfs size in the right units for kickstart (kB)
> + du_cmd = "du -Lbks %s" % rootfs
> + out = exec_cmd(du_cmd)
> + self.size = int(out.split()[0])
> +
> + break
>
> def prepare_rootfs_ext(self, rootfs, oe_builddir, rootfs_dir,
> native_sysroot, pseudo):
> @@ -267,7 +286,7 @@ class Partition():
> (self.fstype, extra_imagecmd, rootfs, label_str, rootfs_dir)
> exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
>
> - mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs)
> + mkfs_cmd = "fsck.%s -fy %s" % (self.fstype, rootfs)
> exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
>
> def prepare_rootfs_btrfs(self, rootfs, oe_builddir, rootfs_dir,
> @@ -294,10 +313,10 @@ class Partition():
> (self.fstype, rootfs_size * 1024, rootfs_dir, label_str, rootfs)
> exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
>
> - def prepare_rootfs_msdos(self, rootfs, oe_builddir, rootfs_dir,
> - native_sysroot, pseudo):
> + def prepare_rootfs_vfat(self, rootfs, oe_builddir, rootfs_dir,
> + native_sysroot, pseudo):
> """
> - Prepare content for a msdos/vfat rootfs partition.
> + Prepare content for a vfat rootfs partition.
> """
> du_cmd = "du -bks %s" % rootfs_dir
> out = exec_cmd(du_cmd)
> @@ -309,12 +328,7 @@ class Partition():
> if self.label:
> label_str = "-n %s" % self.label
>
> - size_str = ""
> - if self.fstype == 'msdos':
> - size_str = "-F 16" # FAT 16
> -
> - dosfs_cmd = "mkdosfs %s -S 512 %s -C %s %d" % (label_str, size_str,
> - rootfs, rootfs_size)
> + dosfs_cmd = "mkdosfs %s -S 512 -C %s %d" % (label_str, rootfs, rootfs_size)
> exec_native_cmd(dosfs_cmd, native_sysroot)
>
> mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir)
> @@ -323,8 +337,6 @@ class Partition():
> chmod_cmd = "chmod 644 %s" % rootfs
> exec_cmd(chmod_cmd)
>
> - prepare_rootfs_vfat = prepare_rootfs_msdos
> -
> def prepare_rootfs_squashfs(self, rootfs, oe_builddir, rootfs_dir,
> native_sysroot, pseudo):
> """
> @@ -370,8 +382,8 @@ class Partition():
> (self.fstype, self.size * 1024, label_str, rootfs)
> exec_native_cmd(mkfs_cmd, native_sysroot)
>
> - def prepare_empty_partition_msdos(self, rootfs, oe_builddir,
> - native_sysroot):
> + def prepare_empty_partition_vfat(self, rootfs, oe_builddir,
> + native_sysroot):
> """
> Prepare an empty vfat partition.
> """
> @@ -381,18 +393,40 @@ class Partition():
> if self.label:
> label_str = "-n %s" % self.label
>
> - size_str = ""
> - if self.fstype == 'msdos':
> - size_str = "-F 16" # FAT 16
> -
> - dosfs_cmd = "mkdosfs %s -S 512 %s -C %s %d" % (label_str, size_str,
> - rootfs, blocks)
> + dosfs_cmd = "mkdosfs %s -S 512 -C %s %d" % (label_str, rootfs, blocks)
> exec_native_cmd(dosfs_cmd, native_sysroot)
>
> chmod_cmd = "chmod 644 %s" % rootfs
> exec_cmd(chmod_cmd)
>
> - prepare_empty_partition_vfat = prepare_empty_partition_msdos
> + def prepare_empty_partition_squashfs(self, cr_workdir, oe_builddir,
> + native_sysroot):
> + """
> + Prepare an empty squashfs partition.
> + """
> + logger.warning("Creating of an empty squashfs %s partition was attempted. "
> + "Proceeding as requested.", self.mountpoint)
> +
> + path = "%s/fs_%s.%s" % (cr_workdir, self.label, self.fstype)
> + if os.path.isfile(path):
> + os.remove(path)
> +
> + # it is not possible to create a squashfs without source data,
> + # thus prepare an empty temp dir that is used as source
> + tmpdir = tempfile.mkdtemp()
> +
> + squashfs_cmd = "mksquashfs %s %s -noappend" % \
> + (tmpdir, path)
> + exec_native_cmd(squashfs_cmd, native_sysroot)
> +
> + os.rmdir(tmpdir)
> +
> + # get the rootfs size in the right units for kickstart (kB)
> + du_cmd = "du -Lbks %s" % path
> + out = exec_cmd(du_cmd)
> + fs_size = out.split()[0]
> +
> + self.size = int(fs_size)
>
> def prepare_swap_partition(self, cr_workdir, oe_builddir, native_sysroot):
> """
> diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
> index f2e6127..7d38ab3 100644
> --- a/scripts/lib/wic/plugins/imager/direct.py
> +++ b/scripts/lib/wic/plugins/imager/direct.py
> @@ -323,7 +323,7 @@ class PartitionedImage():
> if self.ptable_format == 'gpt':
> part.uuid = str(uuid.uuid4())
> else: # msdos partition table
> - part.uuid = '%08x-%02d' % (self.identifier, part.realnum)
> + part.uuid = '%0x-%02d' % (self.identifier, part.realnum)
>
> def prepare(self, imager):
> """Prepare an image. Call prepare method of all image partitions."""
> @@ -487,8 +487,8 @@ class PartitionedImage():
> parted_fs_type = "fat32"
> elif part.fstype == "msdos":
> parted_fs_type = "fat16"
> - if not part.system_id:
> - part.system_id = '0x6' # FAT16
> + elif part.fstype == "ontrackdm6aux3":
> + parted_fs_type = "ontrackdm6aux3"
> else:
> # Type for ext2/ext3/ext4/btrfs
> parted_fs_type = "ext2"
> @@ -538,6 +538,17 @@ class PartitionedImage():
> (self.path, part.num, part.system_id),
> self.native_sysroot)
>
> + # Parted defaults to enabling the lba flag for fat16 partitions,
> + # which causes compatibility issues with some firmware (and really
> + # isn't necessary).
> + if parted_fs_type == "fat16":
> + if self.ptable_format == 'msdos':
> + logger.debug("Disable 'lba' flag for partition '%s' on disk '%s'",
> + part.num, self.path)
> + exec_native_cmd("parted -s %s set %d lba off" % \
> + (self.path, part.num),
> + self.native_sysroot)
> +
> def cleanup(self):
> # remove partition images
> for image in set(self.partimages):
> diff --git a/scripts/lib/wic/plugins/source/bootimg-pcbios.py b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
> index 5890c12..11db304 100644
> --- a/scripts/lib/wic/plugins/source/bootimg-pcbios.py
> +++ b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
> @@ -44,16 +44,19 @@ class BootimgPcbiosPlugin(SourcePlugin):
> name = 'bootimg-pcbios'
>
> @classmethod
> - def _get_bootimg_dir(cls, bootimg_dir, dirname):
> + def _get_syslinux_dir(cls, bootimg_dir):
> """
> - Check if dirname exists in default bootimg_dir or
> - in wic-tools STAGING_DIR.
> + Get path to syslinux from either default bootimg_dir
> + or wic-tools STAGING_DIR.
> """
> - for result in (bootimg_dir, get_bitbake_var("STAGING_DATADIR", "wic-tools")):
> - if os.path.exists("%s/%s" % (result, dirname)):
> - return result
> + for path in (bootimg_dir, get_bitbake_var("STAGING_DATADIR", "wic-tools")):
> + if not path:
> + continue
> + syslinux_dir = os.path.join(path, 'syslinux')
> + if os.path.exists(syslinux_dir):
> + return syslinux_dir
>
> - raise WicError("Couldn't find correct bootimg_dir, exiting")
> + raise WicError("Couldn't find syslinux directory, exiting")
>
> @classmethod
> def do_install_disk(cls, disk, disk_name, creator, workdir, oe_builddir,
> @@ -62,12 +65,11 @@ class BootimgPcbiosPlugin(SourcePlugin):
> Called after all partitions have been prepared and assembled into a
> disk image. In this case, we install the MBR.
> """
> - bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux')
> - mbrfile = "%s/syslinux/" % bootimg_dir
> + syslinux_dir = cls._get_syslinux_dir(bootimg_dir)
> if creator.ptable_format == 'msdos':
> - mbrfile += "mbr.bin"
> + mbrfile = os.path.join(syslinux_dir, "mbr.bin")
> elif creator.ptable_format == 'gpt':
> - mbrfile += "gptmbr.bin"
> + mbrfile = os.path.join(syslinux_dir, "gptmbr.bin")
> else:
> raise WicError("Unsupported partition table: %s" %
> creator.ptable_format)
> @@ -81,8 +83,10 @@ class BootimgPcbiosPlugin(SourcePlugin):
> logger.debug("Installing MBR on disk %s as %s with size %s bytes",
> disk_name, full_path, disk.min_size)
>
> - dd_cmd = "dd if=%s of=%s conv=notrunc" % (mbrfile, full_path)
> - exec_cmd(dd_cmd, native_sysroot)
> + rcode = runner.show(['dd', 'if=%s' % mbrfile,
> + 'of=%s' % full_path, 'conv=notrunc'])
> + if rcode != 0:
> + raise WicError("Unable to set MBR to %s" % full_path)
>
> @classmethod
> def do_configure_partition(cls, part, source_params, creator, cr_workdir,
> @@ -151,7 +155,7 @@ class BootimgPcbiosPlugin(SourcePlugin):
> 'prepares' the partition to be incorporated into the image.
> In this case, prepare content for legacy bios boot partition.
> """
> - bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux')
> + syslinux_dir = cls._get_syslinux_dir(bootimg_dir)
>
> staging_kernel_dir = kernel_dir
>
> @@ -159,14 +163,14 @@ class BootimgPcbiosPlugin(SourcePlugin):
>
> cmds = ("install -m 0644 %s/bzImage %s/vmlinuz" %
> (staging_kernel_dir, hdddir),
> - "install -m 444 %s/syslinux/ldlinux.sys %s/ldlinux.sys" %
> - (bootimg_dir, hdddir),
> - "install -m 0644 %s/syslinux/vesamenu.c32 %s/vesamenu.c32" %
> - (bootimg_dir, hdddir),
> - "install -m 444 %s/syslinux/libcom32.c32 %s/libcom32.c32" %
> - (bootimg_dir, hdddir),
> - "install -m 444 %s/syslinux/libutil.c32 %s/libutil.c32" %
> - (bootimg_dir, hdddir))
> + "install -m 444 %s/ldlinux.sys %s/ldlinux.sys" %
> + (syslinux_dir, hdddir),
> + "install -m 0644 %s/vesamenu.c32 %s/vesamenu.c32" %
> + (syslinux_dir, hdddir),
> + "install -m 444 %s/libcom32.c32 %s/libcom32.c32" %
> + (syslinux_dir, hdddir),
> + "install -m 444 %s/libutil.c32 %s/libutil.c32" %
> + (syslinux_dir, hdddir))
>
> for install_cmd in cmds:
> exec_cmd(install_cmd)
> diff --git a/scripts/lib/wic/plugins/source/fsimage.py b/scripts/lib/wic/plugins/source/fsimage.py
> new file mode 100644
> index 0000000..f781499
> --- /dev/null
> +++ b/scripts/lib/wic/plugins/source/fsimage.py
> @@ -0,0 +1,56 @@
> +# ex:ts=4:sw=4:sts=4:et
> +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License version 2 as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +#
> +
> +import logging
> +import os
> +
> +from wic import WicError
> +from wic.pluginbase import SourcePlugin
> +from wic.utils.misc import get_bitbake_var
> +
> +logger = logging.getLogger('wic')
> +
> +class FSImagePlugin(SourcePlugin):
> + """
> + Add an already existing filesystem image to the partition layout.
> + """
> +
> + name = 'fsimage'
> +
> + @classmethod
> + def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
> + oe_builddir, bootimg_dir, kernel_dir,
> + rootfs_dir, native_sysroot):
> + """
> + Called to do the actual content population for a partition i.e. it
> + 'prepares' the partition to be incorporated into the image.
> + """
> + if not bootimg_dir:
> + bootimg_dir = get_bitbake_var("DEPLOY_DIR_IMAGE")
> + if not bootimg_dir:
> + raise WicError("Couldn't find DEPLOY_DIR_IMAGE, exiting")
> +
> + logger.debug('Bootimg dir: %s', bootimg_dir)
> +
> + if 'file' not in source_params:
> + raise WicError("No file specified")
> +
> + src = os.path.join(bootimg_dir, source_params['file'])
> +
> +
> + logger.debug('Preparing partition using image %s', src)
> + part.prepare_rootfs_from_fs_image(cr_workdir, src, "")
> diff --git a/scripts/lib/wic/utils/misc.py b/scripts/lib/wic/utils/misc.py
> index 4609984..c941112 100644
> --- a/scripts/lib/wic/utils/misc.py
> +++ b/scripts/lib/wic/utils/misc.py
> @@ -59,7 +59,7 @@ NATIVE_RECIPES = {"bmaptool": "bmap-tools",
> "syslinux": "syslinux"
> }
>
> -def _exec_cmd(cmd_and_args, as_shell=False):
> +def _exec_cmd(cmd_and_args, as_shell=False, catch=3):
> """
> Execute command, catching stderr, stdout
>
> @@ -70,9 +70,9 @@ def _exec_cmd(cmd_and_args, as_shell=False):
> logger.debug(args)
>
> if as_shell:
> - ret, out = runner.runtool(cmd_and_args)
> + ret, out = runner.runtool(cmd_and_args, catch)
> else:
> - ret, out = runner.runtool(args)
> + ret, out = runner.runtool(args, catch)
> out = out.strip()
> if ret != 0:
> raise WicError("_exec_cmd: %s returned '%s' instead of 0\noutput: %s" % \
> @@ -84,14 +84,14 @@ def _exec_cmd(cmd_and_args, as_shell=False):
> return ret, out
>
>
> -def exec_cmd(cmd_and_args, as_shell=False):
> +def exec_cmd(cmd_and_args, as_shell=False, catch=3):
> """
> Execute command, return output
> """
> - return _exec_cmd(cmd_and_args, as_shell)[1]
> + return _exec_cmd(cmd_and_args, as_shell, catch)[1]
>
>
> -def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
> +def exec_native_cmd(cmd_and_args, native_sysroot, catch=3, pseudo=""):
> """
> Execute native command, catching stderr, stdout
>
> @@ -118,7 +118,7 @@ def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
>
> # If the command isn't in the native sysroot say we failed.
> if spawn.find_executable(args[0], native_paths):
> - ret, out = _exec_cmd(native_cmd_and_args, True)
> + ret, out = _exec_cmd(native_cmd_and_args, True, catch)
> else:
> ret = 127
> out = "can't find native executable %s in %s" % (args[0], native_paths)
> @@ -131,7 +131,7 @@ def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
> "was not found (see details above).\n\n" % prog
> recipe = NATIVE_RECIPES.get(prog)
> if recipe:
> - msg += "Please make sure wic-tools have %s-native in its DEPENDS, bake it with 'bitbake wic-tools' "\
> + msg += "Please bake it with 'bitbake %s-native' "\
> "and try again.\n" % recipe
> else:
> msg += "Wic failed to find a recipe to build native %s. Please "\
> diff --git a/scripts/lib/wic/utils/runner.py b/scripts/lib/wic/utils/runner.py
> index 4aa00fb..56d7ea3 100644
> --- a/scripts/lib/wic/utils/runner.py
> +++ b/scripts/lib/wic/utils/runner.py
> @@ -14,17 +14,32 @@
> # You should have received a copy of the GNU General Public License along
> # with this program; if not, write to the Free Software Foundation, Inc., 59
> # Temple Place - Suite 330, Boston, MA 02111-1307, USA.
> +
> +import logging
> +import os
> import subprocess
>
> from wic import WicError
>
> -def runtool(cmdln_or_args):
> +logger = logging.getLogger('wic')
> +
> +def runtool(cmdln_or_args, catch=1):
> """ wrapper for most of the subprocess calls
> input:
> cmdln_or_args: can be both args and cmdln str (shell=True)
> + catch: 0, quitely run
> + 1, only STDOUT
> + 2, only STDERR
> + 3, both STDOUT and STDERR
> return:
> - rc, output
> + (rc, output)
> + if catch==0: the output will always None
> """
> +
> + if catch not in (0, 1, 2, 3):
> + # invalid catch selection, will cause exception, that's good
> + return None
> +
> if isinstance(cmdln_or_args, list):
> cmd = cmdln_or_args[0]
> shell = False
> @@ -33,13 +48,26 @@ def runtool(cmdln_or_args):
> cmd = shlex.split(cmdln_or_args)[0]
> shell = True
>
> - sout = subprocess.PIPE
> - serr = subprocess.STDOUT
> + if catch != 3:
> + dev_null = os.open("/dev/null", os.O_WRONLY)
> +
> + if catch == 0:
> + sout = dev_null
> + serr = dev_null
> + elif catch == 1:
> + sout = subprocess.PIPE
> + serr = dev_null
> + elif catch == 2:
> + sout = dev_null
> + serr = subprocess.PIPE
> + elif catch == 3:
> + sout = subprocess.PIPE
> + serr = subprocess.STDOUT
>
> try:
> process = subprocess.Popen(cmdln_or_args, stdout=sout,
> stderr=serr, shell=shell)
> - sout, serr = process.communicate()
> + (sout, serr) = process.communicate()
> # combine stdout and stderr, filter None out and decode
> out = ''.join([out.decode('utf-8') for out in [sout, serr] if out])
> except OSError as err:
> @@ -48,5 +76,39 @@ def runtool(cmdln_or_args):
> raise WicError('Cannot run command: %s, lost dependency?' % cmd)
> else:
> raise # relay
> + finally:
> + if catch != 3:
> + os.close(dev_null)
> +
> + return (process.returncode, out)
> +
> +def show(cmdln_or_args):
> + """Show all messages using logger.debug."""
> +
> + rcode, out = runtool(cmdln_or_args, catch=3)
> +
> + if isinstance(cmdln_or_args, list):
> + cmd = ' '.join(cmdln_or_args)
> + else:
> + cmd = cmdln_or_args
> +
> + msg = 'running command: "%s"' % cmd
> + if out:
> + out = out.strip()
> + if out:
> + msg += ', with output::'
> + msg += '\n +----------------'
> + for line in out.splitlines():
> + msg += '\n | %s' % line
> + msg += '\n +----------------'
> +
> + logger.debug(msg)
> +
> + return rcode
> +
> +def outs(cmdln_or_args, catch=1):
> + # get the outputs of tools
> + return runtool(cmdln_or_args, catch)[1].strip()
>
> - return process.returncode, out
> +def quiet(cmdln_or_args):
> + return runtool(cmdln_or_args, catch=0)[0]
>
--
With best regards,
Alexander Smirnov
ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 51+ messages in thread* Re: [PATCH 9/9] wic: now truly go for the wic version we claim to have
2018-01-31 10:11 ` Alexander Smirnov
@ 2018-01-31 10:55 ` Jan Kiszka
2018-01-31 11:11 ` Alexander Smirnov
0 siblings, 1 reply; 51+ messages in thread
From: Jan Kiszka @ 2018-01-31 10:55 UTC (permalink / raw)
To: Alexander Smirnov, Henning Schild, isar-users
On 2018-01-31 11:11, Alexander Smirnov wrote:
> On 01/31/2018 12:42 PM, Henning Schild wrote:
>> 2b164b18fd639c9 claims to introduce wic hash 131629ca6238ea05
>> This commit really carries that version of wic.
>>
>> Issue:
>> - the wic version in Isar was modified
>> - that causes:
>> - confusion, maintainability and updateability issues
>> - potential quality issues
>>
>> Impact:
>> This patch and the previous reverts get wic back to a state where all
>> these Issues are solved. We could now just update our wic without
>> having
>> to worry about local patches. In case of a wic-update Isar and layers
>> on top would still have to review their plugins.
>
> Does upstream wic script rely on the content from lib/plugins?
>
> For example some recent commit updates both:
>
> https://github.com/openembedded/openembedded-core/commit/00420ec42140c1b752132bda190dede85756d157#diff-df4a70bc146d3159891d7a410f2521a1
>
>
> So due to customized plugins below we should worry and keep this in mind.
Sure, we have to keep an eye on wic<->plugin API changes when updating
wic to the next upstream version. Nothing new, though, when you write
plugins for wic (like we do for firmware update mechanisms, also over
Yocto).
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 9/9] wic: now truly go for the wic version we claim to have
2018-01-31 10:55 ` Jan Kiszka
@ 2018-01-31 11:11 ` Alexander Smirnov
2018-01-31 11:43 ` Jan Kiszka
0 siblings, 1 reply; 51+ messages in thread
From: Alexander Smirnov @ 2018-01-31 11:11 UTC (permalink / raw)
To: Jan Kiszka, isar-users
On 01/31/2018 01:55 PM, Jan Kiszka wrote:
> On 2018-01-31 11:11, Alexander Smirnov wrote:
>> On 01/31/2018 12:42 PM, Henning Schild wrote:
>>> 2b164b18fd639c9 claims to introduce wic hash 131629ca6238ea05
>>> This commit really carries that version of wic.
>>>
>>> Issue:
>>> - the wic version in Isar was modified
>>> - that causes:
>>> - confusion, maintainability and updateability issues
>>> - potential quality issues
>>>
>>> Impact:
>>> This patch and the previous reverts get wic back to a state where all
>>> these Issues are solved. We could now just update our wic without
>>> having
>>> to worry about local patches. In case of a wic-update Isar and layers
>>> on top would still have to review their plugins.
>>
>> Does upstream wic script rely on the content from lib/plugins?
>>
>> For example some recent commit updates both:
>>
>> https://github.com/openembedded/openembedded-core/commit/00420ec42140c1b752132bda190dede85756d157#diff-df4a70bc146d3159891d7a410f2521a1
>>
>>
>> So due to customized plugins below we should worry and keep this in mind.
>
> Sure, we have to keep an eye on wic<->plugin API changes when updating
> wic to the next upstream version. Nothing new, though, when you write
> plugins for wic (like we do for firmware update mechanisms, also over
> Yocto).
This patch claims that the wic maintenance becomes easier. What is the
idea of this claim? Wic is based on both: 'scripts/wic' and 'lib/wic',
they don't work without each other. Whole this series performs migration
of changes from 'scripts/wic' to 'lib/wic', so eventually there is no
unmodified upstream wic support, the only script stays unmodified while
plugins are patched.
So I'm a bit confused, what is the core difference with existing
approach, and how this approach fixes maintainability and updateability
issues?
Alex
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 9/9] wic: now truly go for the wic version we claim to have
2018-01-31 11:11 ` Alexander Smirnov
@ 2018-01-31 11:43 ` Jan Kiszka
2018-01-31 11:53 ` Baurzhan Ismagulov
0 siblings, 1 reply; 51+ messages in thread
From: Jan Kiszka @ 2018-01-31 11:43 UTC (permalink / raw)
To: Alexander Smirnov, isar-users
On 2018-01-31 12:11, Alexander Smirnov wrote:
> On 01/31/2018 01:55 PM, Jan Kiszka wrote:
>> On 2018-01-31 11:11, Alexander Smirnov wrote:
>>> On 01/31/2018 12:42 PM, Henning Schild wrote:
>>>> 2b164b18fd639c9 claims to introduce wic hash 131629ca6238ea05
>>>> This commit really carries that version of wic.
>>>>
>>>> Issue:
>>>> - the wic version in Isar was modified
>>>> - that causes:
>>>> - confusion, maintainability and updateability issues
>>>> - potential quality issues
>>>>
>>>> Impact:
>>>> This patch and the previous reverts get wic back to a state where
>>>> all
>>>> these Issues are solved. We could now just update our wic without
>>>> having
>>>> to worry about local patches. In case of a wic-update Isar and
>>>> layers
>>>> on top would still have to review their plugins.
>>>
>>> Does upstream wic script rely on the content from lib/plugins?
>>>
>>> For example some recent commit updates both:
>>>
>>> https://github.com/openembedded/openembedded-core/commit/00420ec42140c1b752132bda190dede85756d157#diff-df4a70bc146d3159891d7a410f2521a1
>>>
>>>
>>>
>>> So due to customized plugins below we should worry and keep this in
>>> mind.
>>
>> Sure, we have to keep an eye on wic<->plugin API changes when updating
>> wic to the next upstream version. Nothing new, though, when you write
>> plugins for wic (like we do for firmware update mechanisms, also over
>> Yocto).
>
> This patch claims that the wic maintenance becomes easier. What is the
> idea of this claim? Wic is based on both: 'scripts/wic' and 'lib/wic',
> they don't work without each other. Whole this series performs migration
> of changes from 'scripts/wic' to 'lib/wic', so eventually there is no
> unmodified upstream wic support, the only script stays unmodified while
> plugins are patched.
>
> So I'm a bit confused, what is the core difference with existing
> approach, and how this approach fixes maintainability and updateability
> issues?
Study patch 2 again, I think you misunderstood what it changed: all
adaptations are consolidated in those 100 lines now. Now longer a fork
of wic, only few external wrappings (without changing scripts or lib).
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 9/9] wic: now truly go for the wic version we claim to have
2018-01-31 11:43 ` Jan Kiszka
@ 2018-01-31 11:53 ` Baurzhan Ismagulov
2018-01-31 12:01 ` Jan Kiszka
0 siblings, 1 reply; 51+ messages in thread
From: Baurzhan Ismagulov @ 2018-01-31 11:53 UTC (permalink / raw)
To: isar-users
On Wed, Jan 31, 2018 at 12:43:24PM +0100, Jan Kiszka wrote:
> Study patch 2 again, I think you misunderstood what it changed: all
> adaptations are consolidated in those 100 lines now. Now longer a fork
> of wic, only few external wrappings (without changing scripts or lib).
We'll study the patches in more detail, but I'd still like to confirm I
understand the concept correctly. The issue is, we import script/wic and
scripts/lib/wic atomically, as a single whole. Now, patch 9/9 changes:
scripts/lib/wic/canned-wks/qemux86-directdisk.wks | 2 +-
scripts/lib/wic/filemap.py | 6 +-
scripts/lib/wic/help.py | 2 -
scripts/lib/wic/ksparser.py | 4 +-
scripts/lib/wic/partition.py | 114 ++++++++++++++--------
scripts/lib/wic/plugins/imager/direct.py | 17 +++-
scripts/lib/wic/plugins/source/bootimg-pcbios.py | 48 ++++-----
scripts/lib/wic/plugins/source/fsimage.py | 56 +++++++++++
scripts/lib/wic/utils/misc.py | 16 +--
scripts/lib/wic/utils/runner.py | 74 ++++++++++++--
Do I understand correctly that the series reverts changes to bitbake core and
moves them to scripts/lib?
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 9/9] wic: now truly go for the wic version we claim to have
2018-01-31 11:53 ` Baurzhan Ismagulov
@ 2018-01-31 12:01 ` Jan Kiszka
2018-01-31 12:28 ` Baurzhan Ismagulov
0 siblings, 1 reply; 51+ messages in thread
From: Jan Kiszka @ 2018-01-31 12:01 UTC (permalink / raw)
To: isar-users
On 2018-01-31 12:53, Baurzhan Ismagulov wrote:
> On Wed, Jan 31, 2018 at 12:43:24PM +0100, Jan Kiszka wrote:
>> Study patch 2 again, I think you misunderstood what it changed: all
>> adaptations are consolidated in those 100 lines now. Now longer a fork
>> of wic, only few external wrappings (without changing scripts or lib).
>
> We'll study the patches in more detail, but I'd still like to confirm I
> understand the concept correctly. The issue is, we import script/wic and
> scripts/lib/wic atomically, as a single whole. Now, patch 9/9 changes:
>
> scripts/lib/wic/canned-wks/qemux86-directdisk.wks | 2 +-
> scripts/lib/wic/filemap.py | 6 +-
> scripts/lib/wic/help.py | 2 -
> scripts/lib/wic/ksparser.py | 4 +-
> scripts/lib/wic/partition.py | 114 ++++++++++++++--------
> scripts/lib/wic/plugins/imager/direct.py | 17 +++-
> scripts/lib/wic/plugins/source/bootimg-pcbios.py | 48 ++++-----
> scripts/lib/wic/plugins/source/fsimage.py | 56 +++++++++++
> scripts/lib/wic/utils/misc.py | 16 +--
> scripts/lib/wic/utils/runner.py | 74 ++++++++++++--
>
> Do I understand correctly that the series reverts changes to bitbake core and
> moves them to scripts/lib?
To my reading, no changes are moved to lib. lib is restored to the OE
hash the original isar commit only /claimed/ to have used.
Maybe just a mistake in the log of that original commit. But then please
enlighten us what hash was truly used and if that should stay in favor
of this reset.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 9/9] wic: now truly go for the wic version we claim to have
2018-01-31 12:01 ` Jan Kiszka
@ 2018-01-31 12:28 ` Baurzhan Ismagulov
2018-01-31 13:53 ` Henning Schild
0 siblings, 1 reply; 51+ messages in thread
From: Baurzhan Ismagulov @ 2018-01-31 12:28 UTC (permalink / raw)
To: isar-users
On Wed, Jan 31, 2018 at 01:01:57PM +0100, Jan Kiszka wrote:
> To my reading, no changes are moved to lib. lib is restored to the OE
> hash the original isar commit only /claimed/ to have used.
Now I see, thanks. I didn't understand that from the patch description, what
about the following:
Subject: [PATCH 9/9] wic: Fix wic import
Commit 2b164b18fd639c9 claims to introduce wic hash 131629ca6238ea05, but the
former differs from the latter. This change fixes that import.
Issue:
- the wic version in Isar was modified
- that causes:
- confusion, maintainability and updateability issues
- potential quality issues
Impact:
This patch and the previous reverts get wic back to a state where all
these Issues are solved. We could now just update our wic without having
to worry about local patches. In case of a wic-update Isar and layers
on top would still have to review their plugins.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
(I've added the word "commit" because this is what I usually look for searching
for changes fixing other commits).
> Maybe just a mistake in the log of that original commit. But then please
> enlighten us what hash was truly used and if that should stay in favor
> of this reset.
I'm not aware of any other commit. If Henning's version works, I don't see the
point in redoing his work with a different upstream commit.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 9/9] wic: now truly go for the wic version we claim to have
2018-01-31 12:28 ` Baurzhan Ismagulov
@ 2018-01-31 13:53 ` Henning Schild
2018-01-31 14:01 ` Baurzhan Ismagulov
0 siblings, 1 reply; 51+ messages in thread
From: Henning Schild @ 2018-01-31 13:53 UTC (permalink / raw)
To: Baurzhan Ismagulov; +Cc: isar-users
Yes, after the reverts and this patch a diff of wic and lib/wic against
the claimed OE version is empty.
Henning
Am Wed, 31 Jan 2018 13:28:42 +0100
schrieb Baurzhan Ismagulov <ibr@radix50.net>:
> On Wed, Jan 31, 2018 at 01:01:57PM +0100, Jan Kiszka wrote:
> > To my reading, no changes are moved to lib. lib is restored to the
> > OE hash the original isar commit only /claimed/ to have used.
>
> Now I see, thanks. I didn't understand that from the patch
> description, what about the following:
>
>
> Subject: [PATCH 9/9] wic: Fix wic import
>
> Commit 2b164b18fd639c9 claims to introduce wic hash 131629ca6238ea05,
> but the former differs from the latter. This change fixes that import.
>
> Issue:
> - the wic version in Isar was modified
> - that causes:
> - confusion, maintainability and updateability issues
> - potential quality issues
>
> Impact:
> This patch and the previous reverts get wic back to a state where all
> these Issues are solved. We could now just update our wic without
> having to worry about local patches. In case of a wic-update Isar and
> layers on top would still have to review their plugins.
>
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
>
>
> (I've added the word "commit" because this is what I usually look for
> searching for changes fixing other commits).
>
>
> > Maybe just a mistake in the log of that original commit. But then
> > please enlighten us what hash was truly used and if that should
> > stay in favor of this reset.
>
> I'm not aware of any other commit. If Henning's version works, I
> don't see the point in redoing his work with a different upstream
> commit.
>
>
> With kind regards,
> Baurzhan.
>
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 9/9] wic: now truly go for the wic version we claim to have
2018-01-31 13:53 ` Henning Schild
@ 2018-01-31 14:01 ` Baurzhan Ismagulov
2018-01-31 14:21 ` Henning Schild
0 siblings, 1 reply; 51+ messages in thread
From: Baurzhan Ismagulov @ 2018-01-31 14:01 UTC (permalink / raw)
To: isar-users
On Wed, Jan 31, 2018 at 02:53:30PM +0100, Henning Schild wrote:
> Yes, after the reverts and this patch a diff of wic and lib/wic against
> the claimed OE version is empty.
So, ok to use my modified commit subject and description?
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 9/9] wic: now truly go for the wic version we claim to have
2018-01-31 14:01 ` Baurzhan Ismagulov
@ 2018-01-31 14:21 ` Henning Schild
0 siblings, 0 replies; 51+ messages in thread
From: Henning Schild @ 2018-01-31 14:21 UTC (permalink / raw)
To: Baurzhan Ismagulov; +Cc: isar-users
Am Wed, 31 Jan 2018 15:01:47 +0100
schrieb Baurzhan Ismagulov <ibr@radix50.net>:
> On Wed, Jan 31, 2018 at 02:53:30PM +0100, Henning Schild wrote:
> > Yes, after the reverts and this patch a diff of wic and lib/wic
> > against the claimed OE version is empty.
>
> So, ok to use my modified commit subject and description?
Sure, thanks!
Henning
> With kind regards,
> Baurzhan.
>
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 9:41 [PATCH 0/9] first wic integration Henning Schild
` (8 preceding siblings ...)
2018-01-31 9:42 ` [PATCH 9/9] wic: now truly go for the wic version we claim to have Henning Schild
@ 2018-01-31 10:02 ` Alexander Smirnov
2018-01-31 10:12 ` Henning Schild
2018-02-01 12:41 ` [PATCH] images: wic: limit use of sudo and enable manual call again Henning Schild
2018-02-12 17:27 ` [PATCH 0/9] first wic integration Henning Schild
11 siblings, 1 reply; 51+ messages in thread
From: Alexander Smirnov @ 2018-01-31 10:02 UTC (permalink / raw)
To: Henning Schild, isar-users
On 01/31/2018 12:41 PM, Henning Schild wrote:
> This is the first official post of a series that aims to do two things.
> 1. integrate wic into Isar so it does not need to be called after
> bitbake
> 2. get our copy of wic back to an unmodified version from OE
>
> Many people have been asking for the first, so i think i do not need to
> motivate that change. Things this series already brings is control over
> the partition sizes, that we did not have before.
> The choice in working wks files is still limited. The plugins dealing
> with bootloaders (isolinux, syslinux etc.) will probably need to be
> forked (like the efigrub one).
>
> The second one is important so we can update wic in the future and
> developers do not get confused. Having a fork instead of a copy, without
> a very good reason, is not a good idea. I found that wic does not need
> to be changed and plugins can be forked.
>
Are there any changes in user interface? Am I right that old commands
still work?
Alex
> Henning
>
> Henning Schild (9):
> classes: image: introduce size measuring function, for before
> do_*_image
> images: new class wic-img for wic intregration
> wic: add a bootimg-efi-isar plugin outside the wic tree
> Revert "wic: Make the bootimg-efi plugin generate usable images"
> Revert "wic: Introduce the `WicExecError` exception class"
> Revert "wic: Work around mcopy error"
> Revert "wic: Use sudo instead of pseudo"
> Revert "wic: Remove sysroot support"
> wic: now truly go for the wic version we claim to have
>
> .../scripts/lib/wic/canned-wks/sdimage-efi.wks | 2 +-
> .../lib/wic/plugins/source/bootimg-efi-isar.py | 307 +++++++++++++++++++++
> meta/classes/ext4-img.bbclass | 7 +-
> meta/classes/image.bbclass | 19 ++
> meta/classes/wic-img.bbclass | 64 +++++
> scripts/lib/wic/canned-wks/qemux86-directdisk.wks | 2 +-
> scripts/lib/wic/engine.py | 7 +-
> scripts/lib/wic/filemap.py | 6 +-
> scripts/lib/wic/help.py | 2 -
> scripts/lib/wic/ksparser.py | 4 +-
> scripts/lib/wic/partition.py | 197 ++++++++-----
> scripts/lib/wic/pluginbase.py | 11 +-
> scripts/lib/wic/plugins/imager/direct.py | 66 +++--
> scripts/lib/wic/plugins/source/bootimg-efi.py | 78 +-----
> .../lib/wic/plugins/source/bootimg-partition.py | 6 +-
> scripts/lib/wic/plugins/source/bootimg-pcbios.py | 68 ++---
> scripts/lib/wic/plugins/source/fsimage.py | 56 ++++
> .../lib/wic/plugins/source/isoimage-isohybrid.py | 28 +-
> scripts/lib/wic/plugins/source/rawcopy.py | 2 +-
> scripts/lib/wic/plugins/source/rootfs.py | 4 +-
> scripts/lib/wic/utils/misc.py | 33 +--
> scripts/lib/wic/utils/runner.py | 74 ++++-
> scripts/wic | 27 +-
> scripts/wic_fakeroot | 37 +++
> 24 files changed, 846 insertions(+), 261 deletions(-)
> create mode 100644 meta-isar/scripts/lib/wic/plugins/source/bootimg-efi-isar.py
> create mode 100644 meta/classes/wic-img.bbclass
> create mode 100644 scripts/lib/wic/plugins/source/fsimage.py
> create mode 100755 scripts/wic_fakeroot
>
--
With best regards,
Alexander Smirnov
ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 51+ messages in thread* Re: [PATCH 0/9] first wic integration
2018-01-31 10:02 ` [PATCH 0/9] first wic integration Alexander Smirnov
@ 2018-01-31 10:12 ` Henning Schild
2018-01-31 11:24 ` Baurzhan Ismagulov
0 siblings, 1 reply; 51+ messages in thread
From: Henning Schild @ 2018-01-31 10:12 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users
Am Wed, 31 Jan 2018 13:02:53 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> On 01/31/2018 12:41 PM, Henning Schild wrote:
> > This is the first official post of a series that aims to do two
> > things. 1. integrate wic into Isar so it does not need to be called
> > after bitbake
> > 2. get our copy of wic back to an unmodified version from OE
> >
> > Many people have been asking for the first, so i think i do not
> > need to motivate that change. Things this series already brings is
> > control over the partition sizes, that we did not have before.
> > The choice in working wks files is still limited. The plugins
> > dealing with bootloaders (isolinux, syslinux etc.) will probably
> > need to be forked (like the efigrub one).
> >
> > The second one is important so we can update wic in the future and
> > developers do not get confused. Having a fork instead of a copy,
> > without a very good reason, is not a good idea. I found that wic
> > does not need to be changed and plugins can be forked.
> >
>
> Are there any changes in user interface? Am I right that old commands
> still work?
There are no changes. Calling wic manually as non-root does not work
anymore, but that might be unrelated. I think it was broken before. It
is a "du" failing with permission denied, see comment in
scripts/wic_fakeroot. If it did work before, i think it must have been
a hack in the wic-modifications. I also do not fully understand where
the fsck "fail" comes from, wic had modifications there as well.
If the manual non-root call did work before, i will look into that. Or
just change the docs that mention it.
We will need test coverage in CI, but as far as i can tell the whole
wic-side was not covered.
Henning
> Alex
>
> > Henning
> >
> > Henning Schild (9):
> > classes: image: introduce size measuring function, for before
> > do_*_image
> > images: new class wic-img for wic intregration
> > wic: add a bootimg-efi-isar plugin outside the wic tree
> > Revert "wic: Make the bootimg-efi plugin generate usable images"
> > Revert "wic: Introduce the `WicExecError` exception class"
> > Revert "wic: Work around mcopy error"
> > Revert "wic: Use sudo instead of pseudo"
> > Revert "wic: Remove sysroot support"
> > wic: now truly go for the wic version we claim to have
> >
> > .../scripts/lib/wic/canned-wks/sdimage-efi.wks | 2 +-
> > .../lib/wic/plugins/source/bootimg-efi-isar.py | 307
> > +++++++++++++++++++++
> > meta/classes/ext4-img.bbclass | 7 +-
> > meta/classes/image.bbclass | 19 ++
> > meta/classes/wic-img.bbclass | 64 +++++
> > scripts/lib/wic/canned-wks/qemux86-directdisk.wks | 2 +-
> > scripts/lib/wic/engine.py | 7 +-
> > scripts/lib/wic/filemap.py | 6 +-
> > scripts/lib/wic/help.py | 2 -
> > scripts/lib/wic/ksparser.py | 4 +-
> > scripts/lib/wic/partition.py | 197
> > ++++++++----- scripts/lib/wic/pluginbase.py |
> > 11 +- scripts/lib/wic/plugins/imager/direct.py | 66
> > +++-- scripts/lib/wic/plugins/source/bootimg-efi.py | 78
> > +----- .../lib/wic/plugins/source/bootimg-partition.py | 6 +-
> > scripts/lib/wic/plugins/source/bootimg-pcbios.py | 68 ++---
> > scripts/lib/wic/plugins/source/fsimage.py | 56
> > ++++ .../lib/wic/plugins/source/isoimage-isohybrid.py | 28 +-
> > scripts/lib/wic/plugins/source/rawcopy.py | 2 +-
> > scripts/lib/wic/plugins/source/rootfs.py | 4 +-
> > scripts/lib/wic/utils/misc.py | 33 +--
> > scripts/lib/wic/utils/runner.py | 74 ++++-
> > scripts/wic | 27 +-
> > scripts/wic_fakeroot | 37 +++ 24
> > files changed, 846 insertions(+), 261 deletions(-) create mode
> > 100644 meta-isar/scripts/lib/wic/plugins/source/bootimg-efi-isar.py
> > create mode 100644 meta/classes/wic-img.bbclass create mode 100644
> > scripts/lib/wic/plugins/source/fsimage.py create mode 100755
> > scripts/wic_fakeroot
>
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 10:12 ` Henning Schild
@ 2018-01-31 11:24 ` Baurzhan Ismagulov
2018-01-31 11:47 ` Jan Kiszka
2018-01-31 13:05 ` Henning Schild
0 siblings, 2 replies; 51+ messages in thread
From: Baurzhan Ismagulov @ 2018-01-31 11:24 UTC (permalink / raw)
To: isar-users
On Wed, Jan 31, 2018 at 11:12:53AM +0100, Henning Schild wrote:
> There are no changes. Calling wic manually as non-root does not work
> anymore, but that might be unrelated.
Well, that's a change, and a user-visible one.
> If the manual non-root call did work before, i will look into that.
Yes. Yes, please do.
In Yocto, manual non-root call works. The same is our vision for Isar. For now,
it has been workarounded with sudo. We have a working PoC based on PRoot in the
pipeline. So I don't support the idea of changing the user experience because
of this issue.
We'll be looking at your changes in more detail. Before that, I have a
conceptual question. You revert the following patches:
b1511ec wic: Work around mcopy error
17f9196 wic: Introduce the `WicExecError` exception class
All of them are not related to the bitbake core and fix concrete issues with
image generation that were not addressed in the baseline bitbake. Did you
address those issues in your new changes?
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 11:24 ` Baurzhan Ismagulov
@ 2018-01-31 11:47 ` Jan Kiszka
2018-01-31 12:02 ` Baurzhan Ismagulov
2018-01-31 13:05 ` Henning Schild
1 sibling, 1 reply; 51+ messages in thread
From: Jan Kiszka @ 2018-01-31 11:47 UTC (permalink / raw)
To: isar-users
On 2018-01-31 12:24, Baurzhan Ismagulov wrote:
> On Wed, Jan 31, 2018 at 11:12:53AM +0100, Henning Schild wrote:
>> There are no changes. Calling wic manually as non-root does not work
>> anymore, but that might be unrelated.
>
> Well, that's a change, and a user-visible one.
>
>
>> If the manual non-root call did work before, i will look into that.
>
> Yes. Yes, please do.
>
> In Yocto, manual non-root call works. The same is our vision for Isar. For now,
> it has been workarounded with sudo. We have a working PoC based on PRoot in the
> pipeline. So I don't support the idea of changing the user experience because
> of this issue.
>
While I agree we should try to get whole Isar unprivileged, let's not
invest too much into this when it's complex. We are currently far away
from that goal (as you know, proot has several open issues as well), and
this here solves the problem of having to run wic manually at all.
>
> We'll be looking at your changes in more detail. Before that, I have a
> conceptual question. You revert the following patches:
>
> b1511ec wic: Work around mcopy error
> 17f9196 wic: Introduce the `WicExecError` exception class
>
> All of them are not related to the bitbake core and fix concrete issues with
> image generation that were not addressed in the baseline bitbake. Did you
> address those issues in your new changes?
Good points, should be clarified. Maybe an update of the upstream wic
import is the answer if we are about to lose something. Better than
patching locally in Isar, as so far.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 11:47 ` Jan Kiszka
@ 2018-01-31 12:02 ` Baurzhan Ismagulov
2018-01-31 12:15 ` Jan Kiszka
2018-01-31 13:36 ` Henning Schild
0 siblings, 2 replies; 51+ messages in thread
From: Baurzhan Ismagulov @ 2018-01-31 12:02 UTC (permalink / raw)
To: isar-users
On Wed, Jan 31, 2018 at 12:47:08PM +0100, Jan Kiszka wrote:
> >> If the manual non-root call did work before, i will look into that.
> >
> > Yes. Yes, please do.
> >
> > In Yocto, manual non-root call works. The same is our vision for Isar. For now,
> > it has been workarounded with sudo. We have a working PoC based on PRoot in the
> > pipeline. So I don't support the idea of changing the user experience because
> > of this issue.
>
> While I agree we should try to get whole Isar unprivileged, let's not
> invest too much into this when it's complex. We are currently far away
> from that goal (as you know, proot has several open issues as well), and
> this here solves the problem of having to run wic manually at all.
No effort whatsoever. This is already workarounded in master in the following
commit:
bf873d3 wic: Use sudo instead of pseudo
My suggestion is not to revert that. I agree with Henning that we should cover
that in CI. I'll provide an initial testing framework, then we should sync who
does what. The other mcopy and fsck issues are not covered, either.
Regarding the PRoot PoC, I don't agree we are far away. Everything works
except mounting. We've just postponed the work due to higher-prio issues.
> > b1511ec wic: Work around mcopy error
> > 17f9196 wic: Introduce the `WicExecError` exception class
> >
> > All of them are not related to the bitbake core and fix concrete issues with
> > image generation that were not addressed in the baseline bitbake. Did you
> > address those issues in your new changes?
>
> Good points, should be clarified. Maybe an update of the upstream wic
> import is the answer if we are about to lose something. Better than
> patching locally in Isar, as so far.
Do you mean we should initiate upstreaming that to OE? If yes, I agree with
you. The only question is point of time. My idea was to wait till Henning
completes the lifecycle with his OE contribution and learn from his experience.
If you think we should do that now, please let me know.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 12:02 ` Baurzhan Ismagulov
@ 2018-01-31 12:15 ` Jan Kiszka
2018-01-31 13:30 ` Jan Kiszka
` (2 more replies)
2018-01-31 13:36 ` Henning Schild
1 sibling, 3 replies; 51+ messages in thread
From: Jan Kiszka @ 2018-01-31 12:15 UTC (permalink / raw)
To: isar-users
On 2018-01-31 13:02, Baurzhan Ismagulov wrote:
> On Wed, Jan 31, 2018 at 12:47:08PM +0100, Jan Kiszka wrote:
>>>> If the manual non-root call did work before, i will look into that.
>>>
>>> Yes. Yes, please do.
>>>
>>> In Yocto, manual non-root call works. The same is our vision for Isar. For now,
>>> it has been workarounded with sudo. We have a working PoC based on PRoot in the
>>> pipeline. So I don't support the idea of changing the user experience because
>>> of this issue.
>>
>> While I agree we should try to get whole Isar unprivileged, let's not
>> invest too much into this when it's complex. We are currently far away
>> from that goal (as you know, proot has several open issues as well), and
>> this here solves the problem of having to run wic manually at all.
>
> No effort whatsoever. This is already workarounded in master in the following
> commit:
>
> bf873d3 wic: Use sudo instead of pseudo
>
This hacks in wic, and that has to go in order to keep maintainability.
We need to solve that differently, for sure.
> My suggestion is not to revert that. I agree with Henning that we should cover
> that in CI. I'll provide an initial testing framework, then we should sync who
> does what. The other mcopy and fsck issues are not covered, either.
>
>
> Regarding the PRoot PoC, I don't agree we are far away. Everything works
> except mounting. We've just postponed the work due to higher-prio issues.
I'm happy to discuss the effort you see in resolving that last issue and
will re-prioritize.
>
>
>>> b1511ec wic: Work around mcopy error
This issue is not understood yet, thus the commit not ready for
upstream. Furthermore, the log does not describe how to reproduce the
issue. So - how to clarify if we still need it?
>>> 17f9196 wic: Introduce the `WicExecError` exception class
Can you explain why fsck should return an error at all on a freshly
built file system image? This seems to paper over some issue to me.
>>>
>>> All of them are not related to the bitbake core and fix concrete issues with
>>> image generation that were not addressed in the baseline bitbake. Did you
>>> address those issues in your new changes?
>>
>> Good points, should be clarified. Maybe an update of the upstream wic
>> import is the answer if we are about to lose something. Better than
>> patching locally in Isar, as so far.
>
> Do you mean we should initiate upstreaming that to OE? If yes, I agree with
> you. The only question is point of time. My idea was to wait till Henning
> completes the lifecycle with his OE contribution and learn from his experience.
> If you think we should do that now, please let me know.
Local hacks can save some time for a while but will shoot back with more
effort eventually. I think we are already at that point, so let's move
forward. We can help you with this.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 51+ messages in thread* Re: [PATCH 0/9] first wic integration
2018-01-31 12:15 ` Jan Kiszka
@ 2018-01-31 13:30 ` Jan Kiszka
2018-01-31 13:41 ` Baurzhan Ismagulov
2018-01-31 13:35 ` Baurzhan Ismagulov
2018-01-31 13:46 ` Henning Schild
2 siblings, 1 reply; 51+ messages in thread
From: Jan Kiszka @ 2018-01-31 13:30 UTC (permalink / raw)
To: isar-users
On 2018-01-31 13:15, [ext] Jan Kiszka wrote:
> On 2018-01-31 13:02, Baurzhan Ismagulov wrote:
>> Regarding the PRoot PoC, I don't agree we are far away. Everything works
>> except mounting. We've just postponed the work due to higher-prio issues.
>
> I'm happy to discuss the effort you see in resolving that last issue and
> will re-prioritize.
BTW, we also need to address unprivileged or container-compatible
binfmt, or we won't be able to do cross stuff. Probably solvable, maybe
via namespace support for binfmt in the upstream kernel, but far from
reachable in the near future.
Jan
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 13:30 ` Jan Kiszka
@ 2018-01-31 13:41 ` Baurzhan Ismagulov
2018-01-31 14:01 ` Jan Kiszka
0 siblings, 1 reply; 51+ messages in thread
From: Baurzhan Ismagulov @ 2018-01-31 13:41 UTC (permalink / raw)
To: isar-users
On Wed, Jan 31, 2018 at 02:30:33PM +0100, Jan Kiszka wrote:
> BTW, we also need to address unprivileged or container-compatible
> binfmt, or we won't be able to do cross stuff. Probably solvable, maybe
> via namespace support for binfmt in the upstream kernel, but far from
> reachable in the near future.
That is another issue to fix, but it isn't related to hacking wic, is it?
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 13:41 ` Baurzhan Ismagulov
@ 2018-01-31 14:01 ` Jan Kiszka
2018-01-31 15:21 ` Baurzhan Ismagulov
0 siblings, 1 reply; 51+ messages in thread
From: Jan Kiszka @ 2018-01-31 14:01 UTC (permalink / raw)
To: isar-users
On 2018-01-31 14:41, Baurzhan Ismagulov wrote:
> On Wed, Jan 31, 2018 at 02:30:33PM +0100, Jan Kiszka wrote:
>> BTW, we also need to address unprivileged or container-compatible
>> binfmt, or we won't be able to do cross stuff. Probably solvable, maybe
>> via namespace support for binfmt in the upstream kernel, but far from
>> reachable in the near future.
>
> That is another issue to fix, but it isn't related to hacking wic, is it?
It is related to the question if we need to worry about sudo wic right
now or can do this when all the other issues that prevent unprivileged
Isar building are solved. I would say the latter applies here.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 14:01 ` Jan Kiszka
@ 2018-01-31 15:21 ` Baurzhan Ismagulov
2018-01-31 15:46 ` Henning Schild
2018-01-31 16:13 ` Jan Kiszka
0 siblings, 2 replies; 51+ messages in thread
From: Baurzhan Ismagulov @ 2018-01-31 15:21 UTC (permalink / raw)
To: isar-users
On Wed, Jan 31, 2018 at 03:01:48PM +0100, Jan Kiszka wrote:
> >> BTW, we also need to address unprivileged or container-compatible
> >> binfmt, or we won't be able to do cross stuff. Probably solvable, maybe
> >> via namespace support for binfmt in the upstream kernel, but far from
> >> reachable in the near future.
> >
> > That is another issue to fix, but it isn't related to hacking wic, is it?
>
> It is related to the question if we need to worry about sudo wic right
> now or can do this when all the other issues that prevent unprivileged
> Isar building are solved. I would say the latter applies here.
Unprivileged != container-compatible.
Sudo is a hack. Solving it has value.
That said, my concern isn't prioritizing that. My concern is imposing sudo on
wic users when we already have an effective, manageable workaround in master.
There is also an architectural issue with that. All-in sudo would hide the
details why we need it, thus moving unprivileged builds farther away.
At the end, wic should be compatible with Isar and be available without
importing it into Isar. If we want to work with upstream, we should start with
that and not with breaking the existing code. If existing tools were
sufficient, we wouldn't need Isar in the first place; upstreaming should be a
good practical trade-off and not transform into purification that stands in the
way.
Thus my suggestion to keep selective sudo in wic. It doesn't require reworking
the series, we can just drop the patch 7.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 15:21 ` Baurzhan Ismagulov
@ 2018-01-31 15:46 ` Henning Schild
2018-01-31 16:13 ` Jan Kiszka
1 sibling, 0 replies; 51+ messages in thread
From: Henning Schild @ 2018-01-31 15:46 UTC (permalink / raw)
To: Baurzhan Ismagulov; +Cc: isar-users
Am Wed, 31 Jan 2018 16:21:18 +0100
schrieb Baurzhan Ismagulov <ibr@radix50.net>:
> On Wed, Jan 31, 2018 at 03:01:48PM +0100, Jan Kiszka wrote:
> > >> BTW, we also need to address unprivileged or container-compatible
> > >> binfmt, or we won't be able to do cross stuff. Probably
> > >> solvable, maybe via namespace support for binfmt in the upstream
> > >> kernel, but far from reachable in the near future.
> > >
> > > That is another issue to fix, but it isn't related to hacking
> > > wic, is it?
> >
> > It is related to the question if we need to worry about sudo wic
> > right now or can do this when all the other issues that prevent
> > unprivileged Isar building are solved. I would say the latter
> > applies here.
>
> Unprivileged != container-compatible.
>
> Sudo is a hack. Solving it has value.
>
> That said, my concern isn't prioritizing that. My concern is imposing
> sudo on wic users when we already have an effective, manageable
> workaround in master.
No we do not. The stuff in master is a ton of "sudo" and
all-you-can-eat sudo for plugins, totally unsafe like the patch i sent.
> There is also an architectural issue with that. All-in sudo would
> hide the details why we need it, thus moving unprivileged builds
> farther away.
I have a patch ready that takes care of "du" and "mkfs" and therefore
documents their "sudo"-needs.
Henning
> At the end, wic should be compatible with Isar and be available
> without importing it into Isar. If we want to work with upstream, we
> should start with that and not with breaking the existing code. If
> existing tools were sufficient, we wouldn't need Isar in the first
> place; upstreaming should be a good practical trade-off and not
> transform into purification that stands in the way.
>
> Thus my suggestion to keep selective sudo in wic. It doesn't require
> reworking the series, we can just drop the patch 7.
>
> With kind regards,
> Baurzhan.
>
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 15:21 ` Baurzhan Ismagulov
2018-01-31 15:46 ` Henning Schild
@ 2018-01-31 16:13 ` Jan Kiszka
1 sibling, 0 replies; 51+ messages in thread
From: Jan Kiszka @ 2018-01-31 16:13 UTC (permalink / raw)
To: isar-users
On 2018-01-31 16:21, Baurzhan Ismagulov wrote:
> On Wed, Jan 31, 2018 at 03:01:48PM +0100, Jan Kiszka wrote:
>>>> BTW, we also need to address unprivileged or container-compatible
>>>> binfmt, or we won't be able to do cross stuff. Probably solvable, maybe
>>>> via namespace support for binfmt in the upstream kernel, but far from
>>>> reachable in the near future.
>>>
>>> That is another issue to fix, but it isn't related to hacking wic, is it?
>>
>> It is related to the question if we need to worry about sudo wic right
>> now or can do this when all the other issues that prevent unprivileged
>> Isar building are solved. I would say the latter applies here.
>
> Unprivileged != container-compatible.
Unprivileged will imply container-compatible.
In turn, we might become container-compatible prior to becoming
unprivileged, which would be already be a big step forward, but nothing
I see soon either. Also because binfmt is a central, hard-to-share
privileged interface.
Jan
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 12:15 ` Jan Kiszka
2018-01-31 13:30 ` Jan Kiszka
@ 2018-01-31 13:35 ` Baurzhan Ismagulov
2018-01-31 13:47 ` Henning Schild
2018-01-31 13:46 ` Henning Schild
2 siblings, 1 reply; 51+ messages in thread
From: Baurzhan Ismagulov @ 2018-01-31 13:35 UTC (permalink / raw)
To: isar-users
On Wed, Jan 31, 2018 at 01:15:28PM +0100, Jan Kiszka wrote:
> > bf873d3 wic: Use sudo instead of pseudo
>
> This hacks in wic, and that has to go in order to keep maintainability.
> We need to solve that differently, for sure.
That is a trade-off between breaking the user experience (which is strategic)
and temporary workaround that will go away anyway.
Till we get our changes accepted upstream, we have to workaround them locally.
This series should remove the biggest hack, sysroot removal. Other changes are
one-liners that are easy to port to a new bitbake version.
That said, updating wic happens from time to time, but wic is used every day.
That is why I'd like to keep the workaround in master till we see what happens
with the upstream merge.
I'd welcome any suggestions other than "sudo wic".
> > Regarding the PRoot PoC...
>
> I'm happy to discuss the effort you see in resolving that last issue and
> will re-prioritize.
Ok, will come up with that after the current libhello work hits master.
> >>> b1511ec wic: Work around mcopy error
>
> This issue is not understood yet, thus the commit not ready for
> upstream. Furthermore, the log does not describe how to reproduce the
> issue. So - how to clarify if we still need it?
I'll check how to reproduce that.
> >>> 17f9196 wic: Introduce the `WicExecError` exception class
>
> Can you explain why fsck should return an error at all on a freshly
> built file system image? This seems to paper over some issue to me.
At the time, it sounded like a bug somewhere in mkfs or fsck, but we haven't
digged deeper yet. So yes, that's a workaround.
> > Do you mean we should initiate upstreaming that to OE? If yes, I agree with
> > you. The only question is point of time. My idea was to wait till Henning
> > completes the lifecycle with his OE contribution and learn from his experience.
> > If you think we should do that now, please let me know.
>
> Local hacks can save some time for a while but will shoot back with more
> effort eventually.
It wasn't about saving time, it was about providing working code first,
upstreaming second.
> I think we are already at that point, so let's move forward.
Ok, let me analyze the issues and come with a proposal.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 13:35 ` Baurzhan Ismagulov
@ 2018-01-31 13:47 ` Henning Schild
2018-01-31 14:00 ` Baurzhan Ismagulov
0 siblings, 1 reply; 51+ messages in thread
From: Henning Schild @ 2018-01-31 13:47 UTC (permalink / raw)
To: Baurzhan Ismagulov; +Cc: isar-users
Am Wed, 31 Jan 2018 14:35:11 +0100
schrieb Baurzhan Ismagulov <ibr@radix50.net>:
> On Wed, Jan 31, 2018 at 01:15:28PM +0100, Jan Kiszka wrote:
> > > bf873d3 wic: Use sudo instead of pseudo
> >
> > This hacks in wic, and that has to go in order to keep
> > maintainability. We need to solve that differently, for sure.
>
> That is a trade-off between breaking the user experience (which is
> strategic) and temporary workaround that will go away anyway.
>
> Till we get our changes accepted upstream, we have to workaround them
> locally. This series should remove the biggest hack, sysroot removal.
> Other changes are one-liners that are easy to port to a new bitbake
> version.
>
> That said, updating wic happens from time to time, but wic is used
> every day. That is why I'd like to keep the workaround in master till
> we see what happens with the upstream merge.
>
> I'd welcome any suggestions other than "sudo wic".
sudo bash -c wic?
Henning
>
> > > Regarding the PRoot PoC...
> >
> > I'm happy to discuss the effort you see in resolving that last
> > issue and will re-prioritize.
>
> Ok, will come up with that after the current libhello work hits
> master.
>
>
> > >>> b1511ec wic: Work around mcopy error
> >
> > This issue is not understood yet, thus the commit not ready for
> > upstream. Furthermore, the log does not describe how to reproduce
> > the issue. So - how to clarify if we still need it?
>
> I'll check how to reproduce that.
>
>
> > >>> 17f9196 wic: Introduce the `WicExecError` exception class
> >
> > Can you explain why fsck should return an error at all on a freshly
> > built file system image? This seems to paper over some issue to
> > me.
>
> At the time, it sounded like a bug somewhere in mkfs or fsck, but we
> haven't digged deeper yet. So yes, that's a workaround.
>
>
> > > Do you mean we should initiate upstreaming that to OE? If yes, I
> > > agree with you. The only question is point of time. My idea was
> > > to wait till Henning completes the lifecycle with his OE
> > > contribution and learn from his experience. If you think we
> > > should do that now, please let me know.
> >
> > Local hacks can save some time for a while but will shoot back with
> > more effort eventually.
>
> It wasn't about saving time, it was about providing working code
> first, upstreaming second.
>
>
> > I think we are already at that point, so let's move forward.
>
> Ok, let me analyze the issues and come with a proposal.
>
>
> With kind regards,
> Baurzhan.
>
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 13:47 ` Henning Schild
@ 2018-01-31 14:00 ` Baurzhan Ismagulov
0 siblings, 0 replies; 51+ messages in thread
From: Baurzhan Ismagulov @ 2018-01-31 14:00 UTC (permalink / raw)
To: isar-users
On Wed, Jan 31, 2018 at 02:47:41PM +0100, Henning Schild wrote:
> > That is a trade-off between breaking the user experience (which is
> > strategic) and temporary workaround that will go away anyway.
> >
> > Till we get our changes accepted upstream, we have to workaround them
> > locally. This series should remove the biggest hack, sysroot removal.
> > Other changes are one-liners that are easy to port to a new bitbake
> > version.
> >
> > That said, updating wic happens from time to time, but wic is used
> > every day. That is why I'd like to keep the workaround in master till
> > we see what happens with the upstream merge.
> >
> > I'd welcome any suggestions other than "sudo wic".
>
> sudo bash -c wic?
Nope :) .
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 12:15 ` Jan Kiszka
2018-01-31 13:30 ` Jan Kiszka
2018-01-31 13:35 ` Baurzhan Ismagulov
@ 2018-01-31 13:46 ` Henning Schild
2 siblings, 0 replies; 51+ messages in thread
From: Henning Schild @ 2018-01-31 13:46 UTC (permalink / raw)
To: [ext] Jan Kiszka; +Cc: isar-users
Am Wed, 31 Jan 2018 13:15:28 +0100
schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:
> On 2018-01-31 13:02, Baurzhan Ismagulov wrote:
> > On Wed, Jan 31, 2018 at 12:47:08PM +0100, Jan Kiszka wrote:
> >>>> If the manual non-root call did work before, i will look into
> >>>> that.
> >>>
> >>> Yes. Yes, please do.
> >>>
> >>> In Yocto, manual non-root call works. The same is our vision for
> >>> Isar. For now, it has been workarounded with sudo. We have a
> >>> working PoC based on PRoot in the pipeline. So I don't support
> >>> the idea of changing the user experience because of this issue.
> >>
> >> While I agree we should try to get whole Isar unprivileged, let's
> >> not invest too much into this when it's complex. We are currently
> >> far away from that goal (as you know, proot has several open
> >> issues as well), and this here solves the problem of having to run
> >> wic manually at all.
> >
> > No effort whatsoever. This is already workarounded in master in the
> > following commit:
> >
> > bf873d3 wic: Use sudo instead of pseudo
> >
>
> This hacks in wic, and that has to go in order to keep
> maintainability. We need to solve that differently, for sure.
>
> > My suggestion is not to revert that. I agree with Henning that we
> > should cover that in CI. I'll provide an initial testing framework,
> > then we should sync who does what. The other mcopy and fsck issues
> > are not covered, either.
> >
> >
> > Regarding the PRoot PoC, I don't agree we are far away. Everything
> > works except mounting. We've just postponed the work due to
> > higher-prio issues.
>
> I'm happy to discuss the effort you see in resolving that last issue
> and will re-prioritize.
>
> >
> >
> >>> b1511ec wic: Work around mcopy error
>
> This issue is not understood yet, thus the commit not ready for
> upstream. Furthermore, the log does not describe how to reproduce the
> issue. So - how to clarify if we still need it?
I can reproduce it when building images with directdisk.wks, i will
send patches for that.
The so-far supported sdimage-efi does not seem to be affected by it, at
least i never saw it failing.
This one, like the following is likely due to debian having different
versions of the tools than OE code might expect. And what is worse, we
are talking about host-tools here so Isar has to support a relatively
wide range.
> >>> 17f9196 wic: Introduce the `WicExecError` exception class
>
> Can you explain why fsck should return an error at all on a freshly
> built file system image? This seems to paper over some issue to me.
Debian simply uses an older version of e2sck, the bug has been fixed
upstream. I am guessing yocto must be using a more recent version.
The hack that "1" is on ok return value moved into wic_fakeroot.
https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/commit/?id=bf9f3b6d5b10d19218b4ed904c12b22e36ec57dd
> >>>
> >>> All of them are not related to the bitbake core and fix concrete
> >>> issues with image generation that were not addressed in the
> >>> baseline bitbake. Did you address those issues in your new
> >>> changes?
> >>
> >> Good points, should be clarified. Maybe an update of the upstream
> >> wic import is the answer if we are about to lose something. Better
> >> than patching locally in Isar, as so far.
> >
> > Do you mean we should initiate upstreaming that to OE? If yes, I
> > agree with you. The only question is point of time. My idea was to
> > wait till Henning completes the lifecycle with his OE contribution
> > and learn from his experience. If you think we should do that now,
> > please let me know.
>
> Local hacks can save some time for a while but will shoot back with
> more effort eventually. I think we are already at that point, so
> let's move forward. We can help you with this.
>
> Jan
>
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 12:02 ` Baurzhan Ismagulov
2018-01-31 12:15 ` Jan Kiszka
@ 2018-01-31 13:36 ` Henning Schild
2018-01-31 13:40 ` Baurzhan Ismagulov
1 sibling, 1 reply; 51+ messages in thread
From: Henning Schild @ 2018-01-31 13:36 UTC (permalink / raw)
To: Baurzhan Ismagulov; +Cc: isar-users
Am Wed, 31 Jan 2018 13:02:46 +0100
schrieb Baurzhan Ismagulov <ibr@radix50.net>:
> On Wed, Jan 31, 2018 at 12:47:08PM +0100, Jan Kiszka wrote:
> > >> If the manual non-root call did work before, i will look into
> > >> that.
> > >
> > > Yes. Yes, please do.
> > >
> > > In Yocto, manual non-root call works. The same is our vision for
> > > Isar. For now, it has been workarounded with sudo. We have a
> > > working PoC based on PRoot in the pipeline. So I don't support
> > > the idea of changing the user experience because of this issue.
> >
> > While I agree we should try to get whole Isar unprivileged, let's
> > not invest too much into this when it's complex. We are currently
> > far away from that goal (as you know, proot has several open issues
> > as well), and this here solves the problem of having to run wic
> > manually at all.
>
> No effort whatsoever. This is already workarounded in master in the
> following commit:
>
> bf873d3 wic: Use sudo instead of pseudo
This patch did not just replace pseudo with sudo, it also introduced
"sudo"s i.e. for the "du"s that now fail.
I could overlay "du" and "mkfs" with symlinks to wic_fakeroot and use
sudo in there, but i do not really see the point. If you are counting
sudos i replaces a whole lot of them with just one, an almighty one ;).
Henning
> My suggestion is not to revert that. I agree with Henning that we
> should cover that in CI. I'll provide an initial testing framework,
> then we should sync who does what. The other mcopy and fsck issues
> are not covered, either.
>
> Regarding the PRoot PoC, I don't agree we are far away. Everything
> works except mounting. We've just postponed the work due to
> higher-prio issues.
>
>
> > > b1511ec wic: Work around mcopy error
> > > 17f9196 wic: Introduce the `WicExecError` exception class
> > >
> > > All of them are not related to the bitbake core and fix concrete
> > > issues with image generation that were not addressed in the
> > > baseline bitbake. Did you address those issues in your new
> > > changes?
> >
> > Good points, should be clarified. Maybe an update of the upstream
> > wic import is the answer if we are about to lose something. Better
> > than patching locally in Isar, as so far.
>
> Do you mean we should initiate upstreaming that to OE? If yes, I
> agree with you. The only question is point of time. My idea was to
> wait till Henning completes the lifecycle with his OE contribution
> and learn from his experience. If you think we should do that now,
> please let me know.
>
>
> With kind regards,
> Baurzhan.
>
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 13:36 ` Henning Schild
@ 2018-01-31 13:40 ` Baurzhan Ismagulov
0 siblings, 0 replies; 51+ messages in thread
From: Baurzhan Ismagulov @ 2018-01-31 13:40 UTC (permalink / raw)
To: isar-users
On Wed, Jan 31, 2018 at 02:36:43PM +0100, Henning Schild wrote:
> > bf873d3 wic: Use sudo instead of pseudo
>
> This patch did not just replace pseudo with sudo, it also introduced
> "sudo"s i.e. for the "du"s that now fail.
>
> I could overlay "du" and "mkfs" with symlinks to wic_fakeroot and use
> sudo in there, but i do not really see the point. If you are counting
> sudos i replaces a whole lot of them with just one, an almighty one ;).
:)
It isn't about counting sudos, it's against imposing "sudo wic" on the user
when we have a workaround and a solution PoC.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 11:24 ` Baurzhan Ismagulov
2018-01-31 11:47 ` Jan Kiszka
@ 2018-01-31 13:05 ` Henning Schild
1 sibling, 0 replies; 51+ messages in thread
From: Henning Schild @ 2018-01-31 13:05 UTC (permalink / raw)
To: Baurzhan Ismagulov; +Cc: isar-users
Am Wed, 31 Jan 2018 12:24:21 +0100
schrieb Baurzhan Ismagulov <ibr@radix50.net>:
> On Wed, Jan 31, 2018 at 11:12:53AM +0100, Henning Schild wrote:
> > There are no changes. Calling wic manually as non-root does not work
> > anymore, but that might be unrelated.
>
> Well, that's a change, and a user-visible one.
As i said, i do not know when that last worked and would have to test
that.
> > If the manual non-root call did work before, i will look into
> > that.
>
> Yes. Yes, please do.
>
> In Yocto, manual non-root call works. The same is our vision for
> Isar. For now, it has been workarounded with sudo. We have a working
> PoC based on PRoot in the pipeline. So I don't support the idea of
> changing the user experience because of this issue.
>
>
> We'll be looking at your changes in more detail. Before that, I have a
> conceptual question. You revert the following patches:
>
> b1511ec wic: Work around mcopy error
The mcopy fix will come in patches concerning plugins that actually use
mcopy. I have one here that introduces .wks legacy bios with a
bootloader from the rootfs and not the host.
> 17f9196 wic: Introduce the `WicExecError` exception class
That is the fsck. hack that moved into wic_fakeroot.
Henning
> All of them are not related to the bitbake core and fix concrete
> issues with image generation that were not addressed in the baseline
> bitbake. Did you address those issues in your new changes?
>
>
> With kind regards,
> Baurzhan.
>
^ permalink raw reply [flat|nested] 51+ messages in thread
* [PATCH] images: wic: limit use of sudo and enable manual call again
2018-01-31 9:41 [PATCH 0/9] first wic integration Henning Schild
` (9 preceding siblings ...)
2018-01-31 10:02 ` [PATCH 0/9] first wic integration Alexander Smirnov
@ 2018-02-01 12:41 ` Henning Schild
2018-02-01 12:44 ` Henning Schild
2018-02-12 19:07 ` Henning Schild
2018-02-12 17:27 ` [PATCH 0/9] first wic integration Henning Schild
11 siblings, 2 replies; 51+ messages in thread
From: Henning Schild @ 2018-02-01 12:41 UTC (permalink / raw)
To: isar-users; +Cc: Baurzhan Ismagulov, Alexander Smirnov, Henning Schild
Issues:
1. after the wic rework wic was called under a big sudo
2. and calling it manually - like stated in the doc - did not work
anymore
Impact:
This patch solves both issues. Just like before sudo is only used for
"du" and "mkfs". And by applying some tricks and wrapping we now can
call "isar-wic" just like "wic" before.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
doc/user_manual.md | 4 +--
meta/classes/wic-img.bbclass | 6 ++--
meta/recipes-devtools/wic-tools/wic-tools.bb | 4 +++
scripts/isar-wic | 27 ++++++++++++++
scripts/isar-wic-handler | 53 ++++++++++++++++++++++++++++
scripts/wic_fakeroot | 37 -------------------
6 files changed, 90 insertions(+), 41 deletions(-)
create mode 100644 meta/recipes-devtools/wic-tools/wic-tools.bb
create mode 100755 scripts/isar-wic
create mode 100755 scripts/isar-wic-handler
delete mode 100755 scripts/wic_fakeroot
diff --git a/doc/user_manual.md b/doc/user_manual.md
index 969f6d2..e9284a0 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -208,9 +208,9 @@ Once the image artifacts have been built (c.f. previous section), full EFI disk
Currently, only the `i386` and `amd64` target architectures are supported:
```
# Generate an EFI image for the `i386` target architecture
- $ wic create -D sdimage-efi -o . -e multiconfig:qemui386-stretch:isar-image-base
+ $ isar-wic create -D sdimage-efi -o . -e multiconfig:qemui386-stretch:isar-image-base
# Similarly, for the `amd64` target architecture
- $ wic create -D sdimage-efi -o . -e multiconfig:qemuamd64-stretch:isar-image-base
+ $ isar-wic create -D sdimage-efi -o . -e multiconfig:qemuamd64-stretch:isar-image-base
```
In order to run the images with `qemu`, an EFI firmware is required and available at the following address:
diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass
index e8d2678..d4afde6 100644
--- a/meta/classes/wic-img.bbclass
+++ b/meta/classes/wic-img.bbclass
@@ -14,7 +14,7 @@ STAGING_DATADIR ?= "/usr/share/"
STAGING_LIBDIR ?= "/usr/lib/"
STAGING_DIR ?= "${TMPDIR}"
IMAGE_BASENAME ?= "multiconfig:${MACHINE}-${DISTRO}:${PN}"
-FAKEROOTCMD ?= "wic_fakeroot"
+FAKEROOTCMD ?= "/builder/isar/scripts/isar-wic-handler"
RECIPE_SYSROOT_NATIVE ?= "/"
do_wic_image[stamp-extra-info] = "${DISTRO}-${MACHINE}"
@@ -57,8 +57,10 @@ do_rootfs_wicenv[prefuncs] = 'set_image_size'
do_wic_image() {
export BUILDDIR="${BUILDDIR}"
+ export FAKEROOTCMD="${FAKEROOTCMD}"
+ export TMPDIR="${TMPDIR}"
- sudo -E PATH="$PATH:/builder/isar/bitbake/bin:/builder/isar/scripts" /builder/isar/scripts/wic create ${WKS_FILE} --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -o ${DEPLOY_DIR_IMAGE} -e ${IMAGE_BASENAME} ${WIC_CREATE_EXTRA_ARGS}
+ isar-wic create ${WKS_FILE} --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -o ${DEPLOY_DIR_IMAGE} -e ${IMAGE_BASENAME} ${WIC_CREATE_EXTRA_ARGS}
}
addtask wic_image before do_build after do_copy_boot_files
diff --git a/meta/recipes-devtools/wic-tools/wic-tools.bb b/meta/recipes-devtools/wic-tools/wic-tools.bb
new file mode 100644
index 0000000..50ba664
--- /dev/null
+++ b/meta/recipes-devtools/wic-tools/wic-tools.bb
@@ -0,0 +1,4 @@
+# This software is a part of ISAR.
+# Copyright (C) 2018 Siemens AG
+# This is just a dummy because wic might call "bitbake -e wic-tools" to learn wic variables
+inherit wic-img
diff --git a/scripts/isar-wic b/scripts/isar-wic
new file mode 100755
index 0000000..4e4d0dd
--- /dev/null
+++ b/scripts/isar-wic
@@ -0,0 +1,27 @@
+#!/bin/sh
+#
+# This script is a wrapper to wic that prepares everything for Isar specific
+# needs.
+#
+# This software is a part of Isar.
+# Copyright (C) 2018 Siemens AG
+
+set -e
+
+[ -z $FAKEROOTCMD ] && FAKEROOTCMD="/builder/isar/scripts/isar-wic-handler"
+[ -z $TMPDIR ] && TMPDIR=$( mktemp -d )
+
+export MTOOLS_SKIP_CHECK=1
+
+# Play a dirty trick to redirect "du" and "mkfs.*" to FAKEROOTCMD
+TRICK_SYSROOT="${TMPDIR}/trick_wic_sysroot/"
+mkdir -p ${TRICK_SYSROOT}/sbin
+mkdir -p ${TRICK_SYSROOT}/usr/bin
+for fstype in btrfs ext2 ext3 ext4 vfat; do
+ ln -sf ${FAKEROOTCMD} ${TRICK_SYSROOT}/sbin/mkfs.${fstype}
+done
+ln -sf ${FAKEROOTCMD} ${TRICK_SYSROOT}/usr/bin/du
+
+export PATH="${TRICK_SYSROOT}/sbin:${TRICK_SYSROOT}/usr/sbin:${TRICK_SYSROOT}/usr/bin:${PATH}"
+
+exec wic $@
diff --git a/scripts/isar-wic-handler b/scripts/isar-wic-handler
new file mode 100755
index 0000000..01fe4fe
--- /dev/null
+++ b/scripts/isar-wic-handler
@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+#
+# This script is used to handle Isar specifics in wic without having to change
+# wic. It is called in two cases:
+# 1. if wic calls exec_native_cmd with pseudo != ""
+# 2. if wic calls exec_cmd on one of our trick symlinks
+#
+# This software is a part of Isar.
+# Copyright (C) 2018 Siemens AG
+#
+import os
+import sys
+import shutil
+import subprocess
+
+use_sudo = False
+native = False
+
+args = sys.argv
+args[0] = os.path.basename(args[0])
+
+# first thing we do is remove the PATH hack that took us here
+os.environ['PATH'] = ':'.join(os.environ['PATH'].split(':')[3:])
+
+if args[0] == 'isar-wic-handler':
+ native = True
+ args.pop(0)
+
+# run only "mkfs.*" and "du" with sudo, in "exec_native_cmd" and "exec_cmd"
+if (args[0].startswith('mkfs.') or args[0] == 'du'):
+ use_sudo = True
+else:
+ if not native:
+ print('ERROR: wic_fakeroot cmd "%s" not supported in non-native mode.'
+ % args[0], file=sys.stderr)
+ sys.exit(1)
+
+cmd = args[0]
+args.pop(0)
+
+# e2fsck <= 1.43.5 returns 1 on non-errors (stretch and before affected)
+# treat 1 as safe ... the filesystem was successfully repaired and is OK
+if cmd.startswith('fsck.'):
+ ret = subprocess.call([cmd] + args)
+ if ret == 0 or ret == 1:
+ sys.exit(0)
+ sys.exit(ret)
+
+if use_sudo:
+ args = ['-E', 'PATH="%s"' % os.environ['PATH'], cmd ] + args
+ cmd = 'sudo'
+
+os.execv(shutil.which(cmd), args)
diff --git a/scripts/wic_fakeroot b/scripts/wic_fakeroot
deleted file mode 100755
index 9e01c38..0000000
--- a/scripts/wic_fakeroot
+++ /dev/null
@@ -1,37 +0,0 @@
-#!/usr/bin/env python3
-#
-# wic needs a FAKEROOT cmd to run, the default is pseudo. In Isar we do/can not
-# use pseudo. And we call wic as root to begin with, so this script could be a
-# dummy doing nothing. It is almost a dummy ...
-#
-# If the fsck hack ever becomes obsolete, FAKEROOTCMD ?= "true;" can be used
-#
-# This software is a part of Isar.
-# Copyright (C) 2018 Siemens AG
-#
-import os
-import sys
-import shutil
-import subprocess
-
-args = sys.argv
-args.pop(0)
-cmd = args[0]
-
-# expect to be running as root
-# we could loosen that and execv(sudo, args) but even some early
-# "du"s fail, which do not use the fakeroot-wrapper
-# i.e. in wics partition.py the "du -ks" fails on
-# var/cache/apt/archives/partial
-# rootfs/root ...
-assert 'root' == os.environ["USER"]
-
-# e2fsck <= 1.43.5 returns 1 on non-errors (stretch and before affected)
-# treat 1 as safe ... the filesystem was successfully repaired and is OK
-if cmd.startswith('fsck.'):
- ret = subprocess.call(args)
- if ret == 0 or ret == 1:
- sys.exit(0)
- sys.exit(ret)
-
-os.execv(shutil.which(cmd), args)
--
2.13.6
^ permalink raw reply [flat|nested] 51+ messages in thread* Re: [PATCH] images: wic: limit use of sudo and enable manual call again
2018-02-01 12:41 ` [PATCH] images: wic: limit use of sudo and enable manual call again Henning Schild
@ 2018-02-01 12:44 ` Henning Schild
2018-02-01 16:09 ` Baurzhan Ismagulov
2018-02-12 19:07 ` Henning Schild
1 sibling, 1 reply; 51+ messages in thread
From: Henning Schild @ 2018-02-01 12:44 UTC (permalink / raw)
To: isar-users; +Cc: Baurzhan Ismagulov, Alexander Smirnov
This patch addresses the two main issues found in the reviews. The big
"sudo" and the broken "call it manually". The latter one is not fully
solved because users will have to call "isar-wic" instead of "wic".
I would even suggest to not fold this into the series and apply it on
top. It kind of shows some of the hacks required to wrap an unmodified
wic. The patch cleanly applies on top of the series i posted so far.
Henning
Am Thu, 1 Feb 2018 13:41:06 +0100
schrieb Henning Schild <henning.schild@siemens.com>:
> Issues:
> 1. after the wic rework wic was called under a big sudo
> 2. and calling it manually - like stated in the doc - did not work
> anymore
>
> Impact:
> This patch solves both issues. Just like before sudo is only used
> for "du" and "mkfs". And by applying some tricks and wrapping we now
> can call "isar-wic" just like "wic" before.
>
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
> doc/user_manual.md | 4 +--
> meta/classes/wic-img.bbclass | 6 ++--
> meta/recipes-devtools/wic-tools/wic-tools.bb | 4 +++
> scripts/isar-wic | 27 ++++++++++++++
> scripts/isar-wic-handler | 53
> ++++++++++++++++++++++++++++
> scripts/wic_fakeroot | 37 -------------------
> 6 files changed, 90 insertions(+), 41 deletions(-) create mode 100644
> meta/recipes-devtools/wic-tools/wic-tools.bb create mode 100755
> scripts/isar-wic create mode 100755 scripts/isar-wic-handler
> delete mode 100755 scripts/wic_fakeroot
>
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 969f6d2..e9284a0 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -208,9 +208,9 @@ Once the image artifacts have been built (c.f.
> previous section), full EFI disk Currently, only the `i386` and
> `amd64` target architectures are supported: ```
> # Generate an EFI image for the `i386` target architecture
> - $ wic create -D sdimage-efi -o . -e
> multiconfig:qemui386-stretch:isar-image-base
> + $ isar-wic create -D sdimage-efi -o . -e
> multiconfig:qemui386-stretch:isar-image-base # Similarly, for the
> `amd64` target architecture
> - $ wic create -D sdimage-efi -o . -e
> multiconfig:qemuamd64-stretch:isar-image-base
> + $ isar-wic create -D sdimage-efi -o . -e
> multiconfig:qemuamd64-stretch:isar-image-base ```
>
> In order to run the images with `qemu`, an EFI firmware is required
> and available at the following address: diff --git
> a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass index
> e8d2678..d4afde6 100644 --- a/meta/classes/wic-img.bbclass
> +++ b/meta/classes/wic-img.bbclass
> @@ -14,7 +14,7 @@ STAGING_DATADIR ?= "/usr/share/"
> STAGING_LIBDIR ?= "/usr/lib/"
> STAGING_DIR ?= "${TMPDIR}"
> IMAGE_BASENAME ?= "multiconfig:${MACHINE}-${DISTRO}:${PN}"
> -FAKEROOTCMD ?= "wic_fakeroot"
> +FAKEROOTCMD ?= "/builder/isar/scripts/isar-wic-handler"
> RECIPE_SYSROOT_NATIVE ?= "/"
>
> do_wic_image[stamp-extra-info] = "${DISTRO}-${MACHINE}"
> @@ -57,8 +57,10 @@ do_rootfs_wicenv[prefuncs] = 'set_image_size'
>
> do_wic_image() {
> export BUILDDIR="${BUILDDIR}"
> + export FAKEROOTCMD="${FAKEROOTCMD}"
> + export TMPDIR="${TMPDIR}"
>
> - sudo -E
> PATH="$PATH:/builder/isar/bitbake/bin:/builder/isar/scripts" /builder/isar/scripts/wic
> create ${WKS_FILE} --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -o
> ${DEPLOY_DIR_IMAGE} -e ${IMAGE_BASENAME} ${WIC_CREATE_EXTRA_ARGS}
> + isar-wic create ${WKS_FILE} --vars
> "${STAGING_DIR}/${MACHINE}/imgdata/" -o ${DEPLOY_DIR_IMAGE} -e
> ${IMAGE_BASENAME} ${WIC_CREATE_EXTRA_ARGS} }
> addtask wic_image before do_build after do_copy_boot_files
> diff --git a/meta/recipes-devtools/wic-tools/wic-tools.bb
> b/meta/recipes-devtools/wic-tools/wic-tools.bb new file mode 100644
> index 0000000..50ba664
> --- /dev/null
> +++ b/meta/recipes-devtools/wic-tools/wic-tools.bb
> @@ -0,0 +1,4 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2018 Siemens AG
> +# This is just a dummy because wic might call "bitbake -e
> wic-tools" to learn wic variables +inherit wic-img
> diff --git a/scripts/isar-wic b/scripts/isar-wic
> new file mode 100755
> index 0000000..4e4d0dd
> --- /dev/null
> +++ b/scripts/isar-wic
> @@ -0,0 +1,27 @@
> +#!/bin/sh
> +#
> +# This script is a wrapper to wic that prepares everything for Isar
> specific +# needs.
> +#
> +# This software is a part of Isar.
> +# Copyright (C) 2018 Siemens AG
> +
> +set -e
> +
> +[ -z $FAKEROOTCMD ] &&
> FAKEROOTCMD="/builder/isar/scripts/isar-wic-handler" +[ -z $TMPDIR ]
> && TMPDIR=$( mktemp -d ) +
> +export MTOOLS_SKIP_CHECK=1
> +
> +# Play a dirty trick to redirect "du" and "mkfs.*" to FAKEROOTCMD
> +TRICK_SYSROOT="${TMPDIR}/trick_wic_sysroot/"
> +mkdir -p ${TRICK_SYSROOT}/sbin
> +mkdir -p ${TRICK_SYSROOT}/usr/bin
> +for fstype in btrfs ext2 ext3 ext4 vfat; do
> + ln -sf ${FAKEROOTCMD} ${TRICK_SYSROOT}/sbin/mkfs.${fstype}
> +done
> +ln -sf ${FAKEROOTCMD} ${TRICK_SYSROOT}/usr/bin/du
> +
> +export
> PATH="${TRICK_SYSROOT}/sbin:${TRICK_SYSROOT}/usr/sbin:${TRICK_SYSROOT}/usr/bin:${PATH}"
> + +exec wic $@
> diff --git a/scripts/isar-wic-handler b/scripts/isar-wic-handler
> new file mode 100755
> index 0000000..01fe4fe
> --- /dev/null
> +++ b/scripts/isar-wic-handler
> @@ -0,0 +1,53 @@
> +#!/usr/bin/env python3
> +#
> +# This script is used to handle Isar specifics in wic without having
> to change +# wic. It is called in two cases:
> +# 1. if wic calls exec_native_cmd with pseudo != ""
> +# 2. if wic calls exec_cmd on one of our trick symlinks
> +#
> +# This software is a part of Isar.
> +# Copyright (C) 2018 Siemens AG
> +#
> +import os
> +import sys
> +import shutil
> +import subprocess
> +
> +use_sudo = False
> +native = False
> +
> +args = sys.argv
> +args[0] = os.path.basename(args[0])
> +
> +# first thing we do is remove the PATH hack that took us here
> +os.environ['PATH'] = ':'.join(os.environ['PATH'].split(':')[3:])
> +
> +if args[0] == 'isar-wic-handler':
> + native = True
> + args.pop(0)
> +
> +# run only "mkfs.*" and "du" with sudo, in "exec_native_cmd" and
> "exec_cmd" +if (args[0].startswith('mkfs.') or args[0] == 'du'):
> + use_sudo = True
> +else:
> + if not native:
> + print('ERROR: wic_fakeroot cmd "%s" not supported in
> non-native mode.'
> + % args[0], file=sys.stderr)
> + sys.exit(1)
> +
> +cmd = args[0]
> +args.pop(0)
> +
> +# e2fsck <= 1.43.5 returns 1 on non-errors (stretch and before
> affected) +# treat 1 as safe ... the filesystem was successfully
> repaired and is OK +if cmd.startswith('fsck.'):
> + ret = subprocess.call([cmd] + args)
> + if ret == 0 or ret == 1:
> + sys.exit(0)
> + sys.exit(ret)
> +
> +if use_sudo:
> + args = ['-E', 'PATH="%s"' % os.environ['PATH'], cmd ] + args
> + cmd = 'sudo'
> +
> +os.execv(shutil.which(cmd), args)
> diff --git a/scripts/wic_fakeroot b/scripts/wic_fakeroot
> deleted file mode 100755
> index 9e01c38..0000000
> --- a/scripts/wic_fakeroot
> +++ /dev/null
> @@ -1,37 +0,0 @@
> -#!/usr/bin/env python3
> -#
> -# wic needs a FAKEROOT cmd to run, the default is pseudo. In Isar we
> do/can not -# use pseudo. And we call wic as root to begin with, so
> this script could be a -# dummy doing nothing. It is almost a
> dummy ... -#
> -# If the fsck hack ever becomes obsolete, FAKEROOTCMD ?= "true;" can
> be used -#
> -# This software is a part of Isar.
> -# Copyright (C) 2018 Siemens AG
> -#
> -import os
> -import sys
> -import shutil
> -import subprocess
> -
> -args = sys.argv
> -args.pop(0)
> -cmd = args[0]
> -
> -# expect to be running as root
> -# we could loosen that and execv(sudo, args) but even some early
> -# "du"s fail, which do not use the fakeroot-wrapper
> -# i.e. in wics partition.py the "du -ks" fails on
> -# var/cache/apt/archives/partial
> -# rootfs/root ...
> -assert 'root' == os.environ["USER"]
> -
> -# e2fsck <= 1.43.5 returns 1 on non-errors (stretch and before
> affected) -# treat 1 as safe ... the filesystem was successfully
> repaired and is OK -if cmd.startswith('fsck.'):
> - ret = subprocess.call(args)
> - if ret == 0 or ret == 1:
> - sys.exit(0)
> - sys.exit(ret)
> -
> -os.execv(shutil.which(cmd), args)
^ permalink raw reply [flat|nested] 51+ messages in thread* Re: [PATCH] images: wic: limit use of sudo and enable manual call again
2018-02-01 12:44 ` Henning Schild
@ 2018-02-01 16:09 ` Baurzhan Ismagulov
2018-02-01 18:10 ` Henning Schild
0 siblings, 1 reply; 51+ messages in thread
From: Baurzhan Ismagulov @ 2018-02-01 16:09 UTC (permalink / raw)
To: isar-users
On Thu, Feb 01, 2018 at 01:44:59PM +0100, Henning Schild wrote:
> This patch addresses the two main issues found in the reviews. The big
> "sudo" and the broken "call it manually". The latter one is not fully
> solved because users will have to call "isar-wic" instead of "wic".
>
> I would even suggest to not fold this into the series and apply it on
> top. It kind of shows some of the hacks required to wrap an unmodified
> wic. The patch cleanly applies on top of the series i posted so far.
Wow, that's an interesting approach, thanks for the effort. Splitting wic and
isar-wic reduces user surprise, which would be a good thing. Do I understand
correctly, you replace du and mkfs.* with your own versions and execute the
respective command with superuser privileges? Let me read the code in more
detail, but if yes, I'm not sure the complexity is worth the use case.
I haven't finished reading the first series yet. Some background questions:
1. Are there any disadvantages of using wic, compared to the old-school
ext4-img?
2. AFAICT, Yocto still maintains both wic and image-types.bbclass. Why don't
they move to wic completely?
3. With wic-img.bbclass, how does one choose fs type? E.g., quickly generate
the same image on ext4 and ext3.
4. How to add a new fs type to wic?
I see that you add wic-img.bbclass, but no image uses it. Would it be useful to
migrate all images to wic, remove ext4-img, and remove wic usage from the docs?
If that solution would be "clean" and we have answers to the questions above, I
could possibly live with all-in sudo, although I'd still prefer keeping the
sudo hack for a while.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH] images: wic: limit use of sudo and enable manual call again
2018-02-01 16:09 ` Baurzhan Ismagulov
@ 2018-02-01 18:10 ` Henning Schild
2018-02-01 18:55 ` Henning Schild
0 siblings, 1 reply; 51+ messages in thread
From: Henning Schild @ 2018-02-01 18:10 UTC (permalink / raw)
To: Baurzhan Ismagulov; +Cc: isar-users
Am Thu, 1 Feb 2018 17:09:22 +0100
schrieb Baurzhan Ismagulov <ibr@radix50.net>:
> On Thu, Feb 01, 2018 at 01:44:59PM +0100, Henning Schild wrote:
> > This patch addresses the two main issues found in the reviews. The
> > big "sudo" and the broken "call it manually". The latter one is not
> > fully solved because users will have to call "isar-wic" instead of
> > "wic".
> >
> > I would even suggest to not fold this into the series and apply it
> > on top. It kind of shows some of the hacks required to wrap an
> > unmodified wic. The patch cleanly applies on top of the series i
> > posted so far.
>
> Wow, that's an interesting approach, thanks for the effort. Splitting
> wic and isar-wic reduces user surprise, which would be a good thing.
Since i wanted wic to stay untouched i had to introduce isar-wic to
wrap it. It prepares some of the hacks we need in Isar, that code got
moved out of the bbclass into isar-wic.
> Do I understand correctly, you replace du and mkfs.* with your own
> versions and execute the respective command with superuser
> privileges? Let me read the code in more detail, but if yes, I'm not
> sure the complexity is worth the use case.
"du" and "mkfs" are symlinks to "isar-wic-handler". They are placed in
PATH before the real ones so wic calls end up in the handler. The
handler just executes them under sudo.
The other thing the handler does is act as fakeroot cmd where it does
not bail out on ret "1" from fsck.
Both hacks that where previously in wic.
> I haven't finished reading the first series yet. Some background
> questions:
>
> 1. Are there any disadvantages of using wic, compared to the
> old-school ext4-img?
That is still available as image type, however wic has the potential to
replace that eventually. As far as i understood wic builds disks not
partitions and ext4 builds a partition only.
> 2. AFAICT, Yocto still maintains both wic and image-types.bbclass.
> Why don't they move to wic completely?
I think because of disk vs. partition. You can build a disk with just
one partition and use the intermediate file as output, but i think our
wic version does not support that.
My guess is partitions where there before wic and nobody cared to clean
up. And there might be image types that wic can not do i.e. tarball
> 3. With wic-img.bbclass, how does one choose fs type? E.g., quickly
> generate the same image on ext4 and ext3.
You could write an image like that one:
> IMAGE_TYPE = 'wic-img'
> require recipes-core/images/isar-image-base.bb
> WKS_FILE = "sdimage-efi"
or with the other patch i sent
> WKS_FILE = "directdisk-isar"
For i.e. ext3 just copy the wks file and change the fstype in there.
You could move sdimage-efi.wks to sdimage-efi-ext3.wks
> 4. How to add a new fs type to wic?
This can be done with a wic plugin in any layer. Ideally one should
think whether this should be an upstream wic patch, submit it and
eventually update wic in Isar.
But this is more a wic-question, not really an Isar one.
> I see that you add wic-img.bbclass, but no image uses it. Would it be
> useful to migrate all images to wic, remove ext4-img, and remove wic
> usage from the docs? If that solution would be "clean" and we have
> answers to the questions above, I could possibly live with all-in
> sudo, although I'd still prefer keeping the sudo hack for a while.
True, i did not explain how to test this. Please refer to the example i
gave above.
For moving all images to wic, that sounds like a tempting idea. But
that needs more work.
1. the question of partitions vs. disks, as i said an update of wic
could potentially help here, because you can write in your wks that
you want to keep the partition images ... or a disk-plugin that can
handle exactly one partition and does not try to add bootloader/disk
stuff
2. the image file names are generated from wks-file-name, date,
disk-plugin-name and they do not contain all the exta-stamps to
play well with multiconfig ... i would be happy to assign this one
to people that insist on that complexity ;)
btw. all the kernels and initrds that currently end up in the
deploy/images have issues there as well
Next thing on my plate would be to make wic fetch its artifacts only
from buildchroot or rootfs. sdimage-efi actually installs the
bootloader from the host into the target! And enable legacy bios
images, a first patch was already sent.
wic plugins that deal with bootloaders very much depend on the way
there where installed, and debian installs them not the same way OE
does. So these plugins need to get forked.
For dropping manual calls to wic and removing it from the docs:
I made some effort to keep manual wic-usage, it has its value for devs
that might want to create multiple images from one rootfs probaly also
users. I would like to keep that.
Coming up with this version was a lot of trial and error. If it ends up
being not maintainable or causes trouble we can go back to the big
sudo.
For the series and this patch i would like to ask wic-users to try it
out and give feedback. Once wic plays well with multiconfig we can
integrate it into the default builds and CI.
Henning
> With kind regards,
> Baurzhan.
>
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH] images: wic: limit use of sudo and enable manual call again
2018-02-01 18:10 ` Henning Schild
@ 2018-02-01 18:55 ` Henning Schild
0 siblings, 0 replies; 51+ messages in thread
From: Henning Schild @ 2018-02-01 18:55 UTC (permalink / raw)
To: Baurzhan Ismagulov; +Cc: isar-users
Am Thu, 1 Feb 2018 19:10:29 +0100
schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
> Am Thu, 1 Feb 2018 17:09:22 +0100
> schrieb Baurzhan Ismagulov <ibr@radix50.net>:
>
> > On Thu, Feb 01, 2018 at 01:44:59PM +0100, Henning Schild wrote:
> > > This patch addresses the two main issues found in the reviews. The
> > > big "sudo" and the broken "call it manually". The latter one is
> > > not fully solved because users will have to call "isar-wic"
> > > instead of "wic".
> > >
> > > I would even suggest to not fold this into the series and apply it
> > > on top. It kind of shows some of the hacks required to wrap an
> > > unmodified wic. The patch cleanly applies on top of the series i
> > > posted so far.
> >
> > Wow, that's an interesting approach, thanks for the effort.
> > Splitting wic and isar-wic reduces user surprise, which would be a
> > good thing.
>
> Since i wanted wic to stay untouched i had to introduce isar-wic to
> wrap it. It prepares some of the hacks we need in Isar, that code got
> moved out of the bbclass into isar-wic.
>
> > Do I understand correctly, you replace du and mkfs.* with your own
> > versions and execute the respective command with superuser
> > privileges? Let me read the code in more detail, but if yes, I'm not
> > sure the complexity is worth the use case.
>
> "du" and "mkfs" are symlinks to "isar-wic-handler". They are placed in
> PATH before the real ones so wic calls end up in the handler. The
> handler just executes them under sudo.
> The other thing the handler does is act as fakeroot cmd where it does
> not bail out on ret "1" from fsck.
> Both hacks that where previously in wic.
>
> > I haven't finished reading the first series yet. Some background
> > questions:
> >
> > 1. Are there any disadvantages of using wic, compared to the
> > old-school ext4-img?
>
> That is still available as image type, however wic has the potential
> to replace that eventually. As far as i understood wic builds disks
> not partitions and ext4 builds a partition only.
>
> > 2. AFAICT, Yocto still maintains both wic and image-types.bbclass.
> > Why don't they move to wic completely?
>
> I think because of disk vs. partition. You can build a disk with just
> one partition and use the intermediate file as output, but i think our
> wic version does not support that.
> My guess is partitions where there before wic and nobody cared to
> clean up. And there might be image types that wic can not do i.e.
> tarball
>
> > 3. With wic-img.bbclass, how does one choose fs type? E.g., quickly
> > generate the same image on ext4 and ext3.
>
> You could write an image like that one:
> > IMAGE_TYPE = 'wic-img'
> > require recipes-core/images/isar-image-base.bb
> > WKS_FILE = "sdimage-efi"
> or with the other patch i sent
> > WKS_FILE = "directdisk-isar"
>
> For i.e. ext3 just copy the wks file and change the fstype in there.
> You could move sdimage-efi.wks to sdimage-efi-ext3.wks
>
> > 4. How to add a new fs type to wic?
>
> This can be done with a wic plugin in any layer. Ideally one should
> think whether this should be an upstream wic patch, submit it and
> eventually update wic in Isar.
> But this is more a wic-question, not really an Isar one.
>
> > I see that you add wic-img.bbclass, but no image uses it. Would it
> > be useful to migrate all images to wic, remove ext4-img, and remove
> > wic usage from the docs? If that solution would be "clean" and we
> > have answers to the questions above, I could possibly live with
> > all-in sudo, although I'd still prefer keeping the sudo hack for a
> > while.
>
> True, i did not explain how to test this. Please refer to the example
> i gave above.
> For moving all images to wic, that sounds like a tempting idea. But
> that needs more work.
> 1. the question of partitions vs. disks, as i said an update of wic
> could potentially help here, because you can write in your wks that
> you want to keep the partition images ... or a disk-plugin that can
> handle exactly one partition and does not try to add
> bootloader/disk stuff
> 2. the image file names are generated from wks-file-name, date,
> disk-plugin-name and they do not contain all the exta-stamps to
> play well with multiconfig ... i would be happy to assign this one
> to people that insist on that complexity ;)
> btw. all the kernels and initrds that currently end up in the
> deploy/images have issues there as well
>
> Next thing on my plate would be to make wic fetch its artifacts only
> from buildchroot or rootfs. sdimage-efi actually installs the
> bootloader from the host into the target! And enable legacy bios
> images, a first patch was already sent.
> wic plugins that deal with bootloaders very much depend on the way
> there where installed, and debian installs them not the same way OE
> does. So these plugins need to get forked.
An another big issue in Isar is that the kernel and initrd are not
actually taken from the rootfs. They kind of are but you end up with
copies. For ext4-img builds you call qemu with a kernel, initrd and
rootfs while the rootfs contains kernel and initrd. (should be removed
in that case)
For wic builds you create a boot-partition with copies from
the /boot-folder of rootfs, that is done by wic behind the back of
debian. Again you end up with these files twice, and you get an image
where an apt-get kernel update would never work or render the machine
unusable. Debian needs to know about the boot-partition and probably
install the bootloader itself. That can be addressed with tuning the
wks-files and wic-plugins.
So lots to do ... i am just dumping it here for later reference ;).
Henning
> For dropping manual calls to wic and removing it from the docs:
> I made some effort to keep manual wic-usage, it has its value for devs
> that might want to create multiple images from one rootfs probaly also
> users. I would like to keep that.
>
> Coming up with this version was a lot of trial and error. If it ends
> up being not maintainable or causes trouble we can go back to the big
> sudo.
>
> For the series and this patch i would like to ask wic-users to try it
> out and give feedback. Once wic plays well with multiconfig we can
> integrate it into the default builds and CI.
>
> Henning
>
> > With kind regards,
> > Baurzhan.
> >
>
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH] images: wic: limit use of sudo and enable manual call again
2018-02-01 12:41 ` [PATCH] images: wic: limit use of sudo and enable manual call again Henning Schild
2018-02-01 12:44 ` Henning Schild
@ 2018-02-12 19:07 ` Henning Schild
1 sibling, 0 replies; 51+ messages in thread
From: Henning Schild @ 2018-02-12 19:07 UTC (permalink / raw)
To: isar-users; +Cc: Baurzhan Ismagulov, Alexander Smirnov
In this one i tried to get rid of sudo as far as possible. But thinking
about what patches will have to follow in the future, sudo will most
likely be reintroduced. And here is why:
wic has the concept of a native_sysroot where tools like mkfs, parted,
or grub-install etc. are found. These tools have a direct influence on
the image layout and content and should match the image as close as
possible. Right now we just take the tools and the bootloader from the
host, which is clearly wrong. And our patched version wic did not even
support the native_sysroot anymore.
In order to get as close as we possibly can, i suggest using
buildchroot as native_sysroot. And the way to execute stuff in there is
to call "sudo chroot", so a lot of sudos will have to come back. As a
result we will not need grub-efi and other partitioning related things
on the host anymore.
Henning
Am Thu, 1 Feb 2018 13:41:06 +0100
schrieb Henning Schild <henning.schild@siemens.com>:
> Issues:
> 1. after the wic rework wic was called under a big sudo
> 2. and calling it manually - like stated in the doc - did not work
> anymore
>
> Impact:
> This patch solves both issues. Just like before sudo is only used
> for "du" and "mkfs". And by applying some tricks and wrapping we now
> can call "isar-wic" just like "wic" before.
>
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
> doc/user_manual.md | 4 +--
> meta/classes/wic-img.bbclass | 6 ++--
> meta/recipes-devtools/wic-tools/wic-tools.bb | 4 +++
> scripts/isar-wic | 27 ++++++++++++++
> scripts/isar-wic-handler | 53
> ++++++++++++++++++++++++++++
> scripts/wic_fakeroot | 37 -------------------
> 6 files changed, 90 insertions(+), 41 deletions(-) create mode 100644
> meta/recipes-devtools/wic-tools/wic-tools.bb create mode 100755
> scripts/isar-wic create mode 100755 scripts/isar-wic-handler
> delete mode 100755 scripts/wic_fakeroot
>
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 969f6d2..e9284a0 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -208,9 +208,9 @@ Once the image artifacts have been built (c.f.
> previous section), full EFI disk Currently, only the `i386` and
> `amd64` target architectures are supported: ```
> # Generate an EFI image for the `i386` target architecture
> - $ wic create -D sdimage-efi -o . -e
> multiconfig:qemui386-stretch:isar-image-base
> + $ isar-wic create -D sdimage-efi -o . -e
> multiconfig:qemui386-stretch:isar-image-base # Similarly, for the
> `amd64` target architecture
> - $ wic create -D sdimage-efi -o . -e
> multiconfig:qemuamd64-stretch:isar-image-base
> + $ isar-wic create -D sdimage-efi -o . -e
> multiconfig:qemuamd64-stretch:isar-image-base ```
>
> In order to run the images with `qemu`, an EFI firmware is required
> and available at the following address: diff --git
> a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass index
> e8d2678..d4afde6 100644 --- a/meta/classes/wic-img.bbclass
> +++ b/meta/classes/wic-img.bbclass
> @@ -14,7 +14,7 @@ STAGING_DATADIR ?= "/usr/share/"
> STAGING_LIBDIR ?= "/usr/lib/"
> STAGING_DIR ?= "${TMPDIR}"
> IMAGE_BASENAME ?= "multiconfig:${MACHINE}-${DISTRO}:${PN}"
> -FAKEROOTCMD ?= "wic_fakeroot"
> +FAKEROOTCMD ?= "/builder/isar/scripts/isar-wic-handler"
> RECIPE_SYSROOT_NATIVE ?= "/"
>
> do_wic_image[stamp-extra-info] = "${DISTRO}-${MACHINE}"
> @@ -57,8 +57,10 @@ do_rootfs_wicenv[prefuncs] = 'set_image_size'
>
> do_wic_image() {
> export BUILDDIR="${BUILDDIR}"
> + export FAKEROOTCMD="${FAKEROOTCMD}"
> + export TMPDIR="${TMPDIR}"
>
> - sudo -E
> PATH="$PATH:/builder/isar/bitbake/bin:/builder/isar/scripts" /builder/isar/scripts/wic
> create ${WKS_FILE} --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -o
> ${DEPLOY_DIR_IMAGE} -e ${IMAGE_BASENAME} ${WIC_CREATE_EXTRA_ARGS}
> + isar-wic create ${WKS_FILE} --vars
> "${STAGING_DIR}/${MACHINE}/imgdata/" -o ${DEPLOY_DIR_IMAGE} -e
> ${IMAGE_BASENAME} ${WIC_CREATE_EXTRA_ARGS} }
> addtask wic_image before do_build after do_copy_boot_files
> diff --git a/meta/recipes-devtools/wic-tools/wic-tools.bb
> b/meta/recipes-devtools/wic-tools/wic-tools.bb new file mode 100644
> index 0000000..50ba664
> --- /dev/null
> +++ b/meta/recipes-devtools/wic-tools/wic-tools.bb
> @@ -0,0 +1,4 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2018 Siemens AG
> +# This is just a dummy because wic might call "bitbake -e
> wic-tools" to learn wic variables +inherit wic-img
> diff --git a/scripts/isar-wic b/scripts/isar-wic
> new file mode 100755
> index 0000000..4e4d0dd
> --- /dev/null
> +++ b/scripts/isar-wic
> @@ -0,0 +1,27 @@
> +#!/bin/sh
> +#
> +# This script is a wrapper to wic that prepares everything for Isar
> specific +# needs.
> +#
> +# This software is a part of Isar.
> +# Copyright (C) 2018 Siemens AG
> +
> +set -e
> +
> +[ -z $FAKEROOTCMD ] &&
> FAKEROOTCMD="/builder/isar/scripts/isar-wic-handler" +[ -z $TMPDIR ]
> && TMPDIR=$( mktemp -d ) +
> +export MTOOLS_SKIP_CHECK=1
> +
> +# Play a dirty trick to redirect "du" and "mkfs.*" to FAKEROOTCMD
> +TRICK_SYSROOT="${TMPDIR}/trick_wic_sysroot/"
> +mkdir -p ${TRICK_SYSROOT}/sbin
> +mkdir -p ${TRICK_SYSROOT}/usr/bin
> +for fstype in btrfs ext2 ext3 ext4 vfat; do
> + ln -sf ${FAKEROOTCMD} ${TRICK_SYSROOT}/sbin/mkfs.${fstype}
> +done
> +ln -sf ${FAKEROOTCMD} ${TRICK_SYSROOT}/usr/bin/du
> +
> +export
> PATH="${TRICK_SYSROOT}/sbin:${TRICK_SYSROOT}/usr/sbin:${TRICK_SYSROOT}/usr/bin:${PATH}"
> + +exec wic $@
> diff --git a/scripts/isar-wic-handler b/scripts/isar-wic-handler
> new file mode 100755
> index 0000000..01fe4fe
> --- /dev/null
> +++ b/scripts/isar-wic-handler
> @@ -0,0 +1,53 @@
> +#!/usr/bin/env python3
> +#
> +# This script is used to handle Isar specifics in wic without having
> to change +# wic. It is called in two cases:
> +# 1. if wic calls exec_native_cmd with pseudo != ""
> +# 2. if wic calls exec_cmd on one of our trick symlinks
> +#
> +# This software is a part of Isar.
> +# Copyright (C) 2018 Siemens AG
> +#
> +import os
> +import sys
> +import shutil
> +import subprocess
> +
> +use_sudo = False
> +native = False
> +
> +args = sys.argv
> +args[0] = os.path.basename(args[0])
> +
> +# first thing we do is remove the PATH hack that took us here
> +os.environ['PATH'] = ':'.join(os.environ['PATH'].split(':')[3:])
> +
> +if args[0] == 'isar-wic-handler':
> + native = True
> + args.pop(0)
> +
> +# run only "mkfs.*" and "du" with sudo, in "exec_native_cmd" and
> "exec_cmd" +if (args[0].startswith('mkfs.') or args[0] == 'du'):
> + use_sudo = True
> +else:
> + if not native:
> + print('ERROR: wic_fakeroot cmd "%s" not supported in
> non-native mode.'
> + % args[0], file=sys.stderr)
> + sys.exit(1)
> +
> +cmd = args[0]
> +args.pop(0)
> +
> +# e2fsck <= 1.43.5 returns 1 on non-errors (stretch and before
> affected) +# treat 1 as safe ... the filesystem was successfully
> repaired and is OK +if cmd.startswith('fsck.'):
> + ret = subprocess.call([cmd] + args)
> + if ret == 0 or ret == 1:
> + sys.exit(0)
> + sys.exit(ret)
> +
> +if use_sudo:
> + args = ['-E', 'PATH="%s"' % os.environ['PATH'], cmd ] + args
> + cmd = 'sudo'
> +
> +os.execv(shutil.which(cmd), args)
> diff --git a/scripts/wic_fakeroot b/scripts/wic_fakeroot
> deleted file mode 100755
> index 9e01c38..0000000
> --- a/scripts/wic_fakeroot
> +++ /dev/null
> @@ -1,37 +0,0 @@
> -#!/usr/bin/env python3
> -#
> -# wic needs a FAKEROOT cmd to run, the default is pseudo. In Isar we
> do/can not -# use pseudo. And we call wic as root to begin with, so
> this script could be a -# dummy doing nothing. It is almost a
> dummy ... -#
> -# If the fsck hack ever becomes obsolete, FAKEROOTCMD ?= "true;" can
> be used -#
> -# This software is a part of Isar.
> -# Copyright (C) 2018 Siemens AG
> -#
> -import os
> -import sys
> -import shutil
> -import subprocess
> -
> -args = sys.argv
> -args.pop(0)
> -cmd = args[0]
> -
> -# expect to be running as root
> -# we could loosen that and execv(sudo, args) but even some early
> -# "du"s fail, which do not use the fakeroot-wrapper
> -# i.e. in wics partition.py the "du -ks" fails on
> -# var/cache/apt/archives/partial
> -# rootfs/root ...
> -assert 'root' == os.environ["USER"]
> -
> -# e2fsck <= 1.43.5 returns 1 on non-errors (stretch and before
> affected) -# treat 1 as safe ... the filesystem was successfully
> repaired and is OK -if cmd.startswith('fsck.'):
> - ret = subprocess.call(args)
> - if ret == 0 or ret == 1:
> - sys.exit(0)
> - sys.exit(ret)
> -
> -os.execv(shutil.which(cmd), args)
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-01-31 9:41 [PATCH 0/9] first wic integration Henning Schild
` (10 preceding siblings ...)
2018-02-01 12:41 ` [PATCH] images: wic: limit use of sudo and enable manual call again Henning Schild
@ 2018-02-12 17:27 ` Henning Schild
2018-02-12 18:21 ` Alexander Smirnov
11 siblings, 1 reply; 51+ messages in thread
From: Henning Schild @ 2018-02-12 17:27 UTC (permalink / raw)
To: isar-users
This series of patches is working and does apply on current next, the
mentioned open points are not new and therefore will be fixed in follow
up patches. I am waiting from review comments.
Henning
Am Wed, 31 Jan 2018 10:41:51 +0100
schrieb Henning Schild <henning.schild@siemens.com>:
> This is the first official post of a series that aims to do two
> things. 1. integrate wic into Isar so it does not need to be called
> after bitbake
> 2. get our copy of wic back to an unmodified version from OE
>
> Many people have been asking for the first, so i think i do not need
> to motivate that change. Things this series already brings is control
> over the partition sizes, that we did not have before.
> The choice in working wks files is still limited. The plugins dealing
> with bootloaders (isolinux, syslinux etc.) will probably need to be
> forked (like the efigrub one).
>
> The second one is important so we can update wic in the future and
> developers do not get confused. Having a fork instead of a copy,
> without a very good reason, is not a good idea. I found that wic does
> not need to be changed and plugins can be forked.
>
> Henning
>
> Henning Schild (9):
> classes: image: introduce size measuring function, for before
> do_*_image
> images: new class wic-img for wic intregration
> wic: add a bootimg-efi-isar plugin outside the wic tree
> Revert "wic: Make the bootimg-efi plugin generate usable images"
> Revert "wic: Introduce the `WicExecError` exception class"
> Revert "wic: Work around mcopy error"
> Revert "wic: Use sudo instead of pseudo"
> Revert "wic: Remove sysroot support"
> wic: now truly go for the wic version we claim to have
>
> .../scripts/lib/wic/canned-wks/sdimage-efi.wks | 2 +-
> .../lib/wic/plugins/source/bootimg-efi-isar.py | 307
> +++++++++++++++++++++
> meta/classes/ext4-img.bbclass | 7 +-
> meta/classes/image.bbclass | 19 ++
> meta/classes/wic-img.bbclass | 64 +++++
> scripts/lib/wic/canned-wks/qemux86-directdisk.wks | 2 +-
> scripts/lib/wic/engine.py | 7 +-
> scripts/lib/wic/filemap.py | 6 +-
> scripts/lib/wic/help.py | 2 -
> scripts/lib/wic/ksparser.py | 4 +-
> scripts/lib/wic/partition.py | 197
> ++++++++----- scripts/lib/wic/pluginbase.py |
> 11 +- scripts/lib/wic/plugins/imager/direct.py | 66 +++--
> scripts/lib/wic/plugins/source/bootimg-efi.py | 78
> +----- .../lib/wic/plugins/source/bootimg-partition.py | 6 +-
> scripts/lib/wic/plugins/source/bootimg-pcbios.py | 68 ++---
> scripts/lib/wic/plugins/source/fsimage.py | 56
> ++++ .../lib/wic/plugins/source/isoimage-isohybrid.py | 28 +-
> scripts/lib/wic/plugins/source/rawcopy.py | 2 +-
> scripts/lib/wic/plugins/source/rootfs.py | 4 +-
> scripts/lib/wic/utils/misc.py | 33 +--
> scripts/lib/wic/utils/runner.py | 74 ++++-
> scripts/wic | 27 +-
> scripts/wic_fakeroot | 37 +++ 24 files
> changed, 846 insertions(+), 261 deletions(-) create mode 100644
> meta-isar/scripts/lib/wic/plugins/source/bootimg-efi-isar.py create
> mode 100644 meta/classes/wic-img.bbclass create mode 100644
> scripts/lib/wic/plugins/source/fsimage.py create mode 100755
> scripts/wic_fakeroot
>
^ permalink raw reply [flat|nested] 51+ messages in thread* Re: [PATCH 0/9] first wic integration
2018-02-12 17:27 ` [PATCH 0/9] first wic integration Henning Schild
@ 2018-02-12 18:21 ` Alexander Smirnov
2018-02-12 18:30 ` Henning Schild
0 siblings, 1 reply; 51+ messages in thread
From: Alexander Smirnov @ 2018-02-12 18:21 UTC (permalink / raw)
To: Henning Schild, isar-users
On 02/12/2018 08:27 PM, Henning Schild wrote:
> This series of patches is working and does apply on current next, the
> mentioned open points are not new and therefore will be fixed in follow
> up patches. I am waiting from review comments.
I need at least one test for this series to be sure that CI checks wic
image every time Isar is updated. Could you propose the machine to
switch to wic?
Alex
>
> Henning
>
> Am Wed, 31 Jan 2018 10:41:51 +0100
> schrieb Henning Schild <henning.schild@siemens.com>:
>
>> This is the first official post of a series that aims to do two
>> things. 1. integrate wic into Isar so it does not need to be called
>> after bitbake
>> 2. get our copy of wic back to an unmodified version from OE
>>
>> Many people have been asking for the first, so i think i do not need
>> to motivate that change. Things this series already brings is control
>> over the partition sizes, that we did not have before.
>> The choice in working wks files is still limited. The plugins dealing
>> with bootloaders (isolinux, syslinux etc.) will probably need to be
>> forked (like the efigrub one).
>>
>> The second one is important so we can update wic in the future and
>> developers do not get confused. Having a fork instead of a copy,
>> without a very good reason, is not a good idea. I found that wic does
>> not need to be changed and plugins can be forked.
>>
>> Henning
>>
>> Henning Schild (9):
>> classes: image: introduce size measuring function, for before
>> do_*_image
>> images: new class wic-img for wic intregration
>> wic: add a bootimg-efi-isar plugin outside the wic tree
>> Revert "wic: Make the bootimg-efi plugin generate usable images"
>> Revert "wic: Introduce the `WicExecError` exception class"
>> Revert "wic: Work around mcopy error"
>> Revert "wic: Use sudo instead of pseudo"
>> Revert "wic: Remove sysroot support"
>> wic: now truly go for the wic version we claim to have
>>
>> .../scripts/lib/wic/canned-wks/sdimage-efi.wks | 2 +-
>> .../lib/wic/plugins/source/bootimg-efi-isar.py | 307
>> +++++++++++++++++++++
>> meta/classes/ext4-img.bbclass | 7 +-
>> meta/classes/image.bbclass | 19 ++
>> meta/classes/wic-img.bbclass | 64 +++++
>> scripts/lib/wic/canned-wks/qemux86-directdisk.wks | 2 +-
>> scripts/lib/wic/engine.py | 7 +-
>> scripts/lib/wic/filemap.py | 6 +-
>> scripts/lib/wic/help.py | 2 -
>> scripts/lib/wic/ksparser.py | 4 +-
>> scripts/lib/wic/partition.py | 197
>> ++++++++----- scripts/lib/wic/pluginbase.py |
>> 11 +- scripts/lib/wic/plugins/imager/direct.py | 66 +++--
>> scripts/lib/wic/plugins/source/bootimg-efi.py | 78
>> +----- .../lib/wic/plugins/source/bootimg-partition.py | 6 +-
>> scripts/lib/wic/plugins/source/bootimg-pcbios.py | 68 ++---
>> scripts/lib/wic/plugins/source/fsimage.py | 56
>> ++++ .../lib/wic/plugins/source/isoimage-isohybrid.py | 28 +-
>> scripts/lib/wic/plugins/source/rawcopy.py | 2 +-
>> scripts/lib/wic/plugins/source/rootfs.py | 4 +-
>> scripts/lib/wic/utils/misc.py | 33 +--
>> scripts/lib/wic/utils/runner.py | 74 ++++-
>> scripts/wic | 27 +-
>> scripts/wic_fakeroot | 37 +++ 24 files
>> changed, 846 insertions(+), 261 deletions(-) create mode 100644
>> meta-isar/scripts/lib/wic/plugins/source/bootimg-efi-isar.py create
>> mode 100644 meta/classes/wic-img.bbclass create mode 100644
>> scripts/lib/wic/plugins/source/fsimage.py create mode 100755
>> scripts/wic_fakeroot
>>
>
--
With best regards,
Alexander Smirnov
ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 51+ messages in thread
* Re: [PATCH 0/9] first wic integration
2018-02-12 18:21 ` Alexander Smirnov
@ 2018-02-12 18:30 ` Henning Schild
0 siblings, 0 replies; 51+ messages in thread
From: Henning Schild @ 2018-02-12 18:30 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users
Am Mon, 12 Feb 2018 21:21:26 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> On 02/12/2018 08:27 PM, Henning Schild wrote:
> > This series of patches is working and does apply on current next,
> > the mentioned open points are not new and therefore will be fixed
> > in follow up patches. I am waiting from review comments.
>
> I need at least one test for this series to be sure that CI checks
> wic image every time Isar is updated. Could you propose the machine
> to switch to wic?
It is all tested on qemuamd64.
Henning
> Alex
>
> >
> > Henning
> >
> > Am Wed, 31 Jan 2018 10:41:51 +0100
> > schrieb Henning Schild <henning.schild@siemens.com>:
> >
> >> This is the first official post of a series that aims to do two
> >> things. 1. integrate wic into Isar so it does not need to be called
> >> after bitbake
> >> 2. get our copy of wic back to an unmodified version from OE
> >>
> >> Many people have been asking for the first, so i think i do not
> >> need to motivate that change. Things this series already brings is
> >> control over the partition sizes, that we did not have before.
> >> The choice in working wks files is still limited. The plugins
> >> dealing with bootloaders (isolinux, syslinux etc.) will probably
> >> need to be forked (like the efigrub one).
> >>
> >> The second one is important so we can update wic in the future and
> >> developers do not get confused. Having a fork instead of a copy,
> >> without a very good reason, is not a good idea. I found that wic
> >> does not need to be changed and plugins can be forked.
> >>
> >> Henning
> >>
> >> Henning Schild (9):
> >> classes: image: introduce size measuring function, for before
> >> do_*_image
> >> images: new class wic-img for wic intregration
> >> wic: add a bootimg-efi-isar plugin outside the wic tree
> >> Revert "wic: Make the bootimg-efi plugin generate usable images"
> >> Revert "wic: Introduce the `WicExecError` exception class"
> >> Revert "wic: Work around mcopy error"
> >> Revert "wic: Use sudo instead of pseudo"
> >> Revert "wic: Remove sysroot support"
> >> wic: now truly go for the wic version we claim to have
> >>
> >> .../scripts/lib/wic/canned-wks/sdimage-efi.wks | 2 +-
> >> .../lib/wic/plugins/source/bootimg-efi-isar.py | 307
> >> +++++++++++++++++++++
> >> meta/classes/ext4-img.bbclass | 7 +-
> >> meta/classes/image.bbclass | 19 ++
> >> meta/classes/wic-img.bbclass | 64 +++++
> >> scripts/lib/wic/canned-wks/qemux86-directdisk.wks | 2 +-
> >> scripts/lib/wic/engine.py | 7 +-
> >> scripts/lib/wic/filemap.py | 6 +-
> >> scripts/lib/wic/help.py | 2 -
> >> scripts/lib/wic/ksparser.py | 4 +-
> >> scripts/lib/wic/partition.py | 197
> >> ++++++++----- scripts/lib/wic/pluginbase.py |
> >> 11 +- scripts/lib/wic/plugins/imager/direct.py | 66
> >> +++-- scripts/lib/wic/plugins/source/bootimg-efi.py | 78
> >> +----- .../lib/wic/plugins/source/bootimg-partition.py | 6 +-
> >> scripts/lib/wic/plugins/source/bootimg-pcbios.py | 68 ++---
> >> scripts/lib/wic/plugins/source/fsimage.py | 56
> >> ++++ .../lib/wic/plugins/source/isoimage-isohybrid.py | 28 +-
> >> scripts/lib/wic/plugins/source/rawcopy.py | 2 +-
> >> scripts/lib/wic/plugins/source/rootfs.py | 4 +-
> >> scripts/lib/wic/utils/misc.py | 33 +--
> >> scripts/lib/wic/utils/runner.py | 74 ++++-
> >> scripts/wic | 27 +-
> >> scripts/wic_fakeroot | 37 +++ 24
> >> files changed, 846 insertions(+), 261 deletions(-) create mode
> >> 100644
> >> meta-isar/scripts/lib/wic/plugins/source/bootimg-efi-isar.py
> >> create mode 100644 meta/classes/wic-img.bbclass create mode 100644
> >> scripts/lib/wic/plugins/source/fsimage.py create mode 100755
> >> scripts/wic_fakeroot
> >
>
^ permalink raw reply [flat|nested] 51+ messages in thread