[RFC PATCH 0/2] Implementation of UBI

[RFC PATCH 0/2] Implementation of UBI

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Hi,

I send this patch as an RFC, because I would like to have feedback about
it first, before finalizing it with documentation and more.

Here a few remarks about what I would like to have feedback on.

# meta/classes: added ubi-img class

    I would like to know what you think about the template system
employed here. For example usage see the second patch.

    I plan to add some documentation about this into the user manual as
well.

# meta-isar: added qemusabrelite example to demonstrate UBI use

    My primary goal here is to provide some reference and ci test case
for the UBI image creation. Not a real goal for me is to provide a
working BSP for sabrelite running on qemu, but that would be nice to
have.

    Background for this is that qemu supports a `--mtdblock` parameter
for sabrelite, so it might be possible to use the resulting ubi image to
boot a system using this. The problem is however that this doesn't seem
to work. I am not sure if what ubinize outputs can even be directly used
for this parameter or how to generate a valid mtd file to be used for
that. Normally ubiformat is used to write ubi images to mtd partitions
on real hardware, so I guess some emulation needs to happen in order to
write correct mtd image files.

   Another issue is that in order to boot from NAND directy in i.MX6
platforms, the NAND has to contain some propritary infomation that is
written by the kobs [1]. So for me the efford needed to provide a
working qemu or hardware sabrelite support seems a bit to high for this
patchset.

   So instead I think I will just rename the qemusabrelite stuff to
nand-ubi-demo to provide just some reference about how to use this
instead of falsely claim support for some hardware. IMO testing if the
image can be created is more of a goal here than testing if it can be
booted as well.

   Are the other possiblities I don't see here?

Thanks,
Claudius

[1] https://github.com/NXPmicro/imx-kobs 

Claudius Heine (2):
  meta/classes: added ubi-img class
  meta-isar: added qemusabrelite example to demonstrate UBI use

 meta-isar/classes/ubi-ubifs-img.bbclass       |  9 +++
 meta-isar/conf/machine/qemusabrelite.conf     | 11 +++
 .../multiconfig/qemusabrelite-buster.conf     | 13 ++++
 .../images/files/ubinize.cfg.tmpl             | 35 +++++++++
 .../recipes-core/images/isar-image-ubi.bb     | 49 +++++++++++++
 meta/classes/ubi-img.bbclass                  | 71 +++++++++++++++++++
 6 files changed, 188 insertions(+)
 create mode 100644 meta-isar/classes/ubi-ubifs-img.bbclass
 create mode 100644 meta-isar/conf/machine/qemusabrelite.conf
 create mode 100644 meta-isar/conf/multiconfig/qemusabrelite-buster.conf
 create mode 100644 meta-isar/recipes-core/images/files/ubinize.cfg.tmpl
 create mode 100644 meta-isar/recipes-core/images/isar-image-ubi.bb
 create mode 100644 meta/classes/ubi-img.bbclass

-- 
2.20.1

[RFC PATCH 1/2] meta/classes: added ubi-img class

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

This class implements a new image type for UBI formatted nand flashes.

It allows custom ubinize configuration file templates to be placed into
the SRC_URI of the image recipe and has mechanims to replace variables
within those. It does this by using `envsubst` from the gettext project.

The interface for custom template variables is implemented via the
`UBINIZE_CFG_TMPL_VARS` bitbake variable. The value of this variable is
an array of template variable names. Each template variable name
corresponds to a flag of the `UBINIZE_CFG_TMPL_VARS` variable, with
which the value can be set.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/classes/ubi-img.bbclass | 71 ++++++++++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)
 create mode 100644 meta/classes/ubi-img.bbclass

diff --git a/meta/classes/ubi-img.bbclass b/meta/classes/ubi-img.bbclass
new file mode 100644
index 0000000..58059e3
--- /dev/null
+++ b/meta/classes/ubi-img.bbclass
@@ -0,0 +1,71 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+
+python() {
+    if not d.getVar("UBINIZE_ARGS"):
+        bb.fatal("UBINIZE_ARGS must be set")
+}
+
+UBINIZE_CFG_TMPL ??= "ubinize.cfg.tmpl"
+UBINIZE_CFG_TMPL_VARS ??= ""
+UBINIZE_CFG ??= "ubinize.cfg"
+
+inherit image
+
+UBI_IMAGE_FILE ?= "${IMAGE_FULLNAME}.ubi.img"
+
+IMAGER_INSTALL += "mtd-utils"
+
+PP = "/home/builder/${PN}"
+PP_DEPLOY = "${PP}/deploy"
+PP_ROOTFS = "${PP}/rootfs"
+PP_WORK = "${PP}/work"
+
+BUILDROOT = "${BUILDCHROOT_DIR}${PP}"
+BUILDROOT_DEPLOY = "${BUILDCHROOT_DIR}${PP_DEPLOY}"
+BUILDROOT_ROOTFS = "${BUILDCHROOT_DIR}${PP_ROOTFS}"
+BUILDROOT_WORK = "${BUILDCHROOT_DIR}${PP_WORK}"
+
+do_ubi_image[stamp-extra-info] = "${DISTRO}-${MACHINE}"
+
+def get_ubi_exports(d):
+    template_vars = (d.getVar("UBINIZE_CFG_TMPL_VARS", True) or "").split()
+    template_vars_flags = (d.getVarFlags("UBINIZE_CFG_TMPL_VARS") or {})
+    return " ".join("{}=\"{}\"".format(name, template_vars_flags[name])
+                    for name in template_vars)
+
+def get_envsubst_args(d):
+    template_vars = (d.getVar("UBINIZE_CFG_TMPL_VARS", True) or "").split()
+    return " ".join("\\${{{}}}".format(name)
+                    for name in template_vars)
+
+# Generate ubi filesystem image
+do_ubi_image() {
+    if [ ! -e "${WORKDIR}/${UBINIZE_CFG_TMPL}" -a \
+         ! -e "${WORKDIR}/${UBINIZE_CFG}" ]; then
+        die "Neigher UBINIZE_CFG_TMPL nor UBINIZE_CFG contain existing" \
+            "ubinize configuration file"
+    fi
+
+    rm -f ${DEPLOY_DIR_IMAGE}/${UBI_IMAGE_FILE}
+
+    buildchroot_do_mounts
+
+    sudo flock ${MOUNT_LOCKFILE} -c ' \
+        mkdir -p ${BUILDROOT_DEPLOY} ${BUILDROOT_ROOTFS} ${BUILDROOT_WORK}
+        mount --bind ${DEPLOY_DIR_IMAGE} ${BUILDROOT_DEPLOY}
+        mount --bind ${IMAGE_ROOTFS} ${BUILDROOT_ROOTFS}
+        mount --bind ${WORKDIR} ${BUILDROOT_WORK}
+    '
+
+    if [ -e "${WORKDIR}/${UBINIZE_CFG_TMPL}" ]; then
+        export ${@get_ubi_exports(d)}
+        envsubst "${@get_envsubst_args(d)}" \
+            < ${WORKDIR}/${UBINIZE_CFG_TMPL} > ${WORKDIR}/${UBINIZE_CFG} 
+    fi
+
+    # Create ubi image using buildchroot tools
+    sudo chroot ${BUILDCHROOT_DIR} /usr/sbin/ubinize ${UBINIZE_ARGS} \
+                -o "${PP_DEPLOY}/${UBI_IMAGE_FILE}" "${PP_WORK}/${UBINIZE_CFG}"
+}
+addtask ubi_image before do_build after do_copy_boot_files do_install_imager_deps do_unpack
-- 
2.20.1

[RFC PATCH 2/2] meta-isar: added qemusabrelite example to demonstrate UBI use

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta-isar/classes/ubi-ubifs-img.bbclass       |  9 ++++
 meta-isar/conf/machine/qemusabrelite.conf     | 11 +++++
 .../multiconfig/qemusabrelite-buster.conf     | 13 +++++
 .../images/files/ubinize.cfg.tmpl             | 35 +++++++++++++
 .../recipes-core/images/isar-image-ubi.bb     | 49 +++++++++++++++++++
 5 files changed, 117 insertions(+)
 create mode 100644 meta-isar/classes/ubi-ubifs-img.bbclass
 create mode 100644 meta-isar/conf/machine/qemusabrelite.conf
 create mode 100644 meta-isar/conf/multiconfig/qemusabrelite-buster.conf
 create mode 100644 meta-isar/recipes-core/images/files/ubinize.cfg.tmpl
 create mode 100644 meta-isar/recipes-core/images/isar-image-ubi.bb

diff --git a/meta-isar/classes/ubi-ubifs-img.bbclass b/meta-isar/classes/ubi-ubifs-img.bbclass
new file mode 100644
index 0000000..711492e
--- /dev/null
+++ b/meta-isar/classes/ubi-ubifs-img.bbclass
@@ -0,0 +1,9 @@
+# UBI with UBIFS image recipe
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2018 ilbers GmbH
+
+inherit ubi-img ubifs-img
+addtask do_ubi_image after do_ubifs_image
+
+UBINIZE_CFG_TMPL_VARS[UBIFS_IMG] = "${PP_DEPLOY}/${UBIFS_IMAGE_FILE}"
diff --git a/meta-isar/conf/machine/qemusabrelite.conf b/meta-isar/conf/machine/qemusabrelite.conf
new file mode 100644
index 0000000..1f886be
--- /dev/null
+++ b/meta-isar/conf/machine/qemusabrelite.conf
@@ -0,0 +1,11 @@
+# This software is a part of ISAR.
+# Copyright (C) 2015-2017 ilbers GmbH
+
+MACHINE_SERIAL ?= "ttymxc0"
+BAUDRATE_TTY ?= "115200"
+
+MKUBIFS_ARGS := "-m 0x1000 -e 0x3e000 -c 1500"
+UBINIZE_ARGS = "-vv -m 0x1000 -p 0x40000"
+IMAGE_TYPE ?= "ubi-ubifs-img"
+
+DTB_FILE = "imx6q-sabrelite.dtb"
diff --git a/meta-isar/conf/multiconfig/qemusabrelite-buster.conf b/meta-isar/conf/multiconfig/qemusabrelite-buster.conf
new file mode 100644
index 0000000..aa29abb
--- /dev/null
+++ b/meta-isar/conf/multiconfig/qemusabrelite-buster.conf
@@ -0,0 +1,13 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: MIT
+
+MACHINE ?= "qemusabrelite"
+
+DISTRO ?= "debian-buster"
+DISTRO_ARCH ?= "armhf"
+
+KERNEL_NAME ?= "armmp"
+
+IMAGE_INSTALL += "sshd-regen-keys"
diff --git a/meta-isar/recipes-core/images/files/ubinize.cfg.tmpl b/meta-isar/recipes-core/images/files/ubinize.cfg.tmpl
new file mode 100644
index 0000000..f386915
--- /dev/null
+++ b/meta-isar/recipes-core/images/files/ubinize.cfg.tmpl
@@ -0,0 +1,35 @@
+[kernel]
+mode=ubi
+image=${KERNEL_IMG}
+vol_id=1
+vol_type=static
+vol_name=kernel
+vol_size=16MiB
+vol_alignment=1
+
+[initrd]
+mode=ubi
+image=${INITRD_IMG}
+vol_id=2
+vol_type=static
+vol_name=initrd
+vol_size=32MiB
+vol_alignment=1
+
+[dtb]
+mode=ubi
+image=${DTB_IMG}
+vol_id=3
+vol_type=static
+vol_name=dtb
+vol_size=2MiB
+vol_alignment=1
+
+[rootfs]
+mode=ubi
+image=${UBIFS_IMG}
+vol_id=4
+vol_type=dynamic
+vol_name=rootfs
+vol_size=300MiB
+vol_alignment=1
diff --git a/meta-isar/recipes-core/images/isar-image-ubi.bb b/meta-isar/recipes-core/images/isar-image-ubi.bb
new file mode 100644
index 0000000..c5a94d0
--- /dev/null
+++ b/meta-isar/recipes-core/images/isar-image-ubi.bb
@@ -0,0 +1,49 @@
+# UBI image recipe
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2018 ilbers GmbH
+
+DESCRIPTION = "UBI Isar image"
+
+LICENSE = "gpl-2.0"
+LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+
+PV = "1.0"
+
+inherit isar-image
+
+SRC_URI += "file://ubinize.cfg.tmpl"
+
+def find_linux_image(d, root_var):
+    """
+        Find path to the linux-image directory
+
+        Takes the first it can find -> could lead to problems if more than one
+        linux-image package is installed.
+
+        root_var: Name of variable that contains the path to the rootfs
+        returns: found path without leading path to the rootfs or ""
+    """
+    import glob
+
+    root = d.getVar(root_var, True)
+    if not root:
+        bb.fatal("Variable \"{}\" does not exist or is empty!".format(root_var))
+
+    found = glob.glob(os.path.join(root, 'usr/lib/linux-image-*'))
+    if not found:
+        return ""
+
+    return found[0][len(root):]
+
+UBINIZE_CFG_TMPL_VARS ??= "KERNEL_IMG INITRD_IMG DTB_IMG UBIFS_IMG"
+UBINIZE_CFG_TMPL_VARS[KERNEL_IMG] = "${PP_DEPLOY}/${KERNEL_IMAGE}"
+UBINIZE_CFG_TMPL_VARS[INITRD_IMG] = "${PP_DEPLOY}/${INITRD_IMAGE}"
+UBINIZE_CFG_TMPL_VARS[DTB_IMG] = "${PP_ROOTFS}/${@find_linux_image(d, 'IMAGE_ROOTFS')}/${DTB_FILE}"
+
+do_copy_boot_files_append() {
+    DTB="${@find_linux_image(d, 'IMAGE_ROOTFS')}/${DTB_FILE}"
+    if [ -n "${DTB}" ]; then
+        cp -f ${IMAGE_ROOTFS}/${DTB} ${DEPLOY_DIR_IMAGE}/${DTB_FILE}
+    fi
+}
-- 
2.20.1