[PATCH v5 0/5] Debootstrap integration

[PATCH v5 0/5] Debootstrap integration

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Hi,

this is the new version of this patchset, that fixes the
generate_keyring task in isar-bootstrap for systems with read-only
homedir.

best regards,
Claudius

Changes from v4:
  - added 'homedir' parameter to gpg command in isar-bootstrap

Changes from v3:
  - A small modification to isar-bootstrap.bb snuck into the buildchroot
    patch.

Changes from v2:
  - rebased to current next
  - added missing newlines at the end of file
  - use cleandirs with root permissions
    this could be improved in some later patch series, so that this
    feature can be shared.
  - moved main source url of packages to apt source list files of the
    distributions:
    - Implemented some basic parsing of source.lists files in
      isar-bootstrap.
    - Made 'DISTRO_SUITE', 'DISTRO_APT_SOURCE' and 'DISTRO_COMPONENTS'
      variables obsolete.
  - implemented repo key fetching to support rasbian repositories
  - now the clean task removes the symlink of isar-bootstrap deploy task

Changes from v1:
  - rebased to current next
  - added unmounting of /dev and /proc at the end of do_rootfs in isar-image-base
    This was necessary for image file creation, otherwise it tried to
    package stuff from the host.
  - added 'isar-bootstrap:do_deploy' dependency to do_rootfs task in
    image.bbclass.
  - Changed 'RFSDIR' variable name to 'ROOTFSDIR' in
    isar-bootstrap-helper.bbclass to be consistent with the variable
    name in isar-bootstrap.bb
  - Moved 'isar-apt' apt-preference settings from isar-bootstrap.bb to
    isar-bootstrap-helper.bbclass
  - Removed '--no-install-recommends' parameters in favor of adding the
    apt configuration in 'isar-bootstrap.bb'
  - unmount and remove possible existing bootstraped directories before
    bootstrapping in do_bootstrap task of 'isar-bootstrap.bb'
  - changed from /dev ro bind mounts to mounting devtmpfs to /dev to be
    consistent with the other places where /dev is mounted.


Claudius Heine (5):
  implement isar-bootstrap using debootstrap
  meta/isar-bootstrap-helper.bbclass: handle rfs customization centrally
  meta/buildchroot: switch to using isar-bootstrap
  meta-isar/isar-image-base: switch to using isar-bootstrap
  meta-isar/multiconfig: remove multistrap references

 meta-isar/conf/distro/debian-jessie.conf           |   9 +-
 meta-isar/conf/distro/debian-jessie.list           |   3 +
 meta-isar/conf/distro/debian-stretch.conf          |   7 +-
 meta-isar/conf/distro/debian-stretch.list          |   3 +
 meta-isar/conf/distro/debian-wheezy.conf           |   9 +-
 meta-isar/conf/distro/debian-wheezy.list           |   3 +
 meta-isar/conf/distro/raspbian-jessie.conf         |  12 +-
 meta-isar/conf/distro/raspbian-jessie.list         |   1 +
 meta-isar/conf/local.conf.sample                   |   4 +-
 meta-isar/conf/multiconfig/qemuarm-jessie.conf     |   2 +-
 meta-isar/conf/multiconfig/qemuarm-wheezy.conf     |   2 +-
 meta-isar/conf/multiconfig/rpi-jessie.conf         |   2 +-
 .../recipes-core/images/files/download_dev-random  |  24 ---
 .../recipes-core/images/files/multistrap.conf.in   |  38 ----
 meta-isar/recipes-core/images/files/setup.sh       |  77 --------
 meta-isar/recipes-core/images/isar-image-base.bb   |  64 +++----
 meta/classes/image.bbclass                         |   2 +-
 meta/classes/isar-bootstrap-helper.bbclass         |  39 ++++
 .../isar-bootstrap/files/isar-apt.conf             |   2 +
 meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 206 +++++++++++++++++++++
 meta/recipes-devtools/buildchroot/buildchroot.bb   |  59 ++----
 meta/recipes-devtools/buildchroot/files/build.sh   |   2 +-
 .../buildchroot/files/configscript.sh              |  10 -
 .../buildchroot/files/multistrap.conf.in           |  37 ----
 meta/recipes-devtools/buildchroot/files/setup.sh   |  82 --------
 25 files changed, 311 insertions(+), 388 deletions(-)
 create mode 100644 meta-isar/conf/distro/debian-jessie.list
 create mode 100644 meta-isar/conf/distro/debian-stretch.list
 create mode 100644 meta-isar/conf/distro/debian-wheezy.list
 create mode 100644 meta-isar/conf/distro/raspbian-jessie.list
 delete mode 100644 meta-isar/recipes-core/images/files/download_dev-random
 delete mode 100644 meta-isar/recipes-core/images/files/multistrap.conf.in
 delete mode 100755 meta-isar/recipes-core/images/files/setup.sh
 create mode 100644 meta/classes/isar-bootstrap-helper.bbclass
 create mode 100644 meta/recipes-core/isar-bootstrap/files/isar-apt.conf
 create mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap.bb
 delete mode 100644 meta/recipes-devtools/buildchroot/files/multistrap.conf.in
 delete mode 100755 meta/recipes-devtools/buildchroot/files/setup.sh

-- 
2.16.3

[PATCH v5 1/5] implement isar-bootstrap using debootstrap

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Since multistrap is deprecated for some years, it is required to change
to debootstrap.

This patch introduces the 'isar-bootstrap' recipe that implement the
creation of a minimal base root file system using debootstrap.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta-isar/conf/distro/debian-jessie.conf           |  15 +-
 meta-isar/conf/distro/debian-jessie.list           |   3 +
 meta-isar/conf/distro/debian-stretch.conf          |  11 +-
 meta-isar/conf/distro/debian-stretch.list          |   3 +
 meta-isar/conf/distro/debian-wheezy.conf           |  11 +-
 meta-isar/conf/distro/debian-wheezy.list           |   3 +
 meta-isar/conf/distro/raspbian-jessie.conf         |  10 +-
 meta-isar/conf/distro/raspbian-jessie.list         |   1 +
 .../isar-bootstrap/files/isar-apt.conf             |   2 +
 meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 206 +++++++++++++++++++++
 10 files changed, 248 insertions(+), 17 deletions(-)
 create mode 100644 meta-isar/conf/distro/debian-jessie.list
 create mode 100644 meta-isar/conf/distro/debian-stretch.list
 create mode 100644 meta-isar/conf/distro/debian-wheezy.list
 create mode 100644 meta-isar/conf/distro/raspbian-jessie.list
 create mode 100644 meta/recipes-core/isar-bootstrap/files/isar-apt.conf
 create mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap.bb

diff --git a/meta-isar/conf/distro/debian-jessie.conf b/meta-isar/conf/distro/debian-jessie.conf
index 3e62b14..df27fd7 100644
--- a/meta-isar/conf/distro/debian-jessie.conf
+++ b/meta-isar/conf/distro/debian-jessie.conf
@@ -1,16 +1,19 @@
-# Debian jessie multistrap configuration
+# Debian jessie distribution configuration
 #
 # This software is a part of ISAR.
 # Copyright (C) 2015-2016 ilbers GmbH
 
+DISTRO_APT_SOURCES += "conf/distro/debian-jessie.list"
+DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"
+DISTRO_KERNELS ?= "486 4kc-malta 586 5kc-malta 686-pae amd64 arm64 armmp \
+    armmp-lpae ixp4xx kirkwood loongson-2e loongson-2f loongson-3 octeon \
+    orion5x powerpc powerpc-smp powerpc64 powerpc64le r4k-ip22 r5k-ip32 \
+    s390x sb1-bcm91250a versatile"
+
+# No longer needed:
 DISTRO_SUITE ?= "jessie"
 DISTRO_COMPONENTS ?= "main contrib non-free"
 DISTRO_APT_SOURCE ?= "http://ftp.debian.org/debian"
 DISTRO_APT_SOURCE_SEC ?= "http://security.debian.org/debian-security"
-DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"
 DISTRO_MULTICONF_BOOTSTRAP ?= "base updates security"
 DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}"
-DISTRO_KERNELS ?= "486 4kc-malta 586 5kc-malta 686-pae amd64 arm64 armmp \
-    armmp-lpae ixp4xx kirkwood loongson-2e loongson-2f loongson-3 octeon \
-    orion5x powerpc powerpc-smp powerpc64 powerpc64le r4k-ip22 r5k-ip32 \
-    s390x sb1-bcm91250a versatile"
diff --git a/meta-isar/conf/distro/debian-jessie.list b/meta-isar/conf/distro/debian-jessie.list
new file mode 100644
index 0000000..be46a57
--- /dev/null
+++ b/meta-isar/conf/distro/debian-jessie.list
@@ -0,0 +1,3 @@
+deb	http://ftp.de.debian.org/debian	jessie	main contrib non-free
+deb	http://ftp.de.debian.org/debian jessie-updates	main contrib non-free
+deb	http://security.debian.org	jessie/updates	main contrib non-free
diff --git a/meta-isar/conf/distro/debian-stretch.conf b/meta-isar/conf/distro/debian-stretch.conf
index 5a85d52..13c4f94 100644
--- a/meta-isar/conf/distro/debian-stretch.conf
+++ b/meta-isar/conf/distro/debian-stretch.conf
@@ -1,13 +1,16 @@
 # This software is a part of ISAR.
 # Copyright (C) 2017 ilbers GmbH
 
+DISTRO_APT_SOURCES += "conf/distro/debian-stretch.list"
+DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"
+DISTRO_KERNELS ?= "4kc-malta 586 5kc-malta 686 686-pae amd64 arm64 armmp \
+    armmp-lpae kirkwood loongson-3 marvell octeon orion5x powerpc64le \
+    rt-686-pae rt-amd64 s390x"
+
+# No longer needed:
 DISTRO_SUITE ?= "stretch"
 DISTRO_COMPONENTS ?= "main contrib non-free"
 DISTRO_APT_SOURCE ?= "http://ftp.debian.org/debian"
 DISTRO_APT_SOURCE_SEC ?= "http://security.debian.org/debian-security"
-DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"
 DISTRO_MULTICONF_BOOTSTRAP ?= "base updates security"
 DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}"
-DISTRO_KERNELS ?= "4kc-malta 586 5kc-malta 686 686-pae amd64 arm64 armmp \
-    armmp-lpae kirkwood loongson-3 marvell octeon orion5x powerpc64le \
-    rt-686-pae rt-amd64 s390x"
diff --git a/meta-isar/conf/distro/debian-stretch.list b/meta-isar/conf/distro/debian-stretch.list
new file mode 100644
index 0000000..bfa5cf9
--- /dev/null
+++ b/meta-isar/conf/distro/debian-stretch.list
@@ -0,0 +1,3 @@
+deb	http://ftp.de.debian.org/debian stretch	main contrib non-free
+deb	http://ftp.de.debian.org/debian stretch-updates	main contrib non-free
+deb	http://security.debian.org	stretch/updates	main contrib non-free
diff --git a/meta-isar/conf/distro/debian-wheezy.conf b/meta-isar/conf/distro/debian-wheezy.conf
index f5ab6eb..2108f9b 100644
--- a/meta-isar/conf/distro/debian-wheezy.conf
+++ b/meta-isar/conf/distro/debian-wheezy.conf
@@ -1,14 +1,17 @@
-# Debian wheezy multistrap configuration
+# Debian wheezy distribution configuration
 #
 # This software is a part of ISAR.
 # Copyright (C) 2015-2016 ilbers GmbH
 
+DISTRO_APT_SOURCES += "conf/distro/debian-wheezy.list"
+DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"
+DISTRO_KERNELS ?= "486 686-pae amd64 rt-686-pae rt-amd64 versatile vexpress \
+    iop32x ixp4xx kirkwood mv78xx0 mx5 omap orion5x"
+
+# No longer needed:
 DISTRO_SUITE ?= "wheezy"
 DISTRO_COMPONENTS ?= "main contrib non-free"
 DISTRO_APT_SOURCE ?= "http://ftp.debian.org/debian"
 DISTRO_APT_SOURCE_SEC ?= "http://security.debian.org/debian-security"
-DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"
 DISTRO_MULTICONF_BOOTSTRAP ?= "base updates security"
 DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}"
-DISTRO_KERNELS ?= "486 686-pae amd64 rt-686-pae rt-amd64 versatile vexpress \
-    iop32x ixp4xx kirkwood mv78xx0 mx5 omap orion5x"
diff --git a/meta-isar/conf/distro/debian-wheezy.list b/meta-isar/conf/distro/debian-wheezy.list
new file mode 100644
index 0000000..0bfa98e
--- /dev/null
+++ b/meta-isar/conf/distro/debian-wheezy.list
@@ -0,0 +1,3 @@
+deb	http://ftp.de.debian.org/debian wheezy	main contrib non-free
+deb	http://ftp.de.debian.org/debian wheezy-updates	main contrib non-free
+deb	http://security.debian.org	wheezy/updates	main contrib non-free
diff --git a/meta-isar/conf/distro/raspbian-jessie.conf b/meta-isar/conf/distro/raspbian-jessie.conf
index c2a867d..57d0e7d 100644
--- a/meta-isar/conf/distro/raspbian-jessie.conf
+++ b/meta-isar/conf/distro/raspbian-jessie.conf
@@ -1,13 +1,17 @@
-# Raspbian stable multistrap configuration
+# Raspbian stable distribution configuration
 #
 # This software is a part of ISAR.
 # Copyright (C) 2015-2016 ilbers GmbH
 
+DISTRO_APT_SOURCES += "conf/distro/raspbian-jessie.list"
+DISTRO_APT_KEYS += "https://archive.raspbian.org/raspbian.public.key;sha256sum=ca59cd4f2bcbc3a1d41ba6815a02a8dc5c175467a59bd87edeac458f4a5345de"
+DISTRO_CONFIG_SCRIPT?= "raspbian-configscript.sh"
+DISTRO_KERNELS ?= "rpi rpi2 rpi-rpfv rpi2-rpfv"
+
+# No longer needed:
 DISTRO_SUITE ?= "jessie"
 DISTRO_COMPONENTS ?= "main contrib non-free firmware"
 DISTRO_APT_SOURCE ?= "http://archive.raspbian.org/raspbian"
 DISTRO_APT_SOURCE_SEC ?= ""
-DISTRO_CONFIG_SCRIPT ?= "raspbian-configscript.sh"
 DISTRO_MULTICONF_BOOTSTRAP ?= "base"
 DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}"
-DISTRO_KERNELS ?= "rpi rpi2 rpi-rpfv rpi2-rpfv"
diff --git a/meta-isar/conf/distro/raspbian-jessie.list b/meta-isar/conf/distro/raspbian-jessie.list
new file mode 100644
index 0000000..792f9c6
--- /dev/null
+++ b/meta-isar/conf/distro/raspbian-jessie.list
@@ -0,0 +1 @@
+deb	http://archive.raspbian.org/raspbian	jessie	main contrib non-free firmware
diff --git a/meta/recipes-core/isar-bootstrap/files/isar-apt.conf b/meta/recipes-core/isar-bootstrap/files/isar-apt.conf
new file mode 100644
index 0000000..a62feb1
--- /dev/null
+++ b/meta/recipes-core/isar-bootstrap/files/isar-apt.conf
@@ -0,0 +1,2 @@
+APT::Install-Recommends "0";
+APT::Install-Suggests "0";
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb
new file mode 100644
index 0000000..a38dd88
--- /dev/null
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb
@@ -0,0 +1,206 @@
+# Minimal debian root file system
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: MIT
+
+Description = "Minimal debian root file system"
+
+LICENSE = "gpl-2.0"
+LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+FILESPATH_prepend := "${THISDIR}/files:"
+SRC_URI = "file://isar-apt.conf"
+PV = "1.0"
+
+WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
+DEBOOTSTRAP ?= ""
+ROOTFSDIR = "${WORKDIR}/rootfs"
+APTPREFS = "${WORKDIR}/apt-preferences"
+APTSRCS = "${WORKDIR}/apt-sources"
+APTKEYFILES = ""
+APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
+DEBOOTSTRAP_KEYRING = ""
+
+python () {
+    from urllib.parse import urlparse
+
+    debootstrap = d.getVar("DEBOOTSTRAP", True)
+    if not debootstrap:
+        target = d.getVar("DISTRO_ARCH", True)
+        machine = os.uname()[4]
+        m = {
+            "x86_64": ["i386", "amd64"],
+            "x86": ["i386"],
+            }
+        if machine not in m or target not in m[machine]:
+            debootstrap = "qemu-debootstrap"
+        else:
+            debootstrap = "debootstrap"
+        d.setVar("DEBOOTSTRAP", debootstrap)
+
+    distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
+    if distro_apt_keys:
+        d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
+        for key in distro_apt_keys.split():
+            url = urlparse(key)
+            filename = os.path.basename(url.path)
+            d.appendVar("SRC_URI", " %s" % key)
+            d.appendVar("APTKEYFILES", " %s" % filename)
+}
+
+def aggregate_files(d, file_list, file_out):
+    import shutil
+
+    with open(file_out, "wb") as out_fd:
+        for entry in file_list:
+            entry_real = bb.parse.resolve_file(entry, d)
+            with open(entry_real, "rb") as in_fd:
+                 shutil.copyfileobj(in_fd, out_fd, 1024*1024*10)
+            out_fd.write("\n".encode())
+
+def parse_aptsources_list_line(source_list_line):
+    import re
+
+    s = source_list_line.strip()
+
+    if s.startswith("#"):
+        return None
+
+    type, s = re.split("\s+", s, maxsplit=1)
+    if type not in ["deb", "deb-src"]:
+        return None
+
+    options = ""
+    options_match = re.match("\[\s*(\S+=\S+(?=\s))*\s*(\S+=\S+)\s*\]\s+", s)
+    if options_match:
+        options = options_match.group(0).strip()
+        s = s[options_match.end():]
+
+    source, s = re.split("\s+", s, maxsplit=1)
+
+    suite, s = re.split("\s+", s, maxsplit=1)
+
+    components = " ".join(s.split())
+
+    return type, options, source, suite, components
+
+def get_distro_primary_source_entry(d):
+    apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or "").split()
+    for entry in apt_sources_list:
+        entry_real = bb.parse.resolve_file(entry, d)
+        with open(entry_real, "r") as in_fd:
+            for line in in_fd:
+                parsed = parse_aptsources_list_line(line)
+                if parsed:
+                    type, _, source, suite, components = parsed
+                    if type == "deb":
+                        return source, suite, components
+    return "", "", ""
+
+def get_distro_source(d):
+    return get_distro_primary_source_entry(d)[0]
+
+def get_distro_suite(d):
+    return get_distro_primary_source_entry(d)[1]
+
+def get_distro_components_argument(d):
+    components = get_distro_primary_source_entry(d)[2]
+    if components and components.strip():
+        return "--components=%s" % ",".join(components.split())
+    else:
+        return ""
+
+do_generate_keyring[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_generate_keyring[dirs] = "${WORKDIR}"
+do_generate_keyring[vardeps] += "DISTRO_APT_KEYS"
+do_generate_keyring() {
+    if [ -n "${@d.getVar("APTKEYFILES", True) or ""}" ]; then
+        for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
+           gpg --no-default-keyring --keyring "${APTKEYRING}" \
+               --homedir "${WORKDIR}" --import "$keyfile"
+        done
+    fi
+}
+addtask generate_keyring before do_build after do_unpack
+
+do_apt_config_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_apt_config_prepare[vardeps] += "\
+                                   APTPREFS \
+                                   DISTRO_APT_PREFERENCES \
+                                   DEBDISTRONAME \
+                                   APTSRCS \
+                                   DISTRO_APT_SOURCES \
+                                  "
+python do_apt_config_prepare() {
+    apt_preferences_out = d.getVar("APTPREFS", True)
+    apt_preferences_list = (d.getVar("DISTRO_APT_PREFERENCES", True) or ""
+                           ).split()
+    aggregate_files(d, apt_preferences_list, apt_preferences_out)
+
+    apt_sources_out = d.getVar("APTSRCS", True)
+    apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or "").split()
+
+    aggregate_files(d, apt_sources_list, apt_sources_out)
+}
+addtask apt_config_prepare before do_build after do_generate_keyring
+
+do_bootstrap[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_bootstrap[vardeps] += "DISTRO_APT_SOURCES"
+do_bootstrap() {
+    if [ -e "${ROOTFSDIR}" ]; then
+       sudo umount -l "${ROOTFSDIR}/dev" || true
+       sudo umount -l "${ROOTFSDIR}/proc" || true
+       sudo rm -rf "${ROOTFSDIR}"
+    fi
+    E="${@bb.utils.export_proxies(d)}"
+    sudo -E "${DEBOOTSTRAP}" --verbose \
+                             --variant minbase \
+                             --arch "${DISTRO_ARCH}" \
+                             ${@get_distro_components_argument(d)} \
+                             ${DEBOOTSTRAP_KEYRING} \
+                             "${@get_distro_suite(d)}" \
+                             "${ROOTFSDIR}" \
+                             "${@get_distro_source(d)}"
+}
+addtask bootstrap before do_build after do_apt_config_prepare
+
+do_apt_config_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_apt_config_install() {
+    sudo mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"
+    sudo install -v -m644 "${APTPREFS}" \
+                          "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
+    sudo mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
+    sudo install -v -m644 "${APTSRCS}" \
+                          "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
+    sudo rm -f "${ROOTFSDIR}/etc/apt/sources.list"
+    sudo mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"
+    sudo install -v -m644 "${WORKDIR}/isar-apt.conf" \
+                          "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf"
+}
+addtask apt_config_install before do_build after do_bootstrap
+
+do_apt_update[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_apt_update() {
+    sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${ROOTFSDIR}/dev
+    sudo mount -t proc none ${ROOTFSDIR}/proc
+
+    E="${@bb.utils.export_proxies(d)}"
+    export DEBIAN_FRONTEND=noninteractive
+    sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
+    sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
+                                      -o Debug::pkgProblemResolver=yes
+}
+addtask apt_update before do_build after do_apt_config_install
+
+do_deploy[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_deploy[dirs] = "${DEPLOY_DIR_IMAGE}"
+do_deploy() {
+    ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_DIR_IMAGE}/${PN}-${DISTRO}-${DISTRO_ARCH}"
+}
+addtask deploy before do_build after do_apt_update
+
+CLEANFUNCS = "clean_deploy"
+clean_deploy() {
+     rm -f "${DEPLOY_DIR_IMAGE}/${PN}-${DISTRO}-${DISTRO_ARCH}"
+}
-- 
2.16.3

[PATCH v5 2/5] meta/isar-bootstrap-helper.bbclass: handle rfs customization centrally

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

This patch introduces the 'isar-bootstrap-helper' bitbake class to
support recipes that use the basic root file system created by
isar-bootstrap.

In this patch the function 'setup_root_file_system' is defined that
allows to copy the root file system to the directory specified in the
first parameter and installs additional packages specified by the
next parameters. It also adds the local 'isar-apt' repo to the apt
configuration and fetches its index.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/classes/isar-bootstrap-helper.bbclass | 39 ++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
 create mode 100644 meta/classes/isar-bootstrap-helper.bbclass

diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
new file mode 100644
index 0000000..a06116d
--- /dev/null
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -0,0 +1,39 @@
+# Helper functions for using isar-bootstrap
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: MIT
+
+setup_root_file_system() {
+    ROOTFSDIR="$1"
+    shift
+    PACKAGES="$@"
+
+    sudo cp -Trpfx \
+        "${DEPLOY_DIR_IMAGE}/isar-bootstrap-${DISTRO}-${DISTRO_ARCH}/" \
+        "$ROOTFSDIR"
+
+    echo "deb file:///isar-apt ${DEBDISTRONAME} main" | \
+        sudo tee "$ROOTFSDIR/etc/apt/sources.list.d/isar-apt.list" >/dev/null
+
+    echo "Package: *\nPin: release n=${DEBDISTRONAME}\nPin-Priority: 1000" | \
+        sudo tee "$ROOTFSDIR/etc/apt/preferences.d/isar" >/dev/null
+
+    sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} $ROOTFSDIR/isar-apt
+    sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs $ROOTFSDIR/dev
+    sudo mount -t proc none $ROOTFSDIR/proc
+
+    # Install packages:
+    E="${@ bb.utils.export_proxies(d)}"
+    export DEBIAN_FRONTEND=noninteractive
+    sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \
+        -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
+        -o Dir::Etc::sourceparts="-" \
+        -o APT::Get::List-Cleanup="0"
+    sudo -E chroot "$ROOTFSDIR" \
+        /usr/bin/apt-get install -y \
+            --allow-unauthenticated \
+            -o Debug::pkgProblemResolver=yes \
+        $PACKAGES
+}
-- 
2.16.3

[PATCH v5 3/5] meta/buildchroot: switch to using isar-bootstrap

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

This patch removes the multistrap dependency of the buildchroot recipe
and changes it to use the 'isar-bootstrap' and 'isar-bootstrap-helper'
instead.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/recipes-devtools/buildchroot/buildchroot.bb   | 59 +++++-----------
 meta/recipes-devtools/buildchroot/files/build.sh   |  2 +-
 .../buildchroot/files/configscript.sh              | 10 ---
 .../buildchroot/files/multistrap.conf.in           | 37 ----------
 meta/recipes-devtools/buildchroot/files/setup.sh   | 82 ----------------------
 5 files changed, 18 insertions(+), 172 deletions(-)
 delete mode 100644 meta/recipes-devtools/buildchroot/files/multistrap.conf.in
 delete mode 100755 meta/recipes-devtools/buildchroot/files/setup.sh

diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 2fc5297..b16e63a 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -8,13 +8,13 @@ DESCRIPTION = "Multistrap development filesystem"
 LICENSE = "gpl-2.0"
 LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
 
-FILESPATH =. "${LAYERDIR_core}/recipes-devtools/buildchroot/files:"
-SRC_URI = "file://multistrap.conf.in \
-           file://configscript.sh \
-           file://setup.sh \
+FILESPATH_prepend := "${THISDIR}/files:"
+SRC_URI = "file://configscript.sh \
            file://build.sh"
 PV = "1.0"
 
+inherit isar-bootstrap-helper
+
 BUILDCHROOT_PREINSTALL ?= "gcc \
                            make \
                            build-essential \
@@ -31,51 +31,26 @@ BUILDCHROOT_PREINSTALL ?= "gcc \
 WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
 
 do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-do_build[depends] = "isar-apt:do_cache_config"
+do_build[root_cleandirs] = "${BUILDCHROOT_DIR} \
+                            ${BUILDCHROOT_DIR}/isar-apt \
+                            ${BUILDCHROOT_DIR}/downloads \
+                            ${BUILDCHROOT_DIR}/home/builder"
+do_build[depends] = "isar-apt:do_cache_config isar-bootstrap:do_deploy"
 
 do_build() {
-    E="${@ bb.utils.export_proxies(d)}"
-
-    # If buildchroot re-build is triggered, this ensure that we are always
-    # re-building a clean image:
-    sudo rm -rf ${BUILDCHROOT_DIR}
-    mkdir -p ${BUILDCHROOT_DIR}/isar-apt
-    mkdir -p ${BUILDCHROOT_DIR}/downloads
-    mkdir -p ${BUILDCHROOT_DIR}/dev
-    mkdir -p ${BUILDCHROOT_DIR}/proc
-    mkdir -p ${BUILDCHROOT_DIR}/sys
-
-    chmod +x "${WORKDIR}/setup.sh"
-    chmod +x "${WORKDIR}/configscript.sh"
-
-    # Multistrap accepts only relative path in configuration files, so get it:
-    cd ${TOPDIR}
-    WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
-
-    # Adjust multistrap config
-    sed -e 's|##BUILDCHROOT_PREINSTALL##|${BUILDCHROOT_PREINSTALL}|g' \
-        -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
-        -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
-        -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
-        -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
-        -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
-        -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
-        -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \
-        -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
-        -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
-           "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
-
-    sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
-    sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
-    sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
-    sudo mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads
+    CDIRS="${@d.expand(d.getVarFlags("do_build").get("root_cleandirs", ""))}"
+    if [ -n "$CDIRS" ]; then
+        sudo rm -rf $CDIRS
+        mkdir -p $CDIRS
+    fi
 
-    # Create root filesystem
-    sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
+    setup_root_file_system "${BUILDCHROOT_DIR}" ${BUILDCHROOT_PREINSTALL}
 
     # Install package builder script
+    sudo chmod -R a+rw "${BUILDCHROOT_DIR}/home/builder"
     sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
 
     # Configure root filesystem
+    sudo install -m 755 ${WORKDIR}/configscript.sh ${BUILDCHROOT_DIR}
     sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
 }
diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
index 048f76f..06172f7 100644
--- a/meta/recipes-devtools/buildchroot/files/build.sh
+++ b/meta/recipes-devtools/buildchroot/files/build.sh
@@ -29,7 +29,7 @@ install_cmd="${install_cmd} --allow-unauthenticated"
     #   APT::Get::List-Cleanup - do not erase obsolete packages list for
     #                            upstream in '/var/lib/apt/lists'
     apt-get update \
-        -o Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \
+        -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
         -o Dir::Etc::sourceparts="-" \
         -o APT::Get::List-Cleanup="0"
 
diff --git a/meta/recipes-devtools/buildchroot/files/configscript.sh b/meta/recipes-devtools/buildchroot/files/configscript.sh
index a5942b5..c2a599a 100644
--- a/meta/recipes-devtools/buildchroot/files/configscript.sh
+++ b/meta/recipes-devtools/buildchroot/files/configscript.sh
@@ -31,13 +31,3 @@ debconf-set-selections <<END
 locales locales/locales_to_be_generated multiselect en_US.UTF-8 UTF-8
 locales locales/default_environment_locale select en_US.UTF-8
 END
-
-#set up non-interactive configuration
-export DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true
-export LC_ALL=C LANGUAGE=C LANG=C
-
-#run pre installation script
-/var/lib/dpkg/info/dash.preinst install
-
-#configuring packages
-dpkg --configure -a
diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
deleted file mode 100644
index 480a4b8..0000000
--- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
+++ /dev/null
@@ -1,37 +0,0 @@
-# This software is a part of ISAR.
-# Copyright (C) 2015-2017 ilbers GmbH
-
-[General]
-noauth=true
-unpack=true
-ignorenativearch=true
-bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
-aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
-configscript=##CONFIG_SCRIPT##
-setupscript=##SETUP_SCRIPT##
-hookdir=##DIR_HOOKS##
-
-[base]
-source=##DISTRO_APT_SOURCE##
-suite=##DISTRO_SUITE##
-components=##DISTRO_COMPONENTS##
-packages=##BUILDCHROOT_PREINSTALL##
-omitdebsrc=true
-
-[updates]
-source=##DISTRO_APT_SOURCE##
-suite=##DISTRO_SUITE##-updates
-components=##DISTRO_COMPONENTS##
-omitdebsrc=true
-
-[security]
-source=##DISTRO_APT_SOURCE_SEC##
-suite=##DISTRO_SUITE##/updates
-components=##DISTRO_COMPONENTS##
-omitdebsrc=true
-
-[isar-apt]
-source=file:///isar-apt
-suite=isar
-components=main
-omitdebsrc=true
diff --git a/meta/recipes-devtools/buildchroot/files/setup.sh b/meta/recipes-devtools/buildchroot/files/setup.sh
deleted file mode 100755
index 1257739..0000000
--- a/meta/recipes-devtools/buildchroot/files/setup.sh
+++ /dev/null
@@ -1,82 +0,0 @@
-#!/bin/sh
-#
-# This software is a part of ISAR.
-# Copyright (C) 2015-2016 ilbers GmbH
-#
-# Based on multistrap/examples/chroot.sh
-
-set -e
-
-# The script is called with the following arguments:
-# $1 = $DIR  - the top directory of the bootstrapped system
-# $2 = $ARCH - the specified architecture, already checked with
-# dpkg-architecture.
-# setup.sh needs to be executable.
-
-TARGET=$1
-ARCH=$2
-
-# Prevent daemons from starting in buildchroot
-if [ -x "$TARGET/sbin/start-stop-daemon" ]; then
-    echo "initctl: Trying to prevent daemons from starting in $TARGET"
-
-    # Disable start-stop-daemon
-    mv $TARGET/sbin/start-stop-daemon $TARGET/sbin/start-stop-daemon.REAL
-    cat > $TARGET/sbin/start-stop-daemon << EOF
-#!/bin/sh
-echo
-echo Warning: Fake start-stop-daemon called, doing nothing
-EOF
-    chmod 755 $TARGET/sbin/start-stop-daemon
-fi
-
-if [ -x "$TARGET/sbin/initctl" ]; then
-    echo "start-stop-daemon: Trying to prevent daemons from starting in $TARGET"
-
-    # Disable initctl
-    mv "$TARGET/sbin/initctl" "$TARGET/sbin/initctl.REAL"
-    cat > $TARGET/sbin/initctl << EOF
-#!/bin/sh
-echo
-echo "Warning: Fake initctl called, doing nothing"
-EOF
-    chmod 755 $TARGET/sbin/initctl
-fi
-
-# Define sysvinit policy 101 to prevent daemons from starting in buildchroot
-if [ -x "$TARGET/sbin/init" -a ! -f "$TARGET/usr/sbin/policy-rc.d" ]; then
-    echo "sysvinit: Using policy-rc.d to prevent daemons from starting in $TARGET"
-
-    cat > $TARGET/usr/sbin/policy-rc.d << EOF
-#!/bin/sh
-echo "sysvinit: All runlevel operations denied by policy" >&2
-exit 101
-EOF
-    chmod a+x $TARGET/usr/sbin/policy-rc.d
-fi
-
-case $ARCH in
-    armel|armhf)
-        qemu_arch=arm
-        ;;
-    arm64)
-        qemu_arch=aarch64
-        ;;
-esac
-
-# Install QEMU emulator to execute foreign binaries
-if [ ! -x /usr/bin/qemu-${qemu_arch}-static ]; then
-    echo "qemu-${qemu_arch}-static binary not present, unable to execute target binaries"
-else
-    sudo cp /usr/bin/qemu-${qemu_arch}-static ${TARGET}/usr/bin
-fi
-
-# Set hostname
-echo "isar" > $TARGET/etc/hostname
-
-# Create packages build folder
-sudo install -d $TARGET/home/builder
-sudo chmod -R a+rw $TARGET/home/builder
-
-# Install host networking settings
-sudo cp /etc/resolv.conf $TARGET/etc
-- 
2.16.3

[PATCH v5 4/5] meta-isar/isar-image-base: switch to using isar-bootstrap

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

This patch removes the multistrap dependency of the isar-image-base
recipe and changes it to use the 'isar-bootstrap' and
'isar-bootstrap-helper' instead.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 .../recipes-core/images/files/download_dev-random  | 24 -------
 .../recipes-core/images/files/multistrap.conf.in   | 38 -----------
 meta-isar/recipes-core/images/files/setup.sh       | 77 ----------------------
 meta-isar/recipes-core/images/isar-image-base.bb   | 64 ++++++------------
 meta/classes/image.bbclass                         |  2 +-
 5 files changed, 22 insertions(+), 183 deletions(-)
 delete mode 100644 meta-isar/recipes-core/images/files/download_dev-random
 delete mode 100644 meta-isar/recipes-core/images/files/multistrap.conf.in
 delete mode 100755 meta-isar/recipes-core/images/files/setup.sh

diff --git a/meta-isar/recipes-core/images/files/download_dev-random b/meta-isar/recipes-core/images/files/download_dev-random
deleted file mode 100644
index f7f5fe5..0000000
--- a/meta-isar/recipes-core/images/files/download_dev-random
+++ /dev/null
@@ -1,24 +0,0 @@
-#!/bin/sh
-#
-# This software is a part of ISAR.
-# Copyright (C) 2017 ilbers GmbH
-
-set -e
-
-readonly ROOTFS="$1"
-
-if [ ! -c ${ROOTFS}/dev/random ]; then
-    rm -f ${ROOTFS}/dev/random
-    mknod "${ROOTFS}/dev/random" c 1 8
-    chmod 666 "${ROOTFS}/dev/random"
-    chown 0:0 "${ROOTFS}/dev/random"
-fi
-
-if [ ! -c ${ROOTFS}/dev/urandom ]; then
-    rm -f ${ROOTFS}/dev/urandom
-    mknod "${ROOTFS}/dev/urandom" c 1 9
-    chmod 666 "${ROOTFS}/dev/urandom"
-    chown 0:0 "${ROOTFS}/dev/urandom"
-fi
-
-exit 0
diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in
deleted file mode 100644
index 432b6af..0000000
--- a/meta-isar/recipes-core/images/files/multistrap.conf.in
+++ /dev/null
@@ -1,38 +0,0 @@
-# This software is a part of ISAR.
-# Copyright (C) 2015-2017 ilbers GmbH
-
-[General]
-noauth=true
-unpack=true
-ignorenativearch=true
-bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar
-aptsources=##DISTRO_MULTICONF_APTSOURCES##
-configscript=##CONFIG_SCRIPT##
-setupscript=##SETUP_SCRIPT##
-hookdir=##DIR_HOOKS##
-
-[base]
-source=##DISTRO_APT_SOURCE##
-suite=##DISTRO_SUITE##
-components=##DISTRO_COMPONENTS##
-packages=##IMAGE_PREINSTALL##
-omitdebsrc=true
-
-[updates]
-source=##DISTRO_APT_SOURCE##
-suite=##DISTRO_SUITE##-updates
-components=##DISTRO_COMPONENTS##
-omitdebsrc=true
-
-[security]
-source=##DISTRO_APT_SOURCE_SEC##
-suite=##DISTRO_SUITE##/updates
-components=##DISTRO_COMPONENTS##
-omitdebsrc=true
-
-[Isar]
-packages=##IMAGE_INSTALL##
-source=##DEPLOY_DIR_APT##
-suite=##ISAR_DISTRO_SUITE##
-components=main
-omitdebsrc=true
diff --git a/meta-isar/recipes-core/images/files/setup.sh b/meta-isar/recipes-core/images/files/setup.sh
deleted file mode 100755
index 86c065c..0000000
--- a/meta-isar/recipes-core/images/files/setup.sh
+++ /dev/null
@@ -1,77 +0,0 @@
-#!/bin/sh
-#
-# This software is a part of ISAR.
-# Copyright (C) 2015-2016 ilbers GmbH
-#
-# Based on multistrap/examples/chroot.sh
-
-set -e
-
-# The script is called with the following arguments:
-# $1 = $DIR  - the top directory of the bootstrapped system
-# $2 = $ARCH - the specified architecture, already checked with
-# dpkg-architecture.
-# setup.sh needs to be executable.
-
-TARGET=$1
-ARCH=$2
-
-# Prevent daemons from starting in postinstall during the initial "dpkg
-# --configure -a" under QEMU on the build host
-if [ -x "$TARGET/sbin/start-stop-daemon" ]; then
-    echo "initctl: Trying to prevent daemons from starting in $TARGET"
-
-    # Disable start-stop-daemon
-    mv $TARGET/sbin/start-stop-daemon $TARGET/sbin/start-stop-daemon.REAL
-    cat > $TARGET/sbin/start-stop-daemon << EOF
-#!/bin/sh
-echo
-echo Warning: Fake start-stop-daemon called, doing nothing
-EOF
-    chmod 755 $TARGET/sbin/start-stop-daemon
-fi
-
-if [ -x "$TARGET/sbin/initctl" ]; then
-    echo "start-stop-daemon: Trying to prevent daemons from starting in $TARGET"
-
-    # Disable initctl
-    mv "$TARGET/sbin/initctl" "$TARGET/sbin/initctl.REAL"
-    cat > $TARGET/sbin/initctl << EOF
-#!/bin/sh
-echo
-echo "Warning: Fake initctl called, doing nothing"
-EOF
-    chmod 755 $TARGET/sbin/initctl
-fi
-
-# Define sysvinit policy 101 to prevent daemons from starting in postinstall
-# during the initial "dpkg --configure -a" under QEMU on the build host
-if [ -x "$TARGET/sbin/init" -a ! -f "$TARGET/usr/sbin/policy-rc.d" ]; then
-    echo "sysvinit: Using policy-rc.d to prevent daemons from starting in $TARGET"
-
-    cat > $TARGET/usr/sbin/policy-rc.d << EOF
-#!/bin/sh
-echo "sysvinit: All runlevel operations denied by policy" >&2
-exit 101
-EOF
-    chmod a+x $TARGET/usr/sbin/policy-rc.d
-fi
-
-case $ARCH in
-    armel|armhf)
-        qemu_arch=arm
-        ;;
-    arm64)
-        qemu_arch=aarch64
-        ;;
-esac
-
-# Install QEMU emulator to execute foreign binaries
-if [ ! -x /usr/bin/qemu-${qemu_arch}-static ]; then
-    echo "qemu-${qemu_arch}-static binary not present, unable to execute target binaries"
-else
-    sudo cp /usr/bin/qemu-${qemu_arch}-static ${TARGET}/usr/bin
-fi
-
-# Set hostname
-echo "isar" > $TARGET/etc/hostname
diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index 80cb666..c4799d3 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -9,14 +9,12 @@ LICENSE = "gpl-2.0"
 LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
 
 FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:"
-SRC_URI = "file://multistrap.conf.in \
-	   file://${DISTRO_CONFIG_SCRIPT} \
-	   file://setup.sh \
-	   file://download_dev-random"
+SRC_URI = "file://${DISTRO_CONFIG_SCRIPT}"
 
 PV = "1.0"
 
 inherit image
+inherit isar-bootstrap-helper
 
 IMAGE_INSTALL += "linux-image-${KERNEL_NAME}"
 
@@ -28,50 +26,30 @@ IMAGE_PREINSTALL += "apt \
 
 WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
 
-do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap"
+do_rootfs[root_cleandirs] = "${IMAGE_ROOTFS} \
+                             ${IMAGE_ROOTFS}/isar-apt"
 
 do_rootfs() {
-    E="${@ bb.utils.export_proxies(d)}"
+    CDIRS="${@d.expand(d.getVarFlags("do_rootfs").get("root_cleandirs", ""))}"
+    if [ -n "$CDIRS" ]; then
+        sudo rm -rf $CDIRS
+        mkdir -p $CDIRS
+    fi
 
-    sudo rm -rf ${IMAGE_ROOTFS}
-
-    chmod +x "${WORKDIR}/${DISTRO_CONFIG_SCRIPT}"
-    chmod +x "${WORKDIR}/setup.sh"
-    install -m 755 "${WORKDIR}/download_dev-random" "${WORKDIR}/hooks_multistrap/"
-
-    # Multistrap accepts only relative path in configuration files, so get it:
-    cd ${TOPDIR}
-    WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
-
-    # Adjust multistrap config
-    sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \
-        -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
-        -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
-        -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
-        -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
-        -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
-        -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
-        -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
-        -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
-        -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
-        -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
-        -e 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \
-        -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
-           "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
-
-    # Do not use bitbake flag [dirs] here because this folder should have
-    # specific ownership.
-    [ ! -d ${IMAGE_ROOTFS}/proc ] && sudo install -d -o 0 -g 0 -m 555 ${IMAGE_ROOTFS}/proc
-    sudo mount -t proc none ${IMAGE_ROOTFS}/proc
-
-    # Create root filesystem. We must use sudo -E here to preserve the environment
-    # because of proxy settings
-    sudo -E multistrap -a ${DISTRO_ARCH} -d "${IMAGE_ROOTFS}" -f "${WORKDIR}/multistrap.conf"
+    setup_root_file_system "${IMAGE_ROOTFS}" ${IMAGE_PREINSTALL} ${IMAGE_INSTALL}
 
     # Configure root filesystem
+    sudo install -m 755 "${WORKDIR}/${DISTRO_CONFIG_SCRIPT}" "${IMAGE_ROOTFS}"
     sudo chroot ${IMAGE_ROOTFS} /${DISTRO_CONFIG_SCRIPT} ${MACHINE_SERIAL} \
-        ${BAUDRATE_TTY} ${ROOTFS_DEV} ${ROOTFS_TYPE}
-    sudo rm "${IMAGE_ROOTFS}/${DISTRO_CONFIG_SCRIPT}"
+                                                         ${BAUDRATE_TTY} \
+                                                         ${ROOTFS_DEV} \
+                                                         ${ROOTFS_TYPE}
 
-    sudo umount ${IMAGE_ROOTFS}/proc 2>/dev/null || true
+    # Cleanup
+    sudo rm "${IMAGE_ROOTFS}/${DISTRO_CONFIG_SCRIPT}"
+    sudo rm "${IMAGE_ROOTFS}/etc/apt/sources.list.d/isar-apt.list"
+    sudo umount -l ${IMAGE_ROOTFS}/isar-apt
+    sudo rmdir ${IMAGE_ROOTFS}/isar-apt
+    sudo umount -l ${IMAGE_ROOTFS}/dev
+    sudo umount -l ${IMAGE_ROOTFS}/proc
 }
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index e1bb5f6..a6537f9 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -22,7 +22,7 @@ INITRD_IMAGE ?= "${@get_image_name(d, 'initrd.img')}"
 inherit ${IMAGE_TYPE}
 
 do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
-do_rootfs[depends] = "isar-apt:do_cache_config"
+do_rootfs[depends] = "isar-apt:do_cache_config isar-bootstrap:do_deploy"
 
 do_rootfs() {
     die "No root filesystem function defined, please implement in your recipe"
-- 
2.16.3

[PATCH v5 5/5] meta-isar/multiconfig: remove multistrap references

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

This patch removes the left-over references to multistrap from the
configuration files.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta-isar/conf/distro/debian-jessie.conf       | 8 --------
 meta-isar/conf/distro/debian-stretch.conf      | 8 --------
 meta-isar/conf/distro/debian-wheezy.conf       | 8 --------
 meta-isar/conf/distro/raspbian-jessie.conf     | 8 --------
 meta-isar/conf/local.conf.sample               | 4 ++--
 meta-isar/conf/multiconfig/qemuarm-jessie.conf | 2 +-
 meta-isar/conf/multiconfig/qemuarm-wheezy.conf | 2 +-
 meta-isar/conf/multiconfig/rpi-jessie.conf     | 2 +-
 8 files changed, 5 insertions(+), 37 deletions(-)

diff --git a/meta-isar/conf/distro/debian-jessie.conf b/meta-isar/conf/distro/debian-jessie.conf
index df27fd7..ea46be6 100644
--- a/meta-isar/conf/distro/debian-jessie.conf
+++ b/meta-isar/conf/distro/debian-jessie.conf
@@ -9,11 +9,3 @@ DISTRO_KERNELS ?= "486 4kc-malta 586 5kc-malta 686-pae amd64 arm64 armmp \
     armmp-lpae ixp4xx kirkwood loongson-2e loongson-2f loongson-3 octeon \
     orion5x powerpc powerpc-smp powerpc64 powerpc64le r4k-ip22 r5k-ip32 \
     s390x sb1-bcm91250a versatile"
-
-# No longer needed:
-DISTRO_SUITE ?= "jessie"
-DISTRO_COMPONENTS ?= "main contrib non-free"
-DISTRO_APT_SOURCE ?= "http://ftp.debian.org/debian"
-DISTRO_APT_SOURCE_SEC ?= "http://security.debian.org/debian-security"
-DISTRO_MULTICONF_BOOTSTRAP ?= "base updates security"
-DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}"
diff --git a/meta-isar/conf/distro/debian-stretch.conf b/meta-isar/conf/distro/debian-stretch.conf
index 13c4f94..99d1737 100644
--- a/meta-isar/conf/distro/debian-stretch.conf
+++ b/meta-isar/conf/distro/debian-stretch.conf
@@ -6,11 +6,3 @@ DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"
 DISTRO_KERNELS ?= "4kc-malta 586 5kc-malta 686 686-pae amd64 arm64 armmp \
     armmp-lpae kirkwood loongson-3 marvell octeon orion5x powerpc64le \
     rt-686-pae rt-amd64 s390x"
-
-# No longer needed:
-DISTRO_SUITE ?= "stretch"
-DISTRO_COMPONENTS ?= "main contrib non-free"
-DISTRO_APT_SOURCE ?= "http://ftp.debian.org/debian"
-DISTRO_APT_SOURCE_SEC ?= "http://security.debian.org/debian-security"
-DISTRO_MULTICONF_BOOTSTRAP ?= "base updates security"
-DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}"
diff --git a/meta-isar/conf/distro/debian-wheezy.conf b/meta-isar/conf/distro/debian-wheezy.conf
index 2108f9b..5b2e936 100644
--- a/meta-isar/conf/distro/debian-wheezy.conf
+++ b/meta-isar/conf/distro/debian-wheezy.conf
@@ -7,11 +7,3 @@ DISTRO_APT_SOURCES += "conf/distro/debian-wheezy.list"
 DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"
 DISTRO_KERNELS ?= "486 686-pae amd64 rt-686-pae rt-amd64 versatile vexpress \
     iop32x ixp4xx kirkwood mv78xx0 mx5 omap orion5x"
-
-# No longer needed:
-DISTRO_SUITE ?= "wheezy"
-DISTRO_COMPONENTS ?= "main contrib non-free"
-DISTRO_APT_SOURCE ?= "http://ftp.debian.org/debian"
-DISTRO_APT_SOURCE_SEC ?= "http://security.debian.org/debian-security"
-DISTRO_MULTICONF_BOOTSTRAP ?= "base updates security"
-DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}"
diff --git a/meta-isar/conf/distro/raspbian-jessie.conf b/meta-isar/conf/distro/raspbian-jessie.conf
index 57d0e7d..72d6067 100644
--- a/meta-isar/conf/distro/raspbian-jessie.conf
+++ b/meta-isar/conf/distro/raspbian-jessie.conf
@@ -7,11 +7,3 @@ DISTRO_APT_SOURCES += "conf/distro/raspbian-jessie.list"
 DISTRO_APT_KEYS += "https://archive.raspbian.org/raspbian.public.key;sha256sum=ca59cd4f2bcbc3a1d41ba6815a02a8dc5c175467a59bd87edeac458f4a5345de"
 DISTRO_CONFIG_SCRIPT?= "raspbian-configscript.sh"
 DISTRO_KERNELS ?= "rpi rpi2 rpi-rpfv rpi2-rpfv"
-
-# No longer needed:
-DISTRO_SUITE ?= "jessie"
-DISTRO_COMPONENTS ?= "main contrib non-free firmware"
-DISTRO_APT_SOURCE ?= "http://archive.raspbian.org/raspbian"
-DISTRO_APT_SOURCE_SEC ?= ""
-DISTRO_MULTICONF_BOOTSTRAP ?= "base"
-DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}"
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 46bfae0..39875c1 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -23,10 +23,10 @@ MACHINE ??= "qemuarm"
 #
 # Multistrap Configuration Selection
 #
-# You need to select a specific multistrap configuration which will used for both:
+# You need to select a specific distribution configuration which will used for both:
 # generation of buildchroot environment and target root filesystem.
 #
-# This sets the default multistrap configuration:
+# This sets the default distribution configuration:
 DISTRO ??= "debian-jessie"
 DISTRO_ARCH ??= "armhf"
 
diff --git a/meta-isar/conf/multiconfig/qemuarm-jessie.conf b/meta-isar/conf/multiconfig/qemuarm-jessie.conf
index 74fb7d9..fe1d760 100644
--- a/meta-isar/conf/multiconfig/qemuarm-jessie.conf
+++ b/meta-isar/conf/multiconfig/qemuarm-jessie.conf
@@ -9,7 +9,7 @@
 # This sets the default machine to be qemuarm in this configuration:
 MACHINE ?= "qemuarm"
 
-# This sets the default multistrap configuration:
+# This sets the default distribution configuration:
 DISTRO ?= "debian-jessie"
 DISTRO_ARCH ?= "armhf"
 
diff --git a/meta-isar/conf/multiconfig/qemuarm-wheezy.conf b/meta-isar/conf/multiconfig/qemuarm-wheezy.conf
index 11355a4..c573254 100644
--- a/meta-isar/conf/multiconfig/qemuarm-wheezy.conf
+++ b/meta-isar/conf/multiconfig/qemuarm-wheezy.conf
@@ -9,7 +9,7 @@
 # This sets the default machine to be qemuarm in this configuration:
 MACHINE ?= "qemuarm"
 
-# This sets the default multistrap configuration:
+# This sets the default distribution configuration:
 DISTRO ?= "debian-wheezy"
 DISTRO_ARCH ?= "armhf"
 
diff --git a/meta-isar/conf/multiconfig/rpi-jessie.conf b/meta-isar/conf/multiconfig/rpi-jessie.conf
index 3f33731..fcc10d9 100644
--- a/meta-isar/conf/multiconfig/rpi-jessie.conf
+++ b/meta-isar/conf/multiconfig/rpi-jessie.conf
@@ -9,7 +9,7 @@
 # This sets the default machine to be rpi in this configuration:
 MACHINE ?= "rpi"
 
-# This sets the default multistrap configuration:
+# This sets the default distribution configuration:
 DISTRO ?= "raspbian-jessie"
 DISTRO_ARCH ?= "armhf"
 
-- 
2.16.3

Re: [PATCH v5 0/5] Debootstrap integration

[Baurzhan Ismagulov]

On Tue, Apr 03, 2018 at 12:07:57PM +0200, claudius.heine.ext@siemens.com wrote:
> this is the new version of this patchset, that fixes the
> generate_keyring task in isar-bootstrap for systems with read-only
> homedir.

Thanks, worked fine on my host. CI still in progress.


It's unfortunate that the series introduces regressions you wrote about
(changing mirrors, setting hostname). It's always better to fix the issues on
the spot. If there are no objections, I'd like to add TODOs to the patches.
Please let me know whether it's ok, or you would like to address those before
the merge.

What I'd really like to see is an update to doc/user_manual.md. Would you have
time for that in the next days?


If I understand the code correctly, there is also a security issue:

On Tue, Apr 03, 2018 at 12:08:00PM +0200, claudius.heine.ext@siemens.com wrote:
> +    CDIRS="${@d.expand(d.getVarFlags("do_build").get("root_cleandirs", ""))}"
> +    if [ -n "$CDIRS" ]; then
> +        sudo rm -rf $CDIRS
> +        mkdir -p $CDIRS
> +    fi

Should root_cleandirs items be checked for directory traversal ("/", "..") and
mounted filesystems in the subdirectories? If yes, do we want to drop the
feature from this series and address the issue in a separate step?


With kind regards,
Baurzhan.

Re: [PATCH v5 0/5] Debootstrap integration

[Claudius Heine]

Hi Baurzhan,

On 04/04/2018 10:34 PM, Baurzhan Ismagulov wrote:
> On Tue, Apr 03, 2018 at 12:07:57PM +0200, claudius.heine.ext@siemens.com wrote:
>> this is the new version of this patchset, that fixes the
>> generate_keyring task in isar-bootstrap for systems with read-only
>> homedir.
> 
> Thanks, worked fine on my host. CI still in progress.
> 
> 
> It's unfortunate that the series introduces regressions you wrote about
> (changing mirrors, setting hostname). It's always better to fix the issues on
> the spot. If there are no objections, I'd like to add TODOs to the patches.
> Please let me know whether it's ok, or you would like to address those before
> the merge.

I don't know about you, but I prefer having TODOs somewhere outside the 
project, maybe in the github issue tracker. At least in my experience 
TODOs together with code or in a separate file inside the repo are 
seldom updated and easily forgotten. If this project prefers having 
TODOs inside the repo, then sure, I have nothing against adding them 
somewhere.

I do plan of adding more features to this once this is merged. This 
patchset just provides the baseline.

> What I'd really like to see is an update to doc/user_manual.md. Would you have
> time for that in the next days?

I'll try.

> 
> 
> If I understand the code correctly, there is also a security issue:

Not sure if security is really a concern for isar yet. But I get your 
point that we should prevent possible accidents. :)

> 
> On Tue, Apr 03, 2018 at 12:08:00PM +0200, claudius.heine.ext@siemens.com wrote:
>> +    CDIRS="${@d.expand(d.getVarFlags("do_build").get("root_cleandirs", ""))}"
>> +    if [ -n "$CDIRS" ]; then
>> +        sudo rm -rf $CDIRS
>> +        mkdir -p $CDIRS
>> +    fi
> 
> Should root_cleandirs items be checked for directory traversal ("/", "..") and
> mounted filesystems in the subdirectories? If yes, do we want to drop the
> feature from this series and address the issue in a separate step?

This isn't really a new feature of isar yet. Its just the start of a 
general interface, that could be use everywhere when its acknowledged by 
the community and fully implemented. So it has to be improved anyway.

So I would say its good enough in this case, since setting those 
directories in the flag and removing them is currently bundled together 
in the same file. If we later centralized this step somewhere 
(base.bbclass) to make it available for every task, then checking it 
more thoroughly has to be done there.
So maybe add centralization of the 'root_cleandirs' task flag to the 
TODO list as well.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH v5 0/5] Debootstrap integration

[Jan Kiszka]

On 2018-04-05 10:03, [ext] Claudius Heine wrote:
>>
>> If I understand the code correctly, there is also a security issue:
> 
> Not sure if security is really a concern for isar yet. But I get your
> point that we should prevent possible accidents. :)

I would call this a "safety" concern - building images via Isar is not
secure in any way as long as we require root permissions.

Jan

Re: [PATCH v5 0/5] Debootstrap integration

[Jan Kiszka]

Hi all,

On 2018-04-05 10:03, [ext] Claudius Heine wrote:
> Hi Baurzhan,
> 
> On 04/04/2018 10:34 PM, Baurzhan Ismagulov wrote:
>> On Tue, Apr 03, 2018 at 12:07:57PM +0200,
>> claudius.heine.ext@siemens.com wrote:
>>> this is the new version of this patchset, that fixes the
>>> generate_keyring task in isar-bootstrap for systems with read-only
>>> homedir.
>>
>> Thanks, worked fine on my host. CI still in progress.
>>
>>
>> It's unfortunate that the series introduces regressions you wrote about
>> (changing mirrors, setting hostname). It's always better to fix the
>> issues on
>> the spot. If there are no objections, I'd like to add TODOs to the
>> patches.
>> Please let me know whether it's ok, or you would like to address those
>> before
>> the merge.
> 
> I don't know about you, but I prefer having TODOs somewhere outside the
> project, maybe in the github issue tracker. At least in my experience
> TODOs together with code or in a separate file inside the repo are
> seldom updated and easily forgotten. If this project prefers having
> TODOs inside the repo, then sure, I have nothing against adding them
> somewhere.
> 
> I do plan of adding more features to this once this is merged. This
> patchset just provides the baseline.
> 
>> What I'd really like to see is an update to doc/user_manual.md. Would
>> you have
>> time for that in the next days?
> 
> I'll try.
> 
>>
>>
>> If I understand the code correctly, there is also a security issue:
> 
> Not sure if security is really a concern for isar yet. But I get your
> point that we should prevent possible accidents. :)
> 
>>
>> On Tue, Apr 03, 2018 at 12:08:00PM +0200,
>> claudius.heine.ext@siemens.com wrote:
>>> +   
>>> CDIRS="${@d.expand(d.getVarFlags("do_build").get("root_cleandirs",
>>> ""))}"
>>> +    if [ -n "$CDIRS" ]; then
>>> +        sudo rm -rf $CDIRS
>>> +        mkdir -p $CDIRS
>>> +    fi
>>
>> Should root_cleandirs items be checked for directory traversal ("/",
>> "..") and
>> mounted filesystems in the subdirectories? If yes, do we want to drop the
>> feature from this series and address the issue in a separate step?
> 
> This isn't really a new feature of isar yet. Its just the start of a
> general interface, that could be use everywhere when its acknowledged by
> the community and fully implemented. So it has to be improved anyway.
> 
> So I would say its good enough in this case, since setting those
> directories in the flag and removing them is currently bundled together
> in the same file. If we later centralized this step somewhere
> (base.bbclass) to make it available for every task, then checking it
> more thoroughly has to be done there.
> So maybe add centralization of the 'root_cleandirs' task flag to the
> TODO list as well.
> 
> Cheers,
> Claudius
> 

what's the status of this series from upstream perspective now? Claudius
sent a documentation update. Any further requirements that need to be
fulfilled prior to this becoming ready for merge?

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v5 0/5] Debootstrap integration

[Baurzhan Ismagulov]

On Mon, Apr 09, 2018 at 12:50:56PM +0200, Jan Kiszka wrote:
> what's the status of this series from upstream perspective now? Claudius
> sent a documentation update. Any further requirements that need to be
> fulfilled prior to this becoming ready for merge?

Sorry for the delay, I'm still investigating the following issues:

1. CI fails with the series applied, and

2. The series drops Pre-Depends support and daemon starting prevention.


Regarding (1): As the series removes hostname setting, the image inherits the
hostname of the build host. I.e., images generated on different build hosts
have different hostnames. CI already checks the hostname and fails.

I'd suggest to restore at least the one-liner setting the hostname, possibly in
{debian,raspbian}-configscript.sh.


Regarding (2): The motivation for the old configscript.sh / setup.sh
distinction was to support Pre-Depends. "Sometimes, unpacking one package
requires that another package be first unpacked and configured. In this case,
the depending package must specify this dependency in the Pre-Depends control
field." [1].

Besides that, setup.sh prevented daemons from being started during the
configuration, which they otherwise are.

The series happens to work with Isar's minimal set of packages. ATM, I'm not
sure about the implications for real projects with more packages. What should
we do with this?

(I still think that in the end, we'll need more features than debootstrap
provides today, but that's a different story).


In general, I'm willing to move forward quickly. That said, I'd also like to
understand issues and have consensus when which issues are going to be
addressed. Till now, master has been the stable branch. Maybe we should
introduce an additional staging branch other than next. What do you think?


References:

1. https://www.debian.org/doc/debian-policy/


With kind regards,
Baurzhan.

Re: [PATCH v5 0/5] Debootstrap integration

[Jan Kiszka]

On 2018-04-09 14:48, Baurzhan Ismagulov wrote:
> On Mon, Apr 09, 2018 at 12:50:56PM +0200, Jan Kiszka wrote:
>> what's the status of this series from upstream perspective now? Claudius
>> sent a documentation update. Any further requirements that need to be
>> fulfilled prior to this becoming ready for merge?
> 
> Sorry for the delay, I'm still investigating the following issues:
> 
> 1. CI fails with the series applied, and
> 
> 2. The series drops Pre-Depends support and daemon starting prevention.
> 
> 
> Regarding (1): As the series removes hostname setting, the image inherits the
> hostname of the build host. I.e., images generated on different build hosts
> have different hostnames. CI already checks the hostname and fails.
> 
> I'd suggest to restore at least the one-liner setting the hostname, possibly in
> {debian,raspbian}-configscript.sh.

This should go into an exemplary customization package in meta-isar.
Then you can test if that package is properly installed and has the
desired effect of setting the hostname.

> 
> 
> Regarding (2): The motivation for the old configscript.sh / setup.sh
> distinction was to support Pre-Depends. "Sometimes, unpacking one package
> requires that another package be first unpacked and configured. In this case,
> the depending package must specify this dependency in the Pre-Depends control
> field." [1].
> 
> Besides that, setup.sh prevented daemons from being started during the
> configuration, which they otherwise are.
> 
> The series happens to work with Isar's minimal set of packages. ATM, I'm not
> sure about the implications for real projects with more packages. What should
> we do with this?

We either need concrete use cases for both aspects. Can you provide
examples that no longer work as expected or have other limitations now?
Otherwise, I would postpone these topics until the use cases show up.
Maybe that will happen when exposing the patch series to a broader
audience, but so far I heard nothing in this direction from our existing
users.

> 
> (I still think that in the end, we'll need more features than debootstrap
> provides today, but that's a different story).
> 
> 
> In general, I'm willing to move forward quickly. That said, I'd also like to
> understand issues and have consensus when which issues are going to be
> addressed. Till now, master has been the stable branch. Maybe we should
> introduce an additional staging branch other than next. What do you think?
> 

Well, the way master and next are maintained already differs a bit from
common pattern. Master is keep untouched much longer than usual, and
next is only updated after integration in Alex branches matured. The
more branches you have, the harder it gets for contributors to pick the
right one to develop against. If master is updated too infrequently, you
cannot use that. If next does not contain enough integration material,
you are easily forced to look for the latest staging branch to ensure
your patches still apply and work the other day.

So I would recommend to update master more frequently and merge patches
into next as soon as no review comments are pending any more. If you
really do not like to run CI over next (which may enforce some more
rebases of that branch - if only CI reveals issues), have a short-living
CI branch for that purpose, and only that.

Regards,
Jan

> 
> References:
> 
> 1. https://www.debian.org/doc/debian-policy/
> 
> 
> With kind regards,
> Baurzhan.
> 

Re: [PATCH v5 0/5] Debootstrap integration

[Claudius Heine]

Hi,

On 04/09/2018 02:48 PM, Baurzhan Ismagulov wrote:
> On Mon, Apr 09, 2018 at 12:50:56PM +0200, Jan Kiszka wrote:
>> what's the status of this series from upstream perspective now? Claudius
>> sent a documentation update. Any further requirements that need to be
>> fulfilled prior to this becoming ready for merge?
> 
> Sorry for the delay, I'm still investigating the following issues:
> 
> 1. CI fails with the series applied, and
> 
> 2. The series drops Pre-Depends support and daemon starting prevention.
> 
> 
> Regarding (1): As the series removes hostname setting, the image inherits the
> hostname of the build host. I.e., images generated on different build hosts
> have different hostnames. CI already checks the hostname and fails.
> 
> I'd suggest to restore at least the one-liner setting the hostname, possibly in
> {debian,raspbian}-configscript.sh.

I just send a patch that sets the hostname in the example-raw recipe.

> 
> Regarding (2): The motivation for the old configscript.sh / setup.sh
> distinction was to support Pre-Depends. "Sometimes, unpacking one package
> requires that another package be first unpacked and configured. In this case,
> the depending package must specify this dependency in the Pre-Depends control
> field." [1].
> 
> Besides that, setup.sh prevented daemons from being started during the
> configuration, which they otherwise are.
> 
> The series happens to work with Isar's minimal set of packages. ATM, I'm not
> sure about the implications for real projects with more packages. What should
> we do with this?

I took a look at those configscripts and some cleanup there might be 
necessary. For instance I don't know if:

     dpkg --configure -a

is still necessary to be done here, since all packages should be 
installed via apt now.

So maybe I should also prepare a patch that removes all those presumably 
unnecessary commands.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH v5 0/5] Debootstrap integration

[Baurzhan Ismagulov]

On Tue, Apr 10, 2018 at 01:38:26PM +0200, Claudius Heine wrote:
> I just send a patch that sets the hostname in the example-raw recipe.

Thanks, re-testing with CI.

The docs patch also looks good, thanks.

With kind regards,
Baurzhan.

Re: [PATCH v5 0/5] Debootstrap integration

[Baurzhan Ismagulov]

On Tue, Apr 03, 2018 at 12:07:57PM +0200, claudius.heine.ext@siemens.com wrote:
> this is the new version of this patchset, that fixes the
> generate_keyring task in isar-bootstrap for systems with read-only
> homedir.

Thanks, applied to next.

With kind regards,
Baurzhan.

Re: [PATCH v5 0/5] Debootstrap integration

[Baurzhan Ismagulov]

On Thu, Apr 05, 2018 at 11:16:43AM +0200, Jan Kiszka wrote:
> I would call this a "safety" concern - building images via Isar is not
> secure in any way as long as we require root permissions.

However we call it (directory traversal is commonly classified as a security
issue as opposed to safety as in health injury), it doesn't mean we should
easily add more of them just because the existing code already isn't perfect.
I've raised the issue to understand the implications, consider advantages and
disadvantages, and possibly determine TODOs if we end up with a trade-off.

For example, for this specific case, does upstream bitbake and / or OE check
whether cleandirs are within the build directory?

With kind regards,
Baurzhan.

Re: [PATCH v5 0/5] Debootstrap integration

[Jan Kiszka]

On 2018-04-11 08:28, Baurzhan Ismagulov wrote:
> On Thu, Apr 05, 2018 at 11:16:43AM +0200, Jan Kiszka wrote:
>> I would call this a "safety" concern - building images via Isar is not
>> secure in any way as long as we require root permissions.
> 
> However we call it (directory traversal is commonly classified as a security
> issue as opposed to safety as in health injury), it doesn't mean we should

safety: measures against accidental errors, hardware failures etc.
security: measures against malicious attackers (intentional errors)

Isar has currently no own means for the latter due to requiring root
privileges.

> easily add more of them just because the existing code already isn't perfect.
> I've raised the issue to understand the implications, consider advantages and
> disadvantages, and possibly determine TODOs if we end up with a trade-off.
> 
> For example, for this specific case, does upstream bitbake and / or OE check
> whether cleandirs are within the build directory?

That is a good question - though upstream does not have to run as root,
thus can be confined more easily.

Jan

Re: [PATCH v5 0/5] Debootstrap integration

[Claudius Heine]

Hi Baurzhan,

On 04/11/2018 08:28 AM, Baurzhan Ismagulov wrote:
> On Thu, Apr 05, 2018 at 11:16:43AM +0200, Jan Kiszka wrote:
>> I would call this a "safety" concern - building images via Isar is not
>> secure in any way as long as we require root permissions.
> 
> However we call it (directory traversal is commonly classified as a security
> issue as opposed to safety as in health injury), it doesn't mean we should
> easily add more of them just because the existing code already isn't perfect.
> I've raised the issue to understand the implications, consider advantages and
> disadvantages, and possibly determine TODOs if we end up with a trade-off.
> 
> For example, for this specific case, does upstream bitbake and / or OE check
> whether cleandirs are within the build directory?

Please take a look at the 'Centralize root_cleandirs feature' patchset, 
that should address some of those issues you mentioned. Discussion about 
this might be better there as well.

Thanks,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de