[RFC] [PATCH 0/9] Initial cross-compilation support

[RFC] [PATCH 0/9] Initial cross-compilation support

[Alexander Smirnov]

Hi all,

this series introduces basic cross-compilation support for Isar. The implementation
is based around new entity - buildchroot-cross, which is quite similar to original
buildchroot, but has host architecture.

This series tested for the following configuration only:

 $ bitbake multiconfig:qemui386

In this build:
 - libhello is cross-compiled
 - example-hello is compiled traditionally (but it's successfully linked with libhello)

TODO list:
 - Depending on the target architecture, various host tools should be installed. For example
   binutils package has the following names:
   * binutils
   * binutils-x86-64-linux-gnu
   * binutils-arm-linux-gnueabihf
 - Cross-dependency installation. This topic is still open for me because 'mk-build-deps' tool
   doesn't work here correctly. For example package depends on binutils, but instead of
   installation of 'binutils-arm-linux-gnueabihf' it tries to install 'binutils:armhf' and ruins
   the rootfs.

So, as usually, comments and suggestions are welcome :-)

With best redgards,
Alex

Alexander Smirnov (9):
  isar-bootstrap: Add routin to determine host arch
  isar-bootstrap: Move common part to include
  isar-bootstrap: Add host architecture support
  isar-bootstrap-helper: Add parameter to set arch
  buildchroot-cross: Initial implementation
  classes/dpkg*: Relocate dependency from buildchroot
  classes/dpkg*: Drop hardcoded buildchroot blobs
  classes/dpkg-cross: Initial implementation
  [FOR TESTING] libhello: Switch to cross compilation

 meta-isar/recipes-app/libhello/libhello.bb         |   2 +-
 meta/classes/dpkg-base.bbclass                     |  17 +-
 meta/classes/dpkg-cross.bbclass                    |  15 ++
 meta/classes/dpkg-raw.bbclass                      |   7 +-
 meta/classes/dpkg.bbclass                          |   7 +-
 meta/classes/image.bbclass                         |   2 +-
 meta/classes/isar-bootstrap-helper.bbclass         |  13 +-
 meta/conf/isar-bitbake.conf                        |   1 +
 .../isar-bootstrap/isar-bootstrap-host.bb          |  44 ++++
 .../isar-bootstrap/isar-bootstrap-target.bb        |  44 ++++
 meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 259 ---------------------
 .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 223 ++++++++++++++++++
 .../buildchroot/buildchroot-cross.bb               |  53 +++++
 meta/recipes-devtools/buildchroot/buildchroot.bb   |   2 +-
 .../buildchroot/files/build-cross.sh               |  29 +++
 15 files changed, 443 insertions(+), 275 deletions(-)
 create mode 100644 meta/classes/dpkg-cross.bbclass
 create mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
 create mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
 delete mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap.bb
 create mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
 create mode 100644 meta/recipes-devtools/buildchroot/buildchroot-cross.bb
 create mode 100644 meta/recipes-devtools/buildchroot/files/build-cross.sh

-- 
2.1.4

[PATCH 1/9] isar-bootstrap: Add routin to determine host arch

[Alexander Smirnov]

This method will be used to distinguish host and target bootstraps. The
standard python platform module is not robust here, because it provides
the architecture label from kernel. For example running platfotm.machine()
on Fedora kernel will return 'x86_64', what is invalid specificator for
Debian systems.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/isar-bootstrap-helper.bbclass | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 3c10fc7..2598ea3 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -14,6 +14,15 @@ def reverse_bb_array(d, varname):
     array = reversed(array.split())
     return " ".join(i for i in array)
 
+python () {
+    import subprocess
+    host_arch = subprocess.Popen("/usr/bin/dpkg --print-architecture",
+                                 shell=True,
+                                 stdout=subprocess.PIPE
+                                ).stdout.read().decode('utf-8').strip()
+    d.setVar("HOST_ARCH", host_arch);
+}
+
 setup_root_file_system() {
     CLEAN=""
     FSTAB=""
-- 
2.1.4

[PATCH 2/9] isar-bootstrap: Move common part to include

[Alexander Smirnov]

Eventually there should be two bootstraps: host and target. This
patch derives the acrchitecture independent part of original isar-bootstrap
recipe and put it in header file.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/image.bbclass                         |   2 +-
 .../isar-bootstrap/isar-bootstrap-target.bb        |  44 ++++
 meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 259 ---------------------
 .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 223 ++++++++++++++++++
 meta/recipes-devtools/buildchroot/buildchroot.bb   |   2 +-
 5 files changed, 269 insertions(+), 261 deletions(-)
 create mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
 delete mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap.bb
 create mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 3bdcb2f..4738cb8 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -47,7 +47,7 @@ INITRD_IMAGE ?= "${@get_image_name(d, 'initrd.img')[1]}"
 inherit ${IMAGE_TYPE}
 
 do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
-do_rootfs[depends] = "isar-apt:do_cache_config isar-bootstrap:do_deploy"
+do_rootfs[depends] = "isar-apt:do_cache_config isar-bootstrap-target:do_deploy"
 
 do_rootfs() {
     die "No root filesystem function defined, please implement in your recipe"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
new file mode 100644
index 0000000..bb37a68
--- /dev/null
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
@@ -0,0 +1,44 @@
+# Minimal target Debian root file system
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: MIT
+
+Description = "Minimal target Debian root file system"
+
+include isar-bootstrap.inc
+
+do_bootstrap[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_bootstrap[vardeps] += "DISTRO_APT_SOURCES"
+do_bootstrap[vardeps] += "DISTRO_APT_PREMIRRORS"
+do_bootstrap() {
+    if [ -e "${ROOTFSDIR}" ]; then
+       sudo umount -l "${ROOTFSDIR}/dev" || true
+       sudo umount -l "${ROOTFSDIR}/proc" || true
+       sudo rm -rf "${ROOTFSDIR}"
+    fi
+    E="${@bb.utils.export_proxies(d)}"
+    sudo -E "${DEBOOTSTRAP}" --verbose \
+                             --variant=minbase \
+                             --arch="${DISTRO_ARCH}" \
+                             --include=locales \
+                             ${@get_distro_components_argument(d)} \
+                             ${DEBOOTSTRAP_KEYRING} \
+                             "${@get_distro_suite(d)}" \
+                             "${ROOTFSDIR}" \
+                             "${@get_distro_source(d)}"
+}
+addtask bootstrap before do_build after do_generate_keyring
+
+do_deploy[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_deploy[dirs] = "${DEPLOY_DIR_IMAGE}"
+do_deploy() {
+    ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_DIR_IMAGE}/isar-bootstrap-${DISTRO}-${DISTRO_ARCH}"
+}
+addtask deploy before do_build after do_apt_update
+
+CLEANFUNCS = "clean_deploy"
+clean_deploy() {
+     rm -f "${DEPLOY_DIR_IMAGE}/${PN}-${DISTRO}-${DISTRO_ARCH}"
+}
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb
deleted file mode 100644
index 497a4f4..0000000
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb
+++ /dev/null
@@ -1,259 +0,0 @@
-# Minimal debian root file system
-#
-# This software is a part of ISAR.
-# Copyright (c) Siemens AG, 2018
-#
-# SPDX-License-Identifier: MIT
-
-Description = "Minimal debian root file system"
-
-LICENSE = "gpl-2.0"
-LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
-FILESPATH_prepend := "${THISDIR}/files:"
-SRC_URI = " \
-    file://isar-apt.conf \
-    file://isar-apt-fallback.conf \
-    file://locale \
-    file://chroot-setup.sh"
-PV = "1.0"
-
-WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
-DEBOOTSTRAP ?= "qemu-debootstrap"
-ROOTFSDIR = "${WORKDIR}/rootfs"
-APTPREFS = "${WORKDIR}/apt-preferences"
-APTSRCS = "${WORKDIR}/apt-sources"
-APTKEYFILES = ""
-APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
-DEBOOTSTRAP_KEYRING = ""
-
-python () {
-    from urllib.parse import urlparse
-    distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
-    if distro_apt_keys:
-        d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
-        for key in distro_apt_keys.split():
-            url = urlparse(key)
-            filename = os.path.basename(url.path)
-            d.appendVar("SRC_URI", " %s" % key)
-            d.appendVar("APTKEYFILES", " %s" % filename)
-}
-
-def aggregate_files(d, file_list, file_out):
-    import shutil
-
-    with open(file_out, "wb") as out_fd:
-        for entry in file_list:
-            entry_real = bb.parse.resolve_file(entry, d)
-            with open(entry_real, "rb") as in_fd:
-                 shutil.copyfileobj(in_fd, out_fd, 1024*1024*10)
-            out_fd.write("\n".encode())
-
-def parse_aptsources_list_line(source_list_line):
-    import re
-
-    s = source_list_line.strip()
-
-    if s.startswith("#"):
-        return None
-
-    type, s = re.split("\s+", s, maxsplit=1)
-    if type not in ["deb", "deb-src"]:
-        return None
-
-    options = ""
-    options_match = re.match("\[\s*(\S+=\S+(?=\s))*\s*(\S+=\S+)\s*\]\s+", s)
-    if options_match:
-        options = options_match.group(0).strip()
-        s = s[options_match.end():]
-
-    source, s = re.split("\s+", s, maxsplit=1)
-
-    suite, s = re.split("\s+", s, maxsplit=1)
-
-    components = " ".join(s.split())
-
-    return [type, options, source, suite, components]
-
-def get_apt_source_mirror(d, aptsources_entry_list):
-    import re
-
-    premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or ""
-    mirror_list = [entry.split()
-                  for entry in premirrors.split('\\n')
-                  if any(entry)]
-
-    for regex, replace in mirror_list:
-        match = re.search(regex, aptsources_entry_list[2])
-
-        if match:
-            new_aptsources_entry_list = aptsources_entry_list.copy()
-            new_aptsources_entry_list[2] = re.sub(regex, replace,
-                                                  aptsources_entry_list[2],
-                                                  count = 1)
-            return new_aptsources_entry_list
-
-    return aptsources_entry_list
-
-def aggregate_aptsources_list(d, file_list, file_out):
-    import shutil
-
-    with open(file_out, "wb") as out_fd:
-        for entry in file_list:
-            entry_real = bb.parse.resolve_file(entry, d)
-            with open(entry_real, "r") as in_fd:
-                for line in in_fd:
-                    parsed = parse_aptsources_list_line(line)
-                    if parsed:
-                        parsed = get_apt_source_mirror(d, parsed)
-                        out_fd.write(" ".join(parsed).encode())
-                    else:
-                        out_fd.write(line.encode())
-                    out_fd.write("\n".encode())
-            out_fd.write("\n".encode())
-
-def get_distro_primary_source_entry(d):
-    apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or "").split()
-    for entry in apt_sources_list:
-        entry_real = bb.parse.resolve_file(entry, d)
-        with open(entry_real, "r") as in_fd:
-            for line in in_fd:
-                parsed = parse_aptsources_list_line(line)
-                if parsed:
-                    parsed = get_apt_source_mirror(d, parsed)
-                    if parsed[0] == "deb":
-                        return parsed[2:]
-    return ["", "", ""]
-
-def get_distro_source(d):
-    return get_distro_primary_source_entry(d)[0]
-
-def get_distro_suite(d):
-    return get_distro_primary_source_entry(d)[1]
-
-def get_distro_components_argument(d):
-    components = get_distro_primary_source_entry(d)[2]
-    if components and components.strip():
-        return "--components=%s" % ",".join(components.split())
-    else:
-        return ""
-
-do_generate_keyring[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-do_generate_keyring[dirs] = "${WORKDIR}"
-do_generate_keyring[vardeps] += "DISTRO_APT_KEYS"
-do_generate_keyring() {
-    if [ -n "${@d.getVar("APTKEYFILES", True) or ""}" ]; then
-        for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
-           gpg --no-default-keyring --keyring "${APTKEYRING}" \
-               --homedir "${WORKDIR}" --import "$keyfile"
-        done
-    fi
-}
-addtask generate_keyring before do_build after do_unpack
-
-do_apt_config_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-do_apt_config_prepare[dirs] = "${WORKDIR}"
-do_apt_config_prepare[vardeps] += "\
-                                   APTPREFS \
-                                   DISTRO_APT_PREFERENCES \
-                                   DEBDISTRONAME \
-                                   APTSRCS \
-                                   DISTRO_APT_SOURCES \
-                                  "
-python do_apt_config_prepare() {
-    apt_preferences_out = d.getVar("APTPREFS", True)
-    apt_preferences_list = (d.getVar("DISTRO_APT_PREFERENCES", True) or ""
-                           ).split()
-    aggregate_files(d, apt_preferences_list, apt_preferences_out)
-
-    apt_sources_out = d.getVar("APTSRCS", True)
-    apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or "").split()
-
-    aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
-}
-addtask apt_config_prepare before do_build after do_unpack
-
-do_bootstrap[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-do_bootstrap[vardeps] += "DISTRO_APT_SOURCES"
-do_bootstrap[vardeps] += "DISTRO_APT_PREMIRRORS"
-do_bootstrap() {
-    if [ -e "${ROOTFSDIR}" ]; then
-       sudo umount -l "${ROOTFSDIR}/dev" || true
-       sudo umount -l "${ROOTFSDIR}/proc" || true
-       sudo rm -rf "${ROOTFSDIR}"
-    fi
-    E="${@bb.utils.export_proxies(d)}"
-    sudo -E "${DEBOOTSTRAP}" --verbose \
-                             --variant=minbase \
-                             --arch="${DISTRO_ARCH}" \
-                             --include=locales \
-                             ${@get_distro_components_argument(d)} \
-                             ${DEBOOTSTRAP_KEYRING} \
-                             "${@get_distro_suite(d)}" \
-                             "${ROOTFSDIR}" \
-                             "${@get_distro_source(d)}"
-}
-addtask bootstrap before do_build after do_generate_keyring
-
-do_set_locale() {
-    sudo install -v -m644 "${WORKDIR}/locale" "${ROOTFSDIR}/etc/locale"
-
-    sudo sed -i '/en_US.UTF-8 UTF-8/s/^#//g' "${ROOTFSDIR}/etc/locale.gen"
-    sudo -E chroot "${ROOTFSDIR}" /usr/sbin/locale-gen
-}
-addtask set_locale after do_bootstrap
-
-do_setup_chroot() {
-    sudo install -v -m755 "${WORKDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh"
-    sudo "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"
-}
-addtask setup_chroot before do_build after do_bootstrap
-
-def get_host_release():
-    import platform
-    rel = platform.release()
-    return rel
-
-do_apt_config_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-do_apt_config_install() {
-    sudo mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"
-    sudo install -v -m644 "${APTPREFS}" \
-                          "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
-    sudo mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
-    sudo install -v -m644 "${APTSRCS}" \
-                          "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
-    sudo rm -f "${ROOTFSDIR}/etc/apt/sources.list"
-    sudo mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"
-    sudo install -v -m644 "${WORKDIR}/isar-apt.conf" \
-                          "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf"
-
-    if [ "${@get_distro_suite(d)}" = "stretch" ] && [ "${@get_host_release().split('.')[0]}" -lt "4" ]; then
-        sudo install -v -m644 "${WORKDIR}/isar-apt-fallback.conf" \
-                              "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf"
-    fi
-}
-addtask apt_config_install before do_build after do_bootstrap do_apt_config_prepare
-
-do_apt_update[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-do_apt_update() {
-    sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${ROOTFSDIR}/dev
-    sudo mount -t proc none ${ROOTFSDIR}/proc
-
-    E="${@bb.utils.export_proxies(d)}"
-    export DEBIAN_FRONTEND=noninteractive
-    sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
-    sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
-                                      -o Debug::pkgProblemResolver=yes
-}
-addtask apt_update before do_build after do_apt_config_install do_set_locale do_setup_chroot
-
-do_deploy[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-do_deploy[dirs] = "${DEPLOY_DIR_IMAGE}"
-do_deploy() {
-    ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_DIR_IMAGE}/${PN}-${DISTRO}-${DISTRO_ARCH}"
-}
-addtask deploy before do_build after do_apt_update
-
-CLEANFUNCS = "clean_deploy"
-clean_deploy() {
-     rm -f "${DEPLOY_DIR_IMAGE}/${PN}-${DISTRO}-${DISTRO_ARCH}"
-}
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
new file mode 100644
index 0000000..263f8af
--- /dev/null
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -0,0 +1,223 @@
+# Minimal debian root file system
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: MIT
+
+LICENSE = "gpl-2.0"
+LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+FILESPATH_prepend := "${THISDIR}/files:"
+SRC_URI = " \
+    file://isar-apt.conf \
+    file://isar-apt-fallback.conf \
+    file://locale \
+    file://chroot-setup.sh"
+PV = "1.0"
+
+WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
+DEBOOTSTRAP ?= "qemu-debootstrap"
+ROOTFSDIR = "${WORKDIR}/rootfs"
+APTPREFS = "${WORKDIR}/apt-preferences"
+APTSRCS = "${WORKDIR}/apt-sources"
+APTKEYFILES = ""
+APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
+DEBOOTSTRAP_KEYRING = ""
+
+python () {
+    from urllib.parse import urlparse
+    distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
+    if distro_apt_keys:
+        d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
+        for key in distro_apt_keys.split():
+            url = urlparse(key)
+            filename = os.path.basename(url.path)
+            d.appendVar("SRC_URI", " %s" % key)
+            d.appendVar("APTKEYFILES", " %s" % filename)
+}
+
+def aggregate_files(d, file_list, file_out):
+    import shutil
+
+    with open(file_out, "wb") as out_fd:
+        for entry in file_list:
+            entry_real = bb.parse.resolve_file(entry, d)
+            with open(entry_real, "rb") as in_fd:
+                 shutil.copyfileobj(in_fd, out_fd, 1024*1024*10)
+            out_fd.write("\n".encode())
+
+def parse_aptsources_list_line(source_list_line):
+    import re
+
+    s = source_list_line.strip()
+
+    if s.startswith("#"):
+        return None
+
+    type, s = re.split("\s+", s, maxsplit=1)
+    if type not in ["deb", "deb-src"]:
+        return None
+
+    options = ""
+    options_match = re.match("\[\s*(\S+=\S+(?=\s))*\s*(\S+=\S+)\s*\]\s+", s)
+    if options_match:
+        options = options_match.group(0).strip()
+        s = s[options_match.end():]
+
+    source, s = re.split("\s+", s, maxsplit=1)
+
+    suite, s = re.split("\s+", s, maxsplit=1)
+
+    components = " ".join(s.split())
+
+    return [type, options, source, suite, components]
+
+def get_apt_source_mirror(d, aptsources_entry_list):
+    import re
+
+    premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or ""
+    mirror_list = [entry.split()
+                  for entry in premirrors.split('\\n')
+                  if any(entry)]
+
+    for regex, replace in mirror_list:
+        match = re.search(regex, aptsources_entry_list[2])
+
+        if match:
+            new_aptsources_entry_list = aptsources_entry_list.copy()
+            new_aptsources_entry_list[2] = re.sub(regex, replace,
+                                                  aptsources_entry_list[2],
+                                                  count = 1)
+            return new_aptsources_entry_list
+
+    return aptsources_entry_list
+
+def aggregate_aptsources_list(d, file_list, file_out):
+    import shutil
+
+    with open(file_out, "wb") as out_fd:
+        for entry in file_list:
+            entry_real = bb.parse.resolve_file(entry, d)
+            with open(entry_real, "r") as in_fd:
+                for line in in_fd:
+                    parsed = parse_aptsources_list_line(line)
+                    if parsed:
+                        parsed = get_apt_source_mirror(d, parsed)
+                        out_fd.write(" ".join(parsed).encode())
+                    else:
+                        out_fd.write(line.encode())
+                    out_fd.write("\n".encode())
+            out_fd.write("\n".encode())
+
+def get_distro_primary_source_entry(d):
+    apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or "").split()
+    for entry in apt_sources_list:
+        entry_real = bb.parse.resolve_file(entry, d)
+        with open(entry_real, "r") as in_fd:
+            for line in in_fd:
+                parsed = parse_aptsources_list_line(line)
+                if parsed:
+                    parsed = get_apt_source_mirror(d, parsed)
+                    if parsed[0] == "deb":
+                        return parsed[2:]
+    return ["", "", ""]
+
+def get_distro_source(d):
+    return get_distro_primary_source_entry(d)[0]
+
+def get_distro_suite(d):
+    return get_distro_primary_source_entry(d)[1]
+
+def get_distro_components_argument(d):
+    components = get_distro_primary_source_entry(d)[2]
+    if components and components.strip():
+        return "--components=%s" % ",".join(components.split())
+    else:
+        return ""
+
+do_generate_keyring[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_generate_keyring[dirs] = "${WORKDIR}"
+do_generate_keyring[vardeps] += "DISTRO_APT_KEYS"
+do_generate_keyring() {
+    if [ -n "${@d.getVar("APTKEYFILES", True) or ""}" ]; then
+        for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
+           gpg --no-default-keyring --keyring "${APTKEYRING}" \
+               --homedir "${WORKDIR}" --import "$keyfile"
+        done
+    fi
+}
+addtask generate_keyring before do_build after do_unpack
+
+do_apt_config_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_apt_config_prepare[dirs] = "${WORKDIR}"
+do_apt_config_prepare[vardeps] += "\
+                                   APTPREFS \
+                                   DISTRO_APT_PREFERENCES \
+                                   DEBDISTRONAME \
+                                   APTSRCS \
+                                   DISTRO_APT_SOURCES \
+                                  "
+python do_apt_config_prepare() {
+    apt_preferences_out = d.getVar("APTPREFS", True)
+    apt_preferences_list = (d.getVar("DISTRO_APT_PREFERENCES", True) or ""
+                           ).split()
+    aggregate_files(d, apt_preferences_list, apt_preferences_out)
+
+    apt_sources_out = d.getVar("APTSRCS", True)
+    apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or "").split()
+
+    aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
+}
+addtask apt_config_prepare before do_build after do_unpack
+
+do_set_locale() {
+    sudo install -v -m644 "${WORKDIR}/locale" "${ROOTFSDIR}/etc/locale"
+
+    sudo sed -i '/en_US.UTF-8 UTF-8/s/^#//g' "${ROOTFSDIR}/etc/locale.gen"
+    sudo -E chroot "${ROOTFSDIR}" /usr/sbin/locale-gen
+}
+addtask set_locale after do_bootstrap
+
+do_setup_chroot() {
+    sudo install -v -m755 "${WORKDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh"
+    sudo "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"
+}
+addtask setup_chroot before do_build after do_bootstrap
+
+def get_host_release():
+    import platform
+    rel = platform.release()
+    return rel
+
+do_apt_config_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_apt_config_install() {
+    sudo mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"
+    sudo install -v -m644 "${APTPREFS}" \
+                          "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
+    sudo mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
+    sudo install -v -m644 "${APTSRCS}" \
+                          "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
+    sudo rm -f "${ROOTFSDIR}/etc/apt/sources.list"
+    sudo mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"
+    sudo install -v -m644 "${WORKDIR}/isar-apt.conf" \
+                          "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf"
+
+    if [ "${@get_distro_suite(d)}" = "stretch" ] && [ "${@get_host_release().split('.')[0]}" -lt "4" ]; then
+        sudo install -v -m644 "${WORKDIR}/isar-apt-fallback.conf" \
+                              "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf"
+    fi
+}
+addtask apt_config_install before do_build after do_bootstrap do_apt_config_prepare
+
+do_apt_update[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_apt_update() {
+    sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${ROOTFSDIR}/dev
+    sudo mount -t proc none ${ROOTFSDIR}/proc
+
+    E="${@bb.utils.export_proxies(d)}"
+    export DEBIAN_FRONTEND=noninteractive
+    sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
+    sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
+                                      -o Debug::pkgProblemResolver=yes
+}
+addtask apt_update before do_build after do_apt_config_install do_set_locale do_setup_chroot
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 538c577..7db8206 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -66,7 +66,7 @@ do_build[root_cleandirs] = "${BUILDCHROOT_DIR} \
                             ${BUILDCHROOT_DIR}/isar-apt \
                             ${BUILDCHROOT_DIR}/downloads \
                             ${BUILDCHROOT_DIR}/home/builder"
-do_build[depends] = "isar-apt:do_cache_config isar-bootstrap:do_deploy"
+do_build[depends] = "isar-apt:do_cache_config isar-bootstrap-target:do_deploy"
 
 do_build() {
     setup_root_file_system "${BUILDCHROOT_DIR}" ${BUILDCHROOT_PREINSTALL}
-- 
2.1.4

[PATCH 3/9] isar-bootstrap: Add host architecture support

[Alexander Smirnov]

Add recipe to generate bootstrap with the host architecture.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 .../isar-bootstrap/isar-bootstrap-host.bb          | 44 ++++++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
new file mode 100644
index 0000000..c61c665
--- /dev/null
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -0,0 +1,44 @@
+# Minimal host Debian root file system
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: MIT
+
+Description = "Minimal host Debian root file system"
+
+include isar-bootstrap.inc
+inherit isar-bootstrap-helper
+
+do_bootstrap[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_bootstrap[vardeps] += "DISTRO_APT_SOURCES"
+do_bootstrap[vardeps] += "DISTRO_APT_PREMIRRORS"
+do_bootstrap() {
+    if [ -e "${ROOTFSDIR}" ]; then
+       sudo umount -l "${ROOTFSDIR}/dev" || true
+       sudo umount -l "${ROOTFSDIR}/proc" || true
+       sudo rm -rf "${ROOTFSDIR}"
+    fi
+    E="${@bb.utils.export_proxies(d)}"
+    sudo -E "${DEBOOTSTRAP}" --verbose \
+                             --variant=minbase \
+                             --include=locales \
+                             ${@get_distro_components_argument(d)} \
+                             ${DEBOOTSTRAP_KEYRING} \
+                             "${@get_distro_suite(d)}" \
+                             "${ROOTFSDIR}" \
+                             "${@get_distro_source(d)}"
+}
+addtask bootstrap before do_build after do_generate_keyring
+
+do_deploy[stamp-extra-info] = "${DISTRO}-${HOST_ARCH}"
+do_deploy[dirs] = "${DEPLOY_DIR_IMAGE}"
+do_deploy() {
+    ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_DIR_IMAGE}/isar-bootstrap-${DISTRO}-${HOST_ARCH}"
+}
+addtask deploy before do_build after do_apt_update
+
+CLEANFUNCS = "clean_deploy"
+clean_deploy() {
+     rm -f "${DEPLOY_DIR_IMAGE}/${PN}-${DISTRO}-${HOST_ARCH}"
+}
-- 
2.1.4

[PATCH 4/9] isar-bootstrap-helper: Add parameter to set arch

[Alexander Smirnov]

Initially this helper could produce only rootfs with hardcoded target
architecture. This patch adds parameter, so now it's possible to also
produce rootfs with the host architecture.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/isar-bootstrap-helper.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 2598ea3..b288937 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -26,10 +26,12 @@ python () {
 setup_root_file_system() {
     CLEAN=""
     FSTAB=""
+    ROOTFS_ARCH="${DISTRO_ARCH}"
     while true; do
         case "$1" in
         --clean) CLEAN=1 ;;
         --fstab) FSTAB=$2; shift ;;
+        --host-arch) ROOTFS_ARCH=${HOST_ARCH} ;;
         -*) bbfatal "$0: invalid option specified: $1" ;;
         *) break ;;
         esac
@@ -42,7 +44,7 @@ setup_root_file_system() {
     CLEAN_FILES="${ROOTFSDIR}/etc/hostname ${ROOTFSDIR}/etc/resolv.conf"
 
     sudo cp -Trpfx \
-        "${DEPLOY_DIR_IMAGE}/isar-bootstrap-${DISTRO}-${DISTRO_ARCH}/" \
+        "${DEPLOY_DIR_IMAGE}/isar-bootstrap-${DISTRO}-$ROOTFS_ARCH/" \
         "$ROOTFSDIR"
     [ -n "${FSTAB}" ] && cat ${FSTAB} | sudo tee "$ROOTFSDIR/etc/fstab"
 
-- 
2.1.4

[PATCH 5/9] buildchroot-cross: Initial implementation

[Alexander Smirnov]

Add initial generation of buildchroot for cross-compilation.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/conf/isar-bitbake.conf                        |  1 +
 .../buildchroot/buildchroot-cross.bb               | 53 ++++++++++++++++++++++
 .../buildchroot/files/build-cross.sh               | 29 ++++++++++++
 3 files changed, 83 insertions(+)
 create mode 100644 meta/recipes-devtools/buildchroot/buildchroot-cross.bb
 create mode 100644 meta/recipes-devtools/buildchroot/files/build-cross.sh

diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
index ead7798..1412508 100644
--- a/meta/conf/isar-bitbake.conf
+++ b/meta/conf/isar-bitbake.conf
@@ -22,6 +22,7 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}"
 DL_DIR = "${TOPDIR}/downloads"
 SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
 BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
+BUILDCHROOT_CROSS_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-cross/rootfs"
 CACHE = "${TMPDIR}/cache"
 
 OVERRIDES_append = ":${DISTRO}:${DISTRO_ARCH}"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-cross.bb b/meta/recipes-devtools/buildchroot/buildchroot-cross.bb
new file mode 100644
index 0000000..935a626
--- /dev/null
+++ b/meta/recipes-devtools/buildchroot/buildchroot-cross.bb
@@ -0,0 +1,53 @@
+# Root filesystem for packages cross-building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2016 ilbers GmbH
+
+DESCRIPTION = "Isar development cross-filesystem"
+
+LICENSE = "gpl-2.0"
+LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+
+FILESPATH_prepend := "${THISDIR}/files:"
+SRC_URI = "file://configscript.sh \
+           file://build-cross.sh"
+PV = "1.0"
+
+inherit isar-bootstrap-helper
+
+BUILDCHROOT_CROSS_PREINSTALL ?= "gcc-multilib \
+                                 make \
+                                 build-essential \
+                                 debhelper \
+                                 autotools-dev \
+                                 dpkg \
+                                 locales \
+                                 docbook-to-man \
+                                 apt \
+                                 automake \
+                                 devscripts \
+                                 equivs"
+
+# TODO: make this inclusion depending on the target arch
+BUILDCHROOT_CROSS_PREINSTALL += "binutils"
+
+WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
+
+do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_build[root_cleandirs] = "${BUILDCHROOT_CROSS_DIR} \
+                            ${BUILDCHROOT_CROSS_DIR}/isar-apt \
+                            ${BUILDCHROOT_CROSS_DIR}/downloads \
+                            ${BUILDCHROOT_CROSS_DIR}/home/builder"
+do_build[depends] = "isar-apt:do_cache_config isar-bootstrap-host:do_deploy"
+
+do_build() {
+    setup_root_file_system --host-arch "${BUILDCHROOT_CROSS_DIR}" ${BUILDCHROOT_CROSS_PREINSTALL}
+
+    # Install package builder script
+    sudo chmod -R a+rw "${BUILDCHROOT_CROSS_DIR}/home/builder"
+    sudo install -m 755 ${WORKDIR}/build-cross.sh ${BUILDCHROOT_CROSS_DIR}/build.sh
+
+    # Configure root filesystem
+    sudo install -m 755 ${WORKDIR}/configscript.sh ${BUILDCHROOT_CROSS_DIR}
+    sudo chroot ${BUILDCHROOT_CROSS_DIR} /configscript.sh
+}
diff --git a/meta/recipes-devtools/buildchroot/files/build-cross.sh b/meta/recipes-devtools/buildchroot/files/build-cross.sh
new file mode 100644
index 0000000..8c3ddaf
--- /dev/null
+++ b/meta/recipes-devtools/buildchroot/files/build-cross.sh
@@ -0,0 +1,29 @@
+#!/bin/bash
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2017 ilbers GmbH
+# Copyright (c) 2018 Siemens AG
+
+set -e
+
+# Go to build directory
+cd $1
+
+# Add target architecture
+dpkg --add-architecture $2
+
+# To avoid Perl locale warnings:
+export LC_ALL=C
+export LANG=C
+export LANGUAGE=C
+
+# If autotools files have been created, update their timestamp to
+# prevent them from being regenerated
+for i in configure aclocal.m4 Makefile.am Makefile.in; do
+    if [ -f "${i}" ]; then
+        touch "${i}"
+    fi
+done
+
+# Build the package
+dpkg-buildpackage -a$2
-- 
2.1.4

[PATCH 6/9] classes/dpkg*: Relocate dependency from buildchroot

[Alexander Smirnov]

Move dependency from buildchroot to the classes that actually require
buildchroot. This will make possible to re-use dpkg-base class for
buildchroot-cross.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 3 ---
 meta/classes/dpkg-raw.bbclass  | 3 +++
 meta/classes/dpkg.bbclass      | 3 +++
 3 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 75b45a7..24fe2ca 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -14,9 +14,6 @@ do_adjust_git[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 inherit patch
 addtask patch after do_adjust_git before do_build
 
-# Add dependency from buildchroot creation
-do_build[depends] = "buildchroot:do_build"
-
 # Add dependency between Isar recipes
 DEPENDS ?= ""
 do_build[deptask] = "do_deploy_deb"
diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index 28233ac..5c7be6b 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -6,6 +6,9 @@ inherit dpkg-base
 DEBIAN_DEPENDS ?= ""
 MAINTAINER ?= "FIXME Unknown maintainer"
 
+# Add dependency from buildchroot creation
+do_build[depends] = "buildchroot:do_build"
+
 D = "${WORKDIR}/image/"
 
 # Populate folder that will be picked up as package
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index c8d4ac5..c483fe2 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -3,6 +3,9 @@
 
 inherit dpkg-base
 
+# Add dependency from buildchroot creation
+do_build[depends] = "buildchroot:do_build"
+
 # Build package from sources using build script
 dpkg_runbuild() {
     E="${@ bb.utils.export_proxies(d)}"
-- 
2.1.4

[PATCH 7/9] classes/dpkg*: Drop hardcoded buildchroot blobs

[Alexander Smirnov]

This patch makes possible to parametrize the chroot will be actually
used by dpkg-base class.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 14 +++++++-------
 meta/classes/dpkg-raw.bbclass  |  4 +++-
 meta/classes/dpkg.bbclass      |  4 +++-
 3 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 24fe2ca..0668e25 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -31,7 +31,7 @@ def get_package_srcdir(d):
 PP = "/home/builder/${PN}"
 PPS ?= "${@get_package_srcdir(d)}"
 
-BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
+BUILDROOT = "${ROOTFS_DIR}/${PP}"
 do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 # default to "emtpy" implementation
@@ -39,7 +39,7 @@ dpkg_runbuild() {
     die "This should never be called, overwrite it in your derived class"
 }
 
-MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock"
+MOUNT_LOCKFILE = "${ROOTFS_DIR}/mount.lock"
 
 # Wrap the function dpkg_runbuild with the bind mount for buildroot
 do_build() {
@@ -47,11 +47,11 @@ do_build() {
     sudo mount --bind ${WORKDIR} ${BUILDROOT}
 
     sudo flock ${MOUNT_LOCKFILE} -c ' \
-        if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then \
-            mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt; \
-            mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads; \
-            mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev; \
-            mount -t proc none ${BUILDCHROOT_DIR}/proc; \
+        if ! grep -q ${ROOTFS_DIR}/isar-apt /proc/mounts; then \
+            mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${ROOTFS_DIR}/isar-apt; \
+            mount --bind ${DL_DIR} ${ROOTFS_DIR}/downloads; \
+            mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${ROOTFS_DIR}/dev; \
+            mount -t proc none ${ROOTFS_DIR}/proc; \
         fi'
 
     dpkg_runbuild
diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index 5c7be6b..1b661a7 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -9,6 +9,8 @@ MAINTAINER ?= "FIXME Unknown maintainer"
 # Add dependency from buildchroot creation
 do_build[depends] = "buildchroot:do_build"
 
+ROOTFS_DIR = "${BUILDCHROOT_DIR}"
+
 D = "${WORKDIR}/image/"
 
 # Populate folder that will be picked up as package
@@ -59,5 +61,5 @@ addtask deb_package_conffiles after do_deb_package_prepare before do_build
 
 dpkg_runbuild() {
 	sudo chown -R root:root ${D}/DEBIAN/
-	sudo chroot ${BUILDCHROOT_DIR} dpkg-deb --build ${PP}/image ${PP}
+	sudo chroot ${ROOTFS_DIR} dpkg-deb --build ${PP}/image ${PP}
 }
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index c483fe2..195174a 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -6,8 +6,10 @@ inherit dpkg-base
 # Add dependency from buildchroot creation
 do_build[depends] = "buildchroot:do_build"
 
+ROOTFS_DIR = "${BUILDCHROOT_DIR}"
+
 # Build package from sources using build script
 dpkg_runbuild() {
     E="${@ bb.utils.export_proxies(d)}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${PPS}
+    sudo -E chroot ${ROOTFS_DIR} /build.sh ${PP}/${PPS}
 }
-- 
2.1.4

[PATCH 8/9] classes/dpkg-cross: Initial implementation

[Alexander Smirnov]

Add basic class for cross-compilation.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/dpkg-cross.bbclass | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
 create mode 100644 meta/classes/dpkg-cross.bbclass

diff --git a/meta/classes/dpkg-cross.bbclass b/meta/classes/dpkg-cross.bbclass
new file mode 100644
index 0000000..7e5a1fd
--- /dev/null
+++ b/meta/classes/dpkg-cross.bbclass
@@ -0,0 +1,15 @@
+# This software is a part of ISAR.
+# Copyright (C) 2015-2018 ilbers GmbH
+
+inherit dpkg-base
+
+# Add dependency from buildchroot creation
+do_build[depends] = "buildchroot-cross:do_build"
+
+ROOTFS_DIR="${BUILDCHROOT_CROSS_DIR}"
+
+# Build package from sources using build script
+dpkg_runbuild() {
+    E="${@ bb.utils.export_proxies(d)}"
+    sudo -E chroot ${ROOTFS_DIR} /build.sh ${PP}/${PPS} ${DISTRO_ARCH}
+}
-- 
2.1.4

[PATCH 9/9] [FOR TESTING] libhello: Switch to cross compilation

[Alexander Smirnov]

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar/recipes-app/libhello/libhello.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta-isar/recipes-app/libhello/libhello.bb b/meta-isar/recipes-app/libhello/libhello.bb
index 4e75f98..8727aaf 100644
--- a/meta-isar/recipes-app/libhello/libhello.bb
+++ b/meta-isar/recipes-app/libhello/libhello.bb
@@ -13,4 +13,4 @@ PV = "0.1-98f2e41"
 SRC_URI = "git://github.com/ilbers/libhello.git;protocol=https;destsuffix=${P}"
 SRCREV = "98f2e41e7d05ab8d19b0c5d160b104b725c8fd93"
 
-inherit dpkg
+inherit dpkg-cross
-- 
2.1.4

Re: [RFC] [PATCH 0/9] Initial cross-compilation support

[Jan Kiszka]

On 2018-06-13 15:07, Alexander Smirnov wrote:
> Hi all,
> 
> this series introduces basic cross-compilation support for Isar. The implementation
> is based around new entity - buildchroot-cross, which is quite similar to original
> buildchroot, but has host architecture.
> 
> This series tested for the following configuration only:
> 
>  $ bitbake multiconfig:qemui386
> 
> In this build:
>  - libhello is cross-compiled
>  - example-hello is compiled traditionally (but it's successfully linked with libhello)
> 
> TODO list:
>  - Depending on the target architecture, various host tools should be installed. For example
>    binutils package has the following names:
>    * binutils
>    * binutils-x86-64-linux-gnu
>    * binutils-arm-linux-gnueabihf
>  - Cross-dependency installation. This topic is still open for me because 'mk-build-deps' tool
>    doesn't work here correctly. For example package depends on binutils, but instead of
>    installation of 'binutils-arm-linux-gnueabihf' it tries to install 'binutils:armhf' and ruins
>    the rootfs.
> 
> So, as usually, comments and suggestions are welcome :-)
> 

Cool, that comes just in time! We have more and more ARM/ARM64 packages
that would benefit from the speedup.

Concrete question: What do I have to do in order to run, e.g., the
kernel build in [1] for qemuarm64 in this new cross environment?

Jan

[1] https://github.com/siemens/jailhouse-images

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 6/9] classes/dpkg*: Relocate dependency from buildchroot

[Jan Kiszka]

On 2018-06-13 15:08, Alexander Smirnov wrote:
> Move dependency from buildchroot to the classes that actually require
> buildchroot. This will make possible to re-use dpkg-base class for
> buildchroot-cross.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/classes/dpkg-base.bbclass | 3 ---
>  meta/classes/dpkg-raw.bbclass  | 3 +++
>  meta/classes/dpkg.bbclass      | 3 +++
>  3 files changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 75b45a7..24fe2ca 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -14,9 +14,6 @@ do_adjust_git[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>  inherit patch
>  addtask patch after do_adjust_git before do_build
>  
> -# Add dependency from buildchroot creation
> -do_build[depends] = "buildchroot:do_build"
> -
>  # Add dependency between Isar recipes
>  DEPENDS ?= ""
>  do_build[deptask] = "do_deploy_deb"
> diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
> index 28233ac..5c7be6b 100644
> --- a/meta/classes/dpkg-raw.bbclass
> +++ b/meta/classes/dpkg-raw.bbclass
> @@ -6,6 +6,9 @@ inherit dpkg-base
>  DEBIAN_DEPENDS ?= ""
>  MAINTAINER ?= "FIXME Unknown maintainer"
>  
> +# Add dependency from buildchroot creation
> +do_build[depends] = "buildchroot:do_build"
> +
>  D = "${WORKDIR}/image/"
>  
>  # Populate folder that will be picked up as package
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index c8d4ac5..c483fe2 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -3,6 +3,9 @@
>  
>  inherit dpkg-base
>  
> +# Add dependency from buildchroot creation
> +do_build[depends] = "buildchroot:do_build"
> +
>  # Build package from sources using build script
>  dpkg_runbuild() {
>      E="${@ bb.utils.export_proxies(d)}"
> 

You also need to update meta/recipes-kernel/linux/linux-custom.inc. It
inherits dpkg-base as well and, thus, now also needs this explicit
dependency.

Same is true for patch 7 regarding ROOTFS_DIR variable.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [RFC] [PATCH 0/9] Initial cross-compilation support

[Henning Schild]

Am Wed, 13 Jun 2018 15:07:58 +0200
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Hi all,
> 
> this series introduces basic cross-compilation support for Isar. The
> implementation is based around new entity - buildchroot-cross, which
> is quite similar to original buildchroot, but has host architecture.
> 
> This series tested for the following configuration only:
> 
>  $ bitbake multiconfig:qemui386
> 
> In this build:
>  - libhello is cross-compiled
>  - example-hello is compiled traditionally (but it's successfully
> linked with libhello)
> 
> TODO list:
>  - Depending on the target architecture, various host tools should be
> installed. For example binutils package has the following names:
>    * binutils
>    * binutils-x86-64-linux-gnu
>    * binutils-arm-linux-gnueabihf
>  - Cross-dependency installation. This topic is still open for me
> because 'mk-build-deps' tool doesn't work here correctly. For example
> package depends on binutils, but instead of installation of
> 'binutils-arm-linux-gnueabihf' it tries to install 'binutils:armhf'
> and ruins the rootfs.

mk-build-deps supports the following switches "--arch foo --host-arch
foo", maybe they help?

Henning

> So, as usually, comments and suggestions are welcome :-)
> 
> With best redgards,
> Alex
> 
> Alexander Smirnov (9):
>   isar-bootstrap: Add routin to determine host arch
>   isar-bootstrap: Move common part to include
>   isar-bootstrap: Add host architecture support
>   isar-bootstrap-helper: Add parameter to set arch
>   buildchroot-cross: Initial implementation
>   classes/dpkg*: Relocate dependency from buildchroot
>   classes/dpkg*: Drop hardcoded buildchroot blobs
>   classes/dpkg-cross: Initial implementation
>   [FOR TESTING] libhello: Switch to cross compilation
> 
>  meta-isar/recipes-app/libhello/libhello.bb         |   2 +-
>  meta/classes/dpkg-base.bbclass                     |  17 +-
>  meta/classes/dpkg-cross.bbclass                    |  15 ++
>  meta/classes/dpkg-raw.bbclass                      |   7 +-
>  meta/classes/dpkg.bbclass                          |   7 +-
>  meta/classes/image.bbclass                         |   2 +-
>  meta/classes/isar-bootstrap-helper.bbclass         |  13 +-
>  meta/conf/isar-bitbake.conf                        |   1 +
>  .../isar-bootstrap/isar-bootstrap-host.bb          |  44 ++++
>  .../isar-bootstrap/isar-bootstrap-target.bb        |  44 ++++
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 259
> --------------------- .../recipes-core/isar-bootstrap/isar-bootstrap.inc
> | 223
> ++++++++++++++++++ .../buildchroot/buildchroot-cross.bb
> |  53 +++++ meta/recipes-devtools/buildchroot/buildchroot.bb   |   2
> +- .../buildchroot/files/build-cross.sh               |  29 +++ 15
> files changed, 443 insertions(+), 275 deletions(-) create mode 100644
> meta/classes/dpkg-cross.bbclass create mode 100644
> meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb create mode
> 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
> delete mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap.bb
> create mode 100644
> meta/recipes-core/isar-bootstrap/isar-bootstrap.inc create mode
> 100644 meta/recipes-devtools/buildchroot/buildchroot-cross.bb create
> mode 100644 meta/recipes-devtools/buildchroot/files/build-cross.sh
> 

Re: [PATCH 1/9] isar-bootstrap: Add routin to determine host arch

[Henning Schild]

Am Wed, 13 Jun 2018 15:07:59 +0200
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> This method will be used to distinguish host and target bootstraps.
> The standard python platform module is not robust here, because it
> provides the architecture label from kernel. For example running
> platfotm.machine() on Fedora kernel will return 'x86_64', what is
> invalid specificator for Debian systems.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/classes/isar-bootstrap-helper.bbclass | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> b/meta/classes/isar-bootstrap-helper.bbclass index 3c10fc7..2598ea3
> 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
> +++ b/meta/classes/isar-bootstrap-helper.bbclass
> @@ -14,6 +14,15 @@ def reverse_bb_array(d, varname):
>      array = reversed(array.split())
>      return " ".join(i for i in array)
>  
> +python () {
> +    import subprocess
> +    host_arch = subprocess.Popen("/usr/bin/dpkg
> --print-architecture",
> +                                 shell=True,
> +                                 stdout=subprocess.PIPE
> +                                ).stdout.read().decode('utf-8').strip()
> +    d.setVar("HOST_ARCH", host_arch);
> +}

Consider not using an absolute path. That defeats the magic of PATH.

Henning

> +
>  setup_root_file_system() {
>      CLEAN=""
>      FSTAB=""

Re: [PATCH 3/9] isar-bootstrap: Add host architecture support

[Henning Schild]

This + 2/9 looks very much like what Maxim posted. Again i would like
to understand why we need a copy of the debootstrap code. Maybe the
differences can be addressed with introducing variables.

Henning

Am Wed, 13 Jun 2018 15:08:01 +0200
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Add recipe to generate bootstrap with the host architecture.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  .../isar-bootstrap/isar-bootstrap-host.bb          | 44
> ++++++++++++++++++++++ 1 file changed, 44 insertions(+)
>  create mode 100644
> meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
> 
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb new file
> mode 100644 index 0000000..c61c665
> --- /dev/null
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
> @@ -0,0 +1,44 @@
> +# Minimal host Debian root file system
> +#
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2018
> +#
> +# SPDX-License-Identifier: MIT
> +
> +Description = "Minimal host Debian root file system"
> +
> +include isar-bootstrap.inc
> +inherit isar-bootstrap-helper
> +
> +do_bootstrap[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +do_bootstrap[vardeps] += "DISTRO_APT_SOURCES"
> +do_bootstrap[vardeps] += "DISTRO_APT_PREMIRRORS"
> +do_bootstrap() {
> +    if [ -e "${ROOTFSDIR}" ]; then
> +       sudo umount -l "${ROOTFSDIR}/dev" || true
> +       sudo umount -l "${ROOTFSDIR}/proc" || true
> +       sudo rm -rf "${ROOTFSDIR}"
> +    fi
> +    E="${@bb.utils.export_proxies(d)}"
> +    sudo -E "${DEBOOTSTRAP}" --verbose \
> +                             --variant=minbase \
> +                             --include=locales \
> +                             ${@get_distro_components_argument(d)} \
> +                             ${DEBOOTSTRAP_KEYRING} \
> +                             "${@get_distro_suite(d)}" \
> +                             "${ROOTFSDIR}" \
> +                             "${@get_distro_source(d)}"
> +}
> +addtask bootstrap before do_build after do_generate_keyring
> +
> +do_deploy[stamp-extra-info] = "${DISTRO}-${HOST_ARCH}"
> +do_deploy[dirs] = "${DEPLOY_DIR_IMAGE}"
> +do_deploy() {
> +    ln -Tfsr "${ROOTFSDIR}"
> "${DEPLOY_DIR_IMAGE}/isar-bootstrap-${DISTRO}-${HOST_ARCH}" +}
> +addtask deploy before do_build after do_apt_update
> +
> +CLEANFUNCS = "clean_deploy"
> +clean_deploy() {
> +     rm -f "${DEPLOY_DIR_IMAGE}/${PN}-${DISTRO}-${HOST_ARCH}"
> +}

Re: [PATCH 5/9] buildchroot-cross: Initial implementation

[Henning Schild]

Am Wed, 13 Jun 2018 15:08:03 +0200
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Add initial generation of buildchroot for cross-compilation.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/conf/isar-bitbake.conf                        |  1 +
>  .../buildchroot/buildchroot-cross.bb               | 53
> ++++++++++++++++++++++ .../buildchroot/files/build-cross.sh
> | 29 ++++++++++++ 3 files changed, 83 insertions(+)
>  create mode 100644
> meta/recipes-devtools/buildchroot/buildchroot-cross.bb create mode
> 100644 meta/recipes-devtools/buildchroot/files/build-cross.sh
> 
> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
> index ead7798..1412508 100644
> --- a/meta/conf/isar-bitbake.conf
> +++ b/meta/conf/isar-bitbake.conf
> @@ -22,6 +22,7 @@ WORKDIR =
> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}" DL_DIR =
> "${TOPDIR}/downloads" SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
>  BUILDCHROOT_DIR =
> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
> +BUILDCHROOT_CROSS_DIR =
> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-cross/rootfs"
> CACHE = "${TMPDIR}/cache" OVERRIDES_append =
> ":${DISTRO}:${DISTRO_ARCH}" diff --git
> a/meta/recipes-devtools/buildchroot/buildchroot-cross.bb
> b/meta/recipes-devtools/buildchroot/buildchroot-cross.bb new file
> mode 100644 index 0000000..935a626 --- /dev/null
> +++ b/meta/recipes-devtools/buildchroot/buildchroot-cross.bb
> @@ -0,0 +1,53 @@
> +# Root filesystem for packages cross-building
> +#
> +# This software is a part of ISAR.
> +# Copyright (C) 2015-2016 ilbers GmbH
> +
> +DESCRIPTION = "Isar development cross-filesystem"
> +
> +LICENSE = "gpl-2.0"
> +LIC_FILES_CHKSUM =
> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
> + +FILESPATH_prepend := "${THISDIR}/files:"

Same as seen in Maxims patches.
No need to prepend FILESPATH, set PV with proper filename

I guess here some "hacks" are required because other recipes will need
to find that and can not know PV and stuff.

> +SRC_URI = "file://configscript.sh \
> +           file://build-cross.sh"
> +PV = "1.0"
> +
> +inherit isar-bootstrap-helper
> +
> +BUILDCHROOT_CROSS_PREINSTALL ?= "gcc-multilib \
> +                                 make \
> +                                 build-essential \
> +                                 debhelper \
> +                                 autotools-dev \
> +                                 dpkg \
> +                                 locales \
> +                                 docbook-to-man \
> +                                 apt \
> +                                 automake \
> +                                 devscripts \
> +                                 equivs"
> +
> +# TODO: make this inclusion depending on the target arch
> +BUILDCHROOT_CROSS_PREINSTALL += "binutils"
> +
> +WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"

WORKDIR = ${BUILDCHROOT_CROSS_DIR}/../

maybe?

> +do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +do_build[root_cleandirs] = "${BUILDCHROOT_CROSS_DIR} \
> +                            ${BUILDCHROOT_CROSS_DIR}/isar-apt \
> +                            ${BUILDCHROOT_CROSS_DIR}/downloads \
> +                            ${BUILDCHROOT_CROSS_DIR}/home/builder"
> +do_build[depends] = "isar-apt:do_cache_config
> isar-bootstrap-host:do_deploy" +
> +do_build() {
> +    setup_root_file_system --host-arch "${BUILDCHROOT_CROSS_DIR}"
> ${BUILDCHROOT_CROSS_PREINSTALL} +
> +    # Install package builder script
> +    sudo chmod -R a+rw "${BUILDCHROOT_CROSS_DIR}/home/builder"
> +    sudo install -m 755 ${WORKDIR}/build-cross.sh
> ${BUILDCHROOT_CROSS_DIR}/build.sh +
> +    # Configure root filesystem
> +    sudo install -m 755 ${WORKDIR}/configscript.sh
> ${BUILDCHROOT_CROSS_DIR}
> +    sudo chroot ${BUILDCHROOT_CROSS_DIR} /configscript.sh
> +}
> diff --git a/meta/recipes-devtools/buildchroot/files/build-cross.sh
> b/meta/recipes-devtools/buildchroot/files/build-cross.sh new file
> mode 100644 index 0000000..8c3ddaf
> --- /dev/null
> +++ b/meta/recipes-devtools/buildchroot/files/build-cross.sh
> @@ -0,0 +1,29 @@
> +#!/bin/bash
> +#
> +# This software is a part of ISAR.
> +# Copyright (C) 2015-2017 ilbers GmbH
> +# Copyright (c) 2018 Siemens AG
> +
> +set -e
> +
> +# Go to build directory
> +cd $1
> +
> +# Add target architecture
> +dpkg --add-architecture $2
> +
> +# To avoid Perl locale warnings:
> +export LC_ALL=C
> +export LANG=C
> +export LANGUAGE=C
> +
> +# If autotools files have been created, update their timestamp to
> +# prevent them from being regenerated
> +for i in configure aclocal.m4 Makefile.am Makefile.in; do
> +    if [ -f "${i}" ]; then
> +        touch "${i}"
> +    fi
> +done
> +
> +# Build the package
> +dpkg-buildpackage -a$2

Can that be done with changing/extending build.sh and not introducing a
copy?

Henning

Re: [RFC] [PATCH 0/9] Initial cross-compilation support

[Henning Schild]

Cool, i will look into that for an arm image i am working on for some
time now. That might result on some more "hands-on" comments.

Henning

Am Wed, 13 Jun 2018 15:07:58 +0200
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Hi all,
> 
> this series introduces basic cross-compilation support for Isar. The
> implementation is based around new entity - buildchroot-cross, which
> is quite similar to original buildchroot, but has host architecture.
> 
> This series tested for the following configuration only:
> 
>  $ bitbake multiconfig:qemui386
> 
> In this build:
>  - libhello is cross-compiled
>  - example-hello is compiled traditionally (but it's successfully
> linked with libhello)
> 
> TODO list:
>  - Depending on the target architecture, various host tools should be
> installed. For example binutils package has the following names:
>    * binutils
>    * binutils-x86-64-linux-gnu
>    * binutils-arm-linux-gnueabihf
>  - Cross-dependency installation. This topic is still open for me
> because 'mk-build-deps' tool doesn't work here correctly. For example
> package depends on binutils, but instead of installation of
> 'binutils-arm-linux-gnueabihf' it tries to install 'binutils:armhf'
> and ruins the rootfs.
> 
> So, as usually, comments and suggestions are welcome :-)
> 
> With best redgards,
> Alex
> 
> Alexander Smirnov (9):
>   isar-bootstrap: Add routin to determine host arch
>   isar-bootstrap: Move common part to include
>   isar-bootstrap: Add host architecture support
>   isar-bootstrap-helper: Add parameter to set arch
>   buildchroot-cross: Initial implementation
>   classes/dpkg*: Relocate dependency from buildchroot
>   classes/dpkg*: Drop hardcoded buildchroot blobs
>   classes/dpkg-cross: Initial implementation
>   [FOR TESTING] libhello: Switch to cross compilation
> 
>  meta-isar/recipes-app/libhello/libhello.bb         |   2 +-
>  meta/classes/dpkg-base.bbclass                     |  17 +-
>  meta/classes/dpkg-cross.bbclass                    |  15 ++
>  meta/classes/dpkg-raw.bbclass                      |   7 +-
>  meta/classes/dpkg.bbclass                          |   7 +-
>  meta/classes/image.bbclass                         |   2 +-
>  meta/classes/isar-bootstrap-helper.bbclass         |  13 +-
>  meta/conf/isar-bitbake.conf                        |   1 +
>  .../isar-bootstrap/isar-bootstrap-host.bb          |  44 ++++
>  .../isar-bootstrap/isar-bootstrap-target.bb        |  44 ++++
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 259
> --------------------- .../recipes-core/isar-bootstrap/isar-bootstrap.inc
> | 223
> ++++++++++++++++++ .../buildchroot/buildchroot-cross.bb
> |  53 +++++ meta/recipes-devtools/buildchroot/buildchroot.bb   |   2
> +- .../buildchroot/files/build-cross.sh               |  29 +++ 15
> files changed, 443 insertions(+), 275 deletions(-) create mode 100644
> meta/classes/dpkg-cross.bbclass create mode 100644
> meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb create mode
> 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
> delete mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap.bb
> create mode 100644
> meta/recipes-core/isar-bootstrap/isar-bootstrap.inc create mode
> 100644 meta/recipes-devtools/buildchroot/buildchroot-cross.bb create
> mode 100644 meta/recipes-devtools/buildchroot/files/build-cross.sh
> 

Re: [RFC] [PATCH 0/9] Initial cross-compilation support

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 4277 bytes --]

Hi,

On Wed, 2018-06-13 at 15:07 +0200, Alexander Smirnov wrote:
> Hi all,
> 
> this series introduces basic cross-compilation support for Isar. The
> implementation
> is based around new entity - buildchroot-cross, which is quite
> similar to original
> buildchroot, but has host architecture.

Here is a crazy idea from me and Henning:
Have you tried just installing an amd64-arm cross toolchain into the
existing arm buildchroot, then switch to it from the 'native' toolchain
using a combination of PATH variable entries, symlinks and/or wrapper
scripts and let binfmt deal with calling the amd64 binaries of the
toolchain in the arm chroot environment?

This way the arm system would think that you are compiling nativly
while in fact you are using a cross compiler that is not emulated.

Would that work?

Cheers,
Claudius

> 
> This series tested for the following configuration only:
> 
>  $ bitbake multiconfig:qemui386
> 
> In this build:
>  - libhello is cross-compiled
>  - example-hello is compiled traditionally (but it's successfully
> linked with libhello)
> 
> TODO list:
>  - Depending on the target architecture, various host tools should be
> installed. For example
>    binutils package has the following names:
>    * binutils
>    * binutils-x86-64-linux-gnu
>    * binutils-arm-linux-gnueabihf
>  - Cross-dependency installation. This topic is still open for me
> because 'mk-build-deps' tool
>    doesn't work here correctly. For example package depends on
> binutils, but instead of
>    installation of 'binutils-arm-linux-gnueabihf' it tries to install
> 'binutils:armhf' and ruins
>    the rootfs.
> 
> So, as usually, comments and suggestions are welcome :-)
> 
> With best redgards,
> Alex
> 
> Alexander Smirnov (9):
>   isar-bootstrap: Add routin to determine host arch
>   isar-bootstrap: Move common part to include
>   isar-bootstrap: Add host architecture support
>   isar-bootstrap-helper: Add parameter to set arch
>   buildchroot-cross: Initial implementation
>   classes/dpkg*: Relocate dependency from buildchroot
>   classes/dpkg*: Drop hardcoded buildchroot blobs
>   classes/dpkg-cross: Initial implementation
>   [FOR TESTING] libhello: Switch to cross compilation
> 
>  meta-isar/recipes-app/libhello/libhello.bb         |   2 +-
>  meta/classes/dpkg-base.bbclass                     |  17 +-
>  meta/classes/dpkg-cross.bbclass                    |  15 ++
>  meta/classes/dpkg-raw.bbclass                      |   7 +-
>  meta/classes/dpkg.bbclass                          |   7 +-
>  meta/classes/image.bbclass                         |   2 +-
>  meta/classes/isar-bootstrap-helper.bbclass         |  13 +-
>  meta/conf/isar-bitbake.conf                        |   1 +
>  .../isar-bootstrap/isar-bootstrap-host.bb          |  44 ++++
>  .../isar-bootstrap/isar-bootstrap-target.bb        |  44 ++++
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 259 ---------
> ------------
>  .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 223
> ++++++++++++++++++
>  .../buildchroot/buildchroot-cross.bb               |  53 +++++
>  meta/recipes-devtools/buildchroot/buildchroot.bb   |   2 +-
>  .../buildchroot/files/build-cross.sh               |  29 +++
>  15 files changed, 443 insertions(+), 275 deletions(-)
>  create mode 100644 meta/classes/dpkg-cross.bbclass
>  create mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap-
> host.bb
>  create mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap-
> target.bb
>  delete mode 100644 meta/recipes-core/isar-bootstrap/isar-
> bootstrap.bb
>  create mode 100644 meta/recipes-core/isar-bootstrap/isar-
> bootstrap.inc
>  create mode 100644 meta/recipes-devtools/buildchroot/buildchroot-
> cross.bb
>  create mode 100644 meta/recipes-devtools/buildchroot/files/build-
> cross.sh
> 
> -- 
> 2.1.4
> 
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC] [PATCH 0/9] Initial cross-compilation support

[Henning Schild]

Am Thu, 14 Jun 2018 14:04:47 +0200
schrieb Claudius Heine <ch@denx.de>:

> Hi,
> 
> On Wed, 2018-06-13 at 15:07 +0200, Alexander Smirnov wrote:
> > Hi all,
> > 
> > this series introduces basic cross-compilation support for Isar. The
> > implementation
> > is based around new entity - buildchroot-cross, which is quite
> > similar to original
> > buildchroot, but has host architecture.  
> 
> Here is a crazy idea from me and Henning:
> Have you tried just installing an amd64-arm cross toolchain into the
> existing arm buildchroot, then switch to it from the 'native'
> toolchain using a combination of PATH variable entries, symlinks
> and/or wrapper scripts and let binfmt deal with calling the amd64
> binaries of the toolchain in the arm chroot environment?

Another idea that comes very close to the SDK series. How about the
cross_build_chroot contains just the compiler and whatever the compiler
needs. Then we mount buildchroot/rootfs into cross_build_chroot/sysroot
and smuggle in the "--sysroot /sysroot/" i.e. by putting wrapper
scripts for gcc and friends into /usr/local/bin/.

Similar approach, but we do not risk breaking the buildchroot by
forcing an alien compiler into it. And we use less binfmt magic ...
Both the cross-compile and the SDK are very similar and if cross builds
on top of SDK we should be on the right track.

Henning

> This way the arm system would think that you are compiling nativly
> while in fact you are using a cross compiler that is not emulated.
> 
> Would that work?
> 
> Cheers,
> Claudius
> 
> > 
> > This series tested for the following configuration only:
> > 
> >  $ bitbake multiconfig:qemui386
> > 
> > In this build:
> >  - libhello is cross-compiled
> >  - example-hello is compiled traditionally (but it's successfully
> > linked with libhello)
> > 
> > TODO list:
> >  - Depending on the target architecture, various host tools should
> > be installed. For example
> >    binutils package has the following names:
> >    * binutils
> >    * binutils-x86-64-linux-gnu
> >    * binutils-arm-linux-gnueabihf
> >  - Cross-dependency installation. This topic is still open for me
> > because 'mk-build-deps' tool
> >    doesn't work here correctly. For example package depends on
> > binutils, but instead of
> >    installation of 'binutils-arm-linux-gnueabihf' it tries to
> > install 'binutils:armhf' and ruins
> >    the rootfs.
> > 
> > So, as usually, comments and suggestions are welcome :-)
> > 
> > With best redgards,
> > Alex
> > 
> > Alexander Smirnov (9):
> >   isar-bootstrap: Add routin to determine host arch
> >   isar-bootstrap: Move common part to include
> >   isar-bootstrap: Add host architecture support
> >   isar-bootstrap-helper: Add parameter to set arch
> >   buildchroot-cross: Initial implementation
> >   classes/dpkg*: Relocate dependency from buildchroot
> >   classes/dpkg*: Drop hardcoded buildchroot blobs
> >   classes/dpkg-cross: Initial implementation
> >   [FOR TESTING] libhello: Switch to cross compilation
> > 
> >  meta-isar/recipes-app/libhello/libhello.bb         |   2 +-
> >  meta/classes/dpkg-base.bbclass                     |  17 +-
> >  meta/classes/dpkg-cross.bbclass                    |  15 ++
> >  meta/classes/dpkg-raw.bbclass                      |   7 +-
> >  meta/classes/dpkg.bbclass                          |   7 +-
> >  meta/classes/image.bbclass                         |   2 +-
> >  meta/classes/isar-bootstrap-helper.bbclass         |  13 +-
> >  meta/conf/isar-bitbake.conf                        |   1 +
> >  .../isar-bootstrap/isar-bootstrap-host.bb          |  44 ++++
> >  .../isar-bootstrap/isar-bootstrap-target.bb        |  44 ++++
> >  meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 259 ---------
> > ------------
> >  .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 223
> > ++++++++++++++++++
> >  .../buildchroot/buildchroot-cross.bb               |  53 +++++
> >  meta/recipes-devtools/buildchroot/buildchroot.bb   |   2 +-
> >  .../buildchroot/files/build-cross.sh               |  29 +++
> >  15 files changed, 443 insertions(+), 275 deletions(-)
> >  create mode 100644 meta/classes/dpkg-cross.bbclass
> >  create mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap-
> > host.bb
> >  create mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap-
> > target.bb
> >  delete mode 100644 meta/recipes-core/isar-bootstrap/isar-
> > bootstrap.bb
> >  create mode 100644 meta/recipes-core/isar-bootstrap/isar-
> > bootstrap.inc
> >  create mode 100644 meta/recipes-devtools/buildchroot/buildchroot-
> > cross.bb
> >  create mode 100644 meta/recipes-devtools/buildchroot/files/build-
> > cross.sh
> > 
> > -- 
> > 2.1.4
> >   

Re: [RFC] [PATCH 0/9] Initial cross-compilation support

[Jan Kiszka]

On 2018-06-14 14:04, Claudius Heine wrote:
> Hi,
> 
> On Wed, 2018-06-13 at 15:07 +0200, Alexander Smirnov wrote:
>> Hi all,
>>
>> this series introduces basic cross-compilation support for Isar. The
>> implementation
>> is based around new entity - buildchroot-cross, which is quite
>> similar to original
>> buildchroot, but has host architecture.
> 
> Here is a crazy idea from me and Henning:
> Have you tried just installing an amd64-arm cross toolchain into the
> existing arm buildchroot, then switch to it from the 'native' toolchain
> using a combination of PATH variable entries, symlinks and/or wrapper
> scripts and let binfmt deal with calling the amd64 binaries of the
> toolchain in the arm chroot environment?
> 
> This way the arm system would think that you are compiling nativly
> while in fact you are using a cross compiler that is not emulated.
> 
> Would that work?

I just heard some story from SUSE folks doing something probably similar
in the past, but they had to patch binfmt to inject the native tools
into the chroot.

How would you ensure in your proposal that the native compiler finds its
libraries?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [RFC] [PATCH 0/9] Initial cross-compilation support

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 6158 bytes --]

Hi Henning,

On Thu, 2018-06-14 at 14:24 +0200, Henning Schild wrote:
> Am Thu, 14 Jun 2018 14:04:47 +0200
> schrieb Claudius Heine <ch@denx.de>:
> 
> > Hi,
> > 
> > On Wed, 2018-06-13 at 15:07 +0200, Alexander Smirnov wrote:
> > > Hi all,
> > > 
> > > this series introduces basic cross-compilation support for Isar.
> > > The
> > > implementation
> > > is based around new entity - buildchroot-cross, which is quite
> > > similar to original
> > > buildchroot, but has host architecture.  
> > 
> > Here is a crazy idea from me and Henning:
> > Have you tried just installing an amd64-arm cross toolchain into
> > the
> > existing arm buildchroot, then switch to it from the 'native'
> > toolchain using a combination of PATH variable entries, symlinks
> > and/or wrapper scripts and let binfmt deal with calling the amd64
> > binaries of the toolchain in the arm chroot environment?
> 
> Another idea that comes very close to the SDK series. How about the
> cross_build_chroot contains just the compiler and whatever the
> compiler
> needs. Then we mount buildchroot/rootfs into
> cross_build_chroot/sysroot
> and smuggle in the "--sysroot /sysroot/" i.e. by putting wrapper
> scripts for gcc and friends into /usr/local/bin/.
> 
> Similar approach, but we do not risk breaking the buildchroot by
> forcing an alien compiler into it.

Ok, maybe that was a misunderstanding. I meant installing the cross
compiler to a seperate directory inside the buildchroot and only
switching to it if needed by changing the PATH variable.

I currently don't understand how you solve the installation of the
dpkg-build-dependencies that way.

If you put the cross-compiler inside the buildchroot then you can just
chroot into it to build and install dependencies. The other way around
is not so obvious to me.

>  And we use less binfmt magic ...
> Both the cross-compile and the SDK are very similar and if cross
> builds
> on top of SDK we should be on the right track.

I do like the idea to merge the SDK and cross-build patchset somewhat,
since they do have a similar goal.

Cheers,
Claudius

> 
> Henning
> 
> > This way the arm system would think that you are compiling nativly
> > while in fact you are using a cross compiler that is not emulated.
> > 
> > Would that work?
> > 
> > Cheers,
> > Claudius
> > 
> > > 
> > > This series tested for the following configuration only:
> > > 
> > >  $ bitbake multiconfig:qemui386
> > > 
> > > In this build:
> > >  - libhello is cross-compiled
> > >  - example-hello is compiled traditionally (but it's successfully
> > > linked with libhello)
> > > 
> > > TODO list:
> > >  - Depending on the target architecture, various host tools
> > > should
> > > be installed. For example
> > >    binutils package has the following names:
> > >    * binutils
> > >    * binutils-x86-64-linux-gnu
> > >    * binutils-arm-linux-gnueabihf
> > >  - Cross-dependency installation. This topic is still open for me
> > > because 'mk-build-deps' tool
> > >    doesn't work here correctly. For example package depends on
> > > binutils, but instead of
> > >    installation of 'binutils-arm-linux-gnueabihf' it tries to
> > > install 'binutils:armhf' and ruins
> > >    the rootfs.
> > > 
> > > So, as usually, comments and suggestions are welcome :-)
> > > 
> > > With best redgards,
> > > Alex
> > > 
> > > Alexander Smirnov (9):
> > >   isar-bootstrap: Add routin to determine host arch
> > >   isar-bootstrap: Move common part to include
> > >   isar-bootstrap: Add host architecture support
> > >   isar-bootstrap-helper: Add parameter to set arch
> > >   buildchroot-cross: Initial implementation
> > >   classes/dpkg*: Relocate dependency from buildchroot
> > >   classes/dpkg*: Drop hardcoded buildchroot blobs
> > >   classes/dpkg-cross: Initial implementation
> > >   [FOR TESTING] libhello: Switch to cross compilation
> > > 
> > >  meta-isar/recipes-app/libhello/libhello.bb         |   2 +-
> > >  meta/classes/dpkg-base.bbclass                     |  17 +-
> > >  meta/classes/dpkg-cross.bbclass                    |  15 ++
> > >  meta/classes/dpkg-raw.bbclass                      |   7 +-
> > >  meta/classes/dpkg.bbclass                          |   7 +-
> > >  meta/classes/image.bbclass                         |   2 +-
> > >  meta/classes/isar-bootstrap-helper.bbclass         |  13 +-
> > >  meta/conf/isar-bitbake.conf                        |   1 +
> > >  .../isar-bootstrap/isar-bootstrap-host.bb          |  44 ++++
> > >  .../isar-bootstrap/isar-bootstrap-target.bb        |  44 ++++
> > >  meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 259 -----
> > > ----
> > > ------------
> > >  .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 223
> > > ++++++++++++++++++
> > >  .../buildchroot/buildchroot-cross.bb               |  53 +++++
> > >  meta/recipes-devtools/buildchroot/buildchroot.bb   |   2 +-
> > >  .../buildchroot/files/build-cross.sh               |  29 +++
> > >  15 files changed, 443 insertions(+), 275 deletions(-)
> > >  create mode 100644 meta/classes/dpkg-cross.bbclass
> > >  create mode 100644 meta/recipes-core/isar-bootstrap/isar-
> > > bootstrap-
> > > host.bb
> > >  create mode 100644 meta/recipes-core/isar-bootstrap/isar-
> > > bootstrap-
> > > target.bb
> > >  delete mode 100644 meta/recipes-core/isar-bootstrap/isar-
> > > bootstrap.bb
> > >  create mode 100644 meta/recipes-core/isar-bootstrap/isar-
> > > bootstrap.inc
> > >  create mode 100644 meta/recipes-
> > > devtools/buildchroot/buildchroot-
> > > cross.bb
> > >  create mode 100644 meta/recipes-
> > > devtools/buildchroot/files/build-
> > > cross.sh
> > > 
> > > -- 
> > > 2.1.4
> > >   
> 
> 
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC] [PATCH 0/9] Initial cross-compilation support

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 1912 bytes --]

Hi Jan,

On Thu, 2018-06-14 at 14:30 +0200, Jan Kiszka wrote:
> On 2018-06-14 14:04, Claudius Heine wrote:
> > Hi,
> > 
> > On Wed, 2018-06-13 at 15:07 +0200, Alexander Smirnov wrote:
> > > Hi all,
> > > 
> > > this series introduces basic cross-compilation support for Isar.
> > > The
> > > implementation
> > > is based around new entity - buildchroot-cross, which is quite
> > > similar to original
> > > buildchroot, but has host architecture.
> > 
> > Here is a crazy idea from me and Henning:
> > Have you tried just installing an amd64-arm cross toolchain into
> > the
> > existing arm buildchroot, then switch to it from the 'native'
> > toolchain
> > using a combination of PATH variable entries, symlinks and/or
> > wrapper
> > scripts and let binfmt deal with calling the amd64 binaries of the
> > toolchain in the arm chroot environment?
> > 
> > This way the arm system would think that you are compiling nativly
> > while in fact you are using a cross compiler that is not emulated.
> > 
> > Would that work?
> 
> I just heard some story from SUSE folks doing something probably
> similar
> in the past, but they had to patch binfmt to inject the native tools
> into the chroot.
> 
> How would you ensure in your proposal that the native compiler finds
> its
> libraries?

I would try using the wrapper scripts. Setting all the paths to the
toolchain root directory and then calling gcc and friends with '
--sysroot /'.

I would have to try to find out if that works.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC] [PATCH 0/9] Initial cross-compilation support

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 2701 bytes --]

On Thu, 2018-06-14 at 14:42 +0200, Claudius Heine wrote:
> Hi Jan,
> 
> On Thu, 2018-06-14 at 14:30 +0200, Jan Kiszka wrote:
> > On 2018-06-14 14:04, Claudius Heine wrote:
> > > Hi,
> > > 
> > > On Wed, 2018-06-13 at 15:07 +0200, Alexander Smirnov wrote:
> > > > Hi all,
> > > > 
> > > > this series introduces basic cross-compilation support for
> > > > Isar.
> > > > The
> > > > implementation
> > > > is based around new entity - buildchroot-cross, which is quite
> > > > similar to original
> > > > buildchroot, but has host architecture.
> > > 
> > > Here is a crazy idea from me and Henning:
> > > Have you tried just installing an amd64-arm cross toolchain into
> > > the
> > > existing arm buildchroot, then switch to it from the 'native'
> > > toolchain
> > > using a combination of PATH variable entries, symlinks and/or
> > > wrapper
> > > scripts and let binfmt deal with calling the amd64 binaries of
> > > the
> > > toolchain in the arm chroot environment?
> > > 
> > > This way the arm system would think that you are compiling
> > > nativly
> > > while in fact you are using a cross compiler that is not
> > > emulated.
> > > 
> > > Would that work?
> > 
> > I just heard some story from SUSE folks doing something probably
> > similar
> > in the past, but they had to patch binfmt to inject the native
> > tools
> > into the chroot.
> > 
> > How would you ensure in your proposal that the native compiler
> > finds
> > its
> > libraries?
> 
> I would try using the wrapper scripts. Setting all the paths to the
> toolchain root directory and then calling gcc and friends with '
> --sysroot /'.
> 
> I would have to try to find out if that works.

There is also chrpath...

As a more conventional method, you could have a cross-buildchroot with
only the cross compiler and distcc and make it that way available to
the buildchroot.

Claudius

> 
> Cheers,
> Claudius
> 
> -- 
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
> 
>             PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808
> B153
>                               Keyserver: hkp://pool.sks-
> keyservers.net
> 
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 5/9] buildchroot-cross: Initial implementation

[Jan Kiszka]

On 2018-06-13 15:08, Alexander Smirnov wrote:
> Add initial generation of buildchroot for cross-compilation.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/conf/isar-bitbake.conf                        |  1 +
>  .../buildchroot/buildchroot-cross.bb               | 53 ++++++++++++++++++++++
>  .../buildchroot/files/build-cross.sh               | 29 ++++++++++++
>  3 files changed, 83 insertions(+)
>  create mode 100644 meta/recipes-devtools/buildchroot/buildchroot-cross.bb
>  create mode 100644 meta/recipes-devtools/buildchroot/files/build-cross.sh
> 
> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
> index ead7798..1412508 100644
> --- a/meta/conf/isar-bitbake.conf
> +++ b/meta/conf/isar-bitbake.conf
> @@ -22,6 +22,7 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}"
>  DL_DIR = "${TOPDIR}/downloads"
>  SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
>  BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
> +BUILDCHROOT_CROSS_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-cross/rootfs"
>  CACHE = "${TMPDIR}/cache"
>  
>  OVERRIDES_append = ":${DISTRO}:${DISTRO_ARCH}"
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot-cross.bb b/meta/recipes-devtools/buildchroot/buildchroot-cross.bb
> new file mode 100644
> index 0000000..935a626
> --- /dev/null
> +++ b/meta/recipes-devtools/buildchroot/buildchroot-cross.bb
> @@ -0,0 +1,53 @@
> +# Root filesystem for packages cross-building
> +#
> +# This software is a part of ISAR.
> +# Copyright (C) 2015-2016 ilbers GmbH
> +
> +DESCRIPTION = "Isar development cross-filesystem"
> +
> +LICENSE = "gpl-2.0"
> +LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
> +
> +FILESPATH_prepend := "${THISDIR}/files:"
> +SRC_URI = "file://configscript.sh \
> +           file://build-cross.sh"
> +PV = "1.0"
> +
> +inherit isar-bootstrap-helper
> +
> +BUILDCHROOT_CROSS_PREINSTALL ?= "gcc-multilib \
> +                                 make \
> +                                 build-essential \
> +                                 debhelper \
> +                                 autotools-dev \
> +                                 dpkg \
> +                                 locales \
> +                                 docbook-to-man \
> +                                 apt \
> +                                 automake \
> +                                 devscripts \
> +                                 equivs"
> +
> +# TODO: make this inclusion depending on the target arch
> +BUILDCHROOT_CROSS_PREINSTALL += "binutils"
> +
> +WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
> +
> +do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +do_build[root_cleandirs] = "${BUILDCHROOT_CROSS_DIR} \
> +                            ${BUILDCHROOT_CROSS_DIR}/isar-apt \
> +                            ${BUILDCHROOT_CROSS_DIR}/downloads \
> +                            ${BUILDCHROOT_CROSS_DIR}/home/builder"
> +do_build[depends] = "isar-apt:do_cache_config isar-bootstrap-host:do_deploy"
> +
> +do_build() {
> +    setup_root_file_system --host-arch "${BUILDCHROOT_CROSS_DIR}" ${BUILDCHROOT_CROSS_PREINSTALL}
> +
> +    # Install package builder script
> +    sudo chmod -R a+rw "${BUILDCHROOT_CROSS_DIR}/home/builder"
> +    sudo install -m 755 ${WORKDIR}/build-cross.sh ${BUILDCHROOT_CROSS_DIR}/build.sh
> +
> +    # Configure root filesystem
> +    sudo install -m 755 ${WORKDIR}/configscript.sh ${BUILDCHROOT_CROSS_DIR}
> +    sudo chroot ${BUILDCHROOT_CROSS_DIR} /configscript.sh
> +}
> diff --git a/meta/recipes-devtools/buildchroot/files/build-cross.sh b/meta/recipes-devtools/buildchroot/files/build-cross.sh
> new file mode 100644
> index 0000000..8c3ddaf
> --- /dev/null
> +++ b/meta/recipes-devtools/buildchroot/files/build-cross.sh
> @@ -0,0 +1,29 @@
> +#!/bin/bash
> +#
> +# This software is a part of ISAR.
> +# Copyright (C) 2015-2017 ilbers GmbH
> +# Copyright (c) 2018 Siemens AG
> +
> +set -e
> +
> +# Go to build directory
> +cd $1
> +
> +# Add target architecture
> +dpkg --add-architecture $2
> +
> +# To avoid Perl locale warnings:
> +export LC_ALL=C
> +export LANG=C
> +export LANGUAGE=C
> +
> +# If autotools files have been created, update their timestamp to
> +# prevent them from being regenerated
> +for i in configure aclocal.m4 Makefile.am Makefile.in; do
> +    if [ -f "${i}" ]; then
> +        touch "${i}"
> +    fi
> +done
> +
> +# Build the package
> +dpkg-buildpackage -a$2
> 

This build-cross.sh doesn't install any of the build dependencies the
target package lists. How do they come in? Or is this a limitation of
this version? For the kernel cross-build, I'm currently trying to
translate the what is needed for a corresponding build-kernel-cross.sh.

Those packages would go into the cross-buildchroot, or are we also using
the native buildchroot here?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [RFC] [PATCH 0/9] Initial cross-compilation support

[Alexander Smirnov]

On Thu, Jun 14, 2018 at 02:04:47PM +0200, Claudius Heine wrote:

Hi everybody,

> Hi,
> 
> On Wed, 2018-06-13 at 15:07 +0200, Alexander Smirnov wrote:
> > Hi all,
> > 
> > this series introduces basic cross-compilation support for Isar. The
> > implementation
> > is based around new entity - buildchroot-cross, which is quite
> > similar to original
> > buildchroot, but has host architecture.
> 
> Here is a crazy idea from me and Henning:
> Have you tried just installing an amd64-arm cross toolchain into the
> existing arm buildchroot, then switch to it from the 'native' toolchain
> using a combination of PATH variable entries, symlinks and/or wrapper
> scripts and let binfmt deal with calling the amd64 binaries of the
> toolchain in the arm chroot environment?
> 
> This way the arm system would think that you are compiling nativly
> while in fact you are using a cross compiler that is not emulated.
> 
> Would that work?

I'm not sure that cross-compiling is onl about compiler stuff, you also
need:
 - make
 - autotools
 - bash for scripts
 - perl (f�r apt/deb K�chen)
 - possibly something else

The list of tools above with its deps itself is already small rootfs.
Managing mix of these tools for different architecture could be too
costly.

Alex

> 
> Cheers,
> Claudius
> 
> > 
> > This series tested for the following configuration only:
> > 
> >  $ bitbake multiconfig:qemui386
> > 
> > In this build:
> >  - libhello is cross-compiled
> >  - example-hello is compiled traditionally (but it's successfully
> > linked with libhello)
> > 
> > TODO list:
> >  - Depending on the target architecture, various host tools should be
> > installed. For example
> >    binutils package has the following names:
> >    * binutils
> >    * binutils-x86-64-linux-gnu
> >    * binutils-arm-linux-gnueabihf
> >  - Cross-dependency installation. This topic is still open for me
> > because 'mk-build-deps' tool
> >    doesn't work here correctly. For example package depends on
> > binutils, but instead of
> >    installation of 'binutils-arm-linux-gnueabihf' it tries to install
> > 'binutils:armhf' and ruins
> >    the rootfs.
> > 
> > So, as usually, comments and suggestions are welcome :-)
> > 
> > With best redgards,
> > Alex
> > 
> > Alexander Smirnov (9):
> >   isar-bootstrap: Add routin to determine host arch
> >   isar-bootstrap: Move common part to include
> >   isar-bootstrap: Add host architecture support
> >   isar-bootstrap-helper: Add parameter to set arch
> >   buildchroot-cross: Initial implementation
> >   classes/dpkg*: Relocate dependency from buildchroot
> >   classes/dpkg*: Drop hardcoded buildchroot blobs
> >   classes/dpkg-cross: Initial implementation
> >   [FOR TESTING] libhello: Switch to cross compilation
> > 
> >  meta-isar/recipes-app/libhello/libhello.bb         |   2 +-
> >  meta/classes/dpkg-base.bbclass                     |  17 +-
> >  meta/classes/dpkg-cross.bbclass                    |  15 ++
> >  meta/classes/dpkg-raw.bbclass                      |   7 +-
> >  meta/classes/dpkg.bbclass                          |   7 +-
> >  meta/classes/image.bbclass                         |   2 +-
> >  meta/classes/isar-bootstrap-helper.bbclass         |  13 +-
> >  meta/conf/isar-bitbake.conf                        |   1 +
> >  .../isar-bootstrap/isar-bootstrap-host.bb          |  44 ++++
> >  .../isar-bootstrap/isar-bootstrap-target.bb        |  44 ++++
> >  meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 259 ---------
> > ------------
> >  .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 223
> > ++++++++++++++++++
> >  .../buildchroot/buildchroot-cross.bb               |  53 +++++
> >  meta/recipes-devtools/buildchroot/buildchroot.bb   |   2 +-
> >  .../buildchroot/files/build-cross.sh               |  29 +++
> >  15 files changed, 443 insertions(+), 275 deletions(-)
> >  create mode 100644 meta/classes/dpkg-cross.bbclass
> >  create mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap-
> > host.bb
> >  create mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap-
> > target.bb
> >  delete mode 100644 meta/recipes-core/isar-bootstrap/isar-
> > bootstrap.bb
> >  create mode 100644 meta/recipes-core/isar-bootstrap/isar-
> > bootstrap.inc
> >  create mode 100644 meta/recipes-devtools/buildchroot/buildchroot-
> > cross.bb
> >  create mode 100644 meta/recipes-devtools/buildchroot/files/build-
> > cross.sh
> > 
> > -- 
> > 2.1.4
> > 
> -- 
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
> 
>             PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
>                               Keyserver: hkp://pool.sks-keyservers.net

Re: [PATCH 6/9] classes/dpkg*: Relocate dependency from buildchroot

[Alexander Smirnov]

Hi Jan,

> On 2018-06-13 15:08, Alexander Smirnov wrote:
> > Move dependency from buildchroot to the classes that actually require
> > buildchroot. This will make possible to re-use dpkg-base class for
> > buildchroot-cross.
> > 
> > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> > ---
> >  meta/classes/dpkg-base.bbclass | 3 ---
> >  meta/classes/dpkg-raw.bbclass  | 3 +++
> >  meta/classes/dpkg.bbclass      | 3 +++
> >  3 files changed, 6 insertions(+), 3 deletions(-)
> > 
> > diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> > index 75b45a7..24fe2ca 100644
> > --- a/meta/classes/dpkg-base.bbclass
> > +++ b/meta/classes/dpkg-base.bbclass
> > @@ -14,9 +14,6 @@ do_adjust_git[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> >  inherit patch
> >  addtask patch after do_adjust_git before do_build
> >  
> > -# Add dependency from buildchroot creation
> > -do_build[depends] = "buildchroot:do_build"
> > -
> >  # Add dependency between Isar recipes
> >  DEPENDS ?= ""
> >  do_build[deptask] = "do_deploy_deb"
> > diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
> > index 28233ac..5c7be6b 100644
> > --- a/meta/classes/dpkg-raw.bbclass
> > +++ b/meta/classes/dpkg-raw.bbclass
> > @@ -6,6 +6,9 @@ inherit dpkg-base
> >  DEBIAN_DEPENDS ?= ""
> >  MAINTAINER ?= "FIXME Unknown maintainer"
> >  
> > +# Add dependency from buildchroot creation
> > +do_build[depends] = "buildchroot:do_build"
> > +
> >  D = "${WORKDIR}/image/"
> >  
> >  # Populate folder that will be picked up as package
> > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> > index c8d4ac5..c483fe2 100644
> > --- a/meta/classes/dpkg.bbclass
> > +++ b/meta/classes/dpkg.bbclass
> > @@ -3,6 +3,9 @@
> >  
> >  inherit dpkg-base
> >  
> > +# Add dependency from buildchroot creation
> > +do_build[depends] = "buildchroot:do_build"
> > +
> >  # Build package from sources using build script
> >  dpkg_runbuild() {
> >      E="${@ bb.utils.export_proxies(d)}"
> > 
> 
> You also need to update meta/recipes-kernel/linux/linux-custom.inc. It
> inherits dpkg-base as well and, thus, now also needs this explicit
> dependency.
> 
> Same is true for patch 7 regarding ROOTFS_DIR variable.
> 

Aah, totally missed this. Thank you for the hint!

Alex

> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

Re: [PATCH 5/9] buildchroot-cross: Initial implementation

[Alexander Smirnov]

On Thu, Jun 14, 2018 at 05:55:48PM +0200, Jan Kiszka wrote:
> On 2018-06-13 15:08, Alexander Smirnov wrote:
> > Add initial generation of buildchroot for cross-compilation.
> > 
> > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> > ---
> >  meta/conf/isar-bitbake.conf                        |  1 +
> >  .../buildchroot/buildchroot-cross.bb               | 53 ++++++++++++++++++++++
> >  .../buildchroot/files/build-cross.sh               | 29 ++++++++++++
> >  3 files changed, 83 insertions(+)
> >  create mode 100644 meta/recipes-devtools/buildchroot/buildchroot-cross.bb
> >  create mode 100644 meta/recipes-devtools/buildchroot/files/build-cross.sh
> > 
> > diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
> > index ead7798..1412508 100644
> > --- a/meta/conf/isar-bitbake.conf
> > +++ b/meta/conf/isar-bitbake.conf
> > @@ -22,6 +22,7 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}"
> >  DL_DIR = "${TOPDIR}/downloads"
> >  SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
> >  BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
> > +BUILDCHROOT_CROSS_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-cross/rootfs"
> >  CACHE = "${TMPDIR}/cache"
> >  
> >  OVERRIDES_append = ":${DISTRO}:${DISTRO_ARCH}"
> > diff --git a/meta/recipes-devtools/buildchroot/buildchroot-cross.bb b/meta/recipes-devtools/buildchroot/buildchroot-cross.bb
> > new file mode 100644
> > index 0000000..935a626
> > --- /dev/null
> > +++ b/meta/recipes-devtools/buildchroot/buildchroot-cross.bb
> > @@ -0,0 +1,53 @@
> > +# Root filesystem for packages cross-building
> > +#
> > +# This software is a part of ISAR.
> > +# Copyright (C) 2015-2016 ilbers GmbH
> > +
> > +DESCRIPTION = "Isar development cross-filesystem"
> > +
> > +LICENSE = "gpl-2.0"
> > +LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
> > +
> > +FILESPATH_prepend := "${THISDIR}/files:"
> > +SRC_URI = "file://configscript.sh \
> > +           file://build-cross.sh"
> > +PV = "1.0"
> > +
> > +inherit isar-bootstrap-helper
> > +
> > +BUILDCHROOT_CROSS_PREINSTALL ?= "gcc-multilib \
> > +                                 make \
> > +                                 build-essential \
> > +                                 debhelper \
> > +                                 autotools-dev \
> > +                                 dpkg \
> > +                                 locales \
> > +                                 docbook-to-man \
> > +                                 apt \
> > +                                 automake \
> > +                                 devscripts \
> > +                                 equivs"
> > +
> > +# TODO: make this inclusion depending on the target arch
> > +BUILDCHROOT_CROSS_PREINSTALL += "binutils"
> > +
> > +WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
> > +
> > +do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> > +do_build[root_cleandirs] = "${BUILDCHROOT_CROSS_DIR} \
> > +                            ${BUILDCHROOT_CROSS_DIR}/isar-apt \
> > +                            ${BUILDCHROOT_CROSS_DIR}/downloads \
> > +                            ${BUILDCHROOT_CROSS_DIR}/home/builder"
> > +do_build[depends] = "isar-apt:do_cache_config isar-bootstrap-host:do_deploy"
> > +
> > +do_build() {
> > +    setup_root_file_system --host-arch "${BUILDCHROOT_CROSS_DIR}" ${BUILDCHROOT_CROSS_PREINSTALL}
> > +
> > +    # Install package builder script
> > +    sudo chmod -R a+rw "${BUILDCHROOT_CROSS_DIR}/home/builder"
> > +    sudo install -m 755 ${WORKDIR}/build-cross.sh ${BUILDCHROOT_CROSS_DIR}/build.sh
> > +
> > +    # Configure root filesystem
> > +    sudo install -m 755 ${WORKDIR}/configscript.sh ${BUILDCHROOT_CROSS_DIR}
> > +    sudo chroot ${BUILDCHROOT_CROSS_DIR} /configscript.sh
> > +}
> > diff --git a/meta/recipes-devtools/buildchroot/files/build-cross.sh b/meta/recipes-devtools/buildchroot/files/build-cross.sh
> > new file mode 100644
> > index 0000000..8c3ddaf
> > --- /dev/null
> > +++ b/meta/recipes-devtools/buildchroot/files/build-cross.sh
> > @@ -0,0 +1,29 @@
> > +#!/bin/bash
> > +#
> > +# This software is a part of ISAR.
> > +# Copyright (C) 2015-2017 ilbers GmbH
> > +# Copyright (c) 2018 Siemens AG
> > +
> > +set -e
> > +
> > +# Go to build directory
> > +cd $1
> > +
> > +# Add target architecture
> > +dpkg --add-architecture $2
> > +
> > +# To avoid Perl locale warnings:
> > +export LC_ALL=C
> > +export LANG=C
> > +export LANGUAGE=C
> > +
> > +# If autotools files have been created, update their timestamp to
> > +# prevent them from being regenerated
> > +for i in configure aclocal.m4 Makefile.am Makefile.in; do
> > +    if [ -f "${i}" ]; then
> > +        touch "${i}"
> > +    fi
> > +done
> > +
> > +# Build the package
> > +dpkg-buildpackage -a$2
> > 
> 
> This build-cross.sh doesn't install any of the build dependencies the
> target package lists. How do they come in? Or is this a limitation of
> this version? For the kernel cross-build, I'm currently trying to
> translate the what is needed for a corresponding build-kernel-cross.sh.
> 

That's exactly the limitation of this series about which I mentioned in
the cover letter. Attempt to install them in similar way like for
'build.sh' failed, due to it tries to replace native packages like:
build-essentials, binutils by foreign ones :-(. Henning already proposed
to try to play with mk-build-deps params, so I'm going to do this.

> Those packages would go into the cross-buildchroot, or are we also using
> the native buildchroot here?

buildchroot and buildchroot-cross has nothing common except isar-apt
repository.

Alex

> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

Re: [PATCH 3/9] isar-bootstrap: Add host architecture support

[Alexander Smirnov]

On Thu, Jun 14, 2018 at 01:27:37PM +0200, Henning Schild wrote:
Hi Henning,

> This + 2/9 looks very much like what Maxim posted. Again i would like
> to understand why we need a copy of the debootstrap code. Maybe the
> differences can be addressed with introducing variables.
> 

For sure, that's the first idea came to my mind. But after an attempts
to implement this I've faced with the following issues:

 - buildchroot/buildchroot-cross are optional. They should be added to
   pipeline if some recipe requires them, i.e. ihnerits dpkg-* class.
 - Each buildchroot should have its own set of stamps to avoid
   duplications. For example each multiconfig has its own instance of
   target rootfs, but host rootfs should be singletone.

Current Isar implementation contains very comfortable way to chose the way how to build the package - inherit respective class.
Re-use of this approach to implement cross-building looks for me very comfortable way:
 - We should not invent the wheel and introduce new user-interface how
   to implement recipes. For example dpkg.bbclass depends on
buildchroot:do_build, so I simply implement dpkg-cross.bbclass which
depends on buildchroot-cross:do_build.
 - The Isar will stay flexible and builds the only really needed parts

Regarding the duplication - I've tried to reduce it as much as possible.
The shareble part is moved to .inc file, so the recipe payload contains
the only arch-specific stuff.

Alex

> Henning
> 
> Am Wed, 13 Jun 2018 15:08:01 +0200
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
> > Add recipe to generate bootstrap with the host architecture.
> > 
> > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> > ---
> >  .../isar-bootstrap/isar-bootstrap-host.bb          | 44
> > ++++++++++++++++++++++ 1 file changed, 44 insertions(+)
> >  create mode 100644
> > meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
> > 
> > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb new file
> > mode 100644 index 0000000..c61c665
> > --- /dev/null
> > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
> > @@ -0,0 +1,44 @@
> > +# Minimal host Debian root file system
> > +#
> > +# This software is a part of ISAR.
> > +# Copyright (c) Siemens AG, 2018
> > +#
> > +# SPDX-License-Identifier: MIT
> > +
> > +Description = "Minimal host Debian root file system"
> > +
> > +include isar-bootstrap.inc
> > +inherit isar-bootstrap-helper
> > +
> > +do_bootstrap[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> > +do_bootstrap[vardeps] += "DISTRO_APT_SOURCES"
> > +do_bootstrap[vardeps] += "DISTRO_APT_PREMIRRORS"
> > +do_bootstrap() {
> > +    if [ -e "${ROOTFSDIR}" ]; then
> > +       sudo umount -l "${ROOTFSDIR}/dev" || true
> > +       sudo umount -l "${ROOTFSDIR}/proc" || true
> > +       sudo rm -rf "${ROOTFSDIR}"
> > +    fi
> > +    E="${@bb.utils.export_proxies(d)}"
> > +    sudo -E "${DEBOOTSTRAP}" --verbose \
> > +                             --variant=minbase \
> > +                             --include=locales \
> > +                             ${@get_distro_components_argument(d)} \
> > +                             ${DEBOOTSTRAP_KEYRING} \
> > +                             "${@get_distro_suite(d)}" \
> > +                             "${ROOTFSDIR}" \
> > +                             "${@get_distro_source(d)}"
> > +}
> > +addtask bootstrap before do_build after do_generate_keyring
> > +
> > +do_deploy[stamp-extra-info] = "${DISTRO}-${HOST_ARCH}"
> > +do_deploy[dirs] = "${DEPLOY_DIR_IMAGE}"
> > +do_deploy() {
> > +    ln -Tfsr "${ROOTFSDIR}"
> > "${DEPLOY_DIR_IMAGE}/isar-bootstrap-${DISTRO}-${HOST_ARCH}" +}
> > +addtask deploy before do_build after do_apt_update
> > +
> > +CLEANFUNCS = "clean_deploy"
> > +clean_deploy() {
> > +     rm -f "${DEPLOY_DIR_IMAGE}/${PN}-${DISTRO}-${HOST_ARCH}"
> > +}
> 

Re: [PATCH 1/9] isar-bootstrap: Add routin to determine host arch

[Alexander Smirnov]

Hi Henning,

> Am Wed, 13 Jun 2018 15:07:59 +0200
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
> > This method will be used to distinguish host and target bootstraps.
> > The standard python platform module is not robust here, because it
> > provides the architecture label from kernel. For example running
> > platfotm.machine() on Fedora kernel will return 'x86_64', what is
> > invalid specificator for Debian systems.
> > 
> > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> > ---
> >  meta/classes/isar-bootstrap-helper.bbclass | 9 +++++++++
> >  1 file changed, 9 insertions(+)
> > 
> > diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> > b/meta/classes/isar-bootstrap-helper.bbclass index 3c10fc7..2598ea3
> > 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
> > +++ b/meta/classes/isar-bootstrap-helper.bbclass
> > @@ -14,6 +14,15 @@ def reverse_bb_array(d, varname):
> >      array = reversed(array.split())
> >      return " ".join(i for i in array)
> >  
> > +python () {
> > +    import subprocess
> > +    host_arch = subprocess.Popen("/usr/bin/dpkg
> > --print-architecture",
> > +                                 shell=True,
> > +                                 stdout=subprocess.PIPE
> > +                                ).stdout.read().decode('utf-8').strip()
> > +    d.setVar("HOST_ARCH", host_arch);
> > +}
> 
> Consider not using an absolute path. That defeats the magic of PATH.

I agree with you here, but btw, do you (or probably somebody else) know how to workaround this in
python subprocess? An attempt to run just 'dpkg' fails with the error:
command not found.

Alex

> Henning
> 
> > +
> >  setup_root_file_system() {
> >      CLEAN=""
> >      FSTAB=""
> 

Re: [PATCH 5/9] buildchroot-cross: Initial implementation

[Jan Kiszka]

On 2018-06-15 12:56, Alexander Smirnov wrote:
> On Thu, Jun 14, 2018 at 05:55:48PM +0200, Jan Kiszka wrote:
>> On 2018-06-13 15:08, Alexander Smirnov wrote:
>>> Add initial generation of buildchroot for cross-compilation.
>>>
>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>> ---
>>>  meta/conf/isar-bitbake.conf                        |  1 +
>>>  .../buildchroot/buildchroot-cross.bb               | 53 ++++++++++++++++++++++
>>>  .../buildchroot/files/build-cross.sh               | 29 ++++++++++++
>>>  3 files changed, 83 insertions(+)
>>>  create mode 100644 meta/recipes-devtools/buildchroot/buildchroot-cross.bb
>>>  create mode 100644 meta/recipes-devtools/buildchroot/files/build-cross.sh
>>>
>>> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
>>> index ead7798..1412508 100644
>>> --- a/meta/conf/isar-bitbake.conf
>>> +++ b/meta/conf/isar-bitbake.conf
>>> @@ -22,6 +22,7 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}"
>>>  DL_DIR = "${TOPDIR}/downloads"
>>>  SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
>>>  BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
>>> +BUILDCHROOT_CROSS_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-cross/rootfs"
>>>  CACHE = "${TMPDIR}/cache"
>>>  
>>>  OVERRIDES_append = ":${DISTRO}:${DISTRO_ARCH}"
>>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot-cross.bb b/meta/recipes-devtools/buildchroot/buildchroot-cross.bb
>>> new file mode 100644
>>> index 0000000..935a626
>>> --- /dev/null
>>> +++ b/meta/recipes-devtools/buildchroot/buildchroot-cross.bb
>>> @@ -0,0 +1,53 @@
>>> +# Root filesystem for packages cross-building
>>> +#
>>> +# This software is a part of ISAR.
>>> +# Copyright (C) 2015-2016 ilbers GmbH
>>> +
>>> +DESCRIPTION = "Isar development cross-filesystem"
>>> +
>>> +LICENSE = "gpl-2.0"
>>> +LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
>>> +
>>> +FILESPATH_prepend := "${THISDIR}/files:"
>>> +SRC_URI = "file://configscript.sh \
>>> +           file://build-cross.sh"
>>> +PV = "1.0"
>>> +
>>> +inherit isar-bootstrap-helper
>>> +
>>> +BUILDCHROOT_CROSS_PREINSTALL ?= "gcc-multilib \
>>> +                                 make \
>>> +                                 build-essential \
>>> +                                 debhelper \
>>> +                                 autotools-dev \
>>> +                                 dpkg \
>>> +                                 locales \
>>> +                                 docbook-to-man \
>>> +                                 apt \
>>> +                                 automake \
>>> +                                 devscripts \
>>> +                                 equivs"
>>> +
>>> +# TODO: make this inclusion depending on the target arch
>>> +BUILDCHROOT_CROSS_PREINSTALL += "binutils"
>>> +
>>> +WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>>> +
>>> +do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>> +do_build[root_cleandirs] = "${BUILDCHROOT_CROSS_DIR} \
>>> +                            ${BUILDCHROOT_CROSS_DIR}/isar-apt \
>>> +                            ${BUILDCHROOT_CROSS_DIR}/downloads \
>>> +                            ${BUILDCHROOT_CROSS_DIR}/home/builder"
>>> +do_build[depends] = "isar-apt:do_cache_config isar-bootstrap-host:do_deploy"
>>> +
>>> +do_build() {
>>> +    setup_root_file_system --host-arch "${BUILDCHROOT_CROSS_DIR}" ${BUILDCHROOT_CROSS_PREINSTALL}
>>> +
>>> +    # Install package builder script
>>> +    sudo chmod -R a+rw "${BUILDCHROOT_CROSS_DIR}/home/builder"
>>> +    sudo install -m 755 ${WORKDIR}/build-cross.sh ${BUILDCHROOT_CROSS_DIR}/build.sh
>>> +
>>> +    # Configure root filesystem
>>> +    sudo install -m 755 ${WORKDIR}/configscript.sh ${BUILDCHROOT_CROSS_DIR}
>>> +    sudo chroot ${BUILDCHROOT_CROSS_DIR} /configscript.sh
>>> +}
>>> diff --git a/meta/recipes-devtools/buildchroot/files/build-cross.sh b/meta/recipes-devtools/buildchroot/files/build-cross.sh
>>> new file mode 100644
>>> index 0000000..8c3ddaf
>>> --- /dev/null
>>> +++ b/meta/recipes-devtools/buildchroot/files/build-cross.sh
>>> @@ -0,0 +1,29 @@
>>> +#!/bin/bash
>>> +#
>>> +# This software is a part of ISAR.
>>> +# Copyright (C) 2015-2017 ilbers GmbH
>>> +# Copyright (c) 2018 Siemens AG
>>> +
>>> +set -e
>>> +
>>> +# Go to build directory
>>> +cd $1
>>> +
>>> +# Add target architecture
>>> +dpkg --add-architecture $2
>>> +
>>> +# To avoid Perl locale warnings:
>>> +export LC_ALL=C
>>> +export LANG=C
>>> +export LANGUAGE=C
>>> +
>>> +# If autotools files have been created, update their timestamp to
>>> +# prevent them from being regenerated
>>> +for i in configure aclocal.m4 Makefile.am Makefile.in; do
>>> +    if [ -f "${i}" ]; then
>>> +        touch "${i}"
>>> +    fi
>>> +done
>>> +
>>> +# Build the package
>>> +dpkg-buildpackage -a$2
>>>
>>
>> This build-cross.sh doesn't install any of the build dependencies the
>> target package lists. How do they come in? Or is this a limitation of
>> this version? For the kernel cross-build, I'm currently trying to
>> translate the what is needed for a corresponding build-kernel-cross.sh.
>>
> 
> That's exactly the limitation of this series about which I mentioned in
> the cover letter. Attempt to install them in similar way like for
> 'build.sh' failed, due to it tries to replace native packages like:
> build-essentials, binutils by foreign ones :-(. Henning already proposed
> to try to play with mk-build-deps params, so I'm going to do this.

OK, this would be a road-blocker for building anything cross. Already
libhello would lack docbook-to-man (if that is a real dependency).

> 
>> Those packages would go into the cross-buildchroot, or are we also using
>> the native buildchroot here?
> 
> buildchroot and buildchroot-cross has nothing common except isar-apt
> repository.

So the plan is to install all build deps into the buildchroot-cross, right?

Is there an officially documented way in Debian to do cross builds? What
does it say regarding build-deps installation?

Jan

Re: [PATCH 1/9] isar-bootstrap: Add routin to determine host arch

[Jan Kiszka]

On 2018-06-15 13:11, Alexander Smirnov wrote:
> Hi Henning,
> 
>> Am Wed, 13 Jun 2018 15:07:59 +0200
>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>
>>> This method will be used to distinguish host and target bootstraps.
>>> The standard python platform module is not robust here, because it
>>> provides the architecture label from kernel. For example running
>>> platfotm.machine() on Fedora kernel will return 'x86_64', what is
>>> invalid specificator for Debian systems.
>>>
>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>> ---
>>>  meta/classes/isar-bootstrap-helper.bbclass | 9 +++++++++
>>>  1 file changed, 9 insertions(+)
>>>
>>> diff --git a/meta/classes/isar-bootstrap-helper.bbclass
>>> b/meta/classes/isar-bootstrap-helper.bbclass index 3c10fc7..2598ea3
>>> 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
>>> +++ b/meta/classes/isar-bootstrap-helper.bbclass
>>> @@ -14,6 +14,15 @@ def reverse_bb_array(d, varname):
>>>      array = reversed(array.split())
>>>      return " ".join(i for i in array)
>>>  
>>> +python () {
>>> +    import subprocess
>>> +    host_arch = subprocess.Popen("/usr/bin/dpkg
>>> --print-architecture",
>>> +                                 shell=True,
>>> +                                 stdout=subprocess.PIPE
>>> +                                ).stdout.read().decode('utf-8').strip()
>>> +    d.setVar("HOST_ARCH", host_arch);
>>> +}
>>
>> Consider not using an absolute path. That defeats the magic of PATH.
> 
> I agree with you here, but btw, do you (or probably somebody else) know how to workaround this in
> python subprocess? An attempt to run just 'dpkg' fails with the error:
> command not found.

I think your invocation of Popen just misses to pass in the env. Check
other call sites for the right pattern.

Jan

Re: [PATCH 5/9] buildchroot-cross: Initial implementation

[Alexander Smirnov]

Hi Henning,

> Am Wed, 13 Jun 2018 15:08:03 +0200
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
> > Add initial generation of buildchroot for cross-compilation.
> > 
> > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> > ---
> >  meta/conf/isar-bitbake.conf                        |  1 +
> >  .../buildchroot/buildchroot-cross.bb               | 53
> > ++++++++++++++++++++++ .../buildchroot/files/build-cross.sh
> > | 29 ++++++++++++ 3 files changed, 83 insertions(+)
> >  create mode 100644
> > meta/recipes-devtools/buildchroot/buildchroot-cross.bb create mode
> > 100644 meta/recipes-devtools/buildchroot/files/build-cross.sh
> > 
> > diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
> > index ead7798..1412508 100644
> > --- a/meta/conf/isar-bitbake.conf
> > +++ b/meta/conf/isar-bitbake.conf
> > @@ -22,6 +22,7 @@ WORKDIR =
> > "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}" DL_DIR =
> > "${TOPDIR}/downloads" SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
> >  BUILDCHROOT_DIR =
> > "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
> > +BUILDCHROOT_CROSS_DIR =
> > "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-cross/rootfs"
> > CACHE = "${TMPDIR}/cache" OVERRIDES_append =
> > ":${DISTRO}:${DISTRO_ARCH}" diff --git
> > a/meta/recipes-devtools/buildchroot/buildchroot-cross.bb
> > b/meta/recipes-devtools/buildchroot/buildchroot-cross.bb new file
> > mode 100644 index 0000000..935a626 --- /dev/null
> > +++ b/meta/recipes-devtools/buildchroot/buildchroot-cross.bb
> > @@ -0,0 +1,53 @@
> > +# Root filesystem for packages cross-building
> > +#
> > +# This software is a part of ISAR.
> > +# Copyright (C) 2015-2016 ilbers GmbH
> > +
> > +DESCRIPTION = "Isar development cross-filesystem"
> > +
> > +LICENSE = "gpl-2.0"
> > +LIC_FILES_CHKSUM =
> > "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
> > + +FILESPATH_prepend := "${THISDIR}/files:"
> 
> Same as seen in Maxims patches.
> No need to prepend FILESPATH, set PV with proper filename
> 
> I guess here some "hacks" are required because other recipes will need
> to find that and can not know PV and stuff.
> 
> > +SRC_URI = "file://configscript.sh \
> > +           file://build-cross.sh"
> > +PV = "1.0"
> > +
> > +inherit isar-bootstrap-helper
> > +
> > +BUILDCHROOT_CROSS_PREINSTALL ?= "gcc-multilib \
> > +                                 make \
> > +                                 build-essential \
> > +                                 debhelper \
> > +                                 autotools-dev \
> > +                                 dpkg \
> > +                                 locales \
> > +                                 docbook-to-man \
> > +                                 apt \
> > +                                 automake \
> > +                                 devscripts \
> > +                                 equivs"
> > +
> > +# TODO: make this inclusion depending on the target arch
> > +BUILDCHROOT_CROSS_PREINSTALL += "binutils"
> > +
> > +WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
> 
> WORKDIR = ${BUILDCHROOT_CROSS_DIR}/../
> 
> maybe?
> 

For me - no problem :-)

> > +do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> > +do_build[root_cleandirs] = "${BUILDCHROOT_CROSS_DIR} \
> > +                            ${BUILDCHROOT_CROSS_DIR}/isar-apt \
> > +                            ${BUILDCHROOT_CROSS_DIR}/downloads \
> > +                            ${BUILDCHROOT_CROSS_DIR}/home/builder"
> > +do_build[depends] = "isar-apt:do_cache_config
> > isar-bootstrap-host:do_deploy" +
> > +do_build() {
> > +    setup_root_file_system --host-arch "${BUILDCHROOT_CROSS_DIR}"
> > ${BUILDCHROOT_CROSS_PREINSTALL} +
> > +    # Install package builder script
> > +    sudo chmod -R a+rw "${BUILDCHROOT_CROSS_DIR}/home/builder"
> > +    sudo install -m 755 ${WORKDIR}/build-cross.sh
> > ${BUILDCHROOT_CROSS_DIR}/build.sh +
> > +    # Configure root filesystem
> > +    sudo install -m 755 ${WORKDIR}/configscript.sh
> > ${BUILDCHROOT_CROSS_DIR}
> > +    sudo chroot ${BUILDCHROOT_CROSS_DIR} /configscript.sh
> > +}
> > diff --git a/meta/recipes-devtools/buildchroot/files/build-cross.sh
> > b/meta/recipes-devtools/buildchroot/files/build-cross.sh new file
> > mode 100644 index 0000000..8c3ddaf
> > --- /dev/null
> > +++ b/meta/recipes-devtools/buildchroot/files/build-cross.sh
> > @@ -0,0 +1,29 @@
> > +#!/bin/bash
> > +#
> > +# This software is a part of ISAR.
> > +# Copyright (C) 2015-2017 ilbers GmbH
> > +# Copyright (c) 2018 Siemens AG
> > +
> > +set -e
> > +
> > +# Go to build directory
> > +cd $1
> > +
> > +# Add target architecture
> > +dpkg --add-architecture $2
> > +
> > +# To avoid Perl locale warnings:
> > +export LC_ALL=C
> > +export LANG=C
> > +export LANGUAGE=C
> > +
> > +# If autotools files have been created, update their timestamp to
> > +# prevent them from being regenerated
> > +for i in configure aclocal.m4 Makefile.am Makefile.in; do
> > +    if [ -f "${i}" ]; then
> > +        touch "${i}"
> > +    fi
> > +done
> > +
> > +# Build the package
> > +dpkg-buildpackage -a$2
> 
> Can that be done with changing/extending build.sh and not introducing a
> copy?

At the moment I'm in stuck with the deps installation. When I'll find
the clue how to process them - I hope there will be the only one
build.sh.

Alex

> 
> Henning

Re: [RFC] [PATCH 0/9] Initial cross-compilation support

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 6923 bytes --]

Hi Alex,

On Fri, 2018-06-15 at 12:50 +0200, Alexander Smirnov wrote:
> On Thu, Jun 14, 2018 at 02:04:47PM +0200, Claudius Heine wrote:
> 
> Hi everybody,
> 
> > Hi,
> > 
> > On Wed, 2018-06-13 at 15:07 +0200, Alexander Smirnov wrote:
> > > Hi all,
> > > 
> > > this series introduces basic cross-compilation support for Isar.
> > > The
> > > implementation
> > > is based around new entity - buildchroot-cross, which is quite
> > > similar to original
> > > buildchroot, but has host architecture.
> > 
> > Here is a crazy idea from me and Henning:
> > Have you tried just installing an amd64-arm cross toolchain into
> > the
> > existing arm buildchroot, then switch to it from the 'native'
> > toolchain
> > using a combination of PATH variable entries, symlinks and/or
> > wrapper
> > scripts and let binfmt deal with calling the amd64 binaries of the
> > toolchain in the arm chroot environment?
> > 
> > This way the arm system would think that you are compiling nativly
> > while in fact you are using a cross compiler that is not emulated.
> > 
> > Would that work?
> 
> I'm not sure that cross-compiling is onl about compiler stuff, you
> also
> need:
>  - make
>  - autotools
>  - bash for scripts
>  - perl (f�r apt/deb K�chen)
>  - possibly something else
> 
> The list of tools above with its deps itself is already small rootfs.
> Managing mix of these tools for different architecture could be too
> costly.

You are right, but I would just use the compiler non-emulated and the
rest through qemu, since that is the bottleneck. The rest of the system
 (make, autoconf, perl, bash, ...) can think that its compiling native
for arm while in fact it is using a cross-compiler behind the scene.

But if you investigated this and found that this isn't possible or can
think of situations where this doesn't work then then don't do it that
way.

I just thought that cross-compiling debian packages isn't really well
supported and tried to think of different solutions to solve this. One
way is using this trick, another I took from how ArchlinuxARM compiles
packages. They used distcc to send the source code from the ARM
environment to a cross-compiler suite. And since that works there it
could work here as well, I thought.

If there is an issue with doing it that way, I would be interested to
hear about it.

Maybe the situation with cross-compiling debian also changed and that
can be done very easily now. I am not up-to-date on that. But if you
have issues with resolving and installing build dependencies that might
be the way to go.

kind regards,
Claudius

> 
> Alex
> 
> > 
> > Cheers,
> > Claudius
> > 
> > > 
> > > This series tested for the following configuration only:
> > > 
> > >  $ bitbake multiconfig:qemui386
> > > 
> > > In this build:
> > >  - libhello is cross-compiled
> > >  - example-hello is compiled traditionally (but it's successfully
> > > linked with libhello)
> > > 
> > > TODO list:
> > >  - Depending on the target architecture, various host tools
> > > should be
> > > installed. For example
> > >    binutils package has the following names:
> > >    * binutils
> > >    * binutils-x86-64-linux-gnu
> > >    * binutils-arm-linux-gnueabihf
> > >  - Cross-dependency installation. This topic is still open for me
> > > because 'mk-build-deps' tool
> > >    doesn't work here correctly. For example package depends on
> > > binutils, but instead of
> > >    installation of 'binutils-arm-linux-gnueabihf' it tries to
> > > install
> > > 'binutils:armhf' and ruins
> > >    the rootfs.
> > > 
> > > So, as usually, comments and suggestions are welcome :-)
> > > 
> > > With best redgards,
> > > Alex
> > > 
> > > Alexander Smirnov (9):
> > >   isar-bootstrap: Add routin to determine host arch
> > >   isar-bootstrap: Move common part to include
> > >   isar-bootstrap: Add host architecture support
> > >   isar-bootstrap-helper: Add parameter to set arch
> > >   buildchroot-cross: Initial implementation
> > >   classes/dpkg*: Relocate dependency from buildchroot
> > >   classes/dpkg*: Drop hardcoded buildchroot blobs
> > >   classes/dpkg-cross: Initial implementation
> > >   [FOR TESTING] libhello: Switch to cross compilation
> > > 
> > >  meta-isar/recipes-app/libhello/libhello.bb         |   2 +-
> > >  meta/classes/dpkg-base.bbclass                     |  17 +-
> > >  meta/classes/dpkg-cross.bbclass                    |  15 ++
> > >  meta/classes/dpkg-raw.bbclass                      |   7 +-
> > >  meta/classes/dpkg.bbclass                          |   7 +-
> > >  meta/classes/image.bbclass                         |   2 +-
> > >  meta/classes/isar-bootstrap-helper.bbclass         |  13 +-
> > >  meta/conf/isar-bitbake.conf                        |   1 +
> > >  .../isar-bootstrap/isar-bootstrap-host.bb          |  44 ++++
> > >  .../isar-bootstrap/isar-bootstrap-target.bb        |  44 ++++
> > >  meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 259 -----
> > > ----
> > > ------------
> > >  .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 223
> > > ++++++++++++++++++
> > >  .../buildchroot/buildchroot-cross.bb               |  53 +++++
> > >  meta/recipes-devtools/buildchroot/buildchroot.bb   |   2 +-
> > >  .../buildchroot/files/build-cross.sh               |  29 +++
> > >  15 files changed, 443 insertions(+), 275 deletions(-)
> > >  create mode 100644 meta/classes/dpkg-cross.bbclass
> > >  create mode 100644 meta/recipes-core/isar-bootstrap/isar-
> > > bootstrap-
> > > host.bb
> > >  create mode 100644 meta/recipes-core/isar-bootstrap/isar-
> > > bootstrap-
> > > target.bb
> > >  delete mode 100644 meta/recipes-core/isar-bootstrap/isar-
> > > bootstrap.bb
> > >  create mode 100644 meta/recipes-core/isar-bootstrap/isar-
> > > bootstrap.inc
> > >  create mode 100644 meta/recipes-
> > > devtools/buildchroot/buildchroot-
> > > cross.bb
> > >  create mode 100644 meta/recipes-
> > > devtools/buildchroot/files/build-
> > > cross.sh
> > > 
> > > -- 
> > > 2.1.4
> > > 
> > 
> > -- 
> > DENX Software Engineering GmbH,      Managing Director: Wolfgang
> > Denk
> > HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> > Germany
> > Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.
> > de
> > 
> >             PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808
> > B153
> >                               Keyserver: hkp://pool.sks-
> > keyservers.net
> 
> 
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 5/9] buildchroot-cross: Initial implementation

[Alexander Smirnov]

Hi Jan,

On 06/15/2018 02:12 PM, Jan Kiszka wrote:
> On 2018-06-15 12:56, Alexander Smirnov wrote:
>> On Thu, Jun 14, 2018 at 05:55:48PM +0200, Jan Kiszka wrote:
>>> On 2018-06-13 15:08, Alexander Smirnov wrote:
>>>> Add initial generation of buildchroot for cross-compilation.
>>>>
>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>> ---
>>>>   meta/conf/isar-bitbake.conf                        |  1 +
>>>>   .../buildchroot/buildchroot-cross.bb               | 53 ++++++++++++++++++++++
>>>>   .../buildchroot/files/build-cross.sh               | 29 ++++++++++++
>>>>   3 files changed, 83 insertions(+)
>>>>   create mode 100644 meta/recipes-devtools/buildchroot/buildchroot-cross.bb
>>>>   create mode 100644 meta/recipes-devtools/buildchroot/files/build-cross.sh
>>>>
>>>> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
>>>> index ead7798..1412508 100644
>>>> --- a/meta/conf/isar-bitbake.conf
>>>> +++ b/meta/conf/isar-bitbake.conf
>>>> @@ -22,6 +22,7 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}"
>>>>   DL_DIR = "${TOPDIR}/downloads"
>>>>   SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
>>>>   BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
>>>> +BUILDCHROOT_CROSS_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-cross/rootfs"
>>>>   CACHE = "${TMPDIR}/cache"
>>>>   
>>>>   OVERRIDES_append = ":${DISTRO}:${DISTRO_ARCH}"
>>>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot-cross.bb b/meta/recipes-devtools/buildchroot/buildchroot-cross.bb
>>>> new file mode 100644
>>>> index 0000000..935a626
>>>> --- /dev/null
>>>> +++ b/meta/recipes-devtools/buildchroot/buildchroot-cross.bb
>>>> @@ -0,0 +1,53 @@
>>>> +# Root filesystem for packages cross-building
>>>> +#
>>>> +# This software is a part of ISAR.
>>>> +# Copyright (C) 2015-2016 ilbers GmbH
>>>> +
>>>> +DESCRIPTION = "Isar development cross-filesystem"
>>>> +
>>>> +LICENSE = "gpl-2.0"
>>>> +LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
>>>> +
>>>> +FILESPATH_prepend := "${THISDIR}/files:"
>>>> +SRC_URI = "file://configscript.sh \
>>>> +           file://build-cross.sh"
>>>> +PV = "1.0"
>>>> +
>>>> +inherit isar-bootstrap-helper
>>>> +
>>>> +BUILDCHROOT_CROSS_PREINSTALL ?= "gcc-multilib \
>>>> +                                 make \
>>>> +                                 build-essential \
>>>> +                                 debhelper \
>>>> +                                 autotools-dev \
>>>> +                                 dpkg \
>>>> +                                 locales \
>>>> +                                 docbook-to-man \
>>>> +                                 apt \
>>>> +                                 automake \
>>>> +                                 devscripts \
>>>> +                                 equivs"
>>>> +
>>>> +# TODO: make this inclusion depending on the target arch
>>>> +BUILDCHROOT_CROSS_PREINSTALL += "binutils"
>>>> +
>>>> +WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>>>> +
>>>> +do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>>> +do_build[root_cleandirs] = "${BUILDCHROOT_CROSS_DIR} \
>>>> +                            ${BUILDCHROOT_CROSS_DIR}/isar-apt \
>>>> +                            ${BUILDCHROOT_CROSS_DIR}/downloads \
>>>> +                            ${BUILDCHROOT_CROSS_DIR}/home/builder"
>>>> +do_build[depends] = "isar-apt:do_cache_config isar-bootstrap-host:do_deploy"
>>>> +
>>>> +do_build() {
>>>> +    setup_root_file_system --host-arch "${BUILDCHROOT_CROSS_DIR}" ${BUILDCHROOT_CROSS_PREINSTALL}
>>>> +
>>>> +    # Install package builder script
>>>> +    sudo chmod -R a+rw "${BUILDCHROOT_CROSS_DIR}/home/builder"
>>>> +    sudo install -m 755 ${WORKDIR}/build-cross.sh ${BUILDCHROOT_CROSS_DIR}/build.sh
>>>> +
>>>> +    # Configure root filesystem
>>>> +    sudo install -m 755 ${WORKDIR}/configscript.sh ${BUILDCHROOT_CROSS_DIR}
>>>> +    sudo chroot ${BUILDCHROOT_CROSS_DIR} /configscript.sh
>>>> +}
>>>> diff --git a/meta/recipes-devtools/buildchroot/files/build-cross.sh b/meta/recipes-devtools/buildchroot/files/build-cross.sh
>>>> new file mode 100644
>>>> index 0000000..8c3ddaf
>>>> --- /dev/null
>>>> +++ b/meta/recipes-devtools/buildchroot/files/build-cross.sh
>>>> @@ -0,0 +1,29 @@
>>>> +#!/bin/bash
>>>> +#
>>>> +# This software is a part of ISAR.
>>>> +# Copyright (C) 2015-2017 ilbers GmbH
>>>> +# Copyright (c) 2018 Siemens AG
>>>> +
>>>> +set -e
>>>> +
>>>> +# Go to build directory
>>>> +cd $1
>>>> +
>>>> +# Add target architecture
>>>> +dpkg --add-architecture $2
>>>> +
>>>> +# To avoid Perl locale warnings:
>>>> +export LC_ALL=C
>>>> +export LANG=C
>>>> +export LANGUAGE=C
>>>> +
>>>> +# If autotools files have been created, update their timestamp to
>>>> +# prevent them from being regenerated
>>>> +for i in configure aclocal.m4 Makefile.am Makefile.in; do
>>>> +    if [ -f "${i}" ]; then
>>>> +        touch "${i}"
>>>> +    fi
>>>> +done
>>>> +
>>>> +# Build the package
>>>> +dpkg-buildpackage -a$2
>>>>
>>>
>>> This build-cross.sh doesn't install any of the build dependencies the
>>> target package lists. How do they come in? Or is this a limitation of
>>> this version? For the kernel cross-build, I'm currently trying to
>>> translate the what is needed for a corresponding build-kernel-cross.sh.
>>>
>>
>> That's exactly the limitation of this series about which I mentioned in
>> the cover letter. Attempt to install them in similar way like for
>> 'build.sh' failed, due to it tries to replace native packages like:
>> build-essentials, binutils by foreign ones :-(. Henning already proposed
>> to try to play with mk-build-deps params, so I'm going to do this.
> 
> OK, this would be a road-blocker for building anything cross. Already
> libhello would lack docbook-to-man (if that is a real dependency).
> 
>>
>>> Those packages would go into the cross-buildchroot, or are we also using
>>> the native buildchroot here?
>>
>> buildchroot and buildchroot-cross has nothing common except isar-apt
>> repository.
> 
> So the plan is to install all build deps into the buildchroot-cross, right?
> 
> Is there an officially documented way in Debian to do cross builds? What
> does it say regarding build-deps installation?
> 

According to the following link (and some others):

https://wiki.debian.org/Multiarch/HOWTO#Installing_cross-dependencies

The official way to install build deps:

$ apt-get build-dep ...

But this doesn't work for us, because we have no source package yet :-(

Alex

Re: [RFC] [PATCH 0/9] Initial cross-compilation support

[Alexander Smirnov]

Hi Henning,

On 06/14/2018 02:20 PM, Henning Schild wrote:
> Am Wed, 13 Jun 2018 15:07:58 +0200
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Hi all,
>>
>> this series introduces basic cross-compilation support for Isar. The
>> implementation is based around new entity - buildchroot-cross, which
>> is quite similar to original buildchroot, but has host architecture.
>>
>> This series tested for the following configuration only:
>>
>>   $ bitbake multiconfig:qemui386
>>
>> In this build:
>>   - libhello is cross-compiled
>>   - example-hello is compiled traditionally (but it's successfully
>> linked with libhello)
>>
>> TODO list:
>>   - Depending on the target architecture, various host tools should be
>> installed. For example binutils package has the following names:
>>     * binutils
>>     * binutils-x86-64-linux-gnu
>>     * binutils-arm-linux-gnueabihf
>>   - Cross-dependency installation. This topic is still open for me
>> because 'mk-build-deps' tool doesn't work here correctly. For example
>> package depends on binutils, but instead of installation of
>> 'binutils-arm-linux-gnueabihf' it tries to install 'binutils:armhf'
>> and ruins the rootfs.
> 
> mk-build-deps supports the following switches "--arch foo --host-arch
> foo", maybe they help?
> 

Sorry that mess the thread, but the last message in this thread doesn't 
contain you comment :-)

So, I've tried to play with mk-build-deps, but results are really 
disappointed:

1. mk-build-deps support --host-arch and --build-arch starting from 
stretch only, in jessie these parameters are not presented.

2. I've tried to run mk-build-deps in stretch for libhello:

$ mk-build-deps --host-arch amd64 --build-arch i386 debian/control
...
dpkg -I libhello-cross-build-deps-depends_0.1_amd64.deb
  new debian package, version 2.0.
  size 2012 bytes: control archive=516 bytes.
      367 bytes,    11 lines      control
      283 bytes,     3 lines      md5sums
  Package: libhello-cross-build-deps-depends
  Version: 0.1
  Architecture: amd64
  Maintainer: root <root@>
  Installed-Size: 9
  Depends: build-essential:i386, crossbuild-essential-amd64:i386, 
debhelper (>= 9), docbook-to-man
  Section: devel
  Priority: optional
  Multi-Arch: foreign
  Description: build-dependencies for libhello
   Dependency package to build the 'libhello' package

BUT: package 'crossbuild-essential-amd64:i386' does not exists. At least 
I wasn't able to find it via apt (I've also added i386 to source list).

Another issue during attempts to install generated deps:

$ apt-get install build-essential:i386
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
  build-essential:i386 : Depends: libc6-dev:i386 but it is not going to 
be installed or
                                  libc-dev:i386
                         Depends: gcc:i386 (>= 4:6.3) but it is not 
going to be installed
                         Depends: g++:i386 (>= 4:6.3) but it is not 
going to be installed
                         Depends: make:i386
  libhello-cross-build-deps : Depends: crossbuild-essential-amd64:i386 
but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages 
(or specify a solution).

It tries to pull target compiler, what is also incorrect behavior. The 
cross binutils should be installed instead.

So, will continue this investigation :-(

Alex

> Henning
> 
>> So, as usually, comments and suggestions are welcome :-)
>>
>> With best redgards,
>> Alex
>>
>> Alexander Smirnov (9):
>>    isar-bootstrap: Add routin to determine host arch
>>    isar-bootstrap: Move common part to include
>>    isar-bootstrap: Add host architecture support
>>    isar-bootstrap-helper: Add parameter to set arch
>>    buildchroot-cross: Initial implementation
>>    classes/dpkg*: Relocate dependency from buildchroot
>>    classes/dpkg*: Drop hardcoded buildchroot blobs
>>    classes/dpkg-cross: Initial implementation
>>    [FOR TESTING] libhello: Switch to cross compilation
>>
>>   meta-isar/recipes-app/libhello/libhello.bb         |   2 +-
>>   meta/classes/dpkg-base.bbclass                     |  17 +-
>>   meta/classes/dpkg-cross.bbclass                    |  15 ++
>>   meta/classes/dpkg-raw.bbclass                      |   7 +-
>>   meta/classes/dpkg.bbclass                          |   7 +-
>>   meta/classes/image.bbclass                         |   2 +-
>>   meta/classes/isar-bootstrap-helper.bbclass         |  13 +-
>>   meta/conf/isar-bitbake.conf                        |   1 +
>>   .../isar-bootstrap/isar-bootstrap-host.bb          |  44 ++++
>>   .../isar-bootstrap/isar-bootstrap-target.bb        |  44 ++++
>>   meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 259
>> --------------------- .../recipes-core/isar-bootstrap/isar-bootstrap.inc
>> | 223
>> ++++++++++++++++++ .../buildchroot/buildchroot-cross.bb
>> |  53 +++++ meta/recipes-devtools/buildchroot/buildchroot.bb   |   2
>> +- .../buildchroot/files/build-cross.sh               |  29 +++ 15
>> files changed, 443 insertions(+), 275 deletions(-) create mode 100644
>> meta/classes/dpkg-cross.bbclass create mode 100644
>> meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb create mode
>> 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
>> delete mode 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap.bb
>> create mode 100644
>> meta/recipes-core/isar-bootstrap/isar-bootstrap.inc create mode
>> 100644 meta/recipes-devtools/buildchroot/buildchroot-cross.bb create
>> mode 100644 meta/recipes-devtools/buildchroot/files/build-cross.sh
>>
> 

Re: [RFC] [PATCH 0/9] Initial cross-compilation support

[Henning Schild]

Am Sun, 17 Jun 2018 22:41:08 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Hi Henning,
> 
> On 06/14/2018 02:20 PM, Henning Schild wrote:
> > Am Wed, 13 Jun 2018 15:07:58 +0200
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> Hi all,
> >>
> >> this series introduces basic cross-compilation support for Isar.
> >> The implementation is based around new entity - buildchroot-cross,
> >> which is quite similar to original buildchroot, but has host
> >> architecture.
> >>
> >> This series tested for the following configuration only:
> >>
> >>   $ bitbake multiconfig:qemui386
> >>
> >> In this build:
> >>   - libhello is cross-compiled
> >>   - example-hello is compiled traditionally (but it's successfully
> >> linked with libhello)
> >>
> >> TODO list:
> >>   - Depending on the target architecture, various host tools
> >> should be installed. For example binutils package has the
> >> following names:
> >>     * binutils
> >>     * binutils-x86-64-linux-gnu
> >>     * binutils-arm-linux-gnueabihf
> >>   - Cross-dependency installation. This topic is still open for me
> >> because 'mk-build-deps' tool doesn't work here correctly. For
> >> example package depends on binutils, but instead of installation of
> >> 'binutils-arm-linux-gnueabihf' it tries to install 'binutils:armhf'
> >> and ruins the rootfs.  
> > 
> > mk-build-deps supports the following switches "--arch foo
> > --host-arch foo", maybe they help?
> >   
> 
> Sorry that mess the thread, but the last message in this thread
> doesn't contain you comment :-)
> 
> So, I've tried to play with mk-build-deps, but results are really 
> disappointed:
> 
> 1. mk-build-deps support --host-arch and --build-arch starting from 
> stretch only, in jessie these parameters are not presented.
> 
> 2. I've tried to run mk-build-deps in stretch for libhello:
> 
> $ mk-build-deps --host-arch amd64 --build-arch i386 debian/control
> ...
> dpkg -I libhello-cross-build-deps-depends_0.1_amd64.deb
>   new debian package, version 2.0.
>   size 2012 bytes: control archive=516 bytes.
>       367 bytes,    11 lines      control
>       283 bytes,     3 lines      md5sums
>   Package: libhello-cross-build-deps-depends
>   Version: 0.1
>   Architecture: amd64
>   Maintainer: root <root@>
>   Installed-Size: 9
>   Depends: build-essential:i386, crossbuild-essential-amd64:i386, 
> debhelper (>= 9), docbook-to-man
>   Section: devel
>   Priority: optional
>   Multi-Arch: foreign
>   Description: build-dependencies for libhello
>    Dependency package to build the 'libhello' package
> 
> BUT: package 'crossbuild-essential-amd64:i386' does not exists. At
> least I wasn't able to find it via apt (I've also added i386 to
> source list).
> 
> Another issue during attempts to install generated deps:
> 
> $ apt-get install build-essential:i386
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> You might want to run 'apt --fix-broken install' to correct these.
> The following packages have unmet dependencies:
>   build-essential:i386 : Depends: libc6-dev:i386 but it is not going
> to be installed or
>                                   libc-dev:i386
>                          Depends: gcc:i386 (>= 4:6.3) but it is not 
> going to be installed
>                          Depends: g++:i386 (>= 4:6.3) but it is not 
> going to be installed
>                          Depends: make:i386
>   libhello-cross-build-deps : Depends:
> crossbuild-essential-amd64:i386 but it is not installable
> E: Unmet dependencies. Try 'apt --fix-broken install' with no
> packages (or specify a solution).
> 
> It tries to pull target compiler, what is also incorrect behavior.
> The cross binutils should be installed instead.
> 
> So, will continue this investigation :-(

The stretch+ would not be such a big issue, we already have a few
features that will not work for older distros. But since debian
upstream does not really support cross, it is not too surprising that
it is hard to get working.

Since cross will be more of a hack anyways, maybe you can revive the
perl (or was it sed) magic that my mk-build-dep patch replaced. We
could also think about a variable like "CROSS_BUILD_DEBIAN_DEPENDS"
that would by default carry the toolchain etc. and could be extended
with whatever all the recipes for the image might need. So instead of
actually looking at the recipe, just expect that global image-variable
to be all you need. ... Again the whole thing is a hack. Code that
tries to use debian/control to install more is optional on top of the
general mechanism.

Henning

> Alex
> 
> > Henning
> >   
> >> So, as usually, comments and suggestions are welcome :-)
> >>
> >> With best redgards,
> >> Alex
> >>
> >> Alexander Smirnov (9):
> >>    isar-bootstrap: Add routin to determine host arch
> >>    isar-bootstrap: Move common part to include
> >>    isar-bootstrap: Add host architecture support
> >>    isar-bootstrap-helper: Add parameter to set arch
> >>    buildchroot-cross: Initial implementation
> >>    classes/dpkg*: Relocate dependency from buildchroot
> >>    classes/dpkg*: Drop hardcoded buildchroot blobs
> >>    classes/dpkg-cross: Initial implementation
> >>    [FOR TESTING] libhello: Switch to cross compilation
> >>
> >>   meta-isar/recipes-app/libhello/libhello.bb         |   2 +-
> >>   meta/classes/dpkg-base.bbclass                     |  17 +-
> >>   meta/classes/dpkg-cross.bbclass                    |  15 ++
> >>   meta/classes/dpkg-raw.bbclass                      |   7 +-
> >>   meta/classes/dpkg.bbclass                          |   7 +-
> >>   meta/classes/image.bbclass                         |   2 +-
> >>   meta/classes/isar-bootstrap-helper.bbclass         |  13 +-
> >>   meta/conf/isar-bitbake.conf                        |   1 +
> >>   .../isar-bootstrap/isar-bootstrap-host.bb          |  44 ++++
> >>   .../isar-bootstrap/isar-bootstrap-target.bb        |  44 ++++
> >>   meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 259
> >> --------------------- .../recipes-core/isar-bootstrap/isar-bootstrap.inc
> >> | 223
> >> ++++++++++++++++++ .../buildchroot/buildchroot-cross.bb
> >> |  53 +++++ meta/recipes-devtools/buildchroot/buildchroot.bb   |
> >> 2 +- .../buildchroot/files/build-cross.sh               |  29 +++
> >> 15 files changed, 443 insertions(+), 275 deletions(-) create mode
> >> 100644 meta/classes/dpkg-cross.bbclass create mode 100644
> >> meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb create mode
> >> 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
> >> delete mode 100644
> >> meta/recipes-core/isar-bootstrap/isar-bootstrap.bb create mode
> >> 100644 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc create
> >> mode 100644 meta/recipes-devtools/buildchroot/buildchroot-cross.bb
> >> create mode 100644
> >> meta/recipes-devtools/buildchroot/files/build-cross.sh 
> >