[PATCH v4 00/13] Complete backlog: SDK, assorting fixed and cleanups

[PATCH v4 00/13] Complete backlog: SDK, assorting fixed and cleanups

[Jan Kiszka]

Rebased queue of my readily pending patches over next.

Changes:
- fixed "sdk: Make shipping the isar-apt repo opt-in"
- added "u-boot: Deprecate BUILD_DEPENDS..."

Local backlog also contains a new board (STM32MP15x) and TF-A and OP-TEE
support, but the test case for the latter is not yet working. Hope that
this will follow soon once I received the needed information to fix it.

Jan

Jan Kiszka (13):
  sdk: Add support for adding self-defined sdk packages
  sdk: Make all links in the SDK chroot relative
  sdk: Add script to relocate SDK
  sdk: Make shipping the isar-apt repo opt-in
  sdk: Inject sysroot path when calling relocated toolchain
  sdk: Update README.sdk
  isar-bootstrap: Connect to gpg-agent before adding keys
  buildchroot-host: Drop unneeded :native for riscv64 setup
  deb-dl-dir: Fix quoting
  Fix dependencies on isar-apt
  dpkg: Account for changes in mk-build-deps in bullseye
  isar-bootstrap: Align debootstrap calls
  u-boot: Deprecate BUILD_DEPENDS in favor of DEBIAN_BUILD_DEPENDS

 RECIPE-API-CHANGELOG.md                       |  4 ++
 doc/user_manual.md                            |  1 +
 meta/classes/deb-dl-dir.bbclass               |  4 +-
 meta/classes/dpkg-base.bbclass                |  1 -
 meta/classes/image-sdk-extension.bbclass      | 34 ++++++++-
 meta/classes/rootfs.bbclass                   |  2 +-
 .../u-boot/files/debian/control.tmpl          |  2 +-
 meta/recipes-bsp/u-boot/u-boot-custom.inc     | 10 ++-
 .../isar-bootstrap/isar-bootstrap.inc         | 24 ++++---
 .../buildchroot/buildchroot-host.bb           |  6 +-
 .../buildchroot/files/deps.sh                 |  7 +-
 .../sdkchroot/files/README.sdk                | 71 ++++++++++---------
 .../sdkchroot/files/gcc-sysroot-wrapper.sh    | 16 +++++
 .../sdkchroot/files/relocate-sdk.sh           | 44 ++++++++++++
 meta/recipes-devtools/sdkchroot/sdkchroot.bb  | 33 ++++++---
 15 files changed, 188 insertions(+), 71 deletions(-)
 create mode 100755 meta/recipes-devtools/sdkchroot/files/gcc-sysroot-wrapper.sh
 create mode 100755 meta/recipes-devtools/sdkchroot/files/relocate-sdk.sh

-- 
2.26.2

[PATCH v4 01/13] sdk: Add support for adding self-defined sdk packages

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

We do not yet have a good algorithm for automatically adding build
dependencies to the sdk beyond the basic set, let's allow users to
append what they need by appending SDK_PREINSTALL. Analogously to other
images, also allow to install self-built packages, consequently using
SDK_INSTALL.

Based on original patch by Le Jin.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 doc/user_manual.md                           |  1 +
 meta/recipes-devtools/sdkchroot/sdkchroot.bb | 25 ++++++++++++--------
 2 files changed, 16 insertions(+), 10 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 8d04cd25..dd6fc344 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -761,6 +761,7 @@ target binary artifacts. Developer chroots to sdk rootfs and develops applicatio
 
 User manually triggers creation of SDK root filesystem for his target platform by launching the task `do_populate_sdk` for target image, f.e.
 `bitbake -c do_populate_sdk mc:${MACHINE}-${DISTRO}:isar-image-base`.
+Packages that should be additionally installed into the SDK can be appended to `SDK_PREINSTALL` (external repositories) and `SDK_INSTALL` (self-built).
 
 The resulting SDK rootfs is archived into `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz`.
 It is additionally available for direct use under `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}/`.
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index 58420d05..05320a03 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -13,6 +13,10 @@ SRC_URI = " \
     file://README.sdk"
 PV = "0.1"
 
+SDK_INSTALL ?= ""
+
+DEPENDS += "${SDK_INSTALL}"
+
 TOOLCHAIN = "crossbuild-essential-${DISTRO_ARCH}"
 TOOLCHAIN_${HOST_ARCH} = "build-essential"
 TOOLCHAIN_i386 = "build-essential"
@@ -22,7 +26,7 @@ inherit rootfs
 ROOTFS_ARCH = "${HOST_ARCH}"
 ROOTFS_DISTRO = "${HOST_DISTRO}"
 ROOTFSDIR = "${S}"
-ROOTFS_PACKAGES = "${SDKCHROOT_PREINSTALL} ${TOOLCHAIN}"
+ROOTFS_PACKAGES = "${SDK_PREINSTALL} ${SDK_INSTALL} ${TOOLCHAIN}"
 ROOTFS_FEATURES += "clean-package-cache generate-manifest"
 ROOTFS_MANIFEST_DEPLOY_DIR = "${DEPLOY_DIR_SDKCHROOT}"
 
@@ -32,15 +36,16 @@ python() {
             d.getVar("ROOTFS_ARCH")))
 }
 
-SDKCHROOT_PREINSTALL := "debhelper \
-                           autotools-dev \
-                           dpkg \
-                           locales \
-                           docbook-to-man \
-                           apt \
-                           automake \
-                           devscripts \
-                           equivs"
+SDK_PREINSTALL += " \
+    debhelper \
+    autotools-dev \
+    dpkg \
+    locales \
+    docbook-to-man \
+    apt \
+    automake \
+    devscripts \
+    equivs"
 
 S = "${WORKDIR}/rootfs"
 
-- 
2.26.2

[PATCH v4 02/13] sdk: Make all links in the SDK chroot relative

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This, in combination with interp and rpatch rewriting, will allow to use
the SDK outside of its chroot.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/image-sdk-extension.bbclass | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass
index b91fdd64..b6e9ab09 100644
--- a/meta/classes/image-sdk-extension.bbclass
+++ b/meta/classes/image-sdk-extension.bbclass
@@ -18,6 +18,20 @@ do_populate_sdk() {
     # Remove setup scripts
     sudo rm -f ${SDKCHROOT_DIR}/chroot-setup.sh ${SDKCHROOT_DIR}/configscript.sh
 
+    # Make all links relative
+    for link in $(find ${SDKCHROOT_DIR}/ -type l); do
+        target=$(readlink $link)
+
+        if [ "${target#/}" != "${target}" ]; then
+            basedir=$(dirname $link)
+            new_target=$(realpath --no-symlinks -m --relative-to=$basedir ${SDKCHROOT_DIR}/${target})
+
+            # remove first to allow rewriting directory links
+            sudo rm $link
+            sudo ln -s $new_target $link
+        fi
+    done
+
     # Copy mount_chroot.sh for convenience
     sudo cp ${SCRIPTSDIR}/mount_chroot.sh ${SDKCHROOT_DIR}
 
-- 
2.26.2

[PATCH v4 03/13] sdk: Add script to relocate SDK

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

When run inside the unpacked SDK, this script tunes interp and rpath
entry in relevant binaries so that the cross conpilation tool can be
called outside of the chroot, irrespective of the host distribution.
Then only "--sysroot /path/to/sdkroot" needs to be passed to the
compiler.

The script also supports restoring the original chroot-mode when invoked
with the --restore-chroot option.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 .../sdkchroot/files/relocate-sdk.sh           | 41 +++++++++++++++++++
 meta/recipes-devtools/sdkchroot/sdkchroot.bb  |  2 +
 2 files changed, 43 insertions(+)
 create mode 100755 meta/recipes-devtools/sdkchroot/files/relocate-sdk.sh

diff --git a/meta/recipes-devtools/sdkchroot/files/relocate-sdk.sh b/meta/recipes-devtools/sdkchroot/files/relocate-sdk.sh
new file mode 100755
index 00000000..1c9b02fa
--- /dev/null
+++ b/meta/recipes-devtools/sdkchroot/files/relocate-sdk.sh
@@ -0,0 +1,41 @@
+#!/bin/sh
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2020
+#
+# SPDX-License-Identifier: MIT
+
+sdkroot=$(realpath $(dirname $0))
+arch=$(uname -m)
+
+new_sdkroot=$sdkroot
+
+case "$1" in
+--help|-h)
+	echo "Usage: $0 [--restore-chroot|-r]"
+	exit 0
+	;;
+--restore-chroot|-r)
+	new_sdkroot=/
+	;;
+esac
+
+if [ -z $(which patchelf 2>/dev/null) ]; then
+	echo "Please install 'patchelf' package first."
+	exit 1
+fi
+
+echo -n "Adjusting path of SDK to '${new_sdkroot}'... "
+
+for binary in $(find ${sdkroot}/usr/bin ${sdkroot}/usr/sbin ${sdkroot}/usr/lib/gcc* -executable -type f); do
+	interpreter=$(patchelf --print-interpreter ${binary} 2>/dev/null)
+	oldpath=${interpreter%/lib*/ld-linux*}
+	interpreter=${interpreter#${oldpath}}
+	if [ -n "${interpreter}" ]; then
+		patchelf --set-interpreter ${new_sdkroot}${interpreter} \
+			--set-rpath ${new_sdkroot}/usr/lib:${new_sdkroot}/usr/lib/${arch}-linux-gnu \
+			$binary 2>/dev/null
+	fi
+done
+
+echo "done"
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index 05320a03..5fba09c3 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -10,6 +10,7 @@ LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260
 
 SRC_URI = " \
     file://configscript.sh \
+    file://relocate-sdk.sh \
     file://README.sdk"
 PV = "0.1"
 
@@ -59,6 +60,7 @@ ROOTFS_POSTPROCESS_COMMAND =+ "sdkchroot_install_files"
 sdkchroot_install_files() {
     # Configure root filesystem
     sudo install -m 644 ${WORKDIR}/README.sdk ${S}
+    sudo install -m 755 ${WORKDIR}/relocate-sdk.sh ${S}
     sudo install -m 755 ${WORKDIR}/configscript.sh ${S}
     sudo chroot ${S} /configscript.sh  ${DISTRO_ARCH}
 }
-- 
2.26.2

[PATCH v4 04/13] sdk: Make shipping the isar-apt repo opt-in

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Users can add what should be included via SDK_INSTALL now, so this
should be the exception. Keep it around until we decide if we can drop
it completely.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/image-sdk-extension.bbclass     | 12 ++++++++++--
 meta/recipes-devtools/sdkchroot/sdkchroot.bb |  4 +++-
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass
index b6e9ab09..cb4fb55d 100644
--- a/meta/classes/image-sdk-extension.bbclass
+++ b/meta/classes/image-sdk-extension.bbclass
@@ -5,11 +5,19 @@
 #
 # This class extends the image.bbclass to supply the creation of a sdk
 
+SDK_INCLUDE_ISAR_APT ?= "0"
+
 do_populate_sdk[stamp-extra-info] = "${DISTRO}-${MACHINE}"
 do_populate_sdk[depends] = "sdkchroot:do_build"
+do_populate_sdk[vardeps] += "SDK_INCLUDE_ISAR_APT"
 do_populate_sdk() {
-    # Copy isar-apt with deployed Isar packages
-    sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO}  ${SDKCHROOT_DIR}/isar-apt
+    if [ "${SDK_INCLUDE_ISAR_APT}" = "1" ]; then
+        # Copy isar-apt with deployed Isar packages
+        sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${SDKCHROOT_DIR}/isar-apt
+    else
+        # Remove isar-apt repo entry
+        sudo rm -f ${SDKCHROOT_DIR}/etc/apt/sources.list.d/isar-apt.list
+    fi
 
     sudo umount -R ${SDKCHROOT_DIR}/dev || true
     sudo umount ${SDKCHROOT_DIR}/proc || true
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index 5fba09c3..06f8a6e0 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -48,9 +48,11 @@ SDK_PREINSTALL += " \
     devscripts \
     equivs"
 
+SDK_INCLUDE_ISAR_APT ?= "0"
+
 S = "${WORKDIR}/rootfs"
 
-ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt_dir"
+ROOTFS_CONFIGURE_COMMAND += "${@'rootfs_configure_isar_apt_dir' if d.getVar('SDK_INCLUDE_ISAR_APT') == '1' else ''}"
 rootfs_configure_isar_apt_dir() {
     # Copy isar-apt instead of mounting:
     sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${ROOTFSDIR}/isar-apt
-- 
2.26.2

[PATCH v4 05/13] sdk: Inject sysroot path when calling relocated toolchain

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This removes the need to specify --sysroot=/path/to/sdkroot when calling
gcc or ld from the SDK.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/image-sdk-extension.bbclass         |  8 ++++++++
 .../sdkchroot/files/gcc-sysroot-wrapper.sh       | 16 ++++++++++++++++
 .../sdkchroot/files/relocate-sdk.sh              |  3 +++
 meta/recipes-devtools/sdkchroot/sdkchroot.bb     |  2 ++
 4 files changed, 29 insertions(+)
 create mode 100755 meta/recipes-devtools/sdkchroot/files/gcc-sysroot-wrapper.sh

diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass
index cb4fb55d..a8c708a8 100644
--- a/meta/classes/image-sdk-extension.bbclass
+++ b/meta/classes/image-sdk-extension.bbclass
@@ -40,6 +40,14 @@ do_populate_sdk() {
         fi
     done
 
+    # Set up sysroot wrapper
+    for tool_pattern in "gcc-[0-9]*" "g++-[0-9]*" "cpp-[0-9]*" "ld.bfd" "ld.gold"; do
+        for tool in $(find ${SDKCHROOT_DIR}/usr/bin -type f -name "*-linux-gnu*-${tool_pattern}"); do
+            sudo mv "${tool}" "${tool}.bin"
+            sudo ln -sf gcc-sysroot-wrapper.sh ${tool}
+        done
+    done
+
     # Copy mount_chroot.sh for convenience
     sudo cp ${SCRIPTSDIR}/mount_chroot.sh ${SDKCHROOT_DIR}
 
diff --git a/meta/recipes-devtools/sdkchroot/files/gcc-sysroot-wrapper.sh b/meta/recipes-devtools/sdkchroot/files/gcc-sysroot-wrapper.sh
new file mode 100755
index 00000000..feead1a1
--- /dev/null
+++ b/meta/recipes-devtools/sdkchroot/files/gcc-sysroot-wrapper.sh
@@ -0,0 +1,16 @@
+#!/bin/sh
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2020
+#
+# SPDX-License-Identifier: MIT
+
+GCC_SYSROOT=
+
+NEXT_TARGET=$0
+until [ "${NEXT_TARGET##*/}" = "gcc-sysroot-wrapper.sh" ]; do
+	TARGET=${NEXT_TARGET}
+	NEXT_TARGET=$(dirname ${TARGET})/$(readlink ${TARGET})
+done
+
+${TARGET}.bin --sysroot=${GCC_SYSROOT} "$@"
diff --git a/meta/recipes-devtools/sdkchroot/files/relocate-sdk.sh b/meta/recipes-devtools/sdkchroot/files/relocate-sdk.sh
index 1c9b02fa..0d1c6330 100755
--- a/meta/recipes-devtools/sdkchroot/files/relocate-sdk.sh
+++ b/meta/recipes-devtools/sdkchroot/files/relocate-sdk.sh
@@ -38,4 +38,7 @@ for binary in $(find ${sdkroot}/usr/bin ${sdkroot}/usr/sbin ${sdkroot}/usr/lib/g
 	fi
 done
 
+sed -i 's|^GCC_SYSROOT=.*|GCC_SYSROOT="'"${new_sdkroot}"'"|' \
+    ${sdkroot}/usr/bin/gcc-sysroot-wrapper.sh
+
 echo "done"
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index 06f8a6e0..c96cc772 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -11,6 +11,7 @@ LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260
 SRC_URI = " \
     file://configscript.sh \
     file://relocate-sdk.sh \
+    file://gcc-sysroot-wrapper.sh \
     file://README.sdk"
 PV = "0.1"
 
@@ -63,6 +64,7 @@ sdkchroot_install_files() {
     # Configure root filesystem
     sudo install -m 644 ${WORKDIR}/README.sdk ${S}
     sudo install -m 755 ${WORKDIR}/relocate-sdk.sh ${S}
+    sudo install -m 755 ${WORKDIR}/gcc-sysroot-wrapper.sh ${S}/usr/bin
     sudo install -m 755 ${WORKDIR}/configscript.sh ${S}
     sudo chroot ${S} /configscript.sh  ${DISTRO_ARCH}
 }
-- 
2.26.2

[PATCH v4 06/13] sdk: Update README.sdk

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Document the out-of-chroot invocation and make it the preferred option.
Also remove irrelevant information.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 .../sdkchroot/files/README.sdk                | 71 ++++++++++---------
 1 file changed, 36 insertions(+), 35 deletions(-)

diff --git a/meta/recipes-devtools/sdkchroot/files/README.sdk b/meta/recipes-devtools/sdkchroot/files/README.sdk
index 9c1af6d3..3e06d8c5 100644
--- a/meta/recipes-devtools/sdkchroot/files/README.sdk
+++ b/meta/recipes-devtools/sdkchroot/files/README.sdk
@@ -1,42 +1,43 @@
-Building applications for targets in ISAR takes a lot of time as they are built under QEMU.
-SDK helps to develop applications for target platform in crossbuild environment.
+ISAR Target SDK
+===============
 
-SDK contains cross-toolchain for target architecture and a copy of isar-apt repo with
-locally prebuilt target debian packages.
+This SDK helps to develop applications for an ISAR target platform in a
+crossbuild environment. It contains a cross-toolchain and development packages
+corresponding to the original target.
 
- - First one have to mount the system directories for proper operation in chroot environement.
-Just call supplied with sdk tarball  in  udo rootfs as an argument to the script `mount_chroot.sh`:
+The SDK can be used in two ways, described in the following.
 
-$ sudo mount_chroot.sh  <sdk_rootfs>
 
- - chroot to isar SDK rootfs:
+Option 1 (recommended): Use cross-compiler in host environment
+--------------------------------------------------------------
+
+After unpacking the SDK at the desired location, it has to be relocated once:
+
+$ <sdk_rootfs>/relocate-sdk.sh
+
+Now you can add <sdk_rootfs>/usr/bin to the local path or adjust your project
+to call the cross-compiler from the SDK.
+
+
+Option 2 (fallback): Build inside chroot
+----------------------------------------
+
+First you have to mount the system directories for proper operation into the
+chroot environment. Call the helper script supplied with SDK tarball:
+
+$ sudo <sdk_rootfs>/mount_chroot.sh <sdk_rootfs>
+
+Bind-mount the project into the rootfs:
+
+$ sudo mount -o bind /path/to/project <sdk_rootfs>/mnt
+
+If you have relocated the SDK previously for using option 1, you need to call
+this next:
+
+$ <sdk_rootfs>/relocate-sdk.sh --restore-chroot
+
+Then chroot into the SDK rootfs:
 
 $ sudo chroot <sdk_rootfs>
 
- - Check that cross toolchains are installed
-
-:~# dpkg -l | grep crossbuild-essential-armhf
-ii  crossbuild-essential-armhf           12.3                   all          Informational list of cross-build-essential packages
-
- - Install needed prebuilt target packages.
-
-:~# apt-get update
-:~# apt-get install libhello-dev:armhf
-
- - Check the contents of the installed target package
-
-:~# dpkg -L libhello-dev
-/.
-/usr
-/usr/include
-/usr/include/hello.h
-/usr/lib
-/usr/lib/arm-linux-gnueabihf
-/usr/lib/arm-linux-gnueabihf/libhello.a
-/usr/lib/arm-linux-gnueabihf/libhello.la
-/usr/share
-/usr/share/doc
-/usr/share/doc/libhello-dev
-/usr/share/doc/libhello-dev/changelog.gz
-/usr/share/doc/libhello-dev/copyright
-~#
+Now you can build the project under /mnt.
-- 
2.26.2

[PATCH v4 07/13] isar-bootstrap: Connect to gpg-agent before adding keys

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

We have recurring issues in CI with the gpg-agent not being ready when
trying to use apt-key:

gpg: can't connect to the agent: IPC connect call failed

Work around this by connecting upfront to the agent, retrying that in a
bounded loop.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index fbfe669d..1c5dba74 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -303,10 +303,13 @@ isar_bootstrap() {
                              "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf"
             find ${APT_KEYS_DIR}/ -type f | while read keyfile
             do
-                kfn="$(basename $keyfile)"
+                export kfn="$(basename $keyfile)"
                 cp $keyfile "${ROOTFSDIR}/tmp/$kfn"
-                chroot "${ROOTFSDIR}" /usr/bin/apt-key \
-                   --keyring ${THIRD_PARTY_APT_KEYRING} add "/tmp/$kfn"
+                chroot "${ROOTFSDIR}" sh -c '
+                    for i in $(seq 10); do
+                        /usr/bin/apt-key --keyring ${THIRD_PARTY_APT_KEYRING} add "/tmp/$kfn" && break
+                        sleep 1
+                    done'
                 rm "${ROOTFSDIR}/tmp/$kfn"
             done
 
-- 
2.26.2

[PATCH v4 08/13] buildchroot-host: Drop unneeded :native for riscv64 setup

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This is not a control file, this is a list passed to apt, thus "native"
is implicit.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-devtools/buildchroot/buildchroot-host.bb | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
index 8e3a52cf..d8498a56 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
@@ -19,6 +19,6 @@ BUILDCHROOT_PREINSTALL ?= " \
 BUILDCHROOT_PREINSTALL_riscv64 ?= " \
     ${BUILDCHROOT_PREINSTALL_COMMON} \
     libc6:${DISTRO_ARCH} \
-    gcc-riscv64-linux-gnu:native \
-    g++-riscv64-linux-gnu:native \
-    dpkg-cross:native"
+    gcc-riscv64-linux-gnu \
+    g++-riscv64-linux-gnu \
+    dpkg-cross"
-- 
2.26.2

[PATCH v4 09/13] deb-dl-dir: Fix quoting

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

The outer flock call wraps this in '', so this can't be used inside the
block. The current code failed if some .deb file happened to lie around
in the build folder.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/deb-dl-dir.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index e9967036..54c29947 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -14,8 +14,8 @@ deb_dl_dir_import() {
         set -e
         printenv | grep -q BB_VERBOSE_LOGS && set -x
 
-        sudo find "${pc}" -type f -iname '*\.deb' -exec \
-            cp -n --no-preserve=owner -t "${rootfs}"/var/cache/apt/archives/ '{}' +
+        sudo find "${pc}" -type f -iname "*\.deb" -exec \
+            cp -n --no-preserve=owner -t "${rootfs}"/var/cache/apt/archives/ {} +
     '
 }
 
-- 
2.26.2

[PATCH v4 10/13] Fix dependencies on isar-apt

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Currently, we fail if only running isar-bootstrap and no other task that
needs isar-apt. That's because of deb_dl_dir_export expecting the repos
already being available (even if empty).

Fix by letting the isar-bootstrap task depend also on
isar-apt:do_cache_config. Drop redundant dependencies, also for
base-apt.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass                      | 1 -
 meta/classes/rootfs.bbclass                         | 2 +-
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
 3 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 08880d7d..7e12ab0a 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -180,7 +180,6 @@ do_deploy_deb() {
 
 addtask deploy_deb after do_dpkg_build before do_build
 do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
-do_deploy_deb[depends] = "isar-apt:do_cache_config"
 do_deploy_deb[dirs] = "${S}"
 
 python do_devshell() {
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index afec1cbc..00254ef5 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -151,7 +151,7 @@ rootfs_install_pkgs_install() {
 
 do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}"
 do_rootfs_install[vardeps] += "${ROOTFS_CONFIGURE_COMMAND} ${ROOTFS_INSTALL_COMMAND}"
-do_rootfs_install[depends] = "isar-bootstrap-${@'target' if d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build isar-apt:do_cache_config base-apt:do_cache"
+do_rootfs_install[depends] = "isar-bootstrap-${@'target' if d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build"
 do_rootfs_install[deptask] = "do_deploy_deb"
 python do_rootfs_install() {
     configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) or "").split()
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 1c5dba74..b7dc1ab7 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -224,7 +224,7 @@ def get_host_release():
 
 do_bootstrap[vardeps] += "DISTRO_APT_PREMIRRORS ISAR_ENABLE_COMPAT_ARCH"
 do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP}"
-do_bootstrap[depends] = "base-apt:do_cache"
+do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config"
 
 isar_bootstrap() {
     IS_HOST=""
-- 
2.26.2

[PATCH v4 11/13] dpkg: Account for changes in mk-build-deps in bullseye

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

In bullseye, mk-build-deps now emits two log files. When rebuilding
debian packages, those files end up in ${S} and will be recognized by
the package build as deviation from the source file. Avoid this failure
by stepping out of ${S} before calling mk-build-deps.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-devtools/buildchroot/files/deps.sh | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh b/meta/recipes-devtools/buildchroot/files/deps.sh
index 93bc9cf5..1d617bc8 100644
--- a/meta/recipes-devtools/buildchroot/files/deps.sh
+++ b/meta/recipes-devtools/buildchroot/files/deps.sh
@@ -33,17 +33,20 @@ if ! grep "^Architecture:" debian/control | grep -qv "all"; then
     set_arch=""
 fi
 
+control_file=$(pwd)/debian/control
+cd ..
+
 # Install all build deps
 if [ "$3" = "--download-only" ]; then
     # this will not return 0 even when it worked
-    mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control &> \
+    mk-build-deps $set_arch -t "${install_cmd}" -i -r $control_file &> \
         mk-build-deps.output || true
     cat mk-build-deps.output
     # we assume success when we find this
     grep "mk-build-deps: Unable to install all build-dep packages" mk-build-deps.output
     rm -f mk-build-deps.output
 else
-    mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control
+    mk-build-deps $set_arch -t "${install_cmd}" -i -r $control_file
 
     # Upgrade any already installed packages in case we are partially rebuilding
     apt-get upgrade -y --allow-downgrades
-- 
2.26.2

[PATCH v4 12/13] isar-bootstrap: Align debootstrap calls

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

...both indention-wise as well as regarding quoting.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index b7dc1ab7..0b3ad452 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -265,14 +265,13 @@ isar_bootstrap() {
                                "${@get_distro_suite(d, True)}" \
                                "${ROOTFSDIR}" \
                                "${@get_distro_source(d, True)}"
-
             else
-                 "${DEBOOTSTRAP}" $debootstrap_args \
-                                  --arch="${DISTRO_ARCH}" \
-                                  ${@get_distro_components_argument(d, False)} \
-                                  "${@get_distro_suite(d, False)}" \
-                                  "${ROOTFSDIR}" \
-                                  "${@get_distro_source(d, False)}"
+                ${DEBOOTSTRAP} $debootstrap_args \
+                               --arch="${DISTRO_ARCH}" \
+                               ${@get_distro_components_argument(d, False)} \
+                               "${@get_distro_suite(d, False)}" \
+                               "${ROOTFSDIR}" \
+                               "${@get_distro_source(d, False)}"
             fi
 
             # Install apt config
-- 
2.26.2

[PATCH v4 13/13] u-boot: Deprecate BUILD_DEPENDS in favor of DEBIAN_BUILD_DEPENDS

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Aligns this recipe with deb_debianize. Provide a compat path along with
a warning to ease migration.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 RECIPE-API-CHANGELOG.md                           |  4 ++++
 meta/recipes-bsp/u-boot/files/debian/control.tmpl |  2 +-
 meta/recipes-bsp/u-boot/u-boot-custom.inc         | 10 ++++++++--
 3 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index ac9d8d63..6e40fbcd 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -252,3 +252,7 @@ with ${D} which needs to be filled explicitly in do_install as before.
 ISARROOT variable is now removed from the bitbake environment. It is unset
 after the initial setup. It is replaced with dedicated variables like
 BITBAKEDIR, SCRIPTSDIR and TESTSUITEDIR.
+
+### Deprecate BUILD_DEPENDS in u-boot-custom.inc
+
+Use DEBIAN_BUILD_DEPENDS instead, to align with deb_debianize.
diff --git a/meta/recipes-bsp/u-boot/files/debian/control.tmpl b/meta/recipes-bsp/u-boot/files/debian/control.tmpl
index 0ab3a295..9379be7d 100644
--- a/meta/recipes-bsp/u-boot/files/debian/control.tmpl
+++ b/meta/recipes-bsp/u-boot/files/debian/control.tmpl
@@ -2,7 +2,7 @@ Source: ${PN}
 Section: admin
 Priority: optional
 Standards-Version: 3.9.6
-Build-Depends: ${BUILD_DEPENDS}
+Build-Depends: ${DEBIAN_BUILD_DEPENDS}
 Maintainer: ISAR project <isar-users@googlegroups.com>
 
 Package: u-boot-${MACHINE}
diff --git a/meta/recipes-bsp/u-boot/u-boot-custom.inc b/meta/recipes-bsp/u-boot/u-boot-custom.inc
index 41124fc3..34bf91bf 100644
--- a/meta/recipes-bsp/u-boot/u-boot-custom.inc
+++ b/meta/recipes-bsp/u-boot/u-boot-custom.inc
@@ -16,10 +16,16 @@ inherit dpkg
 
 SRC_URI += "file://debian/"
 
-BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler, git"
+python() {
+    if d.getVar('BUILD_DEPENDS'):
+        bb.warn("u-boot-custom: Deprecated use of BUILD_DEPENDS, please switch to DEBIAN_BUILD_DEPENDS")
+        d.setVar('DEBIAN_BUILD_DEPENDS', d.getVar('BUILD_DEPENDS'))
+}
+
+DEBIAN_BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler, git"
 
 TEMPLATE_FILES = "debian/control.tmpl"
-TEMPLATE_VARS += "MACHINE BUILD_DEPENDS"
+TEMPLATE_VARS += "MACHINE DEBIAN_BUILD_DEPENDS"
 
 U_BOOT_TOOLS_PACKAGE ?= "0"
 
-- 
2.26.2

Re: [PATCH v4 03/13] sdk: Add script to relocate SDK

[Baurzhan Ismagulov]

On Wed, Oct 07, 2020 at 04:01:20PM +0200, Jan Kiszka wrote:
> When run inside the unpacked SDK, this script tunes interp and rpath
> entry in relevant binaries so that the cross conpilation tool can be

Suggest "conpilation" -> "compilation".

With kind regards,
Baurzhan.

Re: [PATCH v4 03/13] sdk: Add script to relocate SDK

[Jan Kiszka]

On 15.10.20 20:50, Baurzhan Ismagulov wrote:
> On Wed, Oct 07, 2020 at 04:01:20PM +0200, Jan Kiszka wrote:
>> When run inside the unpacked SDK, this script tunes interp and rpath
>> entry in relevant binaries so that the cross conpilation tool can be
> 
> Suggest "conpilation" -> "compilation".
> 

Missed to fix this. If you take the patches as-is, please fix up.

Thanks,
Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

[PATCH v5 06/13] sdk: Update README.sdk

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Document the out-of-chroot invocation and make it the preferred option.
Remove irrelevant information about a toolchain check - it's always
there. Append additional information in case the isar-apt is shipped and
devel packages are not pre-installed.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---

Partial update only due to the unclear "arm-linux-gnueabihf-gcc" issue,
re-adding notes on dependency installations from isar-apt when 
applicable.

 .../sdkchroot/files/README.sdk                | 71 ++++++++++---------
 meta/recipes-devtools/sdkchroot/sdkchroot.bb  | 27 +++++++
 2 files changed, 63 insertions(+), 35 deletions(-)

diff --git a/meta/recipes-devtools/sdkchroot/files/README.sdk b/meta/recipes-devtools/sdkchroot/files/README.sdk
index 9c1af6d3..3e06d8c5 100644
--- a/meta/recipes-devtools/sdkchroot/files/README.sdk
+++ b/meta/recipes-devtools/sdkchroot/files/README.sdk
@@ -1,42 +1,43 @@
-Building applications for targets in ISAR takes a lot of time as they are built under QEMU.
-SDK helps to develop applications for target platform in crossbuild environment.
+ISAR Target SDK
+===============
 
-SDK contains cross-toolchain for target architecture and a copy of isar-apt repo with
-locally prebuilt target debian packages.
+This SDK helps to develop applications for an ISAR target platform in a
+crossbuild environment. It contains a cross-toolchain and development packages
+corresponding to the original target.
 
- - First one have to mount the system directories for proper operation in chroot environement.
-Just call supplied with sdk tarball  in  udo rootfs as an argument to the script `mount_chroot.sh`:
+The SDK can be used in two ways, described in the following.
 
-$ sudo mount_chroot.sh  <sdk_rootfs>
 
- - chroot to isar SDK rootfs:
+Option 1 (recommended): Use cross-compiler in host environment
+--------------------------------------------------------------
+
+After unpacking the SDK at the desired location, it has to be relocated once:
+
+$ <sdk_rootfs>/relocate-sdk.sh
+
+Now you can add <sdk_rootfs>/usr/bin to the local path or adjust your project
+to call the cross-compiler from the SDK.
+
+
+Option 2 (fallback): Build inside chroot
+----------------------------------------
+
+First you have to mount the system directories for proper operation into the
+chroot environment. Call the helper script supplied with SDK tarball:
+
+$ sudo <sdk_rootfs>/mount_chroot.sh <sdk_rootfs>
+
+Bind-mount the project into the rootfs:
+
+$ sudo mount -o bind /path/to/project <sdk_rootfs>/mnt
+
+If you have relocated the SDK previously for using option 1, you need to call
+this next:
+
+$ <sdk_rootfs>/relocate-sdk.sh --restore-chroot
+
+Then chroot into the SDK rootfs:
 
 $ sudo chroot <sdk_rootfs>
 
- - Check that cross toolchains are installed
-
-:~# dpkg -l | grep crossbuild-essential-armhf
-ii  crossbuild-essential-armhf           12.3                   all          Informational list of cross-build-essential packages
-
- - Install needed prebuilt target packages.
-
-:~# apt-get update
-:~# apt-get install libhello-dev:armhf
-
- - Check the contents of the installed target package
-
-:~# dpkg -L libhello-dev
-/.
-/usr
-/usr/include
-/usr/include/hello.h
-/usr/lib
-/usr/lib/arm-linux-gnueabihf
-/usr/lib/arm-linux-gnueabihf/libhello.a
-/usr/lib/arm-linux-gnueabihf/libhello.la
-/usr/share
-/usr/share/doc
-/usr/share/doc/libhello-dev
-/usr/share/doc/libhello-dev/changelog.gz
-/usr/share/doc/libhello-dev/copyright
-~#
+Now you can build the project under /mnt.
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index c96cc772..3c299202 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -63,6 +63,33 @@ ROOTFS_POSTPROCESS_COMMAND =+ "sdkchroot_install_files"
 sdkchroot_install_files() {
     # Configure root filesystem
     sudo install -m 644 ${WORKDIR}/README.sdk ${S}
+    if [ "${SDK_INCLUDE_ISAR_APT}" = "1" ]; then
+        sudo sh -c "cat <<EOF >>${S}/README.sdk
+
+In case build dependencies have not been pre-installed, you need to do that
+first, e.g.:
+
+:~# apt-get update
+:~# apt-get install libhello-dev:armhf
+
+Check the contents of the installed package like this:
+
+:~# dpkg -L libhello-dev
+/.
+/usr
+/usr/include
+/usr/include/hello.h
+/usr/lib
+/usr/lib/arm-linux-gnueabihf
+/usr/lib/arm-linux-gnueabihf/libhello.a
+/usr/lib/arm-linux-gnueabihf/libhello.la
+/usr/share
+/usr/share/doc
+/usr/share/doc/libhello-dev
+/usr/share/doc/libhello-dev/changelog.gz
+/usr/share/doc/libhello-dev/copyright
+EOF"
+    fi
     sudo install -m 755 ${WORKDIR}/relocate-sdk.sh ${S}
     sudo install -m 755 ${WORKDIR}/gcc-sysroot-wrapper.sh ${S}/usr/bin
     sudo install -m 755 ${WORKDIR}/configscript.sh ${S}
-- 
2.26.2

Re: [PATCH v4 00/13] Complete backlog: SDK, assorting fixed and cleanups

[Baurzhan Ismagulov]

On Wed, Oct 07, 2020 at 04:01:17PM +0200, Jan Kiszka wrote:
> Rebased queue of my readily pending patches over next.

Applied 8, 9, 11, 12 to next, thanks.

With kind regards,
Baurzhan.

Re: [PATCH v4 07/13] isar-bootstrap: Connect to gpg-agent before adding keys

[Jan Kiszka]

On 07.10.20 16:01, [ext] Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> We have recurring issues in CI with the gpg-agent not being ready when
> trying to use apt-key:
> 
> gpg: can't connect to the agent: IPC connect call failed
> 
> Work around this by connecting upfront to the agent, retrying that in a
> bounded loop.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index fbfe669d..1c5dba74 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -303,10 +303,13 @@ isar_bootstrap() {
>                               "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf"
>              find ${APT_KEYS_DIR}/ -type f | while read keyfile
>              do
> -                kfn="$(basename $keyfile)"
> +                export kfn="$(basename $keyfile)"
>                  cp $keyfile "${ROOTFSDIR}/tmp/$kfn"
> -                chroot "${ROOTFSDIR}" /usr/bin/apt-key \
> -                   --keyring ${THIRD_PARTY_APT_KEYRING} add "/tmp/$kfn"
> +                chroot "${ROOTFSDIR}" sh -c '
> +                    for i in $(seq 10); do
> +                        /usr/bin/apt-key --keyring ${THIRD_PARTY_APT_KEYRING} add "/tmp/$kfn" && break
> +                        sleep 1
> +                    done'
>                  rm "${ROOTFSDIR}/tmp/$kfn"
>              done
>  
> 

No more gpg-related CI failures since this has been applied to our
layers. You said you have an even better version, but that wasn't even
shared so far.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [PATCH v4 00/13] Complete backlog: SDK, assorting fixed and cleanups

[Anton Mikanovich]

Applied 10 to next, thanks.

07.10.2020 17:01, Jan Kiszka пишет:
> Rebased queue of my readily pending patches over next.
>
> Changes:
> - fixed "sdk: Make shipping the isar-apt repo opt-in"
> - added "u-boot: Deprecate BUILD_DEPENDS..."
>
> Local backlog also contains a new board (STM32MP15x) and TF-A and OP-TEE
> support, but the test case for the latter is not yet working. Hope that
> this will follow soon once I received the needed information to fix it.
>
> Jan
>
> Jan Kiszka (13):
>    sdk: Add support for adding self-defined sdk packages
>    sdk: Make all links in the SDK chroot relative
>    sdk: Add script to relocate SDK
>    sdk: Make shipping the isar-apt repo opt-in
>    sdk: Inject sysroot path when calling relocated toolchain
>    sdk: Update README.sdk
>    isar-bootstrap: Connect to gpg-agent before adding keys
>    buildchroot-host: Drop unneeded :native for riscv64 setup
>    deb-dl-dir: Fix quoting
>    Fix dependencies on isar-apt
>    dpkg: Account for changes in mk-build-deps in bullseye
>    isar-bootstrap: Align debootstrap calls
>    u-boot: Deprecate BUILD_DEPENDS in favor of DEBIAN_BUILD_DEPENDS
>
>   RECIPE-API-CHANGELOG.md                       |  4 ++
>   doc/user_manual.md                            |  1 +
>   meta/classes/deb-dl-dir.bbclass               |  4 +-
>   meta/classes/dpkg-base.bbclass                |  1 -
>   meta/classes/image-sdk-extension.bbclass      | 34 ++++++++-
>   meta/classes/rootfs.bbclass                   |  2 +-
>   .../u-boot/files/debian/control.tmpl          |  2 +-
>   meta/recipes-bsp/u-boot/u-boot-custom.inc     | 10 ++-
>   .../isar-bootstrap/isar-bootstrap.inc         | 24 ++++---
>   .../buildchroot/buildchroot-host.bb           |  6 +-
>   .../buildchroot/files/deps.sh                 |  7 +-
>   .../sdkchroot/files/README.sdk                | 71 ++++++++++---------
>   .../sdkchroot/files/gcc-sysroot-wrapper.sh    | 16 +++++
>   .../sdkchroot/files/relocate-sdk.sh           | 44 ++++++++++++
>   meta/recipes-devtools/sdkchroot/sdkchroot.bb  | 33 ++++++---
>   15 files changed, 188 insertions(+), 71 deletions(-)
>   create mode 100755 meta/recipes-devtools/sdkchroot/files/gcc-sysroot-wrapper.sh
>   create mode 100755 meta/recipes-devtools/sdkchroot/files/relocate-sdk.sh
>
-- 
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH v4 07/13] isar-bootstrap: Connect to gpg-agent before adding keys

[Jan Kiszka]

On 21.11.20 08:41, [ext] Jan Kiszka wrote:
> On 07.10.20 16:01, [ext] Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> We have recurring issues in CI with the gpg-agent not being ready when
>> trying to use apt-key:
>>
>> gpg: can't connect to the agent: IPC connect call failed
>>
>> Work around this by connecting upfront to the agent, retrying that in a
>> bounded loop.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 9 ++++++---
>>  1 file changed, 6 insertions(+), 3 deletions(-)
>>
>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>> index fbfe669d..1c5dba74 100644
>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>> @@ -303,10 +303,13 @@ isar_bootstrap() {
>>                               "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf"
>>              find ${APT_KEYS_DIR}/ -type f | while read keyfile
>>              do
>> -                kfn="$(basename $keyfile)"
>> +                export kfn="$(basename $keyfile)"
>>                  cp $keyfile "${ROOTFSDIR}/tmp/$kfn"
>> -                chroot "${ROOTFSDIR}" /usr/bin/apt-key \
>> -                   --keyring ${THIRD_PARTY_APT_KEYRING} add "/tmp/$kfn"
>> +                chroot "${ROOTFSDIR}" sh -c '
>> +                    for i in $(seq 10); do
>> +                        /usr/bin/apt-key --keyring ${THIRD_PARTY_APT_KEYRING} add "/tmp/$kfn" && break
>> +                        sleep 1
>> +                    done'
>>                  rm "${ROOTFSDIR}/tmp/$kfn"
>>              done
>>  
>>
> 
> No more gpg-related CI failures since this has been applied to our
> layers. You said you have an even better version, but that wasn't even
> shared so far.
> 

Yet another ping on this.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [PATCH v4 07/13] isar-bootstrap: Connect to gpg-agent before adding keys

[Baurzhan Ismagulov]

On Sat, Nov 21, 2020 at 08:41:57AM +0100, Jan Kiszka wrote:
> No more gpg-related CI failures since this has been applied to our
> layers. You said you have an even better version, but that wasn't even
> shared so far.

I haven't seen this after migrating to a faster server, sorry for the delay.
I'll share the dirty version and send v2 after cleaning up.

With kind regards,
Baurzhan.

[PATCH v1] isar-bootstrap: Run gpg-agent before starting apt-key

[Baurzhan Ismagulov]

From: Yuri Adamov <yadamov@ilbers.de>

Building rpi-stretch natively (under qemu) sometimes fails with:

gpg: can't connect to the agent: IPC connect call failed

gpg starts gpg-agent and times out after 5 s. This value is hard-coded.

Besides, leaving running gpg-agent processes is not clean and prevents
unmounting of filesystems.

This patch starts and stops the agent manually.

Signed-off-by: Yuri Adamov <yadamov@ilbers.de>
---

Notes:
    * Submitting WIP for preview, as cleaning up will require testing time.
    * Remove sleeping.
    * Remove -9 in kill.
    * Maybe check if starting the agent is necessary.
    * Remove OVERRIDES_append and get_distro_needs_gpg_support() if unused.

 .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 4925a45d..74569e5d 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -24,7 +24,7 @@ DISTRO_BOOTSTRAP_KEYFILES = ""
 THIRD_PARTY_APT_KEYFILES = ""
 DEPLOY_ISAR_BOOTSTRAP ?= ""
 DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
-DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg"
+DISTRO_BOOTSTRAP_BASE_PACKAGES_append = ",gnupg"
 DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "${@https_support(d)}"
 
 inherit deb-dl-dir
@@ -307,14 +307,24 @@ isar_bootstrap() {
             mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"
             install -v -m644 "${WORKDIR}/isar-apt.conf" \
                              "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf"
+            MY_GPGHOME=$(chroot "${ROOTFSDIR}" mktemp -d /tmp/gpghomeXXXXXXXXXX)
+            echo "Created temporary directory ${MY_GPGHOME} for gpg-agent"
+            chroot "${ROOTFSDIR}" gpg-agent --homedir "${MY_GPGHOME}" --daemon
             find ${APT_KEYS_DIR}/ -type f | while read keyfile
             do
                 kfn="$(basename $keyfile)"
                 cp $keyfile "${ROOTFSDIR}/tmp/$kfn"
                 chroot "${ROOTFSDIR}" /usr/bin/apt-key \
-                   --keyring ${THIRD_PARTY_APT_KEYRING} add "/tmp/$kfn"
+                    --keyring ${THIRD_PARTY_APT_KEYRING} \
+                    --homedir ${MY_GPGHOME} add "/tmp/$kfn"
                 rm "${ROOTFSDIR}/tmp/$kfn"
             done
+            sleep 4
+            GPG_AGENT_PID=$(ps -aux | grep "gpg-agent.*${MY_GPGHOME}" | grep -v grep | awk  '{print $2}')
+            echo "Killing gpg-agent with pid $GPG_AGENT_PID"
+            /bin/kill -9 ${GPG_AGENT_PID}
+            sleep 4
+            chroot "${ROOTFSDIR}" /bin/rm -rf "${MY_GPGHOME}"
 
             if [ "${@get_distro_suite(d, True)}" = "stretch" ] && [ "${@get_host_release().split('.')[0]}" -lt "4" ]; then
                 install -v -m644 "${WORKDIR}/isar-apt-fallback.conf" \
-- 
2.20.1

Re: [PATCH v1] isar-bootstrap: Run gpg-agent before starting apt-key

[Jan Kiszka]

On 16.12.20 16:53, Baurzhan Ismagulov wrote:
> From: Yuri Adamov <yadamov@ilbers.de>
> 
> Building rpi-stretch natively (under qemu) sometimes fails with:
> 
> gpg: can't connect to the agent: IPC connect call failed
> 
> gpg starts gpg-agent and times out after 5 s. This value is hard-coded.
> 

This is not limited to stretch or rpi. We were seeing this with buster
builds on our CI systems as well - likely when they were overloaded.

> Besides, leaving running gpg-agent processes is not clean and prevents
> unmounting of filesystems.
> 
> This patch starts and stops the agent manually.
> 
> Signed-off-by: Yuri Adamov <yadamov@ilbers.de>
> ---
> 
> Notes:
>     * Submitting WIP for preview, as cleaning up will require testing time.
>     * Remove sleeping.

Yep, that would be good.

>     * Remove -9 in kill.
>     * Maybe check if starting the agent is necessary.
>     * Remove OVERRIDES_append and get_distro_needs_gpg_support() if unused.

That last two points I was wondering as well: Why do we need to make it
unconditionally now? That should at least be explain - or fixed.

> 
>  .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 14 ++++++++++++--
>  1 file changed, 12 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index 4925a45d..74569e5d 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -24,7 +24,7 @@ DISTRO_BOOTSTRAP_KEYFILES = ""
>  THIRD_PARTY_APT_KEYFILES = ""
>  DEPLOY_ISAR_BOOTSTRAP ?= ""
>  DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
> -DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg"
> +DISTRO_BOOTSTRAP_BASE_PACKAGES_append = ",gnupg"
>  DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "${@https_support(d)}"
>  
>  inherit deb-dl-dir
> @@ -307,14 +307,24 @@ isar_bootstrap() {
>              mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"
>              install -v -m644 "${WORKDIR}/isar-apt.conf" \
>                               "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf"
> +            MY_GPGHOME=$(chroot "${ROOTFSDIR}" mktemp -d /tmp/gpghomeXXXXXXXXXX)
> +            echo "Created temporary directory ${MY_GPGHOME} for gpg-agent"
> +            chroot "${ROOTFSDIR}" gpg-agent --homedir "${MY_GPGHOME}" --daemon
>              find ${APT_KEYS_DIR}/ -type f | while read keyfile
>              do
>                  kfn="$(basename $keyfile)"
>                  cp $keyfile "${ROOTFSDIR}/tmp/$kfn"
>                  chroot "${ROOTFSDIR}" /usr/bin/apt-key \
> -                   --keyring ${THIRD_PARTY_APT_KEYRING} add "/tmp/$kfn"
> +                    --keyring ${THIRD_PARTY_APT_KEYRING} \
> +                    --homedir ${MY_GPGHOME} add "/tmp/$kfn"
>                  rm "${ROOTFSDIR}/tmp/$kfn"
>              done
> +            sleep 4
> +            GPG_AGENT_PID=$(ps -aux | grep "gpg-agent.*${MY_GPGHOME}" | grep -v grep | awk  '{print $2}')
> +            echo "Killing gpg-agent with pid $GPG_AGENT_PID"
> +            /bin/kill -9 ${GPG_AGENT_PID}
> +            sleep 4
> +            chroot "${ROOTFSDIR}" /bin/rm -rf "${MY_GPGHOME}"
>  
>              if [ "${@get_distro_suite(d, True)}" = "stretch" ] && [ "${@get_host_release().split('.')[0]}" -lt "4" ]; then
>                  install -v -m644 "${WORKDIR}/isar-apt-fallback.conf" \
> 

I do like the approach of controlling gpg's lifecycle. As you said, some
cleanup is needed, but I'm all for going this direction.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [PATCH v1] isar-bootstrap: Run gpg-agent before starting apt-key

[Jan Kiszka]

On 16.12.20 17:41, [ext] Jan Kiszka wrote:
> On 16.12.20 16:53, Baurzhan Ismagulov wrote:
>> From: Yuri Adamov <yadamov@ilbers.de>
>>
>> Building rpi-stretch natively (under qemu) sometimes fails with:
>>
>> gpg: can't connect to the agent: IPC connect call failed
>>
>> gpg starts gpg-agent and times out after 5 s. This value is hard-coded.
>>
> 
> This is not limited to stretch or rpi. We were seeing this with buster
> builds on our CI systems as well - likely when they were overloaded.
> 
>> Besides, leaving running gpg-agent processes is not clean and prevents
>> unmounting of filesystems.
>>
>> This patch starts and stops the agent manually.
>>
>> Signed-off-by: Yuri Adamov <yadamov@ilbers.de>
>> ---
>>
>> Notes:
>>     * Submitting WIP for preview, as cleaning up will require testing time.
>>     * Remove sleeping.
> 
> Yep, that would be good.
> 
>>     * Remove -9 in kill.
>>     * Maybe check if starting the agent is necessary.
>>     * Remove OVERRIDES_append and get_distro_needs_gpg_support() if unused.
> 
> That last two points I was wondering as well: Why do we need to make it
> unconditionally now? That should at least be explain - or fixed.
> 
>>
>>  .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 14 ++++++++++++--
>>  1 file changed, 12 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>> index 4925a45d..74569e5d 100644
>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>> @@ -24,7 +24,7 @@ DISTRO_BOOTSTRAP_KEYFILES = ""
>>  THIRD_PARTY_APT_KEYFILES = ""
>>  DEPLOY_ISAR_BOOTSTRAP ?= ""
>>  DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
>> -DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg"
>> +DISTRO_BOOTSTRAP_BASE_PACKAGES_append = ",gnupg"
>>  DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "${@https_support(d)}"
>>  
>>  inherit deb-dl-dir
>> @@ -307,14 +307,24 @@ isar_bootstrap() {
>>              mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"
>>              install -v -m644 "${WORKDIR}/isar-apt.conf" \
>>                               "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf"
>> +            MY_GPGHOME=$(chroot "${ROOTFSDIR}" mktemp -d /tmp/gpghomeXXXXXXXXXX)
>> +            echo "Created temporary directory ${MY_GPGHOME} for gpg-agent"
>> +            chroot "${ROOTFSDIR}" gpg-agent --homedir "${MY_GPGHOME}" --daemon
>>              find ${APT_KEYS_DIR}/ -type f | while read keyfile
>>              do
>>                  kfn="$(basename $keyfile)"
>>                  cp $keyfile "${ROOTFSDIR}/tmp/$kfn"
>>                  chroot "${ROOTFSDIR}" /usr/bin/apt-key \
>> -                   --keyring ${THIRD_PARTY_APT_KEYRING} add "/tmp/$kfn"
>> +                    --keyring ${THIRD_PARTY_APT_KEYRING} \
>> +                    --homedir ${MY_GPGHOME} add "/tmp/$kfn"
>>                  rm "${ROOTFSDIR}/tmp/$kfn"
>>              done
>> +            sleep 4
>> +            GPG_AGENT_PID=$(ps -aux | grep "gpg-agent.*${MY_GPGHOME}" | grep -v grep | awk  '{print $2}')
>> +            echo "Killing gpg-agent with pid $GPG_AGENT_PID"
>> +            /bin/kill -9 ${GPG_AGENT_PID}
>> +            sleep 4
>> +            chroot "${ROOTFSDIR}" /bin/rm -rf "${MY_GPGHOME}"
>>  
>>              if [ "${@get_distro_suite(d, True)}" = "stretch" ] && [ "${@get_host_release().split('.')[0]}" -lt "4" ]; then
>>                  install -v -m644 "${WORKDIR}/isar-apt-fallback.conf" \
>>
> 
> I do like the approach of controlling gpg's lifecycle. As you said, some
> cleanup is needed, but I'm all for going this direction.
> 
> Jan
> 

Any news on this?

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux