[PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Jan Kiszka]

This is primarily a repost of most of my pending patches in their tested
order on top of current ibr/devel (I suspect that this is the next chunk
to be merged). As the highest version number of a patch included here
was v4, the series is tagged as v5.

I think this is usefull because I had to inject the isar-bootstrap-host
sharing fix into the middle and adjust a succeeding patch.

As a bonus, this series ports devshell support over from OE core.

Jan

Jan Kiszka (27):
  buildchroot: Properly deploy the build result
  sdk: Use clean-package-cache rather than open-coded cleanup
  sdk: Deploy earlier
  Remove unneeded PF tweakings
  image: Stop changing PF, tune WORKDIR and STAMP directly
  image: Make WORKDIR and STAMPs IMAGE_TYPE-specific
  Adjust STAMP variables to use PN, rather than PF
  Align WORKDIR structure with OE
  isar-bootstrap: Unshare host bootstraps across different target
    distros
  isar-bootstrap: Clean up and fix rebuild
  gitlab-ci: Keep logs as artifacts on failure
  Detect false sharing of recipes
  dpkg-base: Move do_deploy_deb before do_build
  base-apt: Move do_cache_config before do_build
  wic-img: Use private tmp dir
  rootfs: Take isar-apt shared lock during repository operations
  Use deb.debian.org mirrors
  linux-custom: Control linux-libc-dev deployment manually
  dpkg-base: Wait for umount to succeed
  dpkg-base: Permit multiple clean runs
  buildchroot: Factor out common packages
  buildchroot: Do not build cross when there are no arch-specific
    outputs
  Add git-buildpackage support
  dbpg-raw: Only list top-level files in <pn>.install
  Add OE class and library elements for terminal support
  dpkg-base: Add devshell target
  linux-custom: Add libncurses-dev to KBUILD_DEPENDS

 .gitlab-ci.yml                                     |   6 +
 doc/user_manual.md                                 |  10 +-
 meta-isar/recipes-app/cowsay/cowsay_git.bb         |   9 +
 meta/classes/buildchroot.bbclass                   |   2 +-
 meta/classes/dpkg-base.bbclass                     |  36 ++-
 meta/classes/dpkg-gbp.bbclass                      |  24 ++
 meta/classes/dpkg-raw.bbclass                      |   3 +-
 meta/classes/dpkg.bbclass                          |   2 +
 meta/classes/image-sdk-extension.bbclass           |  23 +-
 meta/classes/image.bbclass                         |   5 +-
 meta/classes/isar-events.bbclass                   |  22 ++
 meta/classes/rootfs.bbclass                        |  12 +-
 meta/classes/terminal.bbclass                      | 112 ++++++++
 meta/classes/wic-img.bbclass                       |   7 +-
 meta/conf/bitbake.conf                             |  13 +-
 meta/conf/distro/debian-bullseye.list              |  14 +-
 meta/conf/distro/debian-buster.list                |  14 +-
 meta/conf/distro/debian-stretch.list               |  14 +-
 meta/lib/oe/classutils.py                          |  49 ++++
 meta/lib/oe/data.py                                |  53 ++++
 meta/lib/oe/maketype.py                            | 112 ++++++++
 meta/lib/oe/terminal.py                            | 315 +++++++++++++++++++++
 meta/lib/oe/types.py                               | 187 ++++++++++++
 .../isar-bootstrap/isar-bootstrap-host.bb          |  33 +--
 .../isar-bootstrap/isar-bootstrap-target.bb        |  28 +-
 .../recipes-core/isar-bootstrap/isar-bootstrap.inc |   5 -
 meta/recipes-devtools/base-apt/base-apt.bb         |   2 +-
 .../buildchroot/buildchroot-host.bb                |  19 +-
 .../buildchroot/buildchroot-target.bb              |  20 +-
 meta/recipes-devtools/buildchroot/buildchroot.inc  |  26 ++
 meta/recipes-devtools/buildchroot/files/build.sh   |   2 +-
 meta/recipes-devtools/buildchroot/files/deps.sh    |   6 +
 meta/recipes-devtools/sdkchroot/sdkchroot.bb       |  14 +-
 meta/recipes-kernel/linux/files/build-kernel.sh    |   7 -
 meta/recipes-kernel/linux/linux-custom.inc         |  21 +-
 scripts/ci_build.sh                                |   2 +
 scripts/oe-gnome-terminal-phonehome                |  14 +
 37 files changed, 1113 insertions(+), 130 deletions(-)
 create mode 100644 meta-isar/recipes-app/cowsay/cowsay_git.bb
 create mode 100644 meta/classes/dpkg-gbp.bbclass
 create mode 100644 meta/classes/terminal.bbclass
 create mode 100644 meta/lib/oe/classutils.py
 create mode 100644 meta/lib/oe/data.py
 create mode 100644 meta/lib/oe/maketype.py
 create mode 100644 meta/lib/oe/terminal.py
 create mode 100644 meta/lib/oe/types.py
 create mode 100755 scripts/oe-gnome-terminal-phonehome

-- 
2.16.4

[PATCH v5 01/27] buildchroot: Properly deploy the build result

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This follows the pattern used for isar-bootstrap already: Establish a
stable link to the rootfs in the workdir from a folder in DEPLOY_DIR.
This is cleaner than pointing directly into the workdir via
BUILDCHROOT_HOST/TARGET_DIR, and it avoid having to mess with PF.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/buildchroot.bbclass                        |  2 +-
 meta/conf/bitbake.conf                                  |  5 +++--
 meta/recipes-devtools/buildchroot/buildchroot-host.bb   |  2 ++
 meta/recipes-devtools/buildchroot/buildchroot-target.bb |  2 ++
 meta/recipes-devtools/buildchroot/buildchroot.inc       | 13 +++++++++++++
 5 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index b48b72c..64ac450 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -21,7 +21,7 @@ python __anonymous() {
     d.setVar('BUILDCHROOT_DIR', rootfs)
 }
 
-MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock"
+MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}.lock"
 
 buildchroot_do_mounts() {
     sudo -s <<'EOSUDO'
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 1b9f92b..f25b20f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -47,11 +47,12 @@ GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
 # isar specific config
 WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}"
 DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"
+DEPLOY_DIR_BUILDCHROOT = "${DEPLOY_DIR}/buildchroot"
 DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MACHINE}"
 DL_DIR ?= "${TOPDIR}/downloads"
 SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
-BUILDCHROOT_HOST_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-host-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}/rootfs"
-BUILDCHROOT_TARGET_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-target-${DISTRO}-${DISTRO_ARCH}/rootfs"
+BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"
 CACHE = "${TMPDIR}/cache"
 
 OVERRIDES = "${DISTRO_ARCH}:${MACHINE}:${DISTRO}:forcevariable"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
index 121992c..40a37ea 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
@@ -6,6 +6,8 @@
 DESCRIPTION = "Isar development filesystem for host"
 PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
 
+BUILDCHROOT_VARIANT = "host"
+
 require buildchroot.inc
 ROOTFS_ARCH = "${HOST_ARCH}"
 ROOTFS_DISTRO = "${HOST_DISTRO}"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-target.bb b/meta/recipes-devtools/buildchroot/buildchroot-target.bb
index 5a01258..3b96977 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-target.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-target.bb
@@ -6,6 +6,8 @@
 DESCRIPTION = "Isar development filesystem for target"
 PF = "${PN}-${DISTRO}-${DISTRO_ARCH}"
 
+BUILDCHROOT_VARIANT = "target"
+
 require buildchroot.inc
 
 BUILDCHROOT_PREINSTALL ?= "gcc \
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index 40fc8b0..e704137 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -46,3 +46,16 @@ buildchroot_install_files() {
     sudo chroot ${BUILDCHROOT_DIR} /configscript.sh $USER_ID $GROUP_ID
 
 }
+
+DEPLOY_BUILDCHROOT = "${@d.getVar('BUILDCHROOT_' + d.getVar('BUILDCHROOT_VARIANT').upper() + '_DIR')}"
+
+do_buildchroot_deploy[dirs] = "${DEPLOY_DIR_BUILDCHROOT}-${BUILDCHROOT_VARIANT}"
+do_buildchroot_deploy() {
+    ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_BUILDCHROOT}"
+}
+addtask buildchroot_deploy before do_build after do_rootfs
+
+CLEANFUNCS = "clean_deploy"
+clean_deploy() {
+    rm -f "${DEPLOY_BUILDCHROOT}"
+}
-- 
2.16.4

[PATCH v5 02/27] sdk: Use clean-package-cache rather than open-coded cleanup

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

There is nothing that the sdkchroot will gain after its bootstrap.
Therefore we can clean up the apt cache already there, reusing common
infrastructure.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/image-sdk-extension.bbclass     | 3 ---
 meta/recipes-devtools/sdkchroot/sdkchroot.bb | 2 +-
 2 files changed, 1 insertion(+), 4 deletions(-)

diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass
index 8df3ed1..a53d954 100644
--- a/meta/classes/image-sdk-extension.bbclass
+++ b/meta/classes/image-sdk-extension.bbclass
@@ -13,9 +13,6 @@ do_populate_sdk() {
     # Copy isar-apt with deployed Isar packages
     sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO}  ${SDKCHROOT_DIR}/rootfs/isar-apt
 
-    # Purge apt cache to make image slimmer
-    sudo rm -rf ${SDKCHROOT_DIR}/rootfs/var/cache/apt/*
-
     sudo umount -R ${SDKCHROOT_DIR}/rootfs/dev || true
     sudo umount ${SDKCHROOT_DIR}/rootfs/proc || true
     sudo umount -R ${SDKCHROOT_DIR}/rootfs/sys || true
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index 3cfe05c..d2cc92e 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -23,7 +23,7 @@ ROOTFS_ARCH = "${HOST_ARCH}"
 ROOTFS_DISTRO = "${HOST_DISTRO}"
 ROOTFSDIR = "${S}"
 ROOTFS_PACKAGES = "${SDKCHROOT_PREINSTALL} ${TOOLCHAIN}"
-ROOTFS_FEATURES += ""
+ROOTFS_FEATURES += "clean-package-cache"
 
 python() {
     if d.getVar("HOST_ARCH") not in ['i386', 'amd64']:
-- 
2.16.4

[PATCH v5 03/27] sdk: Deploy earlier

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This avoids having to point from image-sdk-extension directly into the
sdkchroot target workdir. It also aligns the sdk with isar-bootstrap and
buildchroot in this regard.

As we now have a deployment target of the sdk rootfs (SDKCHROOT_DIR), we
can drop the previous link in DEPLOY_DIR_IMAGE.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/image-sdk-extension.bbclass     | 20 ++++++++------------
 meta/conf/bitbake.conf                       |  2 ++
 meta/recipes-devtools/sdkchroot/sdkchroot.bb | 11 +++++++++++
 3 files changed, 21 insertions(+), 12 deletions(-)

diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass
index a53d954..b9f2cf4 100644
--- a/meta/classes/image-sdk-extension.bbclass
+++ b/meta/classes/image-sdk-extension.bbclass
@@ -5,29 +5,25 @@
 #
 # This class extends the image.bbclass to supply the creation of a sdk
 
-SDKCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
-
 do_populate_sdk[stamp-extra-info] = "${DISTRO}-${MACHINE}"
 do_populate_sdk[depends] = "sdkchroot:do_build"
 do_populate_sdk() {
     # Copy isar-apt with deployed Isar packages
-    sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO}  ${SDKCHROOT_DIR}/rootfs/isar-apt
+    sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO}  ${SDKCHROOT_DIR}/isar-apt
 
-    sudo umount -R ${SDKCHROOT_DIR}/rootfs/dev || true
-    sudo umount ${SDKCHROOT_DIR}/rootfs/proc || true
-    sudo umount -R ${SDKCHROOT_DIR}/rootfs/sys || true
+    sudo umount -R ${SDKCHROOT_DIR}/dev || true
+    sudo umount ${SDKCHROOT_DIR}/proc || true
+    sudo umount -R ${SDKCHROOT_DIR}/sys || true
 
     # Remove setup scripts
-    sudo rm -f ${SDKCHROOT_DIR}/rootfs/chroot-setup.sh ${SDKCHROOT_DIR}/rootfs/configscript.sh
+    sudo rm -f ${SDKCHROOT_DIR}/chroot-setup.sh ${SDKCHROOT_DIR}/configscript.sh
 
     # Copy mount_chroot.sh for convenience
-    sudo cp ${ISARROOT}/scripts/mount_chroot.sh ${SDKCHROOT_DIR}/rootfs
+    sudo cp ${ISARROOT}/scripts/mount_chroot.sh ${SDKCHROOT_DIR}
 
     # Create SDK archive
-    sudo tar -C ${SDKCHROOT_DIR} --transform="s|^rootfs|sdk-${DISTRO}-${DISTRO_ARCH}|" \
+    cd -P ${SDKCHROOT_DIR}/..
+    sudo tar --transform="s|^rootfs|sdk-${DISTRO}-${DISTRO_ARCH}|" \
         -c rootfs | xz -T0 > ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz
-
-    # Install deployment link for local use
-    ln -Tfsr ${SDKCHROOT_DIR}/rootfs ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}
 }
 addtask populate_sdk after do_rootfs
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index f25b20f..e5681bf 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -48,11 +48,13 @@ GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
 WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}"
 DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"
 DEPLOY_DIR_BUILDCHROOT = "${DEPLOY_DIR}/buildchroot"
+DEPLOY_DIR_SDKCHROOT = "${DEPLOY_DIR}/sdkchroot"
 DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MACHINE}"
 DL_DIR ?= "${TOPDIR}/downloads"
 SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
 BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
 BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"
+SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${DISTRO}-${DISTRO_ARCH}"
 CACHE = "${TMPDIR}/cache"
 
 OVERRIDES = "${DISTRO_ARCH}:${MACHINE}:${DISTRO}:forcevariable"
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index d2cc92e..84b8808 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -56,3 +56,14 @@ sdkchroot_install_files() {
     sudo install -m 755 ${WORKDIR}/configscript.sh ${S}
     sudo chroot ${S} /configscript.sh  ${DISTRO_ARCH}
 }
+
+do_sdkchroot_deploy[dirs] = "${DEPLOY_DIR_SDKCHROOT}"
+do_sdkchroot_deploy() {
+    ln -Tfsr "${ROOTFSDIR}" "${SDKCHROOT_DIR}"
+}
+addtask sdkchroot_deploy before do_build after do_rootfs
+
+CLEANFUNCS = "clean_deploy"
+clean_deploy() {
+    rm -f "${SDKCHROOT_DIR}"
+}
-- 
2.16.4

[PATCH v5 04/27] Remove unneeded PF tweakings

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

All these recipes now have unique WORKDIRs and STAMPs, and all deploy
their shared artifacts via DEPLOY_DIR.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb   | 1 -
 meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb | 1 -
 meta/recipes-devtools/buildchroot/buildchroot-host.bb     | 1 -
 meta/recipes-devtools/buildchroot/buildchroot-target.bb   | 1 -
 meta/recipes-devtools/sdkchroot/sdkchroot.bb              | 1 -
 5 files changed, 5 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index 8363465..b2dded5 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -6,7 +6,6 @@
 # SPDX-License-Identifier: MIT
 
 Description = "Minimal host Debian root file system"
-PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
 
 DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
 ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}.lock"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
index 77276e9..d7fcdbf 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
@@ -6,7 +6,6 @@
 # SPDX-License-Identifier: MIT
 
 Description = "Minimal target Debian root file system"
-PF = "${PN}-${DISTRO}-${DISTRO_ARCH}"
 
 DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}"
 ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}.lock"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
index 40a37ea..795b77a 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
@@ -4,7 +4,6 @@
 # Copyright (C) 2015-2018 ilbers GmbH
 
 DESCRIPTION = "Isar development filesystem for host"
-PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
 
 BUILDCHROOT_VARIANT = "host"
 
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-target.bb b/meta/recipes-devtools/buildchroot/buildchroot-target.bb
index 3b96977..68aedaa 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-target.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-target.bb
@@ -4,7 +4,6 @@
 # Copyright (C) 2015-2018 ilbers GmbH
 
 DESCRIPTION = "Isar development filesystem for target"
-PF = "${PN}-${DISTRO}-${DISTRO_ARCH}"
 
 BUILDCHROOT_VARIANT = "target"
 
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index 84b8808..e658122 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -12,7 +12,6 @@ SRC_URI = " \
     file://configscript.sh \
     file://README.sdk"
 PV = "0.1"
-PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
 
 TOOLCHAIN = "crossbuild-essential-${DISTRO_ARCH}"
 TOOLCHAIN_${HOST_ARCH} = "build-essential"
-- 
2.16.4

[PATCH v5 05/27] image: Stop changing PF, tune WORKDIR and STAMP directly

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This removes the last fiddling with PF, addressing the issue - a common
image PN name for multiconfig targets while the results are actually
machine-specific - by tuning the critical paths directly. This are just
WORKDIR as well as STAMP and STAMPCLEAN.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/image.bbclass | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index ec6bd39..36e9875 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -1,7 +1,10 @@
 # This software is a part of ISAR.
 # Copyright (C) 2015-2017 ilbers GmbH
 
-PF = "${PN}-${DISTRO}-${MACHINE}"
+# Make workdir and stamps machine-specific without changing common PN target
+WORKDIR .= "-${MACHINE}"
+STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/${PV}-${PR}"
+STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/*-*"
 
 IMAGE_INSTALL ?= ""
 IMAGE_TYPE    ?= "ext4-img"
-- 
2.16.4

[PATCH v6 05/27] image: Stop changing PF, tune WORKDIR and STAMP directly

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This removes the last fiddling with PF, addressing the issue - a common
image PN name for multiconfig targets while the results are actually
machine-specific - by tuning the critical paths directly. This are just
WORKDIR as well as STAMP and STAMPCLEAN.

For keeping the deployment filenames stable, set (and use consistently)
IMAGE_FULLNAME to the former PF value.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---

Changes in v6:
 - avoid kernel/initrd filename changes

 meta/classes/image.bbclass | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index ec6bd39..2dad348 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -1,7 +1,10 @@
 # This software is a part of ISAR.
 # Copyright (C) 2015-2017 ilbers GmbH
 
-PF = "${PN}-${DISTRO}-${MACHINE}"
+# Make workdir and stamps machine-specific without changing common PN target
+WORKDIR .= "-${MACHINE}"
+STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/${PV}-${PR}"
+STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/*-*"
 
 IMAGE_INSTALL ?= ""
 IMAGE_TYPE    ?= "ext4-img"
@@ -10,12 +13,12 @@ IMAGE_ROOTFS ?= "${WORKDIR}/rootfs"
 IMAGE_INSTALL += "${@ ("linux-image-" + d.getVar("KERNEL_NAME", True)) if d.getVar("KERNEL_NAME", True) else ""}"
 
 # Name of the image including distro&machine names
-IMAGE_FULLNAME = "${PF}"
+IMAGE_FULLNAME = "${PN}-${DISTRO}-${MACHINE}"
 
 # These variables are used by wic and start_vm
 KERNEL_FILE  ?= "vmlinuz"
-KERNEL_IMAGE ?= "${PF}-${KERNEL_FILE}"
-INITRD_IMAGE ?= "${PF}-initrd.img"
+KERNEL_IMAGE ?= "${IMAGE_FULLNAME}-${KERNEL_FILE}"
+INITRD_IMAGE ?= "${IMAGE_FULLNAME}-initrd.img"
 
 # Useful variables for imager implementations:
 PP = "/home/builder/${PN}"
-- 
2.16.4

[PATCH v5 06/27] image: Make WORKDIR and STAMPs IMAGE_TYPE-specific

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This fixes false sharing of parallel builds like
multiconfig:qemuamd64-buster:isar-image-base plus
multiconfig:qemuamd64-buster-tgz:isar-image-base.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/image.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 36e9875..89cc5fe 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -2,9 +2,9 @@
 # Copyright (C) 2015-2017 ilbers GmbH
 
 # Make workdir and stamps machine-specific without changing common PN target
-WORKDIR .= "-${MACHINE}"
-STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/${PV}-${PR}"
-STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/*-*"
+WORKDIR .= "-${MACHINE}-${IMAGE_TYPE}"
+STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/${PV}-${PR}"
+STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/*-*"
 
 IMAGE_INSTALL ?= ""
 IMAGE_TYPE    ?= "ext4-img"
-- 
2.16.4

[PATCH v5 07/27] Adjust STAMP variables to use PN, rather than PF

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

The last misuse of PF is gone, we can adjust the defaults to OE.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/conf/bitbake.conf | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index e5681bf..9c9c7fe 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -36,8 +36,8 @@ S = "${WORKDIR}/${P}"
 AUTOREV = "${@bb.fetch2.get_autorev(d)}"
 SRC_URI = "file://${FILE}"
 STAMPS_DIR ?= "${TMPDIR}/stamps"
-STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PF}/${PV}-${PR}"
-STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PF}/*-*"
+STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/${PV}-${PR}"
+STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/*-*"
 SVNDIR = "${DL_DIR}/svn"
 T = "${WORKDIR}/temp"
 TMPDIR = "${TOPDIR}/tmp"
-- 
2.16.4

[PATCH v5 08/27] Align WORKDIR structure with OE

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Pick up the OE pattern to have a per-PV-PR workdir under PN. It also
aligns this with our STAMP setting.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 doc/user_manual.md         | 2 +-
 meta/classes/image.bbclass | 2 +-
 meta/conf/bitbake.conf     | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index c40a214..47f0dd5 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -452,7 +452,7 @@ The user may use `met-isar/recipes-core-images` as a template for new image reci
 ### General Information
 The image recipe in Isar creates a folder with target root filesystem. Its default location is:
 ```
-tmp/work/${DISTRO}-${DISTRO_ARCH}/${MACHINE}/${IMAGE}/rootfs
+tmp/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/${PV}-${PR}/rootfs
 ```
 Every image type in Isar is implemented as a `bitbake` class. The goal of these classes is to pack root filesystem folder to appropriate format.
 
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 89cc5fe..0965f26 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -2,7 +2,7 @@
 # Copyright (C) 2015-2017 ilbers GmbH
 
 # Make workdir and stamps machine-specific without changing common PN target
-WORKDIR .= "-${MACHINE}-${IMAGE_TYPE}"
+WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/${PV}-${PR}"
 STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/${PV}-${PR}"
 STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/*-*"
 
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 9c9c7fe..bca1114 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -45,7 +45,7 @@ PERSISTENT_DIR = "${TMPDIR}/cache"
 GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
 
 # isar specific config
-WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}"
+WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}/${PV}-${PR}"
 DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"
 DEPLOY_DIR_BUILDCHROOT = "${DEPLOY_DIR}/buildchroot"
 DEPLOY_DIR_SDKCHROOT = "${DEPLOY_DIR}/sdkchroot"
-- 
2.16.4

[PATCH v5 09/27] isar-bootstrap: Unshare host bootstraps across different target distros

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

They may come with different settings for the bootstrap environment, and
currently only the first ones win (due to that improper locking in
isar-bootstrap).

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/rootfs.bbclass                             | 2 +-
 meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index c7e0435..5bafe32 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -65,7 +65,7 @@ rootfs_do_qemu() {
     fi
 }
 
-BOOTSTRAP_SRC = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}-${DISTRO_ARCH}"
+BOOTSTRAP_SRC = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}"
 BOOTSTRAP_SRC_${ROOTFS_ARCH} = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}"
 
 rootfs_prepare[weight] = "25"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index b2dded5..d8abaf3 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -7,8 +7,8 @@
 
 Description = "Minimal host Debian root file system"
 
-DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
-ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}.lock"
+DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}"
+ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}.lock"
 
 require isar-bootstrap.inc
 
-- 
2.16.4

[PATCH v5 10/27] isar-bootstrap: Clean up and fix rebuild

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

There is no reason for using a lock anymore. All bootstraps for the same
distro/arch are detected by bitbake as identical targets in multiconfig
scenarios and are not run multiple times. The assumed original reason
for the lock, the bogus sharing of isar-bootstrap-host between Debian
and Raspbian, is fixed now.

In addition, remove the equally bogus re-execution test in
do_apt_config_prepare. This also fixes rebuilds of the isar-bootstrap
target in case some parameter has changed.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 .../isar-bootstrap/isar-bootstrap-host.bb          | 30 ++++++++++------------
 .../isar-bootstrap/isar-bootstrap-target.bb        | 27 +++++++++----------
 .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  5 ----
 3 files changed, 26 insertions(+), 36 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index d8abaf3..c985383 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -8,7 +8,6 @@
 Description = "Minimal host Debian root file system"
 
 DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}"
-ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}.lock"
 
 require isar-bootstrap.inc
 
@@ -25,21 +24,20 @@ do_apt_config_prepare[vardeps] += "\
                                    DEPLOY_ISAR_BOOTSTRAP \
                                   "
 python do_apt_config_prepare() {
-    if not os.path.islink(d.getVar("DEPLOY_ISAR_BOOTSTRAP", True)):
-        apt_preferences_out = d.getVar("APTPREFS", True)
-        apt_preferences_list = (
-            d.getVar("HOST_DISTRO_APT_PREFERENCES", True) or ""
-        ).split()
-        aggregate_files(d, apt_preferences_list, apt_preferences_out)
-
-        apt_sources_out = d.getVar("APTSRCS", True)
-        apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
-        apt_sources_list = (
-            d.getVar("HOST_DISTRO_APT_SOURCES", True) or ""
-        ).split()
-
-        aggregate_files(d, apt_sources_list, apt_sources_init_out)
-        aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
+    apt_preferences_out = d.getVar("APTPREFS", True)
+    apt_preferences_list = (
+        d.getVar("HOST_DISTRO_APT_PREFERENCES", True) or ""
+    ).split()
+    aggregate_files(d, apt_preferences_list, apt_preferences_out)
+
+    apt_sources_out = d.getVar("APTSRCS", True)
+    apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
+    apt_sources_list = (
+        d.getVar("HOST_DISTRO_APT_SOURCES", True) or ""
+    ).split()
+
+    aggregate_files(d, apt_sources_list, apt_sources_init_out)
+    aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
 }
 addtask apt_config_prepare before do_bootstrap after do_unpack
 
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
index d7fcdbf..80e7f40 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
@@ -8,7 +8,6 @@
 Description = "Minimal target Debian root file system"
 
 DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}"
-ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}.lock"
 
 require isar-bootstrap.inc
 
@@ -22,19 +21,18 @@ do_apt_config_prepare[vardeps] += "\
                                    DEPLOY_ISAR_BOOTSTRAP \
                                   "
 python do_apt_config_prepare() {
-    if not os.path.islink(d.getVar("DEPLOY_ISAR_BOOTSTRAP", True)):
-        apt_preferences_out = d.getVar("APTPREFS", True)
-        apt_preferences_list = (
-            d.getVar("DISTRO_APT_PREFERENCES", True) or ""
-        ).split()
-        aggregate_files(d, apt_preferences_list, apt_preferences_out)
-
-        apt_sources_out = d.getVar("APTSRCS", True)
-        apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
-        apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or "").split()
-
-        aggregate_files(d, apt_sources_list, apt_sources_init_out)
-        aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
+    apt_preferences_out = d.getVar("APTPREFS", True)
+    apt_preferences_list = (
+        d.getVar("DISTRO_APT_PREFERENCES", True) or ""
+    ).split()
+    aggregate_files(d, apt_preferences_list, apt_preferences_out)
+
+    apt_sources_out = d.getVar("APTSRCS", True)
+    apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
+    apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or "").split()
+
+    aggregate_files(d, apt_sources_list, apt_sources_init_out)
+    aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
 }
 addtask apt_config_prepare before do_bootstrap after do_unpack
 
@@ -45,4 +43,3 @@ do_bootstrap() {
     isar_bootstrap
 }
 addtask bootstrap before do_build after do_generate_keyrings
-
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index a2bd4a8..9ef5b76 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -210,14 +210,11 @@ do_generate_keyrings() {
 }
 addtask generate_keyrings before do_build after do_unpack
 
-
-
 def get_host_release():
     import platform
     rel = platform.release()
     return rel
 
-
 do_bootstrap[vardeps] += "DISTRO_APT_PREMIRRORS"
 do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP}"
 
@@ -244,7 +241,6 @@ isar_bootstrap() {
     E="${@bb.utils.export_proxies(d)}"
     export IS_HOST debootstrap_args E
     sudo -E -s <<'EOSUDO'
-        ( flock 9
         set -e
         if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then
             rm -rf "${ROOTFSDIR}"
@@ -338,7 +334,6 @@ isar_bootstrap() {
             # Finalize debootstrap by setting the link in deploy
             ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
         fi
-        ) 9>'${ISAR_BOOTSTRAP_LOCK}'
 EOSUDO
 }
 
-- 
2.16.4

[PATCH v5 11/27] gitlab-ci: Keep logs as artifacts on failure

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 .gitlab-ci.yml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 1c6fa84..5c64adc 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -11,3 +11,9 @@ all:
     - export ftp_proxy=$FTP_PROXY
     - export no_proxy=$NO_PROXY
     - scripts/ci_build.sh -q -f
+  artifacts:
+    name: "logs-$CI_JOB_ID"
+    paths:
+      - build/tmp/work/*/*/*/temp
+    when: on_failure
+    expire_in: 1 week
-- 
2.16.4

[PATCH v5 12/27] Detect false sharing of recipes

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

When a task is run more than once per build, this indicates a bug,
usually (though not only) related to incorrect multiconfig use. Such
bugs are hard to find in practice, so they could benefit from some
tooling.

This adds a very simple but effective check for such re-run scenarios
by putting a "once.<taskname>" stamp file into each workdir during a
build. If such a file already exists, an error is thrown. The workdirs
are purged after the parsing phase on the beginning of each build so
that no false positives are raised.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/isar-events.bbclass | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
index 62eb25d..1335356 100644
--- a/meta/classes/isar-events.bbclass
+++ b/meta/classes/isar-events.bbclass
@@ -4,6 +4,28 @@
 # Copyright (C) 2015-2017 ilbers GmbH
 # Copyright (c) Siemens AG, 2018
 
+addhandler parse_completed
+
+python parse_completed() {
+    bb.utils.remove(d.getVar('TMPDIR') + "/work/*/*/*/temp/once.*")
+}
+parse_completed[eventmask] = "bb.event.ParseCompleted"
+
+addhandler task_started
+
+python task_started() {
+    task = d.getVar('BB_RUNTASK')
+    try:
+        f = open("%s/once.%s" % (d.getVar('T'), task), "x")
+        f.close()
+    except FileExistsError:
+        bb.error("Detect multiple executions of %s in %s" %
+                 (task, d.getVar('WORKDIR')))
+        bb.error("Rerun a clean build with empty STAMPCLEAN " \
+                 "and compare the sigdata files")
+}
+task_started[eventmask] = "bb.build.TaskStarted"
+
 addhandler build_completed
 
 python build_completed() {
-- 
2.16.4

[PATCH v6 12/27] Detect false sharing of recipes

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

When a task is run more than once per build, this indicates a bug,
usually (though not only) related to incorrect multiconfig use. Such
bugs are hard to find in practice, so they could benefit from some
tooling.

This adds a very simple but effective check for such re-run scenarios
by putting a "once.<taskname>" stamp file into each workdir during a
build. If such a file already exists, an error is thrown. The workdirs
are purged after the parsing phase on the beginning of each build so
that no false positives are raised. The task stamp is also deleted after
failures so that no (serialized) retries can cause warnings.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---

Changes in v6:
 - avoid warnings after build failures of shared tasks
 - code refactoring

 meta/classes/isar-events.bbclass | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
index 62eb25d..cad1d57 100644
--- a/meta/classes/isar-events.bbclass
+++ b/meta/classes/isar-events.bbclass
@@ -4,6 +4,40 @@
 # Copyright (C) 2015-2017 ilbers GmbH
 # Copyright (c) Siemens AG, 2018
 
+addhandler parse_completed
+
+python parse_completed() {
+    bb.utils.remove(d.getVar('TMPDIR') + "/work/*/*/*/temp/once.*")
+}
+parse_completed[eventmask] = "bb.event.ParseCompleted"
+
+def task_once_stamp(d):
+    return "{temp}/once.{task}".format(temp=d.getVar('T'),
+                                       task=d.getVar('BB_RUNTASK'))
+
+addhandler task_started
+
+python task_started() {
+    try:
+        f = open(task_once_stamp(d), "x")
+        f.close()
+    except FileExistsError:
+        bb.error("Detect multiple executions of %s in %s" %
+                 (d.getVar('BB_RUNTASK'), d.getVar('WORKDIR')))
+        bb.error("Rerun a clean build with empty STAMPCLEAN " \
+                 "and compare the sigdata files")
+}
+task_started[eventmask] = "bb.build.TaskStarted"
+
+addhandler task_failed
+
+python task_failed() {
+    # Avoid false positives if a second target depends on this task and retries
+    # the execution after the first failure.
+    os.remove(task_once_stamp(d))
+}
+task_failed[eventmask] = "bb.build.TaskFailed"
+
 addhandler build_completed
 
 python build_completed() {
-- 
2.16.4

Re: [PATCH v6 12/27] Detect false sharing of recipes

[Baurzhan Ismagulov]

On Tue, Sep 24, 2019 at 08:02:26PM +0200, Jan Kiszka wrote:
> When a task is run more than once per build, this indicates a bug,
> usually (though not only) related to incorrect multiconfig use.

Do I understand correctly:

1. Multiple isar-bootstrap executions (apart from the fact that they are
   unnecessary) could fail because some steps after debootstrap (which was
   locked) were not using the lock and could overwrite each other's results?
   The background is to understand the specific failure mode, because you wrote
   that this locking didn't actually protect from races.

2. Multiple isar-bootstrap executions do not occur now because the same tasks
   that should not be executed twice have the right PF?


> This adds a very simple but effective check for such re-run scenarios
> by putting a "once.<taskname>" stamp file into each workdir during a
> build. If such a file already exists, an error is thrown. The workdirs
> are purged after the parsing phase on the beginning of each build so
> that no false positives are raised. The task stamp is also deleted after
> failures so that no (serialized) retries can cause warnings.

3. Where does this delete the once.* file w.r.t. e.g. do_dpkg_build for
   different multiconfigs?


With kind regards,
Baurzhan.

Re: [PATCH v6 12/27] Detect false sharing of recipes

[Jan Kiszka]

On 29.09.19 16:57, Baurzhan Ismagulov wrote:
> On Tue, Sep 24, 2019 at 08:02:26PM +0200, Jan Kiszka wrote:
>> When a task is run more than once per build, this indicates a bug,
>> usually (though not only) related to incorrect multiconfig use.
> 
> Do I understand correctly:
> 
> 1. Multiple isar-bootstrap executions (apart from the fact that they are
>     unnecessary) could fail because some steps after debootstrap (which was
>     locked) were not using the lock and could overwrite each other's results?
>     The background is to understand the specific failure mode, because you wrote
>     that this locking didn't actually protect from races.
> 
> 2. Multiple isar-bootstrap executions do not occur now because the same tasks
>     that should not be executed twice have the right PF?
> 

isar-bootstrap had two issue:

1. Not rebuilding when it should (on changes to its configuration)

2. False sharing of it deployment directory

None of them were or could have been detected by this instrumentation.

What would have been detected AFTER removing that improper rebuild prevention 
(https://groups.google.com/d/msgid/isar-users/ccad9eb9-fce5-89d1-3188-7043f003ea7e%40siemens.com) 
is 
https://groups.google.com/d/msgid/isar-users/ccad9eb9-fce5-89d1-3188-7043f003ea7e%40siemens.com 
- which is still not in your queue. Lost in your staging tree? I removed it from 
my last repost because it was staged already.

> 
>> This adds a very simple but effective check for such re-run scenarios
>> by putting a "once.<taskname>" stamp file into each workdir during a
>> build. If such a file already exists, an error is thrown. The workdirs
>> are purged after the parsing phase on the beginning of each build so
>> that no false positives are raised. The task stamp is also deleted after
>> failures so that no (serialized) retries can cause warnings.
> 
> 3. Where does this delete the once.* file w.r.t. e.g. do_dpkg_build for
>     different multiconfigs?
> 

At the beginning of the new build (parse_completed) or when a recipe failed to 
build (task_failed).

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v6 12/27] Detect false sharing of recipes

[Baurzhan Ismagulov]

On Mon, Sep 30, 2019 at 08:00:21AM +0200, Jan Kiszka wrote:
> isar-bootstrap had two issue:
> 
> 1. Not rebuilding when it should (on changes to its configuration)

Was this due to the PF tweaking before your series, and is now fixed by your PF
changes?


> 2. False sharing of it deployment directory

Do you mean e.g. using tmp/deploy/bootstrap/debian-stretch-armhf for
qemuarm-stretch and bananapi, in fact having different configurations (bananapi
requesting the buster kernel)?


> None of them were or could have been detected by this instrumentation.

I meant your whole series including the PF changes and bundled the questions in
this reply, sorry for confusion.


> What would have been detected AFTER removing that improper rebuild
> prevention (https://groups.google.com/d/msgid/isar-users/ccad9eb9-fce5-89d1-3188-7043f003ea7e%40siemens.com)
> is https://groups.google.com/d/msgid/isar-users/ccad9eb9-fce5-89d1-3188-7043f003ea7e%40siemens.com
> - which is still not in your queue. Lost in your staging tree?

Not lost, no worries; two patches from the first series are still pending. I'd
like to understand the failure mode because I'm also struggling with a race.
Does the URL refer to [PATCH v2 09/26] meta-isar: Move bananapi to buster,
remove bogus example? My browser opens the URL in the middle of a message.


> > 3. Where does this delete the once.* file w.r.t. e.g. do_dpkg_build for
> >     different multiconfigs?
> 
> At the beginning of the new build (parse_completed) or when a recipe failed
> to build (task_failed).

Yes, this is what I also read from the patch. If I build hello-isar for e.g.
qemuarm-buster and qemuamd64-buster, its do_dpkg_build is called for each
multiconfig. Why doesn't it fail on once.do_dpkg_build? IIUC, parse_completed
has finished in the beginning and no recipes fail. The following output was
taken with your instrumentation:

+ bitbake -v multiconfig:qemuarm-stretch:isar-image-base multiconfig:qemuarm64-stretch:isar-image-base multiconfig:qemuamd64-stretch:isar-image-base multiconfig:qemuarm-buster:isar-image-base
NOTE: Running task 306 of 404 (multiconfig:qemuarm-stretch:/workspace/build/isar_ibr_devel_fast/24/meta-isar/recipes-app/hello-isar/hello-isar.bb:do_dpkg_build)
NOTE: recipe hello-isar-0.3-a18c14c-r0: task do_dpkg_build: Started
NOTE: Running task 310 of 404 (multiconfig:qemuarm64-stretch:/workspace/build/isar_ibr_devel_fast/24/meta-isar/recipes-app/hello-isar/hello-isar.bb:do_dpkg_build)
NOTE: recipe hello-isar-0.3-a18c14c-r0: task do_dpkg_build: Started
NOTE: recipe hello-isar-0.3-a18c14c-r0: task do_dpkg_build: Succeeded
NOTE: recipe hello-isar-0.3-a18c14c-r0: task do_dpkg_build: Succeeded


With kind regards,
Baurzhan.

Re: [PATCH v6 12/27] Detect false sharing of recipes

[Jan Kiszka]

On 30.09.19 11:56, Baurzhan Ismagulov wrote:
> On Mon, Sep 30, 2019 at 08:00:21AM +0200, Jan Kiszka wrote:
>> isar-bootstrap had two issue:
>>
>> 1. Not rebuilding when it should (on changes to its configuration)
> 
> Was this due to the PF tweaking before your series, and is now fixed by your PF
> changes?

https://groups.google.com/d/msg/isar-users/nciPZ484-nI/kf9C-_9tAQAJ
(finding the right messing in googlegroups is a horrific mess...)

> 
> 
>> 2. False sharing of it deployment directory
> 
> Do you mean e.g. using tmp/deploy/bootstrap/debian-stretch-armhf for
> qemuarm-stretch and bananapi, in fact having different configurations (bananapi
> requesting the buster kernel)?
> 

https://groups.google.com/d/msg/isar-users/nciPZ484-nI/vjjYVf9tAQAJ

That bananapi was requesting a different bootstrap setting was a bug of its own, 
fixed by no longer doing that. Not triggered in CI because that target is not built.

> 
>> None of them were or could have been detected by this instrumentation.
> 
> I meant your whole series including the PF changes and bundled the questions in
> this reply, sorry for confusion.
> 
> 
>> What would have been detected AFTER removing that improper rebuild
>> prevention (https://groups.google.com/d/msgid/isar-users/ccad9eb9-fce5-89d1-3188-7043f003ea7e%40siemens.com)
>> is https://groups.google.com/d/msgid/isar-users/ccad9eb9-fce5-89d1-3188-7043f003ea7e%40siemens.com
>> - which is still not in your queue. Lost in your staging tree?
> 
> Not lost, no worries; two patches from the first series are still pending. I'd
> like to understand the failure mode because I'm also struggling with a race.
> Does the URL refer to [PATCH v2 09/26] meta-isar: Move bananapi to buster,
> remove bogus example? My browser opens the URL in the middle of a message.
> 

Yes, sorry, googlegroups... I thought the links you find at the bottom of each 
message would work, but they don't, at least in this thread.

> 
>>> 3. Where does this delete the once.* file w.r.t. e.g. do_dpkg_build for
>>>      different multiconfigs?
>>
>> At the beginning of the new build (parse_completed) or when a recipe failed
>> to build (task_failed).
> 
> Yes, this is what I also read from the patch. If I build hello-isar for e.g.
> qemuarm-buster and qemuamd64-buster, its do_dpkg_build is called for each
> multiconfig. Why doesn't it fail on once.do_dpkg_build? IIUC, parse_completed

If two multiconfs have the same common dependency, that one is only built once. 
Well, except when not. And when not, my instrumentation will tell you that there 
is a fundamental multiconf bug.

> has finished in the beginning and no recipes fail. The following output was
> taken with your instrumentation:
> 
> + bitbake -v multiconfig:qemuarm-stretch:isar-image-base multiconfig:qemuarm64-stretch:isar-image-base multiconfig:qemuamd64-stretch:isar-image-base multiconfig:qemuarm-buster:isar-image-base
> NOTE: Running task 306 of 404 (multiconfig:qemuarm-stretch:/workspace/build/isar_ibr_devel_fast/24/meta-isar/recipes-app/hello-isar/hello-isar.bb:do_dpkg_build)
> NOTE: recipe hello-isar-0.3-a18c14c-r0: task do_dpkg_build: Started
> NOTE: Running task 310 of 404 (multiconfig:qemuarm64-stretch:/workspace/build/isar_ibr_devel_fast/24/meta-isar/recipes-app/hello-isar/hello-isar.bb:do_dpkg_build)
> NOTE: recipe hello-isar-0.3-a18c14c-r0: task do_dpkg_build: Started
> NOTE: recipe hello-isar-0.3-a18c14c-r0: task do_dpkg_build: Succeeded
> NOTE: recipe hello-isar-0.3-a18c14c-r0: task do_dpkg_build: Succeeded
> 
> 
> With kind regards,
> Baurzhan.
> 

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v6 12/27] Detect false sharing of recipes

[Baurzhan Ismagulov]

On Mon, Sep 30, 2019 at 12:14:38PM +0200, Jan Kiszka wrote:
> > > isar-bootstrap had two issue:
> > > 
> > > 1. Not rebuilding when it should (on changes to its configuration)
> 
> https://groups.google.com/d/msg/isar-users/nciPZ484-nI/kf9C-_9tAQAJ
> (finding the right messing in googlegroups is a horrific mess...)
> 
> > > 2. False sharing of it deployment directory
> 
> https://groups.google.com/d/msg/isar-users/nciPZ484-nI/vjjYVf9tAQAJ

Ok, thanks for the explanation. You unshared the host rootfses for Debian armhf
and Raspbian armhf. After that, the locks in isar_bootstrap() and
do_apt_config_prepare() are indeed not necessary.

Actually, sharing both was an intended optimization because the resulting host
rootfses are identical. However, the implementation drops one via locking and
not via bitbake pipeline, and isn't executed on config change. Not sure how
difficult it would be to do that right. I can merge #9 and #10 for now.


> If two multiconfs have the same common dependency, that one is only built
> once. Well, except when not. And when not, my instrumentation will tell you
> that there is a fundamental multiconf bug.

I've forgotten that hello is unpacked for every arch-codename, so each has its
own once.do_dpkg_build. Never mind.


With kind regards,
Baurzhan.

Re: [PATCH v6 12/27] Detect false sharing of recipes

[Jan Kiszka]

On 30.09.19 16:53, Baurzhan Ismagulov wrote:
> On Mon, Sep 30, 2019 at 12:14:38PM +0200, Jan Kiszka wrote:
>>>> isar-bootstrap had two issue:
>>>>
>>>> 1. Not rebuilding when it should (on changes to its configuration)
>>
>> https://groups.google.com/d/msg/isar-users/nciPZ484-nI/kf9C-_9tAQAJ
>> (finding the right messing in googlegroups is a horrific mess...)
>>
>>>> 2. False sharing of it deployment directory
>>
>> https://groups.google.com/d/msg/isar-users/nciPZ484-nI/vjjYVf9tAQAJ
> 
> Ok, thanks for the explanation. You unshared the host rootfses for Debian armhf
> and Raspbian armhf. After that, the locks in isar_bootstrap() and
> do_apt_config_prepare() are indeed not necessary.
> 
> Actually, sharing both was an intended optimization because the resulting host
> rootfses are identical. However, the implementation drops one via locking and
> not via bitbake pipeline, and isn't executed on config change. Not sure how
> difficult it would be to do that right. I can merge #9 and #10 for now.

As the commit log stated, this assumption ("are identical") was wrong in the 
first place.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v6 12/27] Detect false sharing of recipes

[Baurzhan Ismagulov]

On Mon, Sep 30, 2019 at 05:26:07PM +0200, Jan Kiszka wrote:
> > Ok, thanks for the explanation. You unshared the host rootfses for Debian armhf
> > and Raspbian armhf. After that, the locks in isar_bootstrap() and
> > do_apt_config_prepare() are indeed not necessary.
> > 
> > Actually, sharing both was an intended optimization because the resulting host
> > rootfses are identical. However, the implementation drops one via locking and
> > not via bitbake pipeline, and isn't executed on config change. Not sure how
> > difficult it would be to do that right. I can merge #9 and #10 for now.
> 
> As the commit log stated, this assumption ("are identical") was wrong in the
> first place.

To make it clear, that was just an explanation why it had been done, I didn't
intend to challenge your motivation.

With stock Isar and your both series applied, a quick cd tmp/deploy/bootstrap;
sudo diff -Nrq debian-stretch-host_debian-stretch-armhf
debian-stretch-host_raspbian-stretch-armhf results in:

diff: debian-stretch-host_debian-stretch-armhf/lib64/ld-linux-x86-64.so.2: No such file or directory
diff: debian-stretch-host_raspbian-stretch-armhf/lib64/ld-linux-x86-64.so.2: No such file or directory
Files debian-stretch-host_debian-stretch-armhf/var/cache/apt/pkgcache.bin and debian-stretch-host_raspbian-stretch-armhf/var/cache/apt/pkgcache.bin differ
Files debian-stretch-host_debian-stretch-armhf/var/cache/apt/srcpkgcache.bin and debian-stretch-host_raspbian-stretch-armhf/var/cache/apt/srcpkgcache.bin differ
Files debian-stretch-host_debian-stretch-armhf/var/cache/ldconfig/aux-cache and debian-stretch-host_raspbian-stretch-armhf/var/cache/ldconfig/aux-cache differ
Files debian-stretch-host_debian-stretch-armhf/var/log/alternatives.log and debian-stretch-host_raspbian-stretch-armhf/var/log/alternatives.log differ
Files debian-stretch-host_debian-stretch-armhf/var/log/apt/history.log and debian-stretch-host_raspbian-stretch-armhf/var/log/apt/history.log differ
Files debian-stretch-host_debian-stretch-armhf/var/log/apt/term.log and debian-stretch-host_raspbian-stretch-armhf/var/log/apt/term.log differ
Files debian-stretch-host_debian-stretch-armhf/var/log/bootstrap.log and debian-stretch-host_raspbian-stretch-armhf/var/log/bootstrap.log differ
Files debian-stretch-host_debian-stretch-armhf/var/log/dpkg.log and debian-stretch-host_raspbian-stretch-armhf/var/log/dpkg.log differ

I.e., no differences. For my understanding, do you have host bootstrap
differences downstream? What are they? Maybe we could add a test case for that.

With kind regards,
Baurzhan.

Re: [PATCH v6 12/27] Detect false sharing of recipes

[Jan Kiszka]

On 30.09.19 17:39, Baurzhan Ismagulov wrote:
> On Mon, Sep 30, 2019 at 05:26:07PM +0200, Jan Kiszka wrote:
>>> Ok, thanks for the explanation. You unshared the host rootfses for Debian armhf
>>> and Raspbian armhf. After that, the locks in isar_bootstrap() and
>>> do_apt_config_prepare() are indeed not necessary.
>>>
>>> Actually, sharing both was an intended optimization because the resulting host
>>> rootfses are identical. However, the implementation drops one via locking and
>>> not via bitbake pipeline, and isn't executed on config change. Not sure how
>>> difficult it would be to do that right. I can merge #9 and #10 for now.
>>
>> As the commit log stated, this assumption ("are identical") was wrong in the
>> first place.
> 
> To make it clear, that was just an explanation why it had been done, I didn't
> intend to challenge your motivation.
> 
> With stock Isar and your both series applied, a quick cd tmp/deploy/bootstrap;
> sudo diff -Nrq debian-stretch-host_debian-stretch-armhf
> debian-stretch-host_raspbian-stretch-armhf results in:
> 
> diff: debian-stretch-host_debian-stretch-armhf/lib64/ld-linux-x86-64.so.2: No such file or directory
> diff: debian-stretch-host_raspbian-stretch-armhf/lib64/ld-linux-x86-64.so.2: No such file or directory
> Files debian-stretch-host_debian-stretch-armhf/var/cache/apt/pkgcache.bin and debian-stretch-host_raspbian-stretch-armhf/var/cache/apt/pkgcache.bin differ
> Files debian-stretch-host_debian-stretch-armhf/var/cache/apt/srcpkgcache.bin and debian-stretch-host_raspbian-stretch-armhf/var/cache/apt/srcpkgcache.bin differ
> Files debian-stretch-host_debian-stretch-armhf/var/cache/ldconfig/aux-cache and debian-stretch-host_raspbian-stretch-armhf/var/cache/ldconfig/aux-cache differ
> Files debian-stretch-host_debian-stretch-armhf/var/log/alternatives.log and debian-stretch-host_raspbian-stretch-armhf/var/log/alternatives.log differ
> Files debian-stretch-host_debian-stretch-armhf/var/log/apt/history.log and debian-stretch-host_raspbian-stretch-armhf/var/log/apt/history.log differ
> Files debian-stretch-host_debian-stretch-armhf/var/log/apt/term.log and debian-stretch-host_raspbian-stretch-armhf/var/log/apt/term.log differ
> Files debian-stretch-host_debian-stretch-armhf/var/log/bootstrap.log and debian-stretch-host_raspbian-stretch-armhf/var/log/bootstrap.log differ
> Files debian-stretch-host_debian-stretch-armhf/var/log/dpkg.log and debian-stretch-host_raspbian-stretch-armhf/var/log/dpkg.log differ
> 
> I.e., no differences. For my understanding, do you have host bootstrap
> differences downstream? What are they? Maybe we could add a test case for that.

The differences can come from different HOST_DISTRO_* settings. Those are 
applied on a per target distro/arch basis. Locking papered over this. And it 
broke rebuilding on changes.

We didn't trigger that issue with CI, well, maybe except for the case when you 
activated the bananapi.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v6 12/27] Detect false sharing of recipes

[Baurzhan Ismagulov]

On Mon, Sep 30, 2019 at 06:34:02PM +0200, Jan Kiszka wrote:
> The differences can come from different HOST_DISTRO_* settings. Those are
> applied on a per target distro/arch basis.

I see, thanks for the info.

I wonder whether those could be moved down to buildchroot / images in the
future.


> Locking papered over this. And it broke rebuilding on changes.

The implementation was broken, no question. Your patches are necessary to
provide the correct behavior and will be merged.

The question was rather an open one beyond the series, whether one could
determine whether there are no differences and keep sharing in that case -- I
don't have an answer ATM.


With kind regards,
Baurzhan.

Re: [PATCH v6 12/27] Detect false sharing of recipes

[Jan Kiszka]

On 30.09.19 19:20, Baurzhan Ismagulov wrote:
> On Mon, Sep 30, 2019 at 06:34:02PM +0200, Jan Kiszka wrote:
>> The differences can come from different HOST_DISTRO_* settings. Those are
>> applied on a per target distro/arch basis.
> 
> I see, thanks for the info.
> 
> I wonder whether those could be moved down to buildchroot / images in the
> future.
> 

Would mean breaking up the only-once repo updates into chunks so that everything 
we need in addition can also be pulled later on, but still only once. But it may 
not be trivial.

> 
>> Locking papered over this. And it broke rebuilding on changes.
> 
> The implementation was broken, no question. Your patches are necessary to
> provide the correct behavior and will be merged.
> 
> The question was rather an open one beyond the series, whether one could
> determine whether there are no differences and keep sharing in that case -- I
> don't have an answer ATM.

Honestly, I think we have more urgent issues. Correctness first, and then - when 
there is a real need - we can think of optimization.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v6 12/27] Detect false sharing of recipes

[Baurzhan Ismagulov]

On Mon, Sep 30, 2019 at 07:26:22PM +0200, Jan Kiszka wrote:
> > I wonder whether those could be moved down to buildchroot / images in the
> > future.
> 
> Would mean breaking up the only-once repo updates into chunks so that
> everything we need in addition can also be pulled later on, but still only
> once. But it may not be trivial.

Thanks. Sounds complicated indeed.


> > The question was rather an open one beyond the series, whether one could
> > determine whether there are no differences and keep sharing in that case -- I
> > don't have an answer ATM.
> 
> Honestly, I think we have more urgent issues. Correctness first, and then -
> when there is a real need - we can think of optimization.

Not to imply anyone should ever do it, just discussing. Never mind.


With kind regards,
Baurzhan.

[PATCH v5 13/27] dpkg-base: Move do_deploy_deb before do_build

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

There is not good reason to move anything beyond the final do_build task
in a recipe.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 773dced..4c6e02a 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -130,7 +130,7 @@ do_deploy_deb() {
              ${S}/../*.deb
 }
 
-addtask deploy_deb after do_dpkg_build
+addtask deploy_deb after do_dpkg_build before do_build
 do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 do_deploy_deb[depends] = "isar-apt:do_cache_config"
 do_deploy_deb[dirs] = "${S}"
-- 
2.16.4

Re: [PATCH v5 13/27] dpkg-base: Move do_deploy_deb before do_build

[Baurzhan Ismagulov]

On Sun, Sep 22, 2019 at 08:16:57PM +0200, Jan Kiszka wrote:
> There is not good reason to move anything beyond the final do_build task
> in a recipe.

Applied to next, thanks.

With kind regards,
Baurzhan.

[PATCH v5 14/27] base-apt: Move do_cache_config before do_build

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

There is not good reason to move anything beyond the final do_build task
in a recipe. This also aligns base-apt with isar-apt.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-devtools/base-apt/base-apt.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index 74189f1..6acd6e7 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -35,4 +35,4 @@ do_cache_config() {
     fi
 }
 
-addtask cache_config after do_build
+addtask cache_config after do_unpack before do_build
-- 
2.16.4

[PATCH v5 15/27] wic-img: Use private tmp dir

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

The buildchroot is shared between images of the same target arch. Just
using /tmp/<image-name> can cause conflicts and build failures.

Seen with jailhouse-images.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/wic-img.bbclass | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass
index 1ccbe55..795640f 100644
--- a/meta/classes/wic-img.bbclass
+++ b/meta/classes/wic-img.bbclass
@@ -144,14 +144,17 @@ EOSUDO
     export BUILDDIR=${BUILDDIR}
     export MTOOLS_SKIP_CHECK=1
 
+    # create the temp dir in the buildchroot to ensure uniqueness
+    WICTMP=$(cd ${BUILDCHROOT_DIR}; mktemp -d -p tmp)
+
     sudo -E chroot ${BUILDCHROOT_DIR} \
         ${ISARROOT}/scripts/wic create ${WKS_FULL_PATH} \
             --vars "${STAGING_DIR}/${MACHINE}/imgdata/" \
-            -o /tmp/${IMAGE_FULLNAME}.wic/ \
+            -o /$WICTMP/${IMAGE_FULLNAME}.wic/ \
             --bmap \
             -e ${IMAGE_BASENAME} ${WIC_CREATE_EXTRA_ARGS}
     sudo chown -R $(stat -c "%U" ${ISARROOT}) ${ISARROOT}/meta ${ISARROOT}/meta-isar ${ISARROOT}/scripts || true
-    WIC_DIRECT=$(ls -t -1 ${BUILDCHROOT_DIR}/tmp/${IMAGE_FULLNAME}.wic/*.direct | head -1)
+    WIC_DIRECT=$(ls -t -1 ${BUILDCHROOT_DIR}/$WICTMP/${IMAGE_FULLNAME}.wic/*.direct | head -1)
     cp -f ${WIC_DIRECT} ${WIC_IMAGE_FILE}
     cp -f ${WIC_DIRECT}.bmap ${WIC_IMAGE_FILE}.bmap
 }
-- 
2.16.4

[PATCH v5 16/27] rootfs: Take isar-apt shared lock during repository operations

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

We take this lock already for install_imager_deps, but we also need it
conceptually for the rootfs installation. Also here, pulling from
isar-apt can race with not-yet finished deployments of parallel builds,
even if the target package is not shared. See e9a5d8adf6b5 for further
details.

To avoid holding the lock across the whole lengthy package installation,
introduce command flags that define the beginning and the end of the
locked section so that do_rootfs_install can manage that.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/rootfs.bbclass | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 5bafe32..9bada0c 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -93,6 +93,7 @@ EOSUDO
 
 ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update"
 rootfs_install_pkgs_update[weight] = "5"
+rootfs_install_pkgs_update[isar-lock] = "acquire-before"
 rootfs_install_pkgs_update() {
     sudo -E chroot '${ROOTFSDIR}' /usr/bin/apt-get update \
         -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
@@ -108,6 +109,7 @@ rootfs_install_resolvconf() {
 
 ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download"
 rootfs_install_pkgs_download[weight] = "600"
+rootfs_install_pkgs_download[isar-lock] = "release-after"
 rootfs_install_pkgs_download() {
     sudo -E chroot '${ROOTFSDIR}' \
         /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES}
@@ -153,7 +155,15 @@ python do_rootfs_install() {
 
     for cmd in cmds:
         progress_reporter.next_stage()
+
+        if (d.getVarFlag(cmd, 'isar-lock') or "") == "acquire-before":
+            lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
+                                     shared=True)
+
         bb.build.exec_func(cmd, d)
+
+        if (d.getVarFlag(cmd, 'isar-lock') or "") == "release-after":
+            bb.utils.unlockfile(lock)
     progress_reporter.finish()
 }
 addtask rootfs_install before do_rootfs_postprocess after do_unpack
-- 
2.16.4

[PATCH v5 17/27] Use deb.debian.org mirrors

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

See http://deb.debian.org for the background: Addresses the issue of
choosing the best mirror automatically, and it is the official
recommendation.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/conf/distro/debian-bullseye.list | 14 ++++++++------
 meta/conf/distro/debian-buster.list   | 14 ++++++++------
 meta/conf/distro/debian-stretch.list  | 14 ++++++++------
 3 files changed, 24 insertions(+), 18 deletions(-)

diff --git a/meta/conf/distro/debian-bullseye.list b/meta/conf/distro/debian-bullseye.list
index c7f23e9..44b48dd 100644
--- a/meta/conf/distro/debian-bullseye.list
+++ b/meta/conf/distro/debian-bullseye.list
@@ -1,6 +1,8 @@
-deb	http://ftp.debian.org/debian bullseye	main contrib non-free
-deb-src	http://ftp.debian.org/debian bullseye	main contrib non-free
-deb	http://ftp.debian.org/debian bullseye-updates	main contrib non-free
-deb-src	http://ftp.debian.org/debian bullseye-updates	main contrib non-free
-deb	http://security.debian.org	bullseye-security	main contrib non-free
-deb-src	http://security.debian.org	bullseye-security	main contrib non-free
+deb	http://deb.debian.org/debian bullseye main contrib non-free
+deb-src	http://deb.debian.org/debian bullseye main contrib non-free
+
+deb	http://deb.debian.org/debian-security/ bullseye-security/updates main contrib non-free
+deb-src	http://deb.debian.org/debian-security/ bullseye-security/updates main contrib non-free
+
+deb	http://deb.debian.org/debian bullseye-updates main contrib non-free
+deb-src	http://deb.debian.org/debian bullseye-updates main contrib non-free
diff --git a/meta/conf/distro/debian-buster.list b/meta/conf/distro/debian-buster.list
index 206b7a5..ba49649 100644
--- a/meta/conf/distro/debian-buster.list
+++ b/meta/conf/distro/debian-buster.list
@@ -1,6 +1,8 @@
-deb	http://ftp.debian.org/debian buster	main contrib non-free
-deb-src	http://ftp.debian.org/debian buster	main contrib non-free
-deb	http://ftp.debian.org/debian buster-updates	main contrib non-free
-deb-src	http://ftp.debian.org/debian buster-updates	main contrib non-free
-deb	http://security.debian.org	buster/updates	main contrib non-free
-deb-src	http://security.debian.org	buster/updates	main contrib non-free
+deb	http://deb.debian.org/debian buster main contrib non-free
+deb-src	http://deb.debian.org/debian buster main contrib non-free
+
+deb	http://deb.debian.org/debian-security/ buster/updates main contrib non-free
+deb-src	http://deb.debian.org/debian-security/ buster/updates main contrib non-free
+
+deb	http://deb.debian.org/debian buster-updates main contrib non-free
+deb-src	http://deb.debian.org/debian buster-updates main contrib non-free
diff --git a/meta/conf/distro/debian-stretch.list b/meta/conf/distro/debian-stretch.list
index 1ec86fb..8da9a43 100644
--- a/meta/conf/distro/debian-stretch.list
+++ b/meta/conf/distro/debian-stretch.list
@@ -1,6 +1,8 @@
-deb	http://ftp.debian.org/debian stretch	main contrib non-free
-deb-src	http://ftp.debian.org/debian stretch	main contrib non-free
-deb	http://ftp.debian.org/debian stretch-updates	main contrib non-free
-deb-src	http://ftp.debian.org/debian stretch-updates	main contrib non-free
-deb	http://security.debian.org	stretch/updates	main contrib non-free
-deb-src	http://security.debian.org	stretch/updates	main contrib non-free
+deb	http://deb.debian.org/debian stretch main contrib non-free
+deb-src	http://deb.debian.org/debian stretch main contrib non-free
+
+deb	http://deb.debian.org/debian-security/ stretch/updates main contrib non-free
+deb-src	http://deb.debian.org/debian-security/ stretch/updates main contrib non-free
+
+deb	http://deb.debian.org/debian stretch-updates main contrib non-free
+deb-src	http://deb.debian.org/debian stretch-updates main contrib non-free
-- 
2.16.4

Re: [PATCH v5 17/27] Use deb.debian.org mirrors

[Baurzhan Ismagulov]

On Sun, Sep 22, 2019 at 08:17:01PM +0200, Jan Kiszka wrote:
> See http://deb.debian.org for the background: Addresses the issue of
> choosing the best mirror automatically

I'm personally in favor of this. That said, we used to use deb.d.o in the past.
It failed very often with 404. My impression is that its SRV records were
pointing to mirrors with different up-to-dateness. I'd like to hold this patch
and test it on CI for a few days.


> and it is the official recommendation.

Could you please point me to a link? If this is the case, and the problem above
still exists, we could ask for support upstream. I could only find "The Debian
project maintains deb.debian.org for this purpose and you can use this in your
apt sources.list" in https://www.debian.org/mirror/list.


With kind regards,
Baurzhan.

Re: [PATCH v5 17/27] Use deb.debian.org mirrors

[Jan Kiszka]

On 23.09.19 12:03, Baurzhan Ismagulov wrote:
> On Sun, Sep 22, 2019 at 08:17:01PM +0200, Jan Kiszka wrote:
>> See http://deb.debian.org for the background: Addresses the issue of
>> choosing the best mirror automatically
> 
> I'm personally in favor of this. That said, we used to use deb.d.o in the past.
> It failed very often with 404. My impression is that its SRV records were
> pointing to mirrors with different up-to-dateness. I'd like to hold this patch
> and test it on CI for a few days.
> 

No problem. There is no dependency on this IIRC.

> 
>> and it is the official recommendation.
> 
> Could you please point me to a link? If this is the case, and the problem above
> still exists, we could ask for support upstream. I could only find "The Debian
> project maintains deb.debian.org for this purpose and you can use this in your
> apt sources.list" in https://www.debian.org/mirror/list.

https://wiki.debian.org/SourcesList

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v5 17/27] Use deb.debian.org mirrors

[Baurzhan Ismagulov]

On Sun, Sep 22, 2019 at 08:17:01PM +0200, Jan Kiszka wrote:
> See http://deb.debian.org for the background: Addresses the issue of
> choosing the best mirror automatically, and it is the official
> recommendation.

In the last few days, this worked even better than ftp.d.o. Applied to next,
thanks.

That said, network problems are still our top CI failure reason. We'll have to
implement some (pre-)mirroring solution.

With kind regards,
Baurzhan.

[PATCH v5 18/27] linux-custom: Control linux-libc-dev deployment manually

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Deploying a version of linux-libc-dev that is different from the one
Debian uses easily causes problems. We already ran into those when doing
a downgrade, but we can also create deadlocks when doing an update. The
latter happens in common cross-build scenarios when pushing a new
version for the target arch but not providing one for the builder.

Avoid such troubles my making the package deployment opt-in. In most
cases, we will not depend on such an update because we will rarely
exploit new kernel API in userspace packages.

We can revert this behavior once we support building packages for both
target and host.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-kernel/linux/files/build-kernel.sh | 7 -------
 meta/recipes-kernel/linux/linux-custom.inc      | 7 +++++--
 2 files changed, 5 insertions(+), 9 deletions(-)

diff --git a/meta/recipes-kernel/linux/files/build-kernel.sh b/meta/recipes-kernel/linux/files/build-kernel.sh
index 8b7b23b..7b651af 100644
--- a/meta/recipes-kernel/linux/files/build-kernel.sh
+++ b/meta/recipes-kernel/linux/files/build-kernel.sh
@@ -127,10 +127,3 @@ rm -f linux-image-${PV}_${PV}-1_*.deb
 fakeroot dpkg-deb -b "${REPACK_LINUX_HEADERS_DIR}" \
 	linux-headers-${KERNEL_NAME}_${PV}-1_${KERNEL_NAME}.deb
 rm -f linux-headers-${PV}_${PV}-1_*.deb
-
-# linux-libc-dev causes dependency problems if we downgrade
-# remove it after the build so the downgraded version does not get deployed
-LINUX_LIBC_DEV_V=$( dpkg-query --show --showformat '${Version}' linux-libc-dev )
-if dpkg --compare-versions $LINUX_LIBC_DEV_V gt $PV-1; then
-	rm -f linux-libc-dev_${PV}*.deb
-fi
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index c045b89..e75eed1 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -26,6 +26,8 @@ KBUILD_DEPENDS ?= "build-essential:native libssl-dev libelf-dev bc git kmod biso
 KERNEL_DEBIAN_DEPENDS ?= "initramfs-tools | linux-initramfs-tool, kmod, linux-base (>= 4.3~)"
 KERNEL_HEADERS_DEBIAN_DEPENDS ?= "libc6, libssl1.1"
 
+KERNEL_LIBC_DEV_DEPLOY ?= "0"
+
 do_install_builddeps() {
 	dpkg_do_mounts
 	E="${@ bb.utils.export_proxies(d)}"
@@ -61,7 +63,8 @@ dpkg_runbuild() {
 	export KERNEL_HEADERS_DEBIAN_DEPENDS="${KERNEL_HEADERS_DEBIAN_DEPENDS}"
 
 	sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} ${PP}/build-kernel.sh ${PP}/${PPS} ${DISTRO_ARCH}
-	if [ ! -f ${WORKDIR}/linux-libc-dev_${PV}*.deb ]; then
-		bbwarn "Kernel downgrade detected, not deploying linux-libc-dev"
+
+	if [ "${KERNEL_LIBC_DEV_DEPLOY}" != "1" ]; then
+		rm -f ${WORKDIR}/linux-libc-dev_${PV}*.deb
 	fi
 }
-- 
2.16.4

[PATCH v5 19/27] dpkg-base: Wait for umount to succeed

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Rather than ignoring a still busy BUILDROOT mount point, wait for it to
become free. If that should ever get stuck, the user can still forcibly
terminate. But normally, this is just about giving some detached
processes time to finish.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 4c6e02a..c86d0b3 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -77,8 +77,10 @@ dpkg_do_mounts() {
 }
 
 dpkg_undo_mounts() {
-    sudo umount ${BUILDROOT} 2>/dev/null || true
-    sudo rmdir ${BUILDROOT} 2>/dev/null || true
+    while ! sudo umount ${BUILDROOT} 2>/dev/null; do
+        sleep 0.1
+    done
+    sudo rmdir ${BUILDROOT}
 }
 
 # Placeholder for actual dpkg_runbuild() implementation
-- 
2.16.4

Re: [PATCH v5 19/27] dpkg-base: Wait for umount to succeed

[Baurzhan Ismagulov]

On Sun, Sep 22, 2019 at 08:17:03PM +0200, Jan Kiszka wrote:
> Rather than ignoring a still busy BUILDROOT mount point, wait for it to
> become free. If that should ever get stuck, the user can still forcibly
> terminate. But normally, this is just about giving some detached
> processes time to finish.
...
>  dpkg_undo_mounts() {
> -    sudo umount ${BUILDROOT} 2>/dev/null || true
> -    sudo rmdir ${BUILDROOT} 2>/dev/null || true
> +    while ! sudo umount ${BUILDROOT} 2>/dev/null; do
> +        sleep 0.1
> +    done
> +    sudo rmdir ${BUILDROOT}

I agree that we shouldn't ignore failing umounts. However, hanging forever
leads to bad user experience. I suggest to introduce a timeout or a number of
attempts and fail.

With kind regards,
Baurzhan.

Re: [PATCH v5 19/27] dpkg-base: Wait for umount to succeed

[Jan Kiszka]

On 30.09.19 13:07, Baurzhan Ismagulov wrote:
> On Sun, Sep 22, 2019 at 08:17:03PM +0200, Jan Kiszka wrote:
>> Rather than ignoring a still busy BUILDROOT mount point, wait for it to
>> become free. If that should ever get stuck, the user can still forcibly
>> terminate. But normally, this is just about giving some detached
>> processes time to finish.
> ...
>>   dpkg_undo_mounts() {
>> -    sudo umount ${BUILDROOT} 2>/dev/null || true
>> -    sudo rmdir ${BUILDROOT} 2>/dev/null || true
>> +    while ! sudo umount ${BUILDROOT} 2>/dev/null; do
>> +        sleep 0.1
>> +    done
>> +    sudo rmdir ${BUILDROOT}
> 
> I agree that we shouldn't ignore failing umounts. However, hanging forever
> leads to bad user experience. I suggest to introduce a timeout or a number of
> attempts and fail.

Let's not over-engineer: CI runs have top-level timeouts, tuned to the 
requirements of the particular runtime environment. Local builds have ctrl-c. 
And defining such a micro-timeout would not be trivial as well. So let's merge 
and look that this again when this really shows up, based on that concrete case.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v5 19/27] dpkg-base: Wait for umount to succeed

[Baurzhan Ismagulov]

On Mon, Sep 30, 2019 at 02:22:06PM +0200, Jan Kiszka wrote:
> > I agree that we shouldn't ignore failing umounts. However, hanging forever
> > leads to bad user experience. I suggest to introduce a timeout or a number of
> > attempts and fail.
> 
> Let's not over-engineer: CI runs have top-level timeouts, tuned to the
> requirements of the particular runtime environment. Local builds have
> ctrl-c. And defining such a micro-timeout would not be trivial as well. So
> let's merge and look that this again when this really shows up, based on
> that concrete case.

My main concern is that

1. the user isn't immediately aware of the fact that something is going wrong,
   and

2. once aware, it isn't trivial to understand what is going on. Aborting the
   build won't make the user understand the problem.

With wheezy or jessie, we had apt hanging. ATM, Jenkins workspaces leave
gpg-agent running. Similar issues could occur in the chroot at any time. When
it happens, the analysis would take some time; the idea is to reduce it to
zero. This can be as simple as issuing a warning mentioning $0, which dir can't
be umounted, and the fact that we wait infinitely for that, e.g. once a second.
With this, the user could even salvage his build by killing the relevant
processes.

I think this is an important usability issue with a simple mitigation. If
necessary, I could provide a patch.

With kind regards,
Baurzhan.

Re: [PATCH v5 19/27] dpkg-base: Wait for umount to succeed

[Baurzhan Ismagulov]

On Sun, Sep 22, 2019 at 08:17:03PM +0200, Jan Kiszka wrote:
> Rather than ignoring a still busy BUILDROOT mount point, wait for it to
> become free. If that should ever get stuck, the user can still forcibly
> terminate. But normally, this is just about giving some detached
> processes time to finish.

Applied to next, thanks.

With kind regards,
Baurzhan.

[PATCH v5 20/27] dpkg-base: Permit multiple clean runs

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

"find ${S}/.." fails if the is no ${S} anymore.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index c86d0b3..fa904b8 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -102,7 +102,7 @@ addtask dpkg_build before do_build
 CLEANFUNCS += "repo_clean"
 
 repo_clean() {
-    DEBS=$( find ${S}/.. -maxdepth 1 -name "*.deb" )
+    DEBS=$( find ${S}/.. -maxdepth 1 -name "*.deb" || [ ! -d ${S} ] )
     if [ -n "${DEBS}" ]; then
         for d in ${DEBS}; do
             p=$( dpkg-deb --show --showformat '${Package}' ${d} )
-- 
2.16.4

[PATCH v5 21/27] buildchroot: Factor out common packages

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

No need to maintain this list twice.

At this chance, make adduser a common package. be291cd991bd added it
only to the target buildchroot, while it is used by both. That fact
indicates that we only list it here for future cases when it is no
longer implicitly installed.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-devtools/buildchroot/buildchroot-host.bb   | 16 ++++------------
 meta/recipes-devtools/buildchroot/buildchroot-target.bb | 17 ++++-------------
 meta/recipes-devtools/buildchroot/buildchroot.inc       | 13 +++++++++++++
 3 files changed, 21 insertions(+), 25 deletions(-)

diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
index 795b77a..408ad39 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
@@ -11,15 +11,7 @@ require buildchroot.inc
 ROOTFS_ARCH = "${HOST_ARCH}"
 ROOTFS_DISTRO = "${HOST_DISTRO}"
 
-BUILDCHROOT_PREINSTALL ?= "make \
-                           debhelper \
-                           autotools-dev \
-                           dpkg \
-                           locales \
-                           docbook-to-man \
-                           apt \
-                           automake \
-                           devscripts \
-                           equivs \
-                           libc6:${DISTRO_ARCH} \
-                           crossbuild-essential-${DISTRO_ARCH}"
+BUILDCHROOT_PREINSTALL ?= " \
+    ${BUILDCHROOT_PREINSTALL_COMMON} \
+    libc6:${DISTRO_ARCH} \
+    crossbuild-essential-${DISTRO_ARCH}"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-target.bb b/meta/recipes-devtools/buildchroot/buildchroot-target.bb
index 68aedaa..be77044 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-target.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-target.bb
@@ -9,16 +9,7 @@ BUILDCHROOT_VARIANT = "target"
 
 require buildchroot.inc
 
-BUILDCHROOT_PREINSTALL ?= "gcc \
-                           make \
-                           build-essential \
-                           debhelper \
-                           autotools-dev \
-                           dpkg \
-                           locales \
-                           docbook-to-man \
-                           apt \
-                           automake \
-                           devscripts \
-                           equivs \
-                           adduser"
+BUILDCHROOT_PREINSTALL ?= " \
+    ${BUILDCHROOT_PREINSTALL_COMMON} \
+    gcc \
+    build-essential"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index e704137..fc29690 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -20,6 +20,19 @@ ROOTFSDIR = "${BUILDCHROOT_DIR}"
 ROOTFS_PACKAGES = "${BUILDCHROOT_PREINSTALL}"
 ROOTFS_CLEAN_FILES = ""
 
+BUILDCHROOT_PREINSTALL_COMMON = " \
+    make \
+    debhelper \
+    autotools-dev \
+    dpkg \
+    locales \
+    docbook-to-man \
+    apt \
+    automake \
+    devscripts \
+    equivs \
+    adduser"
+
 rootfs_do_mounts_append() {
     sudo -s <<'EOSUDO'
     mkdir -p '${BUILDCHROOT_DIR}/downloads'
-- 
2.16.4

[PATCH v5 22/27] buildchroot: Do not build cross when there are no arch-specific outputs

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

When all binary packages are "Architecture: all", we can avoid the cross
build and use the host architecture directly. This specifically resolves
cross-build issue with dependencies that are not cross-ready.

More concretely, if the build dependency meta package requests an arch-
independent package, this fails because apt seems to look only for
my-dependency:arch, ignoring that there is only :all.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-devtools/buildchroot/files/deps.sh | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh b/meta/recipes-devtools/buildchroot/files/deps.sh
index 19eb9b7..c04acce 100644
--- a/meta/recipes-devtools/buildchroot/files/deps.sh
+++ b/meta/recipes-devtools/buildchroot/files/deps.sh
@@ -25,6 +25,12 @@ apt-get update \
     -o Dir::Etc::sourceparts="-" \
     -o APT::Get::List-Cleanup="0"
 
+# Do not set an architecture when building only 'all' (generic) packages.
+# This can avoid unneeded cross-build issues.
+if ! grep "^Architecture:" debian/control | grep -qv "all"; then
+	set_arch=""
+fi
+
 # Install all build deps
 mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control
 
-- 
2.16.4

[PATCH v5 23/27] Add git-buildpackage support

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This allows to build packages from gbp-compatible repositories like
found on salsa. This is particularly useful for building unreleased
or older packages, and also patching them.

The feature comes with a test case that builds cowsay from its salsa
repo, adding this also to CI.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta-isar/recipes-app/cowsay/cowsay_git.bb       |  9 +++++++++
 meta/classes/dpkg-gbp.bbclass                    | 24 ++++++++++++++++++++++++
 meta/recipes-devtools/buildchroot/files/build.sh |  2 +-
 scripts/ci_build.sh                              |  2 ++
 4 files changed, 36 insertions(+), 1 deletion(-)
 create mode 100644 meta-isar/recipes-app/cowsay/cowsay_git.bb
 create mode 100644 meta/classes/dpkg-gbp.bbclass

diff --git a/meta-isar/recipes-app/cowsay/cowsay_git.bb b/meta-isar/recipes-app/cowsay/cowsay_git.bb
new file mode 100644
index 0000000..7aca1b4
--- /dev/null
+++ b/meta-isar/recipes-app/cowsay/cowsay_git.bb
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2019
+#
+# SPDX-License-Identifier: MIT
+
+inherit dpkg-gbp
+
+SRC_URI = "git://salsa.debian.org/debian/cowsay.git;protocol=https"
+SRCREV = "756f0c41fbf582093c0c1dff9ff77734716cb26f"
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
new file mode 100644
index 0000000..8a8e0f4
--- /dev/null
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -0,0 +1,24 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2019
+#
+# SPDX-License-Identifier: MIT
+
+inherit dpkg
+
+S = "${WORKDIR}/git"
+
+GBP_DEPENDS ?= "git-buildpackage pristine-tar"
+GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
+
+do_install_builddeps_append() {
+    dpkg_do_mounts
+    E="${@ bb.utils.export_proxies(d)}"
+    sudo -E chroot ${BUILDCHROOT_DIR} \
+        apt-get install -y -o Debug::pkgProblemResolver=yes \
+                        --no-install-recommends ${GBP_DEPENDS}
+    dpkg_undo_mounts
+}
+
+dpkg_runbuild_prepend() {
+    export GBP_PREFIX="gbp buildpackage --git-ignore-new ${GBP_EXTRA_OPTIONS} --git-builder="
+}
diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
index 1ba394c..101581d 100644
--- a/meta/recipes-devtools/buildchroot/files/build.sh
+++ b/meta/recipes-devtools/buildchroot/files/build.sh
@@ -14,4 +14,4 @@ for i in configure aclocal.m4 Makefile.am Makefile.in; do
     fi
 done
 
-dpkg-buildpackage -a$target_arch -d --source-option=-I
+${GBP_PREFIX}dpkg-buildpackage -a$target_arch -d --source-option=-I
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index a69cc51..d02326a 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -155,6 +155,8 @@ fi
 
 sed -i -e 's/#IMAGE_INSTALL += "isar-disable-apt-cache"/IMAGE_INSTALL += "isar-disable-apt-cache"/g' conf/local.conf
 
+echo 'IMAGE_INSTALL += "cowsay"' >> conf/local.conf
+
 # Start cross build for the defined set of configurations
 sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
 bitbake $BB_ARGS $CROSS_TARGETS_SET
-- 
2.16.4

[PATCH v5 24/27] dbpg-raw: Only list top-level files in <pn>.install

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

The rest is included automatically. This also avoids that we stumble
over filenames with whitespaces, something dh_install does not seem to
support but also something that we generally do not need at the top
level of a rootfs.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-raw.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index f8dfb2d..3f19f8e 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -21,7 +21,8 @@ addtask install after do_unpack before do_prepare_build
 do_prepare_build[cleandirs] += "${D}/debian"
 do_prepare_build() {
 	cd ${D}
-	find . ! -type d | sed 's:^./::' > ${WORKDIR}/${PN}.install
+	find . -maxdepth 1 ! -name .. -and ! -name . -and ! -name debian | \
+		sed 's:^./::' > ${WORKDIR}/${PN}.install
 	mv ${WORKDIR}/${PN}.install ${D}/debian/
 
 	deb_debianize
-- 
2.16.4

Re: [PATCH v5 24/27] dbpg-raw: Only list top-level files in <pn>.install

[Baurzhan Ismagulov]

On Sun, Sep 22, 2019 at 08:17:08PM +0200, Jan Kiszka wrote:
> The rest is included automatically. This also avoids that we stumble
> over filenames with whitespaces, something dh_install does not seem to
> support but also something that we generally do not need at the top
> level of a rootfs.

"dbpg-raw": Typo in the title?

With kind regards,
Baurzhan.

Re: [PATCH v5 24/27] dbpg-raw: Only list top-level files in <pn>.install

[Jan Kiszka]

On 23.09.19 11:52, Baurzhan Ismagulov wrote:
> On Sun, Sep 22, 2019 at 08:17:08PM +0200, Jan Kiszka wrote:
>> The rest is included automatically. This also avoids that we stumble
>> over filenames with whitespaces, something dh_install does not seem to
>> support but also something that we generally do not need at the top
>> level of a rootfs.
> 
> "dbpg-raw": Typo in the title?
> 

Something with "d..." - yes typo. Can you fix this up on merge, or do you prefer 
me sending an update?

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v5 24/27] dbpg-raw: Only list top-level files in <pn>.install

[Baurzhan Ismagulov]

On Mon, Sep 23, 2019 at 12:29:17PM +0200, Jan Kiszka wrote:
> Something with "d..." - yes typo. Can you fix this up on merge, or do you
> prefer me sending an update?

I can fix it, no need to resend.

With kind regards,
Baurzhan.

[PATCH v5 25/27] Add OE class and library elements for terminal support

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This imports the terminal class from OE core revision 95ad56262963 along
with its dependencies under meta/lib/oe and a helper script. Files are
almost unmodified, just augmented by a note for their origin, SPDX tags
where missing and freed of three whitespace issues.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/terminal.bbclass       | 112 +++++++++++++
 meta/lib/oe/classutils.py           |  49 ++++++
 meta/lib/oe/data.py                 |  53 ++++++
 meta/lib/oe/maketype.py             | 112 +++++++++++++
 meta/lib/oe/terminal.py             | 315 ++++++++++++++++++++++++++++++++++++
 meta/lib/oe/types.py                | 187 +++++++++++++++++++++
 scripts/oe-gnome-terminal-phonehome |  14 ++
 7 files changed, 842 insertions(+)
 create mode 100644 meta/classes/terminal.bbclass
 create mode 100644 meta/lib/oe/classutils.py
 create mode 100644 meta/lib/oe/data.py
 create mode 100644 meta/lib/oe/maketype.py
 create mode 100644 meta/lib/oe/terminal.py
 create mode 100644 meta/lib/oe/types.py
 create mode 100755 scripts/oe-gnome-terminal-phonehome

diff --git a/meta/classes/terminal.bbclass b/meta/classes/terminal.bbclass
new file mode 100644
index 0000000..8436d7f
--- /dev/null
+++ b/meta/classes/terminal.bbclass
@@ -0,0 +1,112 @@
+#
+# Imported from openembedded-core
+#
+# SPDX-License-Identifier: MIT
+#
+
+OE_TERMINAL ?= 'auto'
+OE_TERMINAL[type] = 'choice'
+OE_TERMINAL[choices] = 'auto none \
+                        ${@oe_terminal_prioritized()}'
+
+OE_TERMINAL_EXPORTS += 'EXTRA_OEMAKE CACHED_CONFIGUREVARS CONFIGUREOPTS EXTRA_OECONF'
+OE_TERMINAL_EXPORTS[type] = 'list'
+
+XAUTHORITY ?= "${HOME}/.Xauthority"
+SHELL ?= "bash"
+
+def oe_terminal_prioritized():
+    import oe.terminal
+    return " ".join(o.name for o in oe.terminal.prioritized())
+
+def emit_terminal_func(command, envdata, d):
+    import bb.build
+    cmd_func = 'do_terminal'
+
+    envdata.setVar(cmd_func, 'exec ' + command)
+    envdata.setVarFlag(cmd_func, 'func', '1')
+
+    runfmt = d.getVar('BB_RUNFMT') or "run.{func}.{pid}"
+    runfile = runfmt.format(func=cmd_func, task=cmd_func, taskfunc=cmd_func, pid=os.getpid())
+    runfile = os.path.join(d.getVar('T'), runfile)
+    bb.utils.mkdirhier(os.path.dirname(runfile))
+
+    with open(runfile, 'w') as script:
+        script.write(bb.build.shell_trap_code())
+        bb.data.emit_func(cmd_func, script, envdata)
+        script.write(cmd_func)
+        script.write("\n")
+    os.chmod(runfile, 0o755)
+
+    return runfile
+
+def oe_terminal(command, title, d):
+    import oe.data
+    import oe.terminal
+
+    envdata = bb.data.init()
+
+    for v in os.environ:
+        envdata.setVar(v, os.environ[v])
+        envdata.setVarFlag(v, 'export', '1')
+
+    for export in oe.data.typed_value('OE_TERMINAL_EXPORTS', d):
+        value = d.getVar(export)
+        if value is not None:
+            os.environ[export] = str(value)
+            envdata.setVar(export, str(value))
+            envdata.setVarFlag(export, 'export', '1')
+        if export == "PSEUDO_DISABLED":
+            if "PSEUDO_UNLOAD" in os.environ:
+                del os.environ["PSEUDO_UNLOAD"]
+            envdata.delVar("PSEUDO_UNLOAD")
+
+    # Add in all variables from the user's original environment which
+    # haven't subsequntly been set/changed
+    origbbenv = d.getVar("BB_ORIGENV", False) or {}
+    for key in origbbenv:
+        if key in envdata:
+            continue
+        value = origbbenv.getVar(key)
+        if value is not None:
+            os.environ[key] = str(value)
+            envdata.setVar(key, str(value))
+            envdata.setVarFlag(key, 'export', '1')
+
+    # Use original PATH as a fallback
+    path = d.getVar('PATH') + ":" + origbbenv.getVar('PATH')
+    os.environ['PATH'] = path
+    envdata.setVar('PATH', path)
+
+    # A complex PS1 might need more escaping of chars.
+    # Lets not export PS1 instead.
+    envdata.delVar("PS1")
+
+    # Replace command with an executable wrapper script
+    command = emit_terminal_func(command, envdata, d)
+
+    terminal = oe.data.typed_value('OE_TERMINAL', d).lower()
+    if terminal == 'none':
+        bb.fatal('Devshell usage disabled with OE_TERMINAL')
+    elif terminal != 'auto':
+        try:
+            oe.terminal.spawn(terminal, command, title, None, d)
+            return
+        except oe.terminal.UnsupportedTerminal:
+            bb.warn('Unsupported terminal "%s", defaulting to "auto"' %
+                    terminal)
+        except oe.terminal.ExecutionError as exc:
+            bb.fatal('Unable to spawn terminal %s: %s' % (terminal, exc))
+
+    try:
+        oe.terminal.spawn_preferred(command, title, None, d)
+    except oe.terminal.NoSupportedTerminals as nosup:
+        nosup.terms.remove("false")
+        cmds = '\n\t'.join(nosup.terms).replace("{command}",
+                    "do_terminal").replace("{title}", title)
+        bb.fatal('No valid terminal found, unable to open devshell.\n' +
+                'Tried the following commands:\n\t%s' % cmds)
+    except oe.terminal.ExecutionError as exc:
+        bb.fatal('Unable to spawn terminal %s: %s' % (terminal, exc))
+
+oe_terminal[vardepsexclude] = "BB_ORIGENV"
diff --git a/meta/lib/oe/classutils.py b/meta/lib/oe/classutils.py
new file mode 100644
index 0000000..d879191
--- /dev/null
+++ b/meta/lib/oe/classutils.py
@@ -0,0 +1,49 @@
+#
+# Imported from openembedded-core
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+class ClassRegistryMeta(type):
+    """Give each ClassRegistry their own registry"""
+    def __init__(cls, name, bases, attrs):
+        cls.registry = {}
+        type.__init__(cls, name, bases, attrs)
+
+class ClassRegistry(type, metaclass=ClassRegistryMeta):
+    """Maintain a registry of classes, indexed by name.
+
+Note that this implementation requires that the names be unique, as it uses
+a dictionary to hold the classes by name.
+
+The name in the registry can be overridden via the 'name' attribute of the
+class, and the 'priority' attribute controls priority. The prioritized()
+method returns the registered classes in priority order.
+
+Subclasses of ClassRegistry may define an 'implemented' property to exert
+control over whether the class will be added to the registry (e.g. to keep
+abstract base classes out of the registry)."""
+    priority = 0
+    def __init__(cls, name, bases, attrs):
+        super(ClassRegistry, cls).__init__(name, bases, attrs)
+        try:
+            if not cls.implemented:
+                return
+        except AttributeError:
+            pass
+
+        try:
+            cls.name
+        except AttributeError:
+            cls.name = name
+        cls.registry[cls.name] = cls
+
+    @classmethod
+    def prioritized(tcls):
+        return sorted(list(tcls.registry.values()),
+                      key=lambda v: (v.priority, v.name), reverse=True)
+
+    def unregister(cls):
+        for key in cls.registry.keys():
+            if cls.registry[key] is cls:
+                del cls.registry[key]
diff --git a/meta/lib/oe/data.py b/meta/lib/oe/data.py
new file mode 100644
index 0000000..b832e9b
--- /dev/null
+++ b/meta/lib/oe/data.py
@@ -0,0 +1,53 @@
+#
+# Imported from openembedded-core
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import json
+import oe.maketype
+
+def typed_value(key, d):
+    """Construct a value for the specified metadata variable, using its flags
+    to determine the type and parameters for construction."""
+    var_type = d.getVarFlag(key, 'type')
+    flags = d.getVarFlags(key)
+    if flags is not None:
+        flags = dict((flag, d.expand(value))
+                     for flag, value in list(flags.items()))
+    else:
+        flags = {}
+
+    try:
+        return oe.maketype.create(d.getVar(key) or '', var_type, **flags)
+    except (TypeError, ValueError) as exc:
+        bb.msg.fatal("Data", "%s: %s" % (key, str(exc)))
+
+def export2json(d, json_file, expand=True, searchString="",replaceString=""):
+    data2export = {}
+    keys2export = []
+
+    for key in d.keys():
+        if key.startswith("_"):
+            continue
+        elif key.startswith("BB"):
+            continue
+        elif key.startswith("B_pn"):
+            continue
+        elif key.startswith("do_"):
+            continue
+        elif d.getVarFlag(key, "func"):
+            continue
+
+        keys2export.append(key)
+
+    for key in keys2export:
+        try:
+            data2export[key] = d.getVar(key, expand).replace(searchString,replaceString)
+        except bb.data_smart.ExpansionError:
+            data2export[key] = ''
+        except AttributeError:
+            pass
+
+    with open(json_file, "w") as f:
+        json.dump(data2export, f, skipkeys=True, indent=4, sort_keys=True)
diff --git a/meta/lib/oe/maketype.py b/meta/lib/oe/maketype.py
new file mode 100644
index 0000000..969a22b
--- /dev/null
+++ b/meta/lib/oe/maketype.py
@@ -0,0 +1,112 @@
+#
+# Imported from openembedded-core
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+"""OpenEmbedded variable typing support
+
+Types are defined in the metadata by name, using the 'type' flag on a
+variable.  Other flags may be utilized in the construction of the types.  See
+the arguments of the type's factory for details.
+"""
+
+import inspect
+import oe.types as types
+try:
+    # Python 3.7+
+    from collections.abc import Callable
+except ImportError:
+    # Python < 3.7
+    from collections import Callable
+
+available_types = {}
+
+class MissingFlag(TypeError):
+    """A particular flag is required to construct the type, but has not been
+    provided."""
+    def __init__(self, flag, type):
+        self.flag = flag
+        self.type = type
+        TypeError.__init__(self)
+
+    def __str__(self):
+        return "Type '%s' requires flag '%s'" % (self.type, self.flag)
+
+def factory(var_type):
+    """Return the factory for a specified type."""
+    if var_type is None:
+        raise TypeError("No type specified. Valid types: %s" %
+                        ', '.join(available_types))
+    try:
+        return available_types[var_type]
+    except KeyError:
+        raise TypeError("Invalid type '%s':\n  Valid types: %s" %
+                        (var_type, ', '.join(available_types)))
+
+def create(value, var_type, **flags):
+    """Create an object of the specified type, given the specified flags and
+    string value."""
+    obj = factory(var_type)
+    objflags = {}
+    for flag in obj.flags:
+        if flag not in flags:
+            if flag not in obj.optflags:
+                raise MissingFlag(flag, var_type)
+        else:
+            objflags[flag] = flags[flag]
+
+    return obj(value, **objflags)
+
+def get_callable_args(obj):
+    """Grab all but the first argument of the specified callable, returning
+    the list, as well as a list of which of the arguments have default
+    values."""
+    if type(obj) is type:
+        obj = obj.__init__
+
+    sig = inspect.signature(obj)
+    args = list(sig.parameters.keys())
+    defaults = list(s for s in sig.parameters.keys() if sig.parameters[s].default != inspect.Parameter.empty)
+    flaglist = []
+    if args:
+        if len(args) > 1 and args[0] == 'self':
+            args = args[1:]
+        flaglist.extend(args)
+
+    optional = set()
+    if defaults:
+        optional |= set(flaglist[-len(defaults):])
+    return flaglist, optional
+
+def factory_setup(name, obj):
+    """Prepare a factory for use."""
+    args, optional = get_callable_args(obj)
+    extra_args = args[1:]
+    if extra_args:
+        obj.flags, optional = extra_args, optional
+        obj.optflags = set(optional)
+    else:
+        obj.flags = obj.optflags = ()
+
+    if not hasattr(obj, 'name'):
+        obj.name = name
+
+def register(name, factory):
+    """Register a type, given its name and a factory callable.
+
+    Determines the required and optional flags from the factory's
+    arguments."""
+    factory_setup(name, factory)
+    available_types[factory.name] = factory
+
+
+# Register all our included types
+for name in dir(types):
+    if name.startswith('_'):
+        continue
+
+    obj = getattr(types, name)
+    if not isinstance(obj, Callable):
+        continue
+
+    register(name, obj)
diff --git a/meta/lib/oe/terminal.py b/meta/lib/oe/terminal.py
new file mode 100644
index 0000000..0d875c3
--- /dev/null
+++ b/meta/lib/oe/terminal.py
@@ -0,0 +1,315 @@
+#
+# Imported from openembedded-core
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+import logging
+import oe.classutils
+import shlex
+from bb.process import Popen, ExecutionError
+from distutils.version import LooseVersion
+
+logger = logging.getLogger('BitBake.OE.Terminal')
+
+
+class UnsupportedTerminal(Exception):
+    pass
+
+class NoSupportedTerminals(Exception):
+    def __init__(self, terms):
+        self.terms = terms
+
+
+class Registry(oe.classutils.ClassRegistry):
+    command = None
+
+    def __init__(cls, name, bases, attrs):
+        super(Registry, cls).__init__(name.lower(), bases, attrs)
+
+    @property
+    def implemented(cls):
+        return bool(cls.command)
+
+
+class Terminal(Popen, metaclass=Registry):
+    def __init__(self, sh_cmd, title=None, env=None, d=None):
+        fmt_sh_cmd = self.format_command(sh_cmd, title)
+        try:
+            Popen.__init__(self, fmt_sh_cmd, env=env)
+        except OSError as exc:
+            import errno
+            if exc.errno == errno.ENOENT:
+                raise UnsupportedTerminal(self.name)
+            else:
+                raise
+
+    def format_command(self, sh_cmd, title):
+        fmt = {'title': title or 'Terminal', 'command': sh_cmd, 'cwd': os.getcwd() }
+        if isinstance(self.command, str):
+            return shlex.split(self.command.format(**fmt))
+        else:
+            return [element.format(**fmt) for element in self.command]
+
+class XTerminal(Terminal):
+    def __init__(self, sh_cmd, title=None, env=None, d=None):
+        Terminal.__init__(self, sh_cmd, title, env, d)
+        if not os.environ.get('DISPLAY'):
+            raise UnsupportedTerminal(self.name)
+
+class Gnome(XTerminal):
+    command = 'gnome-terminal -t "{title}" -x {command}'
+    priority = 2
+
+    def __init__(self, sh_cmd, title=None, env=None, d=None):
+        # Recent versions of gnome-terminal does not support non-UTF8 charset:
+        # https://bugzilla.gnome.org/show_bug.cgi?id=732127; as a workaround,
+        # clearing the LC_ALL environment variable so it uses the locale.
+        # Once fixed on the gnome-terminal project, this should be removed.
+        if os.getenv('LC_ALL'): os.putenv('LC_ALL','')
+
+        XTerminal.__init__(self, sh_cmd, title, env, d)
+
+class Mate(XTerminal):
+    command = 'mate-terminal --disable-factory -t "{title}" -x {command}'
+    priority = 2
+
+class Xfce(XTerminal):
+    command = 'xfce4-terminal -T "{title}" -e "{command}"'
+    priority = 2
+
+class Terminology(XTerminal):
+    command = 'terminology -T="{title}" -e {command}'
+    priority = 2
+
+class Konsole(XTerminal):
+    command = 'konsole --separate --workdir . -p tabtitle="{title}" -e {command}'
+    priority = 2
+
+    def __init__(self, sh_cmd, title=None, env=None, d=None):
+        # Check version
+        vernum = check_terminal_version("konsole")
+        if vernum and LooseVersion(vernum) < '2.0.0':
+            # Konsole from KDE 3.x
+            self.command = 'konsole -T "{title}" -e {command}'
+        elif vernum and LooseVersion(vernum) < '16.08.1':
+            # Konsole pre 16.08.01 Has nofork
+            self.command = 'konsole --nofork --workdir . -p tabtitle="{title}" -e {command}'
+        XTerminal.__init__(self, sh_cmd, title, env, d)
+
+class XTerm(XTerminal):
+    command = 'xterm -T "{title}" -e {command}'
+    priority = 1
+
+class Rxvt(XTerminal):
+    command = 'rxvt -T "{title}" -e {command}'
+    priority = 1
+
+class Screen(Terminal):
+    command = 'screen -D -m -t "{title}" -S devshell {command}'
+
+    def __init__(self, sh_cmd, title=None, env=None, d=None):
+        s_id = "devshell_%i" % os.getpid()
+        self.command = "screen -D -m -t \"{title}\" -S %s {command}" % s_id
+        Terminal.__init__(self, sh_cmd, title, env, d)
+        msg = 'Screen started. Please connect in another terminal with ' \
+            '"screen -r %s"' % s_id
+        if (d):
+            bb.event.fire(bb.event.LogExecTTY(msg, "screen -r %s" % s_id,
+                                              0.5, 10), d)
+        else:
+            logger.warning(msg)
+
+class TmuxRunning(Terminal):
+    """Open a new pane in the current running tmux window"""
+    name = 'tmux-running'
+    command = 'tmux split-window -c "{cwd}" "{command}"'
+    priority = 2.75
+
+    def __init__(self, sh_cmd, title=None, env=None, d=None):
+        if not bb.utils.which(os.getenv('PATH'), 'tmux'):
+            raise UnsupportedTerminal('tmux is not installed')
+
+        if not os.getenv('TMUX'):
+            raise UnsupportedTerminal('tmux is not running')
+
+        if not check_tmux_pane_size('tmux'):
+            raise UnsupportedTerminal('tmux pane too small or tmux < 1.9 version is being used')
+
+        Terminal.__init__(self, sh_cmd, title, env, d)
+
+class TmuxNewWindow(Terminal):
+    """Open a new window in the current running tmux session"""
+    name = 'tmux-new-window'
+    command = 'tmux new-window -c "{cwd}" -n "{title}" "{command}"'
+    priority = 2.70
+
+    def __init__(self, sh_cmd, title=None, env=None, d=None):
+        if not bb.utils.which(os.getenv('PATH'), 'tmux'):
+            raise UnsupportedTerminal('tmux is not installed')
+
+        if not os.getenv('TMUX'):
+            raise UnsupportedTerminal('tmux is not running')
+
+        Terminal.__init__(self, sh_cmd, title, env, d)
+
+class Tmux(Terminal):
+    """Start a new tmux session and window"""
+    command = 'tmux new -c "{cwd}" -d -s devshell -n devshell "{command}"'
+    priority = 0.75
+
+    def __init__(self, sh_cmd, title=None, env=None, d=None):
+        if not bb.utils.which(os.getenv('PATH'), 'tmux'):
+            raise UnsupportedTerminal('tmux is not installed')
+
+        # TODO: consider using a 'devshell' session shared amongst all
+        # devshells, if it's already there, add a new window to it.
+        window_name = 'devshell-%i' % os.getpid()
+
+        self.command = 'tmux new -c "{{cwd}}" -d -s {0} -n {0} "{{command}}"'.format(window_name)
+        Terminal.__init__(self, sh_cmd, title, env, d)
+
+        attach_cmd = 'tmux att -t {0}'.format(window_name)
+        msg = 'Tmux started. Please connect in another terminal with `tmux att -t {0}`'.format(window_name)
+        if d:
+            bb.event.fire(bb.event.LogExecTTY(msg, attach_cmd, 0.5, 10), d)
+        else:
+            logger.warning(msg)
+
+class Custom(Terminal):
+    command = 'false' # This is a placeholder
+    priority = 3
+
+    def __init__(self, sh_cmd, title=None, env=None, d=None):
+        self.command = d and d.getVar('OE_TERMINAL_CUSTOMCMD')
+        if self.command:
+            if not '{command}' in self.command:
+                self.command += ' {command}'
+            Terminal.__init__(self, sh_cmd, title, env, d)
+            logger.warning('Custom terminal was started.')
+        else:
+            logger.debug(1, 'No custom terminal (OE_TERMINAL_CUSTOMCMD) set')
+            raise UnsupportedTerminal('OE_TERMINAL_CUSTOMCMD not set')
+
+
+def prioritized():
+    return Registry.prioritized()
+
+def get_cmd_list():
+    terms = Registry.prioritized()
+    cmds = []
+    for term in terms:
+        if term.command:
+            cmds.append(term.command)
+    return cmds
+
+def spawn_preferred(sh_cmd, title=None, env=None, d=None):
+    """Spawn the first supported terminal, by priority"""
+    for terminal in prioritized():
+        try:
+            spawn(terminal.name, sh_cmd, title, env, d)
+            break
+        except UnsupportedTerminal:
+            continue
+    else:
+        raise NoSupportedTerminals(get_cmd_list())
+
+def spawn(name, sh_cmd, title=None, env=None, d=None):
+    """Spawn the specified terminal, by name"""
+    logger.debug(1, 'Attempting to spawn terminal "%s"', name)
+    try:
+        terminal = Registry.registry[name]
+    except KeyError:
+        raise UnsupportedTerminal(name)
+
+    # We need to know when the command completes but some terminals (at least
+    # gnome and tmux) gives us no way to do this. We therefore write the pid
+    # to a file using a "phonehome" wrapper script, then monitor the pid
+    # until it exits.
+    import tempfile
+    import time
+    pidfile = tempfile.NamedTemporaryFile(delete = False).name
+    try:
+        sh_cmd = bb.utils.which(os.getenv('PATH'), "oe-gnome-terminal-phonehome") + " " + pidfile + " " + sh_cmd
+        pipe = terminal(sh_cmd, title, env, d)
+        output = pipe.communicate()[0]
+        if output:
+            output = output.decode("utf-8")
+        if pipe.returncode != 0:
+            raise ExecutionError(sh_cmd, pipe.returncode, output)
+
+        while os.stat(pidfile).st_size <= 0:
+            time.sleep(0.01)
+            continue
+        with open(pidfile, "r") as f:
+            pid = int(f.readline())
+    finally:
+        os.unlink(pidfile)
+
+    while True:
+        try:
+            os.kill(pid, 0)
+            time.sleep(0.1)
+        except OSError:
+           return
+
+def check_tmux_pane_size(tmux):
+    import subprocess as sub
+    # On older tmux versions (<1.9), return false. The reason
+    # is that there is no easy way to get the height of the active panel
+    # on current window without nested formats (available from version 1.9)
+    vernum = check_terminal_version("tmux")
+    if vernum and LooseVersion(vernum) < '1.9':
+        return False
+    try:
+        p = sub.Popen('%s list-panes -F "#{?pane_active,#{pane_height},}"' % tmux,
+                shell=True,stdout=sub.PIPE,stderr=sub.PIPE)
+        out, err = p.communicate()
+        size = int(out.strip())
+    except OSError as exc:
+        import errno
+        if exc.errno == errno.ENOENT:
+            return None
+        else:
+            raise
+
+    return size/2 >= 19
+
+def check_terminal_version(terminalName):
+    import subprocess as sub
+    try:
+        cmdversion = '%s --version' % terminalName
+        if terminalName.startswith('tmux'):
+            cmdversion = '%s -V' % terminalName
+        newenv = os.environ.copy()
+        newenv["LANG"] = "C"
+        p = sub.Popen(['sh', '-c', cmdversion], stdout=sub.PIPE, stderr=sub.PIPE, env=newenv)
+        out, err = p.communicate()
+        ver_info = out.decode().rstrip().split('\n')
+    except OSError as exc:
+        import errno
+        if exc.errno == errno.ENOENT:
+            return None
+        else:
+            raise
+    vernum = None
+    for ver in ver_info:
+        if ver.startswith('Konsole'):
+            vernum = ver.split(' ')[-1]
+        if ver.startswith('GNOME Terminal'):
+            vernum = ver.split(' ')[-1]
+        if ver.startswith('MATE Terminal'):
+            vernum = ver.split(' ')[-1]
+        if ver.startswith('tmux'):
+            vernum = ver.split()[-1]
+        if ver.startswith('tmux next-'):
+            vernum = ver.split()[-1][5:]
+    return vernum
+
+def distro_name():
+    try:
+        p = Popen(['lsb_release', '-i'])
+        out, err = p.communicate()
+        distro = out.split(':')[1].strip().lower()
+    except:
+        distro = "unknown"
+    return distro
diff --git a/meta/lib/oe/types.py b/meta/lib/oe/types.py
new file mode 100644
index 0000000..ad8dd36
--- /dev/null
+++ b/meta/lib/oe/types.py
@@ -0,0 +1,187 @@
+#
+# Imported from openembedded-core
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import errno
+import re
+import os
+
+
+class OEList(list):
+    """OpenEmbedded 'list' type
+
+    Acts as an ordinary list, but is constructed from a string value and a
+    separator (optional), and re-joins itself when converted to a string with
+    str().  Set the variable type flag to 'list' to use this type, and the
+    'separator' flag may be specified (defaulting to whitespace)."""
+
+    name = "list"
+
+    def __init__(self, value, separator = None):
+        if value is not None:
+            list.__init__(self, value.split(separator))
+        else:
+            list.__init__(self)
+
+        if separator is None:
+            self.separator = " "
+        else:
+            self.separator = separator
+
+    def __str__(self):
+        return self.separator.join(self)
+
+def choice(value, choices):
+    """OpenEmbedded 'choice' type
+
+    Acts as a multiple choice for the user.  To use this, set the variable
+    type flag to 'choice', and set the 'choices' flag to a space separated
+    list of valid values."""
+    if not isinstance(value, str):
+        raise TypeError("choice accepts a string, not '%s'" % type(value))
+
+    value = value.lower()
+    choices = choices.lower()
+    if value not in choices.split():
+        raise ValueError("Invalid choice '%s'.  Valid choices: %s" %
+                         (value, choices))
+    return value
+
+class NoMatch(object):
+    """Stub python regex pattern object which never matches anything"""
+    def findall(self, string, flags=0):
+        return None
+
+    def finditer(self, string, flags=0):
+        return None
+
+    def match(self, flags=0):
+        return None
+
+    def search(self, string, flags=0):
+        return None
+
+    def split(self, string, maxsplit=0):
+        return None
+
+    def sub(pattern, repl, string, count=0):
+        return None
+
+    def subn(pattern, repl, string, count=0):
+        return None
+
+NoMatch = NoMatch()
+
+def regex(value, regexflags=None):
+    """OpenEmbedded 'regex' type
+
+    Acts as a regular expression, returning the pre-compiled regular
+    expression pattern object.  To use this type, set the variable type flag
+    to 'regex', and optionally, set the 'regexflags' type to a space separated
+    list of the flags to control the regular expression matching (e.g.
+    FOO[regexflags] += 'ignorecase').  See the python documentation on the
+    're' module for a list of valid flags."""
+
+    flagval = 0
+    if regexflags:
+        for flag in regexflags.split():
+            flag = flag.upper()
+            try:
+                flagval |= getattr(re, flag)
+            except AttributeError:
+                raise ValueError("Invalid regex flag '%s'" % flag)
+
+    if not value:
+        # Let's ensure that the default behavior for an undefined or empty
+        # variable is to match nothing. If the user explicitly wants to match
+        # anything, they can match '.*' instead.
+        return NoMatch
+
+    try:
+        return re.compile(value, flagval)
+    except re.error as exc:
+        raise ValueError("Invalid regex value '%s': %s" %
+                         (value, exc.args[0]))
+
+def boolean(value):
+    """OpenEmbedded 'boolean' type
+
+    Valid values for true: 'yes', 'y', 'true', 't', '1'
+    Valid values for false: 'no', 'n', 'false', 'f', '0', None
+    """
+    if value is None:
+        return False
+
+    if isinstance(value, bool):
+        return value
+
+    if not isinstance(value, str):
+        raise TypeError("boolean accepts a string, not '%s'" % type(value))
+
+    value = value.lower()
+    if value in ('yes', 'y', 'true', 't', '1'):
+        return True
+    elif value in ('no', 'n', 'false', 'f', '0'):
+        return False
+    raise ValueError("Invalid boolean value '%s'" % value)
+
+def integer(value, numberbase=10):
+    """OpenEmbedded 'integer' type
+
+    Defaults to base 10, but this can be specified using the optional
+    'numberbase' flag."""
+
+    return int(value, int(numberbase))
+
+_float = float
+def float(value, fromhex='false'):
+    """OpenEmbedded floating point type
+
+    To use this type, set the type flag to 'float', and optionally set the
+    'fromhex' flag to a true value (obeying the same rules as for the
+    'boolean' type) if the value is in base 16 rather than base 10."""
+
+    if boolean(fromhex):
+        return _float.fromhex(value)
+    else:
+        return _float(value)
+
+def path(value, relativeto='', normalize='true', mustexist='false'):
+    value = os.path.join(relativeto, value)
+
+    if boolean(normalize):
+        value = os.path.normpath(value)
+
+    if boolean(mustexist):
+        try:
+            open(value, 'r')
+        except IOError as exc:
+            if exc.errno == errno.ENOENT:
+                raise ValueError("{0}: {1}".format(value, os.strerror(errno.ENOENT)))
+
+    return value
+
+def is_x86(arch):
+    """
+    Check whether arch is x86 or x86_64
+    """
+    if arch.startswith('x86_') or re.match('i.*86', arch):
+        return True
+    else:
+        return False
+
+def qemu_use_kvm(kvm, target_arch):
+    """
+    Enable kvm if target_arch == build_arch or both of them are x86 archs.
+    """
+
+    use_kvm = False
+    if kvm and boolean(kvm):
+        build_arch = os.uname()[4]
+        if is_x86(build_arch) and is_x86(target_arch):
+            use_kvm = True
+        elif build_arch == target_arch:
+            use_kvm = True
+    return use_kvm
diff --git a/scripts/oe-gnome-terminal-phonehome b/scripts/oe-gnome-terminal-phonehome
new file mode 100755
index 0000000..5a321a2
--- /dev/null
+++ b/scripts/oe-gnome-terminal-phonehome
@@ -0,0 +1,14 @@
+#!/bin/sh
+#
+# Imported from openembedded-core
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Gnome terminal won't tell us which PID a given command is run as
+# or allow a single instance so we can't tell when it completes.
+# This allows us to figure out the PID of the target so we can tell
+# when its done.
+#
+echo $$ > $1
+shift
+exec $@
-- 
2.16.4

[PATCH v5 26/27] dpkg-base: Add devshell target

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Using the imported OE core services, this adds devshell support for dpkg
package builds. The target is put after do_install_builddeps where
available, otherwise do_prepare_build.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 doc/user_manual.md                         |  8 ++++++++
 meta/classes/dpkg-base.bbclass             | 26 +++++++++++++++++++++++++-
 meta/classes/dpkg.bbclass                  |  2 ++
 meta/recipes-kernel/linux/linux-custom.inc |  2 ++
 4 files changed, 37 insertions(+), 1 deletion(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 47f0dd5..c2657da 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -682,6 +682,14 @@ Debian cross-compilation works out of the box starting from Debian stretch distr
  - qemuarm64-buster (for host >= buster)
 
 
+## Examining and debugging package generation inside their buildchroot
+
+Just like OpenEmbedded, Isar supports a devshell target for all dpkg package
+recipes. This target opens a terminal inside the buildchroot that runs the
+package build. To invoke it, just call
+`bitbake multiconfig:${MACHINE}-${DISTRO}:<package_name> -c devshell`.
+
+
 ## Create an ISAR SDK root filesystem
 
 ### Motivation
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index fa904b8..2e8a0d1 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -1,10 +1,11 @@
 # This software is a part of ISAR.
-# Copyright (C) 2017-2018 Siemens AG
+# Copyright (C) 2017-2019 Siemens AG
 #
 # SPDX-License-Identifier: MIT
 
 inherit buildchroot
 inherit debianize
+inherit terminal
 
 DEPENDS ?= ""
 
@@ -136,3 +137,26 @@ addtask deploy_deb after do_dpkg_build before do_build
 do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 do_deploy_deb[depends] = "isar-apt:do_cache_config"
 do_deploy_deb[dirs] = "${S}"
+
+python do_devshell() {
+    import sys
+
+    oe_lib_path = os.path.join(d.getVar('LAYERDIR_core'), 'lib')
+    sys.path.insert(0, oe_lib_path)
+
+    bb.build.exec_func('dpkg_do_mounts', d);
+
+    bb.utils.export_proxies(d)
+
+    buildchroot = d.getVar('BUILDCHROOT_DIR')
+    pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
+    termcmd = "sudo -E chroot {0} sh -c 'cd {1}; $SHELL -i'"
+    oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d)
+
+    bb.build.exec_func('dpkg_undo_mounts', d);
+}
+
+addtask devshell after do_prepare_build
+DEVSHELL_STARTDIR ?= "${S}"
+do_devshell[dirs] = "${DEVSHELL_STARTDIR}"
+do_devshell[nostamp] = "1"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index f5396cb..19d77a2 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -15,6 +15,8 @@ addtask install_builddeps after do_prepare_build before do_dpkg_build
 # apt and reprepro may not run in parallel, acquire the Isar lock
 do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
+addtask devshell after do_install_builddeps
+
 # Build package from sources using build script
 dpkg_runbuild() {
     E="${@ bb.utils.export_proxies(d)}"
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index e75eed1..8ff3836 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -47,6 +47,8 @@ addtask install_builddeps after do_prepare_build before do_dpkg_build
 # apt and reprepro may not run in parallel, acquire the Isar lock
 do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
+addtask devshell after do_install_builddeps
+
 dpkg_runbuild() {
 	chmod +x ${WORKDIR}/build-kernel.sh
 	if [ -n "${KERNEL_DEFCONFIG}" ]; then
-- 
2.16.4

[PATCH v5 27/27] linux-custom: Add libncurses-dev to KBUILD_DEPENDS

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This allows to use menuconfig in the devshell.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-kernel/linux/linux-custom.inc | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 8ff3836..d817158 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -22,7 +22,17 @@ inherit dpkg-base
 
 SRC_URI += "file://build-kernel.sh"
 
-KBUILD_DEPENDS ?= "build-essential:native libssl-dev libelf-dev bc git kmod bison flex cpio"
+KBUILD_DEPENDS ?= " \
+    build-essential:native \
+    libssl-dev \
+    libelf-dev \
+    bc \
+    git \
+    kmod \
+    bison \
+    flex \
+    cpio \
+    libncurses-dev"
 KERNEL_DEBIAN_DEPENDS ?= "initramfs-tools | linux-initramfs-tool, kmod, linux-base (>= 4.3~)"
 KERNEL_HEADERS_DEBIAN_DEPENDS ?= "libc6, libssl1.1"
 
-- 
2.16.4

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Baurzhan Ismagulov]

Hello Jan,

thanks for the update. I was chasing what appears to be a different race in
isar-bootstrap that was randomly triggered by unrelated patches, which blocked
meaningful merging. Unfortunately, it is no more reproducible on any of my
machines. I'll be merging the patches in groups to speed up the process.

With kind regards,
Baurzhan.


On Sun, Sep 22, 2019 at 08:16:44PM +0200, Jan Kiszka wrote:
> This is primarily a repost of most of my pending patches in their tested
> order on top of current ibr/devel (I suspect that this is the next chunk
> to be merged). As the highest version number of a patch included here
> was v4, the series is tagged as v5.
> 
> I think this is usefull because I had to inject the isar-bootstrap-host
> sharing fix into the middle and adjust a succeeding patch.
> 
> As a bonus, this series ports devshell support over from OE core.
> 
> Jan
> 
> Jan Kiszka (27):
>   buildchroot: Properly deploy the build result
>   sdk: Use clean-package-cache rather than open-coded cleanup
>   sdk: Deploy earlier
>   Remove unneeded PF tweakings
>   image: Stop changing PF, tune WORKDIR and STAMP directly
>   image: Make WORKDIR and STAMPs IMAGE_TYPE-specific
>   Adjust STAMP variables to use PN, rather than PF
>   Align WORKDIR structure with OE
>   isar-bootstrap: Unshare host bootstraps across different target
>     distros
>   isar-bootstrap: Clean up and fix rebuild
>   gitlab-ci: Keep logs as artifacts on failure
>   Detect false sharing of recipes
>   dpkg-base: Move do_deploy_deb before do_build
>   base-apt: Move do_cache_config before do_build
>   wic-img: Use private tmp dir
>   rootfs: Take isar-apt shared lock during repository operations
>   Use deb.debian.org mirrors
>   linux-custom: Control linux-libc-dev deployment manually
>   dpkg-base: Wait for umount to succeed
>   dpkg-base: Permit multiple clean runs
>   buildchroot: Factor out common packages
>   buildchroot: Do not build cross when there are no arch-specific
>     outputs
>   Add git-buildpackage support
>   dbpg-raw: Only list top-level files in <pn>.install
>   Add OE class and library elements for terminal support
>   dpkg-base: Add devshell target
>   linux-custom: Add libncurses-dev to KBUILD_DEPENDS
> 
>  .gitlab-ci.yml                                     |   6 +
>  doc/user_manual.md                                 |  10 +-
>  meta-isar/recipes-app/cowsay/cowsay_git.bb         |   9 +
>  meta/classes/buildchroot.bbclass                   |   2 +-
>  meta/classes/dpkg-base.bbclass                     |  36 ++-
>  meta/classes/dpkg-gbp.bbclass                      |  24 ++
>  meta/classes/dpkg-raw.bbclass                      |   3 +-
>  meta/classes/dpkg.bbclass                          |   2 +
>  meta/classes/image-sdk-extension.bbclass           |  23 +-
>  meta/classes/image.bbclass                         |   5 +-
>  meta/classes/isar-events.bbclass                   |  22 ++
>  meta/classes/rootfs.bbclass                        |  12 +-
>  meta/classes/terminal.bbclass                      | 112 ++++++++
>  meta/classes/wic-img.bbclass                       |   7 +-
>  meta/conf/bitbake.conf                             |  13 +-
>  meta/conf/distro/debian-bullseye.list              |  14 +-
>  meta/conf/distro/debian-buster.list                |  14 +-
>  meta/conf/distro/debian-stretch.list               |  14 +-
>  meta/lib/oe/classutils.py                          |  49 ++++
>  meta/lib/oe/data.py                                |  53 ++++
>  meta/lib/oe/maketype.py                            | 112 ++++++++
>  meta/lib/oe/terminal.py                            | 315 +++++++++++++++++++++
>  meta/lib/oe/types.py                               | 187 ++++++++++++
>  .../isar-bootstrap/isar-bootstrap-host.bb          |  33 +--
>  .../isar-bootstrap/isar-bootstrap-target.bb        |  28 +-
>  .../recipes-core/isar-bootstrap/isar-bootstrap.inc |   5 -
>  meta/recipes-devtools/base-apt/base-apt.bb         |   2 +-
>  .../buildchroot/buildchroot-host.bb                |  19 +-
>  .../buildchroot/buildchroot-target.bb              |  20 +-
>  meta/recipes-devtools/buildchroot/buildchroot.inc  |  26 ++
>  meta/recipes-devtools/buildchroot/files/build.sh   |   2 +-
>  meta/recipes-devtools/buildchroot/files/deps.sh    |   6 +
>  meta/recipes-devtools/sdkchroot/sdkchroot.bb       |  14 +-
>  meta/recipes-kernel/linux/files/build-kernel.sh    |   7 -
>  meta/recipes-kernel/linux/linux-custom.inc         |  21 +-
>  scripts/ci_build.sh                                |   2 +
>  scripts/oe-gnome-terminal-phonehome                |  14 +
>  37 files changed, 1113 insertions(+), 130 deletions(-)
>  create mode 100644 meta-isar/recipes-app/cowsay/cowsay_git.bb
>  create mode 100644 meta/classes/dpkg-gbp.bbclass
>  create mode 100644 meta/classes/terminal.bbclass
>  create mode 100644 meta/lib/oe/classutils.py
>  create mode 100644 meta/lib/oe/data.py
>  create mode 100644 meta/lib/oe/maketype.py
>  create mode 100644 meta/lib/oe/terminal.py
>  create mode 100644 meta/lib/oe/types.py
>  create mode 100755 scripts/oe-gnome-terminal-phonehome
> 
> -- 
> 2.16.4
> 
> -- 
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/cover.1569176231.git.jan.kiszka%40siemens.com.

-- 
Baurzhan Ismagulov
ilbers GmbH
Maria-Merian-Str. 8
D-85521 Ottobrunn
+49 (89) 122 67 24-0
ibr@ilbers.de
http://ilbers.de/
Amtsgericht M�nchen, HRB 214197
Gesch�ftsf�hrer: Baurzhan Ismagulov

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Jan Kiszka]

On 23.09.19 08:33, Baurzhan Ismagulov wrote:
> Hello Jan,
> 
> thanks for the update. I was chasing what appears to be a different race in
> isar-bootstrap that was randomly triggered by unrelated patches, which blocked
> meaningful merging. Unfortunately, it is no more reproducible on any of my
> machines. I'll be merging the patches in groups to speed up the process.

Was the issue occurring with or without my lock-removal patch for isar-bootstrap 
applied? I'm asking because the lock removal surfaced the false sharing issue 
that is now (assumed to be) fixed with this series.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Baurzhan Ismagulov]

On Mon, Sep 23, 2019 at 09:33:35AM +0200, Jan Kiszka wrote:
> Was the issue occurring with or without my lock-removal patch for
> isar-bootstrap applied? I'm asking because the lock removal surfaced the
> false sharing issue that is now (assumed to be) fixed with this series.

I had issues with both, but the error messages were different with and without
the lock-removal patch, that is why I'm for now not convinced we can remove it
safely.

With kind regards,
Baurzhan.

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Baurzhan Ismagulov]

On Sun, Sep 22, 2019 at 08:16:44PM +0200, Jan Kiszka wrote:
> This is primarily a repost of most of my pending patches in their tested
> order on top of current ibr/devel (I suspect that this is the next chunk
> to be merged). As the highest version number of a patch included here
> was v4, the series is tagged as v5.
> 
> I think this is usefull because I had to inject the isar-bootstrap-host
> sharing fix into the middle and adjust a succeeding patch.
> 
> As a bonus, this series ports devshell support over from OE core.

Interim update: ci_build.sh -q -r -s fails on the complete series with:

ERROR: mc:qemuamd64-stretch:isar-bootstrap-target-1.0-r0 do_bootstrap: Function failed: do_bootstrap (log file is located at /workspace/build/isar_ibr_devel/54/build/tmp/work/debian-stretch-amd64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.28360)
...
| E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation

The complete log:
http://ci.isar-build.org:8080/job/isar_ibr_devel/54/consoleFull

The failing rootfs doesn't have gnupg installed, only gpgv. Should gnupg be
installed?

With kind regards,
Baurzhan.

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Jan Kiszka]

On 23.09.19 11:50, Baurzhan Ismagulov wrote:
> On Sun, Sep 22, 2019 at 08:16:44PM +0200, Jan Kiszka wrote:
>> This is primarily a repost of most of my pending patches in their tested
>> order on top of current ibr/devel (I suspect that this is the next chunk
>> to be merged). As the highest version number of a patch included here
>> was v4, the series is tagged as v5.
>>
>> I think this is usefull because I had to inject the isar-bootstrap-host
>> sharing fix into the middle and adjust a succeeding patch.
>>
>> As a bonus, this series ports devshell support over from OE core.
> 
> Interim update: ci_build.sh -q -r -s fails on the complete series with:
> 
> ERROR: mc:qemuamd64-stretch:isar-bootstrap-target-1.0-r0 do_bootstrap: Function failed: do_bootstrap (log file is located at /workspace/build/isar_ibr_devel/54/build/tmp/work/debian-stretch-amd64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.28360)
> ...
> | E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation
> 
> The complete log:
> http://ci.isar-build.org:8080/job/isar_ibr_devel/54/consoleFull

Can't check, I don't find my credential.

> 
> The failing rootfs doesn't have gnupg installed, only gpgv. Should gnupg be
> installed?

As this target (qemuamd64-stretch:isar-image-base) builds fine in kas-isar, I 
suspect it's a problem of your host system. Please check if gnupg2 is installed 
and fix that otherwise.

If you think that this case falsely request that package, then we need to 
cross-check that. The decision is made in get_distro_needs_gpg_support.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Baurzhan Ismagulov]

On Mon, Sep 23, 2019 at 12:07:29PM +0200, Jan Kiszka wrote:
> As this target (qemuamd64-stretch:isar-image-base) builds fine in kas-isar,
> I suspect it's a problem of your host system. Please check if gnupg2 is
> installed and fix that otherwise.

If this is the case, we need an update in the user manual.

With kind regards,
Baurzhan.

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Jan Kiszka]

On 23.09.19 12:23, Baurzhan Ismagulov wrote:
> On Mon, Sep 23, 2019 at 12:07:29PM +0200, Jan Kiszka wrote:
>> As this target (qemuamd64-stretch:isar-image-base) builds fine in kas-isar,
>> I suspect it's a problem of your host system. Please check if gnupg2 is
>> installed and fix that otherwise.
> 
> If this is the case, we need an update in the user manual.
> 

Sure. But this is unrelated to this patch series: GPG signing is a pre-existing 
feature. Fix your CI, please.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Baurzhan Ismagulov]

On Mon, Sep 23, 2019 at 12:28:23PM +0200, Jan Kiszka wrote:
> Sure. But this is unrelated to this patch series: GPG signing is a
> pre-existing feature. Fix your CI, please.

An update: This one is not sporadic. Your series fixes the
isar-bootstrap-target package directory for storing the custom keys, which
triggers the problem inside the chroot. The failing command isn't debootstrap,
it's apt-key called later in isar_bootstrap(). That doesn't fail in next
006a6ed "u-boot-custom: Add control for u-boot-tools package build" because its
aptkeys directory is empty for whatever reason, and the keys are not added to
the chroot. gnupg is installed on the host (otherwise debootstrap wouldn't
succeed, I assume).

I've added OVERRIDES_append = ":gnupg" to local.conf for the signed base-apt
test case. ci_build "-f -r -s" worked, waiting for the complete test.

With kind regards,
Baurzhan.

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Jan Kiszka]

On 25.09.19 10:41, Baurzhan Ismagulov wrote:
> On Mon, Sep 23, 2019 at 12:28:23PM +0200, Jan Kiszka wrote:
>> Sure. But this is unrelated to this patch series: GPG signing is a
>> pre-existing feature. Fix your CI, please.
> 
> An update: This one is not sporadic. Your series fixes the
> isar-bootstrap-target package directory for storing the custom keys, which
> triggers the problem inside the chroot. The failing command isn't debootstrap,
> it's apt-key called later in isar_bootstrap(). That doesn't fail in next
> 006a6ed "u-boot-custom: Add control for u-boot-tools package build" because its
> aptkeys directory is empty for whatever reason, and the keys are not added to
> the chroot. gnupg is installed on the host (otherwise debootstrap wouldn't
> succeed, I assume).
> 
> I've added OVERRIDES_append = ":gnupg" to local.conf for the signed base-apt
> test case. ci_build "-f -r -s" worked, waiting for the complete test.

IOW, we are missing a reason to add gnupg to the chroot when building from a 
signed base-apt, right? Let me check if I can reproduce.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Baurzhan Ismagulov]

On Wed, Sep 25, 2019 at 10:51:02AM +0200, Jan Kiszka wrote:
> > I've added OVERRIDES_append = ":gnupg" to local.conf for the signed base-apt
> > test case. ci_build "-f -r -s" worked, waiting for the complete test.
> 
> IOW, we are missing a reason to add gnupg to the chroot when building from a
> signed base-apt, right?

In this wording, I'd say "no" since it doesn't seem to have been designed to
append gnupg to debootstrap --include dynamically if we have custom keys. IMU,
DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg2" suggests that the user
should add the OVERRIDES manually as well. Andreas?

If that could be done implicitly, it would be better. However, in the current
implementation, the keys are checked after debootstrap, so I haven't started
shuffling stuff around.


> Let me check if I can reproduce.

You should be able to trigger it with -r -s.


With kind regards,
Baurzhan.

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Jan Kiszka]

On 25.09.19 11:22, Baurzhan Ismagulov wrote:
> On Wed, Sep 25, 2019 at 10:51:02AM +0200, Jan Kiszka wrote:
>>> I've added OVERRIDES_append = ":gnupg" to local.conf for the signed base-apt
>>> test case. ci_build "-f -r -s" worked, waiting for the complete test.
>>
>> IOW, we are missing a reason to add gnupg to the chroot when building from a
>> signed base-apt, right?
> 
> In this wording, I'd say "no" since it doesn't seem to have been designed to
> append gnupg to debootstrap --include dynamically if we have custom keys. IMU,
> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg2" suggests that the user
> should add the OVERRIDES manually as well. Andreas?

No, this is defined by get_distro_needs_gpg_support.

> 
> If that could be done implicitly, it would be better. However, in the current
> implementation, the keys are checked after debootstrap, so I haven't started
> shuffling stuff around.
> 

It's probably another condition to be added to that function. Checking...

Jan

> 
>> Let me check if I can reproduce.
> 
> You should be able to trigger it with -r -s.
> 
> 
> With kind regards,
> Baurzhan.
> 


-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Baurzhan Ismagulov]

On Wed, Sep 25, 2019 at 11:26:13AM +0200, Jan Kiszka wrote:
> > In this wording, I'd say "no" since it doesn't seem to have been designed to
> > append gnupg to debootstrap --include dynamically if we have custom keys. IMU,
> > DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg2" suggests that the user
> > should add the OVERRIDES manually as well. Andreas?
> 
> No, this is defined by get_distro_needs_gpg_support.

Ah, it's already there.


> > If that could be done implicitly, it would be better. However, in the current
> > implementation, the keys are checked after debootstrap, so I haven't started
> > shuffling stuff around.
> 
> It's probably another condition to be added to that function. Checking...

If conditional OVERRIDES works, this one might not be necessary.


With kind regards,
Baurzhan.

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Jan Kiszka]

On 25.09.19 11:26, [ext] Jan Kiszka wrote:
> On 25.09.19 11:22, Baurzhan Ismagulov wrote:
>> On Wed, Sep 25, 2019 at 10:51:02AM +0200, Jan Kiszka wrote:
>>>> I've added OVERRIDES_append = ":gnupg" to local.conf for the signed base-apt
>>>> test case. ci_build "-f -r -s" worked, waiting for the complete test.
>>>
>>> IOW, we are missing a reason to add gnupg to the chroot when building from a
>>> signed base-apt, right?
>>
>> In this wording, I'd say "no" since it doesn't seem to have been designed to
>> append gnupg to debootstrap --include dynamically if we have custom keys. IMU,
>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg2" suggests that the user
>> should add the OVERRIDES manually as well. Andreas?
> 
> No, this is defined by get_distro_needs_gpg_support.
> 
>>
>> If that could be done implicitly, it would be better. However, in the current
>> implementation, the keys are checked after debootstrap, so I haven't started
>> shuffling stuff around.
>>
> 
> It's probably another condition to be added to that function. Checking...
> 

I suspect this will fix the issue, but the test runs will take some time:

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 215c41b..991410a 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -169,6 +169,8 @@ def get_distro_needs_https_support(d, is_host=False):
 def get_distro_needs_gpg_support(d):
     apt_keys = d.getVar("DISTRO_BOOTSTRAP_KEYS") or ""
     apt_keys += " " + (d.getVar("THIRD_PARTY_APT_KEYS") or "")
+    if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+        apt_keys += " " + (d.getVar("BASE_REPO_KEY") or "")
     if apt_keys != " ":
         return "gnupg"
     return ""

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Baurzhan Ismagulov]

On Wed, Sep 25, 2019 at 11:35:57AM +0200, Jan Kiszka wrote:
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index 215c41b..991410a 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -169,6 +169,8 @@ def get_distro_needs_https_support(d, is_host=False):
>  def get_distro_needs_gpg_support(d):
>      apt_keys = d.getVar("DISTRO_BOOTSTRAP_KEYS") or ""
>      apt_keys += " " + (d.getVar("THIRD_PARTY_APT_KEYS") or "")
> +    if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> +        apt_keys += " " + (d.getVar("BASE_REPO_KEY") or "")
>      if apt_keys != " ":
>          return "gnupg"
>      return ""

That fails to download gnupg2, since ISAR_USE_CACHED_BASE_REPO is 0 during the
download run. What about adding BASE_REPO_KEY unconditionally?

With kind regards,
Baurzhan.

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Jan Kiszka]

On 25.09.19 12:52, Baurzhan Ismagulov wrote:
> On Wed, Sep 25, 2019 at 11:35:57AM +0200, Jan Kiszka wrote:
>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>> index 215c41b..991410a 100644
>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>> @@ -169,6 +169,8 @@ def get_distro_needs_https_support(d, is_host=False):
>>   def get_distro_needs_gpg_support(d):
>>       apt_keys = d.getVar("DISTRO_BOOTSTRAP_KEYS") or ""
>>       apt_keys += " " + (d.getVar("THIRD_PARTY_APT_KEYS") or "")
>> +    if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>> +        apt_keys += " " + (d.getVar("BASE_REPO_KEY") or "")
>>       if apt_keys != " ":
>>           return "gnupg"
>>       return ""
> 
> That fails to download gnupg2, since ISAR_USE_CACHED_BASE_REPO is 0 during the
> download run. What about adding BASE_REPO_KEY unconditionally?

Yes, that sounds reasonable. Will update and test as well.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Baurzhan Ismagulov]

On Wed, Sep 25, 2019 at 01:05:46PM +0200, Jan Kiszka wrote:
> Yes, that sounds reasonable. Will update and test as well.

ci_build.sh "-f -r -s" worked, continuing the tests with the version below.

With kind regards,
Baurzhan.


From: Jan Kiszka <jan.kiszka@siemens.com>
Date: Mon, 26 Aug 2019 08:26:03 +0200
Subject: [PATCH] isar-bootstrap: Fix keyring generation

Due to immediate expansion, APT_KEYS_DIR and DISTRO_BOOTSTRAP_KEYRING
were set to invalid WORKDIR, rather than the value relevant when running
do_generate_keyrings in recipe build context.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index b49c161..24de667 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -169,6 +169,7 @@ def get_distro_needs_https_support(d, is_host=False):
 def get_distro_needs_gpg_support(d):
     apt_keys = d.getVar("DISTRO_BOOTSTRAP_KEYS") or ""
     apt_keys += " " + (d.getVar("THIRD_PARTY_APT_KEYS") or "")
+    apt_keys += " " + (d.getVar("BASE_REPO_KEY") or "")
     if apt_keys != " ":
         return "gnupg"
     return ""
@@ -188,8 +189,8 @@ def get_distro_components_argument(d, is_host):
     else:
         return ""
 
-APT_KEYS_DIR := "${WORKDIR}/aptkeys"
-DISTRO_BOOTSTRAP_KEYRING := "${WORKDIR}/distro-keyring.gpg"
+APT_KEYS_DIR = "${WORKDIR}/aptkeys"
+DISTRO_BOOTSTRAP_KEYRING = "${WORKDIR}/distro-keyring.gpg"
 
 do_generate_keyrings[cleandirs] = "${APT_KEYS_DIR}"
 do_generate_keyrings[dirs] = "${DL_DIR}"
-- 
2.20.1

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Jan Kiszka]

On 25.09.19 14:08, Baurzhan Ismagulov wrote:
> On Wed, Sep 25, 2019 at 01:05:46PM +0200, Jan Kiszka wrote:
>> Yes, that sounds reasonable. Will update and test as well.
> 
> ci_build.sh "-f -r -s" worked, continuing the tests with the version below.
> 

The fix for get_distro_needs_gpg_support looks unrelated to the fix you cite 
below. Below just unbroke the feature. I will send you a proper separate patch.

Jan

> With kind regards,
> Baurzhan.
> 
> 
> From: Jan Kiszka <jan.kiszka@siemens.com>
> Date: Mon, 26 Aug 2019 08:26:03 +0200
> Subject: [PATCH] isar-bootstrap: Fix keyring generation
> 
> Due to immediate expansion, APT_KEYS_DIR and DISTRO_BOOTSTRAP_KEYRING
> were set to invalid WORKDIR, rather than the value relevant when running
> do_generate_keyrings in recipe build context.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>   meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 +++--
>   1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index b49c161..24de667 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -169,6 +169,7 @@ def get_distro_needs_https_support(d, is_host=False):
>   def get_distro_needs_gpg_support(d):
>       apt_keys = d.getVar("DISTRO_BOOTSTRAP_KEYS") or ""
>       apt_keys += " " + (d.getVar("THIRD_PARTY_APT_KEYS") or "")
> +    apt_keys += " " + (d.getVar("BASE_REPO_KEY") or "")
>       if apt_keys != " ":
>           return "gnupg"
>       return ""
> @@ -188,8 +189,8 @@ def get_distro_components_argument(d, is_host):
>       else:
>           return ""
>   
> -APT_KEYS_DIR := "${WORKDIR}/aptkeys"
> -DISTRO_BOOTSTRAP_KEYRING := "${WORKDIR}/distro-keyring.gpg"
> +APT_KEYS_DIR = "${WORKDIR}/aptkeys"
> +DISTRO_BOOTSTRAP_KEYRING = "${WORKDIR}/distro-keyring.gpg"
>   
>   do_generate_keyrings[cleandirs] = "${APT_KEYS_DIR}"
>   do_generate_keyrings[dirs] = "${DL_DIR}"
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Baurzhan Ismagulov]

On Wed, Sep 25, 2019 at 02:29:41PM +0200, Jan Kiszka wrote:
> The fix for get_distro_needs_gpg_support looks unrelated to the fix you cite
> below. Below just unbroke the feature. I will send you a proper separate
> patch.

Makes sense, ok.

With kind regards,
Baurzhan.

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Henning Schild]

Am Mon, 23 Sep 2019 11:50:26 +0200
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> On Sun, Sep 22, 2019 at 08:16:44PM +0200, Jan Kiszka wrote:
> > This is primarily a repost of most of my pending patches in their
> > tested order on top of current ibr/devel (I suspect that this is
> > the next chunk to be merged). As the highest version number of a
> > patch included here was v4, the series is tagged as v5.
> > 
> > I think this is usefull because I had to inject the
> > isar-bootstrap-host sharing fix into the middle and adjust a
> > succeeding patch.
> > 
> > As a bonus, this series ports devshell support over from OE core.  
> 
> Interim update: ci_build.sh -q -r -s fails on the complete series
> with:

That q should probably be removed. We have those CI machines test this
for hours, loosing the results is a shame.

Henning

> ERROR: mc:qemuamd64-stretch:isar-bootstrap-target-1.0-r0
> do_bootstrap: Function failed: do_bootstrap (log file is located
> at /workspace/build/isar_ibr_devel/54/build/tmp/work/debian-stretch-amd64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.28360) ...
> | E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of
> them is required for this operation
> 
> The complete log:
> http://ci.isar-build.org:8080/job/isar_ibr_devel/54/consoleFull
> 
> The failing rootfs doesn't have gnupg installed, only gpgv. Should
> gnupg be installed?
> 
> With kind regards,
> Baurzhan.
> 

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Baurzhan Ismagulov]

On Mon, Sep 23, 2019 at 04:24:47PM +0200, Henning Schild wrote:
> > Interim update: ci_build.sh -q -r -s fails on the complete series
> 
> That q should probably be removed. We have those CI machines test this
> for hours, loosing the results is a shame.

ci_build.sh -q controls bitbake -v, which prints the commands executed and
their output. I personally like the former and not the latter, but the cost of
the results is indeed more important, especially for sporadic races. We did
have complaints about a single huge output file; that is addressed by Avocado.
Removed for master, master_fast, next, next_fast and ibr* jobs.

With kind regards,
Baurzhan.

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Henning Schild]

Something in this series changes the names of deployed images, removing
the suite.

next:
ls out/build/tmp/deploy/images/qemuamd64/
isar-image-base-debian-buster-qemuamd64-initrd.img
isar-image-base-debian-buster-qemuamd64-vmlinuz
isar-image-base-debian-buster-qemuamd64.wic.img
isar-image-base-debian-buster-qemuamd64.wic.img.bmap
isar-image-base-debian-stretch-qemuamd64-initrd.img
isar-image-base-debian-stretch-qemuamd64-vmlinuz
isar-image-base-debian-stretch-qemuamd64.wic.img
isar-image-base-debian-stretch-qemuamd64.wic.img.bmap

71a932796c407f3158536e3ec6d282fbaa585e17
ls out/build/tmp/deploy/images/qemuamd64/
isar-image-base-1.0-r0-initrd.img
isar-image-base-1.0-r0.wic.img
isar-image-base-1.0-r0.manifest
isar-image-base-1.0-r0.wic.img.bmap
isar-image-base-1.0-r0-vmlinuz

The last one is also a build of stretch and buster, not sure who won
(write its images last).

This also breaks the qemu starting in ilbers-ci ...

Henning

Am Sun, 22 Sep 2019 20:16:44 +0200
schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:

> This is primarily a repost of most of my pending patches in their
> tested order on top of current ibr/devel (I suspect that this is the
> next chunk to be merged). As the highest version number of a patch
> included here was v4, the series is tagged as v5.
> 
> I think this is usefull because I had to inject the
> isar-bootstrap-host sharing fix into the middle and adjust a
> succeeding patch.
> 
> As a bonus, this series ports devshell support over from OE core.
> 
> Jan
> 
> Jan Kiszka (27):
>   buildchroot: Properly deploy the build result
>   sdk: Use clean-package-cache rather than open-coded cleanup
>   sdk: Deploy earlier
>   Remove unneeded PF tweakings
>   image: Stop changing PF, tune WORKDIR and STAMP directly
>   image: Make WORKDIR and STAMPs IMAGE_TYPE-specific
>   Adjust STAMP variables to use PN, rather than PF
>   Align WORKDIR structure with OE
>   isar-bootstrap: Unshare host bootstraps across different target
>     distros
>   isar-bootstrap: Clean up and fix rebuild
>   gitlab-ci: Keep logs as artifacts on failure
>   Detect false sharing of recipes
>   dpkg-base: Move do_deploy_deb before do_build
>   base-apt: Move do_cache_config before do_build
>   wic-img: Use private tmp dir
>   rootfs: Take isar-apt shared lock during repository operations
>   Use deb.debian.org mirrors
>   linux-custom: Control linux-libc-dev deployment manually
>   dpkg-base: Wait for umount to succeed
>   dpkg-base: Permit multiple clean runs
>   buildchroot: Factor out common packages
>   buildchroot: Do not build cross when there are no arch-specific
>     outputs
>   Add git-buildpackage support
>   dbpg-raw: Only list top-level files in <pn>.install
>   Add OE class and library elements for terminal support
>   dpkg-base: Add devshell target
>   linux-custom: Add libncurses-dev to KBUILD_DEPENDS
> 
>  .gitlab-ci.yml                                     |   6 +
>  doc/user_manual.md                                 |  10 +-
>  meta-isar/recipes-app/cowsay/cowsay_git.bb         |   9 +
>  meta/classes/buildchroot.bbclass                   |   2 +-
>  meta/classes/dpkg-base.bbclass                     |  36 ++-
>  meta/classes/dpkg-gbp.bbclass                      |  24 ++
>  meta/classes/dpkg-raw.bbclass                      |   3 +-
>  meta/classes/dpkg.bbclass                          |   2 +
>  meta/classes/image-sdk-extension.bbclass           |  23 +-
>  meta/classes/image.bbclass                         |   5 +-
>  meta/classes/isar-events.bbclass                   |  22 ++
>  meta/classes/rootfs.bbclass                        |  12 +-
>  meta/classes/terminal.bbclass                      | 112 ++++++++
>  meta/classes/wic-img.bbclass                       |   7 +-
>  meta/conf/bitbake.conf                             |  13 +-
>  meta/conf/distro/debian-bullseye.list              |  14 +-
>  meta/conf/distro/debian-buster.list                |  14 +-
>  meta/conf/distro/debian-stretch.list               |  14 +-
>  meta/lib/oe/classutils.py                          |  49 ++++
>  meta/lib/oe/data.py                                |  53 ++++
>  meta/lib/oe/maketype.py                            | 112 ++++++++
>  meta/lib/oe/terminal.py                            | 315
> +++++++++++++++++++++
> meta/lib/oe/types.py                               | 187
> ++++++++++++ .../isar-bootstrap/isar-bootstrap-host.bb          |  33
> +-- .../isar-bootstrap/isar-bootstrap-target.bb        |  28
> +- .../recipes-core/isar-bootstrap/isar-bootstrap.inc |   5 -
> meta/recipes-devtools/base-apt/base-apt.bb         |   2
> +- .../buildchroot/buildchroot-host.bb                |  19
> +- .../buildchroot/buildchroot-target.bb              |  20 +-
> meta/recipes-devtools/buildchroot/buildchroot.inc  |  26 ++
> meta/recipes-devtools/buildchroot/files/build.sh   |   2 +-
> meta/recipes-devtools/buildchroot/files/deps.sh    |   6 +
> meta/recipes-devtools/sdkchroot/sdkchroot.bb       |  14 +-
> meta/recipes-kernel/linux/files/build-kernel.sh    |   7 -
> meta/recipes-kernel/linux/linux-custom.inc         |  21 +-
> scripts/ci_build.sh                                |   2 +
> scripts/oe-gnome-terminal-phonehome                |  14 + 37 files
> changed, 1113 insertions(+), 130 deletions(-) create mode 100644
> meta-isar/recipes-app/cowsay/cowsay_git.bb create mode 100644
> meta/classes/dpkg-gbp.bbclass create mode 100644
> meta/classes/terminal.bbclass create mode 100644
> meta/lib/oe/classutils.py create mode 100644 meta/lib/oe/data.py
>  create mode 100644 meta/lib/oe/maketype.py
>  create mode 100644 meta/lib/oe/terminal.py
>  create mode 100644 meta/lib/oe/types.py
>  create mode 100755 scripts/oe-gnome-terminal-phonehome
> 

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Jan Kiszka]

On 25.09.19 12:23, Henning Schild wrote:
> Something in this series changes the names of deployed images, removing
> the suite.
> 
> next:
> ls out/build/tmp/deploy/images/qemuamd64/
> isar-image-base-debian-buster-qemuamd64-initrd.img
> isar-image-base-debian-buster-qemuamd64-vmlinuz
> isar-image-base-debian-buster-qemuamd64.wic.img
> isar-image-base-debian-buster-qemuamd64.wic.img.bmap
> isar-image-base-debian-stretch-qemuamd64-initrd.img
> isar-image-base-debian-stretch-qemuamd64-vmlinuz
> isar-image-base-debian-stretch-qemuamd64.wic.img
> isar-image-base-debian-stretch-qemuamd64.wic.img.bmap
> 
> 71a932796c407f3158536e3ec6d282fbaa585e17
> ls out/build/tmp/deploy/images/qemuamd64/
> isar-image-base-1.0-r0-initrd.img
> isar-image-base-1.0-r0.wic.img
> isar-image-base-1.0-r0.manifest
> isar-image-base-1.0-r0.wic.img.bmap
> isar-image-base-1.0-r0-vmlinuz
> 
> The last one is also a build of stretch and buster, not sure who won
> (write its images last).
> 
> This also breaks the qemu starting in ilbers-ci ...
> 

Thanks for testing. That comes from the PF cleanup (patch 5). Will address.

We problem need to think properly about the paths and file names. Look rather 
messy. Let's check what OE does...

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Baurzhan Ismagulov]

On Wed, Sep 25, 2019 at 01:41:13PM +0200, Jan Kiszka wrote:
> Thanks for testing. That comes from the PF cleanup (patch 5). Will address.

Meanwhile, I'd like to merge 01-03, 11, 14-16, 18, 20-27
(https://github.com/ilbers/isar/commits/ibr/next). You aren't currently working
on those, are you? The PF changes are not included. The rest of the series
rebases cleanly.

With kind regards,
Baurzhan.

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Jan Kiszka]

On 30.09.19 11:29, Baurzhan Ismagulov wrote:
> On Wed, Sep 25, 2019 at 01:41:13PM +0200, Jan Kiszka wrote:
>> Thanks for testing. That comes from the PF cleanup (patch 5). Will address.
> 
> Meanwhile, I'd like to merge 01-03, 11, 14-16, 18, 20-27
> (https://github.com/ilbers/isar/commits/ibr/next). You aren't currently working
> on those, are you? The PF changes are not included. The rest of the series
> rebases cleanly.

Nope, all functional fixes and enhancements were pushed in their latest 
versions. https://github.com/siemens/isar/commits/jan/next passed a full CI run 
this night (5:12...). I'm now just working on making this run only when there 
are changes. Then you will get a patch for that as well (.gitlab-ci.yml).

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Baurzhan Ismagulov]

On Mon, Sep 30, 2019 at 11:29:22AM +0200, Baurzhan Ismagulov wrote:
> Meanwhile, I'd like to merge 01-03, 11, 14-16, 18, 20-27

Merged to next, thanks.

With kind regards,
Baurzhan.

Re: [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell

[Baurzhan Ismagulov]

On Sun, Sep 22, 2019 at 08:16:44PM +0200, Jan Kiszka wrote:
> This is primarily a repost of most of my pending patches in their tested
> order on top of current ibr/devel (I suspect that this is the next chunk
> to be merged). As the highest version number of a patch included here
> was v4, the series is tagged as v5.
> 
> I think this is usefull because I had to inject the isar-bootstrap-host
> sharing fix into the middle and adjust a succeeding patch.
> 
> As a bonus, this series ports devshell support over from OE core.

Applied 04-10, 12 to next, thanks.

With kind regards,
Baurzhan.