[PATCH 00/26] Tons of fixes and cleanups

[PATCH 00/26] Tons of fixes and cleanups

[Jan Kiszka]

Highlights:
- fixed custom DISTRO_BOOTSTRAP_KEYS
- fixed shared lock across dpkg_runbuild (nasty stuff...)
- reworked STAMPing
- fixed follow-up errors on build terminations (error or ctrl-c)
- fixed two false sharings
- detection of false task sharing
- stop mangling PF
- no more tasks beyond do_build
- fixed isar-bootstrap rebuild
- deploy logs on gitlab-ci errors

The queue can also be found at

https://github.com/siemens/isar jan/queue

It passed ci_build.sh -q -f and builds jailhouse-images fine. But these
are many significant changes, so please review carefully.

Jan


CC: Cedric Hombourger <Cedric_Hombourger@mentor.com>

Jan Kiszka (26):
  isar-bootstrap: Consider DISTRO_BOOTSTRAP_KEYS for gnupg installation
  isar-bootstrap: Fix keyring generation
  isar-bootstrap: Install DISTRO_BOOTSTRAP_KEYS
  Rework STAMP layout and cleaning
  dpkg: Fix shared-locking of build task
  dpkg-base: Free do_build task
  base: Clean up build task
  isar-events: Do not start cleanup early
  meta-isar: Move bananapi to buster, remove bogus example
  bitbake.conf: Reorder PV definition
  example-module: Fix build with mipsel target
  buildchroot: Ignore ISAR_CROSS_COMPILE for i386 on stretch
  buildchroot: Fix downgrades when installing dependencies
  buildchroot: Properly deploy the build result
  sdk: Use clean-package-cache rather than open-coded cleanup
  sdk: Deploy earlier
  Remove unneeded PF tweakings
  image: Stop changing PF, tune WORKDIR and STAMP directly
  image: Make WORKDIR and STAMPs IMAGE_TYPE-specific
  Adjust STAMP variables to use PN, rather than PF
  Align WORKDIR structure with OE
  isar-bootstrap: Clean up and fix rebuild
  gitlab-ci: Keep logs as artifacts on failure
  Detect false sharing of recipes
  dpkg-base: Move do_deploy_deb before do_build
  base-apt: Move do_cache_config before do_build

 .gitlab-ci.yml                                     |  6 +++++
 doc/user_manual.md                                 |  2 +-
 .../conf/distro/debian-stretch-backports.list      |  2 --
 meta-isar/conf/local.conf.sample                   |  2 +-
 meta-isar/conf/machine/bananapi.conf               |  3 ---
 ...{bananapi-stretch.conf => bananapi-buster.conf} |  2 +-
 .../conf/multiconfig/preferences.bananapi.conf     |  7 -----
 .../example-module/example-module.bb               |  1 +
 meta/classes/base.bbclass                          | 15 +++++------
 meta/classes/buildchroot.bbclass                   |  6 +++--
 meta/classes/dpkg-base.bbclass                     | 25 +++++++++--------
 meta/classes/dpkg-raw.bbclass                      |  1 -
 meta/classes/dpkg.bbclass                          |  8 +++---
 meta/classes/image-sdk-extension.bbclass           | 23 ++++++----------
 meta/classes/image.bbclass                         |  5 +++-
 meta/classes/isar-events.bbclass                   | 28 ++++++++++++++++---
 meta/classes/patch.bbclass                         |  2 --
 meta/classes/template.bbclass                      |  1 -
 meta/conf/bitbake.conf                             | 15 +++++++----
 .../isar-bootstrap/isar-bootstrap-host.bb          | 31 ++++++++++------------
 .../isar-bootstrap/isar-bootstrap-target.bb        | 28 +++++++++----------
 .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 15 +++++------
 meta/recipes-devtools/base-apt/base-apt.bb         |  2 +-
 .../buildchroot/buildchroot-host.bb                |  3 ++-
 .../buildchroot/buildchroot-target.bb              |  3 ++-
 meta/recipes-devtools/buildchroot/buildchroot.inc  | 13 +++++++++
 meta/recipes-devtools/buildchroot/files/deps.sh    |  2 +-
 meta/recipes-devtools/sdkchroot/sdkchroot.bb       | 14 ++++++++--
 meta/recipes-kernel/linux/linux-custom.inc         |  3 +--
 29 files changed, 146 insertions(+), 122 deletions(-)
 delete mode 100644 meta-isar/conf/distro/debian-stretch-backports.list
 rename meta-isar/conf/multiconfig/{bananapi-stretch.conf => bananapi-buster.conf} (82%)
 delete mode 100644 meta-isar/conf/multiconfig/preferences.bananapi.conf

-- 
2.16.4

[PATCH 01/26] isar-bootstrap: Consider DISTRO_BOOTSTRAP_KEYS for gnupg installation

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

We also need gnupg when using a custom bootstrap key.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 1eb2ac6..bcd644b 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -169,8 +169,9 @@ def get_distro_needs_https_support(d, is_host=False):
         return ""
 
 def get_distro_needs_gpg_support(d):
-    apt_keys = d.getVar("THIRD_PARTY_APT_KEYS")
-    if apt_keys and apt_keys != "":
+    apt_keys = d.getVar("DISTRO_BOOTSTRAP_KEYS") or ""
+    apt_keys += " " + (d.getVar("THIRD_PARTY_APT_KEYS") or "")
+    if apt_keys != " ":
         return "gnupg"
     return ""
 
-- 
2.16.4

[PATCH 02/26] isar-bootstrap: Fix keyring generation

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Due to immediate expansion, APT_KEYS_DIR and DISTRO_BOOTSTRAP_KEYRING
were set to invalid WORKDIR, rather than the value relevant when running
do_generate_keyrings in recipe build context.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index bcd644b..d2210d7 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -190,8 +190,8 @@ def get_distro_components_argument(d, is_host):
     else:
         return ""
 
-APT_KEYS_DIR := "${WORKDIR}/aptkeys"
-DISTRO_BOOTSTRAP_KEYRING := "${WORKDIR}/distro-keyring.gpg"
+APT_KEYS_DIR = "${WORKDIR}/aptkeys"
+DISTRO_BOOTSTRAP_KEYRING = "${WORKDIR}/distro-keyring.gpg"
 
 do_generate_keyrings[cleandirs] = "${APT_KEYS_DIR}"
 do_generate_keyrings[dirs] = "${DL_DIR}"
-- 
2.16.4

Re: [PATCH 02/26] isar-bootstrap: Fix keyring generation

[Baurzhan Ismagulov]

On Mon, Aug 26, 2019 at 08:26:03AM +0200, Jan Kiszka wrote:
> Due to immediate expansion, APT_KEYS_DIR and DISTRO_BOOTSTRAP_KEYRING
> were set to invalid WORKDIR, rather than the value relevant when running
> do_generate_keyrings in recipe build context.

Thanks, merged 02-08, 10, 11 to next.

With kind regards,
Baurzhan.

[PATCH 03/26] isar-bootstrap: Install DISTRO_BOOTSTRAP_KEYS

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

We use the DISTRO_BOOTSTRAP_KEYS already for bootstrapping - but we fail
to install them on the generated rootfs for later use in buildchroots or
target rootfs.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index d2210d7..aea89a9 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -206,6 +206,7 @@ do_generate_keyrings() {
     if [ -n "${@d.getVar("DISTRO_BOOTSTRAP_KEYFILES", True) or ""}" ]; then
         for keyfile in ${@d.getVar("DISTRO_BOOTSTRAP_KEYFILES", True)}; do
            sudo apt-key --keyring "${DISTRO_BOOTSTRAP_KEYRING}" add $keyfile
+           cp "$keyfile" "${APT_KEYS_DIR}"/"$(basename "$keyfile")"
         done
     fi
 }
-- 
2.16.4

[PATCH 04/26] Rework STAMP layout and cleaning

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Follow OE more closely by aligning the stamps folder structure to
workdir. This may resolve false sharing of stamps between recipes of
common PF but different DISTRO/DISTRO_ARCH without the otherwise
required stamp-extra-info. That extra-info can now be removed.

As long as recipes mess with PF, we can't use OE's pattern as-is which
is PN-based.

Furthermore add proper STAMPCLEAN. This ensures that stamp files are
cleaned when a task is rebuilt. Not providing this pattern likely caused
a lot of failures when doing partial rebuilds.

As we are at it, switch do_clean to STAMPCLEAN as well.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/base.bbclass                  | 5 ++---
 meta/classes/dpkg-base.bbclass             | 6 ------
 meta/classes/dpkg-raw.bbclass              | 1 -
 meta/classes/dpkg.bbclass                  | 1 -
 meta/classes/patch.bbclass                 | 2 --
 meta/classes/template.bbclass              | 1 -
 meta/conf/bitbake.conf                     | 4 +++-
 meta/recipes-kernel/linux/linux-custom.inc | 1 -
 8 files changed, 5 insertions(+), 16 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index ca80bf8..1d2852a 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -163,7 +163,6 @@ python do_fetch() {
 addtask fetch before do_build
 
 do_unpack[dirs] = "${WORKDIR}"
-do_unpack[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 # Unpack package and put it into working directory
 python do_unpack() {
@@ -206,8 +205,8 @@ python do_clean() {
     workdir = d.expand("${WORKDIR}")
     subprocess.check_call(["sudo", "rm", "-rf", workdir])
 
-    stamppath = bb.data.expand(d.getVar('STAMP', False), d)
-    stampdirs = glob.glob(stamppath + ".*")
+    stampclean = bb.data.expand(d.getVar('STAMPCLEAN', False), d)
+    stampdirs = glob.glob(stampclean)
     subprocess.check_call(["sudo", "rm", "-rf"] + stampdirs)
 }
 
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 39c8acb..2d26418 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -16,7 +16,6 @@ do_adjust_git() {
 }
 
 addtask adjust_git after do_unpack before do_patch
-do_adjust_git[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 inherit patch
 addtask patch after do_adjust_git before do_build
@@ -42,7 +41,6 @@ do_apt_fetch() {
 
 addtask apt_fetch after do_unpack before do_patch
 do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-do_apt_fetch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 def get_package_srcdir(d):
     s = os.path.abspath(d.getVar("S", True))
@@ -60,15 +58,12 @@ def get_package_srcdir(d):
 PP = "/home/builder/${PN}"
 PPS ?= "${@get_package_srcdir(d)}"
 
-do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-
 # Empty do_prepare_build() implementation, to be overwritten if needed
 do_prepare_build() {
     true
 }
 
 addtask prepare_build after do_patch do_transform_template before do_build
-do_prepare_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 # If Isar recipes depend on each other, they typically need the package
 # deployed to isar-apt
 do_prepare_build[deptask] = "do_deploy_deb"
@@ -132,7 +127,6 @@ do_deploy_deb() {
 }
 
 addtask deploy_deb after do_build
-do_deploy_deb[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 do_deploy_deb[depends] = "isar-apt:do_cache_config"
 do_deploy_deb[dirs] = "${S}"
diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index ea03ea4..f8dfb2d 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -16,7 +16,6 @@ do_install() {
 }
 
 do_install[cleandirs] = "${D}"
-do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 addtask install after do_unpack before do_prepare_build
 
 do_prepare_build[cleandirs] += "${D}/debian"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 997f2f3..dd123c3 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -14,7 +14,6 @@ do_install_builddeps() {
 addtask install_builddeps after do_prepare_build before do_build
 # apt and reprepro may not run in parallel, acquire the Isar lock
 do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-do_install_builddeps[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 # Build package from sources using build script
 dpkg_runbuild() {
diff --git a/meta/classes/patch.bbclass b/meta/classes/patch.bbclass
index 2662702..c5ba463 100644
--- a/meta/classes/patch.bbclass
+++ b/meta/classes/patch.bbclass
@@ -78,5 +78,3 @@ python do_patch() {
         except bb.fetch2.BBFetchException as e:
             raise bb.build.FuncFailed(e)
 }
-
-do_patch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
diff --git a/meta/classes/template.bbclass b/meta/classes/template.bbclass
index f3b5f80..fb9d118 100644
--- a/meta/classes/template.bbclass
+++ b/meta/classes/template.bbclass
@@ -7,7 +7,6 @@ TEMPLATE_FILES ?= ""
 TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER DISTRO_ARCH"
 
 do_transform_template[vardeps] = "TEMPLATE_FILES ${TEMPLATE_VARS}"
-do_transform_template[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 python do_transform_template() {
     import subprocess, contextlib
 
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 3d18223..c41f294 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -35,7 +35,9 @@ PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0
 S = "${WORKDIR}/${P}"
 AUTOREV = "${@bb.fetch2.get_autorev(d)}"
 SRC_URI = "file://${FILE}"
-STAMP = "${TMPDIR}/stamps/${PF}"
+STAMPS_DIR ?= "${TMPDIR}/stamps"
+STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PF}/${PV}-${PR}"
+STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PF}/*-*"
 SVNDIR = "${DL_DIR}/svn"
 T = "${WORKDIR}/temp"
 TMPDIR = "${TOPDIR}/tmp"
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index ee5f20c..12d8d9c 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -44,7 +44,6 @@ do_install_builddeps() {
 addtask install_builddeps after do_prepare_build before do_build
 # apt and reprepro may not run in parallel, acquire the Isar lock
 do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-do_install_builddeps[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 dpkg_runbuild() {
 	chmod +x ${WORKDIR}/build-kernel.sh
-- 
2.16.4

Re: [PATCH 04/26] Rework STAMP layout and cleaning

[Claudius Heine]

Hi Jan,

On 26/08/2019 08.26, [ext] Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> Follow OE more closely by aligning the stamps folder structure to
> workdir. This may resolve false sharing of stamps between recipes of
> common PF but different DISTRO/DISTRO_ARCH without the otherwise
> required stamp-extra-info. That extra-info can now be removed.
> 
> As long as recipes mess with PF, we can't use OE's pattern as-is which
> is PN-based.
> 
> Furthermore add proper STAMPCLEAN. This ensures that stamp files are
> cleaned when a task is rebuilt. Not providing this pattern likely caused
> a lot of failures when doing partial rebuilds.
> 
> As we are at it, switch do_clean to STAMPCLEAN as well.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>   meta/classes/base.bbclass                  | 5 ++---
>   meta/classes/dpkg-base.bbclass             | 6 ------
>   meta/classes/dpkg-raw.bbclass              | 1 -
>   meta/classes/dpkg.bbclass                  | 1 -
>   meta/classes/patch.bbclass                 | 2 --
>   meta/classes/template.bbclass              | 1 -
>   meta/conf/bitbake.conf                     | 4 +++-
>   meta/recipes-kernel/linux/linux-custom.inc | 1 -
>   8 files changed, 5 insertions(+), 16 deletions(-)
> 
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index ca80bf8..1d2852a 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -163,7 +163,6 @@ python do_fetch() {
>   addtask fetch before do_build
>   
>   do_unpack[dirs] = "${WORKDIR}"
> -do_unpack[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>   
>   # Unpack package and put it into working directory
>   python do_unpack() {
> @@ -206,8 +205,8 @@ python do_clean() {
>       workdir = d.expand("${WORKDIR}")
>       subprocess.check_call(["sudo", "rm", "-rf", workdir])
>   
> -    stamppath = bb.data.expand(d.getVar('STAMP', False), d)
> -    stampdirs = glob.glob(stamppath + ".*")
> +    stampclean = bb.data.expand(d.getVar('STAMPCLEAN', False), d)
> +    stampdirs = glob.glob(stampclean)
>       subprocess.check_call(["sudo", "rm", "-rf"] + stampdirs)
>   }
>   
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 39c8acb..2d26418 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -16,7 +16,6 @@ do_adjust_git() {
>   }
>   
>   addtask adjust_git after do_unpack before do_patch
> -do_adjust_git[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>   
>   inherit patch
>   addtask patch after do_adjust_git before do_build
> @@ -42,7 +41,6 @@ do_apt_fetch() {
>   
>   addtask apt_fetch after do_unpack before do_patch
>   do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
> -do_apt_fetch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>   
>   def get_package_srcdir(d):
>       s = os.path.abspath(d.getVar("S", True))
> @@ -60,15 +58,12 @@ def get_package_srcdir(d):
>   PP = "/home/builder/${PN}"
>   PPS ?= "${@get_package_srcdir(d)}"
>   
> -do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> -
>   # Empty do_prepare_build() implementation, to be overwritten if needed
>   do_prepare_build() {
>       true
>   }
>   
>   addtask prepare_build after do_patch do_transform_template before do_build
> -do_prepare_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>   # If Isar recipes depend on each other, they typically need the package
>   # deployed to isar-apt
>   do_prepare_build[deptask] = "do_deploy_deb"
> @@ -132,7 +127,6 @@ do_deploy_deb() {
>   }
>   
>   addtask deploy_deb after do_build
> -do_deploy_deb[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>   do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
>   do_deploy_deb[depends] = "isar-apt:do_cache_config"
>   do_deploy_deb[dirs] = "${S}"
> diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
> index ea03ea4..f8dfb2d 100644
> --- a/meta/classes/dpkg-raw.bbclass
> +++ b/meta/classes/dpkg-raw.bbclass
> @@ -16,7 +16,6 @@ do_install() {
>   }
>   
>   do_install[cleandirs] = "${D}"
> -do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>   addtask install after do_unpack before do_prepare_build
>   
>   do_prepare_build[cleandirs] += "${D}/debian"
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index 997f2f3..dd123c3 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -14,7 +14,6 @@ do_install_builddeps() {
>   addtask install_builddeps after do_prepare_build before do_build
>   # apt and reprepro may not run in parallel, acquire the Isar lock
>   do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
> -do_install_builddeps[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>   
>   # Build package from sources using build script
>   dpkg_runbuild() {
> diff --git a/meta/classes/patch.bbclass b/meta/classes/patch.bbclass
> index 2662702..c5ba463 100644
> --- a/meta/classes/patch.bbclass
> +++ b/meta/classes/patch.bbclass
> @@ -78,5 +78,3 @@ python do_patch() {
>           except bb.fetch2.BBFetchException as e:
>               raise bb.build.FuncFailed(e)
>   }
> -
> -do_patch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> diff --git a/meta/classes/template.bbclass b/meta/classes/template.bbclass
> index f3b5f80..fb9d118 100644
> --- a/meta/classes/template.bbclass
> +++ b/meta/classes/template.bbclass
> @@ -7,7 +7,6 @@ TEMPLATE_FILES ?= ""
>   TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER DISTRO_ARCH"
>   
>   do_transform_template[vardeps] = "TEMPLATE_FILES ${TEMPLATE_VARS}"
> -do_transform_template[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>   python do_transform_template() {
>       import subprocess, contextlib
>   
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 3d18223..c41f294 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -35,7 +35,9 @@ PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0
>   S = "${WORKDIR}/${P}"
>   AUTOREV = "${@bb.fetch2.get_autorev(d)}"
>   SRC_URI = "file://${FILE}"
> -STAMP = "${TMPDIR}/stamps/${PF}"
> +STAMPS_DIR ?= "${TMPDIR}/stamps"
> +STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PF}/${PV}-${PR}"

Can you explain the issue of the PN/PF usage a bit more. I did not 
understand:

> As long as recipes mess with PF, we can't use OE's pattern as-is which
> is PN-based.

The recipes that mess with PF are: buildchroot-*.bb sdkchroot.bb 
isar-bootstrap-*.bb and image.bbclass. All of which do not really have a 
version.

So I would think that:

STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/${PF}"
STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/*-*"

should work. Or am I missing something?

regards,
Claudius


> +STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PF}/*-*"
>   SVNDIR = "${DL_DIR}/svn"
>   T = "${WORKDIR}/temp"
>   TMPDIR = "${TOPDIR}/tmp"
> diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
> index ee5f20c..12d8d9c 100644
> --- a/meta/recipes-kernel/linux/linux-custom.inc
> +++ b/meta/recipes-kernel/linux/linux-custom.inc
> @@ -44,7 +44,6 @@ do_install_builddeps() {
>   addtask install_builddeps after do_prepare_build before do_build
>   # apt and reprepro may not run in parallel, acquire the Isar lock
>   do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
> -do_install_builddeps[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>   
>   dpkg_runbuild() {
>   	chmod +x ${WORKDIR}/build-kernel.sh
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 04/26] Rework STAMP layout and cleaning

[Jan Kiszka]

On 26.08.19 10:30, Claudius Heine wrote:
> Hi Jan,
> 
> On 26/08/2019 08.26, [ext] Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> Follow OE more closely by aligning the stamps folder structure to
>> workdir. This may resolve false sharing of stamps between recipes of
>> common PF but different DISTRO/DISTRO_ARCH without the otherwise
>> required stamp-extra-info. That extra-info can now be removed.
>>
>> As long as recipes mess with PF, we can't use OE's pattern as-is which
>> is PN-based.
>>
>> Furthermore add proper STAMPCLEAN. This ensures that stamp files are
>> cleaned when a task is rebuilt. Not providing this pattern likely caused
>> a lot of failures when doing partial rebuilds.
>>
>> As we are at it, switch do_clean to STAMPCLEAN as well.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>   meta/classes/base.bbclass                  | 5 ++---
>>   meta/classes/dpkg-base.bbclass             | 6 ------
>>   meta/classes/dpkg-raw.bbclass              | 1 -
>>   meta/classes/dpkg.bbclass                  | 1 -
>>   meta/classes/patch.bbclass                 | 2 --
>>   meta/classes/template.bbclass              | 1 -
>>   meta/conf/bitbake.conf                     | 4 +++-
>>   meta/recipes-kernel/linux/linux-custom.inc | 1 -
>>   8 files changed, 5 insertions(+), 16 deletions(-)
>>
>> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
>> index ca80bf8..1d2852a 100644
>> --- a/meta/classes/base.bbclass
>> +++ b/meta/classes/base.bbclass
>> @@ -163,7 +163,6 @@ python do_fetch() {
>>   addtask fetch before do_build
>>   do_unpack[dirs] = "${WORKDIR}"
>> -do_unpack[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   # Unpack package and put it into working directory
>>   python do_unpack() {
>> @@ -206,8 +205,8 @@ python do_clean() {
>>       workdir = d.expand("${WORKDIR}")
>>       subprocess.check_call(["sudo", "rm", "-rf", workdir])
>> -    stamppath = bb.data.expand(d.getVar('STAMP', False), d)
>> -    stampdirs = glob.glob(stamppath + ".*")
>> +    stampclean = bb.data.expand(d.getVar('STAMPCLEAN', False), d)
>> +    stampdirs = glob.glob(stampclean)
>>       subprocess.check_call(["sudo", "rm", "-rf"] + stampdirs)
>>   }
>> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
>> index 39c8acb..2d26418 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -16,7 +16,6 @@ do_adjust_git() {
>>   }
>>   addtask adjust_git after do_unpack before do_patch
>> -do_adjust_git[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   inherit patch
>>   addtask patch after do_adjust_git before do_build
>> @@ -42,7 +41,6 @@ do_apt_fetch() {
>>   addtask apt_fetch after do_unpack before do_patch
>>   do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
>> -do_apt_fetch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   def get_package_srcdir(d):
>>       s = os.path.abspath(d.getVar("S", True))
>> @@ -60,15 +58,12 @@ def get_package_srcdir(d):
>>   PP = "/home/builder/${PN}"
>>   PPS ?= "${@get_package_srcdir(d)}"
>> -do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>> -
>>   # Empty do_prepare_build() implementation, to be overwritten if needed
>>   do_prepare_build() {
>>       true
>>   }
>>   addtask prepare_build after do_patch do_transform_template before do_build
>> -do_prepare_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   # If Isar recipes depend on each other, they typically need the package
>>   # deployed to isar-apt
>>   do_prepare_build[deptask] = "do_deploy_deb"
>> @@ -132,7 +127,6 @@ do_deploy_deb() {
>>   }
>>   addtask deploy_deb after do_build
>> -do_deploy_deb[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
>>   do_deploy_deb[depends] = "isar-apt:do_cache_config"
>>   do_deploy_deb[dirs] = "${S}"
>> diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
>> index ea03ea4..f8dfb2d 100644
>> --- a/meta/classes/dpkg-raw.bbclass
>> +++ b/meta/classes/dpkg-raw.bbclass
>> @@ -16,7 +16,6 @@ do_install() {
>>   }
>>   do_install[cleandirs] = "${D}"
>> -do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   addtask install after do_unpack before do_prepare_build
>>   do_prepare_build[cleandirs] += "${D}/debian"
>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>> index 997f2f3..dd123c3 100644
>> --- a/meta/classes/dpkg.bbclass
>> +++ b/meta/classes/dpkg.bbclass
>> @@ -14,7 +14,6 @@ do_install_builddeps() {
>>   addtask install_builddeps after do_prepare_build before do_build
>>   # apt and reprepro may not run in parallel, acquire the Isar lock
>>   do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
>> -do_install_builddeps[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   # Build package from sources using build script
>>   dpkg_runbuild() {
>> diff --git a/meta/classes/patch.bbclass b/meta/classes/patch.bbclass
>> index 2662702..c5ba463 100644
>> --- a/meta/classes/patch.bbclass
>> +++ b/meta/classes/patch.bbclass
>> @@ -78,5 +78,3 @@ python do_patch() {
>>           except bb.fetch2.BBFetchException as e:
>>               raise bb.build.FuncFailed(e)
>>   }
>> -
>> -do_patch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>> diff --git a/meta/classes/template.bbclass b/meta/classes/template.bbclass
>> index f3b5f80..fb9d118 100644
>> --- a/meta/classes/template.bbclass
>> +++ b/meta/classes/template.bbclass
>> @@ -7,7 +7,6 @@ TEMPLATE_FILES ?= ""
>>   TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER DISTRO_ARCH"
>>   do_transform_template[vardeps] = "TEMPLATE_FILES ${TEMPLATE_VARS}"
>> -do_transform_template[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   python do_transform_template() {
>>       import subprocess, contextlib
>> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
>> index 3d18223..c41f294 100644
>> --- a/meta/conf/bitbake.conf
>> +++ b/meta/conf/bitbake.conf
>> @@ -35,7 +35,9 @@ PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', 
>> False),d)[1] or '1.0
>>   S = "${WORKDIR}/${P}"
>>   AUTOREV = "${@bb.fetch2.get_autorev(d)}"
>>   SRC_URI = "file://${FILE}"
>> -STAMP = "${TMPDIR}/stamps/${PF}"
>> +STAMPS_DIR ?= "${TMPDIR}/stamps"
>> +STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PF}/${PV}-${PR}"
> 
> Can you explain the issue of the PN/PF usage a bit more. I did not understand:
> 
>> As long as recipes mess with PF, we can't use OE's pattern as-is which
>> is PN-based.
> 
> The recipes that mess with PF are: buildchroot-*.bb sdkchroot.bb 
> isar-bootstrap-*.bb and image.bbclass. All of which do not really have a version.
> 
> So I would think that:
> 
> STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/${PF}"
> STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/*-*"
> 
> should work. Or am I missing something?

PF is a effectively read-only. This is fixed later on this series. Then there is 
no need to deviate from OE any more.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[PATCH 05/26] dpkg: Fix shared-locking of build task

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Bitbake file locks are not just exclusive flock calls. For whatever
reason, bitbake prefers to delete the lock file before unlocking them.
That obviously creates a nasty race with other locker requests: Those
already waiting will get access to the removed lock, those coming later
will create a new one. To resolve that race, bitbakes lockfile() tests
if the inode of the acquired lock file is identical to the one that is
currently visible via the lock name. If not, it retries the lock. What a
pointless mess...

Our open-coded "flock -s" does not do that, thus will often acquire only
a stale lock. Fix that by using bitbake's services for locking. That
requires to move the lock acquisition at the top level of the task, ie.
do_build. Thus, also dpkg-raw and kernel builds are now under the shared
(read) lock. The better fix for this issue would actually be private
buildchroots for each build, but that will come later.

CC: Cedric Hombourger <Cedric_Hombourger@mentor.com>
Fixes: 568072765b08 ("dpkg: acquire a read (shared) locks while building packages")
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 11 +++++++----
 meta/classes/dpkg.bbclass      |  5 ++---
 2 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 2d26418..8146328 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -87,10 +87,13 @@ dpkg_runbuild() {
     die "This should never be called, overwrite it in your derived class"
 }
 
-do_build() {
-    dpkg_do_mounts
-    dpkg_runbuild
-    dpkg_undo_mounts
+python do_build() {
+    lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
+                             shared=True)
+    bb.build.exec_func("dpkg_do_mounts", d)
+    bb.build.exec_func("dpkg_runbuild", d)
+    bb.build.exec_func("dpkg_undo_mounts", d)
+    bb.utils.unlockfile(lock)
 }
 
 CLEANFUNCS += "repo_clean"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index dd123c3..df47ced 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -18,7 +18,6 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 # Build package from sources using build script
 dpkg_runbuild() {
     E="${@ bb.utils.export_proxies(d)}"
-    flock -s "${REPO_ISAR_DIR}/isar.lock" \
-        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-             /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH}
+    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+         /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH}
 }
-- 
2.16.4

Re: [PATCH 05/26] dpkg: Fix shared-locking of build task

[Cedric Hombourger]

On 8/26/2019 8:26 AM, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
>
> Bitbake file locks are not just exclusive flock calls. For whatever
> reason, bitbake prefers to delete the lock file before unlocking them.
> That obviously creates a nasty race with other locker requests: Those
> already waiting will get access to the removed lock, those coming later
> will create a new one. To resolve that race, bitbakes lockfile() tests
> if the inode of the acquired lock file is identical to the one that is
> currently visible via the lock name. If not, it retries the lock. What a
> pointless mess...
>
> Our open-coded "flock -s" does not do that, thus will often acquire only
> a stale lock. Fix that by using bitbake's services for locking. That
> requires to move the lock acquisition at the top level of the task, ie.
> do_build. Thus, also dpkg-raw and kernel builds are now under the shared
> (read) lock. The better fix for this issue would actually be private
> buildchroots for each build, but that will come later.

Very good catch! ack

> CC: Cedric Hombourger <Cedric_Hombourger@mentor.com>
> Fixes: 568072765b08 ("dpkg: acquire a read (shared) locks while building packages")
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>   meta/classes/dpkg-base.bbclass | 11 +++++++----
>   meta/classes/dpkg.bbclass      |  5 ++---
>   2 files changed, 9 insertions(+), 7 deletions(-)
>
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 2d26418..8146328 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -87,10 +87,13 @@ dpkg_runbuild() {
>       die "This should never be called, overwrite it in your derived class"
>   }
>   
> -do_build() {
> -    dpkg_do_mounts
> -    dpkg_runbuild
> -    dpkg_undo_mounts
> +python do_build() {
> +    lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
> +                             shared=True)
> +    bb.build.exec_func("dpkg_do_mounts", d)
> +    bb.build.exec_func("dpkg_runbuild", d)
> +    bb.build.exec_func("dpkg_undo_mounts", d)
> +    bb.utils.unlockfile(lock)
>   }
>   
>   CLEANFUNCS += "repo_clean"
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index dd123c3..df47ced 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -18,7 +18,6 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
>   # Build package from sources using build script
>   dpkg_runbuild() {
>       E="${@ bb.utils.export_proxies(d)}"
> -    flock -s "${REPO_ISAR_DIR}/isar.lock" \
> -        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
> -             /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH}
> +    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
> +         /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH}
>   }

Re: [PATCH 05/26] dpkg: Fix shared-locking of build task

[Claudius Heine]

Hi Jan,

On 26/08/2019 08.26, [ext] Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> Bitbake file locks are not just exclusive flock calls. For whatever
> reason, bitbake prefers to delete the lock file before unlocking them.
> That obviously creates a nasty race with other locker requests: Those
> already waiting will get access to the removed lock, those coming later
> will create a new one. To resolve that race, bitbakes lockfile() tests
> if the inode of the acquired lock file is identical to the one that is
> currently visible via the lock name. If not, it retries the lock. What a
> pointless mess...
> 
> Our open-coded "flock -s" does not do that, thus will often acquire only
> a stale lock. Fix that by using bitbake's services for locking. That
> requires to move the lock acquisition at the top level of the task, ie.
> do_build. Thus, also dpkg-raw and kernel builds are now under the shared
> (read) lock. The better fix for this issue would actually be private
> buildchroots for each build, but that will come later.
> 
> CC: Cedric Hombourger <Cedric_Hombourger@mentor.com>
> Fixes: 568072765b08 ("dpkg: acquire a read (shared) locks while building packages")
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>   meta/classes/dpkg-base.bbclass | 11 +++++++----
>   meta/classes/dpkg.bbclass      |  5 ++---
>   2 files changed, 9 insertions(+), 7 deletions(-)
> 
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 2d26418..8146328 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -87,10 +87,13 @@ dpkg_runbuild() {
>       die "This should never be called, overwrite it in your derived class"
>   }
>   
> -do_build() {
> -    dpkg_do_mounts
> -    dpkg_runbuild
> -    dpkg_undo_mounts
> +python do_build() {
> +    lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
> +                             shared=True)
> +    bb.build.exec_func("dpkg_do_mounts", d)
> +    bb.build.exec_func("dpkg_runbuild", d)
> +    bb.build.exec_func("dpkg_undo_mounts", d)
> +    bb.utils.unlockfile(lock)

What is the difference between this and

do_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"

?

regards,
Claudius

>   }
>   
>   CLEANFUNCS += "repo_clean"
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index dd123c3..df47ced 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -18,7 +18,6 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
>   # Build package from sources using build script
>   dpkg_runbuild() {
>       E="${@ bb.utils.export_proxies(d)}"
> -    flock -s "${REPO_ISAR_DIR}/isar.lock" \
> -        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
> -             /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH}
> +    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
> +         /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH}
>   }
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 05/26] dpkg: Fix shared-locking of build task

[Jan Kiszka]

On 26.08.19 10:30, Claudius Heine wrote:
> Hi Jan,
> 
> On 26/08/2019 08.26, [ext] Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> Bitbake file locks are not just exclusive flock calls. For whatever
>> reason, bitbake prefers to delete the lock file before unlocking them.
>> That obviously creates a nasty race with other locker requests: Those
>> already waiting will get access to the removed lock, those coming later
>> will create a new one. To resolve that race, bitbakes lockfile() tests
>> if the inode of the acquired lock file is identical to the one that is
>> currently visible via the lock name. If not, it retries the lock. What a
>> pointless mess...
>>
>> Our open-coded "flock -s" does not do that, thus will often acquire only
>> a stale lock. Fix that by using bitbake's services for locking. That
>> requires to move the lock acquisition at the top level of the task, ie.
>> do_build. Thus, also dpkg-raw and kernel builds are now under the shared
>> (read) lock. The better fix for this issue would actually be private
>> buildchroots for each build, but that will come later.
>>
>> CC: Cedric Hombourger <Cedric_Hombourger@mentor.com>
>> Fixes: 568072765b08 ("dpkg: acquire a read (shared) locks while building 
>> packages")
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>   meta/classes/dpkg-base.bbclass | 11 +++++++----
>>   meta/classes/dpkg.bbclass      |  5 ++---
>>   2 files changed, 9 insertions(+), 7 deletions(-)
>>
>> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
>> index 2d26418..8146328 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -87,10 +87,13 @@ dpkg_runbuild() {
>>       die "This should never be called, overwrite it in your derived class"
>>   }
>> -do_build() {
>> -    dpkg_do_mounts
>> -    dpkg_runbuild
>> -    dpkg_undo_mounts
>> +python do_build() {
>> +    lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
>> +                             shared=True)
>> +    bb.build.exec_func("dpkg_do_mounts", d)
>> +    bb.build.exec_func("dpkg_runbuild", d)
>> +    bb.build.exec_func("dpkg_undo_mounts", d)
>> +    bb.utils.unlockfile(lock)
> 
> What is the difference between this and
> 
> do_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
> 

shared=True.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[PATCH 06/26] dpkg-base: Free do_build task

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Rename do_build to do_dpkg_build so that the build task remains virtual,
just like it's common practice in OE.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass             | 10 ++++++----
 meta/classes/dpkg.bbclass                  |  2 +-
 meta/recipes-kernel/linux/linux-custom.inc |  2 +-
 3 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 8146328..50df06d 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -18,7 +18,7 @@ do_adjust_git() {
 addtask adjust_git after do_unpack before do_patch
 
 inherit patch
-addtask patch after do_adjust_git before do_build
+addtask patch after do_adjust_git before do_dpkg_build
 
 SRC_APT ?= ""
 
@@ -63,7 +63,7 @@ do_prepare_build() {
     true
 }
 
-addtask prepare_build after do_patch do_transform_template before do_build
+addtask prepare_build after do_patch do_transform_template before do_dpkg_build
 # If Isar recipes depend on each other, they typically need the package
 # deployed to isar-apt
 do_prepare_build[deptask] = "do_deploy_deb"
@@ -87,7 +87,7 @@ dpkg_runbuild() {
     die "This should never be called, overwrite it in your derived class"
 }
 
-python do_build() {
+python do_dpkg_build() {
     lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
                              shared=True)
     bb.build.exec_func("dpkg_do_mounts", d)
@@ -96,6 +96,8 @@ python do_build() {
     bb.utils.unlockfile(lock)
 }
 
+addtask dpkg_build before do_build
+
 CLEANFUNCS += "repo_clean"
 
 repo_clean() {
@@ -129,7 +131,7 @@ do_deploy_deb() {
              ${S}/../*.deb
 }
 
-addtask deploy_deb after do_build
+addtask deploy_deb after do_dpkg_build
 do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 do_deploy_deb[depends] = "isar-apt:do_cache_config"
 do_deploy_deb[dirs] = "${S}"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index df47ced..f5396cb 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -11,7 +11,7 @@ do_install_builddeps() {
     dpkg_undo_mounts
 }
 
-addtask install_builddeps after do_prepare_build before do_build
+addtask install_builddeps after do_prepare_build before do_dpkg_build
 # apt and reprepro may not run in parallel, acquire the Isar lock
 do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 12d8d9c..c045b89 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -41,7 +41,7 @@ do_install_builddeps() {
 	dpkg_undo_mounts
 }
 
-addtask install_builddeps after do_prepare_build before do_build
+addtask install_builddeps after do_prepare_build before do_dpkg_build
 # apt and reprepro may not run in parallel, acquire the Isar lock
 do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
-- 
2.16.4

[PATCH 07/26] base: Clean up build task

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This is now an empty one that should no longer be executed. Align with
OE in this regard.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/base.bbclass | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 1d2852a..8c7b021 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -181,14 +181,12 @@ python do_unpack() {
 
 addtask unpack after do_fetch before do_build
 
-addtask build
-do_build[dirs] = "${TOPDIR}"
-python base_do_build () {
-    bb.note("The included, default BB base.bbclass does not define a useful default task.")
-    bb.note("Try running the 'listtasks' task against a .bb to see what tasks are defined.")
+do_build[noexec] = "1"
+do_build () {
+    :
 }
 
-EXPORT_FUNCTIONS do_build
+addtask build
 
 CLEANFUNCS ?= ""
 
-- 
2.16.4

Re: [PATCH 07/26] base: Clean up build task

[Henning Schild]

Am Mon, 26 Aug 2019 08:26:08 +0200
schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:

> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This is now an empty one that should no longer be executed. Align with
> OE in this regard.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/classes/base.bbclass | 10 ++++------
>  1 file changed, 4 insertions(+), 6 deletions(-)
> 
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 1d2852a..8c7b021 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -181,14 +181,12 @@ python do_unpack() {
>  
>  addtask unpack after do_fetch before do_build
>  
> -addtask build
> -do_build[dirs] = "${TOPDIR}"
> -python base_do_build () {
> -    bb.note("The included, default BB base.bbclass does not define a
> useful default task.")
> -    bb.note("Try running the 'listtasks' task against a .bb to see
> what tasks are defined.") +do_build[noexec] = "1"
> +do_build () {
> +    :
>  }

This actually once came from bitbake/classes/base.bbclass

You might want to compare with that.

the default does not have noexec but it does have nostamp

> -EXPORT_FUNCTIONS do_build
> +addtask build

The default bitbake one has both of these lines.

Henning

>  CLEANFUNCS ?= ""
>  

Re: [PATCH 07/26] base: Clean up build task

[Jan Kiszka]

On 26.08.19 10:30, Henning Schild wrote:
> Am Mon, 26 Aug 2019 08:26:08 +0200
> schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:
> 
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This is now an empty one that should no longer be executed. Align with
>> OE in this regard.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>   meta/classes/base.bbclass | 10 ++++------
>>   1 file changed, 4 insertions(+), 6 deletions(-)
>>
>> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
>> index 1d2852a..8c7b021 100644
>> --- a/meta/classes/base.bbclass
>> +++ b/meta/classes/base.bbclass
>> @@ -181,14 +181,12 @@ python do_unpack() {
>>   
>>   addtask unpack after do_fetch before do_build
>>   
>> -addtask build
>> -do_build[dirs] = "${TOPDIR}"
>> -python base_do_build () {
>> -    bb.note("The included, default BB base.bbclass does not define a
>> useful default task.")
>> -    bb.note("Try running the 'listtasks' task against a .bb to see
>> what tasks are defined.") +do_build[noexec] = "1"
>> +do_build () {
>> +    :
>>   }
> 
> This actually once came from bitbake/classes/base.bbclass
> 
> You might want to compare with that.

I compared to OE and took its version. The bitbake one is a template that makes 
no sense here.

> 
> the default does not have noexec but it does have nostamp
> 
>> -EXPORT_FUNCTIONS do_build
>> +addtask build
> 
> The default bitbake one has both of these lines.

Again, OE is closer to our use case.

Jan


-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[PATCH 08/26] isar-events: Do not start cleanup early

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Umounting already on runQueueExitWait just pulls the carpet under the
feet of still running builds, causing follow-up errors when just one
recipe failed. Even worse, terminating builds with ctrl-c does not allow
to finish most of the running tasks because they will stumble over
disappearing mounts.

While at it, give the handler a clearer name, namely that of its
trigger.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/isar-events.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
index 7dd1ed5..62eb25d 100644
--- a/meta/classes/isar-events.bbclass
+++ b/meta/classes/isar-events.bbclass
@@ -4,9 +4,9 @@
 # Copyright (C) 2015-2017 ilbers GmbH
 # Copyright (c) Siemens AG, 2018
 
-addhandler isar_handler
+addhandler build_completed
 
-python isar_handler() {
+python build_completed() {
     import subprocess
 
     tmpdir = d.getVar('TMPDIR', True)
@@ -25,4 +25,4 @@ python isar_handler() {
                 )
 }
 
-isar_handler[eventmask] = "bb.runqueue.runQueueExitWait bb.event.BuildCompleted"
+build_completed[eventmask] = "bb.event.BuildCompleted"
-- 
2.16.4

[PATCH 09/26] meta-isar: Move bananapi to buster, remove bogus example

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

The bananapi machine config contained a "nice" anti-pattern that made
multiconfig explode: By tuning DISTRO_APT_PREFERENCES, it modified the
dependencies of isar-bootstrap-target for all other stretch-armhf
targets. And that nicely exploded when those were built in parallel.

Rather than trying to fix this pattern towards multiconfig
compatibility, which would have been rather complex, simply remove it.
We have buster now, and the buster kernel is what this target want. So
give it to him.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta-isar/conf/distro/debian-stretch-backports.list                | 2 --
 meta-isar/conf/local.conf.sample                                   | 2 +-
 meta-isar/conf/machine/bananapi.conf                               | 3 ---
 .../multiconfig/{bananapi-stretch.conf => bananapi-buster.conf}    | 2 +-
 meta-isar/conf/multiconfig/preferences.bananapi.conf               | 7 -------
 5 files changed, 2 insertions(+), 14 deletions(-)
 delete mode 100644 meta-isar/conf/distro/debian-stretch-backports.list
 rename meta-isar/conf/multiconfig/{bananapi-stretch.conf => bananapi-buster.conf} (82%)
 delete mode 100644 meta-isar/conf/multiconfig/preferences.bananapi.conf

diff --git a/meta-isar/conf/distro/debian-stretch-backports.list b/meta-isar/conf/distro/debian-stretch-backports.list
deleted file mode 100644
index 6b744f3..0000000
--- a/meta-isar/conf/distro/debian-stretch-backports.list
+++ /dev/null
@@ -1,2 +0,0 @@
-deb	http://ftp.debian.org/debian stretch-backports main
-deb-src	http://ftp.debian.org/debian stretch-backports main
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 5b3a0a1..d9379b1 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -47,9 +47,9 @@ BBMULTICONFIG = " \
     qemuamd64-stretch \
     qemumipsel-stretch \
     qemumipsel-buster \
-    bananapi-stretch \
     de0-nano-soc-stretch \
     hikey-stretch \
+    bananapi-buster \
     qemuamd64-buster \
     qemuamd64-buster-tgz \
     nand-ubi-demo-buster \
diff --git a/meta-isar/conf/machine/bananapi.conf b/meta-isar/conf/machine/bananapi.conf
index 7b380c1..cec8057 100644
--- a/meta-isar/conf/machine/bananapi.conf
+++ b/meta-isar/conf/machine/bananapi.conf
@@ -7,9 +7,6 @@ DISTRO_ARCH ?= "armhf"
 
 KERNEL_NAME ?= "armmp"
 
-DISTRO_APT_SOURCES_append = " conf/distro/debian-stretch-backports.list"
-DISTRO_APT_PREFERENCES += "conf/multiconfig/preferences.bananapi.conf"
-
 IMAGE_INSTALL += "u-boot-script"
 
 IMAGE_TYPE ?= "wic-img"
diff --git a/meta-isar/conf/multiconfig/bananapi-stretch.conf b/meta-isar/conf/multiconfig/bananapi-buster.conf
similarity index 82%
rename from meta-isar/conf/multiconfig/bananapi-stretch.conf
rename to meta-isar/conf/multiconfig/bananapi-buster.conf
index 6c80020..6262a07 100644
--- a/meta-isar/conf/multiconfig/bananapi-stretch.conf
+++ b/meta-isar/conf/multiconfig/bananapi-buster.conf
@@ -4,4 +4,4 @@
 # SPDX-License-Identifier: MIT
 
 MACHINE = "bananapi"
-DISTRO = "debian-stretch"
+DISTRO = "debian-buster"
diff --git a/meta-isar/conf/multiconfig/preferences.bananapi.conf b/meta-isar/conf/multiconfig/preferences.bananapi.conf
deleted file mode 100644
index 9f5c196..0000000
--- a/meta-isar/conf/multiconfig/preferences.bananapi.conf
+++ /dev/null
@@ -1,7 +0,0 @@
-Package: linux-image* linux-headers* linux-kbuild*
-Pin: release n=stretch-backports
-Pin-Priority: 501
-
-Package: *
-Pin: release n=stretch-backports
-Pin-Priority: -1
-- 
2.16.4

[PATCH v2 09/26] meta-isar: Move bananapi to buster, remove bogus example

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

The bananapi machine config contained a "nice" anti-pattern that made
multiconfig explode: By tuning DISTRO_APT_PREFERENCES, it modified the
dependencies of isar-bootstrap-target for all other stretch-armhf
targets. And that nicely exploded when those were built in parallel.

Rather than trying to fix this pattern towards multiconfig
compatibility, which would have been rather complex, simply remove it.
We have buster now, and the buster kernel is what this target want. So
give it to him.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---

Changes in v2:
 - rebased over next

The complete series of my pending patches is still at
https://github.com/siemens/isar/commits/jan/queue

 meta-isar/conf/distro/debian-stretch-backports.list                | 2 --
 meta-isar/conf/local.conf.sample                                   | 2 +-
 meta-isar/conf/machine/bananapi.conf                               | 3 ---
 .../multiconfig/{bananapi-stretch.conf => bananapi-buster.conf}    | 2 +-
 meta-isar/conf/multiconfig/preferences.bananapi.conf               | 7 -------
 5 files changed, 2 insertions(+), 14 deletions(-)
 delete mode 100644 meta-isar/conf/distro/debian-stretch-backports.list
 rename meta-isar/conf/multiconfig/{bananapi-stretch.conf => bananapi-buster.conf} (82%)
 delete mode 100644 meta-isar/conf/multiconfig/preferences.bananapi.conf

diff --git a/meta-isar/conf/distro/debian-stretch-backports.list b/meta-isar/conf/distro/debian-stretch-backports.list
deleted file mode 100644
index 6b744f3..0000000
--- a/meta-isar/conf/distro/debian-stretch-backports.list
+++ /dev/null
@@ -1,2 +0,0 @@
-deb	http://ftp.debian.org/debian stretch-backports main
-deb-src	http://ftp.debian.org/debian stretch-backports main
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 60805eb..d73da1d 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -54,7 +54,7 @@ BBMULTICONFIG = " \
     qemumipsel-stretch \
     qemumipsel-buster \
     qemumipsel-bullseye \
-    bananapi-stretch \
+    bananapi-buster \
     de0-nano-soc-stretch \
     hikey-stretch \
     nand-ubi-demo-buster \
diff --git a/meta-isar/conf/machine/bananapi.conf b/meta-isar/conf/machine/bananapi.conf
index 7b380c1..cec8057 100644
--- a/meta-isar/conf/machine/bananapi.conf
+++ b/meta-isar/conf/machine/bananapi.conf
@@ -7,9 +7,6 @@ DISTRO_ARCH ?= "armhf"
 
 KERNEL_NAME ?= "armmp"
 
-DISTRO_APT_SOURCES_append = " conf/distro/debian-stretch-backports.list"
-DISTRO_APT_PREFERENCES += "conf/multiconfig/preferences.bananapi.conf"
-
 IMAGE_INSTALL += "u-boot-script"
 
 IMAGE_TYPE ?= "wic-img"
diff --git a/meta-isar/conf/multiconfig/bananapi-stretch.conf b/meta-isar/conf/multiconfig/bananapi-buster.conf
similarity index 82%
rename from meta-isar/conf/multiconfig/bananapi-stretch.conf
rename to meta-isar/conf/multiconfig/bananapi-buster.conf
index 6c80020..6262a07 100644
--- a/meta-isar/conf/multiconfig/bananapi-stretch.conf
+++ b/meta-isar/conf/multiconfig/bananapi-buster.conf
@@ -4,4 +4,4 @@
 # SPDX-License-Identifier: MIT
 
 MACHINE = "bananapi"
-DISTRO = "debian-stretch"
+DISTRO = "debian-buster"
diff --git a/meta-isar/conf/multiconfig/preferences.bananapi.conf b/meta-isar/conf/multiconfig/preferences.bananapi.conf
deleted file mode 100644
index 9f5c196..0000000
--- a/meta-isar/conf/multiconfig/preferences.bananapi.conf
+++ /dev/null
@@ -1,7 +0,0 @@
-Package: linux-image* linux-headers* linux-kbuild*
-Pin: release n=stretch-backports
-Pin-Priority: 501
-
-Package: *
-Pin: release n=stretch-backports
-Pin-Priority: -1
-- 
2.16.4

[PATCH 10/26] bitbake.conf: Reorder PV definition

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

No functional change, just logical ordering.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index c41f294..1b9f92b 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -29,9 +29,9 @@ GITDIR = "${DL_DIR}/git"
 P = "${PN}-${PV}"
 PF = "${PN}-${PV}-${PR}"
 PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
+PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
 PR = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[2] or 'r0'}"
 PROVIDES = ""
-PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
 S = "${WORKDIR}/${P}"
 AUTOREV = "${@bb.fetch2.get_autorev(d)}"
 SRC_URI = "file://${FILE}"
-- 
2.16.4

[PATCH 11/26] example-module: Fix build with mipsel target

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

That uses the Debian kernel, thus can't build the module cross as well.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta-isar/recipes-kernel/example-module/example-module.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta-isar/recipes-kernel/example-module/example-module.bb b/meta-isar/recipes-kernel/example-module/example-module.bb
index a08a6c5..90db80e 100644
--- a/meta-isar/recipes-kernel/example-module/example-module.bb
+++ b/meta-isar/recipes-kernel/example-module/example-module.bb
@@ -17,6 +17,7 @@ python() {
         'rpi-rpfv',
         'amd64',
         '686-pae',
+        '4kc-malta',
     ]:
         d.setVar('ISAR_CROSS_COMPILE', '0')
 }
-- 
2.16.4

[PATCH 12/26] buildchroot: Ignore ISAR_CROSS_COMPILE for i386 on stretch

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Stretch has no crossbuild-essential-i386, while buster has. That means
we need to use native build for stretch, even if crossbuild was
requested.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/buildchroot.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index b7a8ea5..2ba2cdd 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -8,7 +8,9 @@ ISAR_CROSS_COMPILE ??= "0"
 # Add dependency from the correct buildchroot: host or target
 python __anonymous() {
     mode = d.getVar('ISAR_CROSS_COMPILE', True)
-    if mode == "0" or d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):
+    distro_arch = d.getVar('DISTRO_ARCH')
+    if mode == "0" or d.getVar('HOST_ARCH') ==  distro_arch or \
+       (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
         dep = "buildchroot-target:do_build"
         rootfs = d.getVar('BUILDCHROOT_TARGET_DIR', True)
     else:
-- 
2.16.4

[PATCH 13/26] buildchroot: Fix downgrades when installing dependencies

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Commit 77e62c1e9bbd didn't fully address the issue: There can still be
failing build dependency installations due to "apt-get upgrade" refusing
the downgrade without explicit --allow-downgrades.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-devtools/buildchroot/files/deps.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh b/meta/recipes-devtools/buildchroot/files/deps.sh
index 1963943..19eb9b7 100644
--- a/meta/recipes-devtools/buildchroot/files/deps.sh
+++ b/meta/recipes-devtools/buildchroot/files/deps.sh
@@ -29,4 +29,4 @@ apt-get update \
 mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control
 
 # Upgrade any already installed packages in case we are partially rebuilding
-apt-get upgrade -y
+apt-get upgrade -y --allow-downgrades
-- 
2.16.4

[PATCH 14/26] buildchroot: Properly deploy the build result

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This follows the pattern used for isar-bootstrap already: Establish a
stable link to the rootfs in the workdir from a folder in DEPLOY_DIR.
This is cleaner than pointing directly into the workdir via
BUILDCHROOT_HOST/TARGET_DIR, and it avoid having to mess with PF.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/buildchroot.bbclass                        |  2 +-
 meta/conf/bitbake.conf                                  |  5 +++--
 meta/recipes-devtools/buildchroot/buildchroot-host.bb   |  2 ++
 meta/recipes-devtools/buildchroot/buildchroot-target.bb |  2 ++
 meta/recipes-devtools/buildchroot/buildchroot.inc       | 13 +++++++++++++
 5 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index 2ba2cdd..a87e144 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -21,7 +21,7 @@ python __anonymous() {
     d.setVar('BUILDCHROOT_DIR', rootfs)
 }
 
-MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock"
+MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}.lock"
 
 buildchroot_do_mounts() {
     sudo -s <<'EOSUDO'
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 1b9f92b..f25b20f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -47,11 +47,12 @@ GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
 # isar specific config
 WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}"
 DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"
+DEPLOY_DIR_BUILDCHROOT = "${DEPLOY_DIR}/buildchroot"
 DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MACHINE}"
 DL_DIR ?= "${TOPDIR}/downloads"
 SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
-BUILDCHROOT_HOST_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-host-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}/rootfs"
-BUILDCHROOT_TARGET_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-target-${DISTRO}-${DISTRO_ARCH}/rootfs"
+BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"
 CACHE = "${TMPDIR}/cache"
 
 OVERRIDES = "${DISTRO_ARCH}:${MACHINE}:${DISTRO}:forcevariable"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
index 121992c..40a37ea 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
@@ -6,6 +6,8 @@
 DESCRIPTION = "Isar development filesystem for host"
 PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
 
+BUILDCHROOT_VARIANT = "host"
+
 require buildchroot.inc
 ROOTFS_ARCH = "${HOST_ARCH}"
 ROOTFS_DISTRO = "${HOST_DISTRO}"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-target.bb b/meta/recipes-devtools/buildchroot/buildchroot-target.bb
index 5a01258..3b96977 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-target.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-target.bb
@@ -6,6 +6,8 @@
 DESCRIPTION = "Isar development filesystem for target"
 PF = "${PN}-${DISTRO}-${DISTRO_ARCH}"
 
+BUILDCHROOT_VARIANT = "target"
+
 require buildchroot.inc
 
 BUILDCHROOT_PREINSTALL ?= "gcc \
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index 40fc8b0..e704137 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -46,3 +46,16 @@ buildchroot_install_files() {
     sudo chroot ${BUILDCHROOT_DIR} /configscript.sh $USER_ID $GROUP_ID
 
 }
+
+DEPLOY_BUILDCHROOT = "${@d.getVar('BUILDCHROOT_' + d.getVar('BUILDCHROOT_VARIANT').upper() + '_DIR')}"
+
+do_buildchroot_deploy[dirs] = "${DEPLOY_DIR_BUILDCHROOT}-${BUILDCHROOT_VARIANT}"
+do_buildchroot_deploy() {
+    ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_BUILDCHROOT}"
+}
+addtask buildchroot_deploy before do_build after do_rootfs
+
+CLEANFUNCS = "clean_deploy"
+clean_deploy() {
+    rm -f "${DEPLOY_BUILDCHROOT}"
+}
-- 
2.16.4

[PATCH 15/26] sdk: Use clean-package-cache rather than open-coded cleanup

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

There is nothing that the sdkchroot will gain after its bootstrap.
Therefore we can clean up the apt cache already there, reusing common
infrastructure.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/image-sdk-extension.bbclass     | 3 ---
 meta/recipes-devtools/sdkchroot/sdkchroot.bb | 2 +-
 2 files changed, 1 insertion(+), 4 deletions(-)

diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass
index 8df3ed1..a53d954 100644
--- a/meta/classes/image-sdk-extension.bbclass
+++ b/meta/classes/image-sdk-extension.bbclass
@@ -13,9 +13,6 @@ do_populate_sdk() {
     # Copy isar-apt with deployed Isar packages
     sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO}  ${SDKCHROOT_DIR}/rootfs/isar-apt
 
-    # Purge apt cache to make image slimmer
-    sudo rm -rf ${SDKCHROOT_DIR}/rootfs/var/cache/apt/*
-
     sudo umount -R ${SDKCHROOT_DIR}/rootfs/dev || true
     sudo umount ${SDKCHROOT_DIR}/rootfs/proc || true
     sudo umount -R ${SDKCHROOT_DIR}/rootfs/sys || true
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index 3cfe05c..d2cc92e 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -23,7 +23,7 @@ ROOTFS_ARCH = "${HOST_ARCH}"
 ROOTFS_DISTRO = "${HOST_DISTRO}"
 ROOTFSDIR = "${S}"
 ROOTFS_PACKAGES = "${SDKCHROOT_PREINSTALL} ${TOOLCHAIN}"
-ROOTFS_FEATURES += ""
+ROOTFS_FEATURES += "clean-package-cache"
 
 python() {
     if d.getVar("HOST_ARCH") not in ['i386', 'amd64']:
-- 
2.16.4

[PATCH 16/26] sdk: Deploy earlier

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This avoids having to point from image-sdk-extension directly into the
sdkchroot target workdir. It also aligns the sdk with isar-bootstrap and
buildchroot in this regard.

As we now have a deployment target of the sdk rootfs (SDKCHROOT_DIR), we
can drop the previous link in DEPLOY_DIR_IMAGE.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/image-sdk-extension.bbclass     | 20 ++++++++------------
 meta/conf/bitbake.conf                       |  2 ++
 meta/recipes-devtools/sdkchroot/sdkchroot.bb | 11 +++++++++++
 3 files changed, 21 insertions(+), 12 deletions(-)

diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass
index a53d954..b9f2cf4 100644
--- a/meta/classes/image-sdk-extension.bbclass
+++ b/meta/classes/image-sdk-extension.bbclass
@@ -5,29 +5,25 @@
 #
 # This class extends the image.bbclass to supply the creation of a sdk
 
-SDKCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
-
 do_populate_sdk[stamp-extra-info] = "${DISTRO}-${MACHINE}"
 do_populate_sdk[depends] = "sdkchroot:do_build"
 do_populate_sdk() {
     # Copy isar-apt with deployed Isar packages
-    sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO}  ${SDKCHROOT_DIR}/rootfs/isar-apt
+    sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO}  ${SDKCHROOT_DIR}/isar-apt
 
-    sudo umount -R ${SDKCHROOT_DIR}/rootfs/dev || true
-    sudo umount ${SDKCHROOT_DIR}/rootfs/proc || true
-    sudo umount -R ${SDKCHROOT_DIR}/rootfs/sys || true
+    sudo umount -R ${SDKCHROOT_DIR}/dev || true
+    sudo umount ${SDKCHROOT_DIR}/proc || true
+    sudo umount -R ${SDKCHROOT_DIR}/sys || true
 
     # Remove setup scripts
-    sudo rm -f ${SDKCHROOT_DIR}/rootfs/chroot-setup.sh ${SDKCHROOT_DIR}/rootfs/configscript.sh
+    sudo rm -f ${SDKCHROOT_DIR}/chroot-setup.sh ${SDKCHROOT_DIR}/configscript.sh
 
     # Copy mount_chroot.sh for convenience
-    sudo cp ${ISARROOT}/scripts/mount_chroot.sh ${SDKCHROOT_DIR}/rootfs
+    sudo cp ${ISARROOT}/scripts/mount_chroot.sh ${SDKCHROOT_DIR}
 
     # Create SDK archive
-    sudo tar -C ${SDKCHROOT_DIR} --transform="s|^rootfs|sdk-${DISTRO}-${DISTRO_ARCH}|" \
+    cd -P ${SDKCHROOT_DIR}/..
+    sudo tar --transform="s|^rootfs|sdk-${DISTRO}-${DISTRO_ARCH}|" \
         -c rootfs | xz -T0 > ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz
-
-    # Install deployment link for local use
-    ln -Tfsr ${SDKCHROOT_DIR}/rootfs ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}
 }
 addtask populate_sdk after do_rootfs
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index f25b20f..e5681bf 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -48,11 +48,13 @@ GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
 WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}"
 DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"
 DEPLOY_DIR_BUILDCHROOT = "${DEPLOY_DIR}/buildchroot"
+DEPLOY_DIR_SDKCHROOT = "${DEPLOY_DIR}/sdkchroot"
 DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MACHINE}"
 DL_DIR ?= "${TOPDIR}/downloads"
 SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
 BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
 BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"
+SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${DISTRO}-${DISTRO_ARCH}"
 CACHE = "${TMPDIR}/cache"
 
 OVERRIDES = "${DISTRO_ARCH}:${MACHINE}:${DISTRO}:forcevariable"
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index d2cc92e..84b8808 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -56,3 +56,14 @@ sdkchroot_install_files() {
     sudo install -m 755 ${WORKDIR}/configscript.sh ${S}
     sudo chroot ${S} /configscript.sh  ${DISTRO_ARCH}
 }
+
+do_sdkchroot_deploy[dirs] = "${DEPLOY_DIR_SDKCHROOT}"
+do_sdkchroot_deploy() {
+    ln -Tfsr "${ROOTFSDIR}" "${SDKCHROOT_DIR}"
+}
+addtask sdkchroot_deploy before do_build after do_rootfs
+
+CLEANFUNCS = "clean_deploy"
+clean_deploy() {
+    rm -f "${SDKCHROOT_DIR}"
+}
-- 
2.16.4

[PATCH 17/26] Remove unneeded PF tweakings

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

All these recipes now have unique WORKDIRs and STAMPs, and all deploy
their shared artifacts via DEPLOY_DIR.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb   | 1 -
 meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb | 1 -
 meta/recipes-devtools/buildchroot/buildchroot-host.bb     | 1 -
 meta/recipes-devtools/buildchroot/buildchroot-target.bb   | 1 -
 meta/recipes-devtools/sdkchroot/sdkchroot.bb              | 1 -
 5 files changed, 5 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index 8363465..b2dded5 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -6,7 +6,6 @@
 # SPDX-License-Identifier: MIT
 
 Description = "Minimal host Debian root file system"
-PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
 
 DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
 ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}.lock"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
index 77276e9..d7fcdbf 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
@@ -6,7 +6,6 @@
 # SPDX-License-Identifier: MIT
 
 Description = "Minimal target Debian root file system"
-PF = "${PN}-${DISTRO}-${DISTRO_ARCH}"
 
 DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}"
 ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}.lock"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
index 40a37ea..795b77a 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
@@ -4,7 +4,6 @@
 # Copyright (C) 2015-2018 ilbers GmbH
 
 DESCRIPTION = "Isar development filesystem for host"
-PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
 
 BUILDCHROOT_VARIANT = "host"
 
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-target.bb b/meta/recipes-devtools/buildchroot/buildchroot-target.bb
index 3b96977..68aedaa 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-target.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-target.bb
@@ -4,7 +4,6 @@
 # Copyright (C) 2015-2018 ilbers GmbH
 
 DESCRIPTION = "Isar development filesystem for target"
-PF = "${PN}-${DISTRO}-${DISTRO_ARCH}"
 
 BUILDCHROOT_VARIANT = "target"
 
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index 84b8808..e658122 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -12,7 +12,6 @@ SRC_URI = " \
     file://configscript.sh \
     file://README.sdk"
 PV = "0.1"
-PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
 
 TOOLCHAIN = "crossbuild-essential-${DISTRO_ARCH}"
 TOOLCHAIN_${HOST_ARCH} = "build-essential"
-- 
2.16.4

[PATCH 18/26] image: Stop changing PF, tune WORKDIR and STAMP directly

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This removes the last fiddling with PF, addressing the issue - a common
image PN name for multiconfig targets while the results are actually
machine-specific - by tuning the critical paths directly. This are just
WORKDIR as well as STAMP and STAMPCLEAN.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/image.bbclass | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index ec6bd39..36e9875 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -1,7 +1,10 @@
 # This software is a part of ISAR.
 # Copyright (C) 2015-2017 ilbers GmbH
 
-PF = "${PN}-${DISTRO}-${MACHINE}"
+# Make workdir and stamps machine-specific without changing common PN target
+WORKDIR .= "-${MACHINE}"
+STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/${PV}-${PR}"
+STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/*-*"
 
 IMAGE_INSTALL ?= ""
 IMAGE_TYPE    ?= "ext4-img"
-- 
2.16.4

Re: [PATCH 18/26] image: Stop changing PF, tune WORKDIR and STAMP directly

[Henning Schild]

Am Mon, 26 Aug 2019 08:26:19 +0200
schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:

> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This removes the last fiddling with PF, addressing the issue - a
> common image PN name for multiconfig targets while the results are
> actually machine-specific - by tuning the critical paths directly.
> This are just WORKDIR as well as STAMP and STAMPCLEAN.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/classes/image.bbclass | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index ec6bd39..36e9875 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -1,7 +1,10 @@
>  # This software is a part of ISAR.
>  # Copyright (C) 2015-2017 ilbers GmbH
>  
> -PF = "${PN}-${DISTRO}-${MACHINE}"
> +# Make workdir and stamps machine-specific without changing common
> PN target +WORKDIR .= "-${MACHINE}"
> +STAMP =
> "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/${PV}-${PR}"

.../${PF} ?

Henning

> +STAMPCLEAN =
> "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/*-*"
> IMAGE_INSTALL ?= "" IMAGE_TYPE    ?= "ext4-img"

Re: [PATCH 18/26] image: Stop changing PF, tune WORKDIR and STAMP directly

[Jan Kiszka]

On 26.08.19 10:35, Henning Schild wrote:
> Am Mon, 26 Aug 2019 08:26:19 +0200
> schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:
> 
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This removes the last fiddling with PF, addressing the issue - a
>> common image PN name for multiconfig targets while the results are
>> actually machine-specific - by tuning the critical paths directly.
>> This are just WORKDIR as well as STAMP and STAMPCLEAN.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>   meta/classes/image.bbclass | 5 ++++-
>>   1 file changed, 4 insertions(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index ec6bd39..36e9875 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -1,7 +1,10 @@
>>   # This software is a part of ISAR.
>>   # Copyright (C) 2015-2017 ilbers GmbH
>>   
>> -PF = "${PN}-${DISTRO}-${MACHINE}"
>> +# Make workdir and stamps machine-specific without changing common
>> PN target +WORKDIR .= "-${MACHINE}"
>> +STAMP =
>> "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/${PV}-${PR}"
> 
> .../${PF} ?

Nope, that's already included in PN, PV and PR.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[PATCH 19/26] image: Make WORKDIR and STAMPs IMAGE_TYPE-specific

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This fixes false sharing of parallel builds like
multiconfig:qemuamd64-buster:isar-image-base plus
multiconfig:qemuamd64-buster-tgz:isar-image-base.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/image.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 36e9875..89cc5fe 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -2,9 +2,9 @@
 # Copyright (C) 2015-2017 ilbers GmbH
 
 # Make workdir and stamps machine-specific without changing common PN target
-WORKDIR .= "-${MACHINE}"
-STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/${PV}-${PR}"
-STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/*-*"
+WORKDIR .= "-${MACHINE}-${IMAGE_TYPE}"
+STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/${PV}-${PR}"
+STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/*-*"
 
 IMAGE_INSTALL ?= ""
 IMAGE_TYPE    ?= "ext4-img"
-- 
2.16.4

[PATCH 20/26] Adjust STAMP variables to use PN, rather than PF

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

The last misuse of PF is gone, we can adjust the defaults to OE.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/conf/bitbake.conf | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index e5681bf..9c9c7fe 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -36,8 +36,8 @@ S = "${WORKDIR}/${P}"
 AUTOREV = "${@bb.fetch2.get_autorev(d)}"
 SRC_URI = "file://${FILE}"
 STAMPS_DIR ?= "${TMPDIR}/stamps"
-STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PF}/${PV}-${PR}"
-STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PF}/*-*"
+STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/${PV}-${PR}"
+STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/*-*"
 SVNDIR = "${DL_DIR}/svn"
 T = "${WORKDIR}/temp"
 TMPDIR = "${TOPDIR}/tmp"
-- 
2.16.4

[PATCH 21/26] Align WORKDIR structure with OE

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Pick up the OE pattern to have a per-PV-PR workdir under PN. It also
aligns this with our STAMP setting.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 doc/user_manual.md         | 2 +-
 meta/classes/image.bbclass | 2 +-
 meta/conf/bitbake.conf     | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 289fd1d..eba5f0f 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -452,7 +452,7 @@ The user may use `met-isar/recipes-core-images` as a template for new image reci
 ### General Information
 The image recipe in Isar creates a folder with target root filesystem. Its default location is:
 ```
-tmp/work/${DISTRO}-${DISTRO_ARCH}/${MACHINE}/${IMAGE}/rootfs
+tmp/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/${PV}-${PR}/rootfs
 ```
 Every image type in Isar is implemented as a `bitbake` class. The goal of these classes is to pack root filesystem folder to appropriate format.
 
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 89cc5fe..0965f26 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -2,7 +2,7 @@
 # Copyright (C) 2015-2017 ilbers GmbH
 
 # Make workdir and stamps machine-specific without changing common PN target
-WORKDIR .= "-${MACHINE}-${IMAGE_TYPE}"
+WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/${PV}-${PR}"
 STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/${PV}-${PR}"
 STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/*-*"
 
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 9c9c7fe..bca1114 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -45,7 +45,7 @@ PERSISTENT_DIR = "${TMPDIR}/cache"
 GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
 
 # isar specific config
-WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}"
+WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}/${PV}-${PR}"
 DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"
 DEPLOY_DIR_BUILDCHROOT = "${DEPLOY_DIR}/buildchroot"
 DEPLOY_DIR_SDKCHROOT = "${DEPLOY_DIR}/sdkchroot"
-- 
2.16.4

[PATCH 22/26] isar-bootstrap: Clean up and fix rebuild

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

There is no reason for using a lock here. All bootstraps for the same
distro/arch are detected by bitbake as identical targets in multiconfig
scenarios and not run multiple times. If they are run multiple times,
there is another bug, such as with the recently fixed bananapi machine.

Also remove the equally bogus re-execution test in
do_apt_config_prepare. This also fixes rebuilds of the isar-bootstrap
target in case some parameter has changed.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 .../isar-bootstrap/isar-bootstrap-host.bb          | 30 ++++++++++------------
 .../isar-bootstrap/isar-bootstrap-target.bb        | 27 +++++++++----------
 .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  5 ----
 3 files changed, 26 insertions(+), 36 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index b2dded5..8c211a9 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -8,7 +8,6 @@
 Description = "Minimal host Debian root file system"
 
 DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
-ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}.lock"
 
 require isar-bootstrap.inc
 
@@ -25,21 +24,20 @@ do_apt_config_prepare[vardeps] += "\
                                    DEPLOY_ISAR_BOOTSTRAP \
                                   "
 python do_apt_config_prepare() {
-    if not os.path.islink(d.getVar("DEPLOY_ISAR_BOOTSTRAP", True)):
-        apt_preferences_out = d.getVar("APTPREFS", True)
-        apt_preferences_list = (
-            d.getVar("HOST_DISTRO_APT_PREFERENCES", True) or ""
-        ).split()
-        aggregate_files(d, apt_preferences_list, apt_preferences_out)
-
-        apt_sources_out = d.getVar("APTSRCS", True)
-        apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
-        apt_sources_list = (
-            d.getVar("HOST_DISTRO_APT_SOURCES", True) or ""
-        ).split()
-
-        aggregate_files(d, apt_sources_list, apt_sources_init_out)
-        aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
+    apt_preferences_out = d.getVar("APTPREFS", True)
+    apt_preferences_list = (
+        d.getVar("HOST_DISTRO_APT_PREFERENCES", True) or ""
+    ).split()
+    aggregate_files(d, apt_preferences_list, apt_preferences_out)
+
+    apt_sources_out = d.getVar("APTSRCS", True)
+    apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
+    apt_sources_list = (
+        d.getVar("HOST_DISTRO_APT_SOURCES", True) or ""
+    ).split()
+
+    aggregate_files(d, apt_sources_list, apt_sources_init_out)
+    aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
 }
 addtask apt_config_prepare before do_bootstrap after do_unpack
 
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
index d7fcdbf..80e7f40 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
@@ -8,7 +8,6 @@
 Description = "Minimal target Debian root file system"
 
 DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}"
-ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}.lock"
 
 require isar-bootstrap.inc
 
@@ -22,19 +21,18 @@ do_apt_config_prepare[vardeps] += "\
                                    DEPLOY_ISAR_BOOTSTRAP \
                                   "
 python do_apt_config_prepare() {
-    if not os.path.islink(d.getVar("DEPLOY_ISAR_BOOTSTRAP", True)):
-        apt_preferences_out = d.getVar("APTPREFS", True)
-        apt_preferences_list = (
-            d.getVar("DISTRO_APT_PREFERENCES", True) or ""
-        ).split()
-        aggregate_files(d, apt_preferences_list, apt_preferences_out)
-
-        apt_sources_out = d.getVar("APTSRCS", True)
-        apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
-        apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or "").split()
-
-        aggregate_files(d, apt_sources_list, apt_sources_init_out)
-        aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
+    apt_preferences_out = d.getVar("APTPREFS", True)
+    apt_preferences_list = (
+        d.getVar("DISTRO_APT_PREFERENCES", True) or ""
+    ).split()
+    aggregate_files(d, apt_preferences_list, apt_preferences_out)
+
+    apt_sources_out = d.getVar("APTSRCS", True)
+    apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
+    apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or "").split()
+
+    aggregate_files(d, apt_sources_list, apt_sources_init_out)
+    aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
 }
 addtask apt_config_prepare before do_bootstrap after do_unpack
 
@@ -45,4 +43,3 @@ do_bootstrap() {
     isar_bootstrap
 }
 addtask bootstrap before do_build after do_generate_keyrings
-
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index aea89a9..36cefd2 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -212,14 +212,11 @@ do_generate_keyrings() {
 }
 addtask generate_keyrings before do_build after do_unpack
 
-
-
 def get_host_release():
     import platform
     rel = platform.release()
     return rel
 
-
 do_bootstrap[vardeps] += "DISTRO_APT_PREMIRRORS"
 do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP}"
 
@@ -246,7 +243,6 @@ isar_bootstrap() {
     E="${@bb.utils.export_proxies(d)}"
     export IS_HOST debootstrap_args E
     sudo -E -s <<'EOSUDO'
-        ( flock 9
         set -e
         if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then
             rm -rf "${ROOTFSDIR}"
@@ -340,7 +336,6 @@ isar_bootstrap() {
             # Finalize debootstrap by setting the link in deploy
             ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
         fi
-        ) 9>'${ISAR_BOOTSTRAP_LOCK}'
 EOSUDO
 }
 
-- 
2.16.4

Re: [PATCH 22/26] isar-bootstrap: Clean up and fix rebuild

[Jan Kiszka]

On 26.08.19 08:26, [ext] Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> There is no reason for using a lock here. All bootstraps for the same
> distro/arch are detected by bitbake as identical targets in multiconfig
> scenarios and not run multiple times. If they are run multiple times,
> there is another bug, such as with the recently fixed bananapi machine.
> 
> Also remove the equally bogus re-execution test in
> do_apt_config_prepare. This also fixes rebuilds of the isar-bootstrap
> target in case some parameter has changed.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>   .../isar-bootstrap/isar-bootstrap-host.bb          | 30 ++++++++++------------
>   .../isar-bootstrap/isar-bootstrap-target.bb        | 27 +++++++++----------
>   .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  5 ----
>   3 files changed, 26 insertions(+), 36 deletions(-)
> 
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
> index b2dded5..8c211a9 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
> @@ -8,7 +8,6 @@
>   Description = "Minimal host Debian root file system"
>   
>   DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
> -ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}.lock"
>   
>   require isar-bootstrap.inc
>   
> @@ -25,21 +24,20 @@ do_apt_config_prepare[vardeps] += "\
>                                      DEPLOY_ISAR_BOOTSTRAP \
>                                     "
>   python do_apt_config_prepare() {
> -    if not os.path.islink(d.getVar("DEPLOY_ISAR_BOOTSTRAP", True)):
> -        apt_preferences_out = d.getVar("APTPREFS", True)
> -        apt_preferences_list = (
> -            d.getVar("HOST_DISTRO_APT_PREFERENCES", True) or ""
> -        ).split()
> -        aggregate_files(d, apt_preferences_list, apt_preferences_out)
> -
> -        apt_sources_out = d.getVar("APTSRCS", True)
> -        apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
> -        apt_sources_list = (
> -            d.getVar("HOST_DISTRO_APT_SOURCES", True) or ""
> -        ).split()
> -
> -        aggregate_files(d, apt_sources_list, apt_sources_init_out)
> -        aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
> +    apt_preferences_out = d.getVar("APTPREFS", True)
> +    apt_preferences_list = (
> +        d.getVar("HOST_DISTRO_APT_PREFERENCES", True) or ""
> +    ).split()
> +    aggregate_files(d, apt_preferences_list, apt_preferences_out)
> +
> +    apt_sources_out = d.getVar("APTSRCS", True)
> +    apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
> +    apt_sources_list = (
> +        d.getVar("HOST_DISTRO_APT_SOURCES", True) or ""
> +    ).split()
> +
> +    aggregate_files(d, apt_sources_list, apt_sources_init_out)
> +    aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
>   }
>   addtask apt_config_prepare before do_bootstrap after do_unpack
>   
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
> index d7fcdbf..80e7f40 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
> @@ -8,7 +8,6 @@
>   Description = "Minimal target Debian root file system"
>   
>   DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}"
> -ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}.lock"
>   
>   require isar-bootstrap.inc
>   
> @@ -22,19 +21,18 @@ do_apt_config_prepare[vardeps] += "\
>                                      DEPLOY_ISAR_BOOTSTRAP \
>                                     "
>   python do_apt_config_prepare() {
> -    if not os.path.islink(d.getVar("DEPLOY_ISAR_BOOTSTRAP", True)):
> -        apt_preferences_out = d.getVar("APTPREFS", True)
> -        apt_preferences_list = (
> -            d.getVar("DISTRO_APT_PREFERENCES", True) or ""
> -        ).split()
> -        aggregate_files(d, apt_preferences_list, apt_preferences_out)
> -
> -        apt_sources_out = d.getVar("APTSRCS", True)
> -        apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
> -        apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or "").split()
> -
> -        aggregate_files(d, apt_sources_list, apt_sources_init_out)
> -        aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
> +    apt_preferences_out = d.getVar("APTPREFS", True)
> +    apt_preferences_list = (
> +        d.getVar("DISTRO_APT_PREFERENCES", True) or ""
> +    ).split()
> +    aggregate_files(d, apt_preferences_list, apt_preferences_out)
> +
> +    apt_sources_out = d.getVar("APTSRCS", True)
> +    apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
> +    apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or "").split()
> +
> +    aggregate_files(d, apt_sources_list, apt_sources_init_out)
> +    aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
>   }
>   addtask apt_config_prepare before do_bootstrap after do_unpack
>   
> @@ -45,4 +43,3 @@ do_bootstrap() {
>       isar_bootstrap
>   }
>   addtask bootstrap before do_build after do_generate_keyrings
> -
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index aea89a9..36cefd2 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -212,14 +212,11 @@ do_generate_keyrings() {
>   }
>   addtask generate_keyrings before do_build after do_unpack
>   
> -
> -
>   def get_host_release():
>       import platform
>       rel = platform.release()
>       return rel
>   
> -
>   do_bootstrap[vardeps] += "DISTRO_APT_PREMIRRORS"
>   do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP}"
>   
> @@ -246,7 +243,6 @@ isar_bootstrap() {
>       E="${@bb.utils.export_proxies(d)}"
>       export IS_HOST debootstrap_args E
>       sudo -E -s <<'EOSUDO'
> -        ( flock 9
>           set -e
>           if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then
>               rm -rf "${ROOTFSDIR}"
> @@ -340,7 +336,6 @@ isar_bootstrap() {
>               # Finalize debootstrap by setting the link in deploy
>               ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
>           fi
> -        ) 9>'${ISAR_BOOTSTRAP_LOCK}'
>   EOSUDO
>   }
>   
> 

I just understood (thanks to a rare CI failure) what this lock was papering 
over: Raspbian and Debian builds for the same target distro/arch share their 
host boostrap deployment. That needs to be fixed. Of course, that lock didn't 
solve anything because it forced the second user to live with the bootstrap 
settings of the first one.

Again the time to remind everyone to try harder understanding issues as they 
occur, rather than throwing random stuff at them or even ignore them as "not my 
use case". Those things always tend to bite us again at some point. I think we 
are getting better here, though.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[PATCH 23/26] gitlab-ci: Keep logs as artifacts on failure

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 .gitlab-ci.yml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 1c6fa84..5c64adc 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -11,3 +11,9 @@ all:
     - export ftp_proxy=$FTP_PROXY
     - export no_proxy=$NO_PROXY
     - scripts/ci_build.sh -q -f
+  artifacts:
+    name: "logs-$CI_JOB_ID"
+    paths:
+      - build/tmp/work/*/*/*/temp
+    when: on_failure
+    expire_in: 1 week
-- 
2.16.4

[PATCH 24/26] Detect false sharing of recipes

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

When a task is run more than once per build, this indicates a bug,
usually (though not only) related to incorrect multiconfig use. Such
bugs are hard to find in practice, so they could benefit from some
tooling.

This adds a very simple but effective check for such re-run scenarios
by putting a "once.<taskname>" stamp file into each workdir during a
build. If such a file already exists, an error is thrown. The workdirs
are purged after the parsing phase on the beginning of each build so
that no false positives are raised.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/isar-events.bbclass | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
index 62eb25d..1335356 100644
--- a/meta/classes/isar-events.bbclass
+++ b/meta/classes/isar-events.bbclass
@@ -4,6 +4,28 @@
 # Copyright (C) 2015-2017 ilbers GmbH
 # Copyright (c) Siemens AG, 2018
 
+addhandler parse_completed
+
+python parse_completed() {
+    bb.utils.remove(d.getVar('TMPDIR') + "/work/*/*/*/temp/once.*")
+}
+parse_completed[eventmask] = "bb.event.ParseCompleted"
+
+addhandler task_started
+
+python task_started() {
+    task = d.getVar('BB_RUNTASK')
+    try:
+        f = open("%s/once.%s" % (d.getVar('T'), task), "x")
+        f.close()
+    except FileExistsError:
+        bb.error("Detect multiple executions of %s in %s" %
+                 (task, d.getVar('WORKDIR')))
+        bb.error("Rerun a clean build with empty STAMPCLEAN " \
+                 "and compare the sigdata files")
+}
+task_started[eventmask] = "bb.build.TaskStarted"
+
 addhandler build_completed
 
 python build_completed() {
-- 
2.16.4

[PATCH 25/26] dpkg-base: Move do_deploy_deb before do_build

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

There is not good reason to move anything beyond the final do_build task
in a recipe.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 50df06d..d4541a4 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -131,7 +131,7 @@ do_deploy_deb() {
              ${S}/../*.deb
 }
 
-addtask deploy_deb after do_dpkg_build
+addtask deploy_deb after do_dpkg_build before do_build
 do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 do_deploy_deb[depends] = "isar-apt:do_cache_config"
 do_deploy_deb[dirs] = "${S}"
-- 
2.16.4

Re: [PATCH 25/26] dpkg-base: Move do_deploy_deb before do_build

[Claudius Heine]

Hi Jan,

On 26/08/2019 08.26, [ext] Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> There is not good reason to move anything beyond the final do_build task
> in a recipe.

This is not exactly what this patch does though, so you might want to 
update the description.

What you do here is add do_deploy_deb as a dependency to do_build, 
meaning now deploying a debian package is part of the default build 
target. The main difference is in the definition of when is a recipe 
considered to be 'build': Either after the debian package was created 
inside the work directory or when the debian package is added to a 
repository. I would argue that there are reasons for both perspectives.

It is possible to consider 'do_depoy*' in packages more like 'make 
install', and as such not part of the default target.

regards,
Claudius

> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>   meta/classes/dpkg-base.bbclass | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 50df06d..d4541a4 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -131,7 +131,7 @@ do_deploy_deb() {
>                ${S}/../*.deb
>   }
>   
> -addtask deploy_deb after do_dpkg_build
> +addtask deploy_deb after do_dpkg_build before do_build
>   do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
>   do_deploy_deb[depends] = "isar-apt:do_cache_config"
>   do_deploy_deb[dirs] = "${S}"
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 25/26] dpkg-base: Move do_deploy_deb before do_build

[Baurzhan Ismagulov]

On Wed, Aug 28, 2019 at 09:32:43AM +0200, Claudius Heine wrote:
> > There is not good reason to move anything beyond the final do_build task
> > in a recipe.
> 
> This is not exactly what this patch does though, so you might want to update
> the description.
> 
> What you do here is add do_deploy_deb as a dependency to do_build, meaning
> now deploying a debian package is part of the default build target. The main
> difference is in the definition of when is a recipe considered to be
> 'build': Either after the debian package was created inside the work
> directory or when the debian package is added to a repository. I would argue
> that there are reasons for both perspectives.
> 
> It is possible to consider 'do_depoy*' in packages more like 'make install',
> and as such not part of the default target.

Thanks for the explanation. Yes, there are both perspectives. OE deploys by
default. The description doesn't expand on the implications, which I think is
ok. I'm going to apply this.

With kind regards,
Baurzhan.

[PATCH 26/26] base-apt: Move do_cache_config before do_build

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

There is not good reason to move anything beyond the final do_build task
in a recipe. This also aligns base-apt with isar-apt.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-devtools/base-apt/base-apt.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index 488a44d..f1a0e68 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -37,4 +37,4 @@ do_cache_config() {
     fi
 }
 
-addtask cache_config after do_build
+addtask cache_config after do_unpack before do_build
-- 
2.16.4

[PATCH 27/26] wic-img: Use private tmp dir

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

The buildchroot is shared between images of the same target arch. Just
using /tmp/<image-name> can cause conflicts and build failures.

Seen with jailhouse-images.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/wic-img.bbclass | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass
index 1ccbe55..43e8fd5 100644
--- a/meta/classes/wic-img.bbclass
+++ b/meta/classes/wic-img.bbclass
@@ -144,14 +144,18 @@ EOSUDO
     export BUILDDIR=${BUILDDIR}
     export MTOOLS_SKIP_CHECK=1
 
+    # create the temp dir in the buildchroot to ensure uniqueness
+    BUILDCHROOT_WICTMP=$(mktemp -d -p ${BUILDCHROOT_DIR}/tmp)
+    WICTMP=$(echo $BUILDCHROOT_WICTMP | sed 's|^${BUILDCHROOT_DIR}||')
+
     sudo -E chroot ${BUILDCHROOT_DIR} \
         ${ISARROOT}/scripts/wic create ${WKS_FULL_PATH} \
             --vars "${STAGING_DIR}/${MACHINE}/imgdata/" \
-            -o /tmp/${IMAGE_FULLNAME}.wic/ \
+            -o $WICTMP/${IMAGE_FULLNAME}.wic/ \
             --bmap \
             -e ${IMAGE_BASENAME} ${WIC_CREATE_EXTRA_ARGS}
     sudo chown -R $(stat -c "%U" ${ISARROOT}) ${ISARROOT}/meta ${ISARROOT}/meta-isar ${ISARROOT}/scripts || true
-    WIC_DIRECT=$(ls -t -1 ${BUILDCHROOT_DIR}/tmp/${IMAGE_FULLNAME}.wic/*.direct | head -1)
+    WIC_DIRECT=$(ls -t -1 $BUILDCHROOT_WICTMP/${IMAGE_FULLNAME}.wic/*.direct | head -1)
     cp -f ${WIC_DIRECT} ${WIC_IMAGE_FILE}
     cp -f ${WIC_DIRECT}.bmap ${WIC_IMAGE_FILE}.bmap
 }
-- 

2.16.4

[PATCH v2 27/26] wic-img: Use private tmp dir

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

The buildchroot is shared between images of the same target arch. Just
using /tmp/<image-name> can cause conflicts and build failures.

Seen with jailhouse-images.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---

Changes in v2:
 - avoid back and forth with BUILDCHROOT_WICTMP and WICTMP

 meta/classes/wic-img.bbclass | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass
index 1ccbe55..795640f 100644
--- a/meta/classes/wic-img.bbclass
+++ b/meta/classes/wic-img.bbclass
@@ -144,14 +144,17 @@ EOSUDO
     export BUILDDIR=${BUILDDIR}
     export MTOOLS_SKIP_CHECK=1
 
+    # create the temp dir in the buildchroot to ensure uniqueness
+    WICTMP=$(cd ${BUILDCHROOT_DIR}; mktemp -d -p tmp)
+
     sudo -E chroot ${BUILDCHROOT_DIR} \
         ${ISARROOT}/scripts/wic create ${WKS_FULL_PATH} \
             --vars "${STAGING_DIR}/${MACHINE}/imgdata/" \
-            -o /tmp/${IMAGE_FULLNAME}.wic/ \
+            -o /$WICTMP/${IMAGE_FULLNAME}.wic/ \
             --bmap \
             -e ${IMAGE_BASENAME} ${WIC_CREATE_EXTRA_ARGS}
     sudo chown -R $(stat -c "%U" ${ISARROOT}) ${ISARROOT}/meta ${ISARROOT}/meta-isar ${ISARROOT}/scripts || true
-    WIC_DIRECT=$(ls -t -1 ${BUILDCHROOT_DIR}/tmp/${IMAGE_FULLNAME}.wic/*.direct | head -1)
+    WIC_DIRECT=$(ls -t -1 ${BUILDCHROOT_DIR}/$WICTMP/${IMAGE_FULLNAME}.wic/*.direct | head -1)
     cp -f ${WIC_DIRECT} ${WIC_IMAGE_FILE}
     cp -f ${WIC_DIRECT}.bmap ${WIC_IMAGE_FILE}.bmap
 }
-- 
2.16.4

Re: [PATCH 00/26] Tons of fixes and cleanups

[Baurzhan Ismagulov]

On Mon, Aug 26, 2019 at 08:26:01AM +0200, Jan Kiszka wrote:
> Highlights:
> - fixed custom DISTRO_BOOTSTRAP_KEYS
> - fixed shared lock across dpkg_runbuild (nasty stuff...)
> - reworked STAMPing
> - fixed follow-up errors on build terminations (error or ctrl-c)
> - fixed two false sharings
> - detection of false task sharing
> - stop mangling PF
> - no more tasks beyond do_build
> - fixed isar-bootstrap rebuild
> - deploy logs on gitlab-ci errors

Applied to next, thanks.

With kind regards,
Baurzhan.