[PATCH 00/10] Pending queue: various fixes, improve rebuilds on dependency changes

[PATCH 00/10] Pending queue: various fixes, improve rebuilds on dependency changes

[Jan Kiszka]

Besides various smaller cleanups and fixes, this specifically improves
the tracking of local files that are dependencies of the build so that
we trigger partial rebuilds when they change.

Note that I'm including some previously sent u-boot-custom patches here
again, just to have everything in one series.

Jan

Jan Kiszka (10):
  u-boot-custom: Fix parallel build
  u-boot-custom: Fix comment
  u-boot-custom: Allow to extend or replace U-Boot build dependencies
  meta: Add bbclass to retrieve selected WKS file
  u-boot-script: Switch to wks-file class for WKS_FILE search
  wic-img: Use wks-file class for WKS_FILE lookup
  wic-img: Add WKS_FILE to wic_image task checksum
  base: Improve detection of dependency changes
  linux-custom: Fix typo in KERNEL_NAME_PROVIDED variable
  wic-img: Fix image copy-out in do_wic_image

 meta/classes/base.bbclass                          |  2 ++
 meta/classes/wic-img.bbclass                       | 13 ++++++++--
 meta/classes/wks-file.bbclass                      | 30 ++++++++++++++++++++++
 .../recipes-bsp/u-boot-script/u-boot-script_1.0.bb | 15 +++--------
 meta/recipes-bsp/u-boot/files/debian/control       |  2 +-
 meta/recipes-bsp/u-boot/files/debian/rules         |  4 +--
 meta/recipes-bsp/u-boot/u-boot-custom.inc          |  3 +++
 meta/recipes-kernel/linux/linux-custom.inc         |  4 +--
 8 files changed, 55 insertions(+), 18 deletions(-)
 create mode 100644 meta/classes/wks-file.bbclass

-- 
2.16.4

[PATCH 01/10] u-boot-custom: Fix parallel build

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Do to missing "echo", we were running with "-j" so far, and the build
log contained some complains about unknown commands.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-bsp/u-boot/files/debian/rules | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-bsp/u-boot/files/debian/rules b/meta/recipes-bsp/u-boot/files/debian/rules
index deb7c8e..1d9e645 100755
--- a/meta/recipes-bsp/u-boot/files/debian/rules
+++ b/meta/recipes-bsp/u-boot/files/debian/rules
@@ -12,7 +12,7 @@ export CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)-
 SET_CROSS_BUILD_TOOLS=CROSS_BUILD_TOOLS=y
 endif
 
-PARALLEL_BUILD=-j $(shell $$(($$(nproc) * 2)))
+PARALLEL_BUILD=-j $(shell echo $$(($$(nproc) * 2)))
 
 override_dh_auto_build:
 	$(MAKE) $(PARALLEL_BUILD) $(U_BOOT_CONFIG)
-- 
2.16.4

[PATCH 02/10] u-boot-custom: Fix comment

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-bsp/u-boot/files/debian/rules | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-bsp/u-boot/files/debian/rules b/meta/recipes-bsp/u-boot/files/debian/rules
index 1d9e645..3f7a321 100755
--- a/meta/recipes-bsp/u-boot/files/debian/rules
+++ b/meta/recipes-bsp/u-boot/files/debian/rules
@@ -1,6 +1,6 @@
 #!/usr/bin/make -f
 
-# Debian rules for custom kernel module build
+# Debian rules for custom U-Boot build
 #
 # This software is a part of ISAR.
 # Copyright (c) Siemens AG, 2018
-- 
2.16.4

[PATCH 03/10] u-boot-custom: Allow to extend or replace U-Boot build dependencies

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Some U-Boot configurations may requires additional host-side tools.
Allow to add them without major hacks.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-bsp/u-boot/files/debian/control | 2 +-
 meta/recipes-bsp/u-boot/u-boot-custom.inc    | 3 +++
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-bsp/u-boot/files/debian/control b/meta/recipes-bsp/u-boot/files/debian/control
index df23a69..6b4c839 100644
--- a/meta/recipes-bsp/u-boot/files/debian/control
+++ b/meta/recipes-bsp/u-boot/files/debian/control
@@ -2,7 +2,7 @@ Source: @PN@
 Section: admin
 Priority: optional
 Standards-Version: 3.9.6
-Build-Depends: bc, bison, flex, device-tree-compiler
+Build-Depends: @BUILD_DEPENDS@
 Maintainer: ISAR project <isar-users@googlegroups.com>
 
 Package: u-boot-@MACHINE@
diff --git a/meta/recipes-bsp/u-boot/u-boot-custom.inc b/meta/recipes-bsp/u-boot/u-boot-custom.inc
index f3fca00..583cd47 100644
--- a/meta/recipes-bsp/u-boot/u-boot-custom.inc
+++ b/meta/recipes-bsp/u-boot/u-boot-custom.inc
@@ -15,9 +15,12 @@ inherit dpkg
 
 SRC_URI += "file://debian/"
 
+U_BOOT_BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler"
+
 do_prepare_build() {
     cp -r ${WORKDIR}/debian ${S}/
     sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \
+        -e 's/@BUILD_DEPENDS@/${U_BOOT_BUILD_DEPENDS}/g' \
         -e 's/@MACHINE@/${MACHINE}/g' \
         -e 's/@DISTRO_ARCH@/${DISTRO_ARCH}/g' \
         -e 's/@DESCRIPTION@/${DESCRIPTION}/g' \
-- 
2.16.4

[PATCH 04/10] meta: Add bbclass to retrieve selected WKS file

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This implements a lookup service for the target's WKS_FILE. The goal is
to concentrate the search in a single place in order to provide a
consistent logic.

In addition to permitting absolute paths, get_wks_full_path() implements
almost all search paths of OE:

 - <BBPATH>/wic
 - <BBPATH>/scripts/lib/wic/canned-wks
 - <COREBASE>/scripts/lib/wic/canned-wks

It does not support THISDIR for wic-image recipes because that would
restrict the visibility of that file to a single recipe while we would
like to use it freely, also in other recipes such as u-boot-script.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/wks-file.bbclass | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
 create mode 100644 meta/classes/wks-file.bbclass

diff --git a/meta/classes/wks-file.bbclass b/meta/classes/wks-file.bbclass
new file mode 100644
index 0000000..493fea7
--- /dev/null
+++ b/meta/classes/wks-file.bbclass
@@ -0,0 +1,30 @@
+# This software is a part of ISAR.
+# Copyright (C) 2018 Siemens AG
+#
+# SPDX-License-Identifier: MIT
+
+def get_wks_full_path(d):
+    if (d.getVar('IMAGE_TYPE', True) or '') != 'wic-img':
+        return ""
+
+    wks_full_path = None
+
+    wks_file = d.getVar('WKS_FILE', True)
+    if not wks_file.endswith('.wks'):
+        wks_file += '.wks'
+
+    if os.path.isabs(wks_file):
+        if os.path.exists(wks_file):
+            wks_full_path = wks_file
+    else:
+        bbpaths = d.getVar('BBPATH', True).split(':')
+        corebase = d.getVar('COREBASE', True)
+        search_path = ':'.join('%s/wic' % p for p in bbpaths) + ':' + \
+            ':'.join('%s/scripts/lib/wic/canned-wks' % l \
+                     for l in (bbpaths + [corebase]))
+        wks_full_path = bb.utils.which(search_path, wks_file)
+
+    if not wks_full_path:
+        bb.fatal("WKS_FILE '%s' not found" % wks_file)
+
+    return wks_full_path
-- 
2.16.4

[PATCH 05/10] u-boot-script: Switch to wks-file class for WKS_FILE search

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Use the new central search implementation.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-bsp/u-boot-script/u-boot-script_1.0.bb | 15 ++++-----------
 1 file changed, 4 insertions(+), 11 deletions(-)

diff --git a/meta/recipes-bsp/u-boot-script/u-boot-script_1.0.bb b/meta/recipes-bsp/u-boot-script/u-boot-script_1.0.bb
index 8d1ad25..768a94f 100644
--- a/meta/recipes-bsp/u-boot-script/u-boot-script_1.0.bb
+++ b/meta/recipes-bsp/u-boot-script/u-boot-script_1.0.bb
@@ -6,6 +6,9 @@
 # SPDX-License-Identifier: MIT
 
 inherit dpkg-raw
+inherit wks-file
+
+WKS_FULL_PATH = "${@get_wks_full_path(d)}"
 
 DESCRIPTION = "Boot script generator for U-Boot"
 
@@ -61,17 +64,7 @@ init_config_from_wks() {
 }
 
 do_install() {
-	# Find WKS_FILE specified for the current target.
-	WKS_DIRS=$(dirname $(which wic))/lib/wic/canned-wks
-	for LAYER in ${BBLAYERS}; do
-		WKS_DIRS="${WKS_DIRS} ${LAYER}/scripts/lib/wic/canned-wks"
-	done
-	for DIR in ${WKS_DIRS}; do
-		if [ -f ${DIR}/${WKS_FILE}.wks ]; then
-			init_config_from_wks ${DIR}/${WKS_FILE}.wks
-			break
-		fi
-	done
+	[ -n ${WKS_FULL_PATH} ] && init_config_from_wks "${WKS_FULL_PATH}"
 
 	sudo rm -rf ${D}/etc ${D}/usr
 
-- 
2.16.4

[PATCH 06/10] wic-img: Use wks-file class for WKS_FILE lookup

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Rather than letting wic search only in the canned-wks folders, use
get_wks_full_path search for WKS_FILE at all newly defined locations.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/wic-img.bbclass | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass
index fef8308..c80c927 100644
--- a/meta/classes/wic-img.bbclass
+++ b/meta/classes/wic-img.bbclass
@@ -9,6 +9,10 @@ python () {
         bb.fatal("WKS_FILE must be set")
 }
 
+inherit wks-file
+
+WKS_FULL_PATH = "${@get_wks_full_path(d)}"
+
 # wic comes with reasonable defaults, and the proper interface is the wks file
 ROOTFS_EXTRA ?= "0"
 
@@ -75,7 +79,10 @@ do_wic_image() {
     export BUILDDIR=${BUILDDIR}
     export MTOOLS_SKIP_CHECK=1
 
-    sudo -E chroot ${BUILDCHROOT_DIR} ${ISARROOT}/scripts/wic create ${WKS_FILE} --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -o /tmp/ -e ${IMAGE_BASENAME} ${WIC_CREATE_EXTRA_ARGS}
+    sudo -E chroot ${BUILDCHROOT_DIR} \
+        ${ISARROOT}/scripts/wic create ${WKS_FULL_PATH} \
+            --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -o /tmp/ \
+            -e ${IMAGE_BASENAME} ${WIC_CREATE_EXTRA_ARGS}
     sudo chown -R $(stat -c "%U" ${ISARROOT}) ${ISARROOT}/meta ${ISARROOT}/meta-isar ${ISARROOT}/scripts || true
     cp -f `ls -t -1 ${BUILDCHROOT_DIR}/tmp/${WKS_FILE}*.direct | head -1` ${WIC_IMAGE_FILE}
 }
-- 
2.16.4

[PATCH 07/10] wic-img: Add WKS_FILE to wic_image task checksum

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This ensures the task is rebuilt whenever the wks file changes.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/wic-img.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass
index c80c927..ee7aff1 100644
--- a/meta/classes/wic-img.bbclass
+++ b/meta/classes/wic-img.bbclass
@@ -87,6 +87,7 @@ do_wic_image() {
     cp -f `ls -t -1 ${BUILDCHROOT_DIR}/tmp/${WKS_FILE}*.direct | head -1` ${WIC_IMAGE_FILE}
 }
 
+do_wic_image[file-checksums] += "${WKS_FULL_PATH}:True"
 do_wic_image[depends] = "buildchroot-target:do_build"
 
 addtask wic_image before do_build after do_install_imager_deps
-- 
2.16.4

[PATCH 08/10] base: Improve detection of dependency changes

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This adds two reasons the fetch task should be redone:

 - the content of a local file:// changed
 - a SRCREV value for a repository changed

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/base.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 99730e5..fce1084 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -97,6 +97,8 @@ python() {
 }
 
 do_fetch[dirs] = "${DL_DIR}"
+do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
+do_fetch[vardeps] += "SRCREV"
 
 # Fetch package from the source link
 python do_fetch() {
-- 
2.16.4

[PATCH 09/10] linux-custom: Fix typo in KERNEL_NAME_PROVIDED variable

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

It was documented and intended to be called "KERNEL_NAME_PROVIDED".

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-kernel/linux/linux-custom.inc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 34c9a62..7c6b624 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -9,11 +9,11 @@ FILESPATH =. "${LAYERDIR_core}/recipes-kernel/linux/files:"
 
 DESCRIPTION ?= "Custom kernel"
 
-KERNEL_NAME_PRODVIDED ?= "${@ d.getVar('PN', True).partition('linux-')[2]}"
+KERNEL_NAME_PROVIDED ?= "${@ d.getVar('PN', True).partition('linux-')[2]}"
 KERNEL_DEFCONFIG ?= "defconfig"
 
 python() {
-    kernel_name = d.getVar("KERNEL_NAME_PRODVIDED", True)
+    kernel_name = d.getVar("KERNEL_NAME_PROVIDED", True)
     d.setVar('PROVIDES', 'linux-image-' + kernel_name + ' ' + \
                          'linux-headers-' + kernel_name)
 }
-- 
2.16.4

[PATCH 10/10] wic-img: Fix image copy-out in do_wic_image

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

The magic matching of the output file against WKS_FILE breaks if the
user provided a full path or appended the ".wks" suffix in that
variable. Avoid the fragility by pushing the output into a dedicated,
target-specific tmp directory and picking up the latest .direct file
from there.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/wic-img.bbclass | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass
index ee7aff1..5ff8b45 100644
--- a/meta/classes/wic-img.bbclass
+++ b/meta/classes/wic-img.bbclass
@@ -81,10 +81,11 @@ do_wic_image() {
 
     sudo -E chroot ${BUILDCHROOT_DIR} \
         ${ISARROOT}/scripts/wic create ${WKS_FULL_PATH} \
-            --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -o /tmp/ \
+            --vars "${STAGING_DIR}/${MACHINE}/imgdata/" \
+            -o /tmp/${PN}-${DISTRO}-${MACHINE}.wic/ \
             -e ${IMAGE_BASENAME} ${WIC_CREATE_EXTRA_ARGS}
     sudo chown -R $(stat -c "%U" ${ISARROOT}) ${ISARROOT}/meta ${ISARROOT}/meta-isar ${ISARROOT}/scripts || true
-    cp -f `ls -t -1 ${BUILDCHROOT_DIR}/tmp/${WKS_FILE}*.direct | head -1` ${WIC_IMAGE_FILE}
+    cp -f $(ls -t -1 ${BUILDCHROOT_DIR}/tmp/${PN}-${DISTRO}-${MACHINE}.wic/*.direct | head -1) ${WIC_IMAGE_FILE}
 }
 
 do_wic_image[file-checksums] += "${WKS_FULL_PATH}:True"
-- 
2.16.4

Re: [PATCH 00/10] Pending queue: various fixes, improve rebuilds on dependency changes

[Maxim Yu. Osipov]

On 10/8/18 2:01 PM, Jan Kiszka wrote:
> Besides various smaller cleanups and fixes, this specifically improves
> the tracking of local files that are dependencies of the build so that
> we trigger partial rebuilds when they change.
> 
> Note that I'm including some previously sent u-boot-custom patches here
> again, just to have everything in one series.

Applied to the 'next',

Thanks,
Maxim.

> Jan
> 
> Jan Kiszka (10):
>    u-boot-custom: Fix parallel build
>    u-boot-custom: Fix comment
>    u-boot-custom: Allow to extend or replace U-Boot build dependencies
>    meta: Add bbclass to retrieve selected WKS file
>    u-boot-script: Switch to wks-file class for WKS_FILE search
>    wic-img: Use wks-file class for WKS_FILE lookup
>    wic-img: Add WKS_FILE to wic_image task checksum
>    base: Improve detection of dependency changes
>    linux-custom: Fix typo in KERNEL_NAME_PROVIDED variable
>    wic-img: Fix image copy-out in do_wic_image
> 
>   meta/classes/base.bbclass                          |  2 ++
>   meta/classes/wic-img.bbclass                       | 13 ++++++++--
>   meta/classes/wks-file.bbclass                      | 30 ++++++++++++++++++++++
>   .../recipes-bsp/u-boot-script/u-boot-script_1.0.bb | 15 +++--------
>   meta/recipes-bsp/u-boot/files/debian/control       |  2 +-
>   meta/recipes-bsp/u-boot/files/debian/rules         |  4 +--
>   meta/recipes-bsp/u-boot/u-boot-custom.inc          |  3 +++
>   meta/recipes-kernel/linux/linux-custom.inc         |  4 +--
>   8 files changed, 55 insertions(+), 18 deletions(-)
>   create mode 100644 meta/classes/wks-file.bbclass
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

[PATCH 05/10] u-boot-script: Switch to wks-file class for WKS_FILE search

[chombourger]

[-- Attachment #1: Type: text/plain, Size: 386 bytes --]

Hi Jan,

It may be worth noting that the wks-file class will throw an exception when bitbake goes through available layers to parse recipes that it finds.
This may be an issue in environments / configurations where no WKS_FILE is set even if this particular recipe isn't use. Are we mandating the use of wic now even for users only wanting ISAR to produce eg a plain ext4 image?

Cedric

Re: [PATCH 05/10] u-boot-script: Switch to wks-file class for WKS_FILE search

[Jan Kiszka]

Hi Cedric,

On 13.10.18 08:05, chombourger@gmail.com wrote:
> Hi Jan,
> 
> It may be worth noting that the wks-file class will throw an exception when bitbake goes through available layers to parse recipes that it finds.
> This may be an issue in environments / configurations where no WKS_FILE is set even if this particular recipe isn't use. Are we mandating the use of wic now even for users only wanting ISAR to produce eg a plain ext4 image?

Does your environment incorrectly sets IMAGE_TYPE to wic-img? There is test at 
the beginning of get_wks_full_path that filters out non-wic images based on this 
condition. It shouldn't make sense to set IMAGE_TYPE this way and then not 
defining a proper WKS_FILE.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 05/10] u-boot-script: Switch to wks-file class for WKS_FILE search

[Cedric Hombourger]

Hi Jan,

Thanks for the hint. That was exactly the issue, I had noticed that it was set but should have checked where
Sorry for the noise then!

Cedric

> On Oct 13, 2018, at 3:53 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> 
> Hi Cedric,
> 
> On 13.10.18 08:05, chombourger@gmail.com wrote:
>> Hi Jan,
>> It may be worth noting that the wks-file class will throw an exception when bitbake goes through available layers to parse recipes that it finds.
>> This may be an issue in environments / configurations where no WKS_FILE is set even if this particular recipe isn't use. Are we mandating the use of wic now even for users only wanting ISAR to produce eg a plain ext4 image?
> 
> Does your environment incorrectly sets IMAGE_TYPE to wic-img? There is test at the beginning of get_wks_full_path that filters out non-wic images based on this condition. It shouldn't make sense to set IMAGE_TYPE this way and then not defining a proper WKS_FILE.
> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux