[PATCH 0/6] dpkg class refactorings, API changelog, fixes

[PATCH 0/6] dpkg class refactorings, API changelog, fixes

[Jan Kiszka]

Changes in v2:
 - re-add lost proxy settings to custom kernel build
 - truly fix depmod run of module recipes
 - account for dpkg task split also in custom kernel recipe

Test coverages remarkes:
 - we truly miss a custom kernel + module build in the upstream tests
 - we also lack a test for consistent proxy setting propation

The former is currently only addressed by jailhouse-images, the latter
by running things through our Intranet-hosted CI. Both is not optimal.

Jan

Jan Kiszka (6):
  linux-custom: Add missing proxy settings to prepare task
  linux-module: Run depmod for installed kernel versions
  dpkg-base: Relocate dpkg_runbuild and adjust comments
  dkpg-base: Clarify dependency logic between package recipes
  dpkg, linux-custom: Split dependency installation from build
    preparations
  Add changelog for the recipe API

 RECIPE-API-CHANGELOG.md                            | 59 ++++++++++++++++++++++
 meta/classes/dpkg-base.bbclass                     | 31 +++++-------
 meta/classes/dpkg-raw.bbclass                      |  4 +-
 meta/classes/dpkg.bbclass                          |  8 ++-
 .../linux-module/files/debian/postinst             |  2 +-
 meta/recipes-kernel/linux-module/module.inc        |  2 +-
 meta/recipes-kernel/linux/linux-custom.inc         |  7 ++-
 7 files changed, 88 insertions(+), 25 deletions(-)
 create mode 100644 RECIPE-API-CHANGELOG.md

-- 
2.16.4

[PATCH 1/6] linux-custom: Add missing proxy settings to prepare task

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Lost by a70dd2339cc7.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-kernel/linux/linux-custom.inc | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index d75346e..57f2f82 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -29,6 +29,7 @@ KERNEL_DEBIAN_DEPENDS ?= "initramfs-tools | linux-initramfs-tool, kmod, linux-ba
 KERNEL_HEADERS_DEBIAN_DEPENDS ?= "libc6, libssl1.1"
 
 dpkg_prepare() {
+	E="${@ bb.utils.export_proxies(d)}"
 	sudo -E chroot ${BUILDCHROOT_DIR} \
 		apt-get install \
 			-y -o Debug::pkgProblemResolver=yes \
-- 
2.16.4

[PATCH 2/6] linux-module: Run depmod for installed kernel versions

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

The return code of the postinst script it the one of the last executed
instruction. Unfortunately, depmod tend to fail when run on the host
during rootfs generation because host kernel and target kernel usually
differ.

To overcome this, pick up the kernel version(s) by looking at the
content of /lib/modules.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-kernel/linux-module/files/debian/postinst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-kernel/linux-module/files/debian/postinst b/meta/recipes-kernel/linux-module/files/debian/postinst
index 35d5de0..ed9d312 100755
--- a/meta/recipes-kernel/linux-module/files/debian/postinst
+++ b/meta/recipes-kernel/linux-module/files/debian/postinst
@@ -1,2 +1,2 @@
 #!/bin/sh
-depmod -a
+depmod -a $(ls /lib/modules)
-- 
2.16.4

[PATCH 3/6] dpkg-base: Relocate dpkg_runbuild and adjust comments

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This puts this placeholder in the context of its task definition and
also improves the comments related to dpkg_runbuild as well as
dpkg_prepare. No functional changes.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 5826357..84c56af 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -50,13 +50,10 @@ PPS ?= "${@get_package_srcdir(d)}"
 BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
 do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
-# default to "emtpy" implementation for dpkg_prepare() and dpkg_runbuild()
+# Empty do_prepare() implementation, to be overwritten if needed
 dpkg_prepare() {
     true
 }
-dpkg_runbuild() {
-    die "This should never be called, overwrite it in your derived class"
-}
 
 MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock"
 
@@ -89,6 +86,11 @@ do_prepare() {
 addtask prepare after do_patch before do_build
 do_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
+# Placeholder for actual dpkg_runbuild() implementation
+dpkg_runbuild() {
+    die "This should never be called, overwrite it in your derived class"
+}
+
 do_build() {
     dpkg_do_mounts
     dpkg_runbuild
-- 
2.16.4

[PATCH 4/6] dkpg-base: Clarify dependency logic between package recipes

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

No functional change, just reordering and comment enhancement.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 84c56af..6299ef8 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -3,6 +3,8 @@
 
 ISAR_CROSS_COMPILE ??= "0"
 
+DEPENDS ?= ""
+
 # Add dependency from the correct buildchroot: host or target
 python __anonymous() {
     mode = d.getVar('ISAR_CROSS_COMPILE', True)
@@ -30,10 +32,6 @@ do_adjust_git[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 inherit patch
 addtask patch after do_adjust_git before do_build
 
-# Add dependency between Isar recipes
-DEPENDS ?= ""
-do_prepare[deptask] = "do_deploy_deb"
-
 def get_package_srcdir(d):
     s = d.getVar("S", True)
     workdir = d.getVar("WORKDIR", True)
@@ -85,6 +83,9 @@ do_prepare() {
 
 addtask prepare after do_patch before do_build
 do_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+# If Isar recipes depend on each other, they typically need the package
+# deployed to isar-apt
+do_prepare[deptask] = "do_deploy_deb"
 
 # Placeholder for actual dpkg_runbuild() implementation
 dpkg_runbuild() {
-- 
2.16.4

[PATCH 5/6] dpkg, linux-custom: Split dependency installation from build preparations

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Only the former needs locking, and the latter benefits from clearer
naming: do_prepare_build is supposed to run prior to the build. The
dependency installation task is now from the user perspective part of
the build again, even when run separately directly before it.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass              | 24 +++++++++---------------
 meta/classes/dpkg-raw.bbclass               |  4 ++--
 meta/classes/dpkg.bbclass                   |  8 ++++++--
 meta/recipes-kernel/linux-module/module.inc |  2 +-
 meta/recipes-kernel/linux/linux-custom.inc  |  6 ++++--
 5 files changed, 22 insertions(+), 22 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 6299ef8..a2d2ff0 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -15,7 +15,7 @@ python __anonymous() {
         dep = "buildchroot-host:do_build"
         rootfs = d.getVar('BUILDCHROOT_HOST_DIR', True)
 
-    d.setVarFlag('do_prepare', 'depends', dep)
+    d.setVarFlag('do_prepare_build', 'depends', dep)
     d.setVar('BUILDCHROOT_DIR', rootfs)
 }
 
@@ -48,11 +48,17 @@ PPS ?= "${@get_package_srcdir(d)}"
 BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
 do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
-# Empty do_prepare() implementation, to be overwritten if needed
-dpkg_prepare() {
+# Empty do_prepare_build() implementation, to be overwritten if needed
+do_prepare_build() {
     true
 }
 
+addtask prepare_build after do_patch before do_build
+do_prepare_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+# If Isar recipes depend on each other, they typically need the package
+# deployed to isar-apt
+do_prepare_build[deptask] = "do_deploy_deb"
+
 MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock"
 
 # Wrap the function dpkg_runbuild with the bind mount for buildroot
@@ -75,18 +81,6 @@ dpkg_undo_mounts() {
     sudo rmdir ${BUILDROOT} 2>/dev/null || true
 }
 
-do_prepare() {
-    dpkg_do_mounts
-    dpkg_prepare
-    dpkg_undo_mounts
-}
-
-addtask prepare after do_patch before do_build
-do_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-# If Isar recipes depend on each other, they typically need the package
-# deployed to isar-apt
-do_prepare[deptask] = "do_deploy_deb"
-
 # Placeholder for actual dpkg_runbuild() implementation
 dpkg_runbuild() {
     die "This should never be called, overwrite it in your derived class"
diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index c5f1c88..c848f3d 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -14,7 +14,7 @@ do_install() {
 }
 
 do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-addtask install after do_unpack before do_prepare
+addtask install after do_unpack before do_prepare_build
 
 deb_package_prepare() {
 	sudo rm -rf ${D}/DEBIAN
@@ -48,7 +48,7 @@ deb_package_conffiles() {
 	test -s $CONFFILES || rm $CONFFILES
 }
 
-dpkg_prepare() {
+do_prepare_build() {
 	deb_package_prepare
 	deb_package_conffiles
 }
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 2e53c1b..da0b40d 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -4,13 +4,17 @@
 inherit dpkg-base
 
 # Install build dependencies for package
-dpkg_prepare() {
+do_install_builddeps() {
+    dpkg_do_mounts
     E="${@ bb.utils.export_proxies(d)}"
     sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh ${PP}/${PPS} ${DISTRO_ARCH}
+    dpkg_undo_mounts
 }
 
+addtask install_builddeps after do_prepare_build before do_build
 # apt and reprepro may not run in parallel, acquire the Isar lock
-do_prepare[lockfiles] += "${DEPLOY_DIR_APT}/isar.lock"
+do_install_builddeps[lockfiles] += "${DEPLOY_DIR_APT}/isar.lock"
+do_install_builddeps[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 # Build package from sources using build script
 dpkg_runbuild() {
diff --git a/meta/recipes-kernel/linux-module/module.inc b/meta/recipes-kernel/linux-module/module.inc
index 7bc29a8..08d2c25 100644
--- a/meta/recipes-kernel/linux-module/module.inc
+++ b/meta/recipes-kernel/linux-module/module.inc
@@ -17,7 +17,7 @@ AUTOLOAD ?= "0"
 
 inherit dpkg
 
-do_prepare_prepend() {
+do_prepare_build() {
     cp -r ${WORKDIR}/debian ${S}/
     sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \
            -e 's/@KERNEL_NAME@/${KERNEL_NAME}/g' \
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 57f2f82..4144b67 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -28,7 +28,7 @@ KBUILD_DEPENDS ?= "libssl-dev libelf-dev bc git"
 KERNEL_DEBIAN_DEPENDS ?= "initramfs-tools | linux-initramfs-tool, kmod, linux-base (>= 4.3~)"
 KERNEL_HEADERS_DEBIAN_DEPENDS ?= "libc6, libssl1.1"
 
-dpkg_prepare() {
+do_install_builddeps() {
 	E="${@ bb.utils.export_proxies(d)}"
 	sudo -E chroot ${BUILDCHROOT_DIR} \
 		apt-get install \
@@ -36,8 +36,10 @@ dpkg_prepare() {
 			--no-install-recommends ${KBUILD_DEPENDS}
 }
 
+addtask install_builddeps after do_prepare_build before do_build
 # apt and reprepro may not run in parallel, acquire the Isar lock
-do_prepare[lockfiles] += "${DEPLOY_DIR_APT}/isar.lock"
+do_install_builddeps[lockfiles] += "${DEPLOY_DIR_APT}/isar.lock"
+do_install_builddeps[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 dpkg_runbuild() {
 	# Install package builder script
-- 
2.16.4

[PATCH 6/6] Add changelog for the recipe API

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This uses the last release v0.5 as baseline and documents changes that
affect own recipes and layers using Isar. Hopefully, this caught all of
them since v0.5.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 RECIPE-API-CHANGELOG.md | 59 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 59 insertions(+)
 create mode 100644 RECIPE-API-CHANGELOG.md

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
new file mode 100644
index 0000000..601fc6b
--- /dev/null
+++ b/RECIPE-API-CHANGELOG.md
@@ -0,0 +1,59 @@
+Recipe API Changelog
+====================
+
+Baseline: Release v0.5
+
+Changes in v0.6
+---------------
+
+### Separate do_prepare_build task
+
+All Debianizations of source trees and any further programmatic patching has to
+be moved from the dpkg_runbuild() task to this new task.
+
+### ISAR_CROSS_COMPILE support
+
+Isar now supports cross-compilation. When enabled via `ISAR_CROSS_COMPILE = "1"`
+in a global configuration file, all recipes that do not overwrite this variable
+will build cross-compiled.
+
+If a recipe is not cross-compatible, it should set `ISAR_CROSS_COMPILE = "0"`.
+Then also all its dependencies that are built by Isar must opt out from
+cross-building.
+
+### wic image type, removal of manual wic invocation
+
+Images that are described by wic files are now generated during the build by
+setting `IMAGE_TYPE = "wic-img"`. The manual invocation of wic after the
+bitbake run is no longer needed nor supported.
+
+### Optional kernel installation
+
+The installation of a Linux kernel into the isar-image-base can be skipped by
+setting `KERNEL_NAME = ""`.
+
+### Corrected semantic of `S`
+
+The `S` variable is now defined as NOT including `WORKDIR`, like in OE, Gentoo
+and others. If it was set to, e.g., `S = "sources"` in a recipe so far, it must
+be defined as `S = ${WORKDIR}/sources` from now on.
+
+### DISTRO and DISTRO_ARCH are available as OVERRIDES
+
+Bitbake variables can now also refer to the DISTRO as well as the DISTRO_ARCH
+for overrides.
+
+### Set ISAR_RELEASE_CMD in own top-layer
+
+Isar now populates /etc/os-release with information about the image. In order
+to identify the revision of the top layer that controlled the image build with
+all its dependencies, set ISAR_RELEASE_CMD so that it picks up the required
+information.
+
+If the top-layer is managed in git, set `LAYERDIR_mylayer = "${LAYERDIR}"` in
+`conf/layer.conf` and add something along
+
+    ISAR_RELEASE_CMD = "git -C ${LAYERDIR_mylayer} describe --tags --dirty \
+                            --match 'v[0-9].[0-9]*'"
+
+in the image recipe (or `isar-image-base.bbappend`) of that layer.
-- 
2.16.4

Re: [PATCH 0/6] dpkg class refactorings, API changelog, fixes

[Jan Kiszka]

On 2018-09-02 18:04, [ext] Jan Kiszka wrote:
> Changes in v2:

Err, the "v2" tag is missing in the patches... Sorry.

Jan

>   - re-add lost proxy settings to custom kernel build
>   - truly fix depmod run of module recipes
>   - account for dpkg task split also in custom kernel recipe
> 
> Test coverages remarkes:
>   - we truly miss a custom kernel + module build in the upstream tests
>   - we also lack a test for consistent proxy setting propation
> 
> The former is currently only addressed by jailhouse-images, the latter
> by running things through our Intranet-hosted CI. Both is not optimal.
> 
> Jan
> 
> Jan Kiszka (6):
>    linux-custom: Add missing proxy settings to prepare task
>    linux-module: Run depmod for installed kernel versions
>    dpkg-base: Relocate dpkg_runbuild and adjust comments
>    dkpg-base: Clarify dependency logic between package recipes
>    dpkg, linux-custom: Split dependency installation from build
>      preparations
>    Add changelog for the recipe API
> 
>   RECIPE-API-CHANGELOG.md                            | 59 ++++++++++++++++++++++
>   meta/classes/dpkg-base.bbclass                     | 31 +++++-------
>   meta/classes/dpkg-raw.bbclass                      |  4 +-
>   meta/classes/dpkg.bbclass                          |  8 ++-
>   .../linux-module/files/debian/postinst             |  2 +-
>   meta/recipes-kernel/linux-module/module.inc        |  2 +-
>   meta/recipes-kernel/linux/linux-custom.inc         |  7 ++-
>   7 files changed, 88 insertions(+), 25 deletions(-)
>   create mode 100644 RECIPE-API-CHANGELOG.md
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 0/6] dpkg class refactorings, API changelog, fixes

[Maxim Yu. Osipov]

On 09/02/2018 07:04 PM, Jan Kiszka wrote:
> Changes in v2:
>   - re-add lost proxy settings to custom kernel build
>   - truly fix depmod run of module recipes
>   - account for dpkg task split also in custom kernel recipe
> 
> Test coverages remarkes:
>   - we truly miss a custom kernel + module build in the upstream tests
>   - we also lack a test for consistent proxy setting propation
> 
> The former is currently only addressed by jailhouse-images, the latter
> by running things through our Intranet-hosted CI. Both is not optimal.

Applied to the 'next',

Thanks,
Maxim.

> Jan
> 
> Jan Kiszka (6):
>    linux-custom: Add missing proxy settings to prepare task
>    linux-module: Run depmod for installed kernel versions
>    dpkg-base: Relocate dpkg_runbuild and adjust comments
>    dkpg-base: Clarify dependency logic between package recipes
>    dpkg, linux-custom: Split dependency installation from build
>      preparations
>    Add changelog for the recipe API
> 
>   RECIPE-API-CHANGELOG.md                            | 59 ++++++++++++++++++++++
>   meta/classes/dpkg-base.bbclass                     | 31 +++++-------
>   meta/classes/dpkg-raw.bbclass                      |  4 +-
>   meta/classes/dpkg.bbclass                          |  8 ++-
>   .../linux-module/files/debian/postinst             |  2 +-
>   meta/recipes-kernel/linux-module/module.inc        |  2 +-
>   meta/recipes-kernel/linux/linux-custom.inc         |  7 ++-
>   7 files changed, 88 insertions(+), 25 deletions(-)
>   create mode 100644 RECIPE-API-CHANGELOG.md
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov