[PATCH v10 0/2] Move kernel artifacts deployment from image recipe

[PATCH v10 0/2] Move kernel artifacts deployment from image recipe

[Zhihang Wei]

This patch set should be applied on top of the series "[PATCH 0/3]
Revert "prefix DTB files with PN in deploy dir"".

Image task "do_copy_boot_files" is used to deploy kernel, initrd and
devicetree binaries to deploy directory.

When different images for the same target are built (e.g., "-base" and
"-debug") in parallel, this causes issues with DTB deployment since
they have no any image_name-specific stuff in the path, unlike kernel
and initrd.

Here we make a separate recipe responsible for the DTB deployment.
DTBs are extracted from the kernel package (provided by distro or
self-built).

The recipe 'kernel-deploy' is kernel-dependent, so different DTB_FILES
may be deployed for different machines. At the same time, all these
'kernel-deploy-${MACHINE}' targets depend on the same kernel package,
so it is built only once.

The case when different distros (e.g., bullseye/bookworm) are using the
same name for DTB binaries is covered by the patch 3. It also covers
the case when image uses different kernel suffix (e.g., "realtime"
option for the kernel in some downstreams).

Related topics on maillist:
https://lists.isar-build.org/isar-users/20250403170701.807745-1-badrikesh.prusty@siemens.com/T/#u
https://lists.isar-build.org/isar-users/20240705071856.904-1-cedric.hombourger@siemens.com/T/#u
https://lists.isar-build.org/isar-users/c426ad86-5cc5-42e6-8223-e7a7e5b3424en@googlegroups.com/T/#u
https://lists.isar-build.org/isar-users/20240416100746.5681-1-nicusor_huhulea@mentor.com/T/#u
https://lists.isar-build.org/isar-users/DB3PR10MB690820CF3DD2DCA71D822AF1E63B2@DB3PR10MB6908.EURPRD10.PROD.OUTLOOK.COM/T/#u
https://lists.isar-build.org/isar-users/DB3PR10MB690861E70EB246FEF845E253E64B2@DB3PR10MB6908.EURPRD10.PROD.OUTLOOK.COM/T/#u
https://lists.isar-build.org/isar-users/20240131144459.2386220-1-iskochilov@ilbers.de/T/#u

Changes since v9:
- Base on v6.
- Rebase on next.
- Move also kernel deployment from image recipe.

Changes since v8:
- Rebase on next.
- Document how this approach differs from OpenEmbedded.

Changes since v7:
- Fix SOB.

Changes since v6:
- Fix isoimage-isohybrid-isar WIC plugin.
- Split testcases.
- Update wording.

Changes since v4:
- Rebase on next.
- Squash latest patch to first one.
- Fix deploy paths in initrd tests.

Changes since v3:
- Rebase on next.
- Fix target deploy dir path for installer.

Changes since v2:
- Patch 1: Consider also kernel suffix.
- Patch 2: Use separate testcase for possible dtb clashes.
- Minor cosmetic fixes.

Changes since v1:
- Separate `dtb-files` recipe used instead of linux one
- Included older "[PATCH] Fix do_copy_boot_files error" into the
  series ( https://groups.google.com/g/isar-users/c/Va0Ue-ISYeA )

Anton Mikanovich (2):
  meta: Fix do_copy_boot_files error for different distros of same
    machine
  meta: Move kernel artifacts deployment from image recipe

 RECIPE-API-CHANGELOG.md                       | 27 ++++++++++
 .../installer-add-rootfs.bbclass              |  4 +-
 meta/classes-recipe/image.bbclass             | 39 +++++---------
 meta/conf/bitbake.conf                        |  2 +-
 .../kernel-deploy/kernel-deploy.bb            | 53 +++++++++++++++++++
 testsuite/citest.py                           |  7 +--
 6 files changed, 100 insertions(+), 32 deletions(-)
 create mode 100644 meta/recipes-kernel/kernel-deploy/kernel-deploy.bb

-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260429153829.2668049-1-wzh%40ilbers.de.

[PATCH v10 1/2] meta: Fix do_copy_boot_files error for different distros of same machine

[Zhihang Wei]

From: Anton Mikanovich <amikan@ilbers.de>

When building different distros for the same machine (e.g.,
phyboard-mira-bullseye and phyboard-mira-bookworm), files with the same
name (e.g, DTB files) might be deployed to the same location and this
causes a build error.

Use DISTRO- and KERNEL-dependent deploy directory.

Also fix paths in installer and testsuite.

Fixes test_dtb_deploy_distros testcase:

ERROR: mc:phyboard-mira-bookworm:isar-image-base-1.0-r0 do_copy_boot_files: The recipe isar-image-base is trying to install files into a shared area when those files already exist. Those files and their manifest location are:
NOTE: recipe isar-image-base-1.0-r0: task do_copy_boot_files: Failed
  build/tmp/deploy/images/phyboard-mira/imx6q-phytec-mira-rdk-nand.dtb
    (not matched to any task)
Please verify which recipe should provide the above files.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 RECIPE-API-CHANGELOG.md                           | 15 +++++++++++++++
 .../classes-recipe/installer-add-rootfs.bbclass   |  4 +++-
 meta/conf/bitbake.conf                            |  2 +-
 testsuite/citest.py                               |  7 ++++---
 4 files changed, 23 insertions(+), 5 deletions(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 659a03b0..5723c4c3 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -1067,3 +1067,18 @@ Changes in next
 The DEPLOY_DIR_IMAGE path and artifacts naming changes introduced in
 13cb77dd (included in Isar v1.0) have been reverted. Deployment paths
 and artifact names are the same as pre-v1.0.
+
+### Change DEPLOY_DIR_IMAGE path
+
+Change DEPLOY_DIR_IMAGE from ${DEPLOY_DIR}/images/${MACHINE} to
+${DEPLOY_DIR}/images/${MACHINE}-${DISTRO}-${KERNEL_NAME}. When building
+different distros with the same machine the following error occurs:
+
+do_copy_boot_files: The recipe isar-image-base is trying to install files
+into a shared area when those files already exists. It happens when some
+files have the same names (e.g., dtb files) for different distros.
+
+To prevent this new path is separated also by distro and kernel values.
+
+This change will influence on build artifacts location and should be taken
+into account by downstreams.
diff --git a/meta-isar/classes-recipe/installer-add-rootfs.bbclass b/meta-isar/classes-recipe/installer-add-rootfs.bbclass
index 111506df..0ecb5cc5 100644
--- a/meta-isar/classes-recipe/installer-add-rootfs.bbclass
+++ b/meta-isar/classes-recipe/installer-add-rootfs.bbclass
@@ -13,7 +13,9 @@ INSTALLER_TARGET_IMAGES ??= "${INSTALLER_TARGET_IMAGE}"
 INSTALLER_TARGET_MC ??= "installer-target"
 INSTALLER_TARGET_DISTRO ??= "${DISTRO}"
 INSTALLER_TARGET_MACHINE ??= "${MACHINE}"
-INSTALLER_TARGET_DEPLOY_DIR_IMAGE ??= "${DEPLOY_DIR}/images/${INSTALLER_TARGET_MACHINE}"
+INSTALLER_TARGET_KERNEL_NAME ??= "${KERNEL_NAME}"
+INSTALLER_TARGET_DEPLOY_DIR_POSTFIX ??= "${@ ('-'+d.getVar('INSTALLER_TARGET_KERNEL_NAME')) if d.getVar('INSTALLER_TARGET_KERNEL_NAME') else ''}"
+INSTALLER_TARGET_DEPLOY_DIR_IMAGE ??= "${DEPLOY_DIR}/images/${INSTALLER_TARGET_MACHINE}-${INSTALLER_TARGET_DISTRO}${INSTALLER_TARGET_DEPLOY_DIR_POSTFIX}"
 
 IMAGE_DATA_FILE ??= "${INSTALLER_TARGET_IMAGE}-${INSTALLER_TARGET_DISTRO}-${INSTALLER_TARGET_MACHINE}"
 IMAGE_DATA_POSTFIX ??= "wic.zst"
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 5c71078d..0089942b 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -57,7 +57,7 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}/${PV}-${PR}"
 GIT_DL_LINK_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}"
 DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"
 DEPLOY_DIR_SDKCHROOT = "${DEPLOY_DIR}/sdkchroot"
-DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MACHINE}"
+DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MACHINE}-${DISTRO}${@ ('-'+d.getVar('KERNEL_NAME')) if d.getVar('KERNEL_NAME') else ''}"
 DL_DIR ?= "${TOPDIR}/downloads"
 SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
 SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${DISTRO}-${DISTRO_ARCH}"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index ac45165b..e842b62e 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -498,8 +498,9 @@ class InitRdBaseTest(CIBaseTest):
         super().init()
         self.deploy_dir = os.path.join(self.build_dir, 'tmp', 'deploy')
 
-    def deploy_dir_image(self, machine):
-        return os.path.join(self.deploy_dir, 'images', machine)
+    def deploy_dir_image(self, mc, image):
+        multiconfig = f"{mc}:{image}"
+        return CIUtils.getVars('DEPLOY_DIR_IMAGE', target=multiconfig)
 
     def dracut_in_image(self, targets):
         machine = 'qemuamd64'
@@ -528,7 +529,7 @@ class InitRdBaseTest(CIBaseTest):
                                           bb_should_fail=False):
         mc = f'mc:{machine}-{distro.removeprefix("debian-")}'
         initrd_image = f'{initrd}-{distro}-{machine}-initrd.img'
-        initrd_path = os.path.join(self.deploy_dir_image(machine), initrd_image)
+        initrd_path = os.path.join(self.deploy_dir_image(mc, initrd), initrd_image)
 
         # cleansstate if the initrd image was already built/deployed to verify
         # that a new build does result in the image being deployed
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260429153829.2668049-2-wzh%40ilbers.de.

[PATCH v10 2/2] meta: Move kernel artifacts deployment from image recipe

[Zhihang Wei]

From: Anton Mikanovich <amikan@ilbers.de>

Task do_copy_boot_files deploys DTB files into the same location for
different images (e.g., isar-image-base and isar-image-ci). This causes
a build error.

do_copy_boot_files is called once for every image recipe while dtb
files belong to the kernel which is the same for both images. Performing
dtb deployment once for the same kernel solves the issue.

Introduce `kernel-deploy` recipe responsible for extracting the DTBs and
kernel from the linux-image package and deploying them. The kernel
package is now installed into sbuild chroot. This is achieved via the
respective dependency of `kernel-deploy` recipe.

Fixes test_dtb_deploy_images testcase:

ERROR: mc:phyboard-mira-bookworm:isar-image-base-1.0-r0 do_copy_boot_files: The recipe isar-image-base is trying to install files into a shared area when those files already exist. Those files and their manifest location are:
  build/tmp/deploy/images/phyboard-mira/imx6q-phytec-mira-rdk-nand.dtb
    (not matched to any task)
Please verify which recipe should provide the above files.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 RECIPE-API-CHANGELOG.md                       | 12 +++++
 meta/classes-recipe/image.bbclass             | 39 +++++---------
 .../kernel-deploy/kernel-deploy.bb            | 53 +++++++++++++++++++
 3 files changed, 77 insertions(+), 27 deletions(-)
 create mode 100644 meta/recipes-kernel/kernel-deploy/kernel-deploy.bb

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 5723c4c3..2cecd03b 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -1082,3 +1082,15 @@ To prevent this new path is separated also by distro and kernel values.
 
 This change will influence on build artifacts location and should be taken
 into account by downstreams.
+
+### Move kernel and DTBs deployment from image recipe to kernel related one
+
+To prevent parallel conflicting artifacts deployment from the same kernel
+used by different images, move do_copy_boot_files() task logic from the image
+recipe to the separate one, which is once-per-kernel. With this aproach the
+deployment of any kernel artifacts will be performed only once per kernel.
+
+To left previous kernel image path per-image symlinks are now used.
+
+This deployment is implemented as separated recipe to be compatible both with
+custom and distro kernel types.
diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass
index 26a4ec06..a86ee443 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -1,5 +1,7 @@
 # This software is a part of Isar.
-# Copyright (C) 2015-2017 ilbers GmbH
+# Copyright (C) 2015-2026 ilbers GmbH
+#
+# SPDX-License-Identifier: MIT
 
 # Make workdir and stamps machine-specific without changing common PN target
 WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/${PV}-${PR}"
@@ -9,7 +11,6 @@ STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/*-*"
 
 # Sstate also needs to be machine-specific
 SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${MACHINE}-${DISTRO}-${DISTRO_ARCH}"
-SSTATETASKS += "do_copy_boot_files"
 
 IMAGE_INSTALL ?= ""
 IMAGE_FSTYPES ?= "ext4"
@@ -381,37 +382,21 @@ INITRD_IMG = "${PP_DEPLOY}/${INITRD_DEPLOY_FILE}"
 # only one dtb file supported, pick the first
 DTB_IMG = "${PP_DEPLOY}/${@(d.getVar('DTB_FILES').split() or [''])[0]}"
 
-do_copy_boot_files[cleandirs] += "${DEPLOYDIR}"
-do_copy_boot_files[sstate-inputdirs] = "${DEPLOYDIR}"
-do_copy_boot_files[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
-do_copy_boot_files[network] = "${TASK_USE_SUDO}"
+python() {
+    if d.getVar('DTB_FILES'):
+        d.appendVarFlag("do_copy_boot_files", "depends", "kernel-deploy-${KERNEL_NAME}:do_deploy")
+}
+
+# Associate kernel with image by symlinks
+do_copy_boot_files[dirs] += "${DEPLOY_DIR_IMAGE}"
 do_copy_boot_files() {
-    kernel="$(realpath -q '${IMAGE_ROOTFS}'/vmlinu[xz])"
-    if [ ! -f "$kernel" ]; then
-        kernel="$(realpath -q '${IMAGE_ROOTFS}'/boot/vmlinu[xz])"
-    fi
+    kernel="$(realpath -q '${DEPLOY_DIR_IMAGE}/${KERNEL_NAME}'-vmlinu[xz]*)"
     if [ -f "$kernel" ]; then
-        sudo cat "$kernel" > "${DEPLOYDIR}/${KERNEL_IMAGE}"
+        ln -sfr "$kernel" "${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGE}"
     fi
-
-    for file in ${DTB_FILES}; do
-        dtb="$(find '${IMAGE_ROOTFS}/usr/lib' -type f \
-                    -iwholename '*linux-image-*/'${file} | head -1)"
-
-        if [ -z "$dtb" -o ! -e "$dtb" ]; then
-            die "${file} not found"
-        fi
-
-        cp -f "$dtb" "${DEPLOYDIR}/"
-    done
 }
 addtask copy_boot_files before do_rootfs_postprocess after do_rootfs_install
 
-python do_copy_boot_files_setscene () {
-    sstate_setscene(d)
-}
-addtask do_copy_boot_files_setscene
-
 python do_image_tools() {
     """Virtual task"""
     pass
diff --git a/meta/recipes-kernel/kernel-deploy/kernel-deploy.bb b/meta/recipes-kernel/kernel-deploy/kernel-deploy.bb
new file mode 100644
index 00000000..32521148
--- /dev/null
+++ b/meta/recipes-kernel/kernel-deploy/kernel-deploy.bb
@@ -0,0 +1,53 @@
+# This software is a part of Isar.
+# Copyright (C) ilbers GmbH, 2026
+#
+# SPDX-License-Identifier: MIT
+
+inherit dpkg-raw
+
+MAINTAINER = "isar-users <isar-users@googlegroups.com>"
+
+PN:append = "-${KERNEL_NAME}"
+
+KERNEL_IMAGE_PKG ??= "${@ ("linux-image-" + d.getVar("KERNEL_NAME")) if d.getVar("KERNEL_NAME") else ""}"
+
+DEPENDS = "${KERNEL_IMAGE_PKG}"
+DEBIAN_BUILD_DEPENDS = "${KERNEL_IMAGE_PKG}"
+
+DPKG_ARCH = "${PACKAGE_ARCH}"
+
+do_prepare_build:append() {
+    dir=debian/${PN}/boot
+    cat <<EOF >> ${S}/debian/rules
+	mkdir -p ${dir}
+	realpath -q /boot/vmlinu[xz]
+	kernel="\$\$(realpath -q /vmlinu[xz])" && \
+	if [ ! -f "\$\${kernel}" ]; then kernel="\$\$(realpath -q /boot/vmlinu[xz])"; fi && \
+	if [ -f "\$\${kernel}" ]; then cp "\$\${kernel}" "${dir}/${KERNEL_NAME}-\$\$(basename \$\${kernel})"; fi
+EOF
+
+    for dtb in ${DTB_FILES}; do
+        dir=debian/${PN}/usr/lib/${PN}/$(dirname ${dtb})
+        cat <<EOF >> ${S}/debian/rules
+	mkdir -p ${dir}
+	find /usr/lib/linux-image* -path "*${dtb}" -print -exec cp {} ${dir} \;
+EOF
+    done
+}
+
+DTB_PACKAGE ??= "${PN}_${CHANGELOG_V}_${DISTRO_ARCH}.deb"
+
+do_deploy[dirs] = "${DEPLOY_DIR_IMAGE}"
+do_deploy[cleandirs] = "${WORKDIR}/deploy"
+do_deploy() {
+    dpkg --fsys-tarfile ${WORKDIR}/${DTB_PACKAGE} | \
+    tar --wildcards --extract --directory ${WORKDIR}/deploy ./boot ./usr/lib/${PN}
+    find ${WORKDIR}/deploy/boot -path "*vmlinu*" -print \
+            -exec cp {} ${DEPLOY_DIR_IMAGE}/ \;
+    for dtb in ${DTB_FILES}; do
+        mkdir -p ${DEPLOY_DIR_IMAGE}/$(dirname ${dtb})
+        find ${WORKDIR}/deploy/usr/lib/${PN} -path "*${dtb}" -print \
+            -exec cp {} ${DEPLOY_DIR_IMAGE}/${dtb} \;
+    done
+}
+addtask deploy before do_deploy_deb after do_dpkg_build
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260429153829.2668049-3-wzh%40ilbers.de.