Re: [PATCH v7 3/3] meta: Deploy image build artifacts into distro- and image-specific subdirs

[Zhihang Wei <wzh@ilbers.de>]

On 3/3/26 14:35, Zhihang Wei wrote:
>
>
> On 3/3/26 13:50, Jan Kiszka wrote:
>> On 03.03.26 11:22, Zhihang Wei wrote:
>>>
>>> On 3/2/26 10:45, Jan Kiszka wrote:
>>>> On 26.02.26 17:26, Zhihang Wei wrote:
>>>>> From: Ilia Skochilov <iskochilov@ilbers.de>
>>>>>
>>>>> When building different distros or images for the same machine (e.g.
>>>>> phyboard-mira-bullseye and phyboard-mira-bookworm, or isar-image-ci
>>>>> and isar-image-base), artifacts with identical names (e.g. DTB files)
>>>>> may be deployed to the same location, resulting file conflicts and
>>>>> build failures.
>>>>>
>>>>> Deploy image build artifacts into a subdirectory named
>>>>> ${DISTRO}-${IMAGE_PN} to avoid such collisions.
>>>>>
>>>>> Since the directory now encodes the distro and image name, remove the
>>>>> ${DISTRO} and ${IMAGE_PN} prefixes from artifact filenames to
>>>>> keep the names short and clear.
>>>>>
>>>>> Fixes test_dtb_deploy_distros testcase:
>>>>>
>>>>> ERROR: mc:phyboard-mira-bookworm:isar-image-base-1.0-r0
>>>>> do_copy_boot_files: The recipe isar-image-base is trying to install
>>>>> files into a shared area when those files already exist. Those files
>>>>> and their manifest location are:
>>>>> NOTE: recipe isar-image-base-1.0-r0: task do_copy_boot_files: Failed
>>>>> build/tmp/deploy/images/phyboard-mira/imx6q-phytec-mira-rdk-nand.dtb
>>>>>       (not matched to any task)
>>>>> Please verify which recipe should provide the above files.
>>>>>
>>>>> Signed-off-by: Ilia Skochilov <iskochilov@ilbers.de>
>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
>>>>> Signed-off-by: Badrikesh Prusty <badrikesh.prusty@siemens.com>
>>>>> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
>>>> Were really all those people involved in developing and signing off 
>>>> this
>>>> change?
>>> I'll update the list for this specific version of the patch.
>>>
>>>>> Signed-off-by: Zhihang Wei <wzh@ilbers.de>
>>>>> ---
>>>>>    RECIPE-API-CHANGELOG.md                       | 63 
>>>>> +++++++++++++++++++
>>>>>    .../installer-add-rootfs.bbclass              |  9 +--
>>>>>    meta/classes-recipe/image.bbclass             | 11 ++--
>>>>>    .../imagetypes_container.bbclass              |  2 +-
>>>>>    meta/conf/bitbake.conf                        |  3 +-
>>>>>    .../plugins/source/isoimage-isohybrid-isar.py |  2 +-
>>>>>    testsuite/cibase.py                           |  2 +-
>>>>>    testsuite/citest.py                           |  7 ++-
>>>>>    testsuite/start_vm.py                         |  2 +-
>>>>>    9 files changed, 84 insertions(+), 17 deletions(-)
>>>>>
>>>>> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
>>>>> index 0bad8a44..bc7abb37 100644
>>>>> --- a/RECIPE-API-CHANGELOG.md
>>>>> +++ b/RECIPE-API-CHANGELOG.md
>>>>> @@ -962,3 +962,66 @@ INSTALLER_UNATTENDED_ABORT_ENABLE = "1"
>>>>>    # Optional: set countdown timeout in seconds (default 5)
>>>>>    INSTALLER_UNATTENDED_ABORT_TIMEOUT = "5"
>>>>>    ```
>>>>> +
>>>>> +### Change DEPLOY_DIR_IMAGE path and artifacts naming
>>>>> +
>>>>> +Change DEPLOY_DIR_IMAGE from ${DEPLOY_DIR}/images/${MACHINE} to
>>>>> +${DEPLOY_DIR}/images/${MACHINE}/${DISTRO}-${IMAGE_PN}.
>>>>> +
>>>>> +When building different distros with the same machine the following
>>>>> +error occurs:
>>>>> +
>>>>> +do_copy_boot_files: The recipe isar-image-base is trying to install
>>>>> +files into a shared area when those files already exists. It happens
>>>>> +when some files have the same names (e.g., dtb files) for different
>>>>> +distros.
>>>>> +
>>>>> +To prevent such collisions, image artifacts are now deployed into a
>>>>> +distro- and image-specific subdirectory.
>>>>> +
>>>>> +Additionally, artifact filenames have been shortened by removing the
>>>>> +${DISTRO} and ${IMAGE_PN} prefix, since this information is now
>>>>> +encoded in the directory path.
>>>>> +
>>>>> +Example 1: Build isar-image-base (phyboard-mira, debian-bookworm)
>>>>> +Under "build/tmp/deploy/images/":
>>>>> +Before:
>>>>> +phyboard-mira/imx6q-phytec-mira-rdk-nand.dtb
>>>>> +phyboard-mira/isar-image-base-debian-bookworm-phyboard-mira-initrd.img 
>>>>>
>>>>> +phyboard-mira/isar-image-base-debian-bookworm-phyboard-mira-vmlinuz
>>>>> +phyboard-mira/isar-image-base-debian-bookworm-phyboard-mira.dpkg_status 
>>>>>
>>>>> +phyboard-mira/isar-image-base-debian-bookworm-phyboard-mira.ubi
>>>>> +
>>>>> +After:
>>>>> +phyboard-mira/debian-bookworm-isar-image-base/imx6q-phytec-mira-rdk-
>>>>> nand.dtb
>>>>> +phyboard-mira/debian-bookworm-isar-image-base/initrd.img
>>>>> +phyboard-mira/debian-bookworm-isar-image-base/vmlinuz
>>>>> +phyboard-mira/debian-bookworm-isar-image-base/phyboard-mira.dpkg_status 
>>>>>
>>>>> +phyboard-mira/debian-bookworm-isar-image-base/phyboard-mira.ubi
>>>>> +
>>>>> +Example 2: Build isar-image-ci (qemuamd64, debian-bookworm)
>>>>> +Under "build/tmp/deploy/images/":
>>>>> +Before:
>>>>> +qemuamd64/isar-image-ci-debian-bookworm-qemuamd64-initrd.img
>>>>> +qemuamd64/isar-image-ci-debian-bookworm-qemuamd64-vmlinuz
>>>>> +qemuamd64/isar-image-ci-debian-bookworm-qemuamd64.dpkg_status
>>>>> +qemuamd64/isar-image-ci-debian-bookworm-qemuamd64.manifest
>>>>> +qemuamd64/isar-image-ci-debian-bookworm-qemuamd64.wic
>>>>> +qemuamd64/isar-image-ci-debian-bookworm-qemuamd64.wic.bmap
>>>>> +qemuamd64/isar-image-ci-debian-bookworm-qemuamd64.wic.manifest
>>>>> +
>>>>> +After:
>>>>> +qemuamd64/debian-bookworm-isar-image-ci/initrd.img
>>>>> +qemuamd64/debian-bookworm-isar-image-ci/vmlinuz
>>>>> +qemuamd64/debian-bookworm-isar-image-ci/qemuamd64.dpkg_status
>>>>> +qemuamd64/debian-bookworm-isar-image-ci/qemuamd64.manifest
>>>>> +qemuamd64/debian-bookworm-isar-image-ci/qemuamd64.wic
>>>>> +qemuamd64/debian-bookworm-isar-image-ci/qemuamd64.wic.bmap
>>>>> +qemuamd64/debian-bookworm-isar-image-ci/qemuamd64.wic.manifest
>>>>> +
>>>>> +Artifacts that do not belong to a full image (e.g. isar-image-base,
>>>>> +isar-image-ci) remain unchanged. For example, a customized initramfs
>>>>> +built independently is not affected.
>>>>> +
>>>>> +This change affects the location and naming of build artifacts 
>>>>> and must
>>>>> +be taken into account by downstream users.
>>>> The chosen pattern looks reasonable to me. I'm just wondering what OE
>>>> does when generating multiple images per target, and why it did not 
>>>> use
>>>> something similar. Any ideas?
>>>>
>>>> Jan
>>> I built mc:arm:core-image-minimal and mc:arm:core-image-base in a
>>> single command and enforced DTB deployment. These are the artifacts
>>> generated by OE under "build/tmp/deploy/images/qemuarm":
>>>
>>> core-image-base-qemuarm-20260302155417.qemuboot.conf
>>> core-image-base-qemuarm-20260302155417.rootfs.ext4
>>> core-image-base-qemuarm-20260302155417.rootfs.manifest
>>> core-image-base-qemuarm-20260302155417.rootfs.tar.bz2
>>> core-image-base-qemuarm-20260302155417.testdata.json
>>> core-image-base-qemuarm.ext4 -> core-image-base-
>>> qemuarm-20260302155417.rootfs.ext4
>>> core-image-base-qemuarm.manifest -> core-image-base-
>>> qemuarm-20260302155417.rootfs.manifest
>>> core-image-base-qemuarm.qemuboot.conf -> core-image-base-
>>> qemuarm-20260302155417.qemuboot.conf
>>> core-image-base-qemuarm.tar.bz2 -> core-image-base-
>>> qemuarm-20260302155417.rootfs.tar.bz2
>>> core-image-base-qemuarm.testdata.json -> core-image-base-
>>> qemuarm-20260302155417.testdata.json
>>> core-image-minimal-qemuarm-20260302155417.qemuboot.conf
>>> core-image-minimal-qemuarm-20260302155417.rootfs.ext4
>>> core-image-minimal-qemuarm-20260302155417.rootfs.manifest
>>> core-image-minimal-qemuarm-20260302155417.rootfs.tar.bz2
>>> core-image-minimal-qemuarm-20260302155417.testdata.json
>>> core-image-minimal-qemuarm.ext4 -> core-image-minimal-
>>> qemuarm-20260302155417.rootfs.ext4
>>> core-image-minimal-qemuarm.manifest -> core-image-minimal-
>>> qemuarm-20260302155417.rootfs.manifest
>>> core-image-minimal-qemuarm.qemuboot.conf -> core-image-minimal-
>>> qemuarm-20260302155417.qemuboot.conf
>>> core-image-minimal-qemuarm.tar.bz2 -> core-image-minimal-
>>> qemuarm-20260302155417.rootfs.tar.bz2
>>> core-image-minimal-qemuarm.testdata.json -> core-image-minimal-
>>> qemuarm-20260302155417.testdata.json
>>> modules--5.15.199+git0+78eca082b6_0ea8d4a7d2-r0-qemuarm-20260302155417.tgz 
>>>
>>> modules-qemuarm.tgz -> modules--5.15.199+git0+78eca082b6_0ea8d4a7d2-r0-
>>> qemuarm-20260302155417.tgz
>>> versatile-pb--5.15.199+git0+78eca082b6_0ea8d4a7d2-r0-
>>> qemuarm-20260302155417.dtb
>>> versatile-pb.dtb -> versatile-pb--5.15.199+git0+78eca082b6_0ea8d4a7d2-
>>> r0-qemuarm-20260302155417.dtb
>>> versatile-pb-qemuarm.dtb -> versatile-
>>> pb--5.15.199+git0+78eca082b6_0ea8d4a7d2-r0-qemuarm-20260302155417.dtb
>>> zImage -> zImage--5.15.199+git0+78eca082b6_0ea8d4a7d2-r0-
>>> qemuarm-20260302155417.bin
>>> zImage--5.15.199+git0+78eca082b6_0ea8d4a7d2-r0-qemuarm-20260302155417.bin 
>>>
>>> zImage-qemuarm.bin -> zImage--5.15.199+git0+78eca082b6_0ea8d4a7d2-r0-
>>> qemuarm-20260302155417.bin
>>>
>>> For the same distro but different images, the kernel and DTB are
>>> identical, and they were deployed only once. OE names the artifacts
>>> with a timestamp and then creates symlinks pointing to them using
>>> simpler names.
>> So, we are going do deviate at least from the pattern
>> <image-name>.wic/ext4/etc. by using <image-name>/<machine>.wic/ext4.
>>
>> Can you identify images that also deviate in kernel (configuration) and,
>> thus, deployed images? Does OE solve the duplication of non-identical
>> images then automatically?
> According to [1], OE can deploy multiple non-identical images or
> kernels in the same dir, as each artifact includes a timestamp in its
> filename. The symlinks point to the most recent built.
>
> [1] https://docs.yoctoproject.org/overview-manual/concepts.html#images
>
> Zhihang
>
My earlier explanation regarding the "timestamp" was not accurate. The
timestamp reflects when the BitBake build was started. In a multiconfig 
build
where two configs are built in the same command, the timestamp is identical
and therefore cannot be used to distinguish between the two configs.

OE separates multiconfig builds by using different TMPDIRs, for example by
adding the following to each multiconfig .conf file:
   TMPDIR .= "-${BB_CURRENT_MC}"

This results in different build directories such as "tmp-qemuarm64" and
"tmp-qemuarm64kerneldebug". Build artifacts with the same names but 
different
contents from two configs do not collide with each other.

If we configure two multiconfig .conf files to use the same TMPDIR, 
conflicts
can occur and OE does not handle that. Artifacts with the same name 
overwrite
each other, and the build results become incorrect.

In this way, the logic of the Isar build tmp directory differs from OE. 
Do we
need to document these differences in the commit message?

Zhihang

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/8173c451-26ba-45d1-872d-03313b0e2a7a%40ilbers.de.