public inbox for isar-users@googlegroups.com
 help / color / mirror / Atom feed
From: Zhihang Wei <wzh@ilbers.de>
To: "Moessbauer, Felix" <felix.moessbauer@siemens.com>,
	"Kowalsky, Clara" <clara.kowalsky@siemens.com>,
	"isar-users@googlegroups.com" <isar-users@googlegroups.com>
Cc: "Gylstorff, Quirin" <quirin.gylstorff@siemens.com>
Subject: Re: [PATCH v2 1/2] image: remove unrequested intermediate image types
Date: Wed, 18 Nov 2026 13:09:39 +0100	[thread overview]
Message-ID: <3c563201-bf08-43e2-94f7-1021ad32d0b5@ilbers.de> (raw)
In-Reply-To: <353700093e871f677bb45feb1a6655dcc026f08c.camel@siemens.com>


On 11/18/26 12:55, Moessbauer, Felix wrote:
> On Wed, 2026-11-18 at 09:24 +0000, 'Moessbauer, Felix' via isar-users
> wrote:
>> On Wed, 2026-11-11 at 17:07 +0100, Zhihang Wei wrote:
>>> Applied to next, thanks.
>>>
>>> Zhihang
>>>
>>> On 11/4/26 08:25, 'Clara Kowalsky' via isar-users wrote:
>>>> Images built only as dependencies currently remain in the deploy
>>>> directory even when they are not listed in IMAGE_FSTYPES.
>>>> Fix this by keeping only explicitly requested image outputs and removing
>>>> dependency-only images and conversion intermediates. Users who need a
>>>> dependency image retained should include it in IMAGE_FSTYPES.
>>>>
>>>> Since dependencies are now also removed, run cleanup after all image
>>>> tasks finish rather than at the end of each producing task. This keeps
>>>> dependency images available until their consumers have completed.
>>>> Expanded file paths are recorded, so cleanup also handles type-specific
>>>> filenames and deploy directories.
>>>> Unittests are added to test dependency cleanup and retention of
>>>> requested outputs.
>> Hi,
>>
>> while the intention of the commit is fine, the current implementation
>> has some serious drawbacks, calling for a revert:
>>
>> - it does not remove additional image artifacts (like wic.bmap,
>> wic.spdx.json, ...) and there is no way to register these
>> - it breaks partial rebuilds of a derived image type (e.g. a wic ->
>> squashfs dependency cannot rebuild just the wic type as the squashfs
>> was removed in the meantime).
>>
>> Both aspects cannot be easily fixes implementation wise, as we don't
>> track who deploys what. The proper way to do it would be to deploy via
>> the sstate cache and let the cache handle the deletion of the
>> artifacts, but this requires a major re-design.
>>
>> For the -rc2 release, I recommend to just revert this commit. The
>> "image: remove unrequested intermediate image types" can be kept.
> Oops... I was referring to "imagetypes_vm: add support for standalone
> vmdk images", which can be kept.

Ack, please send the revert.

Zhihang

> Felix
>
>> Best regards,
>> Felix
>>
>>>> Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com>
>>>> ---
>>>>    meta/classes-recipe/image.bbclass             | 24 ++---
>>>>    .../unittests/test_image_type_cleanup.py      | 92 +++++++++++++++++++
>>>>    2 files changed, 104 insertions(+), 12 deletions(-)
>>>>    create mode 100644 testsuite/unittests/test_image_type_cleanup.py
>>>>
>>>> diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass
>>>> index ce19ae1c..229a37fe 100644
>>>> --- a/meta/classes-recipe/image.bbclass
>>>> +++ b/meta/classes-recipe/image.bbclass
>>>> @@ -187,6 +187,8 @@ SUDO_CHROOT = "imager_run -d ${PP_ROOTFS} -u root --"
>>>>    # hook up IMAGE_CMD_*
>>>>    python() {
>>>>        image_types = (d.getVar('IMAGE_FSTYPES') or '').split()
>>>> +    requested_types = set(image_types)
>>>> +    intermediate_files = set()
>>>>        conversions = set(d.getVar('IMAGE_CONVERSIONS').split())
>>>>        chroot_mode = d.getVar('ISAR_CHROOT_MODE')
>>>>    
>>>> @@ -285,8 +287,10 @@ python() {
>>>>    
>>>>            # add conversions
>>>>            conversion_depends = set()
>>>> -        rm_images = set()
>>>>            def create_conversions(t):
>>>> +            localdata.setVar('type', t)
>>>> +            if t not in requested_types:
>>>> +                intermediate_files.add(localdata.getVar('IMAGE_FILE_HOST'))
>>>>                for c in sorted(conversions):
>>>>                    if t.endswith('.' + c):
>>>>                        t = t[:-len(c) - 1]
>>>> @@ -301,20 +305,9 @@ python() {
>>>>                        for dep in (localdata.getVar('CONVERSION_DEPS:' + c) or '').split():
>>>>                            conversion_install.add(dep)
>>>>                            local_conversion_install.add(dep)
>>>> -                    # remove temporary image files
>>>> -                    if t not in image_types:
>>>> -                        rm_images.add(localdata.expand('${IMAGE_FILE_HOST}'))
>>>> -
>>>>            for t in basetypes[bt]:
>>>>                create_conversions(t)
>>>>    
>>>> -        if bt not in image_types:
>>>> -            localdata.setVar('type', t)
>>>> -            rm_images.add(localdata.expand('${IMAGE_FILE_HOST}'))
>>>> -
>>>> -        for image in rm_images:
>>>> -            cmds.append('\trm ' + image)
>>>> -
>>>>            # image type dependencies
>>>>            after = 'do_image_tools'
>>>>            for dep in typedeps[bt]:
>>>> @@ -340,6 +333,7 @@ python() {
>>>>            d.appendVar('INSTALL_image_%s' % bt_clean, ' ' + ' '.join(sorted(local_imager_install | local_conversion_install)))
>>>>            d.appendVarFlag(task, 'vardeps', ' INSTALL_image_%s' % bt_clean)
>>>>    
>>>> +    d.setVar('IMAGE_INTERMEDIATE_FILES', ' '.join(sorted(intermediate_files)))
>>>>        d.appendVar('IMAGER_INSTALL', ' ' + ' '.join(sorted(imager_install | conversion_install)))
>>>>        d.appendVar('IMAGER_BOM', ' ' + ' '.join(sorted(imager_bom)))
>>>>        d.appendVar('IMAGER_BUILD_DEPS', ' ' + ' '.join(sorted(imager_build_deps)))
>>>> @@ -429,6 +423,12 @@ python do_image() {
>>>>        """Virtual task"""
>>>>        pass
>>>>    }
>>>> +python cleanup_image_intermediates() {
>>>> +    for image in (d.getVar('IMAGE_INTERMEDIATE_FILES') or '').split():
>>>> +        bb.utils.remove(image)
>>>> +}
>>>> +do_image[postfuncs] += "cleanup_image_intermediates"
>>>> +do_image[vardeps] += "IMAGE_INTERMEDIATE_FILES"
>>>>    addtask image before do_build after do_image_tools
>>>>    
>>>>    python do_deploy() {
>>>> diff --git a/testsuite/unittests/test_image_type_cleanup.py b/testsuite/unittests/test_image_type_cleanup.py
>>>> new file mode 100644
>>>> index 00000000..1e2ad5b9
>>>> --- /dev/null
>>>> +++ b/testsuite/unittests/test_image_type_cleanup.py
>>>> @@ -0,0 +1,92 @@
>>>> +# This software is a part of Isar.
>>>> +# Copyright (C) Siemens AG, 2026
>>>> +#
>>>> +# SPDX-License-Identifier: MIT
>>>> +
>>>> +import sys
>>>> +import tempfile
>>>> +import unittest
>>>> +from pathlib import Path
>>>> +
>>>> +root = Path(__file__).resolve().parents[2]
>>>> +sys.path.extend([
>>>> +    str(root / 'bitbake/lib'),
>>>> +    str(root / 'meta/lib'),
>>>> +])
>>>> +
>>>> +import bb.data
>>>> +from bb.build import exec_func
>>>> +from bb.parse import handle
>>>> +
>>>> +
>>>> +data = bb.data.init()
>>>> +data.setVar('BBPATH', f'{root}/meta:{root}/bitbake')
>>>> +data.setVar('__bbclasstype', 'recipe')
>>>> +data.setVar('IMGCLASSES', '')
>>>> +parsed = handle(str(root / 'meta/classes-recipe/image.bbclass'), data)
>>>> +data.setVar('IMAGE_TYPEDEP:ova', 'vmdk')
>>>> +data.setVar('IMAGE_TYPEDEP:vmdk', 'wic')
>>>> +data.setVar('IMAGE_CMD:vmdk', ':')
>>>> +namespace = {'bb': bb}
>>>> +exec(parsed['get_base_type'], namespace)
>>>> +image_tasks = next(
>>>> +    function for function in parsed.getVar('__BBANONFUNCS', False)
>>>> +    if 'def collect_image_type(' in parsed.getVar(function, False)
>>>> +)
>>>> +exec(parsed[image_tasks], namespace)
>>>> +
>>>> +
>>>> +class TestImageTypeCleanup(unittest.TestCase):
>>>> +    def generate_image_tasks(self, image_types, deploy_dir='/deploy'):
>>>> +        d = bb.data.createCopy(data)
>>>> +        d.setVar('IMAGE_FSTYPES', image_types)
>>>> +        d.setVar('DEPLOY_DIR_IMAGE', str(deploy_dir))
>>>> +        d.setVar('IMAGE_FULLNAME', 'image')
>>>> +        namespace[image_tasks](d)
>>>> +        return d
>>>> +
>>>> +    def assert_intermediates(self, image_types, expected):
>>>> +        d = self.generate_image_tasks(image_types)
>>>> +        self.assertEqual(
>>>> +            set(d.getVar('IMAGE_INTERMEDIATE_FILES').split()),
>>>> +            {'/deploy/image.' + image_type for image_type in expected},
>>>> +        )
>>>> +
>>>> +    def test_only_unrequested_types_are_intermediates(self):
>>>> +        self.assert_intermediates('ova.zst', {'ova', 'vmdk', 'wic'})
>>>> +        self.assert_intermediates('vmdk ova.zst', {'ova', 'wic'})
>>>> +        self.assert_intermediates('ova ova.zst vmdk wic', set())
>>>> +
>>>> +    def test_unrequested_conversion_stages_are_cleaned(self):
>>>> +        self.assert_intermediates('tar.gz.xz', {'tar', 'tar.gz'})
>>>> +        self.assert_intermediates('tar.gz tar.gz.xz', {'tar'})
>>>> +
>>>> +    def test_cleanup_uses_image_type_filename_overrides(self):
>>>> +        d = bb.data.createCopy(data)
>>>> +        d.setVar('IMAGE_FSTYPES', 'tar.gz')
>>>> +        d.setVar('IMAGE_FULLNAME:tar', 'custom-image')
>>>> +        d.setVar('DEPLOY_DIR_IMAGE:tar', '/custom-deploy')
>>>> +        namespace[image_tasks](d)
>>>> +        self.assertEqual(
>>>> +            d.getVar('IMAGE_INTERMEDIATE_FILES'),
>>>> +            '/custom-deploy/custom-image.tar',
>>>> +        )
>>>> +
>>>> +    def test_cleanup_removes_only_intermediate_files(self):
>>>> +        with tempfile.TemporaryDirectory() as deploy_dir:
>>>> +            deploy_dir = Path(deploy_dir)
>>>> +            for image_type in ('ova', 'wic', 'vmdk', 'ova.zst'):
>>>> +                (deploy_dir / ('image.' + image_type)).touch()
>>>> +
>>>> +            d = self.generate_image_tasks('vmdk ova.zst', deploy_dir)
>>>> +            d.setVar('T', str(deploy_dir / 'temp'))
>>>> +            exec_func('cleanup_image_intermediates', d)
>>>> +
>>>> +            self.assertEqual(
>>>> +                {path.name for path in deploy_dir.glob('image.*')},
>>>> +                {'image.vmdk', 'image.ova.zst'},
>>>> +            )
>>>> +
>>>> +
>>>> +if __name__ == '__main__':
>>>> +    unittest.main()
>>> -- 
>>> You received this message because you are subscribed to the Google Groups "isar-users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
>>> To view this discussion visit https://groups.google.com/d/msgid/isar-users/2f98ffb2-506b-4cd2-a020-75f3d5ee0bef%40ilbers.de.
>> -- 
>> You received this message because you are subscribed to the Google Groups "isar-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
>> To view this discussion visit https://groups.google.com/d/msgid/isar-users/c0f3b81896422d60c9df4b31207fab4111dfb0fd.camel%40siemens.com.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/3c563201-bf08-43e2-94f7-1021ad32d0b5%40ilbers.de.

      reply	other threads:[~2026-11-18 12:11 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-04  7:25 'Clara Kowalsky' via isar-users
2026-11-04  7:26 ` [PATCH v2 2/2] imagetypes_vm: add support for standalone vmdk images 'Clara Kowalsky' via isar-users
2026-11-11 16:07 ` [PATCH v2 1/2] image: remove unrequested intermediate image types Zhihang Wei
2026-11-18  9:24   ` 'Moessbauer, Felix' via isar-users
2026-11-18 11:55     ` 'Moessbauer, Felix' via isar-users
2026-11-18 12:09       ` Zhihang Wei [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=3c563201-bf08-43e2-94f7-1021ad32d0b5@ilbers.de \
    --to=wzh@ilbers.de \
    --cc=clara.kowalsky@siemens.com \
    --cc=felix.moessbauer@siemens.com \
    --cc=isar-users@googlegroups.com \
    --cc=quirin.gylstorff@siemens.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox