Re: [PATCH 2/2] handle DPKG_ARCH=all case for transitive deps

["'Andreas Naumann' via isar-users" <isar-users@googlegroups.com>]

Hi Felix, all

Am 06.10.25 um 16:05 schrieb MOESSBAUER, Felix:
> On Fri, 2025-09-19 at 09:15 +0200, Andreas Naumann wrote:
>> Hi Felix, all
>>
>> Am 16.09.25 um 09:16 schrieb MOESSBAUER, Felix:
>>> On Mon, 2025-09-15 at 19:49 +0200, Andreas Naumann wrote:
>>>> Hi Felix,
>>>>
>>>> Am 20.08.25 um 14:41 schrieb 'Felix Moessbauer' via isar-users:
>>>>> Arch=all packages might build depend on other arch=all packages, hence we
>>>>> need to correctly model the dependency chain. Otherwise the transitive
>>>>> dependencies are built for the distro arch instead of the native arch.
>>>>>
>>>>> We implement this by dispatching the non-native variant of DPKG_ARCH=all
>>>>> packages to the -native variant by adding a dependency. We further
>>>>> replace the non-native do_deploy_dep task with a noop to preserve the
>>>>> dependency chain.
>>>>>
>>>>> Co-developed-by: Adriaan Schmidt<adriaan.schmidt@siemens.com>
>>>>> Signed-off-by: Felix Moessbauer<felix.moessbauer@siemens.com>
>>>>> ---
>>>>>     meta/classes/multiarch.bbclass | 18 +++++++++++++++++-
>>>>>     1 file changed, 17 insertions(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
>>>>> index babdfbd4..c2bba21f 100644
>>>>> --- a/meta/classes/multiarch.bbclass
>>>>> +++ b/meta/classes/multiarch.bbclass
>>>>> @@ -29,7 +29,11 @@ python() {
>>>>>                 d.appendVar('BBCLASSEXTEND', ' compat')
>>>>>     
>>>>>         # build native separately only when it differs from the target variant
>>>>> -    if not archIsAll and archDiffers:
>>>>> +    # We must not short-circuit for DPKG_ARCH=all packages, as they might
>>>>> +    # have transitive dependencies which need to be built for -native.
>>>>> +    # This special handling for DPKG_ARCH=all packages is left to the
>>>>> +    # multiarch_virtclass_handler
>>>>> +    if archDiffers:
>>>>>             d.appendVar('BBCLASSEXTEND', ' native')
>>>>>         else:
>>>>>             extend_provides(pn, 'native', d)
>>>>> @@ -86,6 +90,8 @@ python multiarch_virtclass_handler() {
>>>>>                 d.setVar(var, ' '.join(multiarch_var))
>>>>>     
>>>>>         pn = e.data.getVar('PN')
>>>>> +    archDiffers = d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH')
>>>>> +    archIsAll = d.getVar('DPKG_ARCH') == 'all'
>>>>>         if pn.endswith('-compat'):
>>>>>             e.data.setVar('BPN', pn[:-len('-compat')])
>>>>>             e.data.appendVar('OVERRIDES', ':class-compat')
>>>>> @@ -96,6 +102,16 @@ python multiarch_virtclass_handler() {
>>>>>             e.data.appendVar('OVERRIDES', ':class-native')
>>>>>             fixup_pn_in_vars(e.data)
>>>>>             fixup_depends('-native', e.data)
>>>>> +    elif archIsAll and archDiffers:
>>>>> +        # Arch=all packages might build depend on other arch=all packages,
>>>>> +        # hence we need to correctly model the dependency chain.
>>>>> +        # We implement this by dispatching the non-native variant to the -native
>>>>> +        # variant by adding a dependency. We further replace the non-native
>>>>> +        # do_deploy_dep task with a noop to preserve the dependency chain.
>>>>> +        e.data.setVar('do_deploy_deb', '')
>>>>> +        bb.build.deltask('deploy_deb', e.data)
>>>>> +        bb.build.addtask('deploy_deb', 'do_build', '', e.data)
>>>>> +        e.data.setVarFlag('do_deploy_deb', 'depends', f'{pn}-native:do_deploy_deb')
>>>> I have now done some more testing and unfortunately find that this
>>>> causes failures when building the -native variant of some dpkg-raw
>>>> packages we have.
>>>>
>>> Hi, do you have an example for such a package? Just to be clear: There
>>> is no "native" variant of a dpkg-raw package. The dpkg-raw packages are
>>> always build on the host architecture (in bitbake terms) and of Debian
>>> arch=all.
>>>
>>>> We use them for certain config files and they are
>>>> intended for the target.
>>>>
>>> Do these configs contain data that is not the same on all
>>> architectures? If so, DPKG_ARCH="all" is wrong.
>> After some debugging, it seems to be a combination of your patch and
>> "2ca3a7e dpkg-source: Build source package only once". There is no
>> problem building our dpkg-raw packages explicitly in the non-native
>> form, e.g. in the host environment for the target arch.
>> But through your patch, the package also runs as -native, which then
>> triggers a problem with 2ca3a7e.
>>
>> ... I only now discover that the other part of your patchset "dpkg-raw:
>> add files to source package" fixes this. Ok.
> Ok, so in general this series is fine.

ACK, to get the sbom series going.

However, I have proposed and would like to invite you to comment on my [RFC 0/5] Improving multiarch support for arch-incompatible packages. I think it provides a straightforward, generic solution that works for "all" and non-all packages alike.


>> However, because of this I went digging on why you introduced the patch.
>> And I wonder why sbom-chroot doesn't just depend on
>> python3-debsbom-native directly? (Which still breaks as long as 2ca3a7e
>> sets the dependency on a function of the non-native package).
> This is a more general thing: The chroot is architecture specific. By
> that, the architecture of the packages (DEPENDS) is selected based on
> that without the need for having -native. The internal logic then takes
> care of building the packages in the correct variant (e.g. -compat,
> -native or without any suffix, which refers to the target arch).

I'm not sure I understand what you mean. If python3-debsbom was an "any" 
package, how would the logic guess that it's needed for the host?

best regards,
Andreas

>
> The 2ca3a7e ("dpkg-source: Build source package only once") just
> ensures that we only create a single source package, no matter for
> which (and how many) architectures we build it.
>
>>
>>>> I dont know why exactly they fail and I'm sure
>>>> this could be fixed, but actually there is no need to build those
>>>> packages in the native environment.
>>>>
>>> There is need to do so, as otherwise we emulate the build process which
>>> is super slow.
>> Yes you're right. I meant there is no need to build them *for* the
>> native architecture.
>>>> So I'm somewhat surprised to see
>>>> that and could image that this causes confusion for others that dont
>>>> know about the "all" intricacies as well.
>>> Probably, but in the end we are building a debian system and by that
>>> the users should be aware of the debian architecture specifiers.
>> Yes sure, they were new to me, but no problem, they are not that
>> complicated.
>>
>> But when I deliberately build -native packages (which I assume to be a
>> valid use case according to the doc), and somehow the buildsystem goes
>> back and forth running functions from the native and the base package,
>> it's difficult to follow why that would be necessary. It doesnt make for
>> ease of debugging when working on new packages either.
>>
>> For the use cases I have in sight right now it would be fine if a
>> package build breaks because it cannot be done cross-architecture. I'd
>> hope to being able to fix this by either improving the package or making
>> sure to call it -native only. But if that's not what the expectation is
>> for Debian Crossbuilding I'm eager to learn.
> If there are no further issues with this patchset, I would like to move
> forward and see it merged. It is a pre-condition for the SBOM patches
> for non target!=host arch.
>
> Best regards,
> Felix
>
>> best regards,
>> Andreas
>>
>>>> Another observation that I see is that dpkg-prebuilt "any" packages,
>>>> which are probably in the dependency chain of an "all" package, now are
>>>> always also built as -native, even though they are needed for the target
>>>> only.
>>> Same here, even if bitbake tells you that they are build as "-native",
>>> there is no difference in the output, as the binary is simply copied.
>>>
>>> Best regards,
>>> Felix
>>>
>>>> best,
>>>> Andras
>>>>
>>>>>     }
>>>>>     addhandler multiarch_virtclass_handler
>>>>>     multiarch_virtclass_handler[eventmask] = "bb.event.RecipePreFinalise"
>>>> -- 
>>>> Andreas Naumann
>>>>
>>>> emlix GmbH
>>>> Headquarters: Berliner Str. 12, 37073 Goettingen, Germany
>>>> Phone +49 (0)551 30664-0, e-mailinfo@emlix.com
>>>> District Court of Goettingen, Registry Number HR B 3160
>>>> Managing Directors: Heike Jordan, Dr. Uwe Kracke
>>>> VAT ID No. DE 205 198 055
>>>> Office Berlin: Panoramastr. 1, 10178 Berlin, Germany
>>>> Office Bonn: Bachstr. 6, 53115 Bonn, Germany
>>>> http://www.emlix.com
>> -- 
>> Andreas Naumann
>>
>> emlix GmbH
>> Headquarters: Berliner Str. 12, 37073 Goettingen, Germany
>> Phone +49 (0)551 30664-0, e-mailinfo@emlix.com
>> District Court of Goettingen, Registry Number HR B 3160
>> Managing Directors: Heike Jordan, Dr. Uwe Kracke
>> VAT ID No. DE 205 198 055
>> Office Berlin: Panoramastr. 1, 10178 Berlin, Germany
>> Office Bonn: Bachstr. 6, 53115 Bonn, Germany
>> http://www.emlix.com

-- 
Andreas Naumann

emlix GmbH
Headquarters: Berliner Str. 12, 37073 Goettingen, Germany
Phone +49 (0)551 30664-0, e-mail info@emlix.com
District Court of Goettingen, Registry Number HR B 3160
Managing Directors: Heike Jordan, Dr. Uwe Kracke
VAT ID No. DE 205 198 055
Office Berlin: Panoramastr. 1, 10178 Berlin, Germany
Office Bonn: Bachstr. 6, 53115 Bonn, Germany
http://www.emlix.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/6c272cfc-ca61-4ec9-8913-7497cf43001e%40emlix.com.