Re: [PATCH 2/2] handle DPKG_ARCH=all case for transitive deps

[Andreas Naumann <anaumann@emlix.com>]

Hi Felix, all

Am 16.09.25 um 09:16 schrieb MOESSBAUER, Felix:
> On Mon, 2025-09-15 at 19:49 +0200, Andreas Naumann wrote:
>> Hi Felix,
>>
>> Am 20.08.25 um 14:41 schrieb 'Felix Moessbauer' via isar-users:
>>> Arch=all packages might build depend on other arch=all packages, hence we
>>> need to correctly model the dependency chain. Otherwise the transitive
>>> dependencies are built for the distro arch instead of the native arch.
>>>
>>> We implement this by dispatching the non-native variant of DPKG_ARCH=all
>>> packages to the -native variant by adding a dependency. We further
>>> replace the non-native do_deploy_dep task with a noop to preserve the
>>> dependency chain.
>>>
>>> Co-developed-by: Adriaan Schmidt<adriaan.schmidt@siemens.com>
>>> Signed-off-by: Felix Moessbauer<felix.moessbauer@siemens.com>
>>> ---
>>>    meta/classes/multiarch.bbclass | 18 +++++++++++++++++-
>>>    1 file changed, 17 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
>>> index babdfbd4..c2bba21f 100644
>>> --- a/meta/classes/multiarch.bbclass
>>> +++ b/meta/classes/multiarch.bbclass
>>> @@ -29,7 +29,11 @@ python() {
>>>                d.appendVar('BBCLASSEXTEND', ' compat')
>>>    
>>>        # build native separately only when it differs from the target variant
>>> -    if not archIsAll and archDiffers:
>>> +    # We must not short-circuit for DPKG_ARCH=all packages, as they might
>>> +    # have transitive dependencies which need to be built for -native.
>>> +    # This special handling for DPKG_ARCH=all packages is left to the
>>> +    # multiarch_virtclass_handler
>>> +    if archDiffers:
>>>            d.appendVar('BBCLASSEXTEND', ' native')
>>>        else:
>>>            extend_provides(pn, 'native', d)
>>> @@ -86,6 +90,8 @@ python multiarch_virtclass_handler() {
>>>                d.setVar(var, ' '.join(multiarch_var))
>>>    
>>>        pn = e.data.getVar('PN')
>>> +    archDiffers = d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH')
>>> +    archIsAll = d.getVar('DPKG_ARCH') == 'all'
>>>        if pn.endswith('-compat'):
>>>            e.data.setVar('BPN', pn[:-len('-compat')])
>>>            e.data.appendVar('OVERRIDES', ':class-compat')
>>> @@ -96,6 +102,16 @@ python multiarch_virtclass_handler() {
>>>            e.data.appendVar('OVERRIDES', ':class-native')
>>>            fixup_pn_in_vars(e.data)
>>>            fixup_depends('-native', e.data)
>>> +    elif archIsAll and archDiffers:
>>> +        # Arch=all packages might build depend on other arch=all packages,
>>> +        # hence we need to correctly model the dependency chain.
>>> +        # We implement this by dispatching the non-native variant to the -native
>>> +        # variant by adding a dependency. We further replace the non-native
>>> +        # do_deploy_dep task with a noop to preserve the dependency chain.
>>> +        e.data.setVar('do_deploy_deb', '')
>>> +        bb.build.deltask('deploy_deb', e.data)
>>> +        bb.build.addtask('deploy_deb', 'do_build', '', e.data)
>>> +        e.data.setVarFlag('do_deploy_deb', 'depends', f'{pn}-native:do_deploy_deb')
>> I have now done some more testing and unfortunately find that this
>> causes failures when building the -native variant of some dpkg-raw
>> packages we have.
>>
> Hi, do you have an example for such a package? Just to be clear: There
> is no "native" variant of a dpkg-raw package. The dpkg-raw packages are
> always build on the host architecture (in bitbake terms) and of Debian
> arch=all.
>
>> We use them for certain config files and they are
>> intended for the target.
>>
> Do these configs contain data that is not the same on all
> architectures? If so, DPKG_ARCH="all" is wrong.

After some debugging, it seems to be a combination of your patch and 
"2ca3a7e dpkg-source: Build source package only once". There is no 
problem building our dpkg-raw packages explicitly in the non-native 
form, e.g. in the host environment for the target arch.
But through your patch, the package also runs as -native, which then 
triggers a problem with 2ca3a7e.

... I only now discover that the other part of your patchset "dpkg-raw: 
add files to source package" fixes this. Ok.

However, because of this I went digging on why you introduced the patch. 
And I wonder why sbom-chroot doesn't just depend on 
python3-debsbom-native directly? (Which still breaks as long as 2ca3a7e 
sets the dependency on a function of the non-native package).


>> I dont know why exactly they fail and I'm sure
>> this could be fixed, but actually there is no need to build those
>> packages in the native environment.
>>
> There is need to do so, as otherwise we emulate the build process which
> is super slow.
Yes you're right. I meant there is no need to build them *for* the 
native architecture.
>> So I'm somewhat surprised to see
>> that and could image that this causes confusion for others that dont
>> know about the "all" intricacies as well.
> Probably, but in the end we are building a debian system and by that
> the users should be aware of the debian architecture specifiers.
Yes sure, they were new to me, but no problem, they are not that 
complicated.

But when I deliberately build -native packages (which I assume to be a 
valid use case according to the doc), and somehow the buildsystem goes 
back and forth running functions from the native and the base package, 
it's difficult to follow why that would be necessary. It doesnt make for 
ease of debugging when working on new packages either.

For the use cases I have in sight right now it would be fine if a 
package build breaks because it cannot be done cross-architecture. I'd 
hope to being able to fix this by either improving the package or making 
sure to call it -native only. But if that's not what the expectation is 
for Debian Crossbuilding I'm eager to learn.

best regards,
Andreas

>> Another observation that I see is that dpkg-prebuilt "any" packages,
>> which are probably in the dependency chain of an "all" package, now are
>> always also built as -native, even though they are needed for the target
>> only.
> Same here, even if bitbake tells you that they are build as "-native",
> there is no difference in the output, as the binary is simply copied.
>
> Best regards,
> Felix
>
>> best,
>> Andras
>>
>>>    }
>>>    addhandler multiarch_virtclass_handler
>>>    multiarch_virtclass_handler[eventmask] = "bb.event.RecipePreFinalise"
>> -- 
>> Andreas Naumann
>>
>> emlix GmbH
>> Headquarters: Berliner Str. 12, 37073 Goettingen, Germany
>> Phone +49 (0)551 30664-0, e-mailinfo@emlix.com
>> District Court of Goettingen, Registry Number HR B 3160
>> Managing Directors: Heike Jordan, Dr. Uwe Kracke
>> VAT ID No. DE 205 198 055
>> Office Berlin: Panoramastr. 1, 10178 Berlin, Germany
>> Office Bonn: Bachstr. 6, 53115 Bonn, Germany
>> http://www.emlix.com

-- 
Andreas Naumann

emlix GmbH
Headquarters: Berliner Str. 12, 37073 Goettingen, Germany
Phone +49 (0)551 30664-0, e-mailinfo@emlix.com
District Court of Goettingen, Registry Number HR B 3160
Managing Directors: Heike Jordan, Dr. Uwe Kracke
VAT ID No. DE 205 198 055
Office Berlin: Panoramastr. 1, 10178 Berlin, Germany
Office Bonn: Bachstr. 6, 53115 Bonn, Germany
http://www.emlix.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/45ecefef-bbc3-4506-b9b3-c945d0d77e65%40emlix.com.