Re: [PATCH 2/2] handle DPKG_ARCH=all case for transitive deps

["'MOESSBAUER, Felix' via isar-users" <isar-users@googlegroups.com>]

On Fri, 2025-09-19 at 09:15 +0200, Andreas Naumann wrote:
> Hi Felix, all
> 
> Am 16.09.25 um 09:16 schrieb MOESSBAUER, Felix:
> > On Mon, 2025-09-15 at 19:49 +0200, Andreas Naumann wrote:
> > > Hi Felix,
> > > 
> > > Am 20.08.25 um 14:41 schrieb 'Felix Moessbauer' via isar-users:
> > > > Arch=all packages might build depend on other arch=all packages, hence we
> > > > need to correctly model the dependency chain. Otherwise the transitive
> > > > dependencies are built for the distro arch instead of the native arch.
> > > > 
> > > > We implement this by dispatching the non-native variant of DPKG_ARCH=all
> > > > packages to the -native variant by adding a dependency. We further
> > > > replace the non-native do_deploy_dep task with a noop to preserve the
> > > > dependency chain.
> > > > 
> > > > Co-developed-by: Adriaan Schmidt<adriaan.schmidt@siemens.com>
> > > > Signed-off-by: Felix Moessbauer<felix.moessbauer@siemens.com>
> > > > ---
> > > >    meta/classes/multiarch.bbclass | 18 +++++++++++++++++-
> > > >    1 file changed, 17 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
> > > > index babdfbd4..c2bba21f 100644
> > > > --- a/meta/classes/multiarch.bbclass
> > > > +++ b/meta/classes/multiarch.bbclass
> > > > @@ -29,7 +29,11 @@ python() {
> > > >                d.appendVar('BBCLASSEXTEND', ' compat')
> > > >    
> > > >        # build native separately only when it differs from the target variant
> > > > -    if not archIsAll and archDiffers:
> > > > +    # We must not short-circuit for DPKG_ARCH=all packages, as they might
> > > > +    # have transitive dependencies which need to be built for -native.
> > > > +    # This special handling for DPKG_ARCH=all packages is left to the
> > > > +    # multiarch_virtclass_handler
> > > > +    if archDiffers:
> > > >            d.appendVar('BBCLASSEXTEND', ' native')
> > > >        else:
> > > >            extend_provides(pn, 'native', d)
> > > > @@ -86,6 +90,8 @@ python multiarch_virtclass_handler() {
> > > >                d.setVar(var, ' '.join(multiarch_var))
> > > >    
> > > >        pn = e.data.getVar('PN')
> > > > +    archDiffers = d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH')
> > > > +    archIsAll = d.getVar('DPKG_ARCH') == 'all'
> > > >        if pn.endswith('-compat'):
> > > >            e.data.setVar('BPN', pn[:-len('-compat')])
> > > >            e.data.appendVar('OVERRIDES', ':class-compat')
> > > > @@ -96,6 +102,16 @@ python multiarch_virtclass_handler() {
> > > >            e.data.appendVar('OVERRIDES', ':class-native')
> > > >            fixup_pn_in_vars(e.data)
> > > >            fixup_depends('-native', e.data)
> > > > +    elif archIsAll and archDiffers:
> > > > +        # Arch=all packages might build depend on other arch=all packages,
> > > > +        # hence we need to correctly model the dependency chain.
> > > > +        # We implement this by dispatching the non-native variant to the -native
> > > > +        # variant by adding a dependency. We further replace the non-native
> > > > +        # do_deploy_dep task with a noop to preserve the dependency chain.
> > > > +        e.data.setVar('do_deploy_deb', '')
> > > > +        bb.build.deltask('deploy_deb', e.data)
> > > > +        bb.build.addtask('deploy_deb', 'do_build', '', e.data)
> > > > +        e.data.setVarFlag('do_deploy_deb', 'depends', f'{pn}-native:do_deploy_deb')
> > > I have now done some more testing and unfortunately find that this
> > > causes failures when building the -native variant of some dpkg-raw
> > > packages we have.
> > > 
> > Hi, do you have an example for such a package? Just to be clear: There
> > is no "native" variant of a dpkg-raw package. The dpkg-raw packages are
> > always build on the host architecture (in bitbake terms) and of Debian
> > arch=all.
> > 
> > > We use them for certain config files and they are
> > > intended for the target.
> > > 
> > Do these configs contain data that is not the same on all
> > architectures? If so, DPKG_ARCH="all" is wrong.
> 
> After some debugging, it seems to be a combination of your patch and 
> "2ca3a7e dpkg-source: Build source package only once". There is no 
> problem building our dpkg-raw packages explicitly in the non-native 
> form, e.g. in the host environment for the target arch.
> But through your patch, the package also runs as -native, which then 
> triggers a problem with 2ca3a7e.
> 
> ... I only now discover that the other part of your patchset "dpkg-raw: 
> add files to source package" fixes this. Ok.

Ok, so in general this series is fine.

> 
> However, because of this I went digging on why you introduced the patch. 
> And I wonder why sbom-chroot doesn't just depend on 
> python3-debsbom-native directly? (Which still breaks as long as 2ca3a7e 
> sets the dependency on a function of the non-native package).

This is a more general thing: The chroot is architecture specific. By
that, the architecture of the packages (DEPENDS) is selected based on
that without the need for having -native. The internal logic then takes
care of building the packages in the correct variant (e.g. -compat,
-native or without any suffix, which refers to the target arch).

The 2ca3a7e ("dpkg-source: Build source package only once") just
ensures that we only create a single source package, no matter for
which (and how many) architectures we build it.

> 
> 
> > > I dont know why exactly they fail and I'm sure
> > > this could be fixed, but actually there is no need to build those
> > > packages in the native environment.
> > > 
> > There is need to do so, as otherwise we emulate the build process which
> > is super slow.
> Yes you're right. I meant there is no need to build them *for* the 
> native architecture.
> > > So I'm somewhat surprised to see
> > > that and could image that this causes confusion for others that dont
> > > know about the "all" intricacies as well.
> > Probably, but in the end we are building a debian system and by that
> > the users should be aware of the debian architecture specifiers.
> Yes sure, they were new to me, but no problem, they are not that 
> complicated.
> 
> But when I deliberately build -native packages (which I assume to be a 
> valid use case according to the doc), and somehow the buildsystem goes 
> back and forth running functions from the native and the base package, 
> it's difficult to follow why that would be necessary. It doesnt make for 
> ease of debugging when working on new packages either.
> 
> For the use cases I have in sight right now it would be fine if a 
> package build breaks because it cannot be done cross-architecture. I'd 
> hope to being able to fix this by either improving the package or making 
> sure to call it -native only. But if that's not what the expectation is 
> for Debian Crossbuilding I'm eager to learn.

If there are no further issues with this patchset, I would like to move
forward and see it merged. It is a pre-condition for the SBOM patches
for non target!=host arch.

Best regards,
Felix

> 
> best regards,
> Andreas
> 
> > > Another observation that I see is that dpkg-prebuilt "any" packages,
> > > which are probably in the dependency chain of an "all" package, now are
> > > always also built as -native, even though they are needed for the target
> > > only.
> > Same here, even if bitbake tells you that they are build as "-native",
> > there is no difference in the output, as the binary is simply copied.
> > 
> > Best regards,
> > Felix
> > 
> > > best,
> > > Andras
> > > 
> > > >    }
> > > >    addhandler multiarch_virtclass_handler
> > > >    multiarch_virtclass_handler[eventmask] = "bb.event.RecipePreFinalise"
> > > -- 
> > > Andreas Naumann
> > > 
> > > emlix GmbH
> > > Headquarters: Berliner Str. 12, 37073 Goettingen, Germany
> > > Phone +49 (0)551 30664-0, e-mailinfo@emlix.com
> > > District Court of Goettingen, Registry Number HR B 3160
> > > Managing Directors: Heike Jordan, Dr. Uwe Kracke
> > > VAT ID No. DE 205 198 055
> > > Office Berlin: Panoramastr. 1, 10178 Berlin, Germany
> > > Office Bonn: Bachstr. 6, 53115 Bonn, Germany
> > > http://www.emlix.com
> 
> -- 
> Andreas Naumann
> 
> emlix GmbH
> Headquarters: Berliner Str. 12, 37073 Goettingen, Germany
> Phone +49 (0)551 30664-0, e-mailinfo@emlix.com
> District Court of Goettingen, Registry Number HR B 3160
> Managing Directors: Heike Jordan, Dr. Uwe Kracke
> VAT ID No. DE 205 198 055
> Office Berlin: Panoramastr. 1, 10178 Berlin, Germany
> Office Bonn: Bachstr. 6, 53115 Bonn, Germany
> http://www.emlix.com

-- 
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/56507dd43d6b0f0e309563ebbe2d48327353cdc5.camel%40siemens.com.