Re: [RFC PATCH] native.bbclass

["Moessbauer, Felix" <felix.moessbauer@siemens.com>]

On Wed, 2023-01-18 at 07:09 +0100, Jan Kiszka wrote:
> On 17.01.23 19:03, Adriaan Schmidt wrote:
> > Hi all,
> > 
> > as mentioned by Jan, I have another approach of building
> > native/host/compat
> > arch packages, inspired by OE's native class.
> > 
> > The way it works is:
> > - A recipe can inherit native.bbclass (or, if we like we could even
> > add it
> >   to all recipes via the dpkg class).
> 
> I think we should not start spreading these inherits across random
> recipes when there are no costs and risks (as I believe) in adding
> that
> to dpkg directly, likely even dpkg-base.
> 
> > - For recipes that inherit native.bbclass, bitbake generates a new
> > bitbake
> >   target with suffix `-native`. Recipes can DEPEND on those `-
> > native` packages.
> >   My use case that made me develop this is goreleaser, a build tool
> >   for which I have a bitbake recipe. If I want to crosscompile, I
> > need it
> >   for the host architecture. By DEPENDing on `goreleaser-native`, I
> > can
> >   then add `goreleaser:native` to the build dependencies in my
> > control file
> >   (the `:native` makes dpkg-buildpackage choose the right
> > architecture).
> > - When building the recipe, the only change is that we set
> > PACKAGE_ARCH=HOST_ARCH.
> >   In addition we have the override `class-native`, which is set
> > when building
> >   the native variant of the recipe and lets us make conditional
> > changes to
> >   what is built.
> 
> So, if I decide to pre-populate an image with libfoo not only for the
> target arch but maybe also others (thinking of compat here), I would
> have to do
> 
> IMAGE_INSTALL += "meta-package"
> 
> and meta-package would have
> 
> DEBIAN_DEPENDS += "libfoo, libfoo:${COMPAT_DISTRO_ARCH}"
> DEPENDS += "libfoo libfoo-compat"
> 
> Unless we find a way to teach IMAGE_INSTALL resolving "...:<some-
> arch>"
> into the right -native/-compat DEPENDS.

This is trivial to resolve with python functions.
Currently the IMAGE_INSTALL is added both to DEPENDS, as well as to
apt. For apt, we keep it. For DEPENDS, we replace "<foo>:arch" with
"<foo>-arch", where arch might have to be mapped to "-native / -
compat", depending on how the naming scheme will be implemented.

Felix

> 
> Obviously simpler is to have some 32-bit-only legacy app that states
> 
> PACKAGE_ARCH = "${COMPAT_DISTRO_ARCH}"
> DEBIAN_DEPENDS = "libfoo:${COMPAT_DISTRO_ARCH}"
> DEPENDS = "libfoo-compat"
> 
> libfoo itself would no longer have to worry about becoming compat as
> well.
> 
> > 
> > Some caveats/notes:
> > - Both "normal" target arch and `-native` builds generate ARCH=all
> > packages
> >   and source packages. So there is potential duplication. In theory
> > those
> >   packages should be identical, but we currently don't have any way
> > of knowing.
> >   This is in part due to using shared isar-apt to transfer packages
> > between
> >   jobs, where the last job to push a package always wins.
> >   My earlier RFC on removing (that use of) isar-apt and explicitly
> >   transferring dependent packages might help here. In that case we
> > have full
> >   control of what packages are provided, and can detect such
> > duplicates.
> 
> We have a second duplication issue, and that is around the debian
> source
> package. Again, all targets should normally generate a bit-identical
> version of that so that only identical versions will be found when
> collecting them for potential redistribution. I was assuming Isar
> would
> already build a deb-src repo for isar-apt, but it this feature seems
> to
> be missing. Once we add it, the topic above will become relevant for
> it
> as well.
> 
> > - I'm changing the default overrides to include PACKAGE_ARCH
> > instead of
> >   DISTRO_ARCH. Maybe this is how it always should have been?
> > - This is currently only for "native", but I think we can simply
> > duplicate
> >   the pattern to also support compat arch packages.
> 
> This should be done in the same run, we already need it as urgently
> as
> -native.
> 
> > - Using PN in recipes can become tricky. For example, I sometimes
> > see the
> >   pattern of setting SRC_URI="git://github.com/${PN}/${PN}.git".
> >   This would break when building the native variant, as this will
> > then
> >   have a different PN. Personally, I don't think this is a big
> > problem.
> >   Using ${PN} in SRC_URI may not be a good idea anyways, and if we
> > really
> >   need a variable, there is BPN, which does not have the -native
> > suffix.
> 
> I agree.
> 
> > - We currently don't have a use case for this in meta-isar. But
> > maybe we
> >   can come up with a good example.
> 
> Typical use cases are self-built build tools for other self-built
> packages. Or cleaning up the kbuild issue that also Stefan Koch was
> working on. In fact, kbuild is an existing in-tree -native use case
> that
> we hacked so far to generate only -native packages. When converting
> that, we could add a test case that uses the to-be-generated
> kbuild:target package on the generated image, e.g. to install a dkms
> kernel module live.
> 
> Jan
> 
> > 
> > I'd like to try if this approach works for the other use cases we
> > have.
> > 
> > Adriaan
> > 
> > PS: If you'd like some more examples of the BBCLASSEXTEND feature,
> > have
> >     a look at the SDK generation, which has been refactored in that
> > way.
> > 
> > ---
> >  meta/classes/native.bbclass | 13 +++++++++++++
> >  meta/conf/bitbake.conf      |  5 +++--
> >  2 files changed, 16 insertions(+), 2 deletions(-)
> >  create mode 100644 meta/classes/native.bbclass
> > 
> > diff --git a/meta/classes/native.bbclass
> > b/meta/classes/native.bbclass
> > new file mode 100644
> > index 00000000..896c8a06
> > --- /dev/null
> > +++ b/meta/classes/native.bbclass
> > @@ -0,0 +1,13 @@
> > +BBCLASSEXTEND = "native"
> > +BPN = "${PN}"
> > +
> > +python native_virtclass_handler() {
> > +    pn = e.data.getVar('PN')
> > +    if pn.endswith('-native'):
> > +        e.data.setVar('BPN', pn[:-len('-native')])
> > +        e.data.appendVar('OVERRIDES', ':class-native')
> > +}
> > +addhandler native_virtclass_handler
> > +native_virtclass_handler[eventmask] = "bb.event.RecipePreFinalise"
> > +
> > +PACKAGE_ARCH_class-native = "${HOST_ARCH}"
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > index 98412e02..ef962fc7 100644
> > --- a/meta/conf/bitbake.conf
> > +++ b/meta/conf/bitbake.conf
> > @@ -67,8 +67,8 @@ KERNEL_FILE_mipsel ?= "vmlinux"
> >  KERNEL_FILE_riscv64 ?= "vmlinux"
> >  KERNEL_FILE_arm64 ?= "vmlinux"
> >  
> > -OVERRIDES =
> > "${DISTRO_ARCH}:${COMPAT_OVERRIDE}:${MACHINE}:${DISTRO}:${BASE_DIST
> > RO_CODENAME}:forcevariable"
> > -FILESOVERRIDES = "${DISTRO_ARCH}:${MACHINE}"
> > +OVERRIDES =
> > "${PACKAGE_ARCH}:${COMPAT_OVERRIDE}:${MACHINE}:${DISTRO}:${BASE_DIS
> > TRO_CODENAME}:forcevariable"
> > +FILESOVERRIDES = "${PACKAGE_ARCH}:${MACHINE}"
> >  COMPAT_OVERRIDE = "${@'compat-arch' if
> > d.getVar('ISAR_ENABLE_COMPAT_ARCH') == '1' else ''}"
> >  
> >  # Setting default QEMU_ARCH variables for different DISTRO_ARCH:
> > @@ -81,6 +81,7 @@ QEMU_ARCH_riscv64 = "riscv64"
> >  
> >  # Codename of the repository created by the caching class
> >  DEBDISTRONAME ?= "isar"
> > +NATIVELSBSTRING ?= "isarnative"
> >  
> >  # Strings used in sstate signature files
> >  TARGET_VENDOR = ""
> 
> -- 
> Siemens AG, Technology
> Competence Center Embedded Linux
>