Re: [PATCH v4 2/3] linux-custom: Provide host and target specific kernel kbuild packages

["Koch, Stefan" <stefan-koch@siemens.com>]

On Wed, 2023-10-04 at 18:15 +0200, Jan Kiszka wrote:
> On 04.10.23 17:34, Koch, Stefan (DI PA DCP R&D 3) wrote:
> > On Wed, 2023-09-06 at 06:38 +0000, Schmidt, Adriaan (T CED SES-DE)
> > wrote:
> > > Koch, Stefan (DI PA DCP R&D 3) <stefan-koch@siemens.com>, Monday,
> > > August 21, 2023 3:45 PM:
> > > > When using a cross build this patch does introduce optionally
> > > > host and target specific kernel kbuild packages that
> > > > ship the "scripts" and "tools" binaries.
> > > >
> > > > By default at cross builds a kbuild-cross package will be
> > > > generated that provides the host specific kbuild package.
> > > >
> > > > The "-compat" and "-native" multiarch bitbake targets are
> > > > useable
> > > > to run
> > > > additional target or host specific builds for kbuild scripts
> > > > and
> > > > tools.
> > > >
> > > > Using the "-compat" bitbake target enables the build of
> > > > a target specific kbuild package at cross builds.
> > > >
> > > > Using the "-native" bitbake target enables the build of
> > > > a host specific kbuild package at cross builds.
> > > >
> > > > This solves this from doc/custom_kernel.inc:
> > > > - The kernel headers package has not supported both native
> > > >   and cross compilation of kernel modules when itself was cross
> > > > built
> > > > - Future roadmap: Generate kernel headers package for both host
> > > >   and target when using a cross build
> > > >
> > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
> > > > ---
> > > >  .../linux/files/debian/control.tmpl           | 16 +++++-
> > > >  .../linux/files/debian/isar/build.tmpl        | 12 ++++-
> > > >  .../linux/files/debian/isar/common.tmpl       |  8 +++
> > > >  .../linux/files/debian/isar/install.tmpl      | 26 ++++++----
> > > >  .../linux/files/debian/rules.tmpl             |  2 +-
> > > >  meta/recipes-kernel/linux/linux-custom.inc    | 49
> > > > ++++++++++++++++---
> > > >  6 files changed, 94 insertions(+), 19 deletions(-)
> > > >
> > > > diff --git a/meta/recipes-
> > > > kernel/linux/files/debian/control.tmpl
> > > > b/meta/recipes-kernel/linux/files/debian/control.tmpl
> > > > index b19ca2c9..c012048b 100644
> > > > --- a/meta/recipes-kernel/linux/files/debian/control.tmpl
> > > > +++ b/meta/recipes-kernel/linux/files/debian/control.tmpl
> > > > @@ -6,6 +6,7 @@ Build-Depends: bc, kmod, cpio,
> > > > ${KBUILD_DEPENDS}
> > > >  Homepage:
> > > > http://www.kernel.org/
> > > >
> > > >  Package: linux-image-${KERNEL_NAME_PROVIDED}
> > > > +Build-Profiles: <!targetbuild !hostbuild>
> > > >  Architecture: any
> > > >  Depends: ${KERNEL_DEBIAN_DEPENDS}
> > > >  Description: ${KERNEL_NAME_PROVIDED} Linux kernel, version
> > > > @KR@
> > > > @@ -13,6 +14,7 @@ Description: ${KERNEL_NAME_PROVIDED} Linux
> > > > kernel, version
> > > > @KR@
> > > >   files, version: @KR@.
> > > >
> > > >  Package: linux-headers-${KERNEL_NAME_PROVIDED}
> > > > +Build-Profiles: <!targetbuild !hostbuild>
> > > >  Architecture: any
> > > >  Depends: ${KERNEL_HEADERS_DEBIAN_DEPENDS}, ${perl:Depends},
> > > > ${shlib:Depends}
> > > >  Description: ${KERNEL_NAME_PROVIDED} Linux kernel headers for
> > > > @KR@
> > > > @@ -21,7 +23,7 @@ Description: ${KERNEL_NAME_PROVIDED} Linux
> > > > kernel
> > > > headers
> > > > for @KR@
> > > >   This is useful for people who need to build external modules
> > > >
> > > >  Package: linux-libc-dev
> > > > -Build-Profiles: <!nolibcdev>
> > > > +Build-Profiles: <!nolibcdev !targetbuild !hostbuild>
> > > >  Section: devel
> > > >  Provides: linux-kernel-headers
> > > >  Architecture: any
> > > > @@ -30,6 +32,7 @@ Description: Linux support headers for
> > > > userspace
> > > > development
> > > >   are used by the installed headers for GNU glibc and other
> > > > system
> > > > libraries.
> > > >
> > > >  Package: linux-image-${KERNEL_NAME_PROVIDED}-dbg
> > > > +Build-Profiles: <!targetbuild !hostbuild>
> > > >  Section: debug
> > > >  Architecture: any
> > > >  Description: Linux kernel debugging symbols for @KR@
> > > > @@ -37,8 +40,19 @@ Description: Linux kernel debugging symbols
> > > > for
> > > > @KR@
> > > >   all the necessary debug symbols for the kernel and its
> > > > modules.
> > > >
> > > >  Package: linux-kbuild-${KERNEL_NAME_PROVIDED}
> > > > +Build-Profiles: <targetbuild> <hostbuild> <defaultkbuild>
> > > >  Architecture: any
> > > >  Depends: ${perl:Depends}, ${shlib:Depends}
> > > >  Description: ${KERNEL_NAME_PROVIDED} Linux kbuild scripts and
> > > > tools for @KR@
> > > >   This package provides kernel kbuild scripts and tools for
> > > > @KR@
> > > >   This is useful for people who need to build external modules
> > > > +
> > > > +Package: linux-kbuild-${KERNEL_NAME_PROVIDED}-cross
> > > > +Build-Profiles: <!targetbuild !hostbuild crosskbuild>
> > > > +Architecture: any
> > > > +Depends: ${KERNEL_HEADERS_DEBIAN_DEPENDS}, ${perl:Depends},
> > > > ${shlib:Depends}
> > > > +Description: ${KERNEL_NAME_PROVIDED} Linux kbuild scripts and
> > > > tools for @KR@
> > > > + This package provides kernel kbuild scripts and tools
> > > > + as ${HOST_ARCH} cross binaries for @KR@
> > > > + This is useful for those who need to cross build
> > > > + external modules using ISAR's sbuild-chroot-host
> > > > diff --git a/meta/recipes-
> > > > kernel/linux/files/debian/isar/build.tmpl
> > > > b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
> > > > index 906dc580..7f293528 100644
> > > > --- a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
> > > > +++ b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
> > > > @@ -22,7 +22,17 @@ do_build() {
> > > >      sed -i "s/@KR@/${KR}/g" ${S}/debian/control
> > > > ${S}/debian/linux-
> > > > image-
> > > > ${KERNEL_NAME_PROVIDED}.*
> > > >
> > > >      # Build the Linux kernel
> > > > -    ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE}
> > > > KCFLAGS="${KCFLAGS}"
> > > > KAFLAGS="${KAFLAGS}"
> > > > +    if echo "${DEB_BUILD_PROFILES}" | grep -q -e "targetbuild"
> > > > -e
> > > > "hostbuild"; then # Build kernel scripts and tools
> > > > +        ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE}
> > > > KCFLAGS="${KCFLAGS}"
> > > > KAFLAGS="${KAFLAGS}" scripts
> > > > +        if grep -q -E
> > > > "CONFIG_STACK_VALIDATION=y|CONFIG_HAVE_OBJTOOL=y"
> > > > ${KERNEL_BUILD_DIR}/.config && [ -d "tools/objtool" ]; then
> > > > +            ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE}
> > > > KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" tools/objtool || true
> > > > +        fi
> > > > +        if grep -q "CONFIG_MODULES=y"
> > > > ${KERNEL_BUILD_DIR}/.config;
> > > > then
> > > > +            ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE}
> > > > KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" modules_prepare
> > > > +        fi
> > > > +    else # Build the Linux kernel
> > > > +        ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE}
> > > > KCFLAGS="${KCFLAGS}"
> > > > KAFLAGS="${KAFLAGS}"
> > > > +    fi
> > > >
> > > >      # Stop tracing
> > > >      set +x
> > > > diff --git a/meta/recipes-
> > > > kernel/linux/files/debian/isar/common.tmpl
> > > > b/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
> > > > index f4c0519f..36e01691 100644
> > > > --- a/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
> > > > +++ b/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
> > > > @@ -11,6 +11,14 @@ KERNEL_PKG_KERN_HEADERS=linux-headers-
> > > > ${KERNEL_NAME_PROVIDED}
> > > >  KERNEL_PKG_LIBC_HEADERS=linux-libc-dev
> > > >  KERNEL_PKG_KERN_KBUILD=linux-kbuild-${KERNEL_NAME_PROVIDED}
> > > >
> > > > +if echo "${DEB_BUILD_PROFILES}" | grep -q "hostbuild"; then
> > > > +       # Force creating kernel kbuild debian package with
> > > > valid
> > > > host arch
> > > > +       # Use a cross build to comply with arch specific kernel
> > > > defconfigs
> > > > +       # The scripts and tools are always created for host
> > > > arch
> > > > +       eval $(dpkg-architecture -f -A ${DISTRO_ARCH})
> > > > +       CROSS_COMPILE=${DEB_TARGET_GNU_TYPE}-
> > > > +fi
> > > > +
> > > >  # Constants
> > > >  KCONF=.config
> > > >
> > > > diff --git a/meta/recipes-
> > > > kernel/linux/files/debian/isar/install.tmpl
> > > > b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
> > > > index 236b67c4..08b8a9a5 100644
> > > > --- a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
> > > > +++ b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
> > > > @@ -34,18 +34,24 @@ do_install() {
> > > >      set -x
> > > >
> > > >      # Run the install steps
> > > > -    install_image
> > > > -    if [ "${ARCH}" != "um" ]; then
> > > > -        install_config
> > > > -        install_map
> > > > +    if ! echo "${DEB_BUILD_PROFILES}" | grep -q -e
> > > > "targetbuild" -
> > > > e
> > > > "hostbuild"; then
> > > > +        install_image
> > > > +        if [ "${ARCH}" != "um" ]; then
> > > > +            install_config
> > > > +            install_map
> > > > +        fi
> > > > +        install_hooks
> > > > +        install_dtbs
> > > > +        install_kmods
> > > > +        install_headers
> > > >      fi
> > > > -    install_hooks
> > > > -    install_dtbs
> > > > -    install_kmods
> > > > -    install_headers
> > > >
> > > > -    # Cleanup and install kernel scripts and tools
> > > > -    rm -rf ${deb_kern_kbuild_dir}
> > > > +    if ! echo "${DEB_BUILD_PROFILES}" | grep -q -e
> > > > "targetbuild" -
> > > > e
> > > > "hostbuild" && echo "${DEB_BUILD_PROFILES}" | grep -q
> > > > "crosskbuild"; then
> > > > +        # Install cross kernel scripts and tools
> > > > +        install_kbuild ${deb_kern_kbuild_dir}-cross
> > > > +    fi
> > > > +
> > > > +    # Install kernel scripts and tools
> > > >      install_kbuild ${deb_kern_kbuild_dir}
> > > >
> > > >      # Stop tracing
> > > > diff --git a/meta/recipes-kernel/linux/files/debian/rules.tmpl
> > > > b/meta/recipes-kernel/linux/files/debian/rules.tmpl
> > > > index e8ae3daa..1f838a41 100755
> > > > --- a/meta/recipes-kernel/linux/files/debian/rules.tmpl
> > > > +++ b/meta/recipes-kernel/linux/files/debian/rules.tmpl
> > > > @@ -48,4 +48,4 @@ override_dh_strip_nondeterminism:
> > > >         true
> > > >
> > > >  override_dh_strip:
> > > > -       unset DEB_HOST_GNU_TYPE && dh_strip -Xvmlinu --no-
> > > > automatic-dbgsym
> > > > +       dh_strip -Xvmlinu -Xlinux-kbuild-
> > > > ${KERNEL_NAME_PROVIDED}-
> > > > cross --no-
> > > > automatic-dbgsym
> > > > diff --git a/meta/recipes-kernel/linux/linux-custom.inc
> > > > b/meta/recipes-
> > > > kernel/linux/linux-custom.inc
> > > > index 94ed84a2..85faa2d7 100644
> > > > --- a/meta/recipes-kernel/linux/linux-custom.inc
> > > > +++ b/meta/recipes-kernel/linux/linux-custom.inc
> > > > @@ -85,9 +85,6 @@ TEMPLATE_VARS += "                \
> > > >      DISTRIBUTOR                   \
> > > >  "
> > > >
> > > > -inherit dpkg
> > > > -inherit template
> > > > -
> > > >  # Add custom cflags to the kernel build
> > > >  KCFLAGS ?= "-fdebug-prefix-map=${CURDIR}=."
> > > >  KAFLAGS ?= "-fdebug-prefix-map=${CURDIR}=."
> > > > @@ -95,17 +92,57 @@ KAFLAGS ?= "-fdebug-prefix-map=${CURDIR}=."
> > > >  # Derive name of the kernel packages from the name of this
> > > > recipe
> > > >  KERNEL_NAME_PROVIDED ?= "${@ d.getVar('PN').partition('linux-
> > > > ')[2]}"
> > > >
> > > > -# Make bitbake know we will be producing linux-image and
> > > > linux-
> > > > headers
> > > > packages
> > > >  python() {
> > > >      kernel_name = d.getVar("KERNEL_NAME_PROVIDED")
> > > > -    d.setVar('PROVIDES', 'linux-image-' + kernel_name + ' ' +
> > > > \
> > > > -                         'linux-headers-' + kernel_name)
> > > > +    distro_arch = d.getVar("DISTRO_ARCH")
> > > > +    host_arch = d.getVar("HOST_ARCH")
> > > >      headers_depends = "linux-kbuild-" + kernel_name
> > > >
> > > > +    # The "-compat" and "-native" multiarch bitbake targets
> > > > are
> > > > used to run
> > > > +    # additional target or host specific builds for kbuild
> > > > scripts
> > > > and
> > > > tools.
> > > > +    # The "-compat" bitbake target requests a "target" build
> > > > +    # and the "-native" bitbake target requests a "host"
> > > > build.
> > > > +    #
> > > > +    # For different distro and host archs
> > > > +    # - Add dependency for sbuild-chroot-target
> > > > +    #   to allow building arch specific kbuild scripts and
> > > > tools
> > > > +    # - Set correct variable overrides
> > > > +    is_compat = "compat" in d.getVar("PN", True)
> > > > +    is_native = "native" in d.getVar("PN", True)
> > >
> > > Just a minor thing: The intended/preferred interface for recipes
> > > are
> > > the
> > > "class-native" and "class-compat" overrides set in
> > > multiarch.bbclass.
> > > This is nice for bitbake (just set MY_VAR:class-native =
> > > "value"),
> > > but
> > > unfortunately not so easy to use from Python code. Still, I'd
> > > avoid
> > > looking at PN again in the recipes, and instead go for
> > >
> > > is_compat = "class-compat" in d.getVar("OVERRIDES").split(":")
> > >
> > > > +
> > > > +    # Determine creation of a kbuild-cross or (standard)
> > > > kbuild
> > > > package for
> > > > the host
> > > > +    is_cross_profile = distro_arch != host_arch and
> > > > d.getVar("ISAR_CROSS_COMPILE", True) == "1"
> > > > +
> > > > +    if is_cross_profile:
> > > > +        headers_depends = headers_depends + "-cross | " +
> > > > headers_depends
> > > > +        build_depends = "crossbuild-essential-" + distro_arch
> > > > + "
> > > > [" +
> > > > host_arch + "], "
> > > > +        d.appendVar("DEB_BUILD_PROFILES", " crosskbuild")
> > > > +        d.prependVar("KBUILD_DEPENDS", build_depends)
> > > > +        d.appendVarFlag("do_prepare_build", "depends", "
> > > > sbuild-
> > > > chroot-
> > > > target:do_build")
> > > > +        d.setVar("ISAR_ENABLE_COMPAT_ARCH", "1")
> > > > +        d.setVar("COMPAT_DISTRO_ARCH", distro_arch)
> > >
> > > This feels hacky... it overrides the compat mechanism, resulting
> > > in a
> > > "*-compat" package that is not actually for the compat arch.
> >
> > Yes, it seems a bit hacky...
> >
> > I'm unsure if this special kernel specific approach to rebuild
> > kbuild
> > tools/scripts, forcing non-cross QEMU build even if ISAR cross-
> > build is
> > enabled, legimitates adding a new similar approach like the
> > compat/native mechanism within ISAR.
> >
> > Reusing the "-compat" bitbake target enables the build of
> > a target specific kbuild package at cross builds avoiding deeper
> > changes to ISAR. Forcing a QEMU non-cross build even if
> > ISAR_CROSS_COMPILE is enabled. This is the important part: Running
> > a
> > separate QEMU non-cross build for kernel scripts/tools. This is
> > needed
> > because the tools/scripts kernel build does not support creation
> > for
> > target arch while cross building.
>
> This is the property we should change once we have the means to build
> for the native arch as well.
>
> >
> > Let me know if you have suggestions to archieve that without using
> > the
> > compat mechanism and avoiding deeper changes to ISAR. Important is
> > that
> > a second dpkg build runs separatly within target's schroot (using
> > forced non-cross QEMU) while main build runs cross in host's
> > schroot.
> > Earlier patchsets of this series used a way without using the new
> > compat/native mechanism but made deeper changes to ISAR.
>
> We will need changes in Isar and possibly also in some downstream
> recipes, though I think can be caught by linux-custom-module.inc and
> its
> helpers. This should not be your blocker.
In the past linux-module.inc was not sufficient for building the kbuild
tools. linux-custom.inc was the right place.

What changes to the downstream modules are needed for that purpose?
>
> So far: we build tools for the host, even when building the kernel
> for
> the target.
>
> What we need: we build tools for the target, and if some module
> recipe
> wants to have them for cross-building as well, it asks for -native.

The following example uses arm64 as target and amd64 as host.

a) Yes with the patchset, linux-custom-native generates them for the
cross-build host:
- linux-custom-kbuild_amd64.deb that contains binary amd64 tools.

b) But anyway they are build within the default package (non-compat and
non-native) at cross-build, too. In the default case the host packages
are inside the target package with architecture mismatch:
- linux-custom-kbuild-cross_arm64.deb that contains binary amd64 tools.

Without the patchset
- linux-custom-headers_arm64.deb contains binary amd64 tools.

c) And linux-custom-compat:
linux-custom-kbuild_arm64.deb that contains binary arm64 tools.

>
> Does that make sense?

[PATCH v4 1/3] linux-custom: Split up binaries from kernel headers to
kbuild package

This patch could already merged. Since it keeps the old native/compat
behaviour. And moves the tools/scripts from kernel-headers in a
separated package.

>
> Jan
>

--
Stefan Koch
Siemens AG
http://www.siemens.com/