Re: [PATCH v4 2/3] linux-custom: Provide host and target specific kernel kbuild packages

[Jan Kiszka <jan.kiszka@siemens.com>]

On 04.10.23 17:34, Koch, Stefan (DI PA DCP R&D 3) wrote:
> On Wed, 2023-09-06 at 06:38 +0000, Schmidt, Adriaan (T CED SES-DE)
> wrote:
>> Koch, Stefan (DI PA DCP R&D 3) <stefan-koch@siemens.com>, Monday,
>> August 21, 2023 3:45 PM:
>>> When using a cross build this patch does introduce optionally
>>> host and target specific kernel kbuild packages that
>>> ship the "scripts" and "tools" binaries.
>>>
>>> By default at cross builds a kbuild-cross package will be
>>> generated that provides the host specific kbuild package.
>>>
>>> The "-compat" and "-native" multiarch bitbake targets are useable
>>> to run
>>> additional target or host specific builds for kbuild scripts and
>>> tools.
>>>
>>> Using the "-compat" bitbake target enables the build of
>>> a target specific kbuild package at cross builds.
>>>
>>> Using the "-native" bitbake target enables the build of
>>> a host specific kbuild package at cross builds.
>>>
>>> This solves this from doc/custom_kernel.inc:
>>> - The kernel headers package has not supported both native
>>>   and cross compilation of kernel modules when itself was cross
>>> built
>>> - Future roadmap: Generate kernel headers package for both host
>>>   and target when using a cross build
>>>
>>> Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
>>> ---
>>>  .../linux/files/debian/control.tmpl           | 16 +++++-
>>>  .../linux/files/debian/isar/build.tmpl        | 12 ++++-
>>>  .../linux/files/debian/isar/common.tmpl       |  8 +++
>>>  .../linux/files/debian/isar/install.tmpl      | 26 ++++++----
>>>  .../linux/files/debian/rules.tmpl             |  2 +-
>>>  meta/recipes-kernel/linux/linux-custom.inc    | 49
>>> ++++++++++++++++---
>>>  6 files changed, 94 insertions(+), 19 deletions(-)
>>>
>>> diff --git a/meta/recipes-kernel/linux/files/debian/control.tmpl
>>> b/meta/recipes-kernel/linux/files/debian/control.tmpl
>>> index b19ca2c9..c012048b 100644
>>> --- a/meta/recipes-kernel/linux/files/debian/control.tmpl
>>> +++ b/meta/recipes-kernel/linux/files/debian/control.tmpl
>>> @@ -6,6 +6,7 @@ Build-Depends: bc, kmod, cpio, ${KBUILD_DEPENDS}
>>>  Homepage:
>>> http://www.kernel.org/
>>>
>>>  Package: linux-image-${KERNEL_NAME_PROVIDED}
>>> +Build-Profiles: <!targetbuild !hostbuild>
>>>  Architecture: any
>>>  Depends: ${KERNEL_DEBIAN_DEPENDS}
>>>  Description: ${KERNEL_NAME_PROVIDED} Linux kernel, version @KR@
>>> @@ -13,6 +14,7 @@ Description: ${KERNEL_NAME_PROVIDED} Linux
>>> kernel, version
>>> @KR@
>>>   files, version: @KR@.
>>>
>>>  Package: linux-headers-${KERNEL_NAME_PROVIDED}
>>> +Build-Profiles: <!targetbuild !hostbuild>
>>>  Architecture: any
>>>  Depends: ${KERNEL_HEADERS_DEBIAN_DEPENDS}, ${perl:Depends},
>>> ${shlib:Depends}
>>>  Description: ${KERNEL_NAME_PROVIDED} Linux kernel headers for @KR@
>>> @@ -21,7 +23,7 @@ Description: ${KERNEL_NAME_PROVIDED} Linux kernel
>>> headers
>>> for @KR@
>>>   This is useful for people who need to build external modules
>>>
>>>  Package: linux-libc-dev
>>> -Build-Profiles: <!nolibcdev>
>>> +Build-Profiles: <!nolibcdev !targetbuild !hostbuild>
>>>  Section: devel
>>>  Provides: linux-kernel-headers
>>>  Architecture: any
>>> @@ -30,6 +32,7 @@ Description: Linux support headers for userspace
>>> development
>>>   are used by the installed headers for GNU glibc and other system
>>> libraries.
>>>
>>>  Package: linux-image-${KERNEL_NAME_PROVIDED}-dbg
>>> +Build-Profiles: <!targetbuild !hostbuild>
>>>  Section: debug
>>>  Architecture: any
>>>  Description: Linux kernel debugging symbols for @KR@
>>> @@ -37,8 +40,19 @@ Description: Linux kernel debugging symbols for
>>> @KR@
>>>   all the necessary debug symbols for the kernel and its modules.
>>>
>>>  Package: linux-kbuild-${KERNEL_NAME_PROVIDED}
>>> +Build-Profiles: <targetbuild> <hostbuild> <defaultkbuild>
>>>  Architecture: any
>>>  Depends: ${perl:Depends}, ${shlib:Depends}
>>>  Description: ${KERNEL_NAME_PROVIDED} Linux kbuild scripts and
>>> tools for @KR@
>>>   This package provides kernel kbuild scripts and tools for @KR@
>>>   This is useful for people who need to build external modules
>>> +
>>> +Package: linux-kbuild-${KERNEL_NAME_PROVIDED}-cross
>>> +Build-Profiles: <!targetbuild !hostbuild crosskbuild>
>>> +Architecture: any
>>> +Depends: ${KERNEL_HEADERS_DEBIAN_DEPENDS}, ${perl:Depends},
>>> ${shlib:Depends}
>>> +Description: ${KERNEL_NAME_PROVIDED} Linux kbuild scripts and
>>> tools for @KR@
>>> + This package provides kernel kbuild scripts and tools
>>> + as ${HOST_ARCH} cross binaries for @KR@
>>> + This is useful for those who need to cross build
>>> + external modules using ISAR's sbuild-chroot-host
>>> diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
>>> b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
>>> index 906dc580..7f293528 100644
>>> --- a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
>>> +++ b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
>>> @@ -22,7 +22,17 @@ do_build() {
>>>      sed -i "s/@KR@/${KR}/g" ${S}/debian/control ${S}/debian/linux-
>>> image-
>>> ${KERNEL_NAME_PROVIDED}.*
>>>
>>>      # Build the Linux kernel
>>> -    ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE}
>>> KCFLAGS="${KCFLAGS}"
>>> KAFLAGS="${KAFLAGS}"
>>> +    if echo "${DEB_BUILD_PROFILES}" | grep -q -e "targetbuild" -e
>>> "hostbuild"; then # Build kernel scripts and tools
>>> +        ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE}
>>> KCFLAGS="${KCFLAGS}"
>>> KAFLAGS="${KAFLAGS}" scripts
>>> +        if grep -q -E
>>> "CONFIG_STACK_VALIDATION=y|CONFIG_HAVE_OBJTOOL=y"
>>> ${KERNEL_BUILD_DIR}/.config && [ -d "tools/objtool" ]; then
>>> +            ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE}
>>> KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" tools/objtool || true
>>> +        fi
>>> +        if grep -q "CONFIG_MODULES=y" ${KERNEL_BUILD_DIR}/.config;
>>> then
>>> +            ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE}
>>> KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" modules_prepare
>>> +        fi
>>> +    else # Build the Linux kernel
>>> +        ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE}
>>> KCFLAGS="${KCFLAGS}"
>>> KAFLAGS="${KAFLAGS}"
>>> +    fi
>>>
>>>      # Stop tracing
>>>      set +x
>>> diff --git a/meta/recipes-
>>> kernel/linux/files/debian/isar/common.tmpl
>>> b/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
>>> index f4c0519f..36e01691 100644
>>> --- a/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
>>> +++ b/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
>>> @@ -11,6 +11,14 @@ KERNEL_PKG_KERN_HEADERS=linux-headers-
>>> ${KERNEL_NAME_PROVIDED}
>>>  KERNEL_PKG_LIBC_HEADERS=linux-libc-dev
>>>  KERNEL_PKG_KERN_KBUILD=linux-kbuild-${KERNEL_NAME_PROVIDED}
>>>
>>> +if echo "${DEB_BUILD_PROFILES}" | grep -q "hostbuild"; then
>>> +       # Force creating kernel kbuild debian package with valid
>>> host arch
>>> +       # Use a cross build to comply with arch specific kernel
>>> defconfigs
>>> +       # The scripts and tools are always created for host arch
>>> +       eval $(dpkg-architecture -f -A ${DISTRO_ARCH})
>>> +       CROSS_COMPILE=${DEB_TARGET_GNU_TYPE}-
>>> +fi
>>> +
>>>  # Constants
>>>  KCONF=.config
>>>
>>> diff --git a/meta/recipes-
>>> kernel/linux/files/debian/isar/install.tmpl
>>> b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
>>> index 236b67c4..08b8a9a5 100644
>>> --- a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
>>> +++ b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
>>> @@ -34,18 +34,24 @@ do_install() {
>>>      set -x
>>>
>>>      # Run the install steps
>>> -    install_image
>>> -    if [ "${ARCH}" != "um" ]; then
>>> -        install_config
>>> -        install_map
>>> +    if ! echo "${DEB_BUILD_PROFILES}" | grep -q -e "targetbuild" -
>>> e
>>> "hostbuild"; then
>>> +        install_image
>>> +        if [ "${ARCH}" != "um" ]; then
>>> +            install_config
>>> +            install_map
>>> +        fi
>>> +        install_hooks
>>> +        install_dtbs
>>> +        install_kmods
>>> +        install_headers
>>>      fi
>>> -    install_hooks
>>> -    install_dtbs
>>> -    install_kmods
>>> -    install_headers
>>>
>>> -    # Cleanup and install kernel scripts and tools
>>> -    rm -rf ${deb_kern_kbuild_dir}
>>> +    if ! echo "${DEB_BUILD_PROFILES}" | grep -q -e "targetbuild" -
>>> e
>>> "hostbuild" && echo "${DEB_BUILD_PROFILES}" | grep -q
>>> "crosskbuild"; then
>>> +        # Install cross kernel scripts and tools
>>> +        install_kbuild ${deb_kern_kbuild_dir}-cross
>>> +    fi
>>> +
>>> +    # Install kernel scripts and tools
>>>      install_kbuild ${deb_kern_kbuild_dir}
>>>
>>>      # Stop tracing
>>> diff --git a/meta/recipes-kernel/linux/files/debian/rules.tmpl
>>> b/meta/recipes-kernel/linux/files/debian/rules.tmpl
>>> index e8ae3daa..1f838a41 100755
>>> --- a/meta/recipes-kernel/linux/files/debian/rules.tmpl
>>> +++ b/meta/recipes-kernel/linux/files/debian/rules.tmpl
>>> @@ -48,4 +48,4 @@ override_dh_strip_nondeterminism:
>>>         true
>>>
>>>  override_dh_strip:
>>> -       unset DEB_HOST_GNU_TYPE && dh_strip -Xvmlinu --no-
>>> automatic-dbgsym
>>> +       dh_strip -Xvmlinu -Xlinux-kbuild-${KERNEL_NAME_PROVIDED}-
>>> cross --no-
>>> automatic-dbgsym
>>> diff --git a/meta/recipes-kernel/linux/linux-custom.inc
>>> b/meta/recipes-
>>> kernel/linux/linux-custom.inc
>>> index 94ed84a2..85faa2d7 100644
>>> --- a/meta/recipes-kernel/linux/linux-custom.inc
>>> +++ b/meta/recipes-kernel/linux/linux-custom.inc
>>> @@ -85,9 +85,6 @@ TEMPLATE_VARS += "                \
>>>      DISTRIBUTOR                   \
>>>  "
>>>
>>> -inherit dpkg
>>> -inherit template
>>> -
>>>  # Add custom cflags to the kernel build
>>>  KCFLAGS ?= "-fdebug-prefix-map=${CURDIR}=."
>>>  KAFLAGS ?= "-fdebug-prefix-map=${CURDIR}=."
>>> @@ -95,17 +92,57 @@ KAFLAGS ?= "-fdebug-prefix-map=${CURDIR}=."
>>>  # Derive name of the kernel packages from the name of this recipe
>>>  KERNEL_NAME_PROVIDED ?= "${@ d.getVar('PN').partition('linux-
>>> ')[2]}"
>>>
>>> -# Make bitbake know we will be producing linux-image and linux-
>>> headers
>>> packages
>>>  python() {
>>>      kernel_name = d.getVar("KERNEL_NAME_PROVIDED")
>>> -    d.setVar('PROVIDES', 'linux-image-' + kernel_name + ' ' + \
>>> -                         'linux-headers-' + kernel_name)
>>> +    distro_arch = d.getVar("DISTRO_ARCH")
>>> +    host_arch = d.getVar("HOST_ARCH")
>>>      headers_depends = "linux-kbuild-" + kernel_name
>>>
>>> +    # The "-compat" and "-native" multiarch bitbake targets are
>>> used to run
>>> +    # additional target or host specific builds for kbuild scripts
>>> and
>>> tools.
>>> +    # The "-compat" bitbake target requests a "target" build
>>> +    # and the "-native" bitbake target requests a "host" build.
>>> +    #
>>> +    # For different distro and host archs
>>> +    # - Add dependency for sbuild-chroot-target
>>> +    #   to allow building arch specific kbuild scripts and tools
>>> +    # - Set correct variable overrides
>>> +    is_compat = "compat" in d.getVar("PN", True)
>>> +    is_native = "native" in d.getVar("PN", True)
>>
>> Just a minor thing: The intended/preferred interface for recipes are
>> the
>> "class-native" and "class-compat" overrides set in multiarch.bbclass.
>> This is nice for bitbake (just set MY_VAR:class-native = "value"),
>> but
>> unfortunately not so easy to use from Python code. Still, I'd avoid
>> looking at PN again in the recipes, and instead go for
>>
>> is_compat = "class-compat" in d.getVar("OVERRIDES").split(":")
>>
>>> +
>>> +    # Determine creation of a kbuild-cross or (standard) kbuild
>>> package for
>>> the host
>>> +    is_cross_profile = distro_arch != host_arch and
>>> d.getVar("ISAR_CROSS_COMPILE", True) == "1"
>>> +
>>> +    if is_cross_profile:
>>> +        headers_depends = headers_depends + "-cross | " +
>>> headers_depends
>>> +        build_depends = "crossbuild-essential-" + distro_arch + "
>>> [" +
>>> host_arch + "], "
>>> +        d.appendVar("DEB_BUILD_PROFILES", " crosskbuild")
>>> +        d.prependVar("KBUILD_DEPENDS", build_depends)
>>> +        d.appendVarFlag("do_prepare_build", "depends", " sbuild-
>>> chroot-
>>> target:do_build")
>>> +        d.setVar("ISAR_ENABLE_COMPAT_ARCH", "1")
>>> +        d.setVar("COMPAT_DISTRO_ARCH", distro_arch)
>>
>> This feels hacky... it overrides the compat mechanism, resulting in a
>> "*-compat" package that is not actually for the compat arch.
> 
> Yes, it seems a bit hacky...
> 
> I'm unsure if this special kernel specific approach to rebuild kbuild
> tools/scripts, forcing non-cross QEMU build even if ISAR cross-build is
> enabled, legimitates adding a new similar approach like the
> compat/native mechanism within ISAR.
> 
> Reusing the "-compat" bitbake target enables the build of
> a target specific kbuild package at cross builds avoiding deeper
> changes to ISAR. Forcing a QEMU non-cross build even if
> ISAR_CROSS_COMPILE is enabled. This is the important part: Running a
> separate QEMU non-cross build for kernel scripts/tools. This is needed
> because the tools/scripts kernel build does not support creation for
> target arch while cross building.

This is the property we should change once we have the means to build
for the native arch as well.

> 
> Let me know if you have suggestions to archieve that without using the
> compat mechanism and avoiding deeper changes to ISAR. Important is that
> a second dpkg build runs separatly within target's schroot (using
> forced non-cross QEMU) while main build runs cross in host's schroot.
> Earlier patchsets of this series used a way without using the new
> compat/native mechanism but made deeper changes to ISAR.

We will need changes in Isar and possibly also in some downstream
recipes, though I think can be caught by linux-custom-module.inc and its
helpers. This should not be your blocker.

So far: we build tools for the host, even when building the kernel for
the target.

What we need: we build tools for the target, and if some module recipe
wants to have them for cross-building as well, it asks for -native.

Does that make sense?

Jan

-- 
Siemens AG, Technology
Linux Expert Center