Re: custom-kernel headers: scripts are wrong architecture

["Koch, Stefan" <stefan-koch@siemens.com>]

On Thu, 2024-01-18 at 09:41 +0000, Moessbauer, Felix (T CED OES-DE)
wrote:
> On Thu, 2024-01-18 at 08:52 +0100, Jan Kiszka wrote:
> > On 17.01.24 21:37, Moessbauer, Felix (T CED OES-DE) wrote:
> > > Dear devs,
> > >
> > > while debugging a broken non-cross kernel module build, I found
> > > that
> > > there is a more generic issue about our linux-headers-
> > > ${KERNEL_NAME}
> > > package: It might contain the kernel scripts (ELF binaries) in
> > > the
> > > wrong architecture. This only happens if the kernel itself is
> > > cross-
> > > compiled.
> > >
> > > Let's consider the following example, where we compile for a
> > > debian
> > > bookworm arm64 target. We further look at the 'fixdeps' script
> > > from
> > > the
> > > kernel tools (required during kmod builds).
> > >
> > > Example 1 (stock kernel):
> > >
> > > The script is located in package linux-kbuild-6.1 (dependency of
> > > linux-
> > > headers-<...>).
> > > - file /usr/lib/linux-kbuild-6.1/scripts/basic/fixdep: ELF 64-bit
> > > LSB
> > > pie executable, ARM aarch64, ...
> > > - linux-headers-arm64 : arm64
> > > - linux-kbuild-6.1 : arm64
> > >
> > > Example 2 (custom kernel):
> > >
> > > - file /usr/src/linux-headers-6.1.54-cip6-
> > > rt3/scripts/basic/fixdep:
> > > ELF
> > > 64-bit LSB pie executable, x86-64
> > > - linux-headers-iot2050-rt : arm64
> > >
> > > In example 2, the kmod build is obviously broken, as the emulated
> > > build
> > > will not be able to execute the x64 binary. Also, in an :arm64
> > > package,
> > > there must be no foreign architecture binaries.
> > >
> > > This behavior can also be reproduced, by cross-compiling a target
> > > with
> > > a custom kernel and disabling the cross-build for a module, using
> > > ISAR_CROSS_COMPILE="0" in the module build recipe.
> > >
> > > It would be great, if someone could have a look there. At a first
> > > glance, this issue looks closely related to "linux-custom: Split
> > > up
> > > binaries from kernel headers to kbuild packages". Maybe Stefan
> > > can
> > > comment on this as well.
> >
> > That is exactly what that patch is addressing. Stefan told me
> > recently
> > that he rebased and refreshed that, and I hope we will see an
> > update
> > soon. His aim is also resolving the issue that we currently build
> > the
> > header and later on the kbuild package for the builder arch, rather
> > than
>
> I just rebased the v4 of Stefans series and checked it against this
> issue: Unfortunately also the new approach with having the scripts in
> the kbuild package still contains the scripts in the wrong
> architecture:
>
> linux-kbuild-iot2050-rt-cross : arm64
> file /usr/lib/linux-kbuild-6.1.54-cip6-rt3/scripts/basic/fixdep
> /usr/lib/linux-kbuild-6.1.54-cip6-rt3/scripts/basic/fixdep: ELF 64-
> bit
> LSB pie executable, x86-64
>
> By that I'm pretty sure, the issue here is independent and actually
> arises from how we build the kernel (for building the scripts the
> wrong
> compiler / toolchain is used).
>
> Felix

I'll send a new patch series soon. This series builds in a legacy way:
A separate package for kbuild containing fixdep and other tools with
wrong package arch:
- linux-kbuild-iot2050-rt-amd64-cross:arm64 containing fixdep@amd64

When KERNEL_KBUILD_DEPLOY is 1, then two additional dependend build
steps will run automatically: linux-custom-native (amd64 native build
of tools/scripts like fixdep) and linux-custom-target (arm64-QEMU non-
cross target build of tools/scripts like fixdep) providing these
packages:
- linux-kbuild-iot2050-rt:amd64 containing fixdep@amd64
- linux-kbuild-iot2050-rt:arm64 containing fixdep@arm64

The first package then can be used instead of the amd64-cross package
for cross build purposes. The second one can used on the target, e. g.
to build DKMS modules.

The headers package still remains arm64:
- linux-headers-iot2050-rt:arm64

Maybe this could further be changed from arm64 to all since there are
no binaries inside anymore. But debian itself doesn't provide a linux-
headers:all package.

>
> > for the target. But such an improvement needs more work as it
> > requires
> > multiple kernel build runs for multiple archs while not all should
> > generate full kernels, obviously, only the needed kbuild packages.
> >
> > Jan
> >
> > >
> > > PS: This bug was uncovered today, as the cross-build of a kernel
> > > module
> > > on debian bookworm arm64 is currently broken in upstream debian.
> > > I
> > > already reported this bug in [1].
> > >
> > > [1]
> > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1061075
> > >
> > > Best regards,
> > > Felix
> > >
> >
>

--
Stefan Koch
Siemens AG
http://www.siemens.com/