* custom-kernel headers: scripts are wrong architecture
@ 2024-01-17 20:37 MOESSBAUER, Felix
2024-01-18 7:52 ` Jan Kiszka
2024-04-30 16:54 ` Koch, Stefan
0 siblings, 2 replies; 6+ messages in thread
From: MOESSBAUER, Felix @ 2024-01-17 20:37 UTC (permalink / raw)
To: isar-users; +Cc: Kiszka, Jan, Koch, Stefan
Dear devs,
while debugging a broken non-cross kernel module build, I found that
there is a more generic issue about our linux-headers-${KERNEL_NAME}
package: It might contain the kernel scripts (ELF binaries) in the
wrong architecture. This only happens if the kernel itself is cross-
compiled.
Let's consider the following example, where we compile for a debian
bookworm arm64 target. We further look at the 'fixdeps' script from the
kernel tools (required during kmod builds).
Example 1 (stock kernel):
The script is located in package linux-kbuild-6.1 (dependency of linux-
headers-<...>).
- file /usr/lib/linux-kbuild-6.1/scripts/basic/fixdep: ELF 64-bit LSB
pie executable, ARM aarch64, ...
- linux-headers-arm64 : arm64
- linux-kbuild-6.1 : arm64
Example 2 (custom kernel):
- file /usr/src/linux-headers-6.1.54-cip6-rt3/scripts/basic/fixdep: ELF
64-bit LSB pie executable, x86-64
- linux-headers-iot2050-rt : arm64
In example 2, the kmod build is obviously broken, as the emulated build
will not be able to execute the x64 binary. Also, in an :arm64 package,
there must be no foreign architecture binaries.
This behavior can also be reproduced, by cross-compiling a target with
a custom kernel and disabling the cross-build for a module, using
ISAR_CROSS_COMPILE="0" in the module build recipe.
It would be great, if someone could have a look there. At a first
glance, this issue looks closely related to "linux-custom: Split up
binaries from kernel headers to kbuild packages". Maybe Stefan can
comment on this as well.
PS: This bug was uncovered today, as the cross-build of a kernel module
on debian bookworm arm64 is currently broken in upstream debian. I
already reported this bug in [1].
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1061075
Best regards,
Felix
--
Siemens AG, Technology
Linux Expert Center
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: custom-kernel headers: scripts are wrong architecture
2024-01-17 20:37 custom-kernel headers: scripts are wrong architecture MOESSBAUER, Felix
@ 2024-01-18 7:52 ` Jan Kiszka
2024-01-18 9:41 ` MOESSBAUER, Felix
2024-04-30 16:54 ` Koch, Stefan
1 sibling, 1 reply; 6+ messages in thread
From: Jan Kiszka @ 2024-01-18 7:52 UTC (permalink / raw)
To: Moessbauer, Felix (T CED OES-DE), isar-users
Cc: Koch, Stefan (DI PA DCP R&D 3)
On 17.01.24 21:37, Moessbauer, Felix (T CED OES-DE) wrote:
> Dear devs,
>
> while debugging a broken non-cross kernel module build, I found that
> there is a more generic issue about our linux-headers-${KERNEL_NAME}
> package: It might contain the kernel scripts (ELF binaries) in the
> wrong architecture. This only happens if the kernel itself is cross-
> compiled.
>
> Let's consider the following example, where we compile for a debian
> bookworm arm64 target. We further look at the 'fixdeps' script from the
> kernel tools (required during kmod builds).
>
> Example 1 (stock kernel):
>
> The script is located in package linux-kbuild-6.1 (dependency of linux-
> headers-<...>).
> - file /usr/lib/linux-kbuild-6.1/scripts/basic/fixdep: ELF 64-bit LSB
> pie executable, ARM aarch64, ...
> - linux-headers-arm64 : arm64
> - linux-kbuild-6.1 : arm64
>
> Example 2 (custom kernel):
>
> - file /usr/src/linux-headers-6.1.54-cip6-rt3/scripts/basic/fixdep: ELF
> 64-bit LSB pie executable, x86-64
> - linux-headers-iot2050-rt : arm64
>
> In example 2, the kmod build is obviously broken, as the emulated build
> will not be able to execute the x64 binary. Also, in an :arm64 package,
> there must be no foreign architecture binaries.
>
> This behavior can also be reproduced, by cross-compiling a target with
> a custom kernel and disabling the cross-build for a module, using
> ISAR_CROSS_COMPILE="0" in the module build recipe.
>
> It would be great, if someone could have a look there. At a first
> glance, this issue looks closely related to "linux-custom: Split up
> binaries from kernel headers to kbuild packages". Maybe Stefan can
> comment on this as well.
That is exactly what that patch is addressing. Stefan told me recently
that he rebased and refreshed that, and I hope we will see an update
soon. His aim is also resolving the issue that we currently build the
header and later on the kbuild package for the builder arch, rather than
for the target. But such an improvement needs more work as it requires
multiple kernel build runs for multiple archs while not all should
generate full kernels, obviously, only the needed kbuild packages.
Jan
>
> PS: This bug was uncovered today, as the cross-build of a kernel module
> on debian bookworm arm64 is currently broken in upstream debian. I
> already reported this bug in [1].
>
> [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1061075
>
> Best regards,
> Felix
>
--
Siemens AG, Technology
Linux Expert Center
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: custom-kernel headers: scripts are wrong architecture
2024-01-18 7:52 ` Jan Kiszka
@ 2024-01-18 9:41 ` MOESSBAUER, Felix
2024-01-18 9:43 ` Jan Kiszka
2024-01-18 9:59 ` Koch, Stefan
0 siblings, 2 replies; 6+ messages in thread
From: MOESSBAUER, Felix @ 2024-01-18 9:41 UTC (permalink / raw)
To: isar-users, Kiszka, Jan; +Cc: Koch, Stefan
On Thu, 2024-01-18 at 08:52 +0100, Jan Kiszka wrote:
> On 17.01.24 21:37, Moessbauer, Felix (T CED OES-DE) wrote:
> > Dear devs,
> >
> > while debugging a broken non-cross kernel module build, I found
> > that
> > there is a more generic issue about our linux-headers-
> > ${KERNEL_NAME}
> > package: It might contain the kernel scripts (ELF binaries) in the
> > wrong architecture. This only happens if the kernel itself is
> > cross-
> > compiled.
> >
> > Let's consider the following example, where we compile for a debian
> > bookworm arm64 target. We further look at the 'fixdeps' script from
> > the
> > kernel tools (required during kmod builds).
> >
> > Example 1 (stock kernel):
> >
> > The script is located in package linux-kbuild-6.1 (dependency of
> > linux-
> > headers-<...>).
> > - file /usr/lib/linux-kbuild-6.1/scripts/basic/fixdep: ELF 64-bit
> > LSB
> > pie executable, ARM aarch64, ...
> > - linux-headers-arm64 : arm64
> > - linux-kbuild-6.1 : arm64
> >
> > Example 2 (custom kernel):
> >
> > - file /usr/src/linux-headers-6.1.54-cip6-rt3/scripts/basic/fixdep:
> > ELF
> > 64-bit LSB pie executable, x86-64
> > - linux-headers-iot2050-rt : arm64
> >
> > In example 2, the kmod build is obviously broken, as the emulated
> > build
> > will not be able to execute the x64 binary. Also, in an :arm64
> > package,
> > there must be no foreign architecture binaries.
> >
> > This behavior can also be reproduced, by cross-compiling a target
> > with
> > a custom kernel and disabling the cross-build for a module, using
> > ISAR_CROSS_COMPILE="0" in the module build recipe.
> >
> > It would be great, if someone could have a look there. At a first
> > glance, this issue looks closely related to "linux-custom: Split up
> > binaries from kernel headers to kbuild packages". Maybe Stefan can
> > comment on this as well.
>
> That is exactly what that patch is addressing. Stefan told me
> recently
> that he rebased and refreshed that, and I hope we will see an update
> soon. His aim is also resolving the issue that we currently build the
> header and later on the kbuild package for the builder arch, rather
> than
I just rebased the v4 of Stefans series and checked it against this
issue: Unfortunately also the new approach with having the scripts in
the kbuild package still contains the scripts in the wrong
architecture:
linux-kbuild-iot2050-rt-cross : arm64
file /usr/lib/linux-kbuild-6.1.54-cip6-rt3/scripts/basic/fixdep
/usr/lib/linux-kbuild-6.1.54-cip6-rt3/scripts/basic/fixdep: ELF 64-bit
LSB pie executable, x86-64
By that I'm pretty sure, the issue here is independent and actually
arises from how we build the kernel (for building the scripts the wrong
compiler / toolchain is used).
Felix
> for the target. But such an improvement needs more work as it
> requires
> multiple kernel build runs for multiple archs while not all should
> generate full kernels, obviously, only the needed kbuild packages.
>
> Jan
>
> >
> > PS: This bug was uncovered today, as the cross-build of a kernel
> > module
> > on debian bookworm arm64 is currently broken in upstream debian. I
> > already reported this bug in [1].
> >
> > [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1061075
> >
> > Best regards,
> > Felix
> >
>
--
Siemens AG, Technology
Linux Expert Center
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: custom-kernel headers: scripts are wrong architecture
2024-01-18 9:41 ` MOESSBAUER, Felix
@ 2024-01-18 9:43 ` Jan Kiszka
2024-01-18 9:59 ` Koch, Stefan
1 sibling, 0 replies; 6+ messages in thread
From: Jan Kiszka @ 2024-01-18 9:43 UTC (permalink / raw)
To: Moessbauer, Felix (T CED OES-DE), isar-users
Cc: Koch, Stefan (DI PA DCP R&D 3)
On 18.01.24 10:41, Moessbauer, Felix (T CED OES-DE) wrote:
> On Thu, 2024-01-18 at 08:52 +0100, Jan Kiszka wrote:
>> On 17.01.24 21:37, Moessbauer, Felix (T CED OES-DE) wrote:
>>> Dear devs,
>>>
>>> while debugging a broken non-cross kernel module build, I found
>>> that
>>> there is a more generic issue about our linux-headers-
>>> ${KERNEL_NAME}
>>> package: It might contain the kernel scripts (ELF binaries) in the
>>> wrong architecture. This only happens if the kernel itself is
>>> cross-
>>> compiled.
>>>
>>> Let's consider the following example, where we compile for a debian
>>> bookworm arm64 target. We further look at the 'fixdeps' script from
>>> the
>>> kernel tools (required during kmod builds).
>>>
>>> Example 1 (stock kernel):
>>>
>>> The script is located in package linux-kbuild-6.1 (dependency of
>>> linux-
>>> headers-<...>).
>>> - file /usr/lib/linux-kbuild-6.1/scripts/basic/fixdep: ELF 64-bit
>>> LSB
>>> pie executable, ARM aarch64, ...
>>> - linux-headers-arm64 : arm64
>>> - linux-kbuild-6.1 : arm64
>>>
>>> Example 2 (custom kernel):
>>>
>>> - file /usr/src/linux-headers-6.1.54-cip6-rt3/scripts/basic/fixdep:
>>> ELF
>>> 64-bit LSB pie executable, x86-64
>>> - linux-headers-iot2050-rt : arm64
>>>
>>> In example 2, the kmod build is obviously broken, as the emulated
>>> build
>>> will not be able to execute the x64 binary. Also, in an :arm64
>>> package,
>>> there must be no foreign architecture binaries.
>>>
>>> This behavior can also be reproduced, by cross-compiling a target
>>> with
>>> a custom kernel and disabling the cross-build for a module, using
>>> ISAR_CROSS_COMPILE="0" in the module build recipe.
>>>
>>> It would be great, if someone could have a look there. At a first
>>> glance, this issue looks closely related to "linux-custom: Split up
>>> binaries from kernel headers to kbuild packages". Maybe Stefan can
>>> comment on this as well.
>>
>> That is exactly what that patch is addressing. Stefan told me
>> recently
>> that he rebased and refreshed that, and I hope we will see an update
>> soon. His aim is also resolving the issue that we currently build the
>> header and later on the kbuild package for the builder arch, rather
>> than
>
> I just rebased the v4 of Stefans series and checked it against this
> issue: Unfortunately also the new approach with having the scripts in
> the kbuild package still contains the scripts in the wrong
> architecture:
>
> linux-kbuild-iot2050-rt-cross : arm64
> file /usr/lib/linux-kbuild-6.1.54-cip6-rt3/scripts/basic/fixdep
> /usr/lib/linux-kbuild-6.1.54-cip6-rt3/scripts/basic/fixdep: ELF 64-bit
> LSB pie executable, x86-64
>
> By that I'm pretty sure, the issue here is independent and actually
> arises from how we build the kernel (for building the scripts the wrong
> compiler / toolchain is used).
>
Yes, there is more need than just the cited patch, and it's more complex
than just "do X". That's why it wasn't solved yet. At the same time, we
had not many use cases where the current behaviour of Isar was a
practical problem, and that's why it is like it is for, I don't know, 5
years now.
Jan
--
Siemens AG, Technology
Linux Expert Center
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: custom-kernel headers: scripts are wrong architecture
2024-01-18 9:41 ` MOESSBAUER, Felix
2024-01-18 9:43 ` Jan Kiszka
@ 2024-01-18 9:59 ` Koch, Stefan
1 sibling, 0 replies; 6+ messages in thread
From: Koch, Stefan @ 2024-01-18 9:59 UTC (permalink / raw)
To: isar-users, Kiszka, Jan, MOESSBAUER, Felix
On Thu, 2024-01-18 at 09:41 +0000, Moessbauer, Felix (T CED OES-DE)
wrote:
> On Thu, 2024-01-18 at 08:52 +0100, Jan Kiszka wrote:
> > On 17.01.24 21:37, Moessbauer, Felix (T CED OES-DE) wrote:
> > > Dear devs,
> > >
> > > while debugging a broken non-cross kernel module build, I found
> > > that
> > > there is a more generic issue about our linux-headers-
> > > ${KERNEL_NAME}
> > > package: It might contain the kernel scripts (ELF binaries) in
> > > the
> > > wrong architecture. This only happens if the kernel itself is
> > > cross-
> > > compiled.
> > >
> > > Let's consider the following example, where we compile for a
> > > debian
> > > bookworm arm64 target. We further look at the 'fixdeps' script
> > > from
> > > the
> > > kernel tools (required during kmod builds).
> > >
> > > Example 1 (stock kernel):
> > >
> > > The script is located in package linux-kbuild-6.1 (dependency of
> > > linux-
> > > headers-<...>).
> > > - file /usr/lib/linux-kbuild-6.1/scripts/basic/fixdep: ELF 64-bit
> > > LSB
> > > pie executable, ARM aarch64, ...
> > > - linux-headers-arm64 : arm64
> > > - linux-kbuild-6.1 : arm64
> > >
> > > Example 2 (custom kernel):
> > >
> > > - file /usr/src/linux-headers-6.1.54-cip6-
> > > rt3/scripts/basic/fixdep:
> > > ELF
> > > 64-bit LSB pie executable, x86-64
> > > - linux-headers-iot2050-rt : arm64
> > >
> > > In example 2, the kmod build is obviously broken, as the emulated
> > > build
> > > will not be able to execute the x64 binary. Also, in an :arm64
> > > package,
> > > there must be no foreign architecture binaries.
> > >
> > > This behavior can also be reproduced, by cross-compiling a target
> > > with
> > > a custom kernel and disabling the cross-build for a module, using
> > > ISAR_CROSS_COMPILE="0" in the module build recipe.
> > >
> > > It would be great, if someone could have a look there. At a first
> > > glance, this issue looks closely related to "linux-custom: Split
> > > up
> > > binaries from kernel headers to kbuild packages". Maybe Stefan
> > > can
> > > comment on this as well.
> >
> > That is exactly what that patch is addressing. Stefan told me
> > recently
> > that he rebased and refreshed that, and I hope we will see an
> > update
> > soon. His aim is also resolving the issue that we currently build
> > the
> > header and later on the kbuild package for the builder arch, rather
> > than
>
> I just rebased the v4 of Stefans series and checked it against this
> issue: Unfortunately also the new approach with having the scripts in
> the kbuild package still contains the scripts in the wrong
> architecture:
>
> linux-kbuild-iot2050-rt-cross : arm64
> file /usr/lib/linux-kbuild-6.1.54-cip6-rt3/scripts/basic/fixdep
> /usr/lib/linux-kbuild-6.1.54-cip6-rt3/scripts/basic/fixdep: ELF 64-
> bit
> LSB pie executable, x86-64
>
> By that I'm pretty sure, the issue here is independent and actually
> arises from how we build the kernel (for building the scripts the
> wrong
> compiler / toolchain is used).
>
> Felix
I'll send a new patch series soon. This series builds in a legacy way:
A separate package for kbuild containing fixdep and other tools with
wrong package arch:
- linux-kbuild-iot2050-rt-amd64-cross:arm64 containing fixdep@amd64
When KERNEL_KBUILD_DEPLOY is 1, then two additional dependend build
steps will run automatically: linux-custom-native (amd64 native build
of tools/scripts like fixdep) and linux-custom-target (arm64-QEMU non-
cross target build of tools/scripts like fixdep) providing these
packages:
- linux-kbuild-iot2050-rt:amd64 containing fixdep@amd64
- linux-kbuild-iot2050-rt:arm64 containing fixdep@arm64
The first package then can be used instead of the amd64-cross package
for cross build purposes. The second one can used on the target, e. g.
to build DKMS modules.
The headers package still remains arm64:
- linux-headers-iot2050-rt:arm64
Maybe this could further be changed from arm64 to all since there are
no binaries inside anymore. But debian itself doesn't provide a linux-
headers:all package.
>
> > for the target. But such an improvement needs more work as it
> > requires
> > multiple kernel build runs for multiple archs while not all should
> > generate full kernels, obviously, only the needed kbuild packages.
> >
> > Jan
> >
> > >
> > > PS: This bug was uncovered today, as the cross-build of a kernel
> > > module
> > > on debian bookworm arm64 is currently broken in upstream debian.
> > > I
> > > already reported this bug in [1].
> > >
> > > [1]
> > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1061075
> > >
> > > Best regards,
> > > Felix
> > >
> >
>
--
Stefan Koch
Siemens AG
http://www.siemens.com/
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: custom-kernel headers: scripts are wrong architecture
2024-01-17 20:37 custom-kernel headers: scripts are wrong architecture MOESSBAUER, Felix
2024-01-18 7:52 ` Jan Kiszka
@ 2024-04-30 16:54 ` Koch, Stefan
1 sibling, 0 replies; 6+ messages in thread
From: Koch, Stefan @ 2024-04-30 16:54 UTC (permalink / raw)
To: isar-users, Kiszka, Jan, MOESSBAUER, Felix
Cc: Sudler, Simon, Storm, Christian, Schmidt, Adriaan,
cedric.hombourger, Adler, Michael, ubely
@Jan:
This is fixed by the patch "linux-module: Support emulated module build
with cross-compiled kernel"
On Wed, 2024-01-17 at 20:37 +0000, Moessbauer, Felix (T CED OES-DE)
wrote:
> Dear devs,
>
> while debugging a broken non-cross kernel module build, I found that
> there is a more generic issue about our linux-headers-${KERNEL_NAME}
> package: It might contain the kernel scripts (ELF binaries) in the
> wrong architecture. This only happens if the kernel itself is cross-
> compiled.
>
> Let's consider the following example, where we compile for a debian
> bookworm arm64 target. We further look at the 'fixdeps' script from
> the
> kernel tools (required during kmod builds).
>
> Example 1 (stock kernel):
>
> The script is located in package linux-kbuild-6.1 (dependency of
> linux-
> headers-<...>).
> - file /usr/lib/linux-kbuild-6.1/scripts/basic/fixdep: ELF 64-bit LSB
> pie executable, ARM aarch64, ...
> - linux-headers-arm64 : arm64
> - linux-kbuild-6.1 : arm64
>
> Example 2 (custom kernel):
>
> - file /usr/src/linux-headers-6.1.54-cip6-rt3/scripts/basic/fixdep:
> ELF
> 64-bit LSB pie executable, x86-64
> - linux-headers-iot2050-rt : arm64
>
> In example 2, the kmod build is obviously broken, as the emulated
> build
> will not be able to execute the x64 binary. Also, in an :arm64
> package,
> there must be no foreign architecture binaries.
>
> This behavior can also be reproduced, by cross-compiling a target
> with
> a custom kernel and disabling the cross-build for a module, using
> ISAR_CROSS_COMPILE="0" in the module build recipe.
>
> It would be great, if someone could have a look there. At a first
> glance, this issue looks closely related to "linux-custom: Split up
> binaries from kernel headers to kbuild packages". Maybe Stefan can
> comment on this as well.
>
> PS: This bug was uncovered today, as the cross-build of a kernel
> module
> on debian bookworm arm64 is currently broken in upstream debian. I
> already reported this bug in [1].
>
> [1]
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1061075
>
> Best regards,
> Felix
>
--
Stefan Koch
Siemens AG
http://www.siemens.com/
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-04-30 16:54 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-01-17 20:37 custom-kernel headers: scripts are wrong architecture MOESSBAUER, Felix
2024-01-18 7:52 ` Jan Kiszka
2024-01-18 9:41 ` MOESSBAUER, Felix
2024-01-18 9:43 ` Jan Kiszka
2024-01-18 9:59 ` Koch, Stefan
2024-04-30 16:54 ` Koch, Stefan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox