* [PATCH v2] linux-custom: Speedup build of target specific linux-kbuild package
@ 2024-06-07 14:11 Stefan Koch
2024-06-07 15:07 ` Jan Kiszka
0 siblings, 1 reply; 3+ messages in thread
From: Stefan Koch @ 2024-06-07 14:11 UTC (permalink / raw)
To: isar-users
Cc: stefan-koch, jan.kiszka, christian.storm, michael.adler,
simon.sudler, cedric.hombourger, adriaan.schmidt,
felix.moessbauer, ubely, amikan
Avoids time expensive QEMU-emulated merging of kernel config by using
the existing kernel config for the target specific linux-kbuild
Using DEPENDS instead of RDEPENDS ensures creation of
kernel (including config) before build of target specific linux-kbuild
Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
---
.../linux/files/debian/isar/configure.tmpl | 18 ++++++++++++++----
meta/recipes-kernel/linux/linux-custom.inc | 10 +++++++---
2 files changed, 21 insertions(+), 7 deletions(-)
diff --git a/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl b/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
index 389c9a85..72eae448 100644
--- a/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
@@ -10,10 +10,20 @@ do_configure() {
# Trace what we do here
set -x
- # Process kernel config target and fragments
- ${MAKE} O=${KERNEL_BUILD_DIR} ${KERNEL_CONFIG_TARGET}
- ./scripts/kconfig/merge_config.sh -O ${KERNEL_BUILD_DIR}/ \
- ${KERNEL_BUILD_DIR}/.config ${KERNEL_CONFIG_FRAGMENTS}
+ kernelconfig="$(find /boot -maxdepth 1 -name "config-${PV}*" -print -quit)"
+ if [ -e "${kernelconfig}" ]; then
+ # Prefer existing kernel config
+ # So, very expensive QEMU-emulated merge_config.sh
+ # can be skipped for target specific linux-kbuild package
+ mkdir -p ${KERNEL_BUILD_DIR}
+ cp "${kernelconfig}" ${KERNEL_BUILD_DIR}/.config
+ ${MAKE} O=${KERNEL_BUILD_DIR} olddefconfig
+ else
+ # Process kernel config target and fragments
+ ${MAKE} O=${KERNEL_BUILD_DIR} ${KERNEL_CONFIG_TARGET}
+ ./scripts/kconfig/merge_config.sh -O ${KERNEL_BUILD_DIR}/ \
+ ${KERNEL_BUILD_DIR}/.config ${KERNEL_CONFIG_FRAGMENTS}
+ fi
# Stop tracing
set +x
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 647f09dd..dc354e13 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -25,6 +25,7 @@ KBUILD_DEPENDS ?= "build-essential:native, \
flex, \
git, \
kmod, \
+ linux-image-${KERNEL_NAME_PROVIDED}:${DISTRO_ARCH} <kbuild !kernel>, \
rsync,"
KERNEL_DEBIAN_DEPENDS ?= "initramfs-tools | linux-initramfs-tool, \
@@ -112,7 +113,7 @@ BUILD_PROFILES = "kernel kbuild"
BBCLASSEXTEND:append:cross-profile = " kbuildtarget"
# When cross-profile is active:
-# build only kernel with the default variant of the recipe
+# build only kernel (including config) with the default variant of the recipe
BUILD_PROFILES:cross-profile = "kernel"
# -native: kbuild package for host
@@ -122,14 +123,17 @@ RECIPE_PROVIDES:class-native = " \
linux-kbuild-${KERNEL_NAME_PROVIDED}"
# Use pseudo target to pull in the base variant of the recipe.
# Will be auto-extended with -native by multiarch.bbclass.
-RDEPENDS:class-native += "${BPN}-pseudo"
+# Using DEPENDS, see below
+DEPENDS:class-native += "${BPN}-pseudo"
# -kbuildtarget: kbuild package for target, enforcing non-cross-build
BUILD_PROFILES:class-kbuildtarget = "kbuild"
RECIPE_PROVIDES:class-kbuildtarget = " \
linux-headers-${KERNEL_NAME_PROVIDED} \
linux-kbuild-${KERNEL_NAME_PROVIDED}"
-RDEPENDS:class-kbuildtarget = "${BPN}"
+# Using DEPENDS instead of RDEPENDS to ensure creation of kernel including
+# pregenerated kernel config before target specific linux-kbuild package build
+DEPENDS:class-kbuildtarget = "${BPN}"
ISAR_CROSS_COMPILE:class-kbuildtarget = "0"
# Make bitbake know we will be producing linux-image and linux-headers packages
--
2.39.2
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH v2] linux-custom: Speedup build of target specific linux-kbuild package
2024-06-07 14:11 [PATCH v2] linux-custom: Speedup build of target specific linux-kbuild package Stefan Koch
@ 2024-06-07 15:07 ` Jan Kiszka
2024-06-07 15:37 ` Koch, Stefan
0 siblings, 1 reply; 3+ messages in thread
From: Jan Kiszka @ 2024-06-07 15:07 UTC (permalink / raw)
To: Stefan Koch, isar-users
Cc: christian.storm, michael.adler, simon.sudler, cedric.hombourger,
adriaan.schmidt, felix.moessbauer, ubely, amikan
On 07.06.24 16:11, Stefan Koch wrote:
> Avoids time expensive QEMU-emulated merging of kernel config by using
> the existing kernel config for the target specific linux-kbuild
>
> Using DEPENDS instead of RDEPENDS ensures creation of
> kernel (including config) before build of target specific linux-kbuild
>
> Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
> ---
> .../linux/files/debian/isar/configure.tmpl | 18 ++++++++++++++----
> meta/recipes-kernel/linux/linux-custom.inc | 10 +++++++---
> 2 files changed, 21 insertions(+), 7 deletions(-)
>
> diff --git a/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl b/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
> index 389c9a85..72eae448 100644
> --- a/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
> +++ b/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
> @@ -10,10 +10,20 @@ do_configure() {
> # Trace what we do here
> set -x
>
> - # Process kernel config target and fragments
> - ${MAKE} O=${KERNEL_BUILD_DIR} ${KERNEL_CONFIG_TARGET}
> - ./scripts/kconfig/merge_config.sh -O ${KERNEL_BUILD_DIR}/ \
> - ${KERNEL_BUILD_DIR}/.config ${KERNEL_CONFIG_FRAGMENTS}
> + kernelconfig="$(find /boot -maxdepth 1 -name "config-${PV}*" -print -quit)"
Can we additionally limit this to the case were we expect the config
there (kbuild && !kernel)? Just to reduce the risk that some "this
should never be there" config is ever silently pulled by the base
package build. Only if our dependency is installed, that rather generic
config-<version> file is from there, or we failed on installing the
dependency.
> + if [ -e "${kernelconfig}" ]; then
> + # Prefer existing kernel config
> + # So, very expensive QEMU-emulated merge_config.sh
> + # can be skipped for target specific linux-kbuild package
> + mkdir -p ${KERNEL_BUILD_DIR}
> + cp "${kernelconfig}" ${KERNEL_BUILD_DIR}/.config
> + ${MAKE} O=${KERNEL_BUILD_DIR} olddefconfig
> + else
> + # Process kernel config target and fragments
> + ${MAKE} O=${KERNEL_BUILD_DIR} ${KERNEL_CONFIG_TARGET}
> + ./scripts/kconfig/merge_config.sh -O ${KERNEL_BUILD_DIR}/ \
> + ${KERNEL_BUILD_DIR}/.config ${KERNEL_CONFIG_FRAGMENTS}
> + fi
>
> # Stop tracing
> set +x
> diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
> index 647f09dd..dc354e13 100644
> --- a/meta/recipes-kernel/linux/linux-custom.inc
> +++ b/meta/recipes-kernel/linux/linux-custom.inc
> @@ -25,6 +25,7 @@ KBUILD_DEPENDS ?= "build-essential:native, \
> flex, \
> git, \
> kmod, \
> + linux-image-${KERNEL_NAME_PROVIDED}:${DISTRO_ARCH} <kbuild !kernel>, \
> rsync,"
>
> KERNEL_DEBIAN_DEPENDS ?= "initramfs-tools | linux-initramfs-tool, \
> @@ -112,7 +113,7 @@ BUILD_PROFILES = "kernel kbuild"
> BBCLASSEXTEND:append:cross-profile = " kbuildtarget"
>
> # When cross-profile is active:
> -# build only kernel with the default variant of the recipe
> +# build only kernel (including config) with the default variant of the recipe
> BUILD_PROFILES:cross-profile = "kernel"
>
> # -native: kbuild package for host
> @@ -122,14 +123,17 @@ RECIPE_PROVIDES:class-native = " \
> linux-kbuild-${KERNEL_NAME_PROVIDED}"
> # Use pseudo target to pull in the base variant of the recipe.
> # Will be auto-extended with -native by multiarch.bbclass.
> -RDEPENDS:class-native += "${BPN}-pseudo"
> +# Using DEPENDS, see below
> +DEPENDS:class-native += "${BPN}-pseudo"
Need to help me with remember my own code: Why do we also need this
build dependency? I thought you are only optimizing the kbuildtarget case.
>
> # -kbuildtarget: kbuild package for target, enforcing non-cross-build
> BUILD_PROFILES:class-kbuildtarget = "kbuild"
> RECIPE_PROVIDES:class-kbuildtarget = " \
> linux-headers-${KERNEL_NAME_PROVIDED} \
> linux-kbuild-${KERNEL_NAME_PROVIDED}"
> -RDEPENDS:class-kbuildtarget = "${BPN}"
> +# Using DEPENDS instead of RDEPENDS to ensure creation of kernel including
> +# pregenerated kernel config before target specific linux-kbuild package build
> +DEPENDS:class-kbuildtarget = "${BPN}"
> ISAR_CROSS_COMPILE:class-kbuildtarget = "0"
>
> # Make bitbake know we will be producing linux-image and linux-headers packages
Fine-tuning - this generally looks like a nice optimization now!
Jan
--
Siemens AG, Technology
Linux Expert Center
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH v2] linux-custom: Speedup build of target specific linux-kbuild package
2024-06-07 15:07 ` Jan Kiszka
@ 2024-06-07 15:37 ` Koch, Stefan
0 siblings, 0 replies; 3+ messages in thread
From: Koch, Stefan @ 2024-06-07 15:37 UTC (permalink / raw)
To: isar-users, Kiszka, Jan
Cc: Storm, Christian, cedric.hombourger, Adler, Michael, amikan,
Sudler, Simon, ubely, MOESSBAUER, Felix, Schmidt, Adriaan
On Fri, 2024-06-07 at 17:07 +0200, Jan Kiszka wrote:
> On 07.06.24 16:11, Stefan Koch wrote:
> > Avoids time expensive QEMU-emulated merging of kernel config by
> > using
> > the existing kernel config for the target specific linux-kbuild
> >
> > Using DEPENDS instead of RDEPENDS ensures creation of
> > kernel (including config) before build of target specific linux-
> > kbuild
> >
> > Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
> > ---
> > .../linux/files/debian/isar/configure.tmpl | 18
> > ++++++++++++++----
> > meta/recipes-kernel/linux/linux-custom.inc | 10 +++++++---
> > 2 files changed, 21 insertions(+), 7 deletions(-)
> >
> > diff --git a/meta/recipes-
> > kernel/linux/files/debian/isar/configure.tmpl b/meta/recipes-
> > kernel/linux/files/debian/isar/configure.tmpl
> > index 389c9a85..72eae448 100644
> > --- a/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
> > +++ b/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
> > @@ -10,10 +10,20 @@ do_configure() {
> > # Trace what we do here
> > set -x
> >
> > - # Process kernel config target and fragments
> > - ${MAKE} O=${KERNEL_BUILD_DIR} ${KERNEL_CONFIG_TARGET}
> > - ./scripts/kconfig/merge_config.sh -O ${KERNEL_BUILD_DIR}/ \
> > - ${KERNEL_BUILD_DIR}/.config ${KERNEL_CONFIG_FRAGMENTS}
> > + kernelconfig="$(find /boot -maxdepth 1 -name "config-${PV}*" -
> > print -quit)"
>
> Can we additionally limit this to the case were we expect the config
> there (kbuild && !kernel)? Just to reduce the risk that some "this
> should never be there" config is ever silently pulled by the base
> package build. Only if our dependency is installed, that rather
> generic
> config-<version> file is from there, or we failed on installing the
> dependency.
Build profile can be checked here as second condition, too.
find with "config-${PV}*" is used because some kernel specify "-rt18"
or "-isar" or something else that is not included in PV:
/boot/config-5.10.209-audis-rtp-rt18
/boot/config-6.6.11-isar
Some kernel do not specify an suffix to ${PV}. Maybe it depends on
LOCALVERISON from the kernel config? But as kernel config is not
available, we cannot find out this information...
find returns the first match. When using the old isar-buildchroot
instead of sbuild/schroot: Multiple configs could exist - if multiple
kernels are built. But if ${PV} is identical dpkg should have a
conflict with overwriting an existing config with the same version? (as
long this "-rt18"/"-isar" suffix is not renamed for that build.)
See below for dependency problem.
>
> > + if [ -e "${kernelconfig}" ]; then
> > + # Prefer existing kernel config
> > + # So, very expensive QEMU-emulated merge_config.sh
> > + # can be skipped for target specific linux-kbuild package
> > + mkdir -p ${KERNEL_BUILD_DIR}
> > + cp "${kernelconfig}" ${KERNEL_BUILD_DIR}/.config
> > + ${MAKE} O=${KERNEL_BUILD_DIR} olddefconfig
> > + else
> > + # Process kernel config target and fragments
> > + ${MAKE} O=${KERNEL_BUILD_DIR} ${KERNEL_CONFIG_TARGET}
> > + ./scripts/kconfig/merge_config.sh -O ${KERNEL_BUILD_DIR}/
> > \
> > + ${KERNEL_BUILD_DIR}/.config ${KERNEL_CONFIG_FRAGMENTS}
> > + fi
> >
> > # Stop tracing
> > set +x
> > diff --git a/meta/recipes-kernel/linux/linux-custom.inc
> > b/meta/recipes-kernel/linux/linux-custom.inc
> > index 647f09dd..dc354e13 100644
> > --- a/meta/recipes-kernel/linux/linux-custom.inc
> > +++ b/meta/recipes-kernel/linux/linux-custom.inc
> > @@ -25,6 +25,7 @@ KBUILD_DEPENDS ?= "build-essential:native, \
> > flex, \
> > git, \
> > kmod, \
> > + linux-image-
> > ${KERNEL_NAME_PROVIDED}:${DISTRO_ARCH} <kbuild !kernel>, \
That ensures installation of the dependency.
> > rsync,"
> >
> > KERNEL_DEBIAN_DEPENDS ?= "initramfs-tools | linux-initramfs-tool,
> > \
> > @@ -112,7 +113,7 @@ BUILD_PROFILES = "kernel kbuild"
> > BBCLASSEXTEND:append:cross-profile = " kbuildtarget"
> >
> > # When cross-profile is active:
> > -# build only kernel with the default variant of the recipe
> > +# build only kernel (including config) with the default variant of
> > the recipe
> > BUILD_PROFILES:cross-profile = "kernel"
> >
> > # -native: kbuild package for host
> > @@ -122,14 +123,17 @@ RECIPE_PROVIDES:class-native = " \
> > linux-kbuild-${KERNEL_NAME_PROVIDED}"
> > # Use pseudo target to pull in the base variant of the recipe.
> > # Will be auto-extended with -native by multiarch.bbclass.
> > -RDEPENDS:class-native += "${BPN}-pseudo"
> > +# Using DEPENDS, see below
> > +DEPENDS:class-native += "${BPN}-pseudo"
>
> Need to help me with remember my own code: Why do we also need this
> build dependency? I thought you are only optimizing the kbuildtarget
> case.
Kernel config is also read from linux-image for linux-kbuild-native.
When choosing to call merge_config.sh here again, this change would not
be needed. Then a different build profile must be added, because
<kernel !kbuild> won't match anymore.
>
> >
> > # -kbuildtarget: kbuild package for target, enforcing non-cross-
> > build
> > BUILD_PROFILES:class-kbuildtarget = "kbuild"
> > RECIPE_PROVIDES:class-kbuildtarget = " \
> > linux-headers-${KERNEL_NAME_PROVIDED} \
> > linux-kbuild-${KERNEL_NAME_PROVIDED}"
> > -RDEPENDS:class-kbuildtarget = "${BPN}"
> > +# Using DEPENDS instead of RDEPENDS to ensure creation of kernel
> > including
> > +# pregenerated kernel config before target specific linux-kbuild
> > package build
> > +DEPENDS:class-kbuildtarget = "${BPN}"
> > ISAR_CROSS_COMPILE:class-kbuildtarget = "0"
> >
> > # Make bitbake know we will be producing linux-image and linux-
> > headers packages
>
> Fine-tuning - this generally looks like a nice optimization now!
>
> Jan
>
--
Stefan Koch
Siemens AG
www.siemens.com
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2024-06-07 15:37 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-06-07 14:11 [PATCH v2] linux-custom: Speedup build of target specific linux-kbuild package Stefan Koch
2024-06-07 15:07 ` Jan Kiszka
2024-06-07 15:37 ` Koch, Stefan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox