Re: [PATCH v5 9/9] linux-custom: Move cfg fragments applying to debian/rules

[Uladzimir Bely <ubely@ilbers.de>]

In the email from Thursday, 24 February 2022 15:36:57 +03 user Henning Schild 
wrote:
> Am Thu, 24 Feb 2022 12:09:00 +0300
> 
> schrieb Uladzimir Bely <ubely@ilbers.de>:
> > In the email from Wednesday, 23 February 2022 16:53:03 +03 user
> > 
> > Henning Schild wrote:
> > > I just ran into an issue with that not actually finding my config in
> > > WORKDIR and now trying to go the KBUILD_DEFCONFIG way
> > > 
> > > Am Sun, 13 Feb 2022 08:41:11 +0100
> > > 
> > > schrieb Uladzimir Bely <ubely@ilbers.de>:
> > > > Prepare linux fragments applying in Debian way.
> > > > 
> > > > This allows to avoid pre-build chroot call used for merging
> > > > kernel config file and custom config fragments. Also, shell
> > > > exports are not used anymore in favor of template files.
> > > > 
> > > > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > > > ---
> > > > 
> > > >  .../linux/files/debian/isar/configure.tmpl    | 19 +++++++
> > > >  .../linux/files/debian/rules.tmpl             |  3 ++
> > > >  meta/recipes-kernel/linux/linux-custom.inc    | 53
> > > > 
> > > > ++++++++++++------- 3 files changed, 55 insertions(+), 20
> > > > deletions(-)
> > > > 
> > > >  create mode 100644
> > > > 
> > > > meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
> > > > 
> > > > diff --git
> > > > a/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
> > > > b/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl new
> > > > file mode 100644 index 00000000..83871baa --- /dev/null
> > > > +++ b/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
> > > > @@ -0,0 +1,19 @@
> > > > +#!/bin/bash
> > > > +# Copyright (c) Mentor Graphics, a Siemens business, 2019
> > > > +# SPDX-License-Identifier: MIT
> > > > +
> > > > +# Load common stuff
> > > > +. ${S}/debian/isar/common || exit ${?}
> > > > +
> > > > +do_configure() {
> > > > +
> > > > +    # Process kernel config target and fragments
> > > > +    ${MAKE} O=${KERNEL_BUILD_DIR} ${KERNEL_CONFIG_TARGET} || exit
> > > > ${?}
> > > > +    ./scripts/kconfig/merge_config.sh -O ${KERNEL_BUILD_DIR}/ \
> > > > +    ${KERNEL_BUILD_DIR}/.config ${KERNEL_CONFIG_FRAGMENTS}
> > > > +
> > > > +    # Stop tracing
> > > > +    set +x
> > > > +}
> > > > +
> > > > +main configure ${*}
> > > > diff --git a/meta/recipes-kernel/linux/files/debian/rules.tmpl
> > > > b/meta/recipes-kernel/linux/files/debian/rules.tmpl index
> > > > 05a26fe2..a1166287 100755 ---
> > > > a/meta/recipes-kernel/linux/files/debian/rules.tmpl +++
> > > > b/meta/recipes-kernel/linux/files/debian/rules.tmpl @@ -22,6
> > > > +22,9 @@
> > > > 
> > > > isar_env=$(strip \ override_dh_auto_clean:
> > > >  	$(isar_env) && bash $(deb_top_dir)/isar/clean
> > > > 
> > > > +override_dh_auto_configure:
> > > > +	$(isar_env) && bash $(deb_top_dir)/isar/configure
> > > > +
> > > > 
> > > >  override_dh_auto_build:
> > > >  	$(isar_env) && bash $(deb_top_dir)/isar/build
> > > > 
> > > > diff --git a/meta/recipes-kernel/linux/linux-custom.inc
> > > > b/meta/recipes-kernel/linux/linux-custom.inc index
> > > > 59d42c84..ea1abf76 100644 ---
> > > > a/meta/recipes-kernel/linux/linux-custom.inc +++
> > > > b/meta/recipes-kernel/linux/linux-custom.inc @@ -52,6 +52,7 @@
> > > > TEMPLATE_FILES += "                  \
> > > > 
> > > >      debian/isar/build.tmpl           \
> > > >      debian/isar/clean.tmpl           \
> > > >      debian/isar/common.tmpl          \
> > > > 
> > > > +    debian/isar/configure.tmpl       \
> > > > 
> > > >      debian/isar/install.tmpl         \
> > > >      debian/isar/version.cfg.tmpl     \
> > > >      debian/linux-image.postinst.tmpl \
> > > > 
> > > > @@ -71,6 +72,8 @@ TEMPLATE_VARS += "                \
> > > > 
> > > >      KERNEL_HEADERS_DEBIAN_DEPENDS \
> > > >      LINUX_VERSION_EXTENSION       \
> > > >      KERNEL_NAME_PROVIDED          \
> > > > 
> > > > +    KERNEL_CONFIG_TARGET          \
> > > > +    KERNEL_CONFIG_FRAGMENTS       \
> > > > 
> > > >  "
> > > >  
> > > >  inherit dpkg
> > > > 
> > > > @@ -146,41 +149,51 @@ do_prepare_build_prepend() {
> > > > 
> > > >  # build directory for our "full" kernel build
> > > >  KERNEL_BUILD_DIR = "build-full"
> > > > 
> > > > +def get_kernel_config_target(d):
> > > > +    kernel_defconfig = d.getVar('KERNEL_DEFCONFIG', True)
> > > > +
> > > > +    config_target = kernel_defconfig
> > > > +
> > > > +    if kernel_defconfig:
> > > > +        workdir=d.getVar('WORKDIR', True)
> > > 
> > > > +        if os.path.isfile(workdir + "/" + kernel_defconfig):
> > > Doing that at parse-time when assigning the variable
> > > KERNEL_CONFIG_TARGET can not work. Some task in the task chain might
> > > need to create that file first ... fetch/unpack/fiddle-with
> > > 
> > > Checking if the file exists at parse time seems very wrong. I guess
> > > this patch needs to be reverted and rewritten and i guess the test
> > > suite does not properly cover kernels custom defconfig
> > 
> > Yes, you are right, this moment is wrong.
> > 
> > I guess, it can be fixed by passing KERNEL_CONFIG_TARGET to
> > 'debian/*' by direct appending from the recipe task (instead of
> > template way).
> > 
> > I'll try prepare a solution soon.
> 
> Cool,
> 
> i would even suggest a revert because that breaks kernel builds in
> layers. I reverted just that one and my layer worked again.
> 
> I will look into writing a patch to the testsuite, covering the
> olddefconfig path.
> 
> Henning
> 

Please look at "linux-custom: Set KERNEL_CONFIG_TARGET at build time" patch 
that should solve the issue you've described.

While we have custom configs only for amd64 machines, but we have no in Isar 
amd64 machine with linux-mainline (or linux-cip) that define some custom 
config file in workdir, I just tested it on bullseye-amd64 with separate 
buildi linux-mainline by bitbake.

It seems now it works and properly detects and uses x86_64_defconfig in 
workdir, while the stuff is moved to dpkg_configure_kernel() which technically 
is a part of dpkg_runbuild.

> > > This is how one can rebuild a debian kernel
> > > 
> > > KBUILD_DEPENDS += ",linux-config-${PV}"
> > > 
> > > KERNEL_DEFCONFIG="${P}.conf"
> > > 
> > > do_take_debian_config() {
> > > 
> > >     xzcat
> > > 
> > > ${BUILDCHROOT_DIR}/usr/src/linux-config-${PV}/config.${DISTRO_ARCH}_none
> > > _${
> > > DISTRO_ARCH}.xz > ${S}/${KERNEL_DEFCONFIG} }
> > > 
> > > addtask take_debian_config after do_install_builddeps before
> > > do_dpkg_build
> > > 
> > > Maybe that could be used for writing a kernel test recipe actually
> > > going the "olddefconfig" way in case we do not yet have one.
> > > 
> > > Henning
> > > 
> > > > +            config_target = "olddefconfig"
> > > > +        else:
> > > > +            config_target = "defconfig KBUILD_DEFCONFIG=" +
> > > > kernel_defconfig
> > > > +    else:
> > > > +        config_target = "defconfig"
> > > > +
> > > > +    return config_target
> > > > +
> > > > +KERNEL_CONFIG_TARGET = "${@get_kernel_config_target(d)}"
> > > > +
> > > > +def get_kernel_config_fragments(d):
> > > > +    src_frags = " ".join(config_fragments(d))
> > > > +    out_frags = " ".join(map(lambda frag: 'debian/fragments/' +
> > > > frag, config_fragments(d))) +
> > > > +    linux_version_extension = d.getVar('LINUX_VERSION_EXTENSION',
> > > > True)
> > > > +    if linux_version_extension:
> > > > +        out_frags += " debian/isar/version.cfg"
> > > > +
> > > > +    return out_frags
> > > > +
> > > > +KERNEL_CONFIG_FRAGMENTS = "${@get_kernel_config_fragments(d)}"
> > > > +
> > > > 
> > > >  dpkg_configure_kernel() {
> > > > 
> > > > -	config_target="${KERNEL_DEFCONFIG}"
> > > > 
> > > >  	rm -rf ${S}/${KERNEL_BUILD_DIR} && mkdir -p
> > > > 
> > > > ${S}/${KERNEL_BUILD_DIR} if [ -n "${KERNEL_DEFCONFIG}" ]; then
> > > > 
> > > >  		if [ -e "${WORKDIR}/${KERNEL_DEFCONFIG}" ]; then
> > > >  		
> > > >  			cp ${WORKDIR}/${KERNEL_DEFCONFIG}
> > > > 
> > > > ${S}/${KERNEL_BUILD_DIR}/.config
> > > > -			config_target="olddefconfig"
> > > > -		else
> > > > -			config_target="defconfig
> > > > KBUILD_DEFCONFIG=${KERNEL_DEFCONFIG}" fi
> > > > -	else
> > > > -		config_target="defconfig"
> > > > 
> > > >  	fi
> > > >  	
> > > >  	# copy config fragments over to the kernel tree
> > > >  	src_frags="${@ " ".join(config_fragments(d)) }"
> > > > 
> > > > -	out_frags="${@ " ".join(map(lambda frag:
> > > > 'debian/fragments/'
> > > > + frag, config_fragments(d))) }"
> > > > -	if [ -n "${LINUX_VERSION_EXTENSION}" ]; then
> > > > -		out_frags="${out_frags} debian/isar/version.cfg"
> > > > -	fi
> > > > 
> > > >  	rm -rf ${S}/debian/fragments
> > > >  	if [ -n "${src_frags}" ]; then
> > > >  	
> > > >  		mkdir -p ${S}/debian/fragments
> > > >  		(cd ${WORKDIR} && cp ${src_frags}
> > > > 
> > > > ${S}/debian/fragments/) fi
> > > > -
> > > > -	sudo -E chroot --userspec=$(id -u):$(id -g)
> > > > ${BUILDCHROOT_DIR} sh -c " \
> > > > -		export ARCH=${KERNEL_ARCH} &&
> > > > 
> > > >           \
> > > > 
> > > > -		cd ${PP}/${PPS} &&
> > > > 
> > > >           \
> > > > 
> > > > -		make O=${KERNEL_BUILD_DIR} ${config_target} &&
> > > > 
> > > >           \
> > > > 
> > > > -		./scripts/kconfig/merge_config.sh
> > > > 
> > > >           \
> > > > 
> > > > -			-O ${KERNEL_BUILD_DIR}/
> > > > 
> > > >           \
> > > > 
> > > > -			${KERNEL_BUILD_DIR}/.config
> > > > 
> > > >           \
> > > > 
> > > > -			${out_frags}
> > > > 
> > > >           \
> > > > 
> > > > -        "
> > > > 
> > > >  }
> > > >  
> > > >  dpkg_runbuild_prepend() {


-- 
Uladzimir Bely