Re: [PATCH v5 9/9] linux-custom: Move cfg fragments applying to debian/rules

[Henning Schild <henning.schild@siemens.com>]

Am Thu, 24 Feb 2022 17:49:17 +0300
schrieb Uladzimir Bely <ubely@ilbers.de>:

> In the email from Thursday, 24 February 2022 15:36:57 +03 user
> Henning Schild wrote:
> > Am Thu, 24 Feb 2022 12:09:00 +0300
> > 
> > schrieb Uladzimir Bely <ubely@ilbers.de>:  
> > > In the email from Wednesday, 23 February 2022 16:53:03 +03 user
> > > 
> > > Henning Schild wrote:  
> > > > I just ran into an issue with that not actually finding my
> > > > config in WORKDIR and now trying to go the KBUILD_DEFCONFIG way
> > > > 
> > > > Am Sun, 13 Feb 2022 08:41:11 +0100
> > > > 
> > > > schrieb Uladzimir Bely <ubely@ilbers.de>:  
> > > > > Prepare linux fragments applying in Debian way.
> > > > > 
> > > > > This allows to avoid pre-build chroot call used for merging
> > > > > kernel config file and custom config fragments. Also, shell
> > > > > exports are not used anymore in favor of template files.
> > > > > 
> > > > > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > > > > ---
> > > > > 
> > > > >  .../linux/files/debian/isar/configure.tmpl    | 19 +++++++
> > > > >  .../linux/files/debian/rules.tmpl             |  3 ++
> > > > >  meta/recipes-kernel/linux/linux-custom.inc    | 53
> > > > > 
> > > > > ++++++++++++------- 3 files changed, 55 insertions(+), 20
> > > > > deletions(-)
> > > > > 
> > > > >  create mode 100644
> > > > > 
> > > > > meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
> > > > > 
> > > > > diff --git
> > > > > a/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
> > > > > b/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
> > > > > new file mode 100644 index 00000000..83871baa --- /dev/null
> > > > > +++
> > > > > b/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
> > > > > @@ -0,0 +1,19 @@ +#!/bin/bash
> > > > > +# Copyright (c) Mentor Graphics, a Siemens business, 2019
> > > > > +# SPDX-License-Identifier: MIT
> > > > > +
> > > > > +# Load common stuff
> > > > > +. ${S}/debian/isar/common || exit ${?}
> > > > > +
> > > > > +do_configure() {
> > > > > +
> > > > > +    # Process kernel config target and fragments
> > > > > +    ${MAKE} O=${KERNEL_BUILD_DIR} ${KERNEL_CONFIG_TARGET} ||
> > > > > exit ${?}
> > > > > +    ./scripts/kconfig/merge_config.sh -O
> > > > > ${KERNEL_BUILD_DIR}/ \
> > > > > +    ${KERNEL_BUILD_DIR}/.config ${KERNEL_CONFIG_FRAGMENTS}
> > > > > +
> > > > > +    # Stop tracing
> > > > > +    set +x
> > > > > +}
> > > > > +
> > > > > +main configure ${*}
> > > > > diff --git a/meta/recipes-kernel/linux/files/debian/rules.tmpl
> > > > > b/meta/recipes-kernel/linux/files/debian/rules.tmpl index
> > > > > 05a26fe2..a1166287 100755 ---
> > > > > a/meta/recipes-kernel/linux/files/debian/rules.tmpl +++
> > > > > b/meta/recipes-kernel/linux/files/debian/rules.tmpl @@ -22,6
> > > > > +22,9 @@
> > > > > 
> > > > > isar_env=$(strip \ override_dh_auto_clean:
> > > > >  	$(isar_env) && bash $(deb_top_dir)/isar/clean
> > > > > 
> > > > > +override_dh_auto_configure:
> > > > > +	$(isar_env) && bash $(deb_top_dir)/isar/configure
> > > > > +
> > > > > 
> > > > >  override_dh_auto_build:
> > > > >  	$(isar_env) && bash $(deb_top_dir)/isar/build
> > > > > 
> > > > > diff --git a/meta/recipes-kernel/linux/linux-custom.inc
> > > > > b/meta/recipes-kernel/linux/linux-custom.inc index
> > > > > 59d42c84..ea1abf76 100644 ---
> > > > > a/meta/recipes-kernel/linux/linux-custom.inc +++
> > > > > b/meta/recipes-kernel/linux/linux-custom.inc @@ -52,6 +52,7 @@
> > > > > TEMPLATE_FILES += "                  \
> > > > > 
> > > > >      debian/isar/build.tmpl           \
> > > > >      debian/isar/clean.tmpl           \
> > > > >      debian/isar/common.tmpl          \
> > > > > 
> > > > > +    debian/isar/configure.tmpl       \
> > > > > 
> > > > >      debian/isar/install.tmpl         \
> > > > >      debian/isar/version.cfg.tmpl     \
> > > > >      debian/linux-image.postinst.tmpl \
> > > > > 
> > > > > @@ -71,6 +72,8 @@ TEMPLATE_VARS += "                \
> > > > > 
> > > > >      KERNEL_HEADERS_DEBIAN_DEPENDS \
> > > > >      LINUX_VERSION_EXTENSION       \
> > > > >      KERNEL_NAME_PROVIDED          \
> > > > > 
> > > > > +    KERNEL_CONFIG_TARGET          \
> > > > > +    KERNEL_CONFIG_FRAGMENTS       \
> > > > > 
> > > > >  "
> > > > >  
> > > > >  inherit dpkg
> > > > > 
> > > > > @@ -146,41 +149,51 @@ do_prepare_build_prepend() {
> > > > > 
> > > > >  # build directory for our "full" kernel build
> > > > >  KERNEL_BUILD_DIR = "build-full"
> > > > > 
> > > > > +def get_kernel_config_target(d):
> > > > > +    kernel_defconfig = d.getVar('KERNEL_DEFCONFIG', True)
> > > > > +
> > > > > +    config_target = kernel_defconfig
> > > > > +
> > > > > +    if kernel_defconfig:
> > > > > +        workdir=d.getVar('WORKDIR', True)  
> > > >   
> > > > > +        if os.path.isfile(workdir + "/" + kernel_defconfig):
> > > > >  
> > > > Doing that at parse-time when assigning the variable
> > > > KERNEL_CONFIG_TARGET can not work. Some task in the task chain
> > > > might need to create that file first ...
> > > > fetch/unpack/fiddle-with
> > > > 
> > > > Checking if the file exists at parse time seems very wrong. I
> > > > guess this patch needs to be reverted and rewritten and i guess
> > > > the test suite does not properly cover kernels custom defconfig
> > > >  
> > > 
> > > Yes, you are right, this moment is wrong.
> > > 
> > > I guess, it can be fixed by passing KERNEL_CONFIG_TARGET to
> > > 'debian/*' by direct appending from the recipe task (instead of
> > > template way).
> > > 
> > > I'll try prepare a solution soon.  
> > 
> > Cool,
> > 
> > i would even suggest a revert because that breaks kernel builds in
> > layers. I reverted just that one and my layer worked again.
> > 
> > I will look into writing a patch to the testsuite, covering the
> > olddefconfig path.
> > 
> > Henning
> >   
> 
> Please look at "linux-custom: Set KERNEL_CONFIG_TARGET at build time"
> patch that should solve the issue you've described.
> 
> While we have custom configs only for amd64 machines, but we have no
> in Isar amd64 machine with linux-mainline (or linux-cip) that define
> some custom config file in workdir, I just tested it on
> bullseye-amd64 with separate buildi linux-mainline by bitbake.
> 
> It seems now it works and properly detects and uses x86_64_defconfig
> in workdir, while the stuff is moved to dpkg_configure_kernel() which
> technically is a part of dpkg_runbuild.

I failed to write a patch to test the "olddefconfig" path based on
mainline and cip. These recipes for some reason are not affected, i
guess that python function got executed multiple times.
Probably once before the template stuff kicks in.

One more thing i found is that they use "x86_64_defconfig" as a file
name, which unfortunately also is a valid "make target" which would
just work with no ".config" and not use our custom config.

But that turned out to not be a problem.

I bet there is room for better testing but i currently do not have an
idea on how to do that.

Let me think about it and i might come around with a patch, for now it
seems good and quick turn arounds on layer testing and next fixing seem
to work well. There is only so much that upstream can and should test.

Henning

> > > > This is how one can rebuild a debian kernel
> > > > 
> > > > KBUILD_DEPENDS += ",linux-config-${PV}"
> > > > 
> > > > KERNEL_DEFCONFIG="${P}.conf"
> > > > 
> > > > do_take_debian_config() {
> > > > 
> > > >     xzcat
> > > > 
> > > > ${BUILDCHROOT_DIR}/usr/src/linux-config-${PV}/config.${DISTRO_ARCH}_none
> > > > _${
> > > > DISTRO_ARCH}.xz > ${S}/${KERNEL_DEFCONFIG} }
> > > > 
> > > > addtask take_debian_config after do_install_builddeps before
> > > > do_dpkg_build
> > > > 
> > > > Maybe that could be used for writing a kernel test recipe
> > > > actually going the "olddefconfig" way in case we do not yet
> > > > have one.
> > > > 
> > > > Henning
> > > >   
> > > > > +            config_target = "olddefconfig"
> > > > > +        else:
> > > > > +            config_target = "defconfig KBUILD_DEFCONFIG=" +
> > > > > kernel_defconfig
> > > > > +    else:
> > > > > +        config_target = "defconfig"
> > > > > +
> > > > > +    return config_target
> > > > > +
> > > > > +KERNEL_CONFIG_TARGET = "${@get_kernel_config_target(d)}"
> > > > > +
> > > > > +def get_kernel_config_fragments(d):
> > > > > +    src_frags = " ".join(config_fragments(d))
> > > > > +    out_frags = " ".join(map(lambda frag:
> > > > > 'debian/fragments/' + frag, config_fragments(d))) +
> > > > > +    linux_version_extension =
> > > > > d.getVar('LINUX_VERSION_EXTENSION', True)
> > > > > +    if linux_version_extension:
> > > > > +        out_frags += " debian/isar/version.cfg"
> > > > > +
> > > > > +    return out_frags
> > > > > +
> > > > > +KERNEL_CONFIG_FRAGMENTS =
> > > > > "${@get_kernel_config_fragments(d)}" +
> > > > > 
> > > > >  dpkg_configure_kernel() {
> > > > > 
> > > > > -	config_target="${KERNEL_DEFCONFIG}"
> > > > > 
> > > > >  	rm -rf ${S}/${KERNEL_BUILD_DIR} && mkdir -p
> > > > > 
> > > > > ${S}/${KERNEL_BUILD_DIR} if [ -n "${KERNEL_DEFCONFIG}" ]; then
> > > > > 
> > > > >  		if [ -e "${WORKDIR}/${KERNEL_DEFCONFIG}" ];
> > > > > then 
> > > > >  			cp ${WORKDIR}/${KERNEL_DEFCONFIG}
> > > > > 
> > > > > ${S}/${KERNEL_BUILD_DIR}/.config
> > > > > -			config_target="olddefconfig"
> > > > > -		else
> > > > > -			config_target="defconfig
> > > > > KBUILD_DEFCONFIG=${KERNEL_DEFCONFIG}" fi
> > > > > -	else
> > > > > -		config_target="defconfig"
> > > > > 
> > > > >  	fi
> > > > >  	
> > > > >  	# copy config fragments over to the kernel tree
> > > > >  	src_frags="${@ " ".join(config_fragments(d)) }"
> > > > > 
> > > > > -	out_frags="${@ " ".join(map(lambda frag:
> > > > > 'debian/fragments/'
> > > > > + frag, config_fragments(d))) }"
> > > > > -	if [ -n "${LINUX_VERSION_EXTENSION}" ]; then
> > > > > -		out_frags="${out_frags}
> > > > > debian/isar/version.cfg"
> > > > > -	fi
> > > > > 
> > > > >  	rm -rf ${S}/debian/fragments
> > > > >  	if [ -n "${src_frags}" ]; then
> > > > >  	
> > > > >  		mkdir -p ${S}/debian/fragments
> > > > >  		(cd ${WORKDIR} && cp ${src_frags}
> > > > > 
> > > > > ${S}/debian/fragments/) fi
> > > > > -
> > > > > -	sudo -E chroot --userspec=$(id -u):$(id -g)
> > > > > ${BUILDCHROOT_DIR} sh -c " \
> > > > > -		export ARCH=${KERNEL_ARCH} &&
> > > > > 
> > > > >           \
> > > > > 
> > > > > -		cd ${PP}/${PPS} &&
> > > > > 
> > > > >           \
> > > > > 
> > > > > -		make O=${KERNEL_BUILD_DIR} ${config_target}
> > > > > &&
> > > > > 
> > > > >           \
> > > > > 
> > > > > -		./scripts/kconfig/merge_config.sh
> > > > > 
> > > > >           \
> > > > > 
> > > > > -			-O ${KERNEL_BUILD_DIR}/
> > > > > 
> > > > >           \
> > > > > 
> > > > > -			${KERNEL_BUILD_DIR}/.config
> > > > > 
> > > > >           \
> > > > > 
> > > > > -			${out_frags}
> > > > > 
> > > > >           \
> > > > > 
> > > > > -        "
> > > > > 
> > > > >  }
> > > > >  
> > > > >  dpkg_runbuild_prepend() {  
> 
>