[PATCH v3 0/5] linux-custom: Split up binaries from kernel headers to kbuild packages

[PATCH v3 0/5] linux-custom: Split up binaries from kernel headers to kbuild packages

[Koch, Stefan]

Hi

This updated v3 patchset implement suggestions from reviewed v2 patchset.
It's now a set of five patches:
- linux-custom: Split up binaries from kernel headers to kbuild package
- sbuild: Support overriding configured schroot dir
- dpkg: Allow overriding build and host parameters for sbuild and schroot
- linux-custom: Provide host and target specific kernel kbuild packages
- docs: Update custom_kernel docs for split up of kernel scripts and tools

The main use-case was to swap out the binaries
from the kernel headers into kernel kbuild package.
This is introduced by the first commit
"Split up binaries from kernel headers to kbuild package"

The fourth commit "Provide host and target specific kernel kbuild packages"
introduces that the binaries could be swapped out into host and target
specific kernel kbuild packages.

The main development goals were these:

1. Solve already known isar custom kernel
limitations from doc/custom_kernel.inc
- kernel headers package does not support both native
  and cross compilation of kernel modules when cross built

2. Honor recommendations for future from doc/custom_kernel.inc
- Generate kernel headers packages for both host and target
  when using cross build

3. Add extensions known from debian kernel packages structure
- Generate a kernel headers package without binaries
- Create specific kernel kbuild packages that
  will ship the "scripts" and "tools" binaries
- Use symlinks to point to the "scripts" and "tools" binaries

4. Be user friendly
- Avoid redundant configuration of kernel source definitions with user
  actions to enable kbuild package generation
- Use already known way to include linux-custom.inc in just one
  own bitbake recipe that provides the kernel source definitions
- Keep known user behavior for existing build configurations:
  just update isar, set EXTENDED_KBUILD_MODE to "1" request
  "-compat" (and "-native") bitbake targets and kbuild
  packages for target and host will be created automatically

Best regards

Stefan

Stefan Koch (5):
  linux-custom: Split up binaries from kernel headers to kbuild package
  sbuild: Support overriding configured schroot dir
  dpkg: Allow overriding build and host parameters for sbuild and
    schroot
  linux-custom: Provide host and target specific kernel kbuild packages
  docs: Update custom_kernel docs for split up of kernel scripts and
    tools

 doc/custom_kernel.md                          | 17 +++--
 meta/classes/dpkg-base.bbclass                | 19 ++++-
 meta/classes/dpkg.bbclass                     |  2 +-
 meta/classes/sbuild.bbclass                   |  9 ++-
 .../linux/files/debian/control.tmpl           | 25 ++++++-
 .../linux/files/debian/isar/build.tmpl        | 13 +++-
 .../linux/files/debian/isar/common.tmpl       | 14 +++-
 .../linux/files/debian/isar/configure.tmpl    | 13 +++-
 .../linux/files/debian/isar/install.tmpl      | 73 +++++++++++++-----
 .../linux/files/debian/rules.tmpl             |  2 +-
 meta/recipes-kernel/linux/linux-custom.inc    | 75 +++++++++++++++----
 11 files changed, 211 insertions(+), 51 deletions(-)

-- 
2.39.2

[PATCH v3 1/5] linux-custom: Split up binaries from kernel headers to kbuild package

[Koch, Stefan]

This patch does introduce specific kernel kbuild packages that
ship the "scripts" and "tools" binaries.
The kernel headers fulfill this using symlinks to point
to the "scripts" and "tools" of the kernel kbuild package.

This is already known from debian kernel packages structure:
- Generate a kernel headers package without binaries
- Create kernel kbuild packages that
  ship the "scripts" and "tools" binaries
- Using symlinks to point to the "scripts"
  and "tools" binaries

Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
---
 .../linux/files/debian/control.tmpl           |  9 +++-
 .../linux/files/debian/isar/common.tmpl       |  2 +
 .../linux/files/debian/isar/install.tmpl      | 51 +++++++++++++++----
 meta/recipes-kernel/linux/linux-custom.inc    |  4 ++
 4 files changed, 54 insertions(+), 12 deletions(-)

diff --git a/meta/recipes-kernel/linux/files/debian/control.tmpl b/meta/recipes-kernel/linux/files/debian/control.tmpl
index dd0b6245..b19ca2c9 100644
--- a/meta/recipes-kernel/linux/files/debian/control.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/control.tmpl
@@ -26,7 +26,7 @@ Section: devel
 Provides: linux-kernel-headers
 Architecture: any
 Description: Linux support headers for userspace development
- This package provides userspaces headers from the Linux kernel.  These headers
+ This package provides userspaces headers from the Linux kernel. These headers
  are used by the installed headers for GNU glibc and other system libraries.
 
 Package: linux-image-${KERNEL_NAME_PROVIDED}-dbg
@@ -35,3 +35,10 @@ Architecture: any
 Description: Linux kernel debugging symbols for @KR@
  This package will come in handy if you need to debug the kernel. It provides
  all the necessary debug symbols for the kernel and its modules.
+
+Package: linux-kbuild-${KERNEL_NAME_PROVIDED}
+Architecture: any
+Depends: ${perl:Depends}, ${shlib:Depends}
+Description: ${KERNEL_NAME_PROVIDED} Linux kbuild scripts and tools for @KR@
+ This package provides kernel kbuild scripts and tools for @KR@
+ This is useful for people who need to build external modules
diff --git a/meta/recipes-kernel/linux/files/debian/isar/common.tmpl b/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
index 52ebebb0..f4c0519f 100644
--- a/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
@@ -9,6 +9,7 @@ ARCH=${KERNEL_ARCH}
 KERNEL_PKG_IMAGE=linux-image-${KERNEL_NAME_PROVIDED}
 KERNEL_PKG_KERN_HEADERS=linux-headers-${KERNEL_NAME_PROVIDED}
 KERNEL_PKG_LIBC_HEADERS=linux-libc-dev
+KERNEL_PKG_KERN_KBUILD=linux-kbuild-${KERNEL_NAME_PROVIDED}
 
 # Constants
 KCONF=.config
@@ -19,6 +20,7 @@ deb_img_dir=${deb_top_dir}/${KERNEL_PKG_IMAGE}
 deb_dbg_dir=${deb_img_dir}-dbg
 deb_kern_hdr_dir=${deb_top_dir}/${KERNEL_PKG_KERN_HEADERS}
 deb_libc_hdr_dir=${deb_top_dir}/${KERNEL_PKG_LIBC_HEADERS}
+deb_kern_kbuild_dir=${deb_top_dir}/${KERNEL_PKG_KERN_KBUILD}
 
 # Array of packages to be generated
 declare -A kern_pkgs
diff --git a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
index 8a604e45..236b67c4 100644
--- a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
@@ -33,6 +33,7 @@ do_install() {
     # Trace what we do here
     set -x
 
+    # Run the install steps
     install_image
     if [ "${ARCH}" != "um" ]; then
         install_config
@@ -43,6 +44,10 @@ do_install() {
     install_kmods
     install_headers
 
+    # Cleanup and install kernel scripts and tools
+    rm -rf ${deb_kern_kbuild_dir}
+    install_kbuild ${deb_kern_kbuild_dir}
+
     # Stop tracing
     set +x
 }
@@ -168,21 +173,15 @@ kernel_headers() {
     mkdir -p ${destdir}
     mkdir -p ${deb_kern_hdr_dir}/lib/modules/${krel}
 
-    (cd ${S}; find . -name 'Makefile*' -o -name 'Kconfig*' -o -name '*.pl') >>${src_hdr_files}
-    (cd ${S}; find arch/*/include include scripts -type f -o -type l) >>${src_hdr_files}
+    (cd ${S}; find . -not -path './scripts/*' -a -not -path './tools/*' -a \( -name 'Makefile*' -o -name 'Kconfig*' -o -name '*.pl' \)) >>${src_hdr_files}
+    (cd ${S}; find arch/*/include include -type f -o -type l) >>${src_hdr_files}
     (cd ${S}; find arch/${ARCH} -name module.lds -o -name Kbuild.platforms -o -name Platform) >>${src_hdr_files}
     (cd ${S}; find $(find arch/${ARCH} -name include -o -name scripts -type d) -type f) >>${src_hdr_files}
 
     if [ -n "${CONFIG_MODULES}" ]; then
         echo Module.symvers >> ${obj_hdr_files}
     fi
-    (cd ${O}; find arch/${ARCH}/include include scripts -type f) >>${obj_hdr_files}
-    if [ -n "${CONFIG_STACK_VALIDATION}" -o -n "${CONFIG_HAVE_OBJTOOL}" ]; then
-        (cd ${O}; find tools/objtool -type f -executable) >>${obj_hdr_files}
-    fi
-    if [ -n "${CONFIG_GCC_PLUGINS}" ]; then
-        (cd ${O}; find scripts/gcc-plugins -name *.so -o -name gcc-common.h) >>${obj_hdr_files}
-    fi
+    (cd ${O}; find arch/${ARCH}/include include -type f) >>${obj_hdr_files}
 
     # deploy files that were matched above
     tar -C ${S} -cf - -T - <${src_hdr_files} | tar -C ${destdir} -xf -
@@ -191,8 +190,11 @@ kernel_headers() {
     # add the kernel config
     cp ${O}/${KCONF} ${destdir}/.config
 
-    # handle kernel development tools
-    kernel_tools
+    # add symlink to scripts and tools directories
+    ln -sf ../../lib/linux-kbuild-${krel}/scripts ${destdir}/scripts
+    if [ -n "${CONFIG_STACK_VALIDATION}" -o -n "${CONFIG_HAVE_OBJTOOL}" ]; then
+        ln -sf ../../lib/linux-kbuild-${krel}/tools ${destdir}/tools
+    fi
 
     # create symlinks
     ln -sf /${kernel_headers_dir} ${deb_kern_hdr_dir}/lib/modules/${krel}/build
@@ -206,4 +208,31 @@ install_headers() {
     kernel_headers
 }
 
+install_kbuild() {
+    kernel_kbuild_dir=usr/lib/linux-kbuild-${krel}
+    destdir=${1}/${kernel_kbuild_dir}
+    src_kbuild_files=$(mktemp)
+    obj_kbuild_files=$(mktemp)
+
+    mkdir -p ${destdir}
+
+    (cd ${S}; find . -path './scripts/*' -a -path './tools/*' -a \( -name 'Makefile*' -o -name 'Kconfig*' -o -name '*.pl' \)) >>${src_kbuild_files}
+    (cd ${S}; find scripts -type f -o -type l) >>${src_kbuild_files}
+
+    (cd ${O}; find scripts -type f) >>${obj_kbuild_files}
+    if [ -n "${CONFIG_STACK_VALIDATION}" -o -n "${CONFIG_HAVE_OBJTOOL}" ]; then
+        (cd ${O}; find tools/objtool -type f -executable) >>${obj_kbuild_files}
+    fi
+    if [ -n "${CONFIG_GCC_PLUGINS}" ]; then
+        (cd ${O}; find scripts/gcc-plugins -name *.so -o -name gcc-common.h) >>${obj_kbuild_files}
+    fi
+
+    # deploy files that were matched above
+    tar -C ${S} -cf - -T - <${src_kbuild_files} | tar -C ${destdir} -xf -
+    tar -C ${O} -cf - -T - <${obj_kbuild_files} | tar -C ${destdir} -xf -
+
+    # handle kernel development tools
+    kernel_tools
+}
+
 main install ${*}
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 8b367a2f..94ed84a2 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -100,6 +100,10 @@ python() {
     kernel_name = d.getVar("KERNEL_NAME_PROVIDED")
     d.setVar('PROVIDES', 'linux-image-' + kernel_name + ' ' + \
                          'linux-headers-' + kernel_name)
+    headers_depends = "linux-kbuild-" + kernel_name
+
+    # Set dependency for kernel headers
+    d.appendVar("KERNEL_HEADERS_DEBIAN_DEPENDS", headers_depends)
 }
 
 def get_kernel_arch(d):
-- 
2.39.2

[PATCH v3 2/5] sbuild: Support overriding configured schroot dir

[Koch, Stefan]

This brings support to specify an other than the default schroot dir
as argument when creating the schroot configs.

Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
---
 meta/classes/sbuild.bbclass | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 995f7a54..3c73edd9 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -15,14 +15,19 @@ SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
 SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
 
 schroot_create_configs() {
+    schroot_dir="${SCHROOT_DIR}"
+    if [ -n "${1}" ]; then
+        schroot_dir="${1}"
+    fi
+
     mkdir -p "${TMPDIR}/schroot-overlay"
-    sudo -s <<'EOSUDO'
+    schroot_dir="${schroot_dir}" sudo --preserve-env=schroot_dir -s <<'EOSUDO'
         set -e
 
         cat << EOF > "${SCHROOT_CONF_FILE}"
 [${SBUILD_CHROOT}]
 type=directory
-directory=${SCHROOT_DIR}
+directory=${schroot_dir}
 profile=${SBUILD_CHROOT}
 users=${SCHROOT_USER}
 groups=root,sbuild
-- 
2.39.2

[PATCH v3 3/5] dpkg: Allow overriding build and host parameters for sbuild and schroot

[Koch, Stefan]

This functionality is initially used for custom linux builds using the
"-compat" and "-native" multiarch bitbake targets to run additional
target or host specific builds for kbuild scripts and tools.

Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 19 ++++++++++++++++++-
 meta/classes/dpkg.bbclass      |  2 +-
 2 files changed, 19 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 7b02f378..eaeb0e8b 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -208,7 +208,7 @@ dpkg_runbuild() {
 
 def isar_deb_build_profiles(d):
     deb_build_profiles = d.getVar('DEB_BUILD_PROFILES')
-    if d.getVar('ISAR_CROSS_COMPILE') == "1":
+    if d.getVar('ISAR_CROSS_COMPILE') == "1" and d.getVar('ISAR_SKIP_CROSS_PROFILE') != "1":
         deb_build_profiles += ' cross'
     return deb_build_profiles.strip()
 
@@ -223,6 +223,23 @@ def isar_export_build_settings(d):
     os.environ['DEB_BUILD_PROFILES'] = isar_deb_build_profiles(d)
 
 python do_dpkg_build() {
+    schroot_dir_override = d.getVar('SCHROOT_DIR_OVERRIDE', True)
+    sbuild_build_override = d.getVar('SBUILD_BUILD_OVERRIDE', True)
+    sbuild_host_override = d.getVar('SBUILD_HOST_OVERRIDE', True)
+
+    if schroot_dir_override:
+        d.setVar('SCHROOT_DIR', schroot_dir_override)
+
+    if sbuild_build_override:
+        d.setVar('SBUILD_BUILD', sbuild_build_override)
+    else:
+        d.setVar('SBUILD_BUILD', d.getVar('BUILD_HOST_ARCH', True))
+
+    if sbuild_host_override:
+        d.setVar('SBUILD_HOST', sbuild_host_override)
+    else:
+        d.setVar('SBUILD_HOST', d.getVar('PACKAGE_ARCH', True))
+
     bb.build.exec_func('schroot_create_configs', d)
     try:
         bb.build.exec_func("dpkg_runbuild", d)
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index c596adbf..7d659741 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -102,7 +102,7 @@ dpkg_runbuild() {
     DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -print)
 
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
-        --host=${PACKAGE_ARCH} --build=${BUILD_HOST_ARCH} ${profiles} \
+        --host=${SBUILD_HOST} --build=${SBUILD_BUILD} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \
         --bd-uninstallable-explainer=apt \
         --no-apt-update \
-- 
2.39.2

[PATCH v3 4/5] linux-custom: Provide host and target specific kernel kbuild packages

[Koch, Stefan]

When using a cross build this patch does introduce optionally
host and target specific kernel kbuild packages that
ship the "scripts" and "tools" binaries.

By default at cross builds a kbuild-cross package will be
generated that provides the host specific kbuild package.

When setting EXTENDED_KBUILD_MODE to "1" the "-compat" and "-native"
multiarch bitbake targets are useable to run additional target or host
specific builds for kbuild scripts and tools.

Using the "-compat" bitbake target enables the build of
a target specific kbuild package at cross builds.

Using the "-native" bitbake target enables the build of
a host specific kbuild package at cross builds.

This solves this from doc/custom_kernel.inc:
- The kernel headers package has not supported both native
  and cross compilation of kernel modules when itself was cross built
- Future roadmap: Generate kernel headers package for both host
  and target when using a cross build

Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
---
 .../linux/files/debian/control.tmpl           | 16 ++++-
 .../linux/files/debian/isar/build.tmpl        | 13 +++-
 .../linux/files/debian/isar/common.tmpl       | 12 +++-
 .../linux/files/debian/isar/configure.tmpl    | 13 +++-
 .../linux/files/debian/isar/install.tmpl      | 26 ++++---
 .../linux/files/debian/rules.tmpl             |  2 +-
 meta/recipes-kernel/linux/linux-custom.inc    | 71 +++++++++++++++----
 7 files changed, 123 insertions(+), 30 deletions(-)

diff --git a/meta/recipes-kernel/linux/files/debian/control.tmpl b/meta/recipes-kernel/linux/files/debian/control.tmpl
index b19ca2c9..c012048b 100644
--- a/meta/recipes-kernel/linux/files/debian/control.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/control.tmpl
@@ -6,6 +6,7 @@ Build-Depends: bc, kmod, cpio, ${KBUILD_DEPENDS}
 Homepage: http://www.kernel.org/
 
 Package: linux-image-${KERNEL_NAME_PROVIDED}
+Build-Profiles: <!targetbuild !hostbuild>
 Architecture: any
 Depends: ${KERNEL_DEBIAN_DEPENDS}
 Description: ${KERNEL_NAME_PROVIDED} Linux kernel, version @KR@
@@ -13,6 +14,7 @@ Description: ${KERNEL_NAME_PROVIDED} Linux kernel, version @KR@
  files, version: @KR@.
 
 Package: linux-headers-${KERNEL_NAME_PROVIDED}
+Build-Profiles: <!targetbuild !hostbuild>
 Architecture: any
 Depends: ${KERNEL_HEADERS_DEBIAN_DEPENDS}, ${perl:Depends}, ${shlib:Depends}
 Description: ${KERNEL_NAME_PROVIDED} Linux kernel headers for @KR@
@@ -21,7 +23,7 @@ Description: ${KERNEL_NAME_PROVIDED} Linux kernel headers for @KR@
  This is useful for people who need to build external modules
 
 Package: linux-libc-dev
-Build-Profiles: <!nolibcdev>
+Build-Profiles: <!nolibcdev !targetbuild !hostbuild>
 Section: devel
 Provides: linux-kernel-headers
 Architecture: any
@@ -30,6 +32,7 @@ Description: Linux support headers for userspace development
  are used by the installed headers for GNU glibc and other system libraries.
 
 Package: linux-image-${KERNEL_NAME_PROVIDED}-dbg
+Build-Profiles: <!targetbuild !hostbuild>
 Section: debug
 Architecture: any
 Description: Linux kernel debugging symbols for @KR@
@@ -37,8 +40,19 @@ Description: Linux kernel debugging symbols for @KR@
  all the necessary debug symbols for the kernel and its modules.
 
 Package: linux-kbuild-${KERNEL_NAME_PROVIDED}
+Build-Profiles: <targetbuild> <hostbuild> <defaultkbuild>
 Architecture: any
 Depends: ${perl:Depends}, ${shlib:Depends}
 Description: ${KERNEL_NAME_PROVIDED} Linux kbuild scripts and tools for @KR@
  This package provides kernel kbuild scripts and tools for @KR@
  This is useful for people who need to build external modules
+
+Package: linux-kbuild-${KERNEL_NAME_PROVIDED}-cross
+Build-Profiles: <!targetbuild !hostbuild crosskbuild>
+Architecture: any
+Depends: ${KERNEL_HEADERS_DEBIAN_DEPENDS}, ${perl:Depends}, ${shlib:Depends}
+Description: ${KERNEL_NAME_PROVIDED} Linux kbuild scripts and tools for @KR@
+ This package provides kernel kbuild scripts and tools
+ as ${HOST_ARCH} cross binaries for @KR@
+ This is useful for those who need to cross build
+ external modules using ISAR's sbuild-chroot-host
diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
index 906dc580..7e7cab00 100644
--- a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
@@ -22,7 +22,18 @@ do_build() {
     sed -i "s/@KR@/${KR}/g" ${S}/debian/control ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.*
 
     # Build the Linux kernel
-    ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}"
+    if echo "${DEB_BUILD_PROFILES}" | grep -q -e "targetbuild" -e "hostbuild"; then # Build kernel scripts and tools
+        cat ${KERNEL_BUILD_DIR}/.config | grep RECORDM
+        ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" scripts
+        if [ -d "tools/objtool" ]; then
+            ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" tools/objtool || true
+        fi
+        if grep -q "CONFIG_MODULES=y" ${KERNEL_BUILD_DIR}/.config; then
+            ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" modules_prepare
+        fi
+    else # Build the Linux kernel
+        ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}"
+    fi
 
     # Stop tracing
     set +x
diff --git a/meta/recipes-kernel/linux/files/debian/isar/common.tmpl b/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
index f4c0519f..65fa1fa4 100644
--- a/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
@@ -5,12 +5,22 @@
 set -e
 
 # Isar settings
-ARCH=${KERNEL_ARCH}
 KERNEL_PKG_IMAGE=linux-image-${KERNEL_NAME_PROVIDED}
 KERNEL_PKG_KERN_HEADERS=linux-headers-${KERNEL_NAME_PROVIDED}
 KERNEL_PKG_LIBC_HEADERS=linux-libc-dev
 KERNEL_PKG_KERN_KBUILD=linux-kbuild-${KERNEL_NAME_PROVIDED}
 
+if echo "${DEB_BUILD_PROFILES}" | grep -q "hostbuild"; then
+	# Force creating kernel kbuild debian package with valid host arch
+	# Use a cross build to comply with arch specific kernel defconfigs
+	# scripts and tools are always created in host arch
+	ARCH=${KERNEL_ARCH}
+	eval $(dpkg-architecture -f -A ${DISTRO_ARCH})
+	CROSS_COMPILE=${DEB_TARGET_GNU_TYPE}-
+else
+	ARCH=${KERNEL_ARCH}
+fi
+
 # Constants
 KCONF=.config
 
diff --git a/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl b/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
index 389c9a85..900d5cd8 100644
--- a/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/isar/configure.tmpl
@@ -11,9 +11,16 @@ do_configure() {
     set -x
 
     # Process kernel config target and fragments
-    ${MAKE} O=${KERNEL_BUILD_DIR} ${KERNEL_CONFIG_TARGET}
-    ./scripts/kconfig/merge_config.sh -O ${KERNEL_BUILD_DIR}/ \
-        ${KERNEL_BUILD_DIR}/.config ${KERNEL_CONFIG_FRAGMENTS}
+    if [ -e ${PP}/kernelconfig ]; then
+        mkdir -p ${KERNEL_BUILD_DIR}
+        cp ${PP}/kernelconfig ${KERNEL_BUILD_DIR}/.config
+        ${MAKE} O=${KERNEL_BUILD_DIR} olddefconfig
+    else
+        ${MAKE} O=${KERNEL_BUILD_DIR} ${KERNEL_CONFIG_TARGET}
+        ./scripts/kconfig/merge_config.sh -O ${KERNEL_BUILD_DIR}/ \
+            ${KERNEL_BUILD_DIR}/.config ${KERNEL_CONFIG_FRAGMENTS}
+        cp ${KERNEL_BUILD_DIR}/.config ${PP}/kernelconfig
+    fi
 
     # Stop tracing
     set +x
diff --git a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
index 236b67c4..08b8a9a5 100644
--- a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
@@ -34,18 +34,24 @@ do_install() {
     set -x
 
     # Run the install steps
-    install_image
-    if [ "${ARCH}" != "um" ]; then
-        install_config
-        install_map
+    if ! echo "${DEB_BUILD_PROFILES}" | grep -q -e "targetbuild" -e "hostbuild"; then
+        install_image
+        if [ "${ARCH}" != "um" ]; then
+            install_config
+            install_map
+        fi
+        install_hooks
+        install_dtbs
+        install_kmods
+        install_headers
     fi
-    install_hooks
-    install_dtbs
-    install_kmods
-    install_headers
 
-    # Cleanup and install kernel scripts and tools
-    rm -rf ${deb_kern_kbuild_dir}
+    if ! echo "${DEB_BUILD_PROFILES}" | grep -q -e "targetbuild" -e "hostbuild" && echo "${DEB_BUILD_PROFILES}" | grep -q "crosskbuild"; then
+        # Install cross kernel scripts and tools
+        install_kbuild ${deb_kern_kbuild_dir}-cross
+    fi
+
+    # Install kernel scripts and tools
     install_kbuild ${deb_kern_kbuild_dir}
 
     # Stop tracing
diff --git a/meta/recipes-kernel/linux/files/debian/rules.tmpl b/meta/recipes-kernel/linux/files/debian/rules.tmpl
index e8ae3daa..1f838a41 100755
--- a/meta/recipes-kernel/linux/files/debian/rules.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/rules.tmpl
@@ -48,4 +48,4 @@ override_dh_strip_nondeterminism:
 	true
 
 override_dh_strip:
-	unset DEB_HOST_GNU_TYPE && dh_strip -Xvmlinu --no-automatic-dbgsym
+	dh_strip -Xvmlinu -Xlinux-kbuild-${KERNEL_NAME_PROVIDED}-cross --no-automatic-dbgsym
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 94ed84a2..71f195fb 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -83,11 +83,9 @@ TEMPLATE_VARS += "                \
     KCFLAGS                       \
     KAFLAGS                       \
     DISTRIBUTOR                   \
+    PP                            \
 "
 
-inherit dpkg
-inherit template
-
 # Add custom cflags to the kernel build
 KCFLAGS ?= "-fdebug-prefix-map=${CURDIR}=."
 KAFLAGS ?= "-fdebug-prefix-map=${CURDIR}=."
@@ -95,28 +93,75 @@ KAFLAGS ?= "-fdebug-prefix-map=${CURDIR}=."
 # Derive name of the kernel packages from the name of this recipe
 KERNEL_NAME_PROVIDED ?= "${@ d.getVar('PN').partition('linux-')[2]}"
 
-# Make bitbake know we will be producing linux-image and linux-headers packages
 python() {
     kernel_name = d.getVar("KERNEL_NAME_PROVIDED")
-    d.setVar('PROVIDES', 'linux-image-' + kernel_name + ' ' + \
-                         'linux-headers-' + kernel_name)
+    distro_arch = d.getVar("DISTRO_ARCH")
+    host_arch = d.getVar("HOST_ARCH")
     headers_depends = "linux-kbuild-" + kernel_name
 
+    # Determine creation of a kbuild-cross or (standard) kbuild package for the host
+    is_cross_profile = distro_arch != host_arch and d.getVar("ISAR_CROSS_COMPILE", True) == "1"
+    if is_cross_profile:
+        d.appendVar('DEB_BUILD_PROFILES', ' crosskbuild')
+        headers_depends = headers_depends + "-cross | " + headers_depends
+    else:
+      d.appendVar("DEB_BUILD_PROFILES", " defaultkbuild")
+
+    # By setting EXTENDED_KBUILD_MODE to "1" the "-compat" and "-native"
+    # multiarch bitbake targets are used to run additional
+    # target or host specific builds for kbuild scripts and tools.
+    # In this case the "-compat" bitbake target requests a "target" build
+    # and the "-native" bitbake target requests a "host" build.
+    #
+    # For different distro and host archs
+    # - Add dependency for sbuild-chroot-target
+    #   to allow building arch specific kbuild scripts and tools
+    # - Set correct variable overrides
+    is_compat = "compat" in d.getVar("PN", True)
+    is_native = "native" in d.getVar("PN", True)
+    if d.getVar("EXTENDED_KBUILD_MODE", True) and is_cross_profile:
+        build_depends = "crossbuild-essential-" + distro_arch + " [" + host_arch + "], "
+        d.prependVar("KBUILD_DEPENDS", build_depends)
+        d.appendVarFlag("do_prepare_build", "depends", " sbuild-chroot-target:do_build")
+        d.setVar("ISAR_ENABLE_COMPAT_ARCH", "1")
+        d.setVar("COMPAT_DISTRO_ARCH", distro_arch)
+
+        if is_compat:
+            d.appendVar("DEB_BUILD_PROFILES", " targetbuild")
+            d.setVar("ISAR_SKIP_CROSS_PROFILE", "1")
+            d.setVar("SCHROOT_DIR_OVERRIDE", d.getVar("SCHROOT_TARGET_DIR", True))
+            d.setVar("SBUILD_BUILD_OVERRIDE", d.getVar("PACKAGE_ARCH", True))
+            d.setVar("SBUILD_HOST_OVERRIDE", d.getVar("PACKAGE_ARCH", True))
+        elif is_native:
+            d.appendVar("DEB_BUILD_PROFILES", " hostbuild")
+            d.setVar("SCHROOT_DIR_OVERRIDE", d.getVar("SCHROOT_HOST_DIR", True))
+            d.setVar("SBUILD_BUILD_OVERRIDE", d.getVar("HOST_ARCH", True))
+            d.setVar("SBUILD_HOST_OVERRIDE", d.getVar("HOST_ARCH", True))
+
+    # Make bitbake know we will be producing
+    # linux-image and linux-headers packages
+    if not is_compat and not is_native:
+        d.setVar("PROVIDES", "linux-image-" + kernel_name + " " + \
+                             "linux-headers-" + kernel_name)
+
     # Set dependency for kernel headers
     d.appendVar("KERNEL_HEADERS_DEBIAN_DEPENDS", headers_depends)
 }
 
-def get_kernel_arch(d):
-    distro_arch = d.getVar("DISTRO_ARCH")
-    if distro_arch in ["amd64", "i386"]:
+inherit dpkg
+inherit template
+
+def get_kernel_arch(d, arch="DISTRO_ARCH"):
+    arch = d.getVar(arch)
+    if arch in ["amd64", "i386"]:
         kernel_arch = "x86"
-    elif distro_arch == "arm64":
+    elif arch == "arm64":
         kernel_arch = "arm64"
-    elif distro_arch == "armhf":
+    elif arch == "armhf":
         kernel_arch = "arm"
-    elif distro_arch == "mipsel":
+    elif arch == "mipsel":
         kernel_arch = "mips"
-    elif distro_arch == "riscv64":
+    elif arch == "riscv64":
         kernel_arch = "riscv"
     else:
         kernel_arch = ""
-- 
2.39.2

[PATCH v3 5/5] docs: Update custom_kernel docs for split up of kernel scripts and tools

[Koch, Stefan]

Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
---
 doc/custom_kernel.md | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)

diff --git a/doc/custom_kernel.md b/doc/custom_kernel.md
index 73987a5b..52e03b68 100644
--- a/doc/custom_kernel.md
+++ b/doc/custom_kernel.md
@@ -38,12 +38,17 @@ The linux-custom recipe provides support for:
  7. Allow the name of the kernel image to be changed via `KERNEL_FILE` (defaults
     to `vmlinuz`)
 
- 8. Produce a `linux-headers` package which includes kernel headers and kbuild
-    scripts/tools
+ 8. Produce a `linux-headers` package which includes kernel headers
 
- 9. The `linux-headers` package shall support native and cross compiles of
-    out-of-tree kernel modules. However, when built in cross-compilation mode,
-    it cannot be used on the target so far.
+ 9. Produce a `linux-kbuild` package for both `host` and/or `target` arch
+    which includes kbuild scripts and tools.
+    Setting `EXTENDED_KBUILD_MODE` to `"1"` will allow using the `-compat` and
+    `-native` multiarch bitbake targets at cross building to run additional
+    `target` or `host` specific builds.
+
+    Then the `linux-headers` package supports native and cross compiles of
+    out-of-tree kernel modules. Even, when built in cross-compilation mode,
+    it can be used on the target using the `linux-kbuild` package.
 
  10. Produce a `linux-libc-dev` package to support user-land builds
 
@@ -72,8 +77,6 @@ In the future, the recipe may be extended to:
 
  3. Be compatible with Ubuntu
 
- 4. When cross-building, generate kernel-headers for both host and target
-
 ## Examples
 
 The linux-custom recipe is currently used by the linux-mainline package and is
-- 
2.39.2

Re: [PATCH v3 5/5] docs: Update custom_kernel docs for split up of kernel scripts and tools

[Jan Kiszka]

On 16.08.23 14:14, Koch, Stefan (DI PA DCP R&D 3) wrote:
> Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
> ---
>  doc/custom_kernel.md | 17 ++++++++++-------
>  1 file changed, 10 insertions(+), 7 deletions(-)
> 
> diff --git a/doc/custom_kernel.md b/doc/custom_kernel.md
> index 73987a5b..52e03b68 100644
> --- a/doc/custom_kernel.md
> +++ b/doc/custom_kernel.md
> @@ -38,12 +38,17 @@ The linux-custom recipe provides support for:
>   7. Allow the name of the kernel image to be changed via `KERNEL_FILE` (defaults
>      to `vmlinuz`)
>  
> - 8. Produce a `linux-headers` package which includes kernel headers and kbuild
> -    scripts/tools
> + 8. Produce a `linux-headers` package which includes kernel headers
>  
> - 9. The `linux-headers` package shall support native and cross compiles of
> -    out-of-tree kernel modules. However, when built in cross-compilation mode,
> -    it cannot be used on the target so far.
> + 9. Produce a `linux-kbuild` package for both `host` and/or `target` arch
> +    which includes kbuild scripts and tools.
> +    Setting `EXTENDED_KBUILD_MODE` to `"1"` will allow using the `-compat` and
> +    `-native` multiarch bitbake targets at cross building to run additional
> +    `target` or `host` specific builds.

Why does this need an extra control? And the semantic of that mode is
also by far not clear when looking at the variable name.

Jan

> +
> +    Then the `linux-headers` package supports native and cross compiles of
> +    out-of-tree kernel modules. Even, when built in cross-compilation mode,
> +    it can be used on the target using the `linux-kbuild` package.
>  
>   10. Produce a `linux-libc-dev` package to support user-land builds
>  
> @@ -72,8 +77,6 @@ In the future, the recipe may be extended to:
>  
>   3. Be compatible with Ubuntu
>  
> - 4. When cross-building, generate kernel-headers for both host and target
> -
>  ## Examples
>  
>  The linux-custom recipe is currently used by the linux-mainline package and is

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH v3 3/5] dpkg: Allow overriding build and host parameters for sbuild and schroot

[Jan Kiszka]

On 16.08.23 14:14, Koch, Stefan (DI PA DCP R&D 3) wrote:
> This functionality is initially used for custom linux builds using the
> "-compat" and "-native" multiarch bitbake targets to run additional
> target or host specific builds for kbuild scripts and tools.

But why does the kernel need these extra params? Why wouldn't it be
enough to add arch-specific profile settings for that package so that
linux-headers-mykernel-native will compile the kernel only for the
native headers and tools?

Same question for patch 2, in fact. I'm still missing the big picture.

Jan

> 
> Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
> ---
>  meta/classes/dpkg-base.bbclass | 19 ++++++++++++++++++-
>  meta/classes/dpkg.bbclass      |  2 +-
>  2 files changed, 19 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 7b02f378..eaeb0e8b 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -208,7 +208,7 @@ dpkg_runbuild() {
>  
>  def isar_deb_build_profiles(d):
>      deb_build_profiles = d.getVar('DEB_BUILD_PROFILES')
> -    if d.getVar('ISAR_CROSS_COMPILE') == "1":
> +    if d.getVar('ISAR_CROSS_COMPILE') == "1" and d.getVar('ISAR_SKIP_CROSS_PROFILE') != "1":
>          deb_build_profiles += ' cross'
>      return deb_build_profiles.strip()
>  
> @@ -223,6 +223,23 @@ def isar_export_build_settings(d):
>      os.environ['DEB_BUILD_PROFILES'] = isar_deb_build_profiles(d)
>  
>  python do_dpkg_build() {
> +    schroot_dir_override = d.getVar('SCHROOT_DIR_OVERRIDE', True)
> +    sbuild_build_override = d.getVar('SBUILD_BUILD_OVERRIDE', True)
> +    sbuild_host_override = d.getVar('SBUILD_HOST_OVERRIDE', True)
> +
> +    if schroot_dir_override:
> +        d.setVar('SCHROOT_DIR', schroot_dir_override)
> +
> +    if sbuild_build_override:
> +        d.setVar('SBUILD_BUILD', sbuild_build_override)
> +    else:
> +        d.setVar('SBUILD_BUILD', d.getVar('BUILD_HOST_ARCH', True))
> +
> +    if sbuild_host_override:
> +        d.setVar('SBUILD_HOST', sbuild_host_override)
> +    else:
> +        d.setVar('SBUILD_HOST', d.getVar('PACKAGE_ARCH', True))
> +
>      bb.build.exec_func('schroot_create_configs', d)
>      try:
>          bb.build.exec_func("dpkg_runbuild", d)
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index c596adbf..7d659741 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -102,7 +102,7 @@ dpkg_runbuild() {
>      DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -print)
>  
>      sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
> -        --host=${PACKAGE_ARCH} --build=${BUILD_HOST_ARCH} ${profiles} \
> +        --host=${SBUILD_HOST} --build=${SBUILD_BUILD} ${profiles} \
>          --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \
>          --bd-uninstallable-explainer=apt \
>          --no-apt-update \

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH v3 5/5] docs: Update custom_kernel docs for split up of kernel scripts and tools

[Koch, Stefan]

On Wed, 2023-08-16 at 15:46 +0200, Jan Kiszka wrote:
> On 16.08.23 14:14, Koch, Stefan (DI PA DCP R&D 3) wrote:
> > Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
> > ---
> >  doc/custom_kernel.md | 17 ++++++++++-------
> >  1 file changed, 10 insertions(+), 7 deletions(-)
> > 
> > diff --git a/doc/custom_kernel.md b/doc/custom_kernel.md
> > index 73987a5b..52e03b68 100644
> > --- a/doc/custom_kernel.md
> > +++ b/doc/custom_kernel.md
> > @@ -38,12 +38,17 @@ The linux-custom recipe provides support for:
> >   7. Allow the name of the kernel image to be changed via
> > `KERNEL_FILE` (defaults
> >      to `vmlinuz`)
> >  
> > - 8. Produce a `linux-headers` package which includes kernel
> > headers and kbuild
> > -    scripts/tools
> > + 8. Produce a `linux-headers` package which includes kernel
> > headers
> >  
> > - 9. The `linux-headers` package shall support native and cross
> > compiles of
> > -    out-of-tree kernel modules. However, when built in cross-
> > compilation mode,
> > -    it cannot be used on the target so far.
> > + 9. Produce a `linux-kbuild` package for both `host` and/or
> > `target` arch
> > +    which includes kbuild scripts and tools.
> > +    Setting `EXTENDED_KBUILD_MODE` to `"1"` will allow using the
> > `-compat` and
> > +    `-native` multiarch bitbake targets at cross building to run
> > additional
> > +    `target` or `host` specific builds.
> 
> Why does this need an extra control? And the semantic of that mode is
> also by far not clear when looking at the variable name.

Variable can be dropped and turned on by default.

> 
> Jan
> 
> > +
> > +    Then the `linux-headers` package supports native and cross
> > compiles of
> > +    out-of-tree kernel modules. Even, when built in cross-
> > compilation mode,
> > +    it can be used on the target using the `linux-kbuild` package.
> >  
> >   10. Produce a `linux-libc-dev` package to support user-land
> > builds
> >  
> > @@ -72,8 +77,6 @@ In the future, the recipe may be extended to:
> >  
> >   3. Be compatible with Ubuntu
> >  
> > - 4. When cross-building, generate kernel-headers for both host and
> > target
> > -
> >  ## Examples
> >  
> >  The linux-custom recipe is currently used by the linux-mainline
> > package and is
> 

-- 
Stefan Koch
Siemens AG
www.siemens.com

Re: [PATCH v3 3/5] dpkg: Allow overriding build and host parameters for sbuild and schroot

[Koch, Stefan]

On Wed, 2023-08-16 at 15:55 +0200, Jan Kiszka wrote:
> On 16.08.23 14:14, Koch, Stefan (DI PA DCP R&D 3) wrote:
> > This functionality is initially used for custom linux builds using
> > the
> > "-compat" and "-native" multiarch bitbake targets to run additional
> > target or host specific builds for kbuild scripts and tools.
> 
> But why does the kernel need these extra params? Why wouldn't it be
> enough to add arch-specific profile settings for that package so that
> linux-headers-mykernel-native will compile the kernel only for the
> native headers and tools?
> 

Given a image for a arm64 target cross-built on a amd64 host:



- "bitbake linux" generates with the kbuild-patches:
linux-headers_arm64.deb
linux-image_arm64.deb
linux-kbuild-cross_arm64.deb e. g. containing scripts/fixdep as amd64

=> move to -cross to indicate ARCH mismatch

*Without* the kbuild-patches there are only 
linux-headers_arm64.deb e. g. containing scripts/fixdep as amd64 (!!)
linux-image_arm64.deb

=> *Existing* limitation that ARCH does not match



- "bitbake linux-kbuild-compat" generates (forcing non-cross QEMU
build):
linux-kbuild_arm64.deb e. g. containing scripts/fixdep as arm64

=> scripts utilities can *not* compiled in a cross way
=> this enforces the SBUILD_DIR override to force non-cross QEMU

Maybe, unsetting ISAR_CROSS_COMPILE for the -compat target makes the
SBUILD_DIR override needless. It has to be ensured, that no cross-
compile is used.

What do you think?



- "bitbake linux-native" generates:
linux-kbuild_amd64.deb e. g. containing scripts/fixdep as amd64

=> Optional, only for completeness to have a package matching the
correct ARCH

> Same question for patch 2, in fact. I'm still missing the big
> picture.
> 
> Jan
> 
> > 
> > Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
> > ---
> >  meta/classes/dpkg-base.bbclass | 19 ++++++++++++++++++-
> >  meta/classes/dpkg.bbclass      |  2 +-
> >  2 files changed, 19 insertions(+), 2 deletions(-)
> > 
> > diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-
> > base.bbclass
> > index 7b02f378..eaeb0e8b 100644
> > --- a/meta/classes/dpkg-base.bbclass
> > +++ b/meta/classes/dpkg-base.bbclass
> > @@ -208,7 +208,7 @@ dpkg_runbuild() {
> >  
> >  def isar_deb_build_profiles(d):
> >      deb_build_profiles = d.getVar('DEB_BUILD_PROFILES')
> > -    if d.getVar('ISAR_CROSS_COMPILE') == "1":
> > +    if d.getVar('ISAR_CROSS_COMPILE') == "1" and
> > d.getVar('ISAR_SKIP_CROSS_PROFILE') != "1":
> >          deb_build_profiles += ' cross'
> >      return deb_build_profiles.strip()
> >  
> > @@ -223,6 +223,23 @@ def isar_export_build_settings(d):
> >      os.environ['DEB_BUILD_PROFILES'] = isar_deb_build_profiles(d)
> >  
> >  python do_dpkg_build() {
> > +    schroot_dir_override = d.getVar('SCHROOT_DIR_OVERRIDE', True)
> > +    sbuild_build_override = d.getVar('SBUILD_BUILD_OVERRIDE',
> > True)
> > +    sbuild_host_override = d.getVar('SBUILD_HOST_OVERRIDE', True)
> > +
> > +    if schroot_dir_override:
> > +        d.setVar('SCHROOT_DIR', schroot_dir_override)
> > +
> > +    if sbuild_build_override:
> > +        d.setVar('SBUILD_BUILD', sbuild_build_override)
> > +    else:
> > +        d.setVar('SBUILD_BUILD', d.getVar('BUILD_HOST_ARCH',
> > True))
> > +
> > +    if sbuild_host_override:
> > +        d.setVar('SBUILD_HOST', sbuild_host_override)
> > +    else:
> > +        d.setVar('SBUILD_HOST', d.getVar('PACKAGE_ARCH', True))
> > +
> >      bb.build.exec_func('schroot_create_configs', d)
> >      try:
> >          bb.build.exec_func("dpkg_runbuild", d)
> > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> > index c596adbf..7d659741 100644
> > --- a/meta/classes/dpkg.bbclass
> > +++ b/meta/classes/dpkg.bbclass
> > @@ -102,7 +102,7 @@ dpkg_runbuild() {
> >      DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -
> > print)
> >  
> >      sbuild -A -n -c ${SBUILD_CHROOT} --extra-
> > repository="${ISAR_APT_REPO}" \
> > -        --host=${PACKAGE_ARCH} --build=${BUILD_HOST_ARCH}
> > ${profiles} \
> > +        --host=${SBUILD_HOST} --build=${SBUILD_BUILD} ${profiles}
> > \
> >          --no-run-lintian --no-run-piuparts --no-run-autopkgtest --
> > resolve-alternatives \
> >          --bd-uninstallable-explainer=apt \
> >          --no-apt-update \
> 

-- 
Stefan Koch
Siemens AG
www.siemens.com

Re: [PATCH v3 5/5] docs: Update custom_kernel docs for split up of kernel scripts and tools

[Jan Kiszka]

On 16.08.23 16:25, Koch, Stefan (DI PA DCP R&D 3) wrote:
> On Wed, 2023-08-16 at 15:46 +0200, Jan Kiszka wrote:
>> On 16.08.23 14:14, Koch, Stefan (DI PA DCP R&D 3) wrote:
>>> Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
>>> ---
>>>  doc/custom_kernel.md | 17 ++++++++++-------
>>>  1 file changed, 10 insertions(+), 7 deletions(-)
>>>
>>> diff --git a/doc/custom_kernel.md b/doc/custom_kernel.md
>>> index 73987a5b..52e03b68 100644
>>> --- a/doc/custom_kernel.md
>>> +++ b/doc/custom_kernel.md
>>> @@ -38,12 +38,17 @@ The linux-custom recipe provides support for:
>>>   7. Allow the name of the kernel image to be changed via
>>> `KERNEL_FILE` (defaults
>>>      to `vmlinuz`)
>>>  
>>> - 8. Produce a `linux-headers` package which includes kernel
>>> headers and kbuild
>>> -    scripts/tools
>>> + 8. Produce a `linux-headers` package which includes kernel
>>> headers
>>>  
>>> - 9. The `linux-headers` package shall support native and cross
>>> compiles of
>>> -    out-of-tree kernel modules. However, when built in cross-
>>> compilation mode,
>>> -    it cannot be used on the target so far.
>>> + 9. Produce a `linux-kbuild` package for both `host` and/or
>>> `target` arch
>>> +    which includes kbuild scripts and tools.
>>> +    Setting `EXTENDED_KBUILD_MODE` to `"1"` will allow using the
>>> `-compat` and
>>> +    `-native` multiarch bitbake targets at cross building to run
>>> additional
>>> +    `target` or `host` specific builds.
>>
>> Why does this need an extra control? And the semantic of that mode is
>> also by far not clear when looking at the variable name.
> 
> Variable can be dropped and turned on by default.

What would be different when building with EXTENDED_KBUILD_MODE="0"? The
kernel build be fully built for -native and -compat, right?

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH v3 3/5] dpkg: Allow overriding build and host parameters for sbuild and schroot

[Jan Kiszka]

On 16.08.23 16:27, Koch, Stefan (DI PA DCP R&D 3) wrote:
> On Wed, 2023-08-16 at 15:55 +0200, Jan Kiszka wrote:
>> On 16.08.23 14:14, Koch, Stefan (DI PA DCP R&D 3) wrote:
>>> This functionality is initially used for custom linux builds using
>>> the
>>> "-compat" and "-native" multiarch bitbake targets to run additional
>>> target or host specific builds for kbuild scripts and tools.
>>
>> But why does the kernel need these extra params? Why wouldn't it be
>> enough to add arch-specific profile settings for that package so that
>> linux-headers-mykernel-native will compile the kernel only for the
>> native headers and tools?
>>
> 
> Given a image for a arm64 target cross-built on a amd64 host:
> 
> 
> 
> - "bitbake linux" generates with the kbuild-patches:
> linux-headers_arm64.deb
> linux-image_arm64.deb
> linux-kbuild-cross_arm64.deb e. g. containing scripts/fixdep as amd64
> 
> => move to -cross to indicate ARCH mismatch
> 
> *Without* the kbuild-patches there are only 
> linux-headers_arm64.deb e. g. containing scripts/fixdep as amd64 (!!)
> linux-image_arm64.deb
> 
> => *Existing* limitation that ARCH does not match
> 
> 
> 
> - "bitbake linux-kbuild-compat" generates (forcing non-cross QEMU
> build):
> linux-kbuild_arm64.deb e. g. containing scripts/fixdep as arm64
> 
> => scripts utilities can *not* compiled in a cross way
> => this enforces the SBUILD_DIR override to force non-cross QEMU
> 
> Maybe, unsetting ISAR_CROSS_COMPILE for the -compat target makes the
> SBUILD_DIR override needless. It has to be ensured, that no cross-
> compile is used.
> 
> What do you think?
> 
> 
> 
> - "bitbake linux-native" generates:
> linux-kbuild_amd64.deb e. g. containing scripts/fixdep as amd64
> 
> => Optional, only for completeness to have a package matching the
> correct ARCH
> 

I still need to fully wrap around this, but I'm wondering if we are here
really addressing an issue of the sbuild machinery in Isar or rather an
issue of the kernel build recipe that should better be resolved there.

Maybe you can demonstrate the issue also with a simpler package than the
kernel?

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH v3 3/5] dpkg: Allow overriding build and host parameters for sbuild and schroot

[Koch, Stefan]

On Wed, 2023-08-16 at 16:32 +0200, Jan Kiszka wrote:
> On 16.08.23 16:27, Koch, Stefan (DI PA DCP R&D 3) wrote:
> > On Wed, 2023-08-16 at 15:55 +0200, Jan Kiszka wrote:
> > > On 16.08.23 14:14, Koch, Stefan (DI PA DCP R&D 3) wrote:
> > > > This functionality is initially used for custom linux builds
> > > > using
> > > > the
> > > > "-compat" and "-native" multiarch bitbake targets to run
> > > > additional
> > > > target or host specific builds for kbuild scripts and tools.
> > > 
> > > But why does the kernel need these extra params? Why wouldn't it
> > > be
> > > enough to add arch-specific profile settings for that package so
> > > that
> > > linux-headers-mykernel-native will compile the kernel only for
> > > the
> > > native headers and tools?
> > > 
> > 
> > Given a image for a arm64 target cross-built on a amd64 host:
> > 
> > 
> > 
> > - "bitbake linux" generates with the kbuild-patches:
> > linux-headers_arm64.deb
> > linux-image_arm64.deb
> > linux-kbuild-cross_arm64.deb e. g. containing scripts/fixdep as
> > amd64
> > 
> > => move to -cross to indicate ARCH mismatch
> > 
> > *Without* the kbuild-patches there are only 
> > linux-headers_arm64.deb e. g. containing scripts/fixdep as amd64
> > (!!)
> > linux-image_arm64.deb
> > 
> > => *Existing* limitation that ARCH does not match
> > 
> > 
> > 
> > - "bitbake linux-kbuild-compat" generates (forcing non-cross QEMU
> > build):
> > linux-kbuild_arm64.deb e. g. containing scripts/fixdep as arm64
> > 
> > => scripts utilities can *not* compiled in a cross way
> > => this enforces the SBUILD_DIR override to force non-cross QEMU
> > 
> > Maybe, unsetting ISAR_CROSS_COMPILE for the -compat target makes
> > the
> > SBUILD_DIR override needless. It has to be ensured, that no cross-
> > compile is used.
> > 
> > What do you think?
> > 
> > 
> > 
> > - "bitbake linux-native" generates:
> > linux-kbuild_amd64.deb e. g. containing scripts/fixdep as amd64
> > 
> > => Optional, only for completeness to have a package matching the
> > correct ARCH
> > 
> 
> I still need to fully wrap around this, but I'm wondering if we are
> here
> really addressing an issue of the sbuild machinery in Isar or rather
> an
> issue of the kernel build recipe that should better be resolved
> there.

I have found a solution by setting ISAR_CROSS_COMPILE = 0 in -compat
case. This eliminates patches 2 and 3.

I'll finish and test the patches again, and resubmit a new patchset.

> 
> Maybe you can demonstrate the issue also with a simpler package than
> the
> kernel?
> 
> Jan
> 

-- 
Stefan Koch
Siemens AG
www.siemens.com

Re: [PATCH v3 5/5] docs: Update custom_kernel docs for split up of kernel scripts and tools

[Koch, Stefan]

On Wed, 2023-08-16 at 16:27 +0200, Jan Kiszka wrote:
> On 16.08.23 16:25, Koch, Stefan (DI PA DCP R&D 3) wrote:
> > On Wed, 2023-08-16 at 15:46 +0200, Jan Kiszka wrote:
> > > On 16.08.23 14:14, Koch, Stefan (DI PA DCP R&D 3) wrote:
> > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
> > > > ---
> > > >  doc/custom_kernel.md | 17 ++++++++++-------
> > > >  1 file changed, 10 insertions(+), 7 deletions(-)
> > > > 
> > > > diff --git a/doc/custom_kernel.md b/doc/custom_kernel.md
> > > > index 73987a5b..52e03b68 100644
> > > > --- a/doc/custom_kernel.md
> > > > +++ b/doc/custom_kernel.md
> > > > @@ -38,12 +38,17 @@ The linux-custom recipe provides support
> > > > for:
> > > >   7. Allow the name of the kernel image to be changed via
> > > > `KERNEL_FILE` (defaults
> > > >      to `vmlinuz`)
> > > >  
> > > > - 8. Produce a `linux-headers` package which includes kernel
> > > > headers and kbuild
> > > > -    scripts/tools
> > > > + 8. Produce a `linux-headers` package which includes kernel
> > > > headers
> > > >  
> > > > - 9. The `linux-headers` package shall support native and cross
> > > > compiles of
> > > > -    out-of-tree kernel modules. However, when built in cross-
> > > > compilation mode,
> > > > -    it cannot be used on the target so far.
> > > > + 9. Produce a `linux-kbuild` package for both `host` and/or
> > > > `target` arch
> > > > +    which includes kbuild scripts and tools.
> > > > +    Setting `EXTENDED_KBUILD_MODE` to `"1"` will allow using
> > > > the
> > > > `-compat` and
> > > > +    `-native` multiarch bitbake targets at cross building to
> > > > run
> > > > additional
> > > > +    `target` or `host` specific builds.
> > > 
> > > Why does this need an extra control? And the semantic of that
> > > mode is
> > > also by far not clear when looking at the variable name.
> > 
> > Variable can be dropped and turned on by default.
> 
> What would be different when building with EXTENDED_KBUILD_MODE="0"?
> The
> kernel build be fully built for -native and -compat, right?

This is not set within linux-custom.inc, when it is 0:
d.setVar("ISAR_ENABLE_COMPAT_ARCH", "1")
d.setVar("COMPAT_DISTRO_ARCH", distro_arch)

The next patchset will set both variables by default.
Or should the user do it manually? -> Adding a hint to the docs.

> 
> Jan
> 

-- 
Stefan Koch
Siemens AG
www.siemens.com