[PATCH v2 0/5] More kbuild improvements, single-build source packages, cross profile fix

[PATCH v2 0/5] More kbuild improvements, single-build source packages, cross profile fix

[Jan Kiszka]

This depends on [1] and replaces [2].

Jan

[1] https://patchwork.isar-build.org/project/isar/list/?series=1184
[2] https://patchwork.isar-build.org/project/isar/list/?series=1171

Jan Kiszka (5):
  dpkg-base: Fix enabling of cross build profile
  linux-custom: Model cross-built kbuild package separately
  linux-custom: Resolve native source package differences
  dpkg: Retrieve Debian source name from variable
  dpkg-source: Build source package only once

 meta/classes/dpkg-base.bbclass                |  2 +-
 meta/classes/dpkg-source.bbclass              | 60 ++++++++++++++++---
 meta/classes/dpkg.bbclass                     |  3 +-
 .../linux/files/debian/control.tmpl           | 12 +++-
 .../linux/files/debian/isar/common.tmpl       |  2 +
 .../linux/files/debian/isar/install.tmpl      |  6 +-
 meta/recipes-kernel/linux/linux-custom.inc    |  8 +--
 7 files changed, 77 insertions(+), 16 deletions(-)

-- 
2.35.3

[PATCH v2 1/5] dpkg-base: Fix enabling of cross build profile

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

ISAR_CROSS_COMPILE might be set even if we are not cross-building a
package. Use an output of crossbuild.bbclass, BUILD_ARCH, to find out
if we are actually cross-building a package.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 30caedf9..93321976 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -215,7 +215,7 @@ dpkg_runbuild() {
 
 def isar_deb_build_profiles(d):
     deb_build_profiles = d.getVar('DEB_BUILD_PROFILES')
-    if bb.utils.to_boolean(d.getVar('ISAR_CROSS_COMPILE')):
+    if d.getVar('BUILD_ARCH') != d.getVar('DISTRO_ARCH'):
         deb_build_profiles += ' cross'
     return deb_build_profiles.strip()
 
-- 
2.35.3

[PATCH v2 2/5] linux-custom: Model cross-built kbuild package separately

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Rename to kbuild package containing tools to cross-build modules
linux-kbuild-<name>-<arch>-cross. The headers package now depends on
either the native kbuild or a cross variant, unconditionally. This
allows to have unstable source packages between native and cross and is
generally cleaner to describe the cross version.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-kernel/linux/files/debian/control.tmpl  | 12 +++++++++++-
 .../linux/files/debian/isar/common.tmpl              |  2 ++
 .../linux/files/debian/isar/install.tmpl             |  6 +++++-
 meta/recipes-kernel/linux/linux-custom.inc           |  5 +----
 4 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/meta/recipes-kernel/linux/files/debian/control.tmpl b/meta/recipes-kernel/linux/files/debian/control.tmpl
index 6f8f8afe..e0e932a0 100644
--- a/meta/recipes-kernel/linux/files/debian/control.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/control.tmpl
@@ -51,9 +51,19 @@ Description: Linux kernel debugging symbols for @KR@
  all the necessary debug symbols for the kernel and its modules.
 
 Package: linux-kbuild-${KERNEL_NAME_PROVIDED}
-Build-Profiles: <kbuild>
+Build-Profiles: <kbuild !cross>
 Architecture: any
 Depends: ${perl:Depends}, ${shlib:Depends}
 Description: ${KERNEL_NAME_PROVIDED} Linux kbuild scripts and tools for @KR@
  This package provides kernel kbuild scripts and tools for @KR@
  This is useful for people who need to build external modules
+
+Package: linux-kbuild-${KERNEL_NAME_PROVIDED}-${DISTRO_ARCH}-cross
+Build-Profiles: <kbuild cross>
+Architecture: any
+Multi-Arch: foreign
+Depends: ${perl:Depends}, ${shlib:Depends}
+Conflicts: linux-kbuild-${KERNEL_NAME_PROVIDED}
+Description: ${KERNEL_NAME_PROVIDED} Linux kbuild scripts and tools for @KR@
+ This package provides kernel kbuild scripts and tools for @KR@
+ This is useful for people who need to build external modules
diff --git a/meta/recipes-kernel/linux/files/debian/isar/common.tmpl b/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
index e3a1d8a0..86af70c5 100644
--- a/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/isar/common.tmpl
@@ -11,6 +11,7 @@ KERNEL_PKG_KERN_HEADERS=linux-headers-${KERNEL_NAME_PROVIDED}
 KERNEL_PKG_LIBC_HEADERS=linux-libc-dev
 KERNEL_PKG_LIBC_HEADERS_CROSS=linux-libc-dev-${DISTRO_ARCH}-cross
 KERNEL_PKG_KERN_KBUILD=linux-kbuild-${KERNEL_NAME_PROVIDED}
+KERNEL_PKG_KERN_KBUILD_CROSS=${KERNEL_PKG_KERN_KBUILD}-${DISTRO_ARCH}-cross
 
 # Force creating debian package with valid host arch for -native build
 # Use a cross build to comply with arch specific kernel defconfigs
@@ -32,6 +33,7 @@ deb_kern_hdr_dir=${deb_top_dir}/${KERNEL_PKG_KERN_HEADERS}
 deb_libc_hdr_dir=${deb_top_dir}/${KERNEL_PKG_LIBC_HEADERS}
 deb_libc_hdr_cross_dir=${deb_top_dir}/${KERNEL_PKG_LIBC_HEADERS_CROSS}
 deb_kern_kbuild_dir=${deb_top_dir}/${KERNEL_PKG_KERN_KBUILD}
+deb_kern_kbuild_cross_dir=${deb_top_dir}/${KERNEL_PKG_KERN_KBUILD_CROSS}
 
 # Array of packages to be generated
 declare -A kern_pkgs
diff --git a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
index 77856aee..00011517 100644
--- a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl
@@ -35,7 +35,11 @@ do_install() {
 
     if echo "${DEB_BUILD_PROFILES}" | grep -q "kbuild"; then
         # Install kernel scripts and tools
-        install_kbuild ${deb_kern_kbuild_dir}
+        if echo "${DEB_BUILD_PROFILES}" | grep -q "cross"; then
+            install_kbuild ${deb_kern_kbuild_cross_dir}
+        else
+            install_kbuild ${deb_kern_kbuild_dir}
+        fi
     fi
 
     if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 3fd0c6c2..c54c287f 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -111,9 +111,6 @@ BBCLASSEXTEND:append:cross-profile = " kbuildtarget"
 # build only kernel with the default variant of the recipe
 BUILD_PROFILES:cross-profile = "kernel"
 
-# Select correct kbuild package for isar cross-build
-HEADERS_DEPENDS:cross-profile = ", linux-kbuild-${KERNEL_NAME_PROVIDED}:${HOST_ARCH} | linux-kbuild-${KERNEL_NAME_PROVIDED}"
-
 # -native: kbuild package for host
 BUILD_PROFILES:class-native = "kbuild"
 RECIPE_PROVIDES:class-native = " \
@@ -151,7 +148,7 @@ RECIPE_PROVIDES:remove:cross-profile = " \
     linux-kbuild-${KERNEL_NAME_PROVIDED}"
 
 # Append headers depends
-HEADERS_DEPENDS = ", linux-kbuild-${KERNEL_NAME_PROVIDED}"
+HEADERS_DEPENDS = ", linux-kbuild-${KERNEL_NAME_PROVIDED} | linux-kbuild-${KERNEL_NAME_PROVIDED}-${DISTRO_ARCH}-cross"
 KERNEL_HEADERS_DEBIAN_DEPENDS:append = "${HEADERS_DEPENDS}"
 
 # Append provides
-- 
2.35.3

[PATCH v2 3/5] linux-custom: Resolve native source package differences

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

When building the native package, KERNEL_FILE as defined in bitbake.conf
is wrong as it uses the package arch. Avoid that by redefining
KERNEL_FILE in the kernel recipe using KERNEL_ARCH.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-kernel/linux/linux-custom.inc | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index c54c287f..b53d136c 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -175,6 +175,9 @@ def get_kernel_arch(d):
 
 KERNEL_ARCH ??= "${@get_kernel_arch(d)}"
 
+# set KERNEL_FILE without depending on package arch used in bitbake.conf
+KERNEL_FILE:forcevariable = "${@ 'vmlinux' if d.getVar('KERNEL_ARCH') in ['mipsel', 'riscv', 'arm64'] else 'vmlinuz'}"
+
 def config_fragments(d):
     fragments = []
     sources = d.getVar("SRC_URI").split()
-- 
2.35.3

[PATCH v2 4/5] dpkg: Retrieve Debian source name from variable

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Rather than reading this multiple times back from the changelog, define
DEBIAN_SOURCE, by default BPN, and use that consistently. This will
allow to fetch sources built by the base recipe in native and compat
recipes later on without generating the changelog first.

It also permits to clean up related code. Specifically do_deploy_source
was expecting more than one source file (which was incorrect) and looked
in the wrong folder.

The equivalence between DEBIAN_SOURCE and the actual source field is
checked in do_dpkg_source and enforced by failing the build otherwise.

At this chance, also move global -maxdepth before -name in find
statements.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-source.bbclass | 16 +++++++++++-----
 meta/classes/dpkg.bbclass        |  3 +--
 2 files changed, 12 insertions(+), 7 deletions(-)

diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
index 7fd5d2ed..560f536b 100644
--- a/meta/classes/dpkg-source.bbclass
+++ b/meta/classes/dpkg-source.bbclass
@@ -7,10 +7,15 @@ inherit dpkg-base
 
 DPKG_SOURCE_EXTRA_ARGS ?= "-I"
 
+DEBIAN_SOURCE ?= "${BPN}"
+
 do_dpkg_source() {
     # Create a .dsc file from source directory to use it with sbuild
     DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
-    find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
+    if [ "${DEB_SOURCE_NAME}" != "${DEBIAN_SOURCE}" ]; then
+        bbfatal "DEBIAN_SOURCE (${DEBIAN_SOURCE}) not aligned with source name used in control files (${DEB_SOURCE_NAME})"
+    fi
+    find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete
     sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}"
 }
 addtask dpkg_source after do_prepare_build before do_dpkg_build
@@ -20,12 +25,13 @@ do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 do_deploy_source[dirs] = "${S}"
 do_deploy_source() {
     repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
-        "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
-    find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
+        "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${DEBIAN_SOURCE}"
+    DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc")
+    if [ -n "${DSC_FILE}" ]; then
         repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
             "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
             "${DEBDISTRONAME}" \
-            "${package}"
-    done
+            "${DSC_FILE}"
+    fi
 }
 addtask deploy_source after do_dpkg_source before do_dpkg_build
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 3fa9f604..3cc36c12 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -108,8 +108,7 @@ dpkg_runbuild() {
     echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
     echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG}
 
-    DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
-    DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -print)
+    DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print)
 
     sbuild -A -n -c ${SBUILD_CHROOT} \
         --host=${PACKAGE_ARCH} --build=${BUILD_ARCH} ${profiles} \
-- 
2.35.3

[PATCH v2 5/5] dpkg-source: Build source package only once

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Avoid building the source package multiple times, possibly even
inconsistently. This is achieved by delegating this task to to the base
package and installing the source package from isar-apt in the native
and compat package variants.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-source.bbclass | 44 ++++++++++++++++++++++++++++++--
 1 file changed, 42 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
index 560f536b..6a36081b 100644
--- a/meta/classes/dpkg-source.bbclass
+++ b/meta/classes/dpkg-source.bbclass
@@ -18,7 +18,7 @@ do_dpkg_source() {
     find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete
     sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}"
 }
-addtask dpkg_source after do_prepare_build before do_dpkg_build
+addtask dpkg_source after do_prepare_build
 
 do_deploy_source[depends] += "isar-apt:do_cache_config"
 do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
@@ -34,4 +34,44 @@ do_deploy_source() {
             "${DSC_FILE}"
     fi
 }
-addtask deploy_source after do_dpkg_source before do_dpkg_build
+addtask deploy_source after do_dpkg_source
+
+do_dpkg_build[depends] += "${BPN}:do_deploy_source"
+
+SCHROOT_MOUNTS = "${WORKDIR}:/work ${REPO_ISAR_DIR}/${DISTRO}:/isar-apt"
+
+do_fetch_common_source[depends] += "${SCHROOT_DEP} ${BPN}:do_deploy_source"
+do_fetch_common_source[network] = "${TASK_USE_SUDO}"
+do_fetch_common_source() {
+    schroot_create_configs
+    insert_mounts
+
+    session_id=$(schroot -q -b -c ${SBUILD_CHROOT})
+    echo "Started session: ${session_id}"
+
+    schroot_cleanup() {
+        schroot -q -f -e -c ${session_id} > /dev/null 2>&1
+        remove_mounts > /dev/null 2>&1
+        schroot_delete_configs
+    }
+    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
+    trap 'schroot_cleanup' EXIT
+
+    schroot -r -c ${session_id} -d / -u root -- \
+        apt-get update -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" -o Dir::Etc::SourceParts="-" -o APT::Get::List-Cleanup="0"
+    schroot -r -c ${session_id} -d / -- \
+        sh -c '
+            cd /work
+            apt-get -y --download-only --only-source -o Acquire::Source-Symlinks="false" source ${DEBIAN_SOURCE}'
+
+    schroot -e -c ${session_id}
+    remove_mounts
+    schroot_delete_configs
+}
+addtask fetch_common_source after do_unpack
+
+def is_native_or_compat(d):
+    overrides = d.getVar('OVERRIDES').split(':')
+    return 'class-native' in overrides or 'class-compat' in overrides
+
+do_dpkg_build[depends] += "${@'${PN}:do_fetch_common_source' if is_native_or_compat(d) else ''}"
-- 
2.35.3

Re: [PATCH v2 0/5] More kbuild improvements, single-build source packages, cross profile fix

[Koch, Stefan]

Since applying this series, I get this build error:

ERROR: linux-audis-kbuildtarget-5.10.209-audis-rtp-r0 do_dpkg_build:
ExecutionError('/home/z003jsce/kbuildboards/build/tmp/work/audis-
bullseye-latest-arm64/linux-audis-kbuildtarget/5.10.209-audis-rtp-
r0/temp/run.dpkg_runbuild.77935', 1, None, None)
ERROR: Logfile of failure stored in:
/home/z003jsce/kbuildboards/build/tmp/work/audis-bullseye-latest-
arm64/linux-audis-kbuildtarget/5.10.209-audis-rtp-
r0/temp/log.do_dpkg_build.77935
Log data follows:
| DEBUG: Executing python function sstate_task_prefunc
| DEBUG: Python function sstate_task_prefunc finished
| DEBUG: Executing python function do_dpkg_build
| DEBUG: Executing shell function dpkg_schroot_create_configs
| Creating /etc/schroot/chroot.d/isar-builder-cbda93bb-87d8-4b8d-ae73-
2495897f4aaf-77935
| DEBUG: Shell function dpkg_schroot_create_configs finished
| DEBUG: Executing shell function dpkg_runbuild
| WARNING: Export of 'CCACHE_UMASK=000' detected, please migrate to
templates
| dpkg-parsechangelog: error: cannot open file
/home/z003jsce/kbuildboards/build/debian/changelog: No such file or
directory
| E: Could not parse
/home/z003jsce/kbuildboards/build/debian/changelog:
| WARNING: exit code 1 from a shell command.
| DEBUG: Executing shell function schroot_delete_configs
| Removing /etc/schroot/isar-builder-cbda93bb-87d8-4b8d-ae73-
2495897f4aaf-77935
| Removing /etc/schroot/chroot.d/isar-builder-cbda93bb-87d8-4b8d-ae73-
2495897f4aaf-77935
| DEBUG: Shell function schroot_delete_configs finished
| DEBUG: Python function do_dpkg_build finished
ERROR: Task
(virtual:kbuildtarget:/home/z003jsce/kbuildboards/build/../audis-
isar/recipes-kernel/linux/linux-audis.bb:do_dpkg_build) failed with
exit code '1'
NOTE: Tasks Summary: Attempted 97 tasks of which 0 didn't need to be
rerun and 1 failed.

Stefan

On Wed, 2024-05-15 at 13:08 +0200, Jan Kiszka wrote:
> This depends on [1] and replaces [2].
> 
> Jan
> 
> [1] https://patchwork.isar-build.org/project/isar/list/?series=1184
> [2] https://patchwork.isar-build.org/project/isar/list/?series=1171
> 
> Jan Kiszka (5):
>   dpkg-base: Fix enabling of cross build profile
>   linux-custom: Model cross-built kbuild package separately
>   linux-custom: Resolve native source package differences
>   dpkg: Retrieve Debian source name from variable
>   dpkg-source: Build source package only once
> 
>  meta/classes/dpkg-base.bbclass                |  2 +-
>  meta/classes/dpkg-source.bbclass              | 60 ++++++++++++++++-
> --
>  meta/classes/dpkg.bbclass                     |  3 +-
>  .../linux/files/debian/control.tmpl           | 12 +++-
>  .../linux/files/debian/isar/common.tmpl       |  2 +
>  .../linux/files/debian/isar/install.tmpl      |  6 +-
>  meta/recipes-kernel/linux/linux-custom.inc    |  8 +--
>  7 files changed, 77 insertions(+), 16 deletions(-)
> 

-- 
Stefan Koch
Siemens AG
www.siemens.com

Re: [PATCH v2 0/5] More kbuild improvements, single-build source packages, cross profile fix

[Jan Kiszka]

On 15.05.24 15:47, Koch, Stefan (DI PA DCP R&D 3) wrote:
> Since applying this series, I get this build error:
> 
> ERROR: linux-audis-kbuildtarget-5.10.209-audis-rtp-r0 do_dpkg_build:
> ExecutionError('/home/z003jsce/kbuildboards/build/tmp/work/audis-
> bullseye-latest-arm64/linux-audis-kbuildtarget/5.10.209-audis-rtp-
> r0/temp/run.dpkg_runbuild.77935', 1, None, None)
> ERROR: Logfile of failure stored in:
> /home/z003jsce/kbuildboards/build/tmp/work/audis-bullseye-latest-
> arm64/linux-audis-kbuildtarget/5.10.209-audis-rtp-
> r0/temp/log.do_dpkg_build.77935
> Log data follows:
> | DEBUG: Executing python function sstate_task_prefunc
> | DEBUG: Python function sstate_task_prefunc finished
> | DEBUG: Executing python function do_dpkg_build
> | DEBUG: Executing shell function dpkg_schroot_create_configs
> | Creating /etc/schroot/chroot.d/isar-builder-cbda93bb-87d8-4b8d-ae73-
> 2495897f4aaf-77935
> | DEBUG: Shell function dpkg_schroot_create_configs finished
> | DEBUG: Executing shell function dpkg_runbuild
> | WARNING: Export of 'CCACHE_UMASK=000' detected, please migrate to
> templates
> | dpkg-parsechangelog: error: cannot open file
> /home/z003jsce/kbuildboards/build/debian/changelog: No such file or
> directory
> | E: Could not parse
> /home/z003jsce/kbuildboards/build/debian/changelog:
> | WARNING: exit code 1 from a shell command.
> | DEBUG: Executing shell function schroot_delete_configs
> | Removing /etc/schroot/isar-builder-cbda93bb-87d8-4b8d-ae73-
> 2495897f4aaf-77935
> | Removing /etc/schroot/chroot.d/isar-builder-cbda93bb-87d8-4b8d-ae73-
> 2495897f4aaf-77935
> | DEBUG: Shell function schroot_delete_configs finished
> | DEBUG: Python function do_dpkg_build finished
> ERROR: Task
> (virtual:kbuildtarget:/home/z003jsce/kbuildboards/build/../audis-
> isar/recipes-kernel/linux/linux-audis.bb:do_dpkg_build) failed with
> exit code '1'
> NOTE: Tasks Summary: Attempted 97 tasks of which 0 didn't need to be
> rerun and 1 failed.
> 

Can you find a reproduction path with upstream? Or identify the required
modification for it to trigger this? I suspect some downstream kernel
recipe bits play an central role because we no longer run
dpkg-parsechangelog in upstream do_dpkg_build.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

RE: [PATCH v2 1/5] dpkg-base: Fix enabling of cross build profile

[Schmidt, Adriaan]

Kiszka, Jan (T CED), Mittwoch, 15. Mai 2024 13:08:
> ISAR_CROSS_COMPILE might be set even if we are not cross-building a
> package. Use an output of crossbuild.bbclass, BUILD_ARCH, to find out
> if we are actually cross-building a package.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/classes/dpkg-base.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 30caedf9..93321976 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -215,7 +215,7 @@ dpkg_runbuild() {
> 
>  def isar_deb_build_profiles(d):
>      deb_build_profiles = d.getVar('DEB_BUILD_PROFILES')
> -    if bb.utils.to_boolean(d.getVar('ISAR_CROSS_COMPILE')):
> +    if d.getVar('BUILD_ARCH') != d.getVar('DISTRO_ARCH'):

That's the question "are we _actually_ cross-compiling"?

Elsewhere (multiarch.bbclass, linux-custom.inc) we have that
check implemented as "ISAR_CROSS_COMPILE && DISTRO_ARCH != HOST_ARCH".

As a follow-up we should look into unifying this.
Maybe have crossvars.bbclass figure out how we're building, and set
one variable to be consumed by recipes, e.g.

ISAR_BUILDPROFILE="native|emulated|cross"

Or even add an override (:buildprofile-native, ...) to make it easier to use?

Adriaan


>          deb_build_profiles += ' cross'
>      return deb_build_profiles.strip()
> 
> --
> 2.35.3

Re: [PATCH v2 1/5] dpkg-base: Fix enabling of cross build profile

[Jan Kiszka]

On 16.05.24 08:06, Schmidt, Adriaan (T CED EDC-DE) wrote:
> Kiszka, Jan (T CED), Mittwoch, 15. Mai 2024 13:08:
>> ISAR_CROSS_COMPILE might be set even if we are not cross-building a
>> package. Use an output of crossbuild.bbclass, BUILD_ARCH, to find out
>> if we are actually cross-building a package.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>  meta/classes/dpkg-base.bbclass | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
>> index 30caedf9..93321976 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -215,7 +215,7 @@ dpkg_runbuild() {
>>
>>  def isar_deb_build_profiles(d):
>>      deb_build_profiles = d.getVar('DEB_BUILD_PROFILES')
>> -    if bb.utils.to_boolean(d.getVar('ISAR_CROSS_COMPILE')):
>> +    if d.getVar('BUILD_ARCH') != d.getVar('DISTRO_ARCH'):
> 
> That's the question "are we _actually_ cross-compiling"?
> 
> Elsewhere (multiarch.bbclass, linux-custom.inc) we have that
> check implemented as "ISAR_CROSS_COMPILE && DISTRO_ARCH != HOST_ARCH".
> 
> As a follow-up we should look into unifying this.
> Maybe have crossvars.bbclass figure out how we're building, and set
> one variable to be consumed by recipes, e.g.
> 
> ISAR_BUILDPROFILE="native|emulated|cross"
> 
> Or even add an override (:buildprofile-native, ...) to make it easier to use?
> 

I don't disagree.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH v2 0/5] More kbuild improvements, single-build source packages, cross profile fix

[Jan Kiszka]

On 15.05.24 19:40, 'Jan Kiszka' via isar-users wrote:
> On 15.05.24 15:47, Koch, Stefan (DI PA DCP R&D 3) wrote:
>> Since applying this series, I get this build error:
>>
>> ERROR: linux-audis-kbuildtarget-5.10.209-audis-rtp-r0 do_dpkg_build:
>> ExecutionError('/home/z003jsce/kbuildboards/build/tmp/work/audis-
>> bullseye-latest-arm64/linux-audis-kbuildtarget/5.10.209-audis-rtp-
>> r0/temp/run.dpkg_runbuild.77935', 1, None, None)
>> ERROR: Logfile of failure stored in:
>> /home/z003jsce/kbuildboards/build/tmp/work/audis-bullseye-latest-
>> arm64/linux-audis-kbuildtarget/5.10.209-audis-rtp-
>> r0/temp/log.do_dpkg_build.77935
>> Log data follows:
>> | DEBUG: Executing python function sstate_task_prefunc
>> | DEBUG: Python function sstate_task_prefunc finished
>> | DEBUG: Executing python function do_dpkg_build
>> | DEBUG: Executing shell function dpkg_schroot_create_configs
>> | Creating /etc/schroot/chroot.d/isar-builder-cbda93bb-87d8-4b8d-ae73-
>> 2495897f4aaf-77935
>> | DEBUG: Shell function dpkg_schroot_create_configs finished
>> | DEBUG: Executing shell function dpkg_runbuild
>> | WARNING: Export of 'CCACHE_UMASK=000' detected, please migrate to
>> templates
>> | dpkg-parsechangelog: error: cannot open file
>> /home/z003jsce/kbuildboards/build/debian/changelog: No such file or
>> directory
>> | E: Could not parse
>> /home/z003jsce/kbuildboards/build/debian/changelog:
>> | WARNING: exit code 1 from a shell command.
>> | DEBUG: Executing shell function schroot_delete_configs
>> | Removing /etc/schroot/isar-builder-cbda93bb-87d8-4b8d-ae73-
>> 2495897f4aaf-77935
>> | Removing /etc/schroot/chroot.d/isar-builder-cbda93bb-87d8-4b8d-ae73-
>> 2495897f4aaf-77935
>> | DEBUG: Shell function schroot_delete_configs finished
>> | DEBUG: Python function do_dpkg_build finished
>> ERROR: Task
>> (virtual:kbuildtarget:/home/z003jsce/kbuildboards/build/../audis-
>> isar/recipes-kernel/linux/linux-audis.bb:do_dpkg_build) failed with
>> exit code '1'
>> NOTE: Tasks Summary: Attempted 97 tasks of which 0 didn't need to be
>> rerun and 1 failed.
>>
> 
> Can you find a reproduction path with upstream? Or identify the required
> modification for it to trigger this? I suspect some downstream kernel
> recipe bits play an central role because we no longer run
> dpkg-parsechangelog in upstream do_dpkg_build.
> 

Were you able to clarify what is needed to trigger this? As this series
fixes the remaining issues of the kbuild changes (to my best knowledge),
it would be good to have them ready ASAP.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH v2 0/5] More kbuild improvements, single-build source packages, cross profile fix

[Koch, Stefan]

On Thu, 2024-05-16 at 18:37 +0200, Jan Kiszka wrote:
> On 15.05.24 19:40, 'Jan Kiszka' via isar-users wrote:
> > On 15.05.24 15:47, Koch, Stefan (DI PA DCP R&D 3) wrote:
> > > Since applying this series, I get this build error:
> > > 
> > > ERROR: linux-audis-kbuildtarget-5.10.209-audis-rtp-r0
> > > do_dpkg_build:
> > > ExecutionError('/home/z003jsce/kbuildboards/build/tmp/work/audis-
> > > bullseye-latest-arm64/linux-audis-kbuildtarget/5.10.209-audis-
> > > rtp-
> > > r0/temp/run.dpkg_runbuild.77935', 1, None, None)
> > > ERROR: Logfile of failure stored in:
> > > /home/z003jsce/kbuildboards/build/tmp/work/audis-bullseye-latest-
> > > arm64/linux-audis-kbuildtarget/5.10.209-audis-rtp-
> > > r0/temp/log.do_dpkg_build.77935
> > > Log data follows:
> > > > DEBUG: Executing python function sstate_task_prefunc
> > > > DEBUG: Python function sstate_task_prefunc finished
> > > > DEBUG: Executing python function do_dpkg_build
> > > > DEBUG: Executing shell function dpkg_schroot_create_configs
> > > > Creating /etc/schroot/chroot.d/isar-builder-cbda93bb-87d8-4b8d-
> > > > ae73-
> > > 2495897f4aaf-77935
> > > > DEBUG: Shell function dpkg_schroot_create_configs finished
> > > > DEBUG: Executing shell function dpkg_runbuild
> > > > WARNING: Export of 'CCACHE_UMASK=000' detected, please migrate
> > > > to
> > > templates
> > > > dpkg-parsechangelog: error: cannot open file
> > > /home/z003jsce/kbuildboards/build/debian/changelog: No such file
> > > or
> > > directory
> > > > E: Could not parse
> > > /home/z003jsce/kbuildboards/build/debian/changelog:
> > > > WARNING: exit code 1 from a shell command.
> > > > DEBUG: Executing shell function schroot_delete_configs
> > > > Removing /etc/schroot/isar-builder-cbda93bb-87d8-4b8d-ae73-
> > > 2495897f4aaf-77935
> > > > Removing /etc/schroot/chroot.d/isar-builder-cbda93bb-87d8-4b8d-
> > > > ae73-
> > > 2495897f4aaf-77935
> > > > DEBUG: Shell function schroot_delete_configs finished
> > > > DEBUG: Python function do_dpkg_build finished
> > > ERROR: Task
> > > (virtual:kbuildtarget:/home/z003jsce/kbuildboards/build/../audis-
> > > isar/recipes-kernel/linux/linux-audis.bb:do_dpkg_build) failed
> > > with
> > > exit code '1'
> > > NOTE: Tasks Summary: Attempted 97 tasks of which 0 didn't need to
> > > be
> > > rerun and 1 failed.
> > > 
> > 
> > Can you find a reproduction path with upstream? Or identify the
> > required
> > modification for it to trigger this? I suspect some downstream
> > kernel
> > recipe bits play an central role because we no longer run
> > dpkg-parsechangelog in upstream do_dpkg_build.
> > 
> 
> Were you able to clarify what is needed to trigger this? As this
> series
> fixes the remaining issues of the kbuild changes (to my best
> knowledge),
> it would be good to have them ready ASAP.
> 

The issue seems to come only with commit:
dpkg-source: Build source package only once

But it still needs deeper investigation.


> Jan
> 

-- 
Stefan Koch
Siemens AG
www.siemens.com

Re: [PATCH v2 5/5] dpkg-source: Build source package only once

[Jan Kiszka]

On 15.05.24 13:08, 'Jan Kiszka' via isar-users wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> Avoid building the source package multiple times, possibly even
> inconsistently. This is achieved by delegating this task to to the base
> package and installing the source package from isar-apt in the native
> and compat package variants.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/classes/dpkg-source.bbclass | 44 ++++++++++++++++++++++++++++++--
>  1 file changed, 42 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
> index 560f536b..6a36081b 100644
> --- a/meta/classes/dpkg-source.bbclass
> +++ b/meta/classes/dpkg-source.bbclass
> @@ -18,7 +18,7 @@ do_dpkg_source() {
>      find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete
>      sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}"
>  }
> -addtask dpkg_source after do_prepare_build before do_dpkg_build
> +addtask dpkg_source after do_prepare_build
>  
>  do_deploy_source[depends] += "isar-apt:do_cache_config"
>  do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
> @@ -34,4 +34,44 @@ do_deploy_source() {
>              "${DSC_FILE}"
>      fi
>  }
> -addtask deploy_source after do_dpkg_source before do_dpkg_build
> +addtask deploy_source after do_dpkg_source
> +
> +do_dpkg_build[depends] += "${BPN}:do_deploy_source"
> +
> +SCHROOT_MOUNTS = "${WORKDIR}:/work ${REPO_ISAR_DIR}/${DISTRO}:/isar-apt"
> +
> +do_fetch_common_source[depends] += "${SCHROOT_DEP} ${BPN}:do_deploy_source"
> +do_fetch_common_source[network] = "${TASK_USE_SUDO}"
> +do_fetch_common_source() {
> +    schroot_create_configs
> +    insert_mounts
> +
> +    session_id=$(schroot -q -b -c ${SBUILD_CHROOT})
> +    echo "Started session: ${session_id}"
> +
> +    schroot_cleanup() {
> +        schroot -q -f -e -c ${session_id} > /dev/null 2>&1
> +        remove_mounts > /dev/null 2>&1
> +        schroot_delete_configs
> +    }
> +    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
> +    trap 'schroot_cleanup' EXIT
> +
> +    schroot -r -c ${session_id} -d / -u root -- \
> +        apt-get update -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" -o Dir::Etc::SourceParts="-" -o APT::Get::List-Cleanup="0"
> +    schroot -r -c ${session_id} -d / -- \
> +        sh -c '
> +            cd /work
> +            apt-get -y --download-only --only-source -o Acquire::Source-Symlinks="false" source ${DEBIAN_SOURCE}'
> +
> +    schroot -e -c ${session_id}
> +    remove_mounts
> +    schroot_delete_configs
> +}
> +addtask fetch_common_source after do_unpack
> +
> +def is_native_or_compat(d):
> +    overrides = d.getVar('OVERRIDES').split(':')
> +    return 'class-native' in overrides or 'class-compat' in overrides
> +
> +do_dpkg_build[depends] += "${@'${PN}:do_fetch_common_source' if is_native_or_compat(d) else ''}"

Looks like I lost a good share of my related dependency improvements,
and some appear to be actually needed to avoid regressions. Just found
them again in stash, v3 will follow.

FWIW, they do NOT explain the issue Stefan reported. I tested against
isar-cip-core updated with this, and it works perfectly fine.

Jan

-- 
Siemens AG, Technology
Linux Expert Center