* [PATCH] linux-custom: compile only those specified dtbs @ 2024-08-02 11:41 'Nicusor Huhulea' via isar-users 2024-08-02 17:03 ` 'Jan Kiszka' via isar-users ` (2 more replies) 0 siblings, 3 replies; 17+ messages in thread From: 'Nicusor Huhulea' via isar-users @ 2024-08-02 11:41 UTC (permalink / raw) To: isar-users; +Cc: Nicusor Huhulea The current implementation makes a full kernel build and that means it will compile all dtbs specified by the bsp makefile e.g arch/arm64/boot/dts/ti/Makefile Currently the required dtbs are given by the DTB_FILES. These changes are checking if the DTB_FILES is specified, compile only those and avoids compiling all the dtbs from the bsp directory. Otherwise build all dtbs specified by the kernel source. Preventing the automatic dtb builds is made by using target specific build commands e.g KERNEL_IMAGETYPE The compilation of the dtbs accepts the following formats e.g: a) full path from the architecture directory: arch/${ARCH}/boot/dts/bsp_dir/test.dts arch/${ARCH}/boot/dts/bsp_dir/test.dtb b) relative path from the dts directory: bsp_dir/test.dts c) device tree blob file: bsp_dir/test.dtb Compilation of the dtbs is somewhat similar to KERNEL_DEVICETREE from OE when it comes to declaring the path to the dts, but for now in the DTB_FILES variable only Device Tree Blobs(.dtb) needs to be declared due to the fact that is being used in do_copy_boot_files task. Signed-off-by: Nicusor Huhulea <nicusor.huhulea@siemens.com> --- meta-isar/conf/machine/imx6-sabrelite.conf | 4 +- meta-isar/conf/machine/phyboard-mira.conf | 4 +- .../linux/files/debian/isar/build.tmpl | 43 ++++++++++++++++--- .../linux/files/debian/isar/install.tmpl | 31 ++++++++++++- meta/recipes-kernel/linux/linux-custom.inc | 6 +++ 5 files changed, 80 insertions(+), 8 deletions(-) diff --git a/meta-isar/conf/machine/imx6-sabrelite.conf b/meta-isar/conf/machine/imx6-sabrelite.conf index e48823d6..6ca0a907 100644 --- a/meta-isar/conf/machine/imx6-sabrelite.conf +++ b/meta-isar/conf/machine/imx6-sabrelite.conf @@ -15,4 +15,6 @@ MKUBIFS_ARGS := "-m 0x1000 -e 0x3e000 -c 1500" UBINIZE_ARGS = "-vv -m 0x1000 -p 0x40000" IMAGE_FSTYPES ?= "ubi-ubifs" -DTB_FILES = "imx6q-sabrelite.dtb" +KERNEL_IMAGETYPE = "zImage" + +DTB_FILES = "nxp/imx/imx6q-sabrelite.dtb" diff --git a/meta-isar/conf/machine/phyboard-mira.conf b/meta-isar/conf/machine/phyboard-mira.conf index feec4e54..d64b4624 100644 --- a/meta-isar/conf/machine/phyboard-mira.conf +++ b/meta-isar/conf/machine/phyboard-mira.conf @@ -16,10 +16,12 @@ MKUBIFS_ARGS := "-m 0x800 -e 0x1f000 -c 8012" UBINIZE_ARGS = "-vv -m 0x800 -p 0x20000" IMAGE_FSTYPES ?= "ubi ubifs" -DTB_FILES = "imx6q-phytec-mira-rdk-nand.dtb" +DTB_FILES = "nxp/imx/imx6q-phytec-mira-rdk-nand.dtb" IMAGE_INSTALL += "barebox" +KERNEL_IMAGETYPE = "zImage" + BAREBOX_CONFIG = "imx_v7_defconfig" BAREBOX_ENV = "phytec-mira-env" BAREBOX_BASE_BIN = "barebox-phytec-phycore-imx6q-som-nand-1gib" diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl index bafc5ca4..b18c8f20 100644 --- a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl +++ b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl @@ -21,22 +21,55 @@ do_build() { KR=$(${MAKE} O=${KERNEL_BUILD_DIR} -s --no-print-directory kernelrelease) sed -i "s/@KR@/${KR}/g" ${S}/debian/control ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.* - if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then # Build kernel scripts and tools - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" + MAKE_CMD_PREFIX="${MAKE} O=\"${KERNEL_BUILD_DIR}\" ${PARALLEL_MAKE} KCFLAGS=\"${KCFLAGS}\" KAFLAGS=\"${KAFLAGS}\"" + if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then + if [ -n "${KERNEL_IMAGETYPE}" ]; then # Override the default compilation + eval "${MAKE_CMD_PREFIX} ${KERNEL_EXTRA_BUILDARGS} ${KERNEL_IMAGETYPE} modules" + compile_dtbs + else # Full build + eval "${MAKE_CMD_PREFIX} ${KERNEL_EXTRA_BUILDARGS}" + fi elif echo "${DEB_BUILD_PROFILES}" | grep -q "kbuild"; then # Build kernel scripts and tools - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" scripts + eval "${MAKE_CMD_PREFIX} ${KERNEL_EXTRA_BUILDARGS} scripts" if grep -q -E "CONFIG_STACK_VALIDATION=y|CONFIG_HAVE_OBJTOOL=y" ${KERNEL_BUILD_DIR}/.config && [ -d "tools/objtool" ]; then - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" tools/objtool || true + eval "${MAKE_CMD_PREFIX} tools/objtool" || true fi if grep -q "CONFIG_MODULES=y" ${KERNEL_BUILD_DIR}/.config; then - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" modules_prepare + eval "${MAKE_CMDPREFIX} modules_prepare" fi fi + # Stop tracing set +x } +compile_dtbs() { + prefix="arch/${ARCH}/boot/dts/" + for dtb in ${DTB_FILES}; do + # Check if the path is relative (starts with arch/${ARCH}/) + # and if it does then extract the relative path from the architecture specific path + if [[ "${dtb}" == "${prefix}"* ]]; then + relative_dtb_path="${dtb#${prefix}}" + else + # directly use the given DTB if not prefixed with arch/${ARCH}/" + relative_dtb_path="${dtb}" + fi + + # Check if it's a .dts file + if [[ "${relative_dtb_path}" == *.dts ]]; then + # Change .dts to .dtb + relative_dtb_path="${relative_dtb_path%.dts}.dtb" + fi + + eval "${MAKE_CMD_PREFIX} ${relative_dtb_path}" + if [ $? -ne 0 ]; then + echo "${dtb} failed to compile" >&2 + exit 1 + fi + done +} + print_settings() { cat <<EOF # Build settings: diff --git a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl index 00011517..fd2d3b5b 100644 --- a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl +++ b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl @@ -124,7 +124,36 @@ install_hooks() { install_dtbs() { [ -n "${CONFIG_OF}" ] || return 0 - ${MAKE} O=${O} INSTALL_DTBS_PATH=${deb_img_dir}/usr/lib/linux-image-${krel} dtbs_install + prefix="arch/${ARCH}/boot/dts/" + dtb_bsp_dir="" + src_dir="" + dst_dir="" + + # the assumption is that there's only one BSP in the DTB_FILES + for dtb in ${DTB_FILES}; do + dtb_bsp_dir=$(dirname "${dtb}") + if [ "${dtb_bsp_dir}" == "${prefix}" ]; then + continue + elif [[ "${dtb_bsp_dir}" == "${prefix}"* ]]; then + # get the relative BSP directory + dtb_bsp_dir="${dtb_bsp_dir#"${prefix}"}" + fi + + if [ -n "${dtb_bsp_dir}" ]; then + break + fi + done + + src_dir="${O}/${prefix}${dtb_bsp_dir}" + dst_dir="${deb_img_dir}/usr/lib/linux-image-${krel}/${dtb_bsp_dir}" + mkdir -p "${dst_dir}" + + if find "${src_dir}" -maxdepth 1 -name '*.dtb' | grep -q .; then + cp "${src_dir}"/*.dtb "${dst_dir}/" + echo "All dtbs files have been copied to ${src_dir}." + else + echo "No dtb files found!" + fi } install_kmods() { diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc index 6aa70fd3..cac3c91a 100644 --- a/meta/recipes-kernel/linux/linux-custom.inc +++ b/meta/recipes-kernel/linux/linux-custom.inc @@ -76,6 +76,7 @@ TEMPLATE_VARS += " \ KERNEL_DEBIAN_DEPENDS \ KERNEL_BUILD_DIR \ KERNEL_FILE \ + KERNEL_IMAGETYPE \ KERNEL_HEADERS_DEBIAN_DEPENDS \ LINUX_VERSION_EXTENSION \ KERNEL_NAME_PROVIDED \ @@ -84,12 +85,17 @@ TEMPLATE_VARS += " \ KAFLAGS \ DISTRIBUTOR \ KERNEL_EXTRA_BUILDARGS \ + DTB_FILES \ " inherit dpkg inherit template inherit kbuildtarget +DTB_FILES ?="" + +KERNEL_IMAGETYPE ?= "" + # Add custom cflags to the kernel build KCFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." KAFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." -- 2.39.2 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240802114105.1767411-1-nicusor.huhulea%40siemens.com. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH] linux-custom: compile only those specified dtbs 2024-08-02 11:41 [PATCH] linux-custom: compile only those specified dtbs 'Nicusor Huhulea' via isar-users @ 2024-08-02 17:03 ` 'Jan Kiszka' via isar-users 2024-08-08 12:41 ` 'nicusor.huhulea@siemens.com' via isar-users 2024-08-09 16:57 ` 'Jan Kiszka' via isar-users 2024-08-09 10:59 ` [PATCH v2] " 'Nicusor Huhulea' via isar-users 2024-08-13 11:55 ` [PATCH 1/3] linux-custom: optimization on kernel's make command 'Nicusor Huhulea' via isar-users 2 siblings, 2 replies; 17+ messages in thread From: 'Jan Kiszka' via isar-users @ 2024-08-02 17:03 UTC (permalink / raw) To: Nicusor Huhulea, isar-users On 02.08.24 13:41, 'Nicusor Huhulea' via isar-users wrote: > The current implementation makes a full kernel build and that means it will > compile all dtbs specified by the bsp makefile e.g arch/arm64/boot/dts/ti/Makefile > Currently the required dtbs are given by the DTB_FILES. > > These changes are checking if the DTB_FILES is specified, compile only those > and avoids compiling all the dtbs from the bsp directory. Otherwise build all dtbs > specified by the kernel source. > Preventing the automatic dtb builds is made by using target specific build > commands e.g KERNEL_IMAGETYPE > The compilation of the dtbs accepts the following formats e.g: > a) full path from the architecture directory: > arch/${ARCH}/boot/dts/bsp_dir/test.dts > arch/${ARCH}/boot/dts/bsp_dir/test.dtb > b) relative path from the dts directory: bsp_dir/test.dts > c) device tree blob file: bsp_dir/test.dtb > Compilation of the dtbs is somewhat similar to KERNEL_DEVICETREE from OE when > it comes to declaring the path to the dts, but for now in the DTB_FILES variable > only Device Tree Blobs(.dtb) needs to be declared due to the fact that is being used > in do_copy_boot_files task. Can you provide some numbers on how much can be saved this way? Time or (I rather assume) space? > > Signed-off-by: Nicusor Huhulea <nicusor.huhulea@siemens.com> > --- > meta-isar/conf/machine/imx6-sabrelite.conf | 4 +- > meta-isar/conf/machine/phyboard-mira.conf | 4 +- > .../linux/files/debian/isar/build.tmpl | 43 ++++++++++++++++--- > .../linux/files/debian/isar/install.tmpl | 31 ++++++++++++- > meta/recipes-kernel/linux/linux-custom.inc | 6 +++ > 5 files changed, 80 insertions(+), 8 deletions(-) > > diff --git a/meta-isar/conf/machine/imx6-sabrelite.conf b/meta-isar/conf/machine/imx6-sabrelite.conf > index e48823d6..6ca0a907 100644 > --- a/meta-isar/conf/machine/imx6-sabrelite.conf > +++ b/meta-isar/conf/machine/imx6-sabrelite.conf > @@ -15,4 +15,6 @@ MKUBIFS_ARGS := "-m 0x1000 -e 0x3e000 -c 1500" > UBINIZE_ARGS = "-vv -m 0x1000 -p 0x40000" > IMAGE_FSTYPES ?= "ubi-ubifs" > > -DTB_FILES = "imx6q-sabrelite.dtb" > +KERNEL_IMAGETYPE = "zImage" > + > +DTB_FILES = "nxp/imx/imx6q-sabrelite.dtb" > diff --git a/meta-isar/conf/machine/phyboard-mira.conf b/meta-isar/conf/machine/phyboard-mira.conf > index feec4e54..d64b4624 100644 > --- a/meta-isar/conf/machine/phyboard-mira.conf > +++ b/meta-isar/conf/machine/phyboard-mira.conf > @@ -16,10 +16,12 @@ MKUBIFS_ARGS := "-m 0x800 -e 0x1f000 -c 8012" > UBINIZE_ARGS = "-vv -m 0x800 -p 0x20000" > IMAGE_FSTYPES ?= "ubi ubifs" > > -DTB_FILES = "imx6q-phytec-mira-rdk-nand.dtb" > +DTB_FILES = "nxp/imx/imx6q-phytec-mira-rdk-nand.dtb" This change is only correct when also the kernel is 6.5 or newer (i.e. when the dts folder of arm was split into vendors) - which it was in fact. So, this is a separate fix that should be factored out. > > IMAGE_INSTALL += "barebox" > > +KERNEL_IMAGETYPE = "zImage" > + > BAREBOX_CONFIG = "imx_v7_defconfig" > BAREBOX_ENV = "phytec-mira-env" > BAREBOX_BASE_BIN = "barebox-phytec-phycore-imx6q-som-nand-1gib" > diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > index bafc5ca4..b18c8f20 100644 > --- a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > +++ b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > @@ -21,22 +21,55 @@ do_build() { > KR=$(${MAKE} O=${KERNEL_BUILD_DIR} -s --no-print-directory kernelrelease) > sed -i "s/@KR@/${KR}/g" ${S}/debian/control ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.* > > - if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then # Build kernel scripts and tools > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" > + MAKE_CMD_PREFIX="${MAKE} O=\"${KERNEL_BUILD_DIR}\" ${PARALLEL_MAKE} KCFLAGS=\"${KCFLAGS}\" KAFLAGS=\"${KAFLAGS}\"" > + if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then > + if [ -n "${KERNEL_IMAGETYPE}" ]; then # Override the default compilation > + eval "${MAKE_CMD_PREFIX} ${KERNEL_EXTRA_BUILDARGS} ${KERNEL_IMAGETYPE} modules" Please avoid eval whenever possible, it can give surprising results when the provided variables change. Why do we need it at all? Why not something like ${MAKE} ${MAKE_COMMON_ARGS} ... ? > + compile_dtbs > + else # Full build > + eval "${MAKE_CMD_PREFIX} ${KERNEL_EXTRA_BUILDARGS}" > + fi > elif echo "${DEB_BUILD_PROFILES}" | grep -q "kbuild"; then # Build kernel scripts and tools > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" scripts > + eval "${MAKE_CMD_PREFIX} ${KERNEL_EXTRA_BUILDARGS} scripts" > if grep -q -E "CONFIG_STACK_VALIDATION=y|CONFIG_HAVE_OBJTOOL=y" ${KERNEL_BUILD_DIR}/.config && [ -d "tools/objtool" ]; then > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" tools/objtool || true > + eval "${MAKE_CMD_PREFIX} tools/objtool" || true > fi > if grep -q "CONFIG_MODULES=y" ${KERNEL_BUILD_DIR}/.config; then > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" modules_prepare > + eval "${MAKE_CMDPREFIX} modules_prepare" > fi > fi > > + Stray newline. > # Stop tracing > set +x > } > > +compile_dtbs() { > + prefix="arch/${ARCH}/boot/dts/" > + for dtb in ${DTB_FILES}; do > + # Check if the path is relative (starts with arch/${ARCH}/) > + # and if it does then extract the relative path from the architecture specific path > + if [[ "${dtb}" == "${prefix}"* ]]; then > + relative_dtb_path="${dtb#${prefix}}" > + else > + # directly use the given DTB if not prefixed with arch/${ARCH}/" > + relative_dtb_path="${dtb}" > + fi > + > + # Check if it's a .dts file > + if [[ "${relative_dtb_path}" == *.dts ]]; then > + # Change .dts to .dtb > + relative_dtb_path="${relative_dtb_path%.dts}.dtb" > + fi > + > + eval "${MAKE_CMD_PREFIX} ${relative_dtb_path}" > + if [ $? -ne 0 ]; then > + echo "${dtb} failed to compile" >&2 > + exit 1 > + fi > + done > +} > + > print_settings() { > cat <<EOF > # Build settings: > diff --git a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > index 00011517..fd2d3b5b 100644 > --- a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > +++ b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > @@ -124,7 +124,36 @@ install_hooks() { > > install_dtbs() { > [ -n "${CONFIG_OF}" ] || return 0 > - ${MAKE} O=${O} INSTALL_DTBS_PATH=${deb_img_dir}/usr/lib/linux-image-${krel} dtbs_install > + prefix="arch/${ARCH}/boot/dts/" > + dtb_bsp_dir="" > + src_dir="" > + dst_dir="" > + > + # the assumption is that there's only one BSP in the DTB_FILES > + for dtb in ${DTB_FILES}; do > + dtb_bsp_dir=$(dirname "${dtb}") > + if [ "${dtb_bsp_dir}" == "${prefix}" ]; then > + continue > + elif [[ "${dtb_bsp_dir}" == "${prefix}"* ]]; then > + # get the relative BSP directory > + dtb_bsp_dir="${dtb_bsp_dir#"${prefix}"}" > + fi > + > + if [ -n "${dtb_bsp_dir}" ]; then > + break > + fi > + done > + > + src_dir="${O}/${prefix}${dtb_bsp_dir}" > + dst_dir="${deb_img_dir}/usr/lib/linux-image-${krel}/${dtb_bsp_dir}" > + mkdir -p "${dst_dir}" > + > + if find "${src_dir}" -maxdepth 1 -name '*.dtb' | grep -q .; then > + cp "${src_dir}"/*.dtb "${dst_dir}/" > + echo "All dtbs files have been copied to ${src_dir}." > + else > + echo "No dtb files found!" > + fi > } > > install_kmods() { > diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc > index 6aa70fd3..cac3c91a 100644 > --- a/meta/recipes-kernel/linux/linux-custom.inc > +++ b/meta/recipes-kernel/linux/linux-custom.inc > @@ -76,6 +76,7 @@ TEMPLATE_VARS += " \ > KERNEL_DEBIAN_DEPENDS \ > KERNEL_BUILD_DIR \ > KERNEL_FILE \ > + KERNEL_IMAGETYPE \ > KERNEL_HEADERS_DEBIAN_DEPENDS \ > LINUX_VERSION_EXTENSION \ > KERNEL_NAME_PROVIDED \ > @@ -84,12 +85,17 @@ TEMPLATE_VARS += " \ > KAFLAGS \ > DISTRIBUTOR \ > KERNEL_EXTRA_BUILDARGS \ > + DTB_FILES \ > " > > inherit dpkg > inherit template > inherit kbuildtarget > > +DTB_FILES ?="" Missing space after ?=. > + > +KERNEL_IMAGETYPE ?= "" > + > # Add custom cflags to the kernel build > KCFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." > KAFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." Jan -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/43250db6-2540-48ee-8ba6-afe3fd9b56f9%40web.de. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH] linux-custom: compile only those specified dtbs 2024-08-02 17:03 ` 'Jan Kiszka' via isar-users @ 2024-08-08 12:41 ` 'nicusor.huhulea@siemens.com' via isar-users 2024-08-09 16:57 ` 'Jan Kiszka' via isar-users 1 sibling, 0 replies; 17+ messages in thread From: 'nicusor.huhulea@siemens.com' via isar-users @ 2024-08-08 12:41 UTC (permalink / raw) To: Jan Kiszka, isar-users ________________________________________ From: Jan Kiszka <jan.kiszka@web.de> Sent: Friday, August 2, 2024 8:03 PM To: Huhulea, Nicusor Liviu (DI CTO FDS CES LX SVCS); isar-users@googlegroups.com Subject: Re: [PATCH] linux-custom: compile only those specified dtbs On 02.08.24 13:41, 'Nicusor Huhulea' via isar-users wrote: > The current implementation makes a full kernel build and that means it will > compile all dtbs specified by the bsp makefile e.g arch/arm64/boot/dts/ti/Makefile > Currently the required dtbs are given by the DTB_FILES. > > These changes are checking if the DTB_FILES is specified, compile only those > and avoids compiling all the dtbs from the bsp directory. Otherwise build all dtbs > specified by the kernel source. > Preventing the automatic dtb builds is made by using target specific build > commands e.g KERNEL_IMAGETYPE > The compilation of the dtbs accepts the following formats e.g: > a) full path from the architecture directory: > arch/${ARCH}/boot/dts/bsp_dir/test.dts > arch/${ARCH}/boot/dts/bsp_dir/test.dtb > b) relative path from the dts directory: bsp_dir/test.dts > c) device tree blob file: bsp_dir/test.dtb > Compilation of the dtbs is somewhat similar to KERNEL_DEVICETREE from OE when > it comes to declaring the path to the dts, but for now in the DTB_FILES variable > only Device Tree Blobs(.dtb) needs to be declared due to the fact that is being used > in do_copy_boot_files task. Can you provide some numbers on how much can be saved this way? Time or (I rather assume) space? Yes, I can provide these numbers and the answers for all the other queries in the next update. > > Signed-off-by: Nicusor Huhulea <nicusor.huhulea@siemens.com> > --- > meta-isar/conf/machine/imx6-sabrelite.conf | 4 +- > meta-isar/conf/machine/phyboard-mira.conf | 4 +- > .../linux/files/debian/isar/build.tmpl | 43 ++++++++++++++++--- > .../linux/files/debian/isar/install.tmpl | 31 ++++++++++++- > meta/recipes-kernel/linux/linux-custom.inc | 6 +++ > 5 files changed, 80 insertions(+), 8 deletions(-) > > diff --git a/meta-isar/conf/machine/imx6-sabrelite.conf b/meta-isar/conf/machine/imx6-sabrelite.conf > index e48823d6..6ca0a907 100644 > --- a/meta-isar/conf/machine/imx6-sabrelite.conf > +++ b/meta-isar/conf/machine/imx6-sabrelite.conf > @@ -15,4 +15,6 @@ MKUBIFS_ARGS := "-m 0x1000 -e 0x3e000 -c 1500" > UBINIZE_ARGS = "-vv -m 0x1000 -p 0x40000" > IMAGE_FSTYPES ?= "ubi-ubifs" > > -DTB_FILES = "imx6q-sabrelite.dtb" > +KERNEL_IMAGETYPE = "zImage" > + > +DTB_FILES = "nxp/imx/imx6q-sabrelite.dtb" > diff --git a/meta-isar/conf/machine/phyboard-mira.conf b/meta-isar/conf/machine/phyboard-mira.conf > index feec4e54..d64b4624 100644 > --- a/meta-isar/conf/machine/phyboard-mira.conf > +++ b/meta-isar/conf/machine/phyboard-mira.conf > @@ -16,10 +16,12 @@ MKUBIFS_ARGS := "-m 0x800 -e 0x1f000 -c 8012" > UBINIZE_ARGS = "-vv -m 0x800 -p 0x20000" > IMAGE_FSTYPES ?= "ubi ubifs" > > -DTB_FILES = "imx6q-phytec-mira-rdk-nand.dtb" > +DTB_FILES = "nxp/imx/imx6q-phytec-mira-rdk-nand.dtb" This change is only correct when also the kernel is 6.5 or newer (i.e. when the dts folder of arm was split into vendors) - which it was in fact. So, this is a separate fix that should be factored out. > > IMAGE_INSTALL += "barebox" > > +KERNEL_IMAGETYPE = "zImage" > + > BAREBOX_CONFIG = "imx_v7_defconfig" > BAREBOX_ENV = "phytec-mira-env" > BAREBOX_BASE_BIN = "barebox-phytec-phycore-imx6q-som-nand-1gib" > diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > index bafc5ca4..b18c8f20 100644 > --- a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > +++ b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > @@ -21,22 +21,55 @@ do_build() { > KR=$(${MAKE} O=${KERNEL_BUILD_DIR} -s --no-print-directory kernelrelease) > sed -i "s/@KR@/${KR}/g" ${S}/debian/control ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.* > > - if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then # Build kernel scripts and tools > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" > + MAKE_CMD_PREFIX="${MAKE} O=\"${KERNEL_BUILD_DIR}\" ${PARALLEL_MAKE} KCFLAGS=\"${KCFLAGS}\" KAFLAGS=\"${KAFLAGS}\"" > + if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then > + if [ -n "${KERNEL_IMAGETYPE}" ]; then # Override the default compilation > + eval "${MAKE_CMD_PREFIX} ${KERNEL_EXTRA_BUILDARGS} ${KERNEL_IMAGETYPE} modules" Please avoid eval whenever possible, it can give surprising results when the provided variables change. Why do we need it at all? Why not something like ${MAKE} ${MAKE_COMMON_ARGS} ... ? > + compile_dtbs > + else # Full build > + eval "${MAKE_CMD_PREFIX} ${KERNEL_EXTRA_BUILDARGS}" > + fi > elif echo "${DEB_BUILD_PROFILES}" | grep -q "kbuild"; then # Build kernel scripts and tools > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" scripts > + eval "${MAKE_CMD_PREFIX} ${KERNEL_EXTRA_BUILDARGS} scripts" > if grep -q -E "CONFIG_STACK_VALIDATION=y|CONFIG_HAVE_OBJTOOL=y" ${KERNEL_BUILD_DIR}/.config && [ -d "tools/objtool" ]; then > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" tools/objtool || true > + eval "${MAKE_CMD_PREFIX} tools/objtool" || true > fi > if grep -q "CONFIG_MODULES=y" ${KERNEL_BUILD_DIR}/.config; then > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" modules_prepare > + eval "${MAKE_CMDPREFIX} modules_prepare" > fi > fi > > + Stray newline. > # Stop tracing > set +x > } > > +compile_dtbs() { > + prefix="arch/${ARCH}/boot/dts/" > + for dtb in ${DTB_FILES}; do > + # Check if the path is relative (starts with arch/${ARCH}/) > + # and if it does then extract the relative path from the architecture specific path > + if [[ "${dtb}" == "${prefix}"* ]]; then > + relative_dtb_path="${dtb#${prefix}}" > + else > + # directly use the given DTB if not prefixed with arch/${ARCH}/" > + relative_dtb_path="${dtb}" > + fi > + > + # Check if it's a .dts file > + if [[ "${relative_dtb_path}" == *.dts ]]; then > + # Change .dts to .dtb > + relative_dtb_path="${relative_dtb_path%.dts}.dtb" > + fi > + > + eval "${MAKE_CMD_PREFIX} ${relative_dtb_path}" > + if [ $? -ne 0 ]; then > + echo "${dtb} failed to compile" >&2 > + exit 1 > + fi > + done > +} > + > print_settings() { > cat <<EOF > # Build settings: > diff --git a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > index 00011517..fd2d3b5b 100644 > --- a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > +++ b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > @@ -124,7 +124,36 @@ install_hooks() { > > install_dtbs() { > [ -n "${CONFIG_OF}" ] || return 0 > - ${MAKE} O=${O} INSTALL_DTBS_PATH=${deb_img_dir}/usr/lib/linux-image-${krel} dtbs_install > + prefix="arch/${ARCH}/boot/dts/" > + dtb_bsp_dir="" > + src_dir="" > + dst_dir="" > + > + # the assumption is that there's only one BSP in the DTB_FILES > + for dtb in ${DTB_FILES}; do > + dtb_bsp_dir=$(dirname "${dtb}") > + if [ "${dtb_bsp_dir}" == "${prefix}" ]; then > + continue > + elif [[ "${dtb_bsp_dir}" == "${prefix}"* ]]; then > + # get the relative BSP directory > + dtb_bsp_dir="${dtb_bsp_dir#"${prefix}"}" > + fi > + > + if [ -n "${dtb_bsp_dir}" ]; then > + break > + fi > + done > + > + src_dir="${O}/${prefix}${dtb_bsp_dir}" > + dst_dir="${deb_img_dir}/usr/lib/linux-image-${krel}/${dtb_bsp_dir}" > + mkdir -p "${dst_dir}" > + > + if find "${src_dir}" -maxdepth 1 -name '*.dtb' | grep -q .; then > + cp "${src_dir}"/*.dtb "${dst_dir}/" > + echo "All dtbs files have been copied to ${src_dir}." > + else > + echo "No dtb files found!" > + fi > } > > install_kmods() { > diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc > index 6aa70fd3..cac3c91a 100644 > --- a/meta/recipes-kernel/linux/linux-custom.inc > +++ b/meta/recipes-kernel/linux/linux-custom.inc > @@ -76,6 +76,7 @@ TEMPLATE_VARS += " \ > KERNEL_DEBIAN_DEPENDS \ > KERNEL_BUILD_DIR \ > KERNEL_FILE \ > + KERNEL_IMAGETYPE \ > KERNEL_HEADERS_DEBIAN_DEPENDS \ > LINUX_VERSION_EXTENSION \ > KERNEL_NAME_PROVIDED \ > @@ -84,12 +85,17 @@ TEMPLATE_VARS += " \ > KAFLAGS \ > DISTRIBUTOR \ > KERNEL_EXTRA_BUILDARGS \ > + DTB_FILES \ > " > > inherit dpkg > inherit template > inherit kbuildtarget > > +DTB_FILES ?="" Missing space after ?=. > + > +KERNEL_IMAGETYPE ?= "" > + > # Add custom cflags to the kernel build > KCFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." > KAFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." Jan -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/DB3PR10MB690846A79F483F7E116F30DDE6B92%40DB3PR10MB6908.EURPRD10.PROD.OUTLOOK.COM. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH] linux-custom: compile only those specified dtbs 2024-08-02 17:03 ` 'Jan Kiszka' via isar-users 2024-08-08 12:41 ` 'nicusor.huhulea@siemens.com' via isar-users @ 2024-08-09 16:57 ` 'Jan Kiszka' via isar-users 1 sibling, 0 replies; 17+ messages in thread From: 'Jan Kiszka' via isar-users @ 2024-08-09 16:57 UTC (permalink / raw) To: Jan Kiszka, Nicusor Huhulea, isar-users On 02.08.24 19:03, 'Jan Kiszka' via isar-users wrote: > On 02.08.24 13:41, 'Nicusor Huhulea' via isar-users wrote: >> The current implementation makes a full kernel build and that means it will >> compile all dtbs specified by the bsp makefile e.g arch/arm64/boot/dts/ti/Makefile >> Currently the required dtbs are given by the DTB_FILES. >> >> These changes are checking if the DTB_FILES is specified, compile only those >> and avoids compiling all the dtbs from the bsp directory. Otherwise build all dtbs >> specified by the kernel source. >> Preventing the automatic dtb builds is made by using target specific build >> commands e.g KERNEL_IMAGETYPE >> The compilation of the dtbs accepts the following formats e.g: >> a) full path from the architecture directory: >> arch/${ARCH}/boot/dts/bsp_dir/test.dts >> arch/${ARCH}/boot/dts/bsp_dir/test.dtb >> b) relative path from the dts directory: bsp_dir/test.dts >> c) device tree blob file: bsp_dir/test.dtb >> Compilation of the dtbs is somewhat similar to KERNEL_DEVICETREE from OE when >> it comes to declaring the path to the dts, but for now in the DTB_FILES variable >> only Device Tree Blobs(.dtb) needs to be declared due to the fact that is being used >> in do_copy_boot_files task. > > Can you provide some numbers on how much can be saved this way? Time or > (I rather assume) space? > >> >> Signed-off-by: Nicusor Huhulea <nicusor.huhulea@siemens.com> >> --- >> meta-isar/conf/machine/imx6-sabrelite.conf | 4 +- >> meta-isar/conf/machine/phyboard-mira.conf | 4 +- >> .../linux/files/debian/isar/build.tmpl | 43 ++++++++++++++++--- >> .../linux/files/debian/isar/install.tmpl | 31 ++++++++++++- >> meta/recipes-kernel/linux/linux-custom.inc | 6 +++ >> 5 files changed, 80 insertions(+), 8 deletions(-) >> >> diff --git a/meta-isar/conf/machine/imx6-sabrelite.conf b/meta-isar/conf/machine/imx6-sabrelite.conf >> index e48823d6..6ca0a907 100644 >> --- a/meta-isar/conf/machine/imx6-sabrelite.conf >> +++ b/meta-isar/conf/machine/imx6-sabrelite.conf >> @@ -15,4 +15,6 @@ MKUBIFS_ARGS := "-m 0x1000 -e 0x3e000 -c 1500" >> UBINIZE_ARGS = "-vv -m 0x1000 -p 0x40000" >> IMAGE_FSTYPES ?= "ubi-ubifs" >> >> -DTB_FILES = "imx6q-sabrelite.dtb" >> +KERNEL_IMAGETYPE = "zImage" >> + >> +DTB_FILES = "nxp/imx/imx6q-sabrelite.dtb" >> diff --git a/meta-isar/conf/machine/phyboard-mira.conf b/meta-isar/conf/machine/phyboard-mira.conf >> index feec4e54..d64b4624 100644 >> --- a/meta-isar/conf/machine/phyboard-mira.conf >> +++ b/meta-isar/conf/machine/phyboard-mira.conf >> @@ -16,10 +16,12 @@ MKUBIFS_ARGS := "-m 0x800 -e 0x1f000 -c 8012" >> UBINIZE_ARGS = "-vv -m 0x800 -p 0x20000" >> IMAGE_FSTYPES ?= "ubi ubifs" >> >> -DTB_FILES = "imx6q-phytec-mira-rdk-nand.dtb" >> +DTB_FILES = "nxp/imx/imx6q-phytec-mira-rdk-nand.dtb" > > This change is only correct when also the kernel is 6.5 or newer (i.e. > when the dts folder of arm was split into vendors) - which it was in > fact. So, this is a separate fix that should be factored out. > I think you forgot to send this as separate patch, don't you? Jan -- Siemens AG, Technology Linux Expert Center -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/45999d3d-6432-425f-a909-c4a034e11992%40siemens.com. ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v2] linux-custom: compile only those specified dtbs 2024-08-02 11:41 [PATCH] linux-custom: compile only those specified dtbs 'Nicusor Huhulea' via isar-users 2024-08-02 17:03 ` 'Jan Kiszka' via isar-users @ 2024-08-09 10:59 ` 'Nicusor Huhulea' via isar-users 2024-08-09 10:59 ` [PATCH] " 'Nicusor Huhulea' via isar-users 2024-08-13 11:55 ` [PATCH 1/3] linux-custom: optimization on kernel's make command 'Nicusor Huhulea' via isar-users 2 siblings, 1 reply; 17+ messages in thread From: 'Nicusor Huhulea' via isar-users @ 2024-08-09 10:59 UTC (permalink / raw) To: isar-users; +Cc: Nicusor Huhulea Hello, These changes are trying to solve or offer an alternative to the full build(default build) compilation of the kernel which leads to the compilation of all dtbs. This automatic compilation is due to the kernel configuration that automatically includes the compilation of dtbs in the default build process and the makefiles rules in the bsp directories that builds DTBs along with the kernel by default. Compiling the linux kernel using a specific target stops the automatic build of the dtbs, explicitly specifying dtbs files as make targets will instruct the build process to compile only those files. Certain speed and disk space usage test were made. These tests targeted the phyboard-mira(on Kernel Post-6.x and Kernel Pre-6.x 4.19.88) and iot2050(Kernel Post-6.x): specific target: Build needed 00:02:08, 2219688k disk space - linux-phy kernel 6.x full build: Build needed 00:02:14, 2271528k disk space - linux-phy kernel 6.x specific build : Build needed 00:02:19, 2355400k disk space - linux-iot2050 kernel 6.x full build: Build needed 00:02:23, 2356828k disk space - liux-iot2050 kernel 6.x On linux-phy Kernel 6.x the full build took 6 seconds longer than the specific target build and used approximately 50.6 MB more disk space than the specific target build. The iot2050 data is somewhat similar. Based on the data from the dpkg_build.log the benefits of using specific targets reduces build times and resource efficiency. However there can be some potential pitfalls associated with this approach. When the engineer is not intimate with the bsp certain kernel configurations or some dependencies may not be satisfied. And this is why this is enabled only when KERNEL_IMAGETYPE is present.(although the iot2050 builds went well on my side) Nicu -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240809105933.3040701-1-nicusor.huhulea%40siemens.com. ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH] linux-custom: compile only those specified dtbs 2024-08-09 10:59 ` [PATCH v2] " 'Nicusor Huhulea' via isar-users @ 2024-08-09 10:59 ` 'Nicusor Huhulea' via isar-users 2024-08-09 16:56 ` 'Jan Kiszka' via isar-users 0 siblings, 1 reply; 17+ messages in thread From: 'Nicusor Huhulea' via isar-users @ 2024-08-09 10:59 UTC (permalink / raw) To: isar-users; +Cc: Nicusor Huhulea The current implementation makes a full kernel build and that means it will compile all dtbs specified by the bsp makefile e.g arch/arm64/boot/dts/ti/Makefile Currently the required dtbs are given by the DTB_FILES. These changes are checking if the DTB_FILES is specified, compile only those and avoids compiling all the dtbs from the bsp directory. Otherwise build all dtbs specified by the kernel source. Preventing the automatic dtb builds is made by using target specific build commands e.g KERNEL_IMAGETYPE The compilation of the dtbs accepts the following formats e.g: Kernel Post 6.x: a) full path from the architecture directory: arch/${ARCH}/boot/dts/bsp_dir/test.dts arch/${ARCH}/boot/dts/bsp_dir/test.dtb b) relative path from the dts directory: bsp_dir/test.dts c) device tree blob file: bsp_dir/test.dtb Kernel Pre 6.x: The same as on Kernel Post 6.x but without the bsp_dir The differences when building a full build and when using a specific target build are:(tests were made on the machine phyboard-mira on 6.x and 4.x versions) specific target: Build needed 00:02:08, 2219688k disk space linux-phy 6.x full build: Build needed 00:02:14, 2271528k disk space linux-phy 6.x The time difference: the full build took 6s longer than the specific target build The disk space difference: the full build used 50.6MB more disk space than the specific target build. In conclusion the specific target build is slightly faster and uses less disk space. For now the DTB_FILES variable should accept only Device Tree Blobs(.dtb) because some areas still depends on it e.g: do_copy_boot_files, plugins Signed-off-by: Nicusor Huhulea <nicusor.huhulea@siemens.com> --- meta-isar/conf/machine/imx6-sabrelite.conf | 2 + meta-isar/conf/machine/phyboard-mira.conf | 2 + .../linux/files/debian/isar/build.tmpl | 63 +++++++++++++++++-- .../linux/files/debian/isar/install.tmpl | 12 +++- meta/recipes-kernel/linux/linux-custom.inc | 6 ++ 5 files changed, 79 insertions(+), 6 deletions(-) diff --git a/meta-isar/conf/machine/imx6-sabrelite.conf b/meta-isar/conf/machine/imx6-sabrelite.conf index e48823d6..bffebdc1 100644 --- a/meta-isar/conf/machine/imx6-sabrelite.conf +++ b/meta-isar/conf/machine/imx6-sabrelite.conf @@ -15,4 +15,6 @@ MKUBIFS_ARGS := "-m 0x1000 -e 0x3e000 -c 1500" UBINIZE_ARGS = "-vv -m 0x1000 -p 0x40000" IMAGE_FSTYPES ?= "ubi-ubifs" +KERNEL_IMAGETYPE = "zImage" + DTB_FILES = "imx6q-sabrelite.dtb" diff --git a/meta-isar/conf/machine/phyboard-mira.conf b/meta-isar/conf/machine/phyboard-mira.conf index feec4e54..1d2b3e1c 100644 --- a/meta-isar/conf/machine/phyboard-mira.conf +++ b/meta-isar/conf/machine/phyboard-mira.conf @@ -20,6 +20,8 @@ DTB_FILES = "imx6q-phytec-mira-rdk-nand.dtb" IMAGE_INSTALL += "barebox" +KERNEL_IMAGETYPE = "zImage" + BAREBOX_CONFIG = "imx_v7_defconfig" BAREBOX_ENV = "phytec-mira-env" BAREBOX_BASE_BIN = "barebox-phytec-phycore-imx6q-som-nand-1gib" diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl index bafc5ca4..90222adc 100644 --- a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl +++ b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl @@ -21,15 +21,21 @@ do_build() { KR=$(${MAKE} O=${KERNEL_BUILD_DIR} -s --no-print-directory kernelrelease) sed -i "s/@KR@/${KR}/g" ${S}/debian/control ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.* - if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then # Build kernel scripts and tools - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" + MAKE_COMMON_ARGS="O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS=${KCFLAGS} KAFLAGS=${KAFLAGS}" + if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then + if [ -n "${KERNEL_IMAGETYPE}" ]; then # Override the default compilation + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_IMAGETYPE} modules + compile_dtbs + else # Full build + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} + fi elif echo "${DEB_BUILD_PROFILES}" | grep -q "kbuild"; then # Build kernel scripts and tools - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" scripts + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} scripts if grep -q -E "CONFIG_STACK_VALIDATION=y|CONFIG_HAVE_OBJTOOL=y" ${KERNEL_BUILD_DIR}/.config && [ -d "tools/objtool" ]; then - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" tools/objtool || true + ${MAKE} ${MAKE_COMMON_ARGS} tools/objtool || true fi if grep -q "CONFIG_MODULES=y" ${KERNEL_BUILD_DIR}/.config; then - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" modules_prepare + ${MAKE} ${MAKE_COMMON_ARGS} modules_prepare fi fi @@ -37,6 +43,53 @@ do_build() { set +x } +compile_dtbs() { + local prefix="arch/${ARCH}/boot/dts/" + + for dtb in ${DTB_FILES}; do + # Handle the case where a standalone(with no path .dtb/.dts) file may reside within + # a specific BSP directory or not(kernel pre-6.x) + if [[ ("${dtb}" == *.dtb || "${dtb}" == *.dts) && "${dtb}" != */* ]]; then + bsp_dir="" + dts_path="" + if [[ "${dtb}" == *.dtb ]]; then + dts="${dtb%.dtb}.dts" + else + dts="${dtb}" + fi + + # recursively search for the dts file in all possible BSP directories + while IFS= read -r dts_path; do + if [ -n "${dts_path}" ]; then + # Eliminate the prefix and dts to get only the bsp_dir + bsp_dir="${dts_path#${prefix}}" + bsp_dir="${bsp_dir%${dts}}" + + relative_dtb_path="${bsp_dir}${dtb}" + else + relative_dtb_path="${prefix}${dtb}" + fi + done < <(find "${prefix}" -type f -name "${dts}") + + # Check if the path is relative (starts with arch/${ARCH}/) + # and if it does then extract the relative path from the architecture specific path + elif [[ "${dtb}" == "${prefix}"* ]]; then + relative_dtb_path="${dtb#${prefix}}" + else + # directly use the given dtb + relative_dtb_path="${dtb}" + fi + + # Check if it's a .dts file + if [[ "${relative_dtb_path}" == *.dts ]]; then + # Change .dts to .dtb + relative_dtb_path="${relative_dtb_path%.dts}.dtb" + fi + + ${MAKE} ${MAKE_COMMON_ARGS} ${relative_dtb_path} + done +} + print_settings() { cat <<EOF # Build settings: diff --git a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl index 00011517..c0a60f4e 100644 --- a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl +++ b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl @@ -124,7 +124,17 @@ install_hooks() { install_dtbs() { [ -n "${CONFIG_OF}" ] || return 0 - ${MAKE} O=${O} INSTALL_DTBS_PATH=${deb_img_dir}/usr/lib/linux-image-${krel} dtbs_install + prefix="arch/${ARCH}/boot/dts/" + src_dir="${O}/${prefix}" + dst_dir="" + + dst_dir="${deb_img_dir}/usr/lib/linux-image-${krel}/" + find "${src_dir}" -type f -name '*.dtb' | while IFS= read -r dtb_file; do + local relative_path="${dtb_file#"${src_dir}"}" + mkdir -p "${dst_dir}$(dirname "${relative_path}")" + cp "${dtb_file}" "${dst_dir}/${relative_path}" + echo "Copying ${dtb_file} to ${dest_dir}/${relative_path}" + done } install_kmods() { diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc index 6aa70fd3..31cc480b 100644 --- a/meta/recipes-kernel/linux/linux-custom.inc +++ b/meta/recipes-kernel/linux/linux-custom.inc @@ -76,6 +76,7 @@ TEMPLATE_VARS += " \ KERNEL_DEBIAN_DEPENDS \ KERNEL_BUILD_DIR \ KERNEL_FILE \ + KERNEL_IMAGETYPE \ KERNEL_HEADERS_DEBIAN_DEPENDS \ LINUX_VERSION_EXTENSION \ KERNEL_NAME_PROVIDED \ @@ -84,12 +85,17 @@ TEMPLATE_VARS += " \ KAFLAGS \ DISTRIBUTOR \ KERNEL_EXTRA_BUILDARGS \ + DTB_FILES \ " inherit dpkg inherit template inherit kbuildtarget +DTB_FILES ?= "" + +KERNEL_IMAGETYPE ?= "" + # Add custom cflags to the kernel build KCFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." KAFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." -- 2.39.2 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240809105933.3040701-2-nicusor.huhulea%40siemens.com. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH] linux-custom: compile only those specified dtbs 2024-08-09 10:59 ` [PATCH] " 'Nicusor Huhulea' via isar-users @ 2024-08-09 16:56 ` 'Jan Kiszka' via isar-users 2024-08-12 14:44 ` 'nicusor.huhulea@siemens.com' via isar-users 0 siblings, 1 reply; 17+ messages in thread From: 'Jan Kiszka' via isar-users @ 2024-08-09 16:56 UTC (permalink / raw) To: Nicusor Huhulea, isar-users On 09.08.24 12:59, 'Nicusor Huhulea' via isar-users wrote: > The current implementation makes a full kernel build and that means it will > compile all dtbs specified by the bsp makefile e.g arch/arm64/boot/dts/ti/Makefile > Currently the required dtbs are given by the DTB_FILES. > > These changes are checking if the DTB_FILES is specified, compile only those > and avoids compiling all the dtbs from the bsp directory. Otherwise build all dtbs > specified by the kernel source. > Preventing the automatic dtb builds is made by using target specific build > commands e.g KERNEL_IMAGETYPE Fairly subtle optimization. Can't we identify the default image(s) the kernel would build and fill this automatically? > The compilation of the dtbs accepts the following formats e.g: > Kernel Post 6.x: > a) full path from the architecture directory: > arch/${ARCH}/boot/dts/bsp_dir/test.dts > arch/${ARCH}/boot/dts/bsp_dir/test.dtb > b) relative path from the dts directory: bsp_dir/test.dts > c) device tree blob file: bsp_dir/test.dtb > Kernel Pre 6.x: > The same as on Kernel Post 6.x but without the bsp_dir > > The differences when building a full build and when using a specific target > build are:(tests were made on the machine phyboard-mira on 6.x and 4.x versions) > specific target: Build needed 00:02:08, 2219688k disk space linux-phy 6.x > full build: Build needed 00:02:14, 2271528k disk space linux-phy 6.x > The time difference: the full build took 6s longer than the specific target build > The disk space difference: the full build used 50.6MB more disk space than > the specific target build. That 50M is build-time space, right? What's the size difference on the target? > In conclusion the specific target build is slightly faster and uses less > disk space. > > For now the DTB_FILES variable should accept only Device Tree Blobs(.dtb) > because some areas still depends on it e.g: do_copy_boot_files, plugins We are not yet using dtbo files (overlays) built by the kernel in any projects I know, but those would no longer be built then, right? > > Signed-off-by: Nicusor Huhulea <nicusor.huhulea@siemens.com> > --- > meta-isar/conf/machine/imx6-sabrelite.conf | 2 + > meta-isar/conf/machine/phyboard-mira.conf | 2 + > .../linux/files/debian/isar/build.tmpl | 63 +++++++++++++++++-- > .../linux/files/debian/isar/install.tmpl | 12 +++- > meta/recipes-kernel/linux/linux-custom.inc | 6 ++ > 5 files changed, 79 insertions(+), 6 deletions(-) > > diff --git a/meta-isar/conf/machine/imx6-sabrelite.conf b/meta-isar/conf/machine/imx6-sabrelite.conf > index e48823d6..bffebdc1 100644 > --- a/meta-isar/conf/machine/imx6-sabrelite.conf > +++ b/meta-isar/conf/machine/imx6-sabrelite.conf > @@ -15,4 +15,6 @@ MKUBIFS_ARGS := "-m 0x1000 -e 0x3e000 -c 1500" > UBINIZE_ARGS = "-vv -m 0x1000 -p 0x40000" > IMAGE_FSTYPES ?= "ubi-ubifs" > > +KERNEL_IMAGETYPE = "zImage" > + > DTB_FILES = "imx6q-sabrelite.dtb" > diff --git a/meta-isar/conf/machine/phyboard-mira.conf b/meta-isar/conf/machine/phyboard-mira.conf > index feec4e54..1d2b3e1c 100644 > --- a/meta-isar/conf/machine/phyboard-mira.conf > +++ b/meta-isar/conf/machine/phyboard-mira.conf > @@ -20,6 +20,8 @@ DTB_FILES = "imx6q-phytec-mira-rdk-nand.dtb" > > IMAGE_INSTALL += "barebox" > > +KERNEL_IMAGETYPE = "zImage" > + > BAREBOX_CONFIG = "imx_v7_defconfig" > BAREBOX_ENV = "phytec-mira-env" > BAREBOX_BASE_BIN = "barebox-phytec-phycore-imx6q-som-nand-1gib" > diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > index bafc5ca4..90222adc 100644 > --- a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > +++ b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > @@ -21,15 +21,21 @@ do_build() { > KR=$(${MAKE} O=${KERNEL_BUILD_DIR} -s --no-print-directory kernelrelease) > sed -i "s/@KR@/${KR}/g" ${S}/debian/control ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.* > > - if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then # Build kernel scripts and tools > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" > + MAKE_COMMON_ARGS="O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS=${KCFLAGS} KAFLAGS=${KAFLAGS}" > + if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then > + if [ -n "${KERNEL_IMAGETYPE}" ]; then # Override the default compilation > + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_IMAGETYPE} modules You are losing KERNEL_EXTRA_BUILDARGS. > + compile_dtbs > + else # Full build > + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} > + fi > elif echo "${DEB_BUILD_PROFILES}" | grep -q "kbuild"; then # Build kernel scripts and tools > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" scripts > + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} scripts Why this change and those 2 below? They look wrong. > if grep -q -E "CONFIG_STACK_VALIDATION=y|CONFIG_HAVE_OBJTOOL=y" ${KERNEL_BUILD_DIR}/.config && [ -d "tools/objtool" ]; then > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" tools/objtool || true > + ${MAKE} ${MAKE_COMMON_ARGS} tools/objtool || true > fi > if grep -q "CONFIG_MODULES=y" ${KERNEL_BUILD_DIR}/.config; then > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" modules_prepare > + ${MAKE} ${MAKE_COMMON_ARGS} modules_prepare > fi > fi > > @@ -37,6 +43,53 @@ do_build() { > set +x > } > > +compile_dtbs() { > + local prefix="arch/${ARCH}/boot/dts/" > + > + for dtb in ${DTB_FILES}; do > + # Handle the case where a standalone(with no path .dtb/.dts) file may reside within > + # a specific BSP directory or not(kernel pre-6.x) > + if [[ ("${dtb}" == *.dtb || "${dtb}" == *.dts) && "${dtb}" != */* ]]; then > + bsp_dir="" > + dts_path="" > + if [[ "${dtb}" == *.dtb ]]; then > + dts="${dtb%.dtb}.dts" > + else > + dts="${dtb}" > + fi > + > + # recursively search for the dts file in all possible BSP directories > + while IFS= read -r dts_path; do > + if [ -n "${dts_path}" ]; then > + # Eliminate the prefix and dts to get only the bsp_dir > + bsp_dir="${dts_path#${prefix}}" > + bsp_dir="${bsp_dir%${dts}}" > + > + relative_dtb_path="${bsp_dir}${dtb}" > + else > + relative_dtb_path="${prefix}${dtb}" > + fi > + done < <(find "${prefix}" -type f -name "${dts}") > + > + # Check if the path is relative (starts with arch/${ARCH}/) > + # and if it does then extract the relative path from the architecture specific path > + elif [[ "${dtb}" == "${prefix}"* ]]; then > + relative_dtb_path="${dtb#${prefix}}" > + else > + # directly use the given dtb > + relative_dtb_path="${dtb}" > + fi > + > + # Check if it's a .dts file > + if [[ "${relative_dtb_path}" == *.dts ]]; then > + # Change .dts to .dtb > + relative_dtb_path="${relative_dtb_path%.dts}.dtb" > + fi > + > + ${MAKE} ${MAKE_COMMON_ARGS} ${relative_dtb_path} > + done > +} > + > print_settings() { > cat <<EOF > # Build settings: > diff --git a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > index 00011517..c0a60f4e 100644 > --- a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > +++ b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > @@ -124,7 +124,17 @@ install_hooks() { > > install_dtbs() { > [ -n "${CONFIG_OF}" ] || return 0 > - ${MAKE} O=${O} INSTALL_DTBS_PATH=${deb_img_dir}/usr/lib/linux-image-${krel} dtbs_install > + prefix="arch/${ARCH}/boot/dts/" > + src_dir="${O}/${prefix}" > + dst_dir="" > + > + dst_dir="${deb_img_dir}/usr/lib/linux-image-${krel}/" > + find "${src_dir}" -type f -name '*.dtb' | while IFS= read -r dtb_file; do > + local relative_path="${dtb_file#"${src_dir}"}" > + mkdir -p "${dst_dir}$(dirname "${relative_path}")" > + cp "${dtb_file}" "${dst_dir}/${relative_path}" > + echo "Copying ${dtb_file} to ${dest_dir}/${relative_path}" > + done > } > > install_kmods() { > diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc > index 6aa70fd3..31cc480b 100644 > --- a/meta/recipes-kernel/linux/linux-custom.inc > +++ b/meta/recipes-kernel/linux/linux-custom.inc > @@ -76,6 +76,7 @@ TEMPLATE_VARS += " \ > KERNEL_DEBIAN_DEPENDS \ > KERNEL_BUILD_DIR \ > KERNEL_FILE \ > + KERNEL_IMAGETYPE \ > KERNEL_HEADERS_DEBIAN_DEPENDS \ > LINUX_VERSION_EXTENSION \ > KERNEL_NAME_PROVIDED \ > @@ -84,12 +85,17 @@ TEMPLATE_VARS += " \ > KAFLAGS \ > DISTRIBUTOR \ > KERNEL_EXTRA_BUILDARGS \ > + DTB_FILES \ > " > > inherit dpkg > inherit template > inherit kbuildtarget > > +DTB_FILES ?= "" > + > +KERNEL_IMAGETYPE ?= "" > + > # Add custom cflags to the kernel build > KCFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." > KAFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." Quite a lot of new code for rather moderate optimizations. Maybe we should rather filter the built files when installing? That should be doable in 3 lines or so. Jan -- Siemens AG, Technology Linux Expert Center -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/3bd90633-d978-4a10-82dd-b438098b78d4%40siemens.com. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH] linux-custom: compile only those specified dtbs 2024-08-09 16:56 ` 'Jan Kiszka' via isar-users @ 2024-08-12 14:44 ` 'nicusor.huhulea@siemens.com' via isar-users 2024-08-12 15:16 ` 'Jan Kiszka' via isar-users 0 siblings, 1 reply; 17+ messages in thread From: 'nicusor.huhulea@siemens.com' via isar-users @ 2024-08-12 14:44 UTC (permalink / raw) To: Kiszka, Jan, isar-users ________________________________________ From: Kiszka, Jan (T CED) <jan.kiszka@siemens.com> Sent: Friday, August 9, 2024 7:56 PM To: Huhulea, Nicusor Liviu (DI CTO FDS CES LX SVCS); isar-users@googlegroups.com Subject: Re: [PATCH] linux-custom: compile only those specified dtbs On 09.08.24 12:59, 'Nicusor Huhulea' via isar-users wrote: > The current implementation makes a full kernel build and that means it will > compile all dtbs specified by the bsp makefile e.g arch/arm64/boot/dts/ti/Makefile > Currently the required dtbs are given by the DTB_FILES. > > These changes are checking if the DTB_FILES is specified, compile only those > and avoids compiling all the dtbs from the bsp directory. Otherwise build all dtbs > specified by the kernel source. > Preventing the automatic dtb builds is made by using target specific build > commands e.g KERNEL_IMAGETYPE Fairly subtle optimization. Can't we identify the default image(s) the kernel would build and fill this automatically? Yes, we can do that, but I don't think we want that because if there are multiple targets(image types) enabled we wouldn't know what to choose. The reason being the difference between a full build and a specific target build. So far I have identified 2 differences between a full build and a specific target build: one is the dtbs target and the other one is the different image types that are produced in one build session. e.g: for arm the kernel might have the configuration to build both zImage and Image.gz(full-build case) but we need only one image for deployment(specific target build). However, building all of these images every time might not be necessary, which is where the KERNEL_IMAGETYPE comes into the picture hence the reason for these changes. > The compilation of the dtbs accepts the following formats e.g: > Kernel Post 6.x: > a) full path from the architecture directory: > arch/${ARCH}/boot/dts/bsp_dir/test.dts > arch/${ARCH}/boot/dts/bsp_dir/test.dtb > b) relative path from the dts directory: bsp_dir/test.dts > c) device tree blob file: bsp_dir/test.dtb > Kernel Pre 6.x: > The same as on Kernel Post 6.x but without the bsp_dir > > The differences when building a full build and when using a specific target > build are:(tests were made on the machine phyboard-mira on 6.x and 4.x versions) > specific target: Build needed 00:02:08, 2219688k disk space linux-phy 6.x > full build: Build needed 00:02:14, 2271528k disk space linux-phy 6.x > The time difference: the full build took 6s longer than the specific target build > The disk space difference: the full build used 50.6MB more disk space than > the specific target build. That 50M is build-time space, right? What's the size difference on the target? Yes, 50M disk space. I don't have a phyboard-mira at my disposal, but I can try on a iot2050 board to see the difference on the target. > In conclusion the specific target build is slightly faster and uses less > disk space. > > For now the DTB_FILES variable should accept only Device Tree Blobs(.dtb) > because some areas still depends on it e.g: do_copy_boot_files, plugins We are not yet using dtbo files (overlays) built by the kernel in any projects I know, but those would no longer be built then, right? Exactly, but these changes are still keeping the full-build case in place As you said, because I haven't seen any dtbo files around I left this out for the moment and because the changes were starting to be quite big.(can be treated as a follow up on this as compiling the dtbo is in line with the dtb) > > Signed-off-by: Nicusor Huhulea <nicusor.huhulea@siemens.com> > --- > meta-isar/conf/machine/imx6-sabrelite.conf | 2 + > meta-isar/conf/machine/phyboard-mira.conf | 2 + > .../linux/files/debian/isar/build.tmpl | 63 +++++++++++++++++-- > .../linux/files/debian/isar/install.tmpl | 12 +++- > meta/recipes-kernel/linux/linux-custom.inc | 6 ++ > 5 files changed, 79 insertions(+), 6 deletions(-) > > diff --git a/meta-isar/conf/machine/imx6-sabrelite.conf b/meta-isar/conf/machine/imx6-sabrelite.conf > index e48823d6..bffebdc1 100644 > --- a/meta-isar/conf/machine/imx6-sabrelite.conf > +++ b/meta-isar/conf/machine/imx6-sabrelite.conf > @@ -15,4 +15,6 @@ MKUBIFS_ARGS := "-m 0x1000 -e 0x3e000 -c 1500" > UBINIZE_ARGS = "-vv -m 0x1000 -p 0x40000" > IMAGE_FSTYPES ?= "ubi-ubifs" > > +KERNEL_IMAGETYPE = "zImage" > + > DTB_FILES = "imx6q-sabrelite.dtb" > diff --git a/meta-isar/conf/machine/phyboard-mira.conf b/meta-isar/conf/machine/phyboard-mira.conf > index feec4e54..1d2b3e1c 100644 > --- a/meta-isar/conf/machine/phyboard-mira.conf > +++ b/meta-isar/conf/machine/phyboard-mira.conf > @@ -20,6 +20,8 @@ DTB_FILES = "imx6q-phytec-mira-rdk-nand.dtb" > > IMAGE_INSTALL += "barebox" > > +KERNEL_IMAGETYPE = "zImage" > + > BAREBOX_CONFIG = "imx_v7_defconfig" > BAREBOX_ENV = "phytec-mira-env" > BAREBOX_BASE_BIN = "barebox-phytec-phycore-imx6q-som-nand-1gib" > diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > index bafc5ca4..90222adc 100644 > --- a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > +++ b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > @@ -21,15 +21,21 @@ do_build() { > KR=$(${MAKE} O=${KERNEL_BUILD_DIR} -s --no-print-directory kernelrelease) > sed -i "s/@KR@/${KR}/g" ${S}/debian/control ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.* > > - if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then # Build kernel scripts and tools > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" > + MAKE_COMMON_ARGS="O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS=${KCFLAGS} KAFLAGS=${KAFLAGS}" > + if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then > + if [ -n "${KERNEL_IMAGETYPE}" ]; then # Override the default compilation > + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_IMAGETYPE} modules You are losing KERNEL_EXTRA_BUILDARGS. > + compile_dtbs > + else # Full build > + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} > + fi > elif echo "${DEB_BUILD_PROFILES}" | grep -q "kbuild"; then # Build kernel scripts and tools > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" scripts > + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} scripts Why this change and those 2 below? They look wrong. These changes are some optimizations but after carefully inspecting these I saw that I need to fix some ${KERNEL_EXTRA_BUILDARGS} addition and loosing in some areas. > if grep -q -E "CONFIG_STACK_VALIDATION=y|CONFIG_HAVE_OBJTOOL=y" ${KERNEL_BUILD_DIR}/.config && [ -d "tools/objtool" ]; then > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" tools/objtool || true > + ${MAKE} ${MAKE_COMMON_ARGS} tools/objtool || true > fi > if grep -q "CONFIG_MODULES=y" ${KERNEL_BUILD_DIR}/.config; then > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" modules_prepare > + ${MAKE} ${MAKE_COMMON_ARGS} modules_prepare > fi > fi > > @@ -37,6 +43,53 @@ do_build() { > set +x > } > > +compile_dtbs() { > + local prefix="arch/${ARCH}/boot/dts/" > + > + for dtb in ${DTB_FILES}; do > + # Handle the case where a standalone(with no path .dtb/.dts) file may reside within > + # a specific BSP directory or not(kernel pre-6.x) > + if [[ ("${dtb}" == *.dtb || "${dtb}" == *.dts) && "${dtb}" != */* ]]; then > + bsp_dir="" > + dts_path="" > + if [[ "${dtb}" == *.dtb ]]; then > + dts="${dtb%.dtb}.dts" > + else > + dts="${dtb}" > + fi > + > + # recursively search for the dts file in all possible BSP directories > + while IFS= read -r dts_path; do > + if [ -n "${dts_path}" ]; then > + # Eliminate the prefix and dts to get only the bsp_dir > + bsp_dir="${dts_path#${prefix}}" > + bsp_dir="${bsp_dir%${dts}}" > + > + relative_dtb_path="${bsp_dir}${dtb}" > + else > + relative_dtb_path="${prefix}${dtb}" > + fi > + done < <(find "${prefix}" -type f -name "${dts}") > + > + # Check if the path is relative (starts with arch/${ARCH}/) > + # and if it does then extract the relative path from the architecture specific path > + elif [[ "${dtb}" == "${prefix}"* ]]; then > + relative_dtb_path="${dtb#${prefix}}" > + else > + # directly use the given dtb > + relative_dtb_path="${dtb}" > + fi > + > + # Check if it's a .dts file > + if [[ "${relative_dtb_path}" == *.dts ]]; then > + # Change .dts to .dtb > + relative_dtb_path="${relative_dtb_path%.dts}.dtb" > + fi > + > + ${MAKE} ${MAKE_COMMON_ARGS} ${relative_dtb_path} > + done > +} > + > print_settings() { > cat <<EOF > # Build settings: > diff --git a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > index 00011517..c0a60f4e 100644 > --- a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > +++ b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > @@ -124,7 +124,17 @@ install_hooks() { > > install_dtbs() { > [ -n "${CONFIG_OF}" ] || return 0 > - ${MAKE} O=${O} INSTALL_DTBS_PATH=${deb_img_dir}/usr/lib/linux-image-${krel} dtbs_install > + prefix="arch/${ARCH}/boot/dts/" > + src_dir="${O}/${prefix}" > + dst_dir="" > + > + dst_dir="${deb_img_dir}/usr/lib/linux-image-${krel}/" > + find "${src_dir}" -type f -name '*.dtb' | while IFS= read -r dtb_file; do > + local relative_path="${dtb_file#"${src_dir}"}" > + mkdir -p "${dst_dir}$(dirname "${relative_path}")" > + cp "${dtb_file}" "${dst_dir}/${relative_path}" > + echo "Copying ${dtb_file} to ${dest_dir}/${relative_path}" > + done > } > > install_kmods() { > diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc > index 6aa70fd3..31cc480b 100644 > --- a/meta/recipes-kernel/linux/linux-custom.inc > +++ b/meta/recipes-kernel/linux/linux-custom.inc > @@ -76,6 +76,7 @@ TEMPLATE_VARS += " \ > KERNEL_DEBIAN_DEPENDS \ > KERNEL_BUILD_DIR \ > KERNEL_FILE \ > + KERNEL_IMAGETYPE \ > KERNEL_HEADERS_DEBIAN_DEPENDS \ > LINUX_VERSION_EXTENSION \ > KERNEL_NAME_PROVIDED \ > @@ -84,12 +85,17 @@ TEMPLATE_VARS += " \ > KAFLAGS \ > DISTRIBUTOR \ > KERNEL_EXTRA_BUILDARGS \ > + DTB_FILES \ > " > > inherit dpkg > inherit template > inherit kbuildtarget > > +DTB_FILES ?= "" > + > +KERNEL_IMAGETYPE ?= "" > + > # Add custom cflags to the kernel build > KCFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." > KAFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." Quite a lot of new code for rather moderate optimizations. Maybe we should rather filter the built files when installing? That should be doable in 3 lines or so. Jan -- Siemens AG, Technology Linux Expert Center Yes, indeed seems a lot of code but these changes were made having in mind fixing the compilation of those extra dtbs that were not belonging to that specific target bsp. And the optimizations are just a side effect(maybe as a bonus), they are not on the foreground. The reason because it's a lot of code is primarily because I wanted to keep the DTB_FILES declarations as is in the upper layers and bring new features e.g full architecture path, etc. plus this handles a KERNEL Pre 6.x dtb builds -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/DB3PR10MB690833C214CB30C07972D8CFE6852%40DB3PR10MB6908.EURPRD10.PROD.OUTLOOK.COM. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH] linux-custom: compile only those specified dtbs 2024-08-12 14:44 ` 'nicusor.huhulea@siemens.com' via isar-users @ 2024-08-12 15:16 ` 'Jan Kiszka' via isar-users 0 siblings, 0 replies; 17+ messages in thread From: 'Jan Kiszka' via isar-users @ 2024-08-12 15:16 UTC (permalink / raw) To: Huhulea, Nicusor Liviu (DI CTO FDS CES LX SVCS), isar-users Your email client is not yet properly configured to indent / mark citations. That makes it very hard to find and read your replies. Please fix. On 12.08.24 16:44, Huhulea, Nicusor Liviu (DI CTO FDS CES LX SVCS) wrote: > > > ________________________________________ > From: Kiszka, Jan (T CED) <jan.kiszka@siemens.com> > Sent: Friday, August 9, 2024 7:56 PM > To: Huhulea, Nicusor Liviu (DI CTO FDS CES LX SVCS); isar-users@googlegroups.com > Subject: Re: [PATCH] linux-custom: compile only those specified dtbs > > On 09.08.24 12:59, 'Nicusor Huhulea' via isar-users wrote: >> The current implementation makes a full kernel build and that means it will >> compile all dtbs specified by the bsp makefile e.g arch/arm64/boot/dts/ti/Makefile >> Currently the required dtbs are given by the DTB_FILES. >> >> These changes are checking if the DTB_FILES is specified, compile only those >> and avoids compiling all the dtbs from the bsp directory. Otherwise build all dtbs >> specified by the kernel source. >> Preventing the automatic dtb builds is made by using target specific build >> commands e.g KERNEL_IMAGETYPE > > Fairly subtle optimization. Can't we identify the default image(s) the > kernel would build and fill this automatically? > Yes, we can do that, but I don't think we want that because if there are multiple targets(image types) enabled > we wouldn't know what to choose. The reason being the difference between a full build and a specific target build. > So far I have identified 2 differences between a full build and a specific target build: one is the dtbs target and the > other one is the different image types that are produced in one build session. e.g: for arm the kernel might have the > configuration to build both zImage and Image.gz(full-build case) but we need only one image for deployment(specific target build). > However, building all of these images every time might not be necessary, which is where the KERNEL_IMAGETYPE comes into > the picture hence the reason for these changes. > >> The compilation of the dtbs accepts the following formats e.g: >> Kernel Post 6.x: >> a) full path from the architecture directory: >> arch/${ARCH}/boot/dts/bsp_dir/test.dts >> arch/${ARCH}/boot/dts/bsp_dir/test.dtb >> b) relative path from the dts directory: bsp_dir/test.dts >> c) device tree blob file: bsp_dir/test.dtb >> Kernel Pre 6.x: >> The same as on Kernel Post 6.x but without the bsp_dir >> >> The differences when building a full build and when using a specific target >> build are:(tests were made on the machine phyboard-mira on 6.x and 4.x versions) >> specific target: Build needed 00:02:08, 2219688k disk space linux-phy 6.x >> full build: Build needed 00:02:14, 2271528k disk space linux-phy 6.x >> The time difference: the full build took 6s longer than the specific target build >> The disk space difference: the full build used 50.6MB more disk space than >> the specific target build. > > That 50M is build-time space, right? What's the size difference on the > target? > Yes, 50M disk space. I don't have a phyboard-mira at my disposal, but I can > try on a iot2050 board to see the difference on the target. > >> In conclusion the specific target build is slightly faster and uses less >> disk space. >> >> For now the DTB_FILES variable should accept only Device Tree Blobs(.dtb) >> because some areas still depends on it e.g: do_copy_boot_files, plugins > > We are not yet using dtbo files (overlays) built by the kernel in any > projects I know, but those would no longer be built then, right? > Exactly, but these changes are still keeping the full-build case in place > As you said, because I haven't seen any dtbo files around I left this out for the moment > and because the changes were starting to be quite big.(can be treated as a follow up on this > as compiling the dtbo is in line with the dtb) > >> >> Signed-off-by: Nicusor Huhulea <nicusor.huhulea@siemens.com> >> --- >> meta-isar/conf/machine/imx6-sabrelite.conf | 2 + >> meta-isar/conf/machine/phyboard-mira.conf | 2 + >> .../linux/files/debian/isar/build.tmpl | 63 +++++++++++++++++-- >> .../linux/files/debian/isar/install.tmpl | 12 +++- >> meta/recipes-kernel/linux/linux-custom.inc | 6 ++ >> 5 files changed, 79 insertions(+), 6 deletions(-) >> >> diff --git a/meta-isar/conf/machine/imx6-sabrelite.conf b/meta-isar/conf/machine/imx6-sabrelite.conf >> index e48823d6..bffebdc1 100644 >> --- a/meta-isar/conf/machine/imx6-sabrelite.conf >> +++ b/meta-isar/conf/machine/imx6-sabrelite.conf >> @@ -15,4 +15,6 @@ MKUBIFS_ARGS := "-m 0x1000 -e 0x3e000 -c 1500" >> UBINIZE_ARGS = "-vv -m 0x1000 -p 0x40000" >> IMAGE_FSTYPES ?= "ubi-ubifs" >> >> +KERNEL_IMAGETYPE = "zImage" >> + >> DTB_FILES = "imx6q-sabrelite.dtb" >> diff --git a/meta-isar/conf/machine/phyboard-mira.conf b/meta-isar/conf/machine/phyboard-mira.conf >> index feec4e54..1d2b3e1c 100644 >> --- a/meta-isar/conf/machine/phyboard-mira.conf >> +++ b/meta-isar/conf/machine/phyboard-mira.conf >> @@ -20,6 +20,8 @@ DTB_FILES = "imx6q-phytec-mira-rdk-nand.dtb" >> >> IMAGE_INSTALL += "barebox" >> >> +KERNEL_IMAGETYPE = "zImage" >> + >> BAREBOX_CONFIG = "imx_v7_defconfig" >> BAREBOX_ENV = "phytec-mira-env" >> BAREBOX_BASE_BIN = "barebox-phytec-phycore-imx6q-som-nand-1gib" >> diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl >> index bafc5ca4..90222adc 100644 >> --- a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl >> +++ b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl >> @@ -21,15 +21,21 @@ do_build() { >> KR=$(${MAKE} O=${KERNEL_BUILD_DIR} -s --no-print-directory kernelrelease) >> sed -i "s/@KR@/${KR}/g" ${S}/debian/control ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.* >> >> - if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then # Build kernel scripts and tools >> - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" >> + MAKE_COMMON_ARGS="O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS=${KCFLAGS} KAFLAGS=${KAFLAGS}" >> + if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then >> + if [ -n "${KERNEL_IMAGETYPE}" ]; then # Override the default compilation >> + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_IMAGETYPE} modules > > You are losing KERNEL_EXTRA_BUILDARGS. > >> + compile_dtbs >> + else # Full build >> + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} >> + fi >> elif echo "${DEB_BUILD_PROFILES}" | grep -q "kbuild"; then # Build kernel scripts and tools >> - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" scripts >> + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} scripts > > Why this change and those 2 below? They look wrong. > These changes are some optimizations but after carefully inspecting these I saw that I need to fix some ${KERNEL_EXTRA_BUILDARGS} addition > and loosing in some areas. > Don't mix topic, write separate commits then. >> if grep -q -E "CONFIG_STACK_VALIDATION=y|CONFIG_HAVE_OBJTOOL=y" ${KERNEL_BUILD_DIR}/.config && [ -d "tools/objtool" ]; then >> - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" tools/objtool || true >> + ${MAKE} ${MAKE_COMMON_ARGS} tools/objtool || true >> fi >> if grep -q "CONFIG_MODULES=y" ${KERNEL_BUILD_DIR}/.config; then >> - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" modules_prepare >> + ${MAKE} ${MAKE_COMMON_ARGS} modules_prepare >> fi >> fi >> >> @@ -37,6 +43,53 @@ do_build() { >> set +x >> } >> >> +compile_dtbs() { >> + local prefix="arch/${ARCH}/boot/dts/" >> + >> + for dtb in ${DTB_FILES}; do >> + # Handle the case where a standalone(with no path .dtb/.dts) file may reside within >> + # a specific BSP directory or not(kernel pre-6.x) >> + if [[ ("${dtb}" == *.dtb || "${dtb}" == *.dts) && "${dtb}" != */* ]]; then >> + bsp_dir="" >> + dts_path="" >> + if [[ "${dtb}" == *.dtb ]]; then >> + dts="${dtb%.dtb}.dts" >> + else >> + dts="${dtb}" >> + fi >> + >> + # recursively search for the dts file in all possible BSP directories >> + while IFS= read -r dts_path; do >> + if [ -n "${dts_path}" ]; then >> + # Eliminate the prefix and dts to get only the bsp_dir >> + bsp_dir="${dts_path#${prefix}}" >> + bsp_dir="${bsp_dir%${dts}}" >> + >> + relative_dtb_path="${bsp_dir}${dtb}" >> + else >> + relative_dtb_path="${prefix}${dtb}" >> + fi >> + done < <(find "${prefix}" -type f -name "${dts}") >> + >> + # Check if the path is relative (starts with arch/${ARCH}/) >> + # and if it does then extract the relative path from the architecture specific path >> + elif [[ "${dtb}" == "${prefix}"* ]]; then >> + relative_dtb_path="${dtb#${prefix}}" >> + else >> + # directly use the given dtb >> + relative_dtb_path="${dtb}" >> + fi >> + >> + # Check if it's a .dts file >> + if [[ "${relative_dtb_path}" == *.dts ]]; then >> + # Change .dts to .dtb >> + relative_dtb_path="${relative_dtb_path%.dts}.dtb" >> + fi >> + >> + ${MAKE} ${MAKE_COMMON_ARGS} ${relative_dtb_path} >> + done >> +} >> + >> print_settings() { >> cat <<EOF >> # Build settings: >> diff --git a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl >> index 00011517..c0a60f4e 100644 >> --- a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl >> +++ b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl >> @@ -124,7 +124,17 @@ install_hooks() { >> >> install_dtbs() { >> [ -n "${CONFIG_OF}" ] || return 0 >> - ${MAKE} O=${O} INSTALL_DTBS_PATH=${deb_img_dir}/usr/lib/linux-image-${krel} dtbs_install >> + prefix="arch/${ARCH}/boot/dts/" >> + src_dir="${O}/${prefix}" >> + dst_dir="" >> + >> + dst_dir="${deb_img_dir}/usr/lib/linux-image-${krel}/" >> + find "${src_dir}" -type f -name '*.dtb' | while IFS= read -r dtb_file; do >> + local relative_path="${dtb_file#"${src_dir}"}" >> + mkdir -p "${dst_dir}$(dirname "${relative_path}")" >> + cp "${dtb_file}" "${dst_dir}/${relative_path}" >> + echo "Copying ${dtb_file} to ${dest_dir}/${relative_path}" >> + done >> } >> >> install_kmods() { >> diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc >> index 6aa70fd3..31cc480b 100644 >> --- a/meta/recipes-kernel/linux/linux-custom.inc >> +++ b/meta/recipes-kernel/linux/linux-custom.inc >> @@ -76,6 +76,7 @@ TEMPLATE_VARS += " \ >> KERNEL_DEBIAN_DEPENDS \ >> KERNEL_BUILD_DIR \ >> KERNEL_FILE \ >> + KERNEL_IMAGETYPE \ >> KERNEL_HEADERS_DEBIAN_DEPENDS \ >> LINUX_VERSION_EXTENSION \ >> KERNEL_NAME_PROVIDED \ >> @@ -84,12 +85,17 @@ TEMPLATE_VARS += " \ >> KAFLAGS \ >> DISTRIBUTOR \ >> KERNEL_EXTRA_BUILDARGS \ >> + DTB_FILES \ >> " >> >> inherit dpkg >> inherit template >> inherit kbuildtarget >> >> +DTB_FILES ?= "" >> + >> +KERNEL_IMAGETYPE ?= "" >> + >> # Add custom cflags to the kernel build >> KCFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." >> KAFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." > > Quite a lot of new code for rather moderate optimizations. Maybe we > should rather filter the built files when installing? That should be > doable in 3 lines or so. > > Jan > > -- > Siemens AG, Technology > Linux Expert Center > > Yes, indeed seems a lot of code but these changes were made having in mind fixing the compilation of those extra dtbs that were not > belonging to that specific target bsp. And the optimizations are just a side effect(maybe as a bonus), they are not on the foreground. > The reason because it's a lot of code is primarily because I wanted to keep the DTB_FILES declarations as is in the upper layers and bring > new features e.g full architecture path, etc. plus this handles a KERNEL Pre 6.x dtb builds Again, please split topic-wise. And I'm not yet fully convinced about the value compared to the extra configuration effort. Jan -- Siemens AG, Technology Linux Expert Center -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/160d72b6-d23e-46bf-9315-ed30da3a652e%40siemens.com. ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 1/3] linux-custom: optimization on kernel's make command 2024-08-02 11:41 [PATCH] linux-custom: compile only those specified dtbs 'Nicusor Huhulea' via isar-users 2024-08-02 17:03 ` 'Jan Kiszka' via isar-users 2024-08-09 10:59 ` [PATCH v2] " 'Nicusor Huhulea' via isar-users @ 2024-08-13 11:55 ` 'Nicusor Huhulea' via isar-users 2024-08-13 11:55 ` [PATCH 2/3] linux-custom: compile only those specified dtbs 'Nicusor Huhulea' via isar-users ` (2 more replies) 2 siblings, 3 replies; 17+ messages in thread From: 'Nicusor Huhulea' via isar-users @ 2024-08-13 11:55 UTC (permalink / raw) To: isar-users; +Cc: Nicusor Huhulea Signed-off-by: Nicusor Huhulea <nicusor.huhulea@siemens.com> --- .../recipes-kernel/linux/files/debian/isar/build.tmpl | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl index bafc5ca4..1fd6f948 100644 --- a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl +++ b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl @@ -21,15 +21,16 @@ do_build() { KR=$(${MAKE} O=${KERNEL_BUILD_DIR} -s --no-print-directory kernelrelease) sed -i "s/@KR@/${KR}/g" ${S}/debian/control ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.* - if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then # Build kernel scripts and tools - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" + MAKE_COMMON_ARGS="O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS=${KCFLAGS} KAFLAGS=${KAFLAGS}" + if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then # Full build + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} elif echo "${DEB_BUILD_PROFILES}" | grep -q "kbuild"; then # Build kernel scripts and tools - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" scripts + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} scripts if grep -q -E "CONFIG_STACK_VALIDATION=y|CONFIG_HAVE_OBJTOOL=y" ${KERNEL_BUILD_DIR}/.config && [ -d "tools/objtool" ]; then - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" tools/objtool || true + ${MAKE} ${MAKE_COMMON_ARGS} tools/objtool || true fi if grep -q "CONFIG_MODULES=y" ${KERNEL_BUILD_DIR}/.config; then - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" modules_prepare + ${MAKE} ${MAKE_COMMON_ARGS} modules_prepare fi fi -- 2.39.2 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240813115521.381481-1-nicusor.huhulea%40siemens.com. ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 2/3] linux-custom: compile only those specified dtbs 2024-08-13 11:55 ` [PATCH 1/3] linux-custom: optimization on kernel's make command 'Nicusor Huhulea' via isar-users @ 2024-08-13 11:55 ` 'Nicusor Huhulea' via isar-users 2024-08-13 12:33 ` 'Jan Kiszka' via isar-users 2024-08-13 11:55 ` [PATCH 3/3] phyboard-mira.conf: use a specific target build 'Nicusor Huhulea' via isar-users 2024-08-13 12:31 ` [PATCH 1/3] linux-custom: optimization on kernel's make command 'Jan Kiszka' via isar-users 2 siblings, 1 reply; 17+ messages in thread From: 'Nicusor Huhulea' via isar-users @ 2024-08-13 11:55 UTC (permalink / raw) To: isar-users; +Cc: Nicusor Huhulea The current implementation makes a full kernel build and that means it will compile all dtbs specified by the bsp makefile e.g arch/arm64/boot/dts/ti/Makefile Currently the required dtbs are given by the DTB_FILES. These changes are checking if the DTB_FILES is specified, compile only those and avoids compiling all the dtbs from the bsp directory. Otherwise build all dtbs specified by the kernel source. Preventing the automatic dtb builds is made by using target specific build commands e.g KERNEL_IMAGETYPE The compilation of the dtbs accepts the following formats e.g: Kernel Post 6.x: a) full path from the architecture directory: arch/${ARCH}/boot/dts/bsp_dir/test.dts arch/${ARCH}/boot/dts/bsp_dir/test.dtb b) relative path from the dts directory: bsp_dir/test.dts c) device tree blob file: bsp_dir/test.dtb Kernel Pre 6.x: The same as on Kernel Post 6.x but without the bsp_dir The differences when building a full build and when using a specific target build are:(tests were made on the machine phyboard-mira on 6.x and 4.x versions) specific target: Build needed 00:02:08, 2219688k disk space linux-phy 6.x full build: Build needed 00:02:14, 2271528k disk space linux-phy 6.x The time difference: the full build took 6s longer than the specific target build The disk space difference: the full build used 50.6MB more disk space than the specific target build. In conclusion the specific target build is slightly faster and uses less disk space. For now the DTB_FILES variable should accept only Device Tree Blobs(.dtb) because some areas still depends on it e.g: do_copy_boot_files, plugins Signed-off-by: Nicusor Huhulea <nicusor.huhulea@siemens.com> --- .../linux/files/debian/isar/build.tmpl | 56 ++++++++++++++++++- .../linux/files/debian/isar/install.tmpl | 12 +++- meta/recipes-kernel/linux/linux-custom.inc | 6 ++ 3 files changed, 71 insertions(+), 3 deletions(-) diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl index 1fd6f948..15201686 100644 --- a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl +++ b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl @@ -22,8 +22,13 @@ do_build() { sed -i "s/@KR@/${KR}/g" ${S}/debian/control ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.* MAKE_COMMON_ARGS="O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS=${KCFLAGS} KAFLAGS=${KAFLAGS}" - if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then # Full build - ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} + if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then + if [ -n "${KERNEL_IMAGETYPE}" ]; then # Override the default compilation + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} ${KERNEL_IMAGETYPE} modules + compile_dtbs + else # Full build + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} + fi elif echo "${DEB_BUILD_PROFILES}" | grep -q "kbuild"; then # Build kernel scripts and tools ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} scripts if grep -q -E "CONFIG_STACK_VALIDATION=y|CONFIG_HAVE_OBJTOOL=y" ${KERNEL_BUILD_DIR}/.config && [ -d "tools/objtool" ]; then @@ -38,6 +43,53 @@ do_build() { set +x } +compile_dtbs() { + local prefix="arch/${ARCH}/boot/dts/" + + for dtb in ${DTB_FILES}; do + # Handle the case where a standalone(with no path .dtb/.dts) file may reside within + # a specific BSP directory or not(kernel pre-6.x) + if [[ ("${dtb}" == *.dtb || "${dtb}" == *.dts) && "${dtb}" != */* ]]; then + bsp_dir="" + dts_path="" + if [[ "${dtb}" == *.dtb ]]; then + dts="${dtb%.dtb}.dts" + else + dts="${dtb}" + fi + + # recursively search for the dts file in all possible BSP directories + while IFS= read -r dts_path; do + if [ -n "${dts_path}" ]; then + # Eliminate the prefix and dts to get only the bsp_dir + bsp_dir="${dts_path#${prefix}}" + bsp_dir="${bsp_dir%${dts}}" + + relative_dtb_path="${bsp_dir}${dtb}" + else + relative_dtb_path="${prefix}${dtb}" + fi + done < <(find "${prefix}" -type f -name "${dts}") + + # Check if the path is relative (starts with arch/${ARCH}/) + # and if it does then extract the relative path from the architecture specific path + elif [[ "${dtb}" == "${prefix}"* ]]; then + relative_dtb_path="${dtb#${prefix}}" + else + # directly use the given dtb + relative_dtb_path="${dtb}" + fi + + # Check if it's a .dts file + if [[ "${relative_dtb_path}" == *.dts ]]; then + # Change .dts to .dtb + relative_dtb_path="${relative_dtb_path%.dts}.dtb" + fi + + ${MAKE} ${MAKE_COMMON_ARGS} ${relative_dtb_path} + done +} + print_settings() { cat <<EOF # Build settings: diff --git a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl index 00011517..c0a60f4e 100644 --- a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl +++ b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl @@ -124,7 +124,17 @@ install_hooks() { install_dtbs() { [ -n "${CONFIG_OF}" ] || return 0 - ${MAKE} O=${O} INSTALL_DTBS_PATH=${deb_img_dir}/usr/lib/linux-image-${krel} dtbs_install + prefix="arch/${ARCH}/boot/dts/" + src_dir="${O}/${prefix}" + dst_dir="" + + dst_dir="${deb_img_dir}/usr/lib/linux-image-${krel}/" + find "${src_dir}" -type f -name '*.dtb' | while IFS= read -r dtb_file; do + local relative_path="${dtb_file#"${src_dir}"}" + mkdir -p "${dst_dir}$(dirname "${relative_path}")" + cp "${dtb_file}" "${dst_dir}/${relative_path}" + echo "Copying ${dtb_file} to ${dest_dir}/${relative_path}" + done } install_kmods() { diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc index 6aa70fd3..31cc480b 100644 --- a/meta/recipes-kernel/linux/linux-custom.inc +++ b/meta/recipes-kernel/linux/linux-custom.inc @@ -76,6 +76,7 @@ TEMPLATE_VARS += " \ KERNEL_DEBIAN_DEPENDS \ KERNEL_BUILD_DIR \ KERNEL_FILE \ + KERNEL_IMAGETYPE \ KERNEL_HEADERS_DEBIAN_DEPENDS \ LINUX_VERSION_EXTENSION \ KERNEL_NAME_PROVIDED \ @@ -84,12 +85,17 @@ TEMPLATE_VARS += " \ KAFLAGS \ DISTRIBUTOR \ KERNEL_EXTRA_BUILDARGS \ + DTB_FILES \ " inherit dpkg inherit template inherit kbuildtarget +DTB_FILES ?= "" + +KERNEL_IMAGETYPE ?= "" + # Add custom cflags to the kernel build KCFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." KAFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." -- 2.39.2 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240813115521.381481-2-nicusor.huhulea%40siemens.com. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 2/3] linux-custom: compile only those specified dtbs 2024-08-13 11:55 ` [PATCH 2/3] linux-custom: compile only those specified dtbs 'Nicusor Huhulea' via isar-users @ 2024-08-13 12:33 ` 'Jan Kiszka' via isar-users 2024-08-14 17:37 ` 'nicusor.huhulea@siemens.com' via isar-users 0 siblings, 1 reply; 17+ messages in thread From: 'Jan Kiszka' via isar-users @ 2024-08-13 12:33 UTC (permalink / raw) To: Nicusor Huhulea, isar-users On 13.08.24 13:55, 'Nicusor Huhulea' via isar-users wrote: > The current implementation makes a full kernel build and that means it will > compile all dtbs specified by the bsp makefile e.g arch/arm64/boot/dts/ti/Makefile > Currently the required dtbs are given by the DTB_FILES. > > These changes are checking if the DTB_FILES is specified, compile only those > and avoids compiling all the dtbs from the bsp directory. Otherwise build all dtbs > specified by the kernel source. > Preventing the automatic dtb builds is made by using target specific build > commands e.g KERNEL_IMAGETYPE > The compilation of the dtbs accepts the following formats e.g: > Kernel Post 6.x: > a) full path from the architecture directory: > arch/${ARCH}/boot/dts/bsp_dir/test.dts > arch/${ARCH}/boot/dts/bsp_dir/test.dtb > b) relative path from the dts directory: bsp_dir/test.dts > c) device tree blob file: bsp_dir/test.dtb > Kernel Pre 6.x: > The same as on Kernel Post 6.x but without the bsp_dir > > The differences when building a full build and when using a specific target > build are:(tests were made on the machine phyboard-mira on 6.x and 4.x versions) > specific target: Build needed 00:02:08, 2219688k disk space linux-phy 6.x > full build: Build needed 00:02:14, 2271528k disk space linux-phy 6.x > The time difference: the full build took 6s longer than the specific target build > The disk space difference: the full build used 50.6MB more disk space than > the specific target build. > In conclusion the specific target build is slightly faster and uses less > disk space. > > For now the DTB_FILES variable should accept only Device Tree Blobs(.dtb) > because some areas still depends on it e.g: do_copy_boot_files, plugins > > Signed-off-by: Nicusor Huhulea <nicusor.huhulea@siemens.com> > --- > .../linux/files/debian/isar/build.tmpl | 56 ++++++++++++++++++- > .../linux/files/debian/isar/install.tmpl | 12 +++- > meta/recipes-kernel/linux/linux-custom.inc | 6 ++ > 3 files changed, 71 insertions(+), 3 deletions(-) > > diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > index 1fd6f948..15201686 100644 > --- a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > +++ b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > @@ -22,8 +22,13 @@ do_build() { > sed -i "s/@KR@/${KR}/g" ${S}/debian/control ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.* > > MAKE_COMMON_ARGS="O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS=${KCFLAGS} KAFLAGS=${KAFLAGS}" > - if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then # Full build > - ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} > + if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then > + if [ -n "${KERNEL_IMAGETYPE}" ]; then # Override the default compilation > + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} ${KERNEL_IMAGETYPE} modules > + compile_dtbs > + else # Full build > + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} > + fi > elif echo "${DEB_BUILD_PROFILES}" | grep -q "kbuild"; then # Build kernel scripts and tools > ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} scripts > if grep -q -E "CONFIG_STACK_VALIDATION=y|CONFIG_HAVE_OBJTOOL=y" ${KERNEL_BUILD_DIR}/.config && [ -d "tools/objtool" ]; then > @@ -38,6 +43,53 @@ do_build() { > set +x > } > > +compile_dtbs() { > + local prefix="arch/${ARCH}/boot/dts/" > + > + for dtb in ${DTB_FILES}; do > + # Handle the case where a standalone(with no path .dtb/.dts) file may reside within > + # a specific BSP directory or not(kernel pre-6.x) > + if [[ ("${dtb}" == *.dtb || "${dtb}" == *.dts) && "${dtb}" != */* ]]; then > + bsp_dir="" > + dts_path="" > + if [[ "${dtb}" == *.dtb ]]; then > + dts="${dtb%.dtb}.dts" > + else > + dts="${dtb}" > + fi > + > + # recursively search for the dts file in all possible BSP directories > + while IFS= read -r dts_path; do > + if [ -n "${dts_path}" ]; then > + # Eliminate the prefix and dts to get only the bsp_dir > + bsp_dir="${dts_path#${prefix}}" > + bsp_dir="${bsp_dir%${dts}}" > + > + relative_dtb_path="${bsp_dir}${dtb}" > + else > + relative_dtb_path="${prefix}${dtb}" > + fi > + done < <(find "${prefix}" -type f -name "${dts}") > + > + # Check if the path is relative (starts with arch/${ARCH}/) > + # and if it does then extract the relative path from the architecture specific path > + elif [[ "${dtb}" == "${prefix}"* ]]; then > + relative_dtb_path="${dtb#${prefix}}" > + else > + # directly use the given dtb > + relative_dtb_path="${dtb}" > + fi > + > + # Check if it's a .dts file > + if [[ "${relative_dtb_path}" == *.dts ]]; then > + # Change .dts to .dtb > + relative_dtb_path="${relative_dtb_path%.dts}.dtb" > + fi > + > + ${MAKE} ${MAKE_COMMON_ARGS} ${relative_dtb_path} > + done > +} > + > print_settings() { > cat <<EOF > # Build settings: > diff --git a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > index 00011517..c0a60f4e 100644 > --- a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > +++ b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > @@ -124,7 +124,17 @@ install_hooks() { > > install_dtbs() { > [ -n "${CONFIG_OF}" ] || return 0 > - ${MAKE} O=${O} INSTALL_DTBS_PATH=${deb_img_dir}/usr/lib/linux-image-${krel} dtbs_install > + prefix="arch/${ARCH}/boot/dts/" > + src_dir="${O}/${prefix}" > + dst_dir="" > + > + dst_dir="${deb_img_dir}/usr/lib/linux-image-${krel}/" > + find "${src_dir}" -type f -name '*.dtb' | while IFS= read -r dtb_file; do > + local relative_path="${dtb_file#"${src_dir}"}" > + mkdir -p "${dst_dir}$(dirname "${relative_path}")" > + cp "${dtb_file}" "${dst_dir}/${relative_path}" > + echo "Copying ${dtb_file} to ${dest_dir}/${relative_path}" > + done > } > > install_kmods() { > diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc > index 6aa70fd3..31cc480b 100644 > --- a/meta/recipes-kernel/linux/linux-custom.inc > +++ b/meta/recipes-kernel/linux/linux-custom.inc > @@ -76,6 +76,7 @@ TEMPLATE_VARS += " \ > KERNEL_DEBIAN_DEPENDS \ > KERNEL_BUILD_DIR \ > KERNEL_FILE \ > + KERNEL_IMAGETYPE \ > KERNEL_HEADERS_DEBIAN_DEPENDS \ > LINUX_VERSION_EXTENSION \ > KERNEL_NAME_PROVIDED \ > @@ -84,12 +85,17 @@ TEMPLATE_VARS += " \ > KAFLAGS \ > DISTRIBUTOR \ > KERNEL_EXTRA_BUILDARGS \ > + DTB_FILES \ > " > > inherit dpkg > inherit template > inherit kbuildtarget > > +DTB_FILES ?= "" > + > +KERNEL_IMAGETYPE ?= "" > + > # Add custom cflags to the kernel build > KCFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." > KAFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." I'm still not convinces about effort / gain here. In addition: You already had to address < 6.x (6.0?) vs. newer. Won't this open-coded dtb build break again in future releases? Can't it cause problems with evil vendor kernels? I would rather vote for install-time filtering, not build-time. Jan -- Siemens AG, Technology Linux Expert Center -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/72362b78-e7b1-4409-b364-8106a418d5f6%40siemens.com. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 2/3] linux-custom: compile only those specified dtbs 2024-08-13 12:33 ` 'Jan Kiszka' via isar-users @ 2024-08-14 17:37 ` 'nicusor.huhulea@siemens.com' via isar-users 0 siblings, 0 replies; 17+ messages in thread From: 'nicusor.huhulea@siemens.com' via isar-users @ 2024-08-14 17:37 UTC (permalink / raw) To: Kiszka, Jan, isar-users The most important gain is that there will be alternative to the currently full build approach. The change for Kernel Pre and Post 6.x seems to be in my opinion an improvement to handle the increasing complexity particularly for ARM and happened once as far as I know, therefore I believe that the next major restructuring will happen far in the future. An evil vendor kernel? I don't think so. Based on this https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Makefile?h=v6.11-rc3#n1380 , the Makefile decide where the DTS should be located and how they are processed. So an evil vendor has to modify the Makefile to place his dts in some other location and potentially introduce significant complexity. Second the evil vendor will have serious issues with the upstream compatibility e.g rebasing or merging upstream changes, etc. So my conclusion would be that it's very unlikely for a vendor to deviate from the standard. The Install-time filtering that you suggested won't give you the answer to the question: Why do I see several kernel image types being compiled and why don't I have just the corresponding dtb for my board? I have started another thread summarizing the discussion from this one and other pros for the specific target build. https://groups.google.com/g/isar-users/c/e7EY6zCbZcs ________________________________________ From: Kiszka, Jan (T CED) <jan.kiszka@siemens.com> Sent: Tuesday, August 13, 2024 3:33 PM To: Huhulea, Nicusor Liviu (DI CTO FDS CES LX SVCS); isar-users@googlegroups.com Subject: Re: [PATCH 2/3] linux-custom: compile only those specified dtbs On 13.08.24 13:55, 'Nicusor Huhulea' via isar-users wrote: > The current implementation makes a full kernel build and that means it will > compile all dtbs specified by the bsp makefile e.g arch/arm64/boot/dts/ti/Makefile > Currently the required dtbs are given by the DTB_FILES. > > These changes are checking if the DTB_FILES is specified, compile only those > and avoids compiling all the dtbs from the bsp directory. Otherwise build all dtbs > specified by the kernel source. > Preventing the automatic dtb builds is made by using target specific build > commands e.g KERNEL_IMAGETYPE > The compilation of the dtbs accepts the following formats e.g: > Kernel Post 6.x: > a) full path from the architecture directory: > arch/${ARCH}/boot/dts/bsp_dir/test.dts > arch/${ARCH}/boot/dts/bsp_dir/test.dtb > b) relative path from the dts directory: bsp_dir/test.dts > c) device tree blob file: bsp_dir/test.dtb > Kernel Pre 6.x: > The same as on Kernel Post 6.x but without the bsp_dir > > The differences when building a full build and when using a specific target > build are:(tests were made on the machine phyboard-mira on 6.x and 4.x versions) > specific target: Build needed 00:02:08, 2219688k disk space linux-phy 6.x > full build: Build needed 00:02:14, 2271528k disk space linux-phy 6.x > The time difference: the full build took 6s longer than the specific target build > The disk space difference: the full build used 50.6MB more disk space than > the specific target build. > In conclusion the specific target build is slightly faster and uses less > disk space. > > For now the DTB_FILES variable should accept only Device Tree Blobs(.dtb) > because some areas still depends on it e.g: do_copy_boot_files, plugins > > Signed-off-by: Nicusor Huhulea <nicusor.huhulea@siemens.com> > --- > .../linux/files/debian/isar/build.tmpl | 56 ++++++++++++++++++- > .../linux/files/debian/isar/install.tmpl | 12 +++- > meta/recipes-kernel/linux/linux-custom.inc | 6 ++ > 3 files changed, 71 insertions(+), 3 deletions(-) > > diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > index 1fd6f948..15201686 100644 > --- a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > +++ b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > @@ -22,8 +22,13 @@ do_build() { > sed -i "s/@KR@/${KR}/g" ${S}/debian/control ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.* > > MAKE_COMMON_ARGS="O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS=${KCFLAGS} KAFLAGS=${KAFLAGS}" > - if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then # Full build > - ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} > + if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then > + if [ -n "${KERNEL_IMAGETYPE}" ]; then # Override the default compilation > + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} ${KERNEL_IMAGETYPE} modules > + compile_dtbs > + else # Full build > + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} > + fi > elif echo "${DEB_BUILD_PROFILES}" | grep -q "kbuild"; then # Build kernel scripts and tools > ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} scripts > if grep -q -E "CONFIG_STACK_VALIDATION=y|CONFIG_HAVE_OBJTOOL=y" ${KERNEL_BUILD_DIR}/.config && [ -d "tools/objtool" ]; then > @@ -38,6 +43,53 @@ do_build() { > set +x > } > > +compile_dtbs() { > + local prefix="arch/${ARCH}/boot/dts/" > + > + for dtb in ${DTB_FILES}; do > + # Handle the case where a standalone(with no path .dtb/.dts) file may reside within > + # a specific BSP directory or not(kernel pre-6.x) > + if [[ ("${dtb}" == *.dtb || "${dtb}" == *.dts) && "${dtb}" != */* ]]; then > + bsp_dir="" > + dts_path="" > + if [[ "${dtb}" == *.dtb ]]; then > + dts="${dtb%.dtb}.dts" > + else > + dts="${dtb}" > + fi > + > + # recursively search for the dts file in all possible BSP directories > + while IFS= read -r dts_path; do > + if [ -n "${dts_path}" ]; then > + # Eliminate the prefix and dts to get only the bsp_dir > + bsp_dir="${dts_path#${prefix}}" > + bsp_dir="${bsp_dir%${dts}}" > + > + relative_dtb_path="${bsp_dir}${dtb}" > + else > + relative_dtb_path="${prefix}${dtb}" > + fi > + done < <(find "${prefix}" -type f -name "${dts}") > + > + # Check if the path is relative (starts with arch/${ARCH}/) > + # and if it does then extract the relative path from the architecture specific path > + elif [[ "${dtb}" == "${prefix}"* ]]; then > + relative_dtb_path="${dtb#${prefix}}" > + else > + # directly use the given dtb > + relative_dtb_path="${dtb}" > + fi > + > + # Check if it's a .dts file > + if [[ "${relative_dtb_path}" == *.dts ]]; then > + # Change .dts to .dtb > + relative_dtb_path="${relative_dtb_path%.dts}.dtb" > + fi > + > + ${MAKE} ${MAKE_COMMON_ARGS} ${relative_dtb_path} > + done > +} > + > print_settings() { > cat <<EOF > # Build settings: > diff --git a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > index 00011517..c0a60f4e 100644 > --- a/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > +++ b/meta/recipes-kernel/linux/files/debian/isar/install.tmpl > @@ -124,7 +124,17 @@ install_hooks() { > > install_dtbs() { > [ -n "${CONFIG_OF}" ] || return 0 > - ${MAKE} O=${O} INSTALL_DTBS_PATH=${deb_img_dir}/usr/lib/linux-image-${krel} dtbs_install > + prefix="arch/${ARCH}/boot/dts/" > + src_dir="${O}/${prefix}" > + dst_dir="" > + > + dst_dir="${deb_img_dir}/usr/lib/linux-image-${krel}/" > + find "${src_dir}" -type f -name '*.dtb' | while IFS= read -r dtb_file; do > + local relative_path="${dtb_file#"${src_dir}"}" > + mkdir -p "${dst_dir}$(dirname "${relative_path}")" > + cp "${dtb_file}" "${dst_dir}/${relative_path}" > + echo "Copying ${dtb_file} to ${dest_dir}/${relative_path}" > + done > } > > install_kmods() { > diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc > index 6aa70fd3..31cc480b 100644 > --- a/meta/recipes-kernel/linux/linux-custom.inc > +++ b/meta/recipes-kernel/linux/linux-custom.inc > @@ -76,6 +76,7 @@ TEMPLATE_VARS += " \ > KERNEL_DEBIAN_DEPENDS \ > KERNEL_BUILD_DIR \ > KERNEL_FILE \ > + KERNEL_IMAGETYPE \ > KERNEL_HEADERS_DEBIAN_DEPENDS \ > LINUX_VERSION_EXTENSION \ > KERNEL_NAME_PROVIDED \ > @@ -84,12 +85,17 @@ TEMPLATE_VARS += " \ > KAFLAGS \ > DISTRIBUTOR \ > KERNEL_EXTRA_BUILDARGS \ > + DTB_FILES \ > " > > inherit dpkg > inherit template > inherit kbuildtarget > > +DTB_FILES ?= "" > + > +KERNEL_IMAGETYPE ?= "" > + > # Add custom cflags to the kernel build > KCFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." > KAFLAGS ?= "-fdebug-prefix-map=${CURDIR}=." I'm still not convinces about effort / gain here. In addition: You already had to address < 6.x (6.0?) vs. newer. Won't this open-coded dtb build break again in future releases? Can't it cause problems with evil vendor kernels? I would rather vote for install-time filtering, not build-time. Jan -- Siemens AG, Technology Linux Expert Center -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/DB3PR10MB69082B90DA26153EBB5D9F81E6872%40DB3PR10MB6908.EURPRD10.PROD.OUTLOOK.COM. ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 3/3] phyboard-mira.conf: use a specific target build 2024-08-13 11:55 ` [PATCH 1/3] linux-custom: optimization on kernel's make command 'Nicusor Huhulea' via isar-users 2024-08-13 11:55 ` [PATCH 2/3] linux-custom: compile only those specified dtbs 'Nicusor Huhulea' via isar-users @ 2024-08-13 11:55 ` 'Nicusor Huhulea' via isar-users 2024-08-13 12:10 ` 'nicusor.huhulea@siemens.com' via isar-users 2024-08-13 12:34 ` 'Jan Kiszka' via isar-users 2024-08-13 12:31 ` [PATCH 1/3] linux-custom: optimization on kernel's make command 'Jan Kiszka' via isar-users 2 siblings, 2 replies; 17+ messages in thread From: 'Nicusor Huhulea' via isar-users @ 2024-08-13 11:55 UTC (permalink / raw) To: isar-users; +Cc: Nicusor Huhulea The reason for this change is that phyboard-mira full build produces several tens of dtbs from the nxp bsp dir, therefore use a target build to get only the desired dtb Signed-off-by: Nicusor Huhulea <nicusor.huhulea@siemens.com> --- meta-isar/conf/machine/phyboard-mira.conf | 2 ++ 1 file changed, 2 insertions(+) diff --git a/meta-isar/conf/machine/phyboard-mira.conf b/meta-isar/conf/machine/phyboard-mira.conf index feec4e54..1d2b3e1c 100644 --- a/meta-isar/conf/machine/phyboard-mira.conf +++ b/meta-isar/conf/machine/phyboard-mira.conf @@ -20,6 +20,8 @@ DTB_FILES = "imx6q-phytec-mira-rdk-nand.dtb" IMAGE_INSTALL += "barebox" +KERNEL_IMAGETYPE = "zImage" + BAREBOX_CONFIG = "imx_v7_defconfig" BAREBOX_ENV = "phytec-mira-env" BAREBOX_BASE_BIN = "barebox-phytec-phycore-imx6q-som-nand-1gib" -- 2.39.2 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240813115521.381481-3-nicusor.huhulea%40siemens.com. ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: [PATCH 3/3] phyboard-mira.conf: use a specific target build 2024-08-13 11:55 ` [PATCH 3/3] phyboard-mira.conf: use a specific target build 'Nicusor Huhulea' via isar-users @ 2024-08-13 12:10 ` 'nicusor.huhulea@siemens.com' via isar-users 2024-08-13 12:34 ` 'Jan Kiszka' via isar-users 1 sibling, 0 replies; 17+ messages in thread From: 'nicusor.huhulea@siemens.com' via isar-users @ 2024-08-13 12:10 UTC (permalink / raw) To: isar-users I can add another con to the cons already mentioned in this thread, particularly for ARM architectures. So, it's better to use a targeted build needed for the target hardware and a full build should be only reserved for the entire range of hardware supported by the bsp dir. e.g: for me it make sense to have a full build if I'm going to do a test on all the bsps from arch/arm/nxp/freescale/*.dtbs but it does not make sense to have a full build when I'm using one bsp. >>-----Original Message----- >>From: Huhulea, Nicusor Liviu (DI CTO FDS CES LX SVCS) >><nicusor.huhulea@siemens.com> >>Sent: Tuesday, August 13, 2024 2:55 PM >>To: isar-users@googlegroups.com >>Cc: Huhulea, Nicusor Liviu (DI CTO FDS CES LX SVCS) >><nicusor.huhulea@siemens.com> >>Subject: [PATCH 3/3] phyboard-mira.conf: use a specific target build >> >>The reason for this change is that phyboard-mira full build produces several tens >>of dtbs from the nxp bsp dir, therefore use a target build to get only the desired >>dtb >> >>Signed-off-by: Nicusor Huhulea <nicusor.huhulea@siemens.com> >>--- >> meta-isar/conf/machine/phyboard-mira.conf | 2 ++ >> 1 file changed, 2 insertions(+) >> >>diff --git a/meta-isar/conf/machine/phyboard-mira.conf b/meta- >>isar/conf/machine/phyboard-mira.conf >>index feec4e54..1d2b3e1c 100644 >>--- a/meta-isar/conf/machine/phyboard-mira.conf >>+++ b/meta-isar/conf/machine/phyboard-mira.conf >>@@ -20,6 +20,8 @@ DTB_FILES = "imx6q-phytec-mira-rdk-nand.dtb" >> >> IMAGE_INSTALL += "barebox" >> >>+KERNEL_IMAGETYPE = "zImage" >>+ >> BAREBOX_CONFIG = "imx_v7_defconfig" >> BAREBOX_ENV = "phytec-mira-env" >> BAREBOX_BASE_BIN = "barebox-phytec-phycore-imx6q-som-nand-1gib" >>-- >>2.39.2 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/DB3PR10MB690863F9825DE317E5F92AC9E6862%40DB3PR10MB6908.EURPRD10.PROD.OUTLOOK.COM. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 3/3] phyboard-mira.conf: use a specific target build 2024-08-13 11:55 ` [PATCH 3/3] phyboard-mira.conf: use a specific target build 'Nicusor Huhulea' via isar-users 2024-08-13 12:10 ` 'nicusor.huhulea@siemens.com' via isar-users @ 2024-08-13 12:34 ` 'Jan Kiszka' via isar-users 1 sibling, 0 replies; 17+ messages in thread From: 'Jan Kiszka' via isar-users @ 2024-08-13 12:34 UTC (permalink / raw) To: Nicusor Huhulea, isar-users On 13.08.24 13:55, 'Nicusor Huhulea' via isar-users wrote: > The reason for this change is that phyboard-mira full build produces > several tens of dtbs from the nxp bsp dir, therefore use a target build > to get only the desired dtb > It's also needed so that we have at least one test in-tree. > Signed-off-by: Nicusor Huhulea <nicusor.huhulea@siemens.com> > --- > meta-isar/conf/machine/phyboard-mira.conf | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/meta-isar/conf/machine/phyboard-mira.conf b/meta-isar/conf/machine/phyboard-mira.conf > index feec4e54..1d2b3e1c 100644 > --- a/meta-isar/conf/machine/phyboard-mira.conf > +++ b/meta-isar/conf/machine/phyboard-mira.conf > @@ -20,6 +20,8 @@ DTB_FILES = "imx6q-phytec-mira-rdk-nand.dtb" > > IMAGE_INSTALL += "barebox" > > +KERNEL_IMAGETYPE = "zImage" > + > BAREBOX_CONFIG = "imx_v7_defconfig" > BAREBOX_ENV = "phytec-mira-env" > BAREBOX_BASE_BIN = "barebox-phytec-phycore-imx6q-som-nand-1gib" BTW, you are missing patch 4 in your series. I reminded of it in V1 of your submission. Jan -- Siemens AG, Technology Linux Expert Center -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/d554fecd-e54b-49ae-ba0c-6997ddfd3257%40siemens.com. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 1/3] linux-custom: optimization on kernel's make command 2024-08-13 11:55 ` [PATCH 1/3] linux-custom: optimization on kernel's make command 'Nicusor Huhulea' via isar-users 2024-08-13 11:55 ` [PATCH 2/3] linux-custom: compile only those specified dtbs 'Nicusor Huhulea' via isar-users 2024-08-13 11:55 ` [PATCH 3/3] phyboard-mira.conf: use a specific target build 'Nicusor Huhulea' via isar-users @ 2024-08-13 12:31 ` 'Jan Kiszka' via isar-users 2 siblings, 0 replies; 17+ messages in thread From: 'Jan Kiszka' via isar-users @ 2024-08-13 12:31 UTC (permalink / raw) To: Nicusor Huhulea, isar-users - series is missing a cover letter - series is missing a version update (we are at v3 IIRC) - series is sent as reply to older patches which may cause them to be overlooked - patch commit message provides no reason and is also not more specific on what aspect of make it optimizes Jan On 13.08.24 13:55, 'Nicusor Huhulea' via isar-users wrote: > Signed-off-by: Nicusor Huhulea <nicusor.huhulea@siemens.com> > --- > .../recipes-kernel/linux/files/debian/isar/build.tmpl | 11 ++++++----- > 1 file changed, 6 insertions(+), 5 deletions(-) > > diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > index bafc5ca4..1fd6f948 100644 > --- a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > +++ b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl > @@ -21,15 +21,16 @@ do_build() { > KR=$(${MAKE} O=${KERNEL_BUILD_DIR} -s --no-print-directory kernelrelease) > sed -i "s/@KR@/${KR}/g" ${S}/debian/control ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.* > > - if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then # Build kernel scripts and tools > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" > + MAKE_COMMON_ARGS="O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS=${KCFLAGS} KAFLAGS=${KAFLAGS}" > + if echo "${DEB_BUILD_PROFILES}" | grep -q "kernel"; then # Full build > + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} > elif echo "${DEB_BUILD_PROFILES}" | grep -q "kbuild"; then # Build kernel scripts and tools > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} ${KERNEL_EXTRA_BUILDARGS} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" scripts > + ${MAKE} ${MAKE_COMMON_ARGS} ${KERNEL_EXTRA_BUILDARGS} scripts > if grep -q -E "CONFIG_STACK_VALIDATION=y|CONFIG_HAVE_OBJTOOL=y" ${KERNEL_BUILD_DIR}/.config && [ -d "tools/objtool" ]; then > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" tools/objtool || true > + ${MAKE} ${MAKE_COMMON_ARGS} tools/objtool || true > fi > if grep -q "CONFIG_MODULES=y" ${KERNEL_BUILD_DIR}/.config; then > - ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} KCFLAGS="${KCFLAGS}" KAFLAGS="${KAFLAGS}" modules_prepare > + ${MAKE} ${MAKE_COMMON_ARGS} modules_prepare > fi > fi > -- Siemens AG, Technology Linux Expert Center -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/3a698550-4698-4d73-bf57-c91eacc14c4a%40siemens.com. ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2024-08-14 17:37 UTC | newest] Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-08-02 11:41 [PATCH] linux-custom: compile only those specified dtbs 'Nicusor Huhulea' via isar-users 2024-08-02 17:03 ` 'Jan Kiszka' via isar-users 2024-08-08 12:41 ` 'nicusor.huhulea@siemens.com' via isar-users 2024-08-09 16:57 ` 'Jan Kiszka' via isar-users 2024-08-09 10:59 ` [PATCH v2] " 'Nicusor Huhulea' via isar-users 2024-08-09 10:59 ` [PATCH] " 'Nicusor Huhulea' via isar-users 2024-08-09 16:56 ` 'Jan Kiszka' via isar-users 2024-08-12 14:44 ` 'nicusor.huhulea@siemens.com' via isar-users 2024-08-12 15:16 ` 'Jan Kiszka' via isar-users 2024-08-13 11:55 ` [PATCH 1/3] linux-custom: optimization on kernel's make command 'Nicusor Huhulea' via isar-users 2024-08-13 11:55 ` [PATCH 2/3] linux-custom: compile only those specified dtbs 'Nicusor Huhulea' via isar-users 2024-08-13 12:33 ` 'Jan Kiszka' via isar-users 2024-08-14 17:37 ` 'nicusor.huhulea@siemens.com' via isar-users 2024-08-13 11:55 ` [PATCH 3/3] phyboard-mira.conf: use a specific target build 'Nicusor Huhulea' via isar-users 2024-08-13 12:10 ` 'nicusor.huhulea@siemens.com' via isar-users 2024-08-13 12:34 ` 'Jan Kiszka' via isar-users 2024-08-13 12:31 ` [PATCH 1/3] linux-custom: optimization on kernel's make command 'Jan Kiszka' via isar-users
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox