Re: [PATCH] Consolidate over PARALLEL_MAKE

[Henning Schild <henning.schild@siemens.com>]

On Wed, 3 Jun 2020 08:37:57 +0200
"[ext] Jan Kiszka" <jan.kiszka@siemens.com> wrote:

> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> Replace self-invented / open-coded parallel build rules over the OE
> pattern PARALLEL_MAKE. Should specifically reduce the spread of
> PARALLEL_BUILD definitions in downstream layers. Mea culpa.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules | 5 ++---
>  meta/classes/dpkg.bbclass                              | 1 +
>  meta/conf/bitbake.conf                                 | 4 ++++
>  meta/recipes-bsp/u-boot/files/debian/rules             | 8 +++-----
>  meta/recipes-kernel/linux/files/debian/isar/build.tmpl | 2 +-
>  5 files changed, 11 insertions(+), 9 deletions(-)
> 
> diff --git a/meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules
> b/meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules index
> c8feb715..c3fc50a9 100644 ---
> a/meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules +++
> b/meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules @@ -4,10
> +4,9 @@ ifneq ($(DEB_BUILD_GNU_TYPE),$(DEB_HOST_GNU_TYPE)) export
> CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)- endif
>  
> -PARALLEL_BUILD=-j $(shell echo $$(($$(nproc) * 2)))
> -
>  override_dh_auto_build:
> -	CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_BUILD)
> PLATFORM=sifive/fu540
> FW_PAYLOAD_PATH=/usr/lib/u-boot/sifive_fu540/u-boot.bin
> +	CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE)
> PLATFORM=sifive/fu540 \
> +
> FW_PAYLOAD_PATH=/usr/lib/u-boot/sifive_fu540/u-boot.bin 
>  %:
>  	dh $@
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index 585365ee..acccc1f2 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -25,6 +25,7 @@ addtask devshell after do_install_builddeps
>  # Build package from sources using build script
>  dpkg_runbuild() {
>      E="${@ isar_export_proxies(d)}"
> +    export PARALLEL_MAKE="${PARALLEL_MAKE}"
>      sudo -E chroot --userspec=$( id -u ):$( id -g )
> ${BUILDCHROOT_DIR} \ /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH}
>  }
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 848e085a..0e0c79e0 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -101,8 +101,12 @@ BB_HASHCONFIG_WHITELIST ?=
> "${BB_HASHBASE_WHITELIST} DATE TIME SSH_AGENT_PID \
> BB_SIGNATURE_HANDLER ?= "basichash" BB_STAMP_POLICY ?= "full"
>  
> +# Default to setting automatically based on cpu count
>  BB_NUMBER_THREADS ?= "${@bb.utils.cpu_count()}"
>  
> +# Default to setting automatically based on cpu count
> +PARALLEL_MAKE ?= "-j ${@bb.utils.cpu_count()}"

Maybe derive it from the other one "-j ${BB_NUMBER_THREADS}" ?

For make it is often suggested to use slightly more threads than
available CPUs, when some threads are doing IO you still get to use all
CPUs. There was a "*2" ...

Henning

>  BBINCLUDELOGS ??= "yes"
>  
>  # Add event handlers for bitbake
> diff --git a/meta/recipes-bsp/u-boot/files/debian/rules
> b/meta/recipes-bsp/u-boot/files/debian/rules index 3f7a3218..92c07b17
> 100755 --- a/meta/recipes-bsp/u-boot/files/debian/rules
> +++ b/meta/recipes-bsp/u-boot/files/debian/rules
> @@ -12,12 +12,10 @@ export CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)-
>  SET_CROSS_BUILD_TOOLS=CROSS_BUILD_TOOLS=y
>  endif
>  
> -PARALLEL_BUILD=-j $(shell echo $$(($$(nproc) * 2)))
> -
>  override_dh_auto_build:
> -	$(MAKE) $(PARALLEL_BUILD) $(U_BOOT_CONFIG)
> -	$(MAKE) $(PARALLEL_BUILD) ${U_BOOT_BIN}
> -	$(MAKE) $(PARALLEL_BUILD) $(SET_CROSS_BUILD_TOOLS) NO_SDL=1
> tools-only envtools
> +	$(MAKE) $(PARALLEL_MAKE) $(U_BOOT_CONFIG)
> +	$(MAKE) $(PARALLEL_MAKE) ${U_BOOT_BIN}
> +	$(MAKE) $(PARALLEL_MAKE) $(SET_CROSS_BUILD_TOOLS) NO_SDL=1
> tools-only envtools 
>  override_dh_auto_install:
>  	mv tools/env/lib.a tools/env/libubootenv.a
> diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
> b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl index
> 17983e4d..b7749cec 100644 ---
> a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl +++
> b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl @@ -22,7
> +22,7 @@ do_build() { set -x
>  
>      # Build the Linux kernel
> -    ${MAKE} O=${KERNEL_BUILD_DIR} -j $(nproc) || exit ${?}
> +    ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} || exit ${?}
>  
>      # Stop tracing
>      set +x