* [PATCH] Consolidate over PARALLEL_MAKE
@ 2020-06-03 6:37 Jan Kiszka
2020-06-03 9:49 ` Henning Schild
2020-06-21 19:02 ` Baurzhan Ismagulov
0 siblings, 2 replies; 4+ messages in thread
From: Jan Kiszka @ 2020-06-03 6:37 UTC (permalink / raw)
To: isar-users
From: Jan Kiszka <jan.kiszka@siemens.com>
Replace self-invented / open-coded parallel build rules over the OE
pattern PARALLEL_MAKE. Should specifically reduce the spread of
PARALLEL_BUILD definitions in downstream layers. Mea culpa.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules | 5 ++---
meta/classes/dpkg.bbclass | 1 +
meta/conf/bitbake.conf | 4 ++++
meta/recipes-bsp/u-boot/files/debian/rules | 8 +++-----
meta/recipes-kernel/linux/files/debian/isar/build.tmpl | 2 +-
5 files changed, 11 insertions(+), 9 deletions(-)
diff --git a/meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules b/meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules
index c8feb715..c3fc50a9 100644
--- a/meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules
+++ b/meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules
@@ -4,10 +4,9 @@ ifneq ($(DEB_BUILD_GNU_TYPE),$(DEB_HOST_GNU_TYPE))
export CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)-
endif
-PARALLEL_BUILD=-j $(shell echo $$(($$(nproc) * 2)))
-
override_dh_auto_build:
- CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_BUILD) PLATFORM=sifive/fu540 FW_PAYLOAD_PATH=/usr/lib/u-boot/sifive_fu540/u-boot.bin
+ CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE) PLATFORM=sifive/fu540 \
+ FW_PAYLOAD_PATH=/usr/lib/u-boot/sifive_fu540/u-boot.bin
%:
dh $@
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 585365ee..acccc1f2 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -25,6 +25,7 @@ addtask devshell after do_install_builddeps
# Build package from sources using build script
dpkg_runbuild() {
E="${@ isar_export_proxies(d)}"
+ export PARALLEL_MAKE="${PARALLEL_MAKE}"
sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
/isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH}
}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 848e085a..0e0c79e0 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -101,8 +101,12 @@ BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SSH_AGENT_PID \
BB_SIGNATURE_HANDLER ?= "basichash"
BB_STAMP_POLICY ?= "full"
+# Default to setting automatically based on cpu count
BB_NUMBER_THREADS ?= "${@bb.utils.cpu_count()}"
+# Default to setting automatically based on cpu count
+PARALLEL_MAKE ?= "-j ${@bb.utils.cpu_count()}"
+
BBINCLUDELOGS ??= "yes"
# Add event handlers for bitbake
diff --git a/meta/recipes-bsp/u-boot/files/debian/rules b/meta/recipes-bsp/u-boot/files/debian/rules
index 3f7a3218..92c07b17 100755
--- a/meta/recipes-bsp/u-boot/files/debian/rules
+++ b/meta/recipes-bsp/u-boot/files/debian/rules
@@ -12,12 +12,10 @@ export CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)-
SET_CROSS_BUILD_TOOLS=CROSS_BUILD_TOOLS=y
endif
-PARALLEL_BUILD=-j $(shell echo $$(($$(nproc) * 2)))
-
override_dh_auto_build:
- $(MAKE) $(PARALLEL_BUILD) $(U_BOOT_CONFIG)
- $(MAKE) $(PARALLEL_BUILD) ${U_BOOT_BIN}
- $(MAKE) $(PARALLEL_BUILD) $(SET_CROSS_BUILD_TOOLS) NO_SDL=1 tools-only envtools
+ $(MAKE) $(PARALLEL_MAKE) $(U_BOOT_CONFIG)
+ $(MAKE) $(PARALLEL_MAKE) ${U_BOOT_BIN}
+ $(MAKE) $(PARALLEL_MAKE) $(SET_CROSS_BUILD_TOOLS) NO_SDL=1 tools-only envtools
override_dh_auto_install:
mv tools/env/lib.a tools/env/libubootenv.a
diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
index 17983e4d..b7749cec 100644
--- a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
@@ -22,7 +22,7 @@ do_build() {
set -x
# Build the Linux kernel
- ${MAKE} O=${KERNEL_BUILD_DIR} -j $(nproc) || exit ${?}
+ ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} || exit ${?}
# Stop tracing
set +x
--
2.26.2
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [PATCH] Consolidate over PARALLEL_MAKE
2020-06-03 6:37 [PATCH] Consolidate over PARALLEL_MAKE Jan Kiszka
@ 2020-06-03 9:49 ` Henning Schild
2020-06-03 9:56 ` Jan Kiszka
2020-06-21 19:02 ` Baurzhan Ismagulov
1 sibling, 1 reply; 4+ messages in thread
From: Henning Schild @ 2020-06-03 9:49 UTC (permalink / raw)
To: [ext] Jan Kiszka; +Cc: isar-users
On Wed, 3 Jun 2020 08:37:57 +0200
"[ext] Jan Kiszka" <jan.kiszka@siemens.com> wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
>
> Replace self-invented / open-coded parallel build rules over the OE
> pattern PARALLEL_MAKE. Should specifically reduce the spread of
> PARALLEL_BUILD definitions in downstream layers. Mea culpa.
>
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
> meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules | 5 ++---
> meta/classes/dpkg.bbclass | 1 +
> meta/conf/bitbake.conf | 4 ++++
> meta/recipes-bsp/u-boot/files/debian/rules | 8 +++-----
> meta/recipes-kernel/linux/files/debian/isar/build.tmpl | 2 +-
> 5 files changed, 11 insertions(+), 9 deletions(-)
>
> diff --git a/meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules
> b/meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules index
> c8feb715..c3fc50a9 100644 ---
> a/meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules +++
> b/meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules @@ -4,10
> +4,9 @@ ifneq ($(DEB_BUILD_GNU_TYPE),$(DEB_HOST_GNU_TYPE)) export
> CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)- endif
>
> -PARALLEL_BUILD=-j $(shell echo $$(($$(nproc) * 2)))
> -
> override_dh_auto_build:
> - CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_BUILD)
> PLATFORM=sifive/fu540
> FW_PAYLOAD_PATH=/usr/lib/u-boot/sifive_fu540/u-boot.bin
> + CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE)
> PLATFORM=sifive/fu540 \
> +
> FW_PAYLOAD_PATH=/usr/lib/u-boot/sifive_fu540/u-boot.bin
> %:
> dh $@
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index 585365ee..acccc1f2 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -25,6 +25,7 @@ addtask devshell after do_install_builddeps
> # Build package from sources using build script
> dpkg_runbuild() {
> E="${@ isar_export_proxies(d)}"
> + export PARALLEL_MAKE="${PARALLEL_MAKE}"
> sudo -E chroot --userspec=$( id -u ):$( id -g )
> ${BUILDCHROOT_DIR} \ /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH}
> }
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 848e085a..0e0c79e0 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -101,8 +101,12 @@ BB_HASHCONFIG_WHITELIST ?=
> "${BB_HASHBASE_WHITELIST} DATE TIME SSH_AGENT_PID \
> BB_SIGNATURE_HANDLER ?= "basichash" BB_STAMP_POLICY ?= "full"
>
> +# Default to setting automatically based on cpu count
> BB_NUMBER_THREADS ?= "${@bb.utils.cpu_count()}"
>
> +# Default to setting automatically based on cpu count
> +PARALLEL_MAKE ?= "-j ${@bb.utils.cpu_count()}"
Maybe derive it from the other one "-j ${BB_NUMBER_THREADS}" ?
For make it is often suggested to use slightly more threads than
available CPUs, when some threads are doing IO you still get to use all
CPUs. There was a "*2" ...
Henning
> BBINCLUDELOGS ??= "yes"
>
> # Add event handlers for bitbake
> diff --git a/meta/recipes-bsp/u-boot/files/debian/rules
> b/meta/recipes-bsp/u-boot/files/debian/rules index 3f7a3218..92c07b17
> 100755 --- a/meta/recipes-bsp/u-boot/files/debian/rules
> +++ b/meta/recipes-bsp/u-boot/files/debian/rules
> @@ -12,12 +12,10 @@ export CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)-
> SET_CROSS_BUILD_TOOLS=CROSS_BUILD_TOOLS=y
> endif
>
> -PARALLEL_BUILD=-j $(shell echo $$(($$(nproc) * 2)))
> -
> override_dh_auto_build:
> - $(MAKE) $(PARALLEL_BUILD) $(U_BOOT_CONFIG)
> - $(MAKE) $(PARALLEL_BUILD) ${U_BOOT_BIN}
> - $(MAKE) $(PARALLEL_BUILD) $(SET_CROSS_BUILD_TOOLS) NO_SDL=1
> tools-only envtools
> + $(MAKE) $(PARALLEL_MAKE) $(U_BOOT_CONFIG)
> + $(MAKE) $(PARALLEL_MAKE) ${U_BOOT_BIN}
> + $(MAKE) $(PARALLEL_MAKE) $(SET_CROSS_BUILD_TOOLS) NO_SDL=1
> tools-only envtools
> override_dh_auto_install:
> mv tools/env/lib.a tools/env/libubootenv.a
> diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
> b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl index
> 17983e4d..b7749cec 100644 ---
> a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl +++
> b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl @@ -22,7
> +22,7 @@ do_build() { set -x
>
> # Build the Linux kernel
> - ${MAKE} O=${KERNEL_BUILD_DIR} -j $(nproc) || exit ${?}
> + ${MAKE} O=${KERNEL_BUILD_DIR} ${PARALLEL_MAKE} || exit ${?}
>
> # Stop tracing
> set +x
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [PATCH] Consolidate over PARALLEL_MAKE
2020-06-03 9:49 ` Henning Schild
@ 2020-06-03 9:56 ` Jan Kiszka
0 siblings, 0 replies; 4+ messages in thread
From: Jan Kiszka @ 2020-06-03 9:56 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On 03.06.20 11:49, Henning Schild wrote:
> On Wed, 3 Jun 2020 08:37:57 +0200
> "[ext] Jan Kiszka" <jan.kiszka@siemens.com> wrote:
>
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> Replace self-invented / open-coded parallel build rules over the OE
>> pattern PARALLEL_MAKE. Should specifically reduce the spread of
>> PARALLEL_BUILD definitions in downstream layers. Mea culpa.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>> meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules | 5 ++---
>> meta/classes/dpkg.bbclass | 1 +
>> meta/conf/bitbake.conf | 4 ++++
>> meta/recipes-bsp/u-boot/files/debian/rules | 8 +++-----
>> meta/recipes-kernel/linux/files/debian/isar/build.tmpl | 2 +-
>> 5 files changed, 11 insertions(+), 9 deletions(-)
>>
>> diff --git a/meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules
>> b/meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules index
>> c8feb715..c3fc50a9 100644 ---
>> a/meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules +++
>> b/meta-isar/recipes-bsp/opensbi/files/sifive-fu540-rules @@ -4,10
>> +4,9 @@ ifneq ($(DEB_BUILD_GNU_TYPE),$(DEB_HOST_GNU_TYPE)) export
>> CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)- endif
>>
>> -PARALLEL_BUILD=-j $(shell echo $$(($$(nproc) * 2)))
>> -
>> override_dh_auto_build:
>> - CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_BUILD)
>> PLATFORM=sifive/fu540
>> FW_PAYLOAD_PATH=/usr/lib/u-boot/sifive_fu540/u-boot.bin
>> + CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE)
>> PLATFORM=sifive/fu540 \
>> +
>> FW_PAYLOAD_PATH=/usr/lib/u-boot/sifive_fu540/u-boot.bin
>> %:
>> dh $@
>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>> index 585365ee..acccc1f2 100644
>> --- a/meta/classes/dpkg.bbclass
>> +++ b/meta/classes/dpkg.bbclass
>> @@ -25,6 +25,7 @@ addtask devshell after do_install_builddeps
>> # Build package from sources using build script
>> dpkg_runbuild() {
>> E="${@ isar_export_proxies(d)}"
>> + export PARALLEL_MAKE="${PARALLEL_MAKE}"
>> sudo -E chroot --userspec=$( id -u ):$( id -g )
>> ${BUILDCHROOT_DIR} \ /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH}
>> }
>> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
>> index 848e085a..0e0c79e0 100644
>> --- a/meta/conf/bitbake.conf
>> +++ b/meta/conf/bitbake.conf
>> @@ -101,8 +101,12 @@ BB_HASHCONFIG_WHITELIST ?=
>> "${BB_HASHBASE_WHITELIST} DATE TIME SSH_AGENT_PID \
>> BB_SIGNATURE_HANDLER ?= "basichash" BB_STAMP_POLICY ?= "full"
>>
>> +# Default to setting automatically based on cpu count
>> BB_NUMBER_THREADS ?= "${@bb.utils.cpu_count()}"
>>
>> +# Default to setting automatically based on cpu count
>> +PARALLEL_MAKE ?= "-j ${@bb.utils.cpu_count()}"
>
> Maybe derive it from the other one "-j ${BB_NUMBER_THREADS}" ?
>
I'm just using the OE pattern here.
> For make it is often suggested to use slightly more threads than
> available CPUs, when some threads are doing IO you still get to use all
> CPUs. There was a "*2" ...
I know, but same reasoning as above.
And given that cpu_count is cores*threads with, typically, threads >= 2,
this overcommittment is less important today.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] Consolidate over PARALLEL_MAKE
2020-06-03 6:37 [PATCH] Consolidate over PARALLEL_MAKE Jan Kiszka
2020-06-03 9:49 ` Henning Schild
@ 2020-06-21 19:02 ` Baurzhan Ismagulov
1 sibling, 0 replies; 4+ messages in thread
From: Baurzhan Ismagulov @ 2020-06-21 19:02 UTC (permalink / raw)
To: isar-users
On Wed, Jun 03, 2020 at 08:37:57AM +0200, Jan Kiszka wrote:
> Replace self-invented / open-coded parallel build rules over the OE
> pattern PARALLEL_MAKE. Should specifically reduce the spread of
> PARALLEL_BUILD definitions in downstream layers. Mea culpa.
Applied to next, thanks.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-06-21 19:03 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-03 6:37 [PATCH] Consolidate over PARALLEL_MAKE Jan Kiszka
2020-06-03 9:49 ` Henning Schild
2020-06-03 9:56 ` Jan Kiszka
2020-06-21 19:02 ` Baurzhan Ismagulov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox