public inbox for isar-users@googlegroups.com
 help / color / mirror / Atom feed
* [PATCH 0/4] CI improvements
@ 2019-02-14 16:08 Maxim Yu. Osipov
  2019-02-14 16:08 ` [PATCH 1/4] ci: Introduce TARGETS_SET variable Maxim Yu. Osipov
                   ` (4 more replies)
  0 siblings, 5 replies; 11+ messages in thread
From: Maxim Yu. Osipov @ 2019-02-14 16:08 UTC (permalink / raw)
  To: isar-users

Hello everybody,

At the moment there is a confusion regarding CI build.
"Fast" CI build (called with parameter '-f') _cross_ builds 
3 most common (at least we think that) configurations
with cached base repository feature enabled:

 * multiconfig:qemuarm-stretch:isar-image-base
 * multiconfig:qemuarm64-stretch:isar-image-base
 * multiconfig:qemuamd64-stretch:isar-image-base

When default (w/o parameters) CI is called it _natively_ builds 
the following set of configurations:

 * multiconfig:qemuarm-jessie:isar-image-base
 * multiconfig:qemuarm-stretch:isar-image-base 
 * multiconfig:qemuarm-buster:isar-image-base
 * multiconfig:qemuarm64-stretch:isar-image-base
 * multiconfig:qemui386-jessie:isar-image-base
 * multiconfig:qemui386-stretch:isar-image-base
 * multiconfig:qemui386-buster:isar-image-base
 * multiconfig:qemuamd64-jessie:isar-image-base
 * multiconfig:qemuamd64-stretch:isar-image-base
 * multiconfig:qemuamd64-buster:isar-image-base
 * multiconfig:qemuamd64-buster-tgz:isar-image-base
 * multiconfig:rpi-jessie:isar-image-base

Unfortunately this set doesn't include supported configurations 
(real boards) 
 * multiconfig:bananapi-stretch.conf
 * hikey-stretch.conf
 * de0-nano-soc-stretch.conf

The reason is that these configurations include build of Linux kernel/U-Boot so 
the native build will last huge amount of time.

So the problem with two flavours (fast and default) of CI build is that they 
test different features for the subset of all supported configurations.

After upgrade of CI server we intend to increase a test coverage - 
 * Include cross builds for mentioned boards hikey, de0-nano-soc etc.
 * To use default build for overnight builds with different combination of features enabled 
 (native and cross, with or without used cached base repo etc.)

This patchset is the first step towards this direction.

Kind regards,
Maxim.


Maxim Yu. Osipov (4):
  ci: Introduce TARGETS_SET variable
  ci: Introduce --repro parameter to enable use of cached base
    repository
  ci: Include de0-nano-soc target into fast build
  ci: Disable use of cached base repository for fast build by default

 scripts/ci_build.sh | 71 +++++++++++++++++++++++++++++++----------------------
 1 file changed, 41 insertions(+), 30 deletions(-)

-- 
2.11.0


^ permalink raw reply	[flat|nested] 11+ messages in thread

* [PATCH 1/4] ci: Introduce TARGETS_SET variable
  2019-02-14 16:08 [PATCH 0/4] CI improvements Maxim Yu. Osipov
@ 2019-02-14 16:08 ` Maxim Yu. Osipov
  2019-02-14 16:08 ` [PATCH 2/4] ci: Introduce --repro parameter to enable use of cached base repository Maxim Yu. Osipov
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 11+ messages in thread
From: Maxim Yu. Osipov @ 2019-02-14 16:08 UTC (permalink / raw)
  To: isar-users

This improves code readibility.

Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
---
 scripts/ci_build.sh | 50 ++++++++++++++++++++++++++------------------------
 1 file changed, 26 insertions(+), 24 deletions(-)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 1789183..846349f 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -20,6 +20,23 @@ BUILD_DIR=./build
 
 BB_ARGS="-v"
 
+TARGETS_SET="\
+            multiconfig:qemuarm-jessie:isar-image-base \
+            multiconfig:qemuarm-stretch:isar-image-base \
+            multiconfig:qemuarm-buster:isar-image-base \
+            multiconfig:qemuarm64-stretch:isar-image-base \
+            multiconfig:qemui386-jessie:isar-image-base \
+            multiconfig:qemui386-stretch:isar-image-base \
+            multiconfig:qemui386-buster:isar-image-base \
+            multiconfig:qemuamd64-jessie:isar-image-base \
+            multiconfig:qemuamd64-stretch:isar-image-base \
+            multiconfig:qemuamd64-buster:isar-image-base \
+            multiconfig:qemuamd64-buster-tgz:isar-image-base \
+            multiconfig:rpi-jessie:isar-image-base"
+          # qemu-user-static of <= buster too old to build that
+          # multiconfig:qemuarm64-buster:isar-image-base
+
+
 show_help() {
     echo "This script builds the default Isar images."
     echo
@@ -62,6 +79,10 @@ do
         ;;
     -f|--fast)
         FAST_BUILD="1"
+        TARGETS_SET="\
+                     multiconfig:qemuarm-stretch:isar-image-base \
+                     multiconfig:qemuarm64-stretch:isar-image-base \
+                     multiconfig:qemuamd64-stretch:isar-image-base"
         ;;
     -q|--quiet)
         BB_ARGS=""
@@ -90,34 +111,15 @@ if [ -n "$FAST_BUILD" ]; then
     # Enforce cross-compilation to speed up the build
     # Enable use of cached base repository
     sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
-    bitbake $BB_ARGS -c cache_base_repo \
-        multiconfig:qemuarm-stretch:isar-image-base \
-        multiconfig:qemuarm64-stretch:isar-image-base \
-        multiconfig:qemuamd64-stretch:isar-image-base
+    bitbake $BB_ARGS -c cache_base_repo $TARGETS_SET
     while [ -e bitbake.sock ]; do sleep 1; done
     sudo rm -rf tmp
     sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
-    bitbake $BB_ARGS \
-        multiconfig:qemuarm-stretch:isar-image-base \
-        multiconfig:qemuarm64-stretch:isar-image-base \
-        multiconfig:qemuamd64-stretch:isar-image-base
+    bitbake $BB_ARGS $TARGETS_SET
 else
-    # Start build for all possible configurations
-    bitbake $BB_ARGS \
-        multiconfig:qemuarm-jessie:isar-image-base \
-        multiconfig:qemuarm-stretch:isar-image-base \
-        multiconfig:qemuarm-buster:isar-image-base \
-        multiconfig:qemuarm64-stretch:isar-image-base \
-        multiconfig:qemui386-jessie:isar-image-base \
-        multiconfig:qemui386-stretch:isar-image-base \
-        multiconfig:qemui386-buster:isar-image-base \
-        multiconfig:qemuamd64-jessie:isar-image-base \
-        multiconfig:qemuamd64-stretch:isar-image-base \
-        multiconfig:qemuamd64-buster:isar-image-base \
-        multiconfig:qemuamd64-buster-tgz:isar-image-base \
-        multiconfig:rpi-jessie:isar-image-base
-    # qemu-user-static of <= buster too old to build that
-    #multiconfig:qemuarm64-buster:isar-image-base
+    # Start build for the full set of configurations
+    bitbake $BB_ARGS $TARGETS_SET
+
 fi
 
 cp -a "${ISARROOT}/meta/classes/dpkg-base.bbclass" "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup"
-- 
2.11.0


^ permalink raw reply	[flat|nested] 11+ messages in thread

* [PATCH 2/4] ci: Introduce --repro parameter to enable use of cached base repository
  2019-02-14 16:08 [PATCH 0/4] CI improvements Maxim Yu. Osipov
  2019-02-14 16:08 ` [PATCH 1/4] ci: Introduce TARGETS_SET variable Maxim Yu. Osipov
@ 2019-02-14 16:08 ` Maxim Yu. Osipov
  2019-02-14 16:08 ` [PATCH 3/4] ci: Include de0-nano-soc target into fast build Maxim Yu. Osipov
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 11+ messages in thread
From: Maxim Yu. Osipov @ 2019-02-14 16:08 UTC (permalink / raw)
  To: isar-users

This patch allows to test cached base repository functionality
for not only fast build case.

Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
---
 scripts/ci_build.sh | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 846349f..acce152 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -48,8 +48,9 @@ show_help() {
     echo "                          the build will be started in current path."
     echo "    -c, --cross           enable cross-compilation."
     echo "    -d, --debug           enable debug bitbake output."
-    echo "    -f, --fast            build reduced set of configurations."
+    echo "    -f, --fast            cross build reduced set of configurations."
     echo "    -q, --quiet           suppress verbose bitbake output."
+    echo "    -r, --repro           enable use of cached base repository."
     echo "    --help                display this message and exit."
     echo
     echo "Exit status:"
@@ -78,7 +79,12 @@ do
         BB_ARGS="$BB_ARGS -d"
         ;;
     -f|--fast)
+        # Start build for the reduced set of configurations
+        # Enforce cross-compilation to speed up the build
+        # Enable use of cached base repository
         FAST_BUILD="1"
+        CROSS_BUILD="1"
+        REPRO_BUILD="1"
         TARGETS_SET="\
                      multiconfig:qemuarm-stretch:isar-image-base \
                      multiconfig:qemuarm64-stretch:isar-image-base \
@@ -87,6 +93,9 @@ do
     -q|--quiet)
         BB_ARGS=""
         ;;
+    -r|--repro)
+        REPRO_BUILD="1"
+        ;;
     *)
         echo "error: invalid parameter '$key', please try '--help' to get list of supported parameters"
         exit $ES_BUG
@@ -106,22 +115,17 @@ if [ -n "$CROSS_BUILD" ]; then
     sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
 fi
 
-if [ -n "$FAST_BUILD" ]; then
-    # Start build for the reduced set of configurations
-    # Enforce cross-compilation to speed up the build
+if [ -n "$REPRO_BUILD" ]; then
     # Enable use of cached base repository
-    sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
-    bitbake $BB_ARGS -c cache_base_repo $TARGETS_SET
+    bitbake $BB_ARGS -c cache_base_repo  $TARGETS_SET
     while [ -e bitbake.sock ]; do sleep 1; done
     sudo rm -rf tmp
     sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
-    bitbake $BB_ARGS $TARGETS_SET
-else
-    # Start build for the full set of configurations
-    bitbake $BB_ARGS $TARGETS_SET
-
 fi
 
+# Start build for the defined set of configurations
+bitbake $BB_ARGS $TARGETS_SET
+
 cp -a "${ISARROOT}/meta/classes/dpkg-base.bbclass" "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup"
 echo -e "do_fetch_append() {\n\n}" >> "${ISARROOT}/meta/classes/dpkg-base.bbclass"
 
-- 
2.11.0


^ permalink raw reply	[flat|nested] 11+ messages in thread

* [PATCH 3/4] ci: Include de0-nano-soc target into fast build
  2019-02-14 16:08 [PATCH 0/4] CI improvements Maxim Yu. Osipov
  2019-02-14 16:08 ` [PATCH 1/4] ci: Introduce TARGETS_SET variable Maxim Yu. Osipov
  2019-02-14 16:08 ` [PATCH 2/4] ci: Introduce --repro parameter to enable use of cached base repository Maxim Yu. Osipov
@ 2019-02-14 16:08 ` Maxim Yu. Osipov
  2019-02-15  7:20   ` Jan Kiszka
  2019-02-14 16:08 ` [PATCH 4/4] ci: Disable use of cached base repository for fast build by default Maxim Yu. Osipov
  2019-02-15 14:42 ` [PATCH 0/4] CI improvements Maxim Yu. Osipov
  4 siblings, 1 reply; 11+ messages in thread
From: Maxim Yu. Osipov @ 2019-02-14 16:08 UTC (permalink / raw)
  To: isar-users

Note: So far de0-nano-soc build is launched separately as
parallel build with the same target arch (qemuarm) fails.
While the problem with parallel build is being investigated
de0-nano-soc build has to be checked by CI due to high
importance.

Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
---
 scripts/ci_build.sh | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index acce152..9c2322f 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -126,6 +126,13 @@ fi
 # Start build for the defined set of configurations
 bitbake $BB_ARGS $TARGETS_SET
 
+# Note: de0-nano-soc build is launched separately as
+# parallel build with the same target arch (armhf) fails.
+# The problem is being investigated
+if [ -n "$FAST_BUILD" ]; then
+    bitbake $BB_ARGS multiconfig:de0-nano-soc-stretch:isar-image-base
+fi
+
 cp -a "${ISARROOT}/meta/classes/dpkg-base.bbclass" "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup"
 echo -e "do_fetch_append() {\n\n}" >> "${ISARROOT}/meta/classes/dpkg-base.bbclass"
 
-- 
2.11.0


^ permalink raw reply	[flat|nested] 11+ messages in thread

* [PATCH 4/4] ci: Disable use of cached base repository for fast build by default
  2019-02-14 16:08 [PATCH 0/4] CI improvements Maxim Yu. Osipov
                   ` (2 preceding siblings ...)
  2019-02-14 16:08 ` [PATCH 3/4] ci: Include de0-nano-soc target into fast build Maxim Yu. Osipov
@ 2019-02-14 16:08 ` Maxim Yu. Osipov
  2019-02-15 14:42 ` [PATCH 0/4] CI improvements Maxim Yu. Osipov
  4 siblings, 0 replies; 11+ messages in thread
From: Maxim Yu. Osipov @ 2019-02-14 16:08 UTC (permalink / raw)
  To: isar-users

Build of de0-nano-soc introduced by previous commit
takes a lot of time as it results builds of Linux kernel
and u-boot. Disable the cached base repository usage for fast build case
by default to reduce overall build time. Pass '-r' parameter to ci_build.sh
if you want to use cached base repository.

Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
---
 scripts/ci_build.sh | 2 --
 1 file changed, 2 deletions(-)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 9c2322f..665bbd0 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -81,10 +81,8 @@ do
     -f|--fast)
         # Start build for the reduced set of configurations
         # Enforce cross-compilation to speed up the build
-        # Enable use of cached base repository
         FAST_BUILD="1"
         CROSS_BUILD="1"
-        REPRO_BUILD="1"
         TARGETS_SET="\
                      multiconfig:qemuarm-stretch:isar-image-base \
                      multiconfig:qemuarm64-stretch:isar-image-base \
-- 
2.11.0


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 3/4] ci: Include de0-nano-soc target into fast build
  2019-02-14 16:08 ` [PATCH 3/4] ci: Include de0-nano-soc target into fast build Maxim Yu. Osipov
@ 2019-02-15  7:20   ` Jan Kiszka
  2019-02-18  8:59     ` Claudius Heine
  2019-02-18 10:18     ` Henning Schild
  0 siblings, 2 replies; 11+ messages in thread
From: Jan Kiszka @ 2019-02-15  7:20 UTC (permalink / raw)
  To: Maxim Yu. Osipov, isar-users

On 14.02.19 17:08, Maxim Yu. Osipov wrote:
> Note: So far de0-nano-soc build is launched separately as
> parallel build with the same target arch (qemuarm) fails.

jailhouse-images builds multiple images out of the same arch for quite a while 
now. You may want to cross-check and adopt the patterns there.

Even if unrelated to this issue: A proper multiconfig file only contains two 
assignments: MACHINE and DISTRO. This should be eventually cleaned up in 
meta-isar so that user do not copy the wrong patterns.

Jan

> While the problem with parallel build is being investigated
> de0-nano-soc build has to be checked by CI due to high
> importance.
> 
> Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
> ---
>   scripts/ci_build.sh | 7 +++++++
>   1 file changed, 7 insertions(+)
> 
> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> index acce152..9c2322f 100755
> --- a/scripts/ci_build.sh
> +++ b/scripts/ci_build.sh
> @@ -126,6 +126,13 @@ fi
>   # Start build for the defined set of configurations
>   bitbake $BB_ARGS $TARGETS_SET
>   
> +# Note: de0-nano-soc build is launched separately as
> +# parallel build with the same target arch (armhf) fails.
> +# The problem is being investigated
> +if [ -n "$FAST_BUILD" ]; then
> +    bitbake $BB_ARGS multiconfig:de0-nano-soc-stretch:isar-image-base
> +fi
> +
>   cp -a "${ISARROOT}/meta/classes/dpkg-base.bbclass" "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup"
>   echo -e "do_fetch_append() {\n\n}" >> "${ISARROOT}/meta/classes/dpkg-base.bbclass"
>   
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 0/4] CI improvements
  2019-02-14 16:08 [PATCH 0/4] CI improvements Maxim Yu. Osipov
                   ` (3 preceding siblings ...)
  2019-02-14 16:08 ` [PATCH 4/4] ci: Disable use of cached base repository for fast build by default Maxim Yu. Osipov
@ 2019-02-15 14:42 ` Maxim Yu. Osipov
  4 siblings, 0 replies; 11+ messages in thread
From: Maxim Yu. Osipov @ 2019-02-15 14:42 UTC (permalink / raw)
  To: isar-users

On 2/14/19 5:08 PM, Maxim Yu. Osipov wrote:
> Hello everybody,
> 
> At the moment there is a confusion regarding CI build.
> "Fast" CI build (called with parameter '-f') _cross_ builds
> 3 most common (at least we think that) configurations
> with cached base repository feature enabled:
> 
>   * multiconfig:qemuarm-stretch:isar-image-base
>   * multiconfig:qemuarm64-stretch:isar-image-base
>   * multiconfig:qemuamd64-stretch:isar-image-base
> 
> When default (w/o parameters) CI is called it _natively_ builds
> the following set of configurations:
> 
>   * multiconfig:qemuarm-jessie:isar-image-base
>   * multiconfig:qemuarm-stretch:isar-image-base
>   * multiconfig:qemuarm-buster:isar-image-base
>   * multiconfig:qemuarm64-stretch:isar-image-base
>   * multiconfig:qemui386-jessie:isar-image-base
>   * multiconfig:qemui386-stretch:isar-image-base
>   * multiconfig:qemui386-buster:isar-image-base
>   * multiconfig:qemuamd64-jessie:isar-image-base
>   * multiconfig:qemuamd64-stretch:isar-image-base
>   * multiconfig:qemuamd64-buster:isar-image-base
>   * multiconfig:qemuamd64-buster-tgz:isar-image-base
>   * multiconfig:rpi-jessie:isar-image-base
> 
> Unfortunately this set doesn't include supported configurations
> (real boards)
>   * multiconfig:bananapi-stretch.conf
>   * hikey-stretch.conf
>   * de0-nano-soc-stretch.conf
> 
> The reason is that these configurations include build of Linux kernel/U-Boot so
> the native build will last huge amount of time.
> 
> So the problem with two flavours (fast and default) of CI build is that they
> test different features for the subset of all supported configurations.
> 
> After upgrade of CI server we intend to increase a test coverage -
>   * Include cross builds for mentioned boards hikey, de0-nano-soc etc.
>   * To use default build for overnight builds with different combination of features enabled
>   (native and cross, with or without used cached base repo etc.)
> 
> This patchset is the first step towards this direction.
> 
> Kind regards,
> Maxim.
> 

Applied to the 'next'.


> Maxim Yu. Osipov (4):
>    ci: Introduce TARGETS_SET variable
>    ci: Introduce --repro parameter to enable use of cached base
>      repository
>    ci: Include de0-nano-soc target into fast build
>    ci: Disable use of cached base repository for fast build by default
> 
>   scripts/ci_build.sh | 71 +++++++++++++++++++++++++++++++----------------------
>   1 file changed, 41 insertions(+), 30 deletions(-)
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 3/4] ci: Include de0-nano-soc target into fast build
  2019-02-15  7:20   ` Jan Kiszka
@ 2019-02-18  8:59     ` Claudius Heine
  2019-02-18 10:18     ` Henning Schild
  1 sibling, 0 replies; 11+ messages in thread
From: Claudius Heine @ 2019-02-18  8:59 UTC (permalink / raw)
  To: [ext] Jan Kiszka, Maxim Yu. Osipov, isar-users

Hi,

On 15/02/2019 08.20, [ext] Jan Kiszka wrote:
> On 14.02.19 17:08, Maxim Yu. Osipov wrote:
>> Note: So far de0-nano-soc build is launched separately as
>> parallel build with the same target arch (qemuarm) fails.
> 
> jailhouse-images builds multiple images out of the same arch for quite a 
> while now. You may want to cross-check and adopt the patterns there.
> 
> Even if unrelated to this issue: A proper multiconfig file only contains 
> two assignments: MACHINE and DISTRO. This should be eventually cleaned 
> up in meta-isar so that user do not copy the wrong patterns.

For tracking purposes I opened this issue on gh:
https://github.com/ilbers/isar/issues/44

regards,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 3/4] ci: Include de0-nano-soc target into fast build
  2019-02-15  7:20   ` Jan Kiszka
  2019-02-18  8:59     ` Claudius Heine
@ 2019-02-18 10:18     ` Henning Schild
  2019-02-18 10:23       ` Jan Kiszka
  2019-02-18 10:29       ` Claudius Heine
  1 sibling, 2 replies; 11+ messages in thread
From: Henning Schild @ 2019-02-18 10:18 UTC (permalink / raw)
  To: [ext] Jan Kiszka; +Cc: Maxim Yu. Osipov, isar-users

Am Fri, 15 Feb 2019 08:20:19 +0100
schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:

> On 14.02.19 17:08, Maxim Yu. Osipov wrote:
> > Note: So far de0-nano-soc build is launched separately as
> > parallel build with the same target arch (qemuarm) fails.  
> 
> jailhouse-images builds multiple images out of the same arch for
> quite a while now. You may want to cross-check and adopt the patterns
> there.
> 
> Even if unrelated to this issue: A proper multiconfig file only
> contains two assignments: MACHINE and DISTRO. This should be
> eventually cleaned up in meta-isar so that user do not copy the wrong
> patterns.

It is most likely false-sharing on multiconfig. Now if we look for
"[stamp-extra-info]" in the source-tree we will find several
combinations.

"${DISTRO}-${DISTRO_ARCH}"
"${DISTRO}-${MACHINE}"
"${MACHINE}-${DISTRO}"
"${DISTRO}"

And possibly several tasks that should have an extra-stamp but do not.
Avoiding multiconfig is a very good idea, not just for CI ;).

Henning

> Jan
> 
> > While the problem with parallel build is being investigated
> > de0-nano-soc build has to be checked by CI due to high
> > importance.
> > 
> > Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
> > ---
> >   scripts/ci_build.sh | 7 +++++++
> >   1 file changed, 7 insertions(+)
> > 
> > diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> > index acce152..9c2322f 100755
> > --- a/scripts/ci_build.sh
> > +++ b/scripts/ci_build.sh
> > @@ -126,6 +126,13 @@ fi
> >   # Start build for the defined set of configurations
> >   bitbake $BB_ARGS $TARGETS_SET
> >   
> > +# Note: de0-nano-soc build is launched separately as
> > +# parallel build with the same target arch (armhf) fails.
> > +# The problem is being investigated
> > +if [ -n "$FAST_BUILD" ]; then
> > +    bitbake $BB_ARGS
> > multiconfig:de0-nano-soc-stretch:isar-image-base +fi
> > +
> >   cp -a "${ISARROOT}/meta/classes/dpkg-base.bbclass"
> > "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup" echo -e
> > "do_fetch_append() {\n\n}" >>
> > "${ISARROOT}/meta/classes/dpkg-base.bbclass" 
> 


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 3/4] ci: Include de0-nano-soc target into fast build
  2019-02-18 10:18     ` Henning Schild
@ 2019-02-18 10:23       ` Jan Kiszka
  2019-02-18 10:29       ` Claudius Heine
  1 sibling, 0 replies; 11+ messages in thread
From: Jan Kiszka @ 2019-02-18 10:23 UTC (permalink / raw)
  To: Henning Schild; +Cc: Maxim Yu. Osipov, isar-users

On 18.02.19 11:18, Henning Schild wrote:
> Am Fri, 15 Feb 2019 08:20:19 +0100
> schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:
> 
>> On 14.02.19 17:08, Maxim Yu. Osipov wrote:
>>> Note: So far de0-nano-soc build is launched separately as
>>> parallel build with the same target arch (qemuarm) fails.
>>
>> jailhouse-images builds multiple images out of the same arch for
>> quite a while now. You may want to cross-check and adopt the patterns
>> there.
>>
>> Even if unrelated to this issue: A proper multiconfig file only
>> contains two assignments: MACHINE and DISTRO. This should be
>> eventually cleaned up in meta-isar so that user do not copy the wrong
>> patterns.
> 
> It is most likely false-sharing on multiconfig. Now if we look for
> "[stamp-extra-info]" in the source-tree we will find several
> combinations.
> 
> "${DISTRO}-${DISTRO_ARCH}"
> "${DISTRO}-${MACHINE}"
> "${MACHINE}-${DISTRO}"
> "${DISTRO}"
> 

There are likely broken stamps. We should sort them according to differentiating 
variables into subfolders, like oe does. And we should review the tagging at 
that chance, if there are no false sharings hidden.

> And possibly several tasks that should have an extra-stamp but do not.
> Avoiding multiconfig is a very good idea, not just for CI ;).

This remains wrong, even if you repeat it. It's not perfect, but it works in 
practice, solving way more problems than it introduces. But it remains an expert 
feature that should be advertised as such.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 3/4] ci: Include de0-nano-soc target into fast build
  2019-02-18 10:18     ` Henning Schild
  2019-02-18 10:23       ` Jan Kiszka
@ 2019-02-18 10:29       ` Claudius Heine
  1 sibling, 0 replies; 11+ messages in thread
From: Claudius Heine @ 2019-02-18 10:29 UTC (permalink / raw)
  To: [ext] Henning Schild, [ext] Jan Kiszka; +Cc: Maxim Yu. Osipov, isar-users

Hi Henning,

On 18/02/2019 11.18, [ext] Henning Schild wrote:
> Am Fri, 15 Feb 2019 08:20:19 +0100
> schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:
> 
>> On 14.02.19 17:08, Maxim Yu. Osipov wrote:
>>> Note: So far de0-nano-soc build is launched separately as
>>> parallel build with the same target arch (qemuarm) fails.
>>
>> jailhouse-images builds multiple images out of the same arch for
>> quite a while now. You may want to cross-check and adopt the patterns
>> there.
>>
>> Even if unrelated to this issue: A proper multiconfig file only
>> contains two assignments: MACHINE and DISTRO. This should be
>> eventually cleaned up in meta-isar so that user do not copy the wrong
>> patterns.
> 
> It is most likely false-sharing on multiconfig. Now if we look for
> "[stamp-extra-info]" in the source-tree we will find several
> combinations.
> 
> "${DISTRO}-${DISTRO_ARCH}"
> "${DISTRO}-${MACHINE}"
> "${MACHINE}-${DISTRO}"
> "${DISTRO}"
> 
> And possibly several tasks that should have an extra-stamp but do not.

Good point. That is also something that should be looked at before a 1.0 
release ;)

> Avoiding multiconfig is a very good idea, not just for CI ;).

I agree. Another way would be to set the tmp directory in the 
multiconfig like the bitbake manual documents:

https://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html#executing-a-multiple-configuration-build

regards,
Claudius

> 
> Henning
> 
>> Jan
>>
>>> While the problem with parallel build is being investigated
>>> de0-nano-soc build has to be checked by CI due to high
>>> importance.
>>>
>>> Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
>>> ---
>>>    scripts/ci_build.sh | 7 +++++++
>>>    1 file changed, 7 insertions(+)
>>>
>>> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
>>> index acce152..9c2322f 100755
>>> --- a/scripts/ci_build.sh
>>> +++ b/scripts/ci_build.sh
>>> @@ -126,6 +126,13 @@ fi
>>>    # Start build for the defined set of configurations
>>>    bitbake $BB_ARGS $TARGETS_SET
>>>    
>>> +# Note: de0-nano-soc build is launched separately as
>>> +# parallel build with the same target arch (armhf) fails.
>>> +# The problem is being investigated
>>> +if [ -n "$FAST_BUILD" ]; then
>>> +    bitbake $BB_ARGS
>>> multiconfig:de0-nano-soc-stretch:isar-image-base +fi
>>> +
>>>    cp -a "${ISARROOT}/meta/classes/dpkg-base.bbclass"
>>> "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup" echo -e
>>> "do_fetch_append() {\n\n}" >>
>>> "${ISARROOT}/meta/classes/dpkg-base.bbclass"
>>
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2019-02-18 10:30 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-02-14 16:08 [PATCH 0/4] CI improvements Maxim Yu. Osipov
2019-02-14 16:08 ` [PATCH 1/4] ci: Introduce TARGETS_SET variable Maxim Yu. Osipov
2019-02-14 16:08 ` [PATCH 2/4] ci: Introduce --repro parameter to enable use of cached base repository Maxim Yu. Osipov
2019-02-14 16:08 ` [PATCH 3/4] ci: Include de0-nano-soc target into fast build Maxim Yu. Osipov
2019-02-15  7:20   ` Jan Kiszka
2019-02-18  8:59     ` Claudius Heine
2019-02-18 10:18     ` Henning Schild
2019-02-18 10:23       ` Jan Kiszka
2019-02-18 10:29       ` Claudius Heine
2019-02-14 16:08 ` [PATCH 4/4] ci: Disable use of cached base repository for fast build by default Maxim Yu. Osipov
2019-02-15 14:42 ` [PATCH 0/4] CI improvements Maxim Yu. Osipov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox