[PATCH] scripts/ci_build.sh: Don't fail on bullseye build errors

[PATCH] scripts/ci_build.sh: Don't fail on bullseye build errors

[Baurzhan Ismagulov]

From: Baurzhan Ismagulov <ibr@ilbers.de>

Bullseye targets fail to build with the following error message:

ERROR: mc:qemuarm-bullseye:libhello-0.1-98f2e41-r0 do_install_builddeps: Execution of '/workspace/build/isar_master/447/build/tmp/work/debian-bullseye-armhf/libhello/0.1-98f2e41-r0/temp/run.do_install_builddeps.29112' failed with exit code 2:
...
dpkg-checkbuilddeps: error: Unmet build dependencies: build-essential:native

Workaround: If bullseye doesn't build, print a message but not mark the
build as failed.

Signed-off-by: Vlad Serebrennikov <vserebr@ilbers.de>
Signed-off-by: Baurzhan Ismagulov <ibr@ilbers.de>
---
 scripts/ci_build.sh | 29 ++++++++++++++++++++++++-----
 1 file changed, 24 insertions(+), 5 deletions(-)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index adc403b..d2c707b 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -22,33 +22,39 @@ BB_ARGS="-v"
 TARGETS_SET="\
             mc:qemuarm-stretch:isar-image-base \
             mc:qemuarm-buster:isar-image-base \
-            mc:qemuarm-bullseye:isar-image-base \
             mc:qemuarm64-stretch:isar-image-base \
             mc:qemui386-stretch:isar-image-base \
             mc:qemui386-buster:isar-image-base \
-            mc:qemui386-bullseye:isar-image-base \
             mc:qemuamd64-stretch:isar-image-base \
             mc:qemuamd64-buster:isar-image-base \
             mc:qemuamd64-buster-tgz:isar-image-base \
-            mc:qemuamd64-bullseye:isar-image-base \
             mc:qemumipsel-stretch:isar-image-base \
             mc:qemumipsel-buster:isar-image-base \
-            mc:qemumipsel-bullseye:isar-image-base \
             mc:nand-ubi-demo-buster:isar-image-ubi \
             mc:rpi-stretch:isar-image-base"
           # qemu-user-static of <= buster too old to build that
           # mc:qemuarm64-buster:isar-image-base
           # mc:qemuarm64-bullseye:isar-image-base
 
+TARGETS_SET_BULLSEYE="\
+    mc:qemuamd64-bullseye:isar-image-base \
+    mc:qemuarm-bullseye:isar-image-base \
+    mc:qemui386-bullseye:isar-image-base \
+    mc:qemumipsel-bullseye:isar-image-base \
+"
+
 CROSS_TARGETS_SET="\
                   mc:qemuarm-stretch:isar-image-base \
                   mc:qemuarm-buster:isar-image-base \
-                  mc:qemuarm-bullseye:isar-image-base \
                   mc:qemuarm64-stretch:isar-image-base \
                   mc:qemuamd64-stretch:isar-image-base \
                   mc:de0-nano-soc-stretch:isar-image-base \
                   mc:rpi-stretch:isar-image-base"
 
+CROSS_TARGETS_SET_BULLSEYE="\
+    mc:qemuarm-bullseye:isar-image-base \
+"
+
 REPRO_TARGETS_SET_SIGNED="\
             mc:de0-nano-soc-stretch:isar-image-base \
             mc:qemuarm64-stretch:isar-image-base"
@@ -176,6 +182,11 @@ fi
 sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
 bitbake $BB_ARGS $CROSS_TARGETS_SET
 while [ -e bitbake.sock ]; do sleep 1; done
+if bitbake $BB_ARGS $CROSS_TARGETS_SET_BULLSEYE; then
+    echo "bullseye cross: PASSED"
+else
+    echo "bullseye cross: KFAIL"
+fi
 # In addition test SDK creation
 bitbake $BB_ARGS -c do_populate_sdk mc:qemuarm-stretch:isar-image-base
 while [ -e bitbake.sock ]; do sleep 1; done
@@ -185,6 +196,14 @@ if [ -z "$FAST_BUILD" ]; then
     sudo rm -rf tmp
     sed -i -e 's/ISAR_CROSS_COMPILE ?= "1"/ISAR_CROSS_COMPILE ?= "0"/g' conf/local.conf
     bitbake $BB_ARGS $TARGETS_SET
+    while [ -e bitbake.sock ]; do sleep 1; done
+
+    if bitbake $BB_ARGS $TARGETS_SET_BULLSEYE; then
+        echo "bullseye: PASSED"
+    else
+	echo "bullseye: KFAIL"
+    fi
+    while [ -e bitbake.sock ]; do sleep 1; done
 fi
 
 cp -a "${ISARROOT}/meta/classes/dpkg-base.bbclass" "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup"
-- 
2.20.1

Re: [PATCH] scripts/ci_build.sh: Don't fail on bullseye build errors

[Jan Kiszka]

On 21.06.20 22:08, Baurzhan Ismagulov wrote:
> From: Baurzhan Ismagulov <ibr@ilbers.de>
> 
> Bullseye targets fail to build with the following error message:
> 
> ERROR: mc:qemuarm-bullseye:libhello-0.1-98f2e41-r0 do_install_builddeps: Execution of '/workspace/build/isar_master/447/build/tmp/work/debian-bullseye-armhf/libhello/0.1-98f2e41-r0/temp/run.do_install_builddeps.29112' failed with exit code 2:
> ...
> dpkg-checkbuilddeps: error: Unmet build dependencies: build-essential:native
> 
> Workaround: If bullseye doesn't build, print a message but not mark the
> build as failed.
> 

I'm carrying a disabling patch for bullseye in my queue as well.

There are two issues, in fact. The bigger one is the above. The smaller
one is that cowsay needs a refreshment to build again, but only for
bullseye.

Jan

> Signed-off-by: Vlad Serebrennikov <vserebr@ilbers.de>
> Signed-off-by: Baurzhan Ismagulov <ibr@ilbers.de>
> ---
>  scripts/ci_build.sh | 29 ++++++++++++++++++++++++-----
>  1 file changed, 24 insertions(+), 5 deletions(-)
> 
> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> index adc403b..d2c707b 100755
> --- a/scripts/ci_build.sh
> +++ b/scripts/ci_build.sh
> @@ -22,33 +22,39 @@ BB_ARGS="-v"
>  TARGETS_SET="\
>              mc:qemuarm-stretch:isar-image-base \
>              mc:qemuarm-buster:isar-image-base \
> -            mc:qemuarm-bullseye:isar-image-base \
>              mc:qemuarm64-stretch:isar-image-base \
>              mc:qemui386-stretch:isar-image-base \
>              mc:qemui386-buster:isar-image-base \
> -            mc:qemui386-bullseye:isar-image-base \
>              mc:qemuamd64-stretch:isar-image-base \
>              mc:qemuamd64-buster:isar-image-base \
>              mc:qemuamd64-buster-tgz:isar-image-base \
> -            mc:qemuamd64-bullseye:isar-image-base \
>              mc:qemumipsel-stretch:isar-image-base \
>              mc:qemumipsel-buster:isar-image-base \
> -            mc:qemumipsel-bullseye:isar-image-base \
>              mc:nand-ubi-demo-buster:isar-image-ubi \
>              mc:rpi-stretch:isar-image-base"
>            # qemu-user-static of <= buster too old to build that
>            # mc:qemuarm64-buster:isar-image-base
>            # mc:qemuarm64-bullseye:isar-image-base
>  
> +TARGETS_SET_BULLSEYE="\
> +    mc:qemuamd64-bullseye:isar-image-base \
> +    mc:qemuarm-bullseye:isar-image-base \
> +    mc:qemui386-bullseye:isar-image-base \
> +    mc:qemumipsel-bullseye:isar-image-base \
> +"
> +
>  CROSS_TARGETS_SET="\
>                    mc:qemuarm-stretch:isar-image-base \
>                    mc:qemuarm-buster:isar-image-base \
> -                  mc:qemuarm-bullseye:isar-image-base \
>                    mc:qemuarm64-stretch:isar-image-base \
>                    mc:qemuamd64-stretch:isar-image-base \
>                    mc:de0-nano-soc-stretch:isar-image-base \
>                    mc:rpi-stretch:isar-image-base"
>  
> +CROSS_TARGETS_SET_BULLSEYE="\
> +    mc:qemuarm-bullseye:isar-image-base \
> +"
> +
>  REPRO_TARGETS_SET_SIGNED="\
>              mc:de0-nano-soc-stretch:isar-image-base \
>              mc:qemuarm64-stretch:isar-image-base"
> @@ -176,6 +182,11 @@ fi
>  sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
>  bitbake $BB_ARGS $CROSS_TARGETS_SET
>  while [ -e bitbake.sock ]; do sleep 1; done
> +if bitbake $BB_ARGS $CROSS_TARGETS_SET_BULLSEYE; then
> +    echo "bullseye cross: PASSED"
> +else
> +    echo "bullseye cross: KFAIL"
> +fi
>  # In addition test SDK creation
>  bitbake $BB_ARGS -c do_populate_sdk mc:qemuarm-stretch:isar-image-base
>  while [ -e bitbake.sock ]; do sleep 1; done
> @@ -185,6 +196,14 @@ if [ -z "$FAST_BUILD" ]; then
>      sudo rm -rf tmp
>      sed -i -e 's/ISAR_CROSS_COMPILE ?= "1"/ISAR_CROSS_COMPILE ?= "0"/g' conf/local.conf
>      bitbake $BB_ARGS $TARGETS_SET
> +    while [ -e bitbake.sock ]; do sleep 1; done
> +
> +    if bitbake $BB_ARGS $TARGETS_SET_BULLSEYE; then
> +        echo "bullseye: PASSED"
> +    else
> +	echo "bullseye: KFAIL"
> +    fi
> +    while [ -e bitbake.sock ]; do sleep 1; done
>  fi
>  
>  cp -a "${ISARROOT}/meta/classes/dpkg-base.bbclass" "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup"
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH] scripts/ci_build.sh: Don't fail on bullseye build errors

[Henning Schild]

We need a real fix for this. And if we go for a temporary skip we need
a proper testing framework ...

With this hack all we do is potentially introduce more problems for
bullseye which will hit us when we have the fix, which we need anyways.

Henning

Am Sun, 21 Jun 2020 22:08:43 +0200
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> From: Baurzhan Ismagulov <ibr@ilbers.de>
> 
> Bullseye targets fail to build with the following error message:
> 
> ERROR: mc:qemuarm-bullseye:libhello-0.1-98f2e41-r0
> do_install_builddeps: Execution of
> '/workspace/build/isar_master/447/build/tmp/work/debian-bullseye-armhf/libhello/0.1-98f2e41-r0/temp/run.do_install_builddeps.29112'
> failed with exit code 2: ... dpkg-checkbuilddeps: error: Unmet build
> dependencies: build-essential:native
> 
> Workaround: If bullseye doesn't build, print a message but not mark
> the build as failed.
> 
> Signed-off-by: Vlad Serebrennikov <vserebr@ilbers.de>
> Signed-off-by: Baurzhan Ismagulov <ibr@ilbers.de>
> ---
>  scripts/ci_build.sh | 29 ++++++++++++++++++++++++-----
>  1 file changed, 24 insertions(+), 5 deletions(-)
> 
> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> index adc403b..d2c707b 100755
> --- a/scripts/ci_build.sh
> +++ b/scripts/ci_build.sh
> @@ -22,33 +22,39 @@ BB_ARGS="-v"
>  TARGETS_SET="\
>              mc:qemuarm-stretch:isar-image-base \
>              mc:qemuarm-buster:isar-image-base \
> -            mc:qemuarm-bullseye:isar-image-base \
>              mc:qemuarm64-stretch:isar-image-base \
>              mc:qemui386-stretch:isar-image-base \
>              mc:qemui386-buster:isar-image-base \
> -            mc:qemui386-bullseye:isar-image-base \
>              mc:qemuamd64-stretch:isar-image-base \
>              mc:qemuamd64-buster:isar-image-base \
>              mc:qemuamd64-buster-tgz:isar-image-base \
> -            mc:qemuamd64-bullseye:isar-image-base \
>              mc:qemumipsel-stretch:isar-image-base \
>              mc:qemumipsel-buster:isar-image-base \
> -            mc:qemumipsel-bullseye:isar-image-base \
>              mc:nand-ubi-demo-buster:isar-image-ubi \
>              mc:rpi-stretch:isar-image-base"
>            # qemu-user-static of <= buster too old to build that
>            # mc:qemuarm64-buster:isar-image-base
>            # mc:qemuarm64-bullseye:isar-image-base
>  
> +TARGETS_SET_BULLSEYE="\
> +    mc:qemuamd64-bullseye:isar-image-base \
> +    mc:qemuarm-bullseye:isar-image-base \
> +    mc:qemui386-bullseye:isar-image-base \
> +    mc:qemumipsel-bullseye:isar-image-base \
> +"
> +
>  CROSS_TARGETS_SET="\
>                    mc:qemuarm-stretch:isar-image-base \
>                    mc:qemuarm-buster:isar-image-base \
> -                  mc:qemuarm-bullseye:isar-image-base \
>                    mc:qemuarm64-stretch:isar-image-base \
>                    mc:qemuamd64-stretch:isar-image-base \
>                    mc:de0-nano-soc-stretch:isar-image-base \
>                    mc:rpi-stretch:isar-image-base"
>  
> +CROSS_TARGETS_SET_BULLSEYE="\
> +    mc:qemuarm-bullseye:isar-image-base \
> +"
> +
>  REPRO_TARGETS_SET_SIGNED="\
>              mc:de0-nano-soc-stretch:isar-image-base \
>              mc:qemuarm64-stretch:isar-image-base"
> @@ -176,6 +182,11 @@ fi
>  sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g'
> conf/local.conf bitbake $BB_ARGS $CROSS_TARGETS_SET
>  while [ -e bitbake.sock ]; do sleep 1; done
> +if bitbake $BB_ARGS $CROSS_TARGETS_SET_BULLSEYE; then
> +    echo "bullseye cross: PASSED"
> +else
> +    echo "bullseye cross: KFAIL"
> +fi
>  # In addition test SDK creation
>  bitbake $BB_ARGS -c do_populate_sdk
> mc:qemuarm-stretch:isar-image-base while [ -e bitbake.sock ]; do
> sleep 1; done @@ -185,6 +196,14 @@ if [ -z "$FAST_BUILD" ]; then
>      sudo rm -rf tmp
>      sed -i -e 's/ISAR_CROSS_COMPILE ?= "1"/ISAR_CROSS_COMPILE ?=
> "0"/g' conf/local.conf bitbake $BB_ARGS $TARGETS_SET
> +    while [ -e bitbake.sock ]; do sleep 1; done
> +
> +    if bitbake $BB_ARGS $TARGETS_SET_BULLSEYE; then
> +        echo "bullseye: PASSED"
> +    else
> +	echo "bullseye: KFAIL"
> +    fi
> +    while [ -e bitbake.sock ]; do sleep 1; done
>  fi
>  
>  cp -a "${ISARROOT}/meta/classes/dpkg-base.bbclass"
> "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup"

Re: [PATCH] scripts/ci_build.sh: Don't fail on bullseye build errors

[Baurzhan Ismagulov]

On Mon, Jun 22, 2020 at 12:42:51PM +0200, Henning Schild wrote:
> We need a real fix for this. And if we go for a temporary skip we need
> a proper testing framework ...
> 
> With this hack all we do is potentially introduce more problems for
> bullseye which will hit us when we have the fix, which we need anyways.

I agree that bullseye must be fixed. The problem is upstream, so that will take
time.

I also agree that we need a proper testing framework. It is actually in place.
As desired, every scenario has its own test case (a, b, c). However,
ci_build.sh tests many things in at once (a + b + c). Due to that, the test
cases don't result in the same coverage. Just adding the two would double the
time needed (a, b, c, a + b + c). So, I'd like to find a good compromise.

I agree that KFAIL may potentially hide further problems with bullseye.
However, as you mention in the other thread, it also blocks CI -- that is, any
work. That is why I think that unblocking CI has priority over fixing bullseye.

Applied to next.

With kind regards,
Baurzhan.

Re: [PATCH] scripts/ci_build.sh: Don't fail on bullseye build errors

[Baurzhan Ismagulov]

Hello Jan,

On Mon, Jun 22, 2020 at 09:39:04AM +0200, Jan Kiszka wrote:
> I'm carrying a disabling patch for bullseye in my queue as well.
> 
> There are two issues, in fact. The bigger one is the above. The smaller
> one is that cowsay needs a refreshment to build again, but only for
> bullseye.

Could you please share the cowsay one, too?

With kind regards,
Baurzhan.

Re: [PATCH] scripts/ci_build.sh: Don't fail on bullseye build errors

[Jan Kiszka]

On 30.07.20 18:09, Baurzhan Ismagulov wrote:
> Hello Jan,
> 
> On Mon, Jun 22, 2020 at 09:39:04AM +0200, Jan Kiszka wrote:
>> I'm carrying a disabling patch for bullseye in my queue as well.
>>
>> There are two issues, in fact. The bigger one is the above. The smaller
>> one is that cowsay needs a refreshment to build again, but only for
>> bullseye.
> 
> Could you please share the cowsay one, too?

We need to bump the cowsay git uri to bullseye compatible revision while
keeping it at the current one for buster and before.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH] scripts/ci_build.sh: Don't fail on bullseye build errors

[Henning Schild]

On Thu, 30 Jul 2020 18:08:55 +0200
Baurzhan Ismagulov <ibr@radix50.net> wrote:

> On Mon, Jun 22, 2020 at 12:42:51PM +0200, Henning Schild wrote:
> > We need a real fix for this. And if we go for a temporary skip we
> > need a proper testing framework ...
> > 
> > With this hack all we do is potentially introduce more problems for
> > bullseye which will hit us when we have the fix, which we need
> > anyways.  
> 
> I agree that bullseye must be fixed. The problem is upstream, so that
> will take time.

Did you report it upstream? Where is the bug? With you hiding the issue
it is you who needs to do the upstream work.

Henning

> I also agree that we need a proper testing framework. It is actually
> in place. As desired, every scenario has its own test case (a, b, c).
> However, ci_build.sh tests many things in at once (a + b + c). Due to
> that, the test cases don't result in the same coverage. Just adding
> the two would double the time needed (a, b, c, a + b + c). So, I'd
> like to find a good compromise.
> 
> I agree that KFAIL may potentially hide further problems with
> bullseye. However, as you mention in the other thread, it also blocks
> CI -- that is, any work. That is why I think that unblocking CI has
> priority over fixing bullseye.
> 
> Applied to next.
> 
> With kind regards,
> Baurzhan.
>