[PATCH 0/3] Update scripts

[PATCH 0/3] Update scripts

[Alexander Smirnov]

Hi all,

the key purpose of this series is to make ci and test scripts more
generic and easy to use by generic user. Also this series adds optional
verbose mode for these scripts.

After applying this series, Jenkins server will be updated to support
verbose output.

Now to build all Isar configuration user may run:

 $ scripts/ci_build.sh [-v]

To test all these configurations:

 $ cd build
 $ vm_smoke_test [-v]

With best regards,
Alex

Alexander Smirnov (3):
  scripts/vm_smoke_test: Add verbosity parameter
  scripts/vm_smoke_test: Check if QEMU didn't start
  scripts/ci_build: Make CI script more generic

 scripts/ci_build.sh   | 66 +++++++++++++++++++++++++++++++-------
 scripts/vm_smoke_test | 89 ++++++++++++++++++++++++++++++++++++++++++---------
 2 files changed, 128 insertions(+), 27 deletions(-)

-- 
2.1.4

[PATCH 1/3] scripts/vm_smoke_test: Add verbosity parameter

[Alexander Smirnov]

Issues:
1. Currently this script displays boot log for all the images. This could
   bring some inconviniences for users, who want to perform quick test and
   get just messages FAILED/PASSED.
2. Also the boot log is displayed if test case failed. So in the worst case
   when all cases for current image failed, the boot log will be displayed
   3 times.

Solution:
1. If at least one test case failed for current image, the boot log will be
   displayed (only one time, despite on number of cases failed).
2. Boot log is disabled by default. It could be enabled for all the images
   by '-v' option passed to this script.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 scripts/vm_smoke_test | 62 ++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 54 insertions(+), 8 deletions(-)

diff --git a/scripts/vm_smoke_test b/scripts/vm_smoke_test
index 274ec1d..92fda2d 100755
--- a/scripts/vm_smoke_test
+++ b/scripts/vm_smoke_test
@@ -5,13 +5,19 @@
 
 CONSOLE_OUTPUT=/tmp/isar_console
 PID_FILE=/tmp/qemu.pid
+VERBOSE=0
 
-RET=0
+# Error codes:
+ES_OK=0
+ES_FAIL=1
+ES_BUG=3
+
+RET=$ES_OK
 
 dump_boot_log() {
-    echo "Boot log: 8<--"
+    echo "Boot log:\n8<--"
     cat $CONSOLE_OUTPUT
-    echo -e "\n8<--"
+    echo "\n8<--"
 }
 
 check_login_prompt() {
@@ -23,8 +29,8 @@ check_login_prompt() {
         echo "PASSED"
     else
         echo "FAIL"
-        dump_boot_log
-        RET=1
+        RET=$ES_FAIL
+        FAIL=1
     fi
 }
 
@@ -37,8 +43,8 @@ check_example_module() {
         echo "PASSED"
     else
         echo "FAIL"
-        dump_boot_log
-        RET=1
+        RET=$ES_FAIL
+        FAIL=1
     fi
 }
 
@@ -54,19 +60,59 @@ run_test () {
     sleep 60
     kill `cat $PID_FILE`
 
+    FAIL=0
+
     # Check login prompt
     check_login_prompt
 
     # Check kernel module
     check_example_module
 
-    dump_boot_log
+    [ $VERBOSE -eq 1 -o $FAIL -eq 1 ] && dump_boot_log
 
     # Clean up test artifacts
     rm $CONSOLE_OUTPUT
     rm $PID_FILE
 }
 
+show_help() {
+    echo "This script tests all the default Isar images in QEMU."
+    echo
+    echo "Usage:"
+    echo "    $0 [params]"
+    echo
+    echo "Parameters:"
+    echo "    -v, --verbose         display boot logs for all the targets."
+    echo "    -h, --help            display this message and exit."
+    echo
+    echo "Exit status:"
+    echo " 0  if OK,"
+    echo " 1  if test failed,"
+    echo " 3  if invalid parameters are passed."
+}
+
+# Parse command line to get user configuration
+while [ $# -gt 0 ]
+do
+    key="$1"
+
+    case $key in
+    -h|--help)
+        show_help
+        exit 0
+        ;;
+    -v|--verbose)
+        VERBOSE=1
+        ;;
+    *)
+        echo "error: invalid parameter '$key', please try '--help' to get list of supported parameters"
+        exit $ES_BUG
+        ;;
+    esac
+
+    shift
+done
+
 # ARM machine
 run_test arm wheezy
 run_test arm jessie
-- 
2.1.4

Re: [PATCH 1/3] scripts/vm_smoke_test: Add verbosity parameter

[Henning Schild]

Am Tue, 1 May 2018 20:55:09 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Issues:
> 1. Currently this script displays boot log for all the images. This
> could bring some inconviniences for users, who want to perform quick
> test and get just messages FAILED/PASSED.
> 2. Also the boot log is displayed if test case failed. So in the
> worst case when all cases for current image failed, the boot log will
> be displayed 3 times.

Both are features not issues. If one really wants less output you could
introduce an argument to reduce verbosity (-nv). Or please add the "-v"
to your CI setup. A CI build takes quite some time/ressources so it is
a good idea to keep logs by default to later understand why things
failed.

Henning

> Solution:
> 1. If at least one test case failed for current image, the boot log
> will be displayed (only one time, despite on number of cases failed).
> 2. Boot log is disabled by default. It could be enabled for all the
> images by '-v' option passed to this script.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  scripts/vm_smoke_test | 62
> ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed,
> 54 insertions(+), 8 deletions(-)
> 
> diff --git a/scripts/vm_smoke_test b/scripts/vm_smoke_test
> index 274ec1d..92fda2d 100755
> --- a/scripts/vm_smoke_test
> +++ b/scripts/vm_smoke_test
> @@ -5,13 +5,19 @@
>  
>  CONSOLE_OUTPUT=/tmp/isar_console
>  PID_FILE=/tmp/qemu.pid
> +VERBOSE=0
>  
> -RET=0
> +# Error codes:
> +ES_OK=0
> +ES_FAIL=1
> +ES_BUG=3
> +
> +RET=$ES_OK
>  
>  dump_boot_log() {
> -    echo "Boot log: 8<--"
> +    echo "Boot log:\n8<--"
>      cat $CONSOLE_OUTPUT
> -    echo -e "\n8<--"
> +    echo "\n8<--"
>  }
>  
>  check_login_prompt() {
> @@ -23,8 +29,8 @@ check_login_prompt() {
>          echo "PASSED"
>      else
>          echo "FAIL"
> -        dump_boot_log
> -        RET=1
> +        RET=$ES_FAIL
> +        FAIL=1
>      fi
>  }
>  
> @@ -37,8 +43,8 @@ check_example_module() {
>          echo "PASSED"
>      else
>          echo "FAIL"
> -        dump_boot_log
> -        RET=1
> +        RET=$ES_FAIL
> +        FAIL=1
>      fi
>  }
>  
> @@ -54,19 +60,59 @@ run_test () {
>      sleep 60
>      kill `cat $PID_FILE`
>  
> +    FAIL=0
> +
>      # Check login prompt
>      check_login_prompt
>  
>      # Check kernel module
>      check_example_module
>  
> -    dump_boot_log
> +    [ $VERBOSE -eq 1 -o $FAIL -eq 1 ] && dump_boot_log
>  
>      # Clean up test artifacts
>      rm $CONSOLE_OUTPUT
>      rm $PID_FILE
>  }
>  
> +show_help() {
> +    echo "This script tests all the default Isar images in QEMU."
> +    echo
> +    echo "Usage:"
> +    echo "    $0 [params]"
> +    echo
> +    echo "Parameters:"
> +    echo "    -v, --verbose         display boot logs for all the
> targets."
> +    echo "    -h, --help            display this message and exit."
> +    echo
> +    echo "Exit status:"
> +    echo " 0  if OK,"
> +    echo " 1  if test failed,"
> +    echo " 3  if invalid parameters are passed."
> +}
> +
> +# Parse command line to get user configuration
> +while [ $# -gt 0 ]
> +do
> +    key="$1"
> +
> +    case $key in
> +    -h|--help)
> +        show_help
> +        exit 0
> +        ;;
> +    -v|--verbose)
> +        VERBOSE=1
> +        ;;
> +    *)
> +        echo "error: invalid parameter '$key', please try '--help'
> to get list of supported parameters"
> +        exit $ES_BUG
> +        ;;
> +    esac
> +
> +    shift
> +done
> +
>  # ARM machine
>  run_test arm wheezy
>  run_test arm jessie

[PATCH 2/3] scripts/vm_smoke_test: Check if QEMU didn't start

[Alexander Smirnov]

Add check for QEMU start and display error message if it didn't start.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 scripts/vm_smoke_test | 31 +++++++++++++++++++++----------
 1 file changed, 21 insertions(+), 10 deletions(-)

diff --git a/scripts/vm_smoke_test b/scripts/vm_smoke_test
index 92fda2d..006ab53 100755
--- a/scripts/vm_smoke_test
+++ b/scripts/vm_smoke_test
@@ -57,21 +57,32 @@ run_test () {
 
     # Start QEMU with Isar image
     start_vm -a $ARCH -d $DISTRO -o $CONSOLE_OUTPUT -p $PID_FILE > /dev/null 2>&1 &
-    sleep 60
-    kill `cat $PID_FILE`
+    sleep 5
 
-    FAIL=0
+    if [ -z `ps -p $! -o pid=` ]; then
+        echo "QEMU start: FAILED"
+        RET=$ES_FAIL
+        echo "Command output:\n8<--"
+        start_vm -a $ARCH -d $DISTRO -o $CONSOLE_OUTPUT -p $PID_FILE
+        echo "\n8<--"
+    else
+        sleep 60
+        kill `cat $PID_FILE`
+
+        FAIL=0
 
-    # Check login prompt
-    check_login_prompt
+        # Check login prompt
+        check_login_prompt
 
-    # Check kernel module
-    check_example_module
+        # Check kernel module
+        check_example_module
 
-    [ $VERBOSE -eq 1 -o $FAIL -eq 1 ] && dump_boot_log
+        [ $VERBOSE -eq 1 -o $FAIL -eq 1 ] && dump_boot_log
+
+        # Clean up test artifacts
+        rm $CONSOLE_OUTPUT
+    fi
 
-    # Clean up test artifacts
-    rm $CONSOLE_OUTPUT
     rm $PID_FILE
 }
 
-- 
2.1.4

Re: [PATCH 2/3] scripts/vm_smoke_test: Check if QEMU didn't start

[Henning Schild]

Am Tue, 1 May 2018 20:55:10 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Add check for QEMU start and display error message if it didn't start.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  scripts/vm_smoke_test | 31 +++++++++++++++++++++----------
>  1 file changed, 21 insertions(+), 10 deletions(-)
> 
> diff --git a/scripts/vm_smoke_test b/scripts/vm_smoke_test
> index 92fda2d..006ab53 100755
> --- a/scripts/vm_smoke_test
> +++ b/scripts/vm_smoke_test
> @@ -57,21 +57,32 @@ run_test () {
>  
>      # Start QEMU with Isar image
>      start_vm -a $ARCH -d $DISTRO -o $CONSOLE_OUTPUT -p $PID_FILE
> > /dev/null 2>&1 &
> -    sleep 60
> -    kill `cat $PID_FILE`
> +    sleep 5
>  
> -    FAIL=0
> +    if [ -z `ps -p $! -o pid=` ]; then

[ -d /proc/$! ]

> +        echo "QEMU start: FAILED"
> +        RET=$ES_FAIL
> +        echo "Command output:\n8<--"
> +        start_vm -a $ARCH -d $DISTRO -o $CONSOLE_OUTPUT -p $PID_FILE
> +        echo "\n8<--"
> +    else
> +        sleep 60
> +        kill `cat $PID_FILE`
> +
> +        FAIL=0
>  
> -    # Check login prompt
> -    check_login_prompt
> +        # Check login prompt
> +        check_login_prompt

i think the function name says all there is to say
 
> -    # Check kernel module
> -    check_example_module
> +        # Check kernel module
> +        check_example_module

same here

Henning

> -    [ $VERBOSE -eq 1 -o $FAIL -eq 1 ] && dump_boot_log
> +        [ $VERBOSE -eq 1 -o $FAIL -eq 1 ] && dump_boot_log
> +
> +        # Clean up test artifacts
> +        rm $CONSOLE_OUTPUT
> +    fi
>  
> -    # Clean up test artifacts
> -    rm $CONSOLE_OUTPUT
>      rm $PID_FILE
>  }
>  

[PATCH 3/3] scripts/ci_build: Make CI script more generic

[Alexander Smirnov]

Make this script more generic, so it could be used by any CI and by user
as well. Also add parameter to specify build folder.

Usage:
  ci_build [-v] -b BUILD_DIR

And this command will build all the default Isar images.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 scripts/ci_build.sh | 66 ++++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 55 insertions(+), 11 deletions(-)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index e941f8c..6b5871e 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -1,28 +1,72 @@
-# Script for Jenkins build
+# Script for CI system build
 #
 # Alexander Smirnov <asmirnov@ilbers.de>
-# Copyright (c) 2016-2017 ilbers GmbH
+# Copyright (c) 2016-2018 ilbers GmbH
 
 #!/bin/sh
 
+ES_BUG=3
+
 # Export $PATH to use 'parted' tool
 export PATH=$PATH:/sbin
 
-# Get parameters from the command line
-WORKSPACE=$1
-GIT_COMMIT=$2
-
 # Go to Isar root
 cd $(dirname $0)/..
 
-# Setup build folder for current revision
-if [ ! -d /build/$WORKSPACE/$GIT_COMMIT ]; then
-        mkdir -p /build/$WORKSPACE/$GIT_COMMIT
+# Start build in Isar tree by default
+BUILD_DIR=./build
+
+show_help() {
+    echo "This script builds all the default Isar images."
+    echo
+    echo "Usage:"
+    echo "    $0 [params]"
+    echo
+    echo "Parameters:"
+    echo "    -b, --build BUILD_DIR set path to build directory. If not set,"
+    echo "                          the build will be started in current path."
+    echo "    -v, --verbose         set verbose level of bitbake output."
+    echo "    --help                display this message and exit."
+    echo
+    echo "Exit status:"
+    echo " 0  if OK,"
+    echo " 3  if invalid parameters are passed."
+}
+
+# Parse command line to get user configuration
+while [ $# -gt 0 ]
+do
+    key="$1"
+
+    case $key in
+    -h|--help)
+        show_help
+        exit 0
+        ;;
+    -b|--build)
+        BUILD_DIR="$2"
+        shift
+        ;;
+    -v|--verbose)
+        BB_ARGS="$BB_ARGS -v"
+        ;;
+    *)
+        echo "error: invalid parameter '$key', please try '--help' to get list of supported parameters"
+        exit $ES_BUG
+        ;;
+    esac
+
+    shift
+done
+
+# Setup build folder for the current build
+if [ ! -d $BUILD_DIR ]; then
+        mkdir -p $BUILD_DIR
 fi
-source isar-init-build-env /build/$WORKSPACE/$GIT_COMMIT
+source isar-init-build-env $BUILD_DIR
 
 # Start build for all possible configurations
-bitbake -v \
+bitbake $BB_ARGS \
         multiconfig:qemuarm-wheezy:isar-image-base \
         multiconfig:qemuarm-jessie:isar-image-base \
         multiconfig:qemuarm-stretch:isar-image-base \
-- 
2.1.4

Re: [PATCH 3/3] scripts/ci_build: Make CI script more generic

[Henning Schild]

Am Tue, 1 May 2018 20:55:11 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Make this script more generic, so it could be used by any CI and by
> user as well. Also add parameter to specify build folder.
> 
> Usage:
>   ci_build [-v] -b BUILD_DIR
> 
> And this command will build all the default Isar images.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  scripts/ci_build.sh | 66
> ++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed,
> 55 insertions(+), 11 deletions(-)
> 
> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> index e941f8c..6b5871e 100755
> --- a/scripts/ci_build.sh
> +++ b/scripts/ci_build.sh
> @@ -1,28 +1,72 @@
> -# Script for Jenkins build
> +# Script for CI system build
>  #
>  # Alexander Smirnov <asmirnov@ilbers.de>
> -# Copyright (c) 2016-2017 ilbers GmbH
> +# Copyright (c) 2016-2018 ilbers GmbH
>  
>  #!/bin/sh
>  
> +ES_BUG=3
> +
>  # Export $PATH to use 'parted' tool
>  export PATH=$PATH:/sbin
>  
> -# Get parameters from the command line
> -WORKSPACE=$1
> -GIT_COMMIT=$2
> -
>  # Go to Isar root
>  cd $(dirname $0)/..
>  
> -# Setup build folder for current revision
> -if [ ! -d /build/$WORKSPACE/$GIT_COMMIT ]; then
> -        mkdir -p /build/$WORKSPACE/$GIT_COMMIT
> +# Start build in Isar tree by default
> +BUILD_DIR=./build
> +
> +show_help() {
> +    echo "This script builds all the default Isar images."
> +    echo
> +    echo "Usage:"
> +    echo "    $0 [params]"
> +    echo
> +    echo "Parameters:"
> +    echo "    -b, --build BUILD_DIR set path to build directory. If
> not set,"
> +    echo "                          the build will be started in
> current path."
> +    echo "    -v, --verbose         set verbose level of bitbake
> output."
> +    echo "    --help                display this message and exit."
> +    echo
> +    echo "Exit status:"
> +    echo " 0  if OK,"
> +    echo " 3  if invalid parameters are passed."
> +}
> +
> +# Parse command line to get user configuration
> +while [ $# -gt 0 ]
> +do
> +    key="$1"
> +
> +    case $key in
> +    -h|--help)
> +        show_help
> +        exit 0
> +        ;;
> +    -b|--build)
> +        BUILD_DIR="$2"
> +        shift
> +        ;;
> +    -v|--verbose)
> +        BB_ARGS="$BB_ARGS -v"
> +        ;;
> +    *)
> +        echo "error: invalid parameter '$key', please try '--help'
> to get list of supported parameters"
> +        exit $ES_BUG
> +        ;;
> +    esac
> +
> +    shift
> +done
> +
> +# Setup build folder for the current build
> +if [ ! -d $BUILD_DIR ]; then
> +        mkdir -p $BUILD_DIR
>  fi
> -source isar-init-build-env /build/$WORKSPACE/$GIT_COMMIT
> +source isar-init-build-env $BUILD_DIR
>  
>  # Start build for all possible configurations
> -bitbake -v \
> +bitbake $BB_ARGS \

Looks to me like there is a case where BB_ARGS is not initialized and
that the default "-v" got dropped. Again this output is valuable and
should probably be there by default.

Henning

>          multiconfig:qemuarm-wheezy:isar-image-base \
>          multiconfig:qemuarm-jessie:isar-image-base \
>          multiconfig:qemuarm-stretch:isar-image-base \

Re: [PATCH 0/3] Update scripts

[Henning Schild]

Am Tue, 1 May 2018 20:55:08 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Hi all,
> 
> the key purpose of this series is to make ci and test scripts more
> generic and easy to use by generic user. Also this series adds
> optional verbose mode for these scripts.
> 
> After applying this series, Jenkins server will be updated to support
> verbose output.

Read this last, sorry. If jenkins will add the "-v" it is OK but i
would still prefer the other way around (-nv). Other people setting up
their CI might forget the vs only to find them later.

Henning

> Now to build all Isar configuration user may run:
> 
>  $ scripts/ci_build.sh [-v]
> 
> To test all these configurations:
> 
>  $ cd build
>  $ vm_smoke_test [-v]
> 
> With best regards,
> Alex
> 
> Alexander Smirnov (3):
>   scripts/vm_smoke_test: Add verbosity parameter
>   scripts/vm_smoke_test: Check if QEMU didn't start
>   scripts/ci_build: Make CI script more generic
> 
>  scripts/ci_build.sh   | 66 +++++++++++++++++++++++++++++++-------
>  scripts/vm_smoke_test | 89
> ++++++++++++++++++++++++++++++++++++++++++--------- 2 files changed,
> 128 insertions(+), 27 deletions(-)
>