[PATCH v2 0/4] CI rework of gitlab fast job

[PATCH v2 0/4] CI rework of gitlab fast job

[Henning Schild]

changes since v1:
 - squash v1p4-v1p8 into v2p4 and add some generic reasoning

This series contains some changes to the CI as done on gitlab but also
affects testing in general. The main idea is to run qemu basically every
time we build an image (only fast for gitlab at the moment), so we do
not just build it without testing.
On the way i also improved shell style and removed some tests from the
fast set, since they seem more advanced and just waste time on "fast"

Henning Schild (4):
  CI: move to avocado to 99.0
  CI: fix shell coding style
  CI: install qemu-system when qemu testing is requested
  testsuite: remove tests from "fast" set

 .gitlab-ci.yml      |  2 +-
 scripts/ci_build.sh | 26 ++++++++++++++++----------
 testsuite/citest.py | 10 +++++-----
 3 files changed, 22 insertions(+), 16 deletions(-)

-- 
2.38.2

[PATCH v2 1/4] CI: move to avocado to 99.0

[Henning Schild]

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 scripts/ci_build.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index fa7208c2b64b..84a3e472e784 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -21,7 +21,7 @@ if ! command -v avocado > /dev/null; then
     rm -rf /tmp/avocado_venv
     virtualenv --python python3 /tmp/avocado_venv
     source /tmp/avocado_venv/bin/activate
-    pip install avocado-framework==96.0
+    pip install avocado-framework==99.0
 fi
 
 # Get Avocado build tests path
-- 
2.38.2

[PATCH v2 2/4] CI: fix shell coding style

[Henning Schild]

Apply some suggestions from shellcheck and make sure that runs fine.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 scripts/ci_build.sh | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 84a3e472e784..e5f20d4f0a1f 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -20,6 +20,7 @@ if ! command -v avocado > /dev/null; then
     sudo apt-get install -y virtualenv
     rm -rf /tmp/avocado_venv
     virtualenv --python python3 /tmp/avocado_venv
+    # shellcheck source=/dev/null
     source /tmp/avocado_venv/bin/activate
     pip install avocado-framework==99.0
 fi
@@ -33,8 +34,8 @@ BASE_DIR=./build
 # Check dependencies
 DEPENDENCIES="umoci skopeo"
 for prog in ${DEPENDENCIES} ; do
-    if [ ! -x "$(which $prog)" ] ; then
-        echo "missing $prog in PATH" >&2
+    if ! command -v "${prog}" > /dev/null; then
+        echo "missing ${prog} in PATH" >&2
     fi
 done
 
@@ -132,16 +133,16 @@ fi
 mkdir -p .config/avocado
 cat <<EOF > .config/avocado/avocado.conf
 [datadir.paths]
-base_dir = $(realpath $BASE_DIR)/
-test_dir = $(realpath $BASE_DIR)/tests
-data_dir = $(realpath $BASE_DIR)/data
-logs_dir = $(realpath $BASE_DIR)/job-results
+base_dir = $(realpath "${BASE_DIR}")/
+test_dir = $(realpath "${BASE_DIR}")/tests
+data_dir = $(realpath "${BASE_DIR}")/data
+logs_dir = $(realpath "${BASE_DIR}")/job-results
 EOF
 export VIRTUAL_ENV="./"
 
 # the real stuff starts here, trace commands from now on
 set -x
 
-avocado $VERBOSE run "$TESTSUITE_DIR/citest.py" \
-    -t $TAGS --nrunner-max-parallel-tasks=1 --disable-sysinfo \
-    -p quiet=$QUIET -p cross=$CROSS_BUILD -p time_to_wait=$TIMEOUT
+avocado ${VERBOSE} run "${TESTSUITE_DIR}/citest.py" \
+    -t "${TAGS}" --nrunner-max-parallel-tasks=1 --disable-sysinfo \
+    -p quiet="${QUIET}" -p cross="${CROSS_BUILD}" -p time_to_wait="${TIMEOUT}"
-- 
2.38.2

[PATCH v2 3/4] CI: install qemu-system when qemu testing is requested

[Henning Schild]

And also change gitlab ci config to request qemu testing.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 .gitlab-ci.yml      | 2 +-
 scripts/ci_build.sh | 5 +++++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index fcdefa87bf66..dc8a01b9d1d6 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -27,7 +27,7 @@ fast-ci:
   except:
     - schedules
   script:
-    - scripts/ci_build.sh -q -f -n
+    - scripts/ci_build.sh -q -f
 
 full-ci:
   <<: *common-build
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index e5f20d4f0a1f..1704ec17af89 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -127,6 +127,11 @@ fi
 
 if [ -n "$NORUN" ]; then
     TAGS="$TAGS,-startvm"
+else
+    if ! command -v qemu-system-x86 > /dev/null; then
+        sudo apt-get update -qq
+        sudo apt-get install -y qemu-system
+    fi
 fi
 
 # Provide working path
-- 
2.38.2

[PATCH v2 4/4] testsuite: remove tests from "fast" set

[Henning Schild]

Ccache, Sdk, ContainerImage, ContainerSdk, Sstate

All of those are rather pedantic or test infrastructure bits or
corner-cases. They should not be done when we try to be "fast".

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 testsuite/citest.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 7aa1e6b5d12c..9315fe78e71d 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -53,7 +53,7 @@ class CcacheTest(CIBaseTest):
     """
     Test rebuild speed improve with ccache
 
-    :avocado: tags=ccache,fast,full
+    :avocado: tags=ccache,full
     """
     def test_ccache_rebuild(self):
         targets = ['mc:qemuamd64-bullseye:hello-isar']
@@ -118,7 +118,7 @@ class SdkTest(CIBaseTest):
     """
     In addition test SDK creation
 
-    :avocado: tags=sdk,fast,full
+    :avocado: tags=sdk,full
     """
     def test_sdk(self):
         targets = ['mc:qemuarm-bullseye:isar-image-base']
@@ -251,7 +251,7 @@ class ContainerImageTest(CIBaseTest):
     """
     Test containerized images creation
 
-    :avocado: tags=containerbuild,fast,full,container
+    :avocado: tags=containerbuild,full,container
     """
     @skipUnless(UMOCI_AVAILABLE and SKOPEO_AVAILABLE, 'umoci/skopeo not found')
     def test_container_image(self):
@@ -269,7 +269,7 @@ class ContainerSdkTest(CIBaseTest):
     """
     Test SDK container image creation
 
-    :avocado: tags=containersdk,fast,full,container
+    :avocado: tags=containersdk,full,container
     """
     @skipUnless(UMOCI_AVAILABLE and SKOPEO_AVAILABLE, 'umoci/skopeo not found')
     def test_container_sdk(self):
@@ -283,7 +283,7 @@ class SstateTest(CIBaseTest):
     """
     Test builds with artifacts taken from sstate cache
 
-    :avocado: tags=sstate,fast,full
+    :avocado: tags=sstate,full
     """
     def test_sstate(self):
         image_target = 'mc:qemuamd64-bullseye:isar-image-base'
-- 
2.38.2

Re: [PATCH v2 3/4] CI: install qemu-system when qemu testing is requested

[Uladzimir Bely]

In the email from Friday, 13 January 2023 02:59:28 +03 user Henning Schild 
wrote:
> And also change gitlab ci config to request qemu testing.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>  .gitlab-ci.yml      | 2 +-
>  scripts/ci_build.sh | 5 +++++
>  2 files changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index fcdefa87bf66..dc8a01b9d1d6 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -27,7 +27,7 @@ fast-ci:
>    except:
>      - schedules
>    script:
> -    - scripts/ci_build.sh -q -f -n
> +    - scripts/ci_build.sh -q -f
> 
>  full-ci:
>    <<: *common-build
> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> index e5f20d4f0a1f..1704ec17af89 100755
> --- a/scripts/ci_build.sh
> +++ b/scripts/ci_build.sh
> @@ -127,6 +127,11 @@ fi
> 
>  if [ -n "$NORUN" ]; then
>      TAGS="$TAGS,-startvm"
> +else
> +    if ! command -v qemu-system-x86 > /dev/null; then
> +        sudo apt-get update -qq
> +        sudo apt-get install -y qemu-system

This seems to be incorrect. In our CI chroot we have `qemu-system-x86_64` and 
`/usr/bin/qemu-system-i386` (among other archs), but not `-x86` one. So, the 
check always fails and apt-get is always called (with "qemu-system is already 
the newest version" message from apt).

> +    fi
>  fi
> 
>  # Provide working path

Re: [PATCH v2 3/4] CI: install qemu-system when qemu testing is requested

[Henning Schild]

Am Sat, 14 Jan 2023 23:15:54 +0300
schrieb Uladzimir Bely <ubely@ilbers.de>:

> In the email from Friday, 13 January 2023 02:59:28 +03 user Henning
> Schild wrote:
> > And also change gitlab ci config to request qemu testing.
> > 
> > Signed-off-by: Henning Schild <henning.schild@siemens.com>
> > ---
> >  .gitlab-ci.yml      | 2 +-
> >  scripts/ci_build.sh | 5 +++++
> >  2 files changed, 6 insertions(+), 1 deletion(-)
> > 
> > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > index fcdefa87bf66..dc8a01b9d1d6 100644
> > --- a/.gitlab-ci.yml
> > +++ b/.gitlab-ci.yml
> > @@ -27,7 +27,7 @@ fast-ci:
> >    except:
> >      - schedules
> >    script:
> > -    - scripts/ci_build.sh -q -f -n
> > +    - scripts/ci_build.sh -q -f
> > 
> >  full-ci:
> >    <<: *common-build
> > diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> > index e5f20d4f0a1f..1704ec17af89 100755
> > --- a/scripts/ci_build.sh
> > +++ b/scripts/ci_build.sh
> > @@ -127,6 +127,11 @@ fi
> > 
> >  if [ -n "$NORUN" ]; then
> >      TAGS="$TAGS,-startvm"
> > +else
> > +    if ! command -v qemu-system-x86 > /dev/null; then
> > +        sudo apt-get update -qq
> > +        sudo apt-get install -y qemu-system  
> 
> This seems to be incorrect. In our CI chroot we have
> `qemu-system-x86_64` and `/usr/bin/qemu-system-i386` (among other
> archs), but not `-x86` one. So, the check always fails and apt-get is
> always called (with "qemu-system is already the newest version"
> message from apt).

Right. I will switch this over to testing for "qemu-system-x86_64" but
still install "all qemus" with the meta package "qemu-system".

Or maybe "test -f /usr/share/doc/qemu-system/copyright" to check for
that meta-package.

Strictly speaking we do not need all of the qemus, but maintaining the
list of our arches seems too much effort.

Henning

> 
> > +    fi
> >  fi
> > 
> >  # Provide working path  
> 
> 
> 
> 

Re: [PATCH v2 3/4] CI: install qemu-system when qemu testing is requested

[Anton Mikanovich]

23/01/2023 10:51, Henning Schild wrote:
> Right. I will switch this over to testing for "qemu-system-x86_64" but
> still install "all qemus" with the meta package "qemu-system".
>
> Or maybe "test -f /usr/share/doc/qemu-system/copyright" to check for
> that meta-package.
>
> Strictly speaking we do not need all of the qemus, but maintaining the
> list of our arches seems too much effort.
>
> Henning

Hello, I've just send '[PATCH v2 00/15] Add developers test' with this 
commit
fixed inside. This is how I see test cases splitting between developers and
maintainers.