[PATCH 0/6] Unify build and run test cases

[PATCH 0/6] Unify build and run test cases

[Anton Mikanovich]

Make both build and start_vm CI tests run from one entry point to be
able to manage CI in more flexible way.
Also move QEMU boot logs into subdir to not flood ./build.

Anton Mikanovich (6):
  CI: Reorganize testsuite folders
  CI: Migrate all tests to one location
  CI: Adopt vm_start test logging
  CI: Do not run QEMU start tests from Gitlab
  CI: Rename build_test.py to citest.py
  CI: Change QEMU boot logs format

 .gitlab-ci.yml                                |   4 +-
 scripts/ci_build.sh                           |  12 +-
 scripts/vm_smoke_test                         |   4 +-
 testsuite/README.md                           |  26 +++-
 testsuite/build_test/run_fast.sh              |   3 -
 testsuite/build_test/run_full.sh              |   3 -
 testsuite/{build_test => }/cibase.py          |   4 +-
 testsuite/{build_test => }/cibuilder.py       |  80 ++++++++++-
 .../{build_test/build_test.py => citest.py}   |  46 ++++++
 testsuite/{ => keys}/base-apt/test_priv.key   |   0
 testsuite/{ => keys}/base-apt/test_pub.key    |   0
 testsuite/vm_boot_test/run_fast.sh            |   3 -
 testsuite/vm_boot_test/run_full.sh            |   3 -
 testsuite/vm_boot_test/vm_boot_test.py        | 133 ------------------
 14 files changed, 164 insertions(+), 157 deletions(-)
 delete mode 100755 testsuite/build_test/run_fast.sh
 delete mode 100755 testsuite/build_test/run_full.sh
 rename testsuite/{build_test => }/cibase.py (97%)
 mode change 100644 => 100755
 rename testsuite/{build_test => }/cibuilder.py (69%)
 mode change 100644 => 100755
 rename testsuite/{build_test/build_test.py => citest.py} (87%)
 mode change 100644 => 100755
 rename testsuite/{ => keys}/base-apt/test_priv.key (100%)
 rename testsuite/{ => keys}/base-apt/test_pub.key (100%)
 delete mode 100755 testsuite/vm_boot_test/run_fast.sh
 delete mode 100755 testsuite/vm_boot_test/run_full.sh
 delete mode 100644 testsuite/vm_boot_test/vm_boot_test.py

-- 
2.25.1

[PATCH 1/6] CI: Reorganize testsuite folders

[Anton Mikanovich]

Move all tests into one directory to be able to unify them.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 scripts/ci_build.sh                          |  4 +--
 scripts/vm_smoke_test                        |  4 +--
 testsuite/README.md                          | 26 +++++++++++++++++---
 testsuite/{build_test => }/build_test.py     |  0
 testsuite/build_test/run_fast.sh             |  3 ---
 testsuite/build_test/run_full.sh             |  3 ---
 testsuite/{build_test => }/cibase.py         |  4 +--
 testsuite/{build_test => }/cibuilder.py      |  2 +-
 testsuite/{ => keys}/base-apt/test_priv.key  |  0
 testsuite/{ => keys}/base-apt/test_pub.key   |  0
 testsuite/{vm_boot_test => }/vm_boot_test.py |  1 -
 testsuite/vm_boot_test/run_fast.sh           |  3 ---
 testsuite/vm_boot_test/run_full.sh           |  3 ---
 13 files changed, 30 insertions(+), 23 deletions(-)
 rename testsuite/{build_test => }/build_test.py (100%)
 mode change 100644 => 100755
 delete mode 100755 testsuite/build_test/run_fast.sh
 delete mode 100755 testsuite/build_test/run_full.sh
 rename testsuite/{build_test => }/cibase.py (97%)
 mode change 100644 => 100755
 rename testsuite/{build_test => }/cibuilder.py (99%)
 mode change 100644 => 100755
 rename testsuite/{ => keys}/base-apt/test_priv.key (100%)
 rename testsuite/{ => keys}/base-apt/test_pub.key (100%)
 rename testsuite/{vm_boot_test => }/vm_boot_test.py (98%)
 mode change 100644 => 100755
 delete mode 100755 testsuite/vm_boot_test/run_fast.sh
 delete mode 100755 testsuite/vm_boot_test/run_full.sh

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 339ebca..fe139d1 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -25,7 +25,7 @@ if ! command -v avocado > /dev/null; then
 fi
 
 # Get Avocado build tests path
-BUILD_TEST_DIR="$(pwd)/testsuite/build_test"
+TESTSUITE_DIR="$(pwd)/testsuite"
 
 # Start tests in current path by default
 BASE_DIR=./build
@@ -127,6 +127,6 @@ export VIRTUAL_ENV="./"
 # the real stuff starts here, trace commands from now on
 set -x
 
-avocado $VERBOSE run "$BUILD_TEST_DIR/build_test.py" \
+avocado $VERBOSE run "$TESTSUITE_DIR/build_test.py" \
     -t $TAGS --test-runner=runner --disable-sysinfo \
     -p quiet=$QUIET -p cross=$CROSS_BUILD
diff --git a/scripts/vm_smoke_test b/scripts/vm_smoke_test
index 6416250..50051a7 100755
--- a/scripts/vm_smoke_test
+++ b/scripts/vm_smoke_test
@@ -14,7 +14,7 @@ ES_BUG=3
 RET=$ES_FAIL
 
 # Get Avocado QEMU tests path
-VM_TEST_DIR="$(dirname "$0")/../testsuite/vm_boot_test"
+TESTSUITE_DIR="$(dirname "$0")/../testsuite"
 
 # Go to Isar root
 cd "$(dirname "$0")/.."
@@ -95,7 +95,7 @@ logs_dir = $BUILD_DIR/job-results
 EOF
 export VIRTUAL_ENV="./"
 
-if avocado $VERBOSE run "$VM_TEST_DIR/vm_boot_test.py" -t $TAGS \
+if avocado $VERBOSE run "$TESTSUITE_DIR/vm_boot_test.py" -t $TAGS \
     --test-runner=runner --disable-sysinfo \
     -p build_dir="$BUILD_DIR" -p time_to_wait=$TIMEOUT; then
     RET=$ES_OK
diff --git a/testsuite/README.md b/testsuite/README.md
index fcbea1e..b83a8f9 100644
--- a/testsuite/README.md
+++ b/testsuite/README.md
@@ -10,9 +10,29 @@ The framework could be installed by using standard HOWTO:
 
 # Run test
 
-Each testsuite directory contains:
- - run_*.sh - script to start tests
- - *.py - test case
+## Fast build test
+
+```
+$ avocado run build_test.py -t fast -p quiet=1 -p cross=1
+```
+
+## Full build test
+
+```
+$ avocado run build_test.py -t full -p quiet=1
+```
+
+## Fast boot test
+
+```
+$ avocado run vm_boot_test.py -t fast -p build_dir="$BUILDDIR" -p time_to_wait=300
+```
+
+## Full boot test
+
+```
+$ avocado run vm_boot_test.py -t full -p build_dir="$BUILDDIR" -p time_to_wait=300
+```
 
 # Other
 
diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test.py
old mode 100644
new mode 100755
similarity index 100%
rename from testsuite/build_test/build_test.py
rename to testsuite/build_test.py
diff --git a/testsuite/build_test/run_fast.sh b/testsuite/build_test/run_fast.sh
deleted file mode 100755
index 4d1bf44..0000000
--- a/testsuite/build_test/run_fast.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/bash
-
-avocado run build_test.py -t fast -p quiet=1 -p cross=1
diff --git a/testsuite/build_test/run_full.sh b/testsuite/build_test/run_full.sh
deleted file mode 100755
index af5ec59..0000000
--- a/testsuite/build_test/run_full.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/bash
-
-avocado run build_test.py -t full -p quiet=1
diff --git a/testsuite/build_test/cibase.py b/testsuite/cibase.py
old mode 100644
new mode 100755
similarity index 97%
rename from testsuite/build_test/cibase.py
rename to testsuite/cibase.py
index ae03aa7..35d3c65
--- a/testsuite/build_test/cibase.py
+++ b/testsuite/cibase.py
@@ -18,8 +18,8 @@ class CIBaseTest(CIBuilder):
         self.bitbake(targets, **kwargs)
 
     def perform_repro_test(self, targets, signed=False, **kwargs):
-        gpg_pub_key = os.path.dirname(__file__) + '/../base-apt/test_pub.key'
-        gpg_priv_key = os.path.dirname(__file__) + '/../base-apt/test_priv.key'
+        gpg_pub_key = os.path.dirname(__file__) + '/../keys/base-apt/test_pub.key'
+        gpg_priv_key = os.path.dirname(__file__) + '/../keys/base-apt/test_priv.key'
 
         self.configure(gpg_pub_key=gpg_pub_key if signed else None, **kwargs)
 
diff --git a/testsuite/build_test/cibuilder.py b/testsuite/cibuilder.py
old mode 100644
new mode 100755
similarity index 99%
rename from testsuite/build_test/cibuilder.py
rename to testsuite/cibuilder.py
index 2cfb6ec..f0c4bc3
--- a/testsuite/build_test/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -10,7 +10,7 @@ from avocado import Test
 from avocado.utils import path
 from avocado.utils import process
 
-isar_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))
+isar_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
 backup_prefix = '.ci-backup'
 
 app_log = logging.getLogger("avocado.app")
diff --git a/testsuite/base-apt/test_priv.key b/testsuite/keys/base-apt/test_priv.key
similarity index 100%
rename from testsuite/base-apt/test_priv.key
rename to testsuite/keys/base-apt/test_priv.key
diff --git a/testsuite/base-apt/test_pub.key b/testsuite/keys/base-apt/test_pub.key
similarity index 100%
rename from testsuite/base-apt/test_pub.key
rename to testsuite/keys/base-apt/test_pub.key
diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test.py
old mode 100644
new mode 100755
similarity index 98%
rename from testsuite/vm_boot_test/vm_boot_test.py
rename to testsuite/vm_boot_test.py
index f8c655c..6c4e979
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test.py
@@ -8,7 +8,6 @@ import time
 import tempfile
 
 from os.path import dirname
-sys.path.append(dirname(__file__) + '/..')
 
 import start_vm
 
diff --git a/testsuite/vm_boot_test/run_fast.sh b/testsuite/vm_boot_test/run_fast.sh
deleted file mode 100755
index 0fc77b3..0000000
--- a/testsuite/vm_boot_test/run_fast.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/bash
-
-avocado run vm_boot_test.py -t fast -p build_dir="$BUILDDIR" -p time_to_wait=300
diff --git a/testsuite/vm_boot_test/run_full.sh b/testsuite/vm_boot_test/run_full.sh
deleted file mode 100755
index a561a36..0000000
--- a/testsuite/vm_boot_test/run_full.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/bash
-
-avocado run vm_boot_test.py -t full -p build_dir="$BUILDDIR" -p time_to_wait=300
-- 
2.25.1

[PATCH 2/6] CI: Migrate all tests to one location

[Anton Mikanovich]

Move QEMU start tests to the same class as build tests to be able to
run everything from the single avocado command.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 scripts/vm_smoke_test     |   2 +-
 testsuite/build_test.py   |  46 +++++++++++++
 testsuite/cibuilder.py    |  68 ++++++++++++++++++++
 testsuite/vm_boot_test.py | 132 --------------------------------------
 4 files changed, 115 insertions(+), 133 deletions(-)
 delete mode 100755 testsuite/vm_boot_test.py

diff --git a/scripts/vm_smoke_test b/scripts/vm_smoke_test
index 50051a7..28c959a 100755
--- a/scripts/vm_smoke_test
+++ b/scripts/vm_smoke_test
@@ -95,7 +95,7 @@ logs_dir = $BUILD_DIR/job-results
 EOF
 export VIRTUAL_ENV="./"
 
-if avocado $VERBOSE run "$TESTSUITE_DIR/vm_boot_test.py" -t $TAGS \
+if avocado $VERBOSE run "$TESTSUITE_DIR/build_test.py" -t $TAGS,startvm \
     --test-runner=runner --disable-sysinfo \
     -p build_dir="$BUILD_DIR" -p time_to_wait=$TIMEOUT; then
     RET=$ES_OK
diff --git a/testsuite/build_test.py b/testsuite/build_test.py
index 988ae2f..2818a2d 100755
--- a/testsuite/build_test.py
+++ b/testsuite/build_test.py
@@ -244,3 +244,49 @@ class SstateTest(CIBaseTest):
 
         self.init('build-sstate')
         self.perform_sstate_test(image_target, package_target)
+
+class VmBootTestFast(CIBaseTest):
+
+    """
+    Test QEMU image start (fast)
+
+    :avocado: tags=startvm,fast,full
+    """
+    def test_arm_bullseye(self):
+        self.init()
+        self.vm_start('arm','bullseye')
+
+    def test_arm_buster(self):
+        self.init()
+        self.vm_start('arm','buster')
+
+    def test_arm64_bullseye(self):
+        self.init()
+        self.vm_start('arm64','bullseye')
+
+    def test_amd64_bullseye(self):
+        self.init()
+        self.vm_start('amd64','bullseye')
+
+class VmBootTestFull(CIBaseTest):
+
+    """
+    Test QEMU image start (full)
+
+    :avocado: tags=startvm,full
+    """
+    def test_i386_stretch(self):
+        self.init()
+        self.vm_start('i386','stretch')
+
+    def test_i386_buster(self):
+        self.init()
+        self.vm_start('i386','buster')
+
+    def test_amd64_buster(self):
+        self.init()
+        self.vm_start('amd64','buster')
+
+    def test_amd64_focal(self):
+        self.init()
+        self.vm_start('amd64','focal')
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index f0c4bc3..2170ea6 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -5,6 +5,10 @@ import os
 import select
 import shutil
 import subprocess
+import time
+import tempfile
+
+import start_vm
 
 from avocado import Test
 from avocado.utils import path
@@ -15,6 +19,9 @@ backup_prefix = '.ci-backup'
 
 app_log = logging.getLogger("avocado.app")
 
+class CanBeFinished(Exception):
+    pass
+
 class CIBuilder(Test):
     def setUp(self):
         super(CIBuilder, self).setUp()
@@ -181,3 +188,64 @@ class CIBuilder(Test):
                     for x in output.splitlines() if x != ''))
 
         return env['LAYERDIR_' + layer].strip('"')
+
+    def vm_start(self, arch='amd64', distro='buster'):
+        time_to_wait = self.params.get('time_to_wait', default=60)
+
+        self.log.info('===================================================')
+        self.log.info('Running Isar VM boot test for (' + distro + '-' + arch + ')')
+        self.log.info('Isar build folder is: ' + self.build_dir)
+        self.log.info('===================================================')
+
+        self.check_init()
+
+        fd, output_file = tempfile.mkstemp(suffix='_log.txt',
+                                           prefix='vm_start_' + distro + '_' +
+                                           arch + '_', dir=self.build_dir, text=True)
+        os.chmod(output_file, 0o644)
+
+        cmdline = start_vm.format_qemu_cmdline(arch, self.build_dir, distro,
+                                               output_file, None)
+        cmdline.insert(1, '-nographic')
+
+        self.log.info('QEMU boot line: ' + str(cmdline))
+
+        login_prompt = b'isar login:'
+        service_prompt = b'Just an example'
+
+        timeout = time.time() + int(time_to_wait)
+
+        p1 = subprocess.Popen(cmdline, stdout=subprocess.PIPE,
+                              stderr=subprocess.PIPE)
+        try:
+            poller = select.poll()
+            poller.register(p1.stdout, select.POLLIN)
+            poller.register(p1.stderr, select.POLLIN)
+            while time.time() < timeout and p1.poll() is None:
+                events = poller.poll(1000 * (timeout - time.time()))
+                for fd, event in events:
+                    if fd == p1.stdout.fileno():
+                        # Wait for the complete string if it is read in chunks
+                        # like "i", "sar", " login:"
+                        time.sleep(0.01)
+                        data = os.read(fd, 1024)
+                        if login_prompt in data:
+                            raise CanBeFinished
+                    if fd == p1.stderr.fileno():
+                        self.log.error(p1.stderr.readline())
+        except CanBeFinished:
+            self.log.debug('Got login prompt')
+        finally:
+            if p1.poll() is None:
+                p1.kill()
+            p1.wait()
+
+        if os.path.exists(output_file):
+            with open(output_file, "rb") as f1:
+                data = f1.read()
+                if service_prompt in data and login_prompt in data:
+                    return
+                else:
+                    self.log.error(data)
+
+        self.fail('Log ' + output_file)
diff --git a/testsuite/vm_boot_test.py b/testsuite/vm_boot_test.py
deleted file mode 100755
index 6c4e979..0000000
--- a/testsuite/vm_boot_test.py
+++ /dev/null
@@ -1,132 +0,0 @@
-#!/usr/bin/env python3
-
-import os
-import select
-import subprocess
-import sys
-import time
-import tempfile
-
-from os.path import dirname
-
-import start_vm
-
-from avocado import Test
-from avocado.utils import process
-from avocado.utils import path
-
-class CanBeFinished(Exception):
-    pass
-
-class VmBase(Test):
-
-    def check_bitbake(self, build_dir):
-        try:
-            path.find_command('bitbake')
-        except path.CmdNotFoundError:
-            out = process.getoutput('/bin/bash -c "cd ../.. && \
-                                    source isar-init-build-env \
-                                    %s 2>&1 >/dev/null; env"' % build_dir)
-            env = dict(((x.split('=', 1) + [''])[:2] \
-                        for x in output.splitlines() if x != ''))
-            os.environ.update(env)
-
-    def vm_start(self, arch='amd64', distro='buster'):
-        build_dir = self.params.get('build_dir', default='.')
-        time_to_wait = self.params.get('time_to_wait', default=60)
-
-        self.log.info('===================================================')
-        self.log.info('Running Isar VM boot test for (' + distro + '-' + arch + ')')
-        self.log.info('Isar build folder is: ' + build_dir)
-        self.log.info('===================================================')
-
-        self.check_bitbake(build_dir)
-
-        fd, output_file = tempfile.mkstemp(suffix='_log.txt',
-                                           prefix='vm_start_' + distro + '_' +
-                                           arch + '_', dir=build_dir, text=True)
-        os.chmod(output_file, 0o644)
-
-        cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro,
-                                               output_file, None)
-        cmdline.insert(1, '-nographic')
-
-        self.log.info('QEMU boot line: ' + str(cmdline))
-
-        login_prompt = b'isar login:'
-        service_prompt = b'Just an example'
-
-        timeout = time.time() + int(time_to_wait)
-
-        p1 = subprocess.Popen(cmdline, stdout=subprocess.PIPE,
-                              stderr=subprocess.PIPE)
-        try:
-            poller = select.poll()
-            poller.register(p1.stdout, select.POLLIN)
-            poller.register(p1.stderr, select.POLLIN)
-            while time.time() < timeout and p1.poll() is None:
-                events = poller.poll(1000 * (timeout - time.time()))
-                for fd, event in events:
-                    if fd == p1.stdout.fileno():
-                        # Wait for the complete string if it is read in chunks
-                        # like "i", "sar", " login:"
-                        time.sleep(0.01)
-                        data = os.read(fd, 1024)
-                        if login_prompt in data:
-                            raise CanBeFinished
-                    if fd == p1.stderr.fileno():
-                        self.log.error(p1.stderr.readline())
-        except CanBeFinished:
-            self.log.debug('Got login prompt')
-        finally:
-            if p1.poll() is None:
-                p1.kill()
-            p1.wait()
-
-        if os.path.exists(output_file):
-            with open(output_file, "rb") as f1:
-                data = f1.read()
-                if service_prompt in data and login_prompt in data:
-                    return
-                else:
-                    self.log.error(data)
-
-        self.fail('Log ' + output_file)
-
-class VmBootTestFast(VmBase):
-
-    """
-    Test QEMU image start (fast)
-
-    :avocado: tags=fast,full
-    """
-    def test_arm_bullseye(self):
-        self.vm_start('arm','bullseye')
-
-    def test_arm_buster(self):
-        self.vm_start('arm','buster')
-
-    def test_arm64_bullseye(self):
-        self.vm_start('arm64','bullseye')
-
-    def test_amd64_bullseye(self):
-        self.vm_start('amd64','bullseye')
-
-class VmBootTestFull(VmBase):
-
-    """
-    Test QEMU image start (full)
-
-    :avocado: tags=full
-    """
-    def test_i386_stretch(self):
-        self.vm_start('i386','stretch')
-
-    def test_i386_buster(self):
-        self.vm_start('i386','buster')
-
-    def test_amd64_buster(self):
-        self.vm_start('amd64','buster')
-
-    def test_amd64_focal(self):
-        self.vm_start('amd64','focal')
-- 
2.25.1

[PATCH 3/6] CI: Adopt vm_start test logging

[Anton Mikanovich]

Adopt stderr output format to made QEMU start tests output logs in the
same way like build tests are.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/cibuilder.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 2170ea6..52bb2a5 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -216,7 +216,7 @@ class CIBuilder(Test):
         timeout = time.time() + int(time_to_wait)
 
         p1 = subprocess.Popen(cmdline, stdout=subprocess.PIPE,
-                              stderr=subprocess.PIPE)
+                              stderr=subprocess.PIPE, universal_newlines=True)
         try:
             poller = select.poll()
             poller.register(p1.stdout, select.POLLIN)
@@ -224,6 +224,8 @@ class CIBuilder(Test):
             while time.time() < timeout and p1.poll() is None:
                 events = poller.poll(1000 * (timeout - time.time()))
                 for fd, event in events:
+                    if event != select.POLLIN:
+                        continue
                     if fd == p1.stdout.fileno():
                         # Wait for the complete string if it is read in chunks
                         # like "i", "sar", " login:"
@@ -232,7 +234,7 @@ class CIBuilder(Test):
                         if login_prompt in data:
                             raise CanBeFinished
                     if fd == p1.stderr.fileno():
-                        self.log.error(p1.stderr.readline())
+                        app_log.error(p1.stderr.readline().rstrip())
         except CanBeFinished:
             self.log.debug('Got login prompt')
         finally:
@@ -240,12 +242,12 @@ class CIBuilder(Test):
                 p1.kill()
             p1.wait()
 
-        if os.path.exists(output_file):
+        if os.path.exists(output_file) and os.path.getsize(output_file) > 0:
             with open(output_file, "rb") as f1:
                 data = f1.read()
                 if service_prompt in data and login_prompt in data:
                     return
                 else:
-                    self.log.error(data)
+                    app_log.error(data.decode(errors='replace'))
 
         self.fail('Log ' + output_file)
-- 
2.25.1

[PATCH 4/6] CI: Do not run QEMU start tests from Gitlab

[Anton Mikanovich]

To leave existing behaviour of vm_start tests execution introduce
'--norun' flag to use it in .gitlab-ci.yml scripts.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .gitlab-ci.yml      | 4 ++--
 scripts/ci_build.sh | 8 ++++++++
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 37882ef..d0488a0 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -27,7 +27,7 @@ fast-ci:
   except:
     - schedules
   script:
-    - scripts/ci_build.sh -q -f -d
+    - scripts/ci_build.sh -q -f -d -n
 
 full-ci:
   <<: *common-build
@@ -37,7 +37,7 @@ full-ci:
     - PREVIOUS_SHA="$(cat .CI_COMMIT_SHA || true)"
     - if [ "$CI_COMMIT_SHA" != "$PREVIOUS_SHA" ]; then
           echo "$CI_COMMIT_SHA" > .CI_COMMIT_SHA;
-          scripts/ci_build.sh -q -c -r -d;
+          scripts/ci_build.sh -q -c -r -d -n;
       fi
   cache:
     key: "$CI_COMMIT_REF_SLUG"
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index fe139d1..ea69553 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -52,6 +52,7 @@ show_help() {
     echo "    -f, --fast               cross build reduced set of configurations."
     echo "    -q, --quiet              suppress verbose bitbake output."
     echo "    -r, --repro              enable use of cached base repository."
+    echo "    -n, --norun              do not execute QEMU run tests."
     echo "    --help                   display this message and exit."
     echo
     echo "Exit status:"
@@ -100,6 +101,9 @@ do
         -s|--sign) shift ;;
         esac
         ;;
+    -n|--norun)
+        NORUN="1"
+        ;;
     *)
         echo "error: invalid parameter '$key', please try '--help' to get list of supported parameters"
         exit $ES_BUG
@@ -113,6 +117,10 @@ if [ -z "$REPRO_BUILD" ]; then
     TAGS="$TAGS,-repro"
 fi
 
+if [ -n "$NORUN" ]; then
+    TAGS="$TAGS,-startvm"
+fi
+
 # Provide working path
 mkdir -p .config/avocado
 cat <<EOF > .config/avocado/avocado.conf
-- 
2.25.1

[PATCH 5/6] CI: Rename build_test.py to citest.py

[Anton Mikanovich]

It is no more just build test after adding QEMU start testing.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 scripts/ci_build.sh                    | 2 +-
 scripts/vm_smoke_test                  | 2 +-
 testsuite/{build_test.py => citest.py} | 0
 3 files changed, 2 insertions(+), 2 deletions(-)
 rename testsuite/{build_test.py => citest.py} (100%)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index ea69553..41257cb 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -135,6 +135,6 @@ export VIRTUAL_ENV="./"
 # the real stuff starts here, trace commands from now on
 set -x
 
-avocado $VERBOSE run "$TESTSUITE_DIR/build_test.py" \
+avocado $VERBOSE run "$TESTSUITE_DIR/citest.py" \
     -t $TAGS --test-runner=runner --disable-sysinfo \
     -p quiet=$QUIET -p cross=$CROSS_BUILD
diff --git a/scripts/vm_smoke_test b/scripts/vm_smoke_test
index 28c959a..5c16d86 100755
--- a/scripts/vm_smoke_test
+++ b/scripts/vm_smoke_test
@@ -95,7 +95,7 @@ logs_dir = $BUILD_DIR/job-results
 EOF
 export VIRTUAL_ENV="./"
 
-if avocado $VERBOSE run "$TESTSUITE_DIR/build_test.py" -t $TAGS,startvm \
+if avocado $VERBOSE run "$TESTSUITE_DIR/citest.py" -t $TAGS,startvm \
     --test-runner=runner --disable-sysinfo \
     -p build_dir="$BUILD_DIR" -p time_to_wait=$TIMEOUT; then
     RET=$ES_OK
diff --git a/testsuite/build_test.py b/testsuite/citest.py
similarity index 100%
rename from testsuite/build_test.py
rename to testsuite/citest.py
-- 
2.25.1

[PATCH 6/6] CI: Change QEMU boot logs format

[Anton Mikanovich]

Move startvm boot logs into separate location with log datetime and
'latest' log symlinks.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/cibuilder.py | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 52bb2a5..069a1d9 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -199,10 +199,18 @@ class CIBuilder(Test):
 
         self.check_init()
 
-        fd, output_file = tempfile.mkstemp(suffix='_log.txt',
-                                           prefix='vm_start_' + distro + '_' +
-                                           arch + '_', dir=self.build_dir, text=True)
+        logdir = '%s/vm_start' % self.build_dir
+        if not os.path.exists(logdir):
+            os.mkdir(logdir)
+        prefix = '%s-vm_start_%s_%s_' % (time.strftime('%Y%m%d-%H%M%S'),
+                                         distro, arch)
+        fd, output_file = tempfile.mkstemp(suffix='_log.txt', prefix=prefix,
+                                           dir=logdir, text=True)
         os.chmod(output_file, 0o644)
+        latest_link = '%s/vm_start_%s_%s_latest.txt' % (logdir, distro, arch)
+        if os.path.exists(latest_link):
+            os.unlink(latest_link)
+        os.symlink(os.path.basename(output_file), latest_link)
 
         cmdline = start_vm.format_qemu_cmdline(arch, self.build_dir, distro,
                                                output_file, None)
-- 
2.25.1