[PATCH v8 00/17] Update Avocado testsuite

[PATCH v8 00/17] Update Avocado testsuite

[Anton Mikanovich]

This patchset provides implementation of build and vm_start tests
functionality in Python with help of Avocado framework.
Scripts ci_build and vm_smoke_test are rewritten to call Avocado.
ci_build and vm_smoke_test are left for compatibility and
should be replaced by direct avocado calls later.
Original ci_build and vm_smoke_test are renamed to *_legacy.
Build and run logs are hidden now until an error occurs.

---
Changes since v7:
- Remove code duplication.
- Fix build_dir parsing.
- Add missed protect files backup logic.
Changes since v6:
- Improve environment parsing.
- Protect files backup logic.
- Rebuild full log output option.
- Protect ubuntu target with KFAIL.
Changes since v5:
- Rebase on current next.
Changes since v4:
- Add container tests.
- Implement testcase skipping logic.
- Separate and protect Bullseye tests.
- Add checks for container dependencies.
- Add Ilbers repo key.
- Improve error handling.
- Fix Ubuntu qemu image name.
Changes since v3:
- Add automatic bitbake init.
- Fix line lengths.
- Fix build without repro.
- Fix build artifact paths.
- Prepare for Gitlab build.
- Change debug settings.
- Move log splitting to start_vm.
- Add deb package install.
Changes since v2:
- Make CI call Avocado tests.
- Wait less on vm_start test.
- Split Roman's vm_start commit.
- Put logs to build folder.
- Rearrange patches.
- Split build test to classes.
- Add test case tagging.
- Get rid of avocado-framework-plugin-varianter-yaml-to-mux.
- Get rid of python-subprocess32.
- Improve logging.
Changes since v1:
- Merge start_vm rebuild patches.
- Fix patch comments.

Anton Mikanovich (14):
  start_vm.py: Fix ubuntu image name
  vm_boot_test: Fix log file path in vm_boot_test
  vm_boot_test: Remove external varianter
  vm_boot_test: Improve QEMU images checking
  build_test: Refactoring build tests cases
  testsuite: Add Python generations for testsuite in gitignore
  testsuite: Fix test suite prepare guide
  gitlab-ci: Add Avocado build artifacts
  gitlab-ci: Add debug flag
  vm_boot_test: Add automatic bitbake init
  build_test: Protect ubuntu target with KFAIL
  ci_build: Migrate to Avocado
  vm_smoke_test: Migrate to Avocado
  ci_build: Install Avocado if needed

Roman Pletnev (3):
  start_vm.py: Fix target name handling
  start_vm.py: Add output and PID file vm_start.py options
  start_vm.py: Add MIPS support

 .gitlab-ci.yml                         |   5 +-
 scripts/ci_build.sh                    | 213 ++++----------------
 scripts/ci_build_legacy.sh             | 267 +++++++++++++++++++++++++
 scripts/vm_smoke_test                  | 110 ++--------
 scripts/vm_smoke_test_legacy           | 168 ++++++++++++++++
 testsuite/.gitignore                   |   1 +
 testsuite/README.md                    |  21 +-
 testsuite/build_test/build_test.py     | 217 ++++++++++++++++++--
 testsuite/build_test/cibase.py         | 110 ++++++++++
 testsuite/build_test/cibuilder.py      | 103 ++++++++++
 testsuite/build_test/run.sh            |   3 -
 testsuite/build_test/run_fast.sh       |   3 +
 testsuite/build_test/run_full.sh       |   3 +
 testsuite/build_test/variant.yaml      |  22 --
 testsuite/start_vm.py                  |  27 ++-
 testsuite/vm_boot_test/run.sh          |   3 -
 testsuite/vm_boot_test/run_fast.sh     |   3 +
 testsuite/vm_boot_test/run_full.sh     |   3 +
 testsuite/vm_boot_test/variant.yaml    |  22 --
 testsuite/vm_boot_test/vm_boot_test.py | 121 +++++++++--
 20 files changed, 1043 insertions(+), 382 deletions(-)
 create mode 100755 scripts/ci_build_legacy.sh
 create mode 100755 scripts/vm_smoke_test_legacy
 create mode 100644 testsuite/.gitignore
 create mode 100644 testsuite/build_test/cibase.py
 create mode 100644 testsuite/build_test/cibuilder.py
 delete mode 100755 testsuite/build_test/run.sh
 create mode 100755 testsuite/build_test/run_fast.sh
 create mode 100755 testsuite/build_test/run_full.sh
 delete mode 100644 testsuite/build_test/variant.yaml
 delete mode 100755 testsuite/vm_boot_test/run.sh
 create mode 100755 testsuite/vm_boot_test/run_fast.sh
 create mode 100755 testsuite/vm_boot_test/run_full.sh
 delete mode 100644 testsuite/vm_boot_test/variant.yaml

-- 
2.25.1

[PATCH v8 01/17] start_vm.py: Fix target name handling

[Anton Mikanovich]

From: Roman Pletnev <rpletnev@ilbers.de>

This patch fix bb_output search (issue with wrong arch name being
selected for qemu) which caused qemu-system-riscv64 to be used for all
the targets:

>ERROR: [Errno 2] No such file or directory: u'qemu-system-riscv64' (1.05 s)

Signed-off-by: Roman Pletnev <rpletnev@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/start_vm.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index a3e32ac..c10db67 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -17,7 +17,7 @@ def get_bitbake_env(arch, distro):
 def get_bitbake_var(output, var):
     ret = ''
     for line in output.splitlines():
-        if line.startswith(var):
+        if line.startswith(var + '='):
             ret = line.split('"')[1]
     return ret
 
-- 
2.25.1

[PATCH v8 02/17] start_vm.py: Add output and PID file vm_start.py options

[Anton Mikanovich]

From: Roman Pletnev <rpletnev@ilbers.de>

This patch adds options -o (output file) and -p (pid file).
Implement virtual machine log output both to the file and stdout.

Signed-off-by: Roman Pletnev <rpletnev@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/start_vm.py                  | 18 ++++++++++++++----
 testsuite/vm_boot_test/vm_boot_test.py |  5 ++---
 2 files changed, 16 insertions(+), 7 deletions(-)

diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index c10db67..3736e51 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -21,7 +21,7 @@ def get_bitbake_var(output, var):
             ret = line.split('"')[1]
     return ret
 
-def format_qemu_cmdline(arch, build, distro):
+def format_qemu_cmdline(arch, build, distro, out, pid):
     bb_output = get_bitbake_env(arch, distro).decode()
 
     rootfs_image = ''
@@ -57,6 +57,13 @@ def format_qemu_cmdline(arch, build, distro):
     qemu_cpu = get_bitbake_var(bb_output, 'QEMU_CPU')
     qemu_disk_args = get_bitbake_var(bb_output, 'QEMU_DISK_ARGS')
 
+    if out:
+        extra_args.extend(['-chardev','stdio,id=ch0,logfile=' + out])
+        extra_args.extend(['-serial','chardev:ch0'])
+        extra_args.extend(['-monitor','none'])
+    if pid:
+        extra_args.extend(['-pidfile', pid])
+
     qemu_disk_args = qemu_disk_args.replace('##ROOTFS_IMAGE##', deploy_dir_image + '/' + rootfs_image).split()
 
     cmd = ['qemu-system-' + qemu_arch, '-m', '1024M']
@@ -72,10 +79,11 @@ def format_qemu_cmdline(arch, build, distro):
 
     return cmd
 
-def start_qemu(arch, build, distro):
-    cmdline = format_qemu_cmdline(arch, build, distro)
+def start_qemu(arch, build, distro, out, pid):
+    cmdline = format_qemu_cmdline(arch, build, distro, out, pid)
     cmdline.insert(1, '-nographic')
 
+    print(cmdline)
     p1 = subprocess.call(cmdline)
 
 if __name__ == "__main__":
@@ -83,6 +91,8 @@ if __name__ == "__main__":
     parser.add_argument('-a', '--arch', choices=['arm', 'arm64', 'amd64', 'i386'], help='set isar machine architecture.', default='arm')
     parser.add_argument('-b', '--build', help='set path to build directory.', default=os.getcwd())
     parser.add_argument('-d', '--distro', choices=['jessie', 'stretch'], help='set isar Debian distribution.', default='stretch')
+    parser.add_argument('-o', '--out', help='Route QEMU console output to specified file.')
+    parser.add_argument('-p', '--pid', help='Store QEMU pid to specified file.')
     args = parser.parse_args()
 
-    start_qemu(args.arch, args.build, args.distro)
+    start_qemu(args.arch, args.build, args.distro, args.out, args.pid)
diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index d4849c7..96e02f7 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -30,10 +30,9 @@ class VmBootTest(Test):
         if os.path.exists(output_file):
             os.remove(output_file)
 
-        cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro)
+        cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro,
+                                               output_file, None)
         cmdline.insert(1, '-nographic')
-        cmdline.append('-serial')
-        cmdline.append('file:' + output_file)
 
         self.log.info('QEMU boot line: ' + str(cmdline))
 
-- 
2.25.1

[PATCH v8 03/17] start_vm.py: Add MIPS support

[Anton Mikanovich]

From: Roman Pletnev <rpletnev@ilbers.de>

This patch adds mipsel to the list of supported architectures.

Signed-off-by: Roman Pletnev <rpletnev@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/start_vm.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index 3736e51..face747 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -88,7 +88,7 @@ def start_qemu(arch, build, distro, out, pid):
 
 if __name__ == "__main__":
     parser = argparse.ArgumentParser()
-    parser.add_argument('-a', '--arch', choices=['arm', 'arm64', 'amd64', 'i386'], help='set isar machine architecture.', default='arm')
+    parser.add_argument('-a', '--arch', choices=['arm', 'arm64', 'amd64', 'i386', 'mipsel'], help='set isar machine architecture.', default='arm')
     parser.add_argument('-b', '--build', help='set path to build directory.', default=os.getcwd())
     parser.add_argument('-d', '--distro', choices=['jessie', 'stretch'], help='set isar Debian distribution.', default='stretch')
     parser.add_argument('-o', '--out', help='Route QEMU console output to specified file.')
-- 
2.25.1

[PATCH v8 04/17] start_vm.py: Fix ubuntu image name

[Anton Mikanovich]

Choose 'ubuntu' base name for 'focal' target, not 'debian'.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/start_vm.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index face747..8f0ccd6 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -30,8 +30,9 @@ def format_qemu_cmdline(arch, build, distro, out, pid):
 
     image_type = get_bitbake_var(bb_output, 'IMAGE_TYPE')
     deploy_dir_image = get_bitbake_var(bb_output, 'DEPLOY_DIR_IMAGE')
+    base = 'ubuntu' if distro in ['focal', 'bionic'] else 'debian'
     if image_type == 'ext4-img':
-        rootfs_image = 'isar-image-base-debian-' + distro + '-qemu' + arch + '.ext4.img'
+        rootfs_image = 'isar-image-base-' + base + '-' + distro + '-qemu' + arch + '.ext4.img'
         kernel_image = deploy_dir_image + '/' + get_bitbake_var(bb_output, 'KERNEL_IMAGE')
         initrd_image = get_bitbake_var(bb_output, 'INITRD_IMAGE')
 
@@ -47,7 +48,7 @@ def format_qemu_cmdline(arch, build, distro, out, pid):
         extra_args = ['-kernel', kernel_image, '-initrd', initrd_image]
         extra_args.extend(kargs)
     elif image_type == 'wic-img':
-        rootfs_image = 'isar-image-base-debian-' + distro + '-qemu' + arch + '.wic.img'
+        rootfs_image = 'isar-image-base-' + base + '-' + distro + '-qemu' + arch + '.wic.img'
         extra_args = ['-snapshot']
     else:
         raise ValueError('Invalid image type: ' + str(image_type))
-- 
2.25.1

[PATCH v8 05/17] vm_boot_test: Fix log file path in vm_boot_test

[Anton Mikanovich]

Make log filename random and store it in build folder.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/vm_boot_test/vm_boot_test.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index 96e02f7..64dfda6 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -4,6 +4,7 @@ import os
 import subprocess32
 import sys
 import time
+import tempfile
 
 from os.path import dirname
 sys.path.append(dirname(__file__) + '/..')
@@ -26,9 +27,9 @@ class VmBootTest(Test):
         self.log.info('Isar build folder is: ' + build_dir)
         self.log.info('===================================================')
 
-        output_file = '/tmp/vm_boot_test.log'
-        if os.path.exists(output_file):
-            os.remove(output_file)
+        fd, output_file = tempfile.mkstemp(suffix='_log.txt',
+                                           prefix='vm_start_' + distro + '_' +
+                                           arch + '_', dir=build_dir, text=True)
 
         cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro,
                                                output_file, None)
-- 
2.25.1

[PATCH v8 06/17] vm_boot_test: Remove external varianter

[Anton Mikanovich]

To get rid of avocado-framework-plugin-varianter-yaml-to-mux pip
dependency we should not use yaml-to-mux parameters.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/vm_boot_test/run.sh          |  3 --
 testsuite/vm_boot_test/run_fast.sh     |  3 ++
 testsuite/vm_boot_test/run_full.sh     |  3 ++
 testsuite/vm_boot_test/variant.yaml    | 22 -------------
 testsuite/vm_boot_test/vm_boot_test.py | 45 +++++++++++++++++++++++---
 5 files changed, 46 insertions(+), 30 deletions(-)
 delete mode 100755 testsuite/vm_boot_test/run.sh
 create mode 100755 testsuite/vm_boot_test/run_fast.sh
 create mode 100755 testsuite/vm_boot_test/run_full.sh
 delete mode 100644 testsuite/vm_boot_test/variant.yaml

diff --git a/testsuite/vm_boot_test/run.sh b/testsuite/vm_boot_test/run.sh
deleted file mode 100755
index 9fdda95..0000000
--- a/testsuite/vm_boot_test/run.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/bash
-
-avocado run vm_boot_test.py --mux-yaml test:variant.yaml --mux-inject build_dir:$BUILDDIR time_to_wait:300
diff --git a/testsuite/vm_boot_test/run_fast.sh b/testsuite/vm_boot_test/run_fast.sh
new file mode 100755
index 0000000..0fc77b3
--- /dev/null
+++ b/testsuite/vm_boot_test/run_fast.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+avocado run vm_boot_test.py -t fast -p build_dir="$BUILDDIR" -p time_to_wait=300
diff --git a/testsuite/vm_boot_test/run_full.sh b/testsuite/vm_boot_test/run_full.sh
new file mode 100755
index 0000000..a561a36
--- /dev/null
+++ b/testsuite/vm_boot_test/run_full.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+avocado run vm_boot_test.py -t full -p build_dir="$BUILDDIR" -p time_to_wait=300
diff --git a/testsuite/vm_boot_test/variant.yaml b/testsuite/vm_boot_test/variant.yaml
deleted file mode 100644
index 9ddc634..0000000
--- a/testsuite/vm_boot_test/variant.yaml
+++ /dev/null
@@ -1,22 +0,0 @@
-variants: !mux
-    stretch-amd64:
-        distro: "stretch"
-        arch:   "amd64"
-    stretch-i386:
-        distro: "stretch"
-        arch:   "i386"
-    stretch-arm:
-        distro: "stretch"
-        arch:   "arm"
-    stretch-arm64:
-        distro: "stretch"
-        arch:   "arm64"
-    buster-amd64:
-        distro: "buster"
-        arch:   "amd64"
-    buster-i386:
-        distro: "buster"
-        arch:   "i386"
-    buster-arm:
-        distro: "buster"
-        arch:   "arm"
diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index 64dfda6..eee14b2 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -13,13 +13,10 @@ import start_vm
 
 from avocado import Test
 
-class VmBootTest(Test):
+class VmBase(Test):
 
-    def test(self):
-        # TODO: add default values
+    def vm_start(self, arch='amd64', distro='buster'):
         build_dir = self.params.get('build_dir', default='.')
-        arch = self.params.get('arch', default='arm')
-        distro = self.params.get('distro', default='stretch')
         time_to_wait = self.params.get('time_to_wait', default=60)
 
         self.log.info('===================================================')
@@ -49,3 +46,41 @@ class VmBootTest(Test):
                 return
 
         self.fail('Test failed')
+
+class VmBootTestFast(VmBase):
+
+    """
+    Test QEMU image start (fast)
+
+    :avocado: tags=fast,full
+    """
+    def test_arm_stretch(self):
+        self.vm_start('arm','stretch')
+
+    def test_arm_buster(self):
+        self.vm_start('arm','buster')
+
+    def test_arm64_stretch(self):
+        self.vm_start('arm64','stretch')
+
+    def test_amd64_stretch(self):
+        self.vm_start('amd64','stretch')
+
+class VmBootTestFull(VmBase):
+
+    """
+    Test QEMU image start (full)
+
+    :avocado: tags=full
+    """
+    def test_i386_stretch(self):
+        self.vm_start('i386','stretch')
+
+    def test_i386_buster(self):
+        self.vm_start('i386','buster')
+
+    def test_amd64_buster(self):
+        self.vm_start('amd64','buster')
+
+    def test_amd64_focal(self):
+        self.vm_start('amd64','focal')
-- 
2.25.1

[PATCH v8 07/17] vm_boot_test: Improve QEMU images checking

[Anton Mikanovich]

Makes QEMU start test to analyze boot log in real-time. It helps test
cases to finish as soon as booting succeeds and do not wasting time on
waiting.
Get rid of python-subprocess32 backport package.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/vm_boot_test/vm_boot_test.py | 53 ++++++++++++++++++++------
 1 file changed, 42 insertions(+), 11 deletions(-)

diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index eee14b2..6e580d0 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -1,7 +1,8 @@
 #!/usr/bin/env python3
 
 import os
-import subprocess32
+import select
+import subprocess
 import sys
 import time
 import tempfile
@@ -13,6 +14,9 @@ import start_vm
 
 from avocado import Test
 
+class CanBeFinished(Exception):
+    pass
+
 class VmBase(Test):
 
     def vm_start(self, arch='amd64', distro='buster'):
@@ -34,18 +38,45 @@ class VmBase(Test):
 
         self.log.info('QEMU boot line: ' + str(cmdline))
 
-        devnull = open(os.devnull, 'w')
-
-        p1 = subprocess32.Popen(cmdline, stdout=devnull, stderr=devnull)
-        time.sleep(int(time_to_wait))
-        p1.kill()
-        p1.wait()
+        login_prompt = b'isar login:'
+        service_prompt = b'Just an example'
+
+        timeout = time.time() + int(time_to_wait)
+
+        p1 = subprocess.Popen(cmdline, stdout=subprocess.PIPE,
+                              stderr=subprocess.PIPE)
+        try:
+            poller = select.poll()
+            poller.register(p1.stdout, select.POLLIN)
+            poller.register(p1.stderr, select.POLLIN)
+            while time.time() < timeout and p1.poll() is None:
+                events = poller.poll(1000 * (timeout - time.time()))
+                for fd, event in events:
+                    if fd == p1.stdout.fileno():
+                        # Wait for the complete string if it is read in chunks
+                        # like "i", "sar", " login:"
+                        time.sleep(0.01)
+                        data = os.read(fd, 1024)
+                        if login_prompt in data:
+                            raise CanBeFinished
+                    if fd == p1.stderr.fileno():
+                        self.log.error(p1.stderr.readline())
+        except CanBeFinished:
+            self.log.debug('Got login prompt')
+        finally:
+            if p1.poll() is None:
+                p1.kill()
+            p1.wait()
 
         if os.path.exists(output_file):
-            if 'isar login:' in open(output_file).read():
-                return
-
-        self.fail('Test failed')
+            with open(output_file, "rb") as f1:
+                data = f1.read()
+                if service_prompt in data and login_prompt in data:
+                    return
+                else:
+                    self.log.error(data)
+
+        self.fail('Log ' + output_file)
 
 class VmBootTestFast(VmBase):
 
-- 
2.25.1

[PATCH v8 08/17] build_test: Refactoring build tests cases

[Anton Mikanovich]

Split build test into classes and separate cases. Implement test cases
tagging for external configuring. Also remove the usage of external
varianter plugin.
In this step, all ci_build test cases were copied as-is without
functional changes.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/build_test/build_test.py | 208 ++++++++++++++++++++++++++---
 testsuite/build_test/cibase.py     | 110 +++++++++++++++
 testsuite/build_test/cibuilder.py  | 103 ++++++++++++++
 testsuite/build_test/run.sh        |   3 -
 testsuite/build_test/run_fast.sh   |   3 +
 testsuite/build_test/run_full.sh   |   3 +
 testsuite/build_test/variant.yaml  |  22 ---
 7 files changed, 407 insertions(+), 45 deletions(-)
 create mode 100644 testsuite/build_test/cibase.py
 create mode 100644 testsuite/build_test/cibuilder.py
 delete mode 100755 testsuite/build_test/run.sh
 create mode 100755 testsuite/build_test/run_fast.sh
 create mode 100755 testsuite/build_test/run_full.sh
 delete mode 100644 testsuite/build_test/variant.yaml

diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
index 7a55c2f..9038ec5 100644
--- a/testsuite/build_test/build_test.py
+++ b/testsuite/build_test/build_test.py
@@ -1,29 +1,197 @@
 #!/usr/bin/env python3
 
 import os
-import subprocess32
-import sys
-from os.path import dirname
 
-from avocado import Test
+from avocado import skipUnless
+from avocado.utils import path
+from cibase import CIBaseTest
 
-class BuildTest(Test):
+UMOCI_AVAILABLE = True
+SKOPEO_AVAILABLE = True
+try:
+    path.find_command('umoci')
+except path.CmdNotFoundError:
+    UMOCI_AVAILABLE = False
+try:
+    path.find_command('skopeo')
+except path.CmdNotFoundError:
+    SKOPEO_AVAILABLE = False
 
-    def test(self):
-        # TODO: add default values
-        build_dir = self.params.get('build_dir', default='.')
-        arch = self.params.get('arch', default='arm')
-        distro = self.params.get('distro', default='stretch')
+class ReproTest(CIBaseTest):
 
-        self.log.info('===================================================')
-        self.log.info('Running Isar build test for (' + distro + '-' + arch + ')')
-        self.log.info('Isar build folder is: ' + build_dir)
-        self.log.info('===================================================')
+    """
+    Test cached base repository
 
-        #isar_root = dirname(__file__) + '/..'
-        os.chdir(build_dir)
-        cmdline = ['bitbake', 'mc:qemu' + arch + '-' + distro + ':isar-image-base']
-        p1 = subprocess32.run(cmdline)
+    :avocado: tags=repro,fast,full
+    """
+    def test_repro_signed(self):
+        targets = [
+            'mc:de0-nano-soc-buster:isar-image-base',
+            'mc:qemuarm64-stretch:isar-image-base'
+                  ]
 
-        if p1.returncode:
-            self.fail('Test failed')
+        self.perform_repro_test(targets, 1)
+
+    def test_repro_unsigned(self):
+        targets = [
+            'mc:qemuamd64-stretch:isar-image-base',
+            'mc:qemuarm-stretch:isar-image-base'
+                  ]
+
+        self.perform_repro_test(targets, 0)
+
+class CrossTest(CIBaseTest):
+
+    """
+    Start cross build for the defined set of configurations
+
+    :avocado: tags=cross,fast,full
+    """
+    def test_cross(self):
+        targets = [
+            'mc:qemuarm-stretch:isar-image-base',
+            'mc:qemuarm-buster:isar-image-base',
+            'mc:qemuarm64-stretch:isar-image-base',
+            'mc:qemuamd64-stretch:isar-image-base',
+            'mc:de0-nano-soc-buster:isar-image-base',
+            'mc:stm32mp15x-buster:isar-image-base',
+            'mc:rpi-stretch:isar-image-base',
+            'mc:qemuarm64-focal:isar-image-base'
+                  ]
+
+        self.perform_build_test(targets, 1, None)
+
+    def test_cross_bullseye(self):
+        targets = [
+            'mc:qemuarm-bullseye:isar-image-base'
+                  ]
+
+        try:
+            self.perform_build_test(targets, 1, None)
+        except:
+            self.cancel('KFAIL')
+
+class SdkTest(CIBaseTest):
+
+    """
+    In addition test SDK creation
+
+    :avocado: tags=sdk,fast,full
+    """
+    def test_sdk(self):
+        targets = ['mc:qemuarm-stretch:isar-image-base']
+
+        self.perform_build_test(targets, 1, 'do_populate_sdk')
+
+class NoCrossTest(CIBaseTest):
+
+    """
+    Start non-cross build for the defined set of configurations
+
+    :avocado: tags=nocross,full
+    """
+    def test_nocross(self):
+        targets = [
+            'mc:qemuarm-stretch:isar-image-base',
+            'mc:qemuarm-buster:isar-image-base',
+            'mc:qemuarm64-stretch:isar-image-base',
+            'mc:qemui386-stretch:isar-image-base',
+            'mc:qemui386-buster:isar-image-base',
+            'mc:qemuamd64-stretch:isar-image-base',
+            'mc:qemuamd64-buster:isar-image-base',
+            'mc:qemuamd64-buster-tgz:isar-image-base',
+            'mc:qemuamd64-buster:isar-initramfs',
+            'mc:qemumipsel-stretch:isar-image-base',
+            'mc:qemumipsel-buster:isar-image-base',
+            'mc:nand-ubi-demo-buster:isar-image-ubi',
+            'mc:rpi-stretch:isar-image-base',
+            'mc:qemuamd64-focal:isar-image-base'
+                  ]
+
+        # Cleanup after cross build
+        self.deletetmp(self.params.get('build_dir',
+                       default=os.path.dirname(__file__) + '/../../build'))
+
+        self.perform_build_test(targets, 0, None)
+
+    def test_nocross_bullseye(self):
+        targets = [
+            'mc:qemuamd64-bullseye:isar-image-base',
+            'mc:qemuarm-bullseye:isar-image-base',
+            'mc:qemui386-bullseye:isar-image-base',
+            'mc:qemumipsel-bullseye:isar-image-base'
+                  ]
+
+        try:
+            self.perform_build_test(targets, 0, None)
+        except:
+            self.cancel('KFAIL')
+
+class RebuildTest(CIBaseTest):
+
+    """
+    Test image rebuild
+
+    :avocado: tags=rebuild,fast,full
+    """
+    def test_rebuild(self):
+        is_cross_build = int(self.params.get('cross', default=0))
+
+        layerdir_core = self.getlayerdir('core')
+
+        dpkgbase_file = layerdir_core + '/classes/dpkg-base.bbclass'
+
+        self.backupfile(dpkgbase_file)
+        with open(dpkgbase_file, 'a') as file:
+            file.write('do_fetch_append() {\n\n}')
+
+        try:
+            self.perform_build_test('mc:qemuamd64-stretch:isar-image-base',
+                                    is_cross_build, None)
+        finally:
+            self.restorefile(dpkgbase_file)
+
+class WicTest(CIBaseTest):
+
+    """
+    Test wic --exclude-path
+
+    :avocado: tags=wic,fast,full
+    """
+    def test_wic_exclude(self):
+        # TODO: remove hardcoded filenames
+        wks_path = '/scripts/lib/wic/canned-wks/sdimage-efi.wks'
+        wic_path = '/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img'
+
+        self.perform_wic_test('mc:qemuamd64-stretch:isar-image-base',
+                              wks_path, wic_path)
+
+class ContainerImageTest(CIBaseTest):
+
+    """
+    Test containerized images creation
+
+    :avocado: tags=containerbuild,fast,full,container
+    """
+    @skipUnless(UMOCI_AVAILABLE and SKOPEO_AVAILABLE, 'umoci/skopeo not found')
+    def test_nocross(self):
+        targets = [
+            'mc:container-amd64-stretch:isar-image-base',
+            'mc:container-amd64-buster:isar-image-base',
+            'mc:container-amd64-bullseye:isar-image-base'
+                  ]
+
+        self.perform_container_test(targets, None)
+
+class ContainerSdkTest(CIBaseTest):
+
+    """
+    Test SDK container image creation
+
+    :avocado: tags=containersdk,fast,full,container
+    """
+    @skipUnless(UMOCI_AVAILABLE and SKOPEO_AVAILABLE, 'umoci/skopeo not found')
+    def test_container_sdk(self):
+        targets = ['mc:container-amd64-stretch:isar-image-base']
+
+        self.perform_container_test(targets, 'do_populate_sdk')
diff --git a/testsuite/build_test/cibase.py b/testsuite/build_test/cibase.py
new file mode 100644
index 0000000..1a89f43
--- /dev/null
+++ b/testsuite/build_test/cibase.py
@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+
+import os
+import re
+import tempfile
+
+from cibuilder import CIBuilder
+from avocado.utils import process
+
+isar_root = os.path.dirname(__file__) + '/../..'
+
+class CIBaseTest(CIBuilder):
+
+    def prep(self, testname, targets, cross, debsrc_cache):
+        build_dir = self.params.get('build_dir', default=isar_root + '/build')
+        build_dir = os.path.realpath(build_dir)
+        quiet = int(self.params.get('quiet', default=0))
+        bitbake_args = '-v'
+
+        if quiet:
+            bitbake_args = ''
+
+        self.log.info('===================================================')
+        self.log.info('Running ' + testname + ' test for:')
+        self.log.info(targets)
+        self.log.info('Isar build folder is: ' + build_dir)
+        self.log.info('===================================================')
+
+        self.init(build_dir)
+        self.confprepare(build_dir, 1, cross, debsrc_cache)
+
+        return build_dir, bitbake_args;
+
+    def perform_build_test(self, targets, cross, bitbake_cmd):
+        build_dir, bb_args = self.prep('Isar build', targets, cross, 1)
+
+        self.log.info('Starting build...')
+
+        self.bitbake(build_dir, targets, bitbake_cmd, bb_args)
+
+    def perform_repro_test(self, targets, signed):
+        cross = int(self.params.get('cross', default=0))
+        build_dir, bb_args = self.prep('repro Isar build', targets, cross, 0)
+
+        gpg_pub_key = os.path.dirname(__file__) + '/../base-apt/test_pub.key'
+        gpg_priv_key = os.path.dirname(__file__) + '/../base-apt/test_priv.key'
+
+        if signed:
+            with open(build_dir + '/conf/ci_build.conf', 'a') as file:
+                # Enable use of signed cached base repository
+                file.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
+
+        os.chdir(build_dir)
+
+        os.environ['GNUPGHOME'] = tempfile.mkdtemp()
+        result = process.run('gpg --import %s %s' % (gpg_pub_key, gpg_priv_key))
+
+        if result.exit_status:
+            self.fail('GPG import failed')
+
+        self.bitbake(build_dir, targets, None, bb_args)
+
+        self.deletetmp(build_dir)
+        with open(build_dir + '/conf/ci_build.conf', 'a') as file:
+            file.write('ISAR_USE_CACHED_BASE_REPO = "1"\n')
+            file.write('BB_NO_NETWORK = "1"\n')
+
+        self.bitbake(build_dir, targets, None, bb_args)
+
+        # Cleanup and disable use of signed cached base repository
+        self.deletetmp(build_dir)
+        self.confcleanup(build_dir)
+
+    def perform_wic_test(self, targets, wks_path, wic_path):
+        cross = int(self.params.get('cross', default=0))
+        build_dir, bb_args = self.prep('WIC exclude build', targets, cross, 1)
+
+        layerdir_isar = self.getlayerdir('isar')
+
+        wks_file = layerdir_isar + wks_path
+        wic_img = build_dir + wic_path
+
+        if not os.path.isfile(wic_img):
+            self.fail('No build started before: ' + wic_img + ' not exist')
+
+        self.backupfile(wks_file)
+        self.backupmove(wic_img)
+
+        with open(wks_file, 'r') as file:
+            lines = file.readlines()
+        with open(wks_file, 'w') as file:
+            for line in lines:
+                file.write(re.sub(r'part \/ ', 'part \/ --exclude-path usr ',
+                                  line))
+
+        try:
+            self.bitbake(build_dir, targets, None, bb_args)
+        finally:
+            self.restorefile(wks_file)
+
+        self.restorefile(wic_img)
+
+    def perform_container_test(self, targets, bitbake_cmd):
+        cross = int(self.params.get('cross', default=0))
+        build_dir, bb_args = self.prep('Isar Container', targets, cross, 1)
+
+        self.containerprep(build_dir)
+
+        self.bitbake(build_dir, targets, bitbake_cmd, bb_args)
+
diff --git a/testsuite/build_test/cibuilder.py b/testsuite/build_test/cibuilder.py
new file mode 100644
index 0000000..f5cc173
--- /dev/null
+++ b/testsuite/build_test/cibuilder.py
@@ -0,0 +1,103 @@
+#!/usr/bin/env python3
+
+import os
+import re
+import shutil
+
+from avocado import Test
+from avocado.utils import path
+from avocado.utils import process
+
+isar_root = os.path.dirname(__file__) + '/../..'
+backup_prefix = '.ci-backup'
+
+class CIBuilder(Test):
+
+    def init(self, build_dir):
+        os.chdir(isar_root)
+        path.usable_rw_dir(build_dir)
+        output = process.getoutput('/bin/bash -c "source isar-init-build-env \
+                                    %s 2>&1 >/dev/null; env"' % build_dir)
+        env = dict(((x.split('=', 1) + [''])[:2] \
+                    for x in output.splitlines() if x != ''))
+        os.environ.update(env)
+
+    def confprepare(self, build_dir, compat_arch, cross, debsrc_cache):
+        with open(build_dir + '/conf/ci_build.conf', 'w') as f:
+            if compat_arch:
+                f.write('ISAR_ENABLE_COMPAT_ARCH_amd64 = "1"\n')
+                f.write('ISAR_ENABLE_COMPAT_ARCH_arm64 = "1"\n')
+                f.write('ISAR_ENABLE_COMPAT_ARCH_debian-stretch_amd64 = "0"\n')
+                f.write('IMAGE_INSTALL += "kselftest"\n')
+            if cross:
+                f.write('ISAR_CROSS_COMPILE = "1"\n')
+            if debsrc_cache:
+                f.write('BASE_REPO_FEATURES = "cache-deb-src"\n')
+
+        with open(build_dir + '/conf/local.conf', 'r+') as f:
+            for line in f:
+                if 'include ci_build.conf' in line:
+                    break
+            else:
+                f.write('\ninclude ci_build.conf')
+
+    def containerprep(self, build_dir):
+        with open(build_dir + '/conf/ci_build.conf', 'a') as f:
+            f.write('SDK_FORMATS = "docker-archive"\n')
+            f.write('IMAGE_INSTALL_remove = "example-module-${KERNEL_NAME} enable-fsck"\n')
+
+    def confcleanup(self, build_dir):
+        open(build_dir + '/conf/ci_build.conf', 'w').close()
+
+    def deletetmp(self, build_dir):
+        process.run('rm -rf ' + build_dir + '/tmp', sudo=True)
+
+    def bitbake(self, build_dir, target, cmd, args):
+        os.chdir(build_dir)
+        cmdline = ['bitbake']
+        if args:
+            cmdline.append(args)
+        if cmd:
+            cmdline.append('-c')
+            cmdline.append(cmd)
+        if isinstance(target, list):
+            cmdline.extend(target)
+        else:
+            cmdline.append(target)
+
+        result = process.run(" ".join(cmdline), ignore_status=True,
+                             allow_output_check='stderr')
+        if result.exit_status:
+            self.fail(result.stderr)
+
+    def backupfile(self, path):
+        try:
+            shutil.copy2(path, path + backup_prefix)
+        except FileNotFoundError:
+            self.log.warn(path + ' not exist')
+
+    def backupmove(self, path):
+        try:
+            shutil.move(path, path + backup_prefix)
+        except FileNotFoundError:
+            self.log.warn(path + ' not exist')
+
+    def restorefile(self, path):
+        try:
+            shutil.move(path + backup_prefix, path)
+        except FileNotFoundError:
+            self.log.warn(path + backup_prefix + ' not exist')
+
+    def getlayerdir(self, layer):
+        try:
+            path.find_command('bitbake')
+        except path.CmdNotFoundError:
+            build_dir = self.params.get('build_dir',
+                                        default=isar_root + '/build')
+            self.init(build_dir)
+        output = process.getoutput('bitbake -e | grep "^LAYERDIR_.*="')
+        env = dict(((x.split('=', 1) + [''])[:2] \
+                    for x in output.splitlines() if x != ''))
+
+        return env['LAYERDIR_' + layer].strip('"')
+
diff --git a/testsuite/build_test/run.sh b/testsuite/build_test/run.sh
deleted file mode 100755
index a8ea9cc..0000000
--- a/testsuite/build_test/run.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/bash
-
-avocado run build_test.py --mux-yaml test:variant.yaml --mux-inject build_dir:$BUILDDIR
diff --git a/testsuite/build_test/run_fast.sh b/testsuite/build_test/run_fast.sh
new file mode 100755
index 0000000..4d1bf44
--- /dev/null
+++ b/testsuite/build_test/run_fast.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+avocado run build_test.py -t fast -p quiet=1 -p cross=1
diff --git a/testsuite/build_test/run_full.sh b/testsuite/build_test/run_full.sh
new file mode 100755
index 0000000..af5ec59
--- /dev/null
+++ b/testsuite/build_test/run_full.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+avocado run build_test.py -t full -p quiet=1
diff --git a/testsuite/build_test/variant.yaml b/testsuite/build_test/variant.yaml
deleted file mode 100644
index 9ddc634..0000000
--- a/testsuite/build_test/variant.yaml
+++ /dev/null
@@ -1,22 +0,0 @@
-variants: !mux
-    stretch-amd64:
-        distro: "stretch"
-        arch:   "amd64"
-    stretch-i386:
-        distro: "stretch"
-        arch:   "i386"
-    stretch-arm:
-        distro: "stretch"
-        arch:   "arm"
-    stretch-arm64:
-        distro: "stretch"
-        arch:   "arm64"
-    buster-amd64:
-        distro: "buster"
-        arch:   "amd64"
-    buster-i386:
-        distro: "buster"
-        arch:   "i386"
-    buster-arm:
-        distro: "buster"
-        arch:   "arm"
-- 
2.25.1

[PATCH v8 09/17] testsuite: Add Python generations for testsuite in gitignore

[Anton Mikanovich]

Ignore Python byte code auto generated files inside testsuite.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/.gitignore | 1 +
 1 file changed, 1 insertion(+)
 create mode 100644 testsuite/.gitignore

diff --git a/testsuite/.gitignore b/testsuite/.gitignore
new file mode 100644
index 0000000..0d20b64
--- /dev/null
+++ b/testsuite/.gitignore
@@ -0,0 +1 @@
+*.pyc
-- 
2.25.1

[PATCH v8 10/17] testsuite: Fix test suite prepare guide

[Anton Mikanovich]

Remove deprecated yaml-to-mux plugin and yml variants from the doc.
Also remove subprocess32 backport installation which is not used.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/README.md | 21 +++------------------
 1 file changed, 3 insertions(+), 18 deletions(-)

diff --git a/testsuite/README.md b/testsuite/README.md
index 5e64223..f91ea29 100644
--- a/testsuite/README.md
+++ b/testsuite/README.md
@@ -4,29 +4,14 @@ The framework could be installed by using standard HOWTO:
 
   https://github.com/avocado-framework/avocado#installing-with-standard-python-tools
 
-Then you need to install varianter yaml-to-mux plugin by following these instructions:
-
-  https://github.com/avocado-framework/avocado/tree/master/optional_plugins
-
-## For Debian 9.x
-
-        $ sudo apt-get install python-pip
-        $ pip install --user subprocess32
-        $ pip install --user avocado-framework
-        $ pip install --user avocado-framework-plugin-varianter-yaml-to-mux
-
-# Pre
-
-        $ export PATH=$PATH:~/.local/bin
-        $ cd isar
-        $ source isar-init-build-env
+## For Debian (tested on Debian 10.x)
 
+        $ sudo dpkg -i avocado_85.0_all.deb
 
 # Run test
 
 Each testsuite directory contains:
- - run.sh - script to start tests
- - variants.yaml - set of input data
+ - run_*.sh - script to start tests
  - *.py - test case
 
 # Other
-- 
2.25.1

[PATCH v8 11/17] gitlab-ci: Add Avocado build artifacts

[Anton Mikanovich]

Save build logs in case of Avocado CI build.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .gitlab-ci.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 7fc1612..56e98dc 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -14,6 +14,7 @@ variables:
     name: "logs-$CI_JOB_ID"
     paths:
       - build/tmp/work/*/*/*/temp
+      - build/job-results
     when: on_failure
     expire_in: 1 week
 
-- 
2.25.1

[PATCH v8 12/17] gitlab-ci: Add debug flag

[Anton Mikanovich]

Gitlab users expect full build log output witch was enabled by default.
Debug flag was ignored and never used, now it becomes functional.
To keed compatibility add '-d' flag to ci_build.sh call.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .gitlab-ci.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 56e98dc..ac5e0c4 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -23,7 +23,7 @@ fast-ci:
   except:
     - schedules
   script:
-    - scripts/ci_build.sh -q -f
+    - scripts/ci_build.sh -q -f -d
 
 full-ci:
   <<: *common-build
@@ -33,7 +33,7 @@ full-ci:
     - PREVIOUS_SHA="$(cat .CI_COMMIT_SHA || true)"
     - if [ "$CI_COMMIT_SHA" != "$PREVIOUS_SHA" ]; then
           echo "$CI_COMMIT_SHA" > .CI_COMMIT_SHA;
-          scripts/ci_build.sh -q -c -r;
+          scripts/ci_build.sh -q -c -r -d;
       fi
   cache:
     key: "$CI_COMMIT_REF_SLUG"
-- 
2.25.1

[PATCH v8 13/17] vm_boot_test: Add automatic bitbake init

[Anton Mikanovich]

Implement source of isar-init-build-env in case there is no bitbake
command found.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/vm_boot_test/vm_boot_test.py | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index 6e580d0..01623c5 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -13,12 +13,25 @@ sys.path.append(dirname(__file__) + '/..')
 import start_vm
 
 from avocado import Test
+from avocado.utils import process
+from avocado.utils import path
 
 class CanBeFinished(Exception):
     pass
 
 class VmBase(Test):
 
+    def check_bitbake(self, build_dir):
+        try:
+            path.find_command('bitbake')
+        except path.CmdNotFoundError:
+            out = process.getoutput('/bin/bash -c "cd ../.. && \
+                                    source isar-init-build-env \
+                                    %s 2>&1 >/dev/null; env"' % build_dir)
+            env = dict(((x.split('=', 1) + [''])[:2] \
+                        for x in output.splitlines() if x != ''))
+            os.environ.update(env)
+
     def vm_start(self, arch='amd64', distro='buster'):
         build_dir = self.params.get('build_dir', default='.')
         time_to_wait = self.params.get('time_to_wait', default=60)
@@ -28,6 +41,8 @@ class VmBase(Test):
         self.log.info('Isar build folder is: ' + build_dir)
         self.log.info('===================================================')
 
+        self.check_bitbake(build_dir)
+
         fd, output_file = tempfile.mkstemp(suffix='_log.txt',
                                            prefix='vm_start_' + distro + '_' +
                                            arch + '_', dir=build_dir, text=True)
-- 
2.25.1

[PATCH v8 14/17] build_test: Protect ubuntu target with KFAIL

[Anton Mikanovich]

We experience the same QEMU-related semop issue on qemuarm64-focal:

| ERROR: mc:qemuarm64-focal:kselftest-5.6.13-r0 do_install_builddeps: Execution of '/build/tmp/work/ubuntu-focal-arm64/kselftest/5.6.13-r0/temp/run.do_install_builddeps.33809' failed with exit code 1:
| /bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
| hostname: Temporary failure in name resolution
| semop(1): encountered an error: Function not implemented
| Error in the build process: exit status 1
| dpkg: error: cannot access archive 'kselftest-build-deps_5.6.13_arm64.deb': No such file or directory

So we need to protect with KFAIL not only bullseye targets, but Ubuntu
Focal targets also.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/build_test/build_test.py | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
index 9038ec5..63ea301 100644
--- a/testsuite/build_test/build_test.py
+++ b/testsuite/build_test/build_test.py
@@ -55,12 +55,21 @@ class CrossTest(CIBaseTest):
             'mc:qemuamd64-stretch:isar-image-base',
             'mc:de0-nano-soc-buster:isar-image-base',
             'mc:stm32mp15x-buster:isar-image-base',
-            'mc:rpi-stretch:isar-image-base',
-            'mc:qemuarm64-focal:isar-image-base'
+            'mc:rpi-stretch:isar-image-base'
                   ]
 
         self.perform_build_test(targets, 1, None)
 
+    def test_cross_ubuntu(self):
+        targets = [
+            'mc:qemuarm64-focal:isar-image-base'
+                  ]
+
+        try:
+            self.perform_build_test(targets, 1, None)
+        except:
+            self.cancel('KFAIL')
+
     def test_cross_bullseye(self):
         targets = [
             'mc:qemuarm-bullseye:isar-image-base'
-- 
2.25.1

[PATCH v8 15/17] ci_build: Migrate to Avocado

[Anton Mikanovich]

Call Avocado test cases instead of shell based.
Build artifacts are placed in build directory.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 scripts/ci_build.sh        | 198 ++++-----------------------
 scripts/ci_build_legacy.sh | 267 +++++++++++++++++++++++++++++++++++++
 2 files changed, 293 insertions(+), 172 deletions(-)
 create mode 100755 scripts/ci_build_legacy.sh

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 49d6aa8..a406dcf 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -14,6 +14,9 @@ export PATH=$PATH:/sbin
 # Go to Isar root
 cd "$(dirname "$0")/.."
 
+# Get Avocado build tests path
+BUILD_TEST_DIR="$(pwd)/testsuite/build_test"
+
 # Start build in Isar tree by default
 BUILD_DIR=./build
 
@@ -21,68 +24,10 @@ BUILD_DIR=./build
 DEPENDENCIES="umoci skopeo"
 for prog in ${DEPENDENCIES} ; do
     if [ ! -x "$(which $prog)" ] ; then
-        echo "missing $prog in PATH, exiting" >&2
-        exit 1
+        echo "missing $prog in PATH" >&2
     fi
 done
 
-BB_ARGS="-v"
-
-TARGETS_SET="\
-            mc:qemuarm-stretch:isar-image-base \
-            mc:qemuarm-buster:isar-image-base \
-            mc:qemuarm64-stretch:isar-image-base \
-            mc:qemui386-stretch:isar-image-base \
-            mc:qemui386-buster:isar-image-base \
-            mc:qemuamd64-stretch:isar-image-base \
-            mc:qemuamd64-buster:isar-image-base \
-            mc:qemuamd64-buster-tgz:isar-image-base \
-            mc:qemuamd64-buster:isar-initramfs \
-            mc:qemumipsel-stretch:isar-image-base \
-            mc:qemumipsel-buster:isar-image-base \
-            mc:nand-ubi-demo-buster:isar-image-ubi \
-            mc:rpi-stretch:isar-image-base \
-            mc:qemuamd64-focal:isar-image-base \
-            "
-          # qemu-user-static of <= buster too old to build that
-          # mc:qemuarm64-buster:isar-image-base
-          # mc:qemuarm64-bullseye:isar-image-base
-
-TARGETS_SET_BULLSEYE="\
-    mc:qemuamd64-bullseye:isar-image-base \
-    mc:qemuarm-bullseye:isar-image-base \
-    mc:qemui386-bullseye:isar-image-base \
-    mc:qemumipsel-bullseye:isar-image-base \
-"
-
-TARGETS_CONTAINERS="\
-    mc:container-amd64-stretch:isar-image-base \
-    mc:container-amd64-buster:isar-image-base \
-    mc:container-amd64-bullseye:isar-image-base \
-"
-
-CROSS_TARGETS_SET="\
-                  mc:qemuarm-stretch:isar-image-base \
-                  mc:qemuarm-buster:isar-image-base \
-                  mc:qemuarm64-stretch:isar-image-base \
-                  mc:qemuamd64-stretch:isar-image-base \
-                  mc:de0-nano-soc-buster:isar-image-base \
-                  mc:stm32mp15x-buster:isar-image-base \
-                  mc:rpi-stretch:isar-image-base \
-                  mc:qemuarm64-focal:isar-image-base"
-
-CROSS_TARGETS_SET_BULLSEYE="\
-    mc:qemuarm-bullseye:isar-image-base \
-"
-
-REPRO_TARGETS_SET_SIGNED="\
-            mc:de0-nano-soc-buster:isar-image-base \
-            mc:qemuarm64-stretch:isar-image-base"
-
-REPRO_TARGETS_SET="\
-            mc:qemuamd64-stretch:isar-image-base \
-            mc:qemuarm-buster:isar-image-base"
-
 show_help() {
     echo "This script builds the default Isar images."
     echo
@@ -104,6 +49,10 @@ show_help() {
     echo " 3  if invalid parameters are passed."
 }
 
+TAGS="full"
+CROSS_BUILD="0"
+QUIET="0"
+
 # Parse command line to get user configuration
 while [ $# -gt 0 ]
 do
@@ -122,16 +71,16 @@ do
         CROSS_BUILD="1"
         ;;
     -d|--debug)
-        BB_ARGS="$BB_ARGS -D"
+        VERBOSE="--show=test"
         ;;
     -f|--fast)
         # Start build for the reduced set of configurations
         # Enforce cross-compilation to speed up the build
-        FAST_BUILD="1"
+        TAGS="fast"
         CROSS_BUILD="1"
         ;;
     -q|--quiet)
-        BB_ARGS=""
+        QUIET="1"
         ;;
     -r|--repro)
         REPRO_BUILD="1"
@@ -150,118 +99,23 @@ do
     shift
 done
 
+if [ -z "$REPRO_BUILD" ]; then
+    TAGS="$TAGS,-repro"
+fi
+
 # the real stuff starts here, trace commands from now on
 set -x
 
-# Setup build folder for the current build
-if [ ! -d "$BUILD_DIR" ]; then
-        mkdir -p "$BUILD_DIR"
-fi
-source isar-init-build-env "$BUILD_DIR"
-
-cat >>conf/local.conf <<EOF
-ISAR_ENABLE_COMPAT_ARCH_amd64 = "1"
-ISAR_ENABLE_COMPAT_ARCH_arm64 = "1"
-ISAR_ENABLE_COMPAT_ARCH_debian-stretch_amd64 = "0"
-IMAGE_INSTALL += "kselftest"
+# Provide working path
+mkdir -p .config/avocado
+cat <<EOF > .config/avocado/avocado.conf
+[datadir.paths]
+base_dir = ./
+test_dir = ./tests
+data_dir = ./data
+logs_dir = ./job-results
 EOF
+export VIRTUAL_ENV="./"
 
-if [ -n "$CROSS_BUILD" ]; then
-    sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
-fi
-
-if [ -n "$REPRO_BUILD" ]; then
-    ISAR_TESTSUITE_GPG_PUB_KEY_FILE="$TESTSUITEDIR/base-apt/test_pub.key"
-    ISAR_TESTSUITE_GPG_PRIV_KEY_FILE="$TESTSUITEDIR/base-apt/test_priv.key"
-    export GNUPGHOME=$(mktemp -d)
-    gpg --import $ISAR_TESTSUITE_GPG_PUB_KEY_FILE $ISAR_TESTSUITE_GPG_PRIV_KEY_FILE
-
-    # Enable use of signed cached base repository
-    echo BASE_REPO_KEY=\"file://$ISAR_TESTSUITE_GPG_PUB_KEY_FILE\" >> conf/local.conf
-    bitbake $BB_ARGS $REPRO_TARGETS_SET_SIGNED
-    while [ -e bitbake.sock ]; do sleep 1; done
-    sudo rm -rf tmp
-    sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
-    sed -i -e 's/^#BB_NO_NETWORK/BB_NO_NETWORK/g' conf/local.conf
-    bitbake $BB_ARGS $REPRO_TARGETS_SET_SIGNED
-    while [ -e bitbake.sock ]; do sleep 1; done
-    # Cleanup and disable use of signed cached base repository
-    sudo rm -rf tmp
-    sed -i -e 's/ISAR_USE_CACHED_BASE_REPO ?= "1"/#ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
-    sed -i -e 's/^BB_NO_NETWORK/#BB_NO_NETWORK/g' conf/local.conf
-    sed -i -e 's/^BASE_REPO_KEY/#BASE_REPO_KEY/g' conf/local.conf
-
-    # Enable use of unsigned cached base repository
-    bitbake $BB_ARGS $REPRO_TARGETS_SET
-    while [ -e bitbake.sock ]; do sleep 1; done
-    sudo rm -rf tmp
-    sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
-    sed -i -e 's/^#BB_NO_NETWORK/BB_NO_NETWORK/g' conf/local.conf
-    bitbake $BB_ARGS $REPRO_TARGETS_SET
-    while [ -e bitbake.sock ]; do sleep 1; done
-    # Cleanup and disable use of unsigned cached base repository
-    sudo rm -rf tmp
-    sed -i -e 's/ISAR_USE_CACHED_BASE_REPO ?= "1"/#ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
-    sed -i -e 's/^BB_NO_NETWORK/#BB_NO_NETWORK/g' conf/local.conf
-fi
-
-sed -i -e 's/^#BASE_REPO_FEATURES ?= "cache-deb-src"/BASE_REPO_FEATURES ?= "cache-deb-src"/g' conf/local.conf
-# Start cross build for the defined set of configurations
-sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
-bitbake $BB_ARGS $CROSS_TARGETS_SET
-while [ -e bitbake.sock ]; do sleep 1; done
-if bitbake $BB_ARGS $CROSS_TARGETS_SET_BULLSEYE; then
-    echo "bullseye cross: PASSED"
-else
-    echo "bullseye cross: KFAIL"
-fi
-# In addition test SDK creation
-bitbake $BB_ARGS -c do_populate_sdk mc:qemuarm-stretch:isar-image-base
-while [ -e bitbake.sock ]; do sleep 1; done
-
-if [ -z "$FAST_BUILD" ]; then
-    # Cleanup and disable cross build
-    sudo rm -rf tmp
-    sed -i -e 's/ISAR_CROSS_COMPILE ?= "1"/ISAR_CROSS_COMPILE ?= "0"/g' conf/local.conf
-    bitbake $BB_ARGS $TARGETS_SET
-    while [ -e bitbake.sock ]; do sleep 1; done
-
-    if bitbake $BB_ARGS $TARGETS_SET_BULLSEYE; then
-        echo "bullseye: PASSED"
-    else
-	echo "bullseye: KFAIL"
-    fi
-    while [ -e bitbake.sock ]; do sleep 1; done
-fi
-
-eval $(bitbake -e | grep "^LAYERDIR_core=")
-eval $(bitbake -e | grep "^LAYERDIR_isar=")
-cp -a "${LAYERDIR_core}/classes/dpkg-base.bbclass" "${LAYERDIR_core}/classes/dpkg-base.bbclass.ci-backup"
-echo -e "do_fetch_append() {\n\n}" >> "${LAYERDIR_core}/classes/dpkg-base.bbclass"
-
-bitbake $BB_ARGS mc:qemuamd64-stretch:isar-image-base
-
-mv "${LAYERDIR_core}/classes/dpkg-base.bbclass.ci-backup" "${LAYERDIR_core}/classes/dpkg-base.bbclass"
-
-# Test wic --exclude-path
-cp -a "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks" "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks.ci-backup"
-mv ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img \
-    ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img.ci-backup
-sed -i -e 's/part \/ /part \/ --exclude-path usr /g' "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks"
-
-bitbake $BB_ARGS mc:qemuamd64-stretch:isar-image-base
-
-mv "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks.ci-backup" "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks"
-mv ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img.ci-backup \
-    ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img
-
-# Finalize with containerized images, since they remove some not-needed packages from the local.conf
-sed -i -e 's/\(IMAGE_INSTALL = .*\) example-module-${KERNEL_NAME}\(.*\)/\1\2/g' conf/local.conf
-sed -i -e 's/\(IMAGE_INSTALL = .*\) enable-fsck\(.*\)/\1\2/g' conf/local.conf
-bitbake $BB_ARGS $TARGETS_CONTAINERS
-while [ -e bitbake.sock ]; do sleep 1; done
-# and SDK container image creation
-echo 'SDK_FORMATS = "docker-archive"' >> conf/local.conf
-bitbake $BB_ARGS -c do_populate_sdk mc:container-amd64-stretch:isar-image-base
-while [ -e bitbake.sock ]; do sleep 1; done
-
+avocado $VERBOSE run "$BUILD_TEST_DIR/build_test.py" -t $TAGS \
+    -p build_dir="$BUILD_DIR" -p quiet=$QUIET -p cross=$CROSS_BUILD
diff --git a/scripts/ci_build_legacy.sh b/scripts/ci_build_legacy.sh
new file mode 100755
index 0000000..49d6aa8
--- /dev/null
+++ b/scripts/ci_build_legacy.sh
@@ -0,0 +1,267 @@
+#!/usr/bin/env bash
+# Script for CI system build
+#
+# Alexander Smirnov <asmirnov@ilbers.de>
+# Copyright (c) 2016-2018 ilbers GmbH
+
+set -e
+
+ES_BUG=3
+
+# Export $PATH to use 'parted' tool
+export PATH=$PATH:/sbin
+
+# Go to Isar root
+cd "$(dirname "$0")/.."
+
+# Start build in Isar tree by default
+BUILD_DIR=./build
+
+# Check dependencies
+DEPENDENCIES="umoci skopeo"
+for prog in ${DEPENDENCIES} ; do
+    if [ ! -x "$(which $prog)" ] ; then
+        echo "missing $prog in PATH, exiting" >&2
+        exit 1
+    fi
+done
+
+BB_ARGS="-v"
+
+TARGETS_SET="\
+            mc:qemuarm-stretch:isar-image-base \
+            mc:qemuarm-buster:isar-image-base \
+            mc:qemuarm64-stretch:isar-image-base \
+            mc:qemui386-stretch:isar-image-base \
+            mc:qemui386-buster:isar-image-base \
+            mc:qemuamd64-stretch:isar-image-base \
+            mc:qemuamd64-buster:isar-image-base \
+            mc:qemuamd64-buster-tgz:isar-image-base \
+            mc:qemuamd64-buster:isar-initramfs \
+            mc:qemumipsel-stretch:isar-image-base \
+            mc:qemumipsel-buster:isar-image-base \
+            mc:nand-ubi-demo-buster:isar-image-ubi \
+            mc:rpi-stretch:isar-image-base \
+            mc:qemuamd64-focal:isar-image-base \
+            "
+          # qemu-user-static of <= buster too old to build that
+          # mc:qemuarm64-buster:isar-image-base
+          # mc:qemuarm64-bullseye:isar-image-base
+
+TARGETS_SET_BULLSEYE="\
+    mc:qemuamd64-bullseye:isar-image-base \
+    mc:qemuarm-bullseye:isar-image-base \
+    mc:qemui386-bullseye:isar-image-base \
+    mc:qemumipsel-bullseye:isar-image-base \
+"
+
+TARGETS_CONTAINERS="\
+    mc:container-amd64-stretch:isar-image-base \
+    mc:container-amd64-buster:isar-image-base \
+    mc:container-amd64-bullseye:isar-image-base \
+"
+
+CROSS_TARGETS_SET="\
+                  mc:qemuarm-stretch:isar-image-base \
+                  mc:qemuarm-buster:isar-image-base \
+                  mc:qemuarm64-stretch:isar-image-base \
+                  mc:qemuamd64-stretch:isar-image-base \
+                  mc:de0-nano-soc-buster:isar-image-base \
+                  mc:stm32mp15x-buster:isar-image-base \
+                  mc:rpi-stretch:isar-image-base \
+                  mc:qemuarm64-focal:isar-image-base"
+
+CROSS_TARGETS_SET_BULLSEYE="\
+    mc:qemuarm-bullseye:isar-image-base \
+"
+
+REPRO_TARGETS_SET_SIGNED="\
+            mc:de0-nano-soc-buster:isar-image-base \
+            mc:qemuarm64-stretch:isar-image-base"
+
+REPRO_TARGETS_SET="\
+            mc:qemuamd64-stretch:isar-image-base \
+            mc:qemuarm-buster:isar-image-base"
+
+show_help() {
+    echo "This script builds the default Isar images."
+    echo
+    echo "Usage:"
+    echo "    $0 [params]"
+    echo
+    echo "Parameters:"
+    echo "    -b, --build BUILD_DIR    set path to build directory. If not set,"
+    echo "                             the build will be started in current path."
+    echo "    -c, --cross              enable cross-compilation."
+    echo "    -d, --debug              enable debug bitbake output."
+    echo "    -f, --fast               cross build reduced set of configurations."
+    echo "    -q, --quiet              suppress verbose bitbake output."
+    echo "    -r, --repro              enable use of cached base repository."
+    echo "    --help                   display this message and exit."
+    echo
+    echo "Exit status:"
+    echo " 0  if OK,"
+    echo " 3  if invalid parameters are passed."
+}
+
+# Parse command line to get user configuration
+while [ $# -gt 0 ]
+do
+    key="$1"
+
+    case $key in
+    -h|--help)
+        show_help
+        exit 0
+        ;;
+    -b|--build)
+        BUILD_DIR="$2"
+        shift
+        ;;
+    -c|--cross)
+        CROSS_BUILD="1"
+        ;;
+    -d|--debug)
+        BB_ARGS="$BB_ARGS -D"
+        ;;
+    -f|--fast)
+        # Start build for the reduced set of configurations
+        # Enforce cross-compilation to speed up the build
+        FAST_BUILD="1"
+        CROSS_BUILD="1"
+        ;;
+    -q|--quiet)
+        BB_ARGS=""
+        ;;
+    -r|--repro)
+        REPRO_BUILD="1"
+        # This switch is deprecated, just here to not cause failing CI on
+        # legacy configs
+        case "$2" in
+        -s|--sign) shift ;;
+        esac
+        ;;
+    *)
+        echo "error: invalid parameter '$key', please try '--help' to get list of supported parameters"
+        exit $ES_BUG
+        ;;
+    esac
+
+    shift
+done
+
+# the real stuff starts here, trace commands from now on
+set -x
+
+# Setup build folder for the current build
+if [ ! -d "$BUILD_DIR" ]; then
+        mkdir -p "$BUILD_DIR"
+fi
+source isar-init-build-env "$BUILD_DIR"
+
+cat >>conf/local.conf <<EOF
+ISAR_ENABLE_COMPAT_ARCH_amd64 = "1"
+ISAR_ENABLE_COMPAT_ARCH_arm64 = "1"
+ISAR_ENABLE_COMPAT_ARCH_debian-stretch_amd64 = "0"
+IMAGE_INSTALL += "kselftest"
+EOF
+
+if [ -n "$CROSS_BUILD" ]; then
+    sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
+fi
+
+if [ -n "$REPRO_BUILD" ]; then
+    ISAR_TESTSUITE_GPG_PUB_KEY_FILE="$TESTSUITEDIR/base-apt/test_pub.key"
+    ISAR_TESTSUITE_GPG_PRIV_KEY_FILE="$TESTSUITEDIR/base-apt/test_priv.key"
+    export GNUPGHOME=$(mktemp -d)
+    gpg --import $ISAR_TESTSUITE_GPG_PUB_KEY_FILE $ISAR_TESTSUITE_GPG_PRIV_KEY_FILE
+
+    # Enable use of signed cached base repository
+    echo BASE_REPO_KEY=\"file://$ISAR_TESTSUITE_GPG_PUB_KEY_FILE\" >> conf/local.conf
+    bitbake $BB_ARGS $REPRO_TARGETS_SET_SIGNED
+    while [ -e bitbake.sock ]; do sleep 1; done
+    sudo rm -rf tmp
+    sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
+    sed -i -e 's/^#BB_NO_NETWORK/BB_NO_NETWORK/g' conf/local.conf
+    bitbake $BB_ARGS $REPRO_TARGETS_SET_SIGNED
+    while [ -e bitbake.sock ]; do sleep 1; done
+    # Cleanup and disable use of signed cached base repository
+    sudo rm -rf tmp
+    sed -i -e 's/ISAR_USE_CACHED_BASE_REPO ?= "1"/#ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
+    sed -i -e 's/^BB_NO_NETWORK/#BB_NO_NETWORK/g' conf/local.conf
+    sed -i -e 's/^BASE_REPO_KEY/#BASE_REPO_KEY/g' conf/local.conf
+
+    # Enable use of unsigned cached base repository
+    bitbake $BB_ARGS $REPRO_TARGETS_SET
+    while [ -e bitbake.sock ]; do sleep 1; done
+    sudo rm -rf tmp
+    sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
+    sed -i -e 's/^#BB_NO_NETWORK/BB_NO_NETWORK/g' conf/local.conf
+    bitbake $BB_ARGS $REPRO_TARGETS_SET
+    while [ -e bitbake.sock ]; do sleep 1; done
+    # Cleanup and disable use of unsigned cached base repository
+    sudo rm -rf tmp
+    sed -i -e 's/ISAR_USE_CACHED_BASE_REPO ?= "1"/#ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
+    sed -i -e 's/^BB_NO_NETWORK/#BB_NO_NETWORK/g' conf/local.conf
+fi
+
+sed -i -e 's/^#BASE_REPO_FEATURES ?= "cache-deb-src"/BASE_REPO_FEATURES ?= "cache-deb-src"/g' conf/local.conf
+# Start cross build for the defined set of configurations
+sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
+bitbake $BB_ARGS $CROSS_TARGETS_SET
+while [ -e bitbake.sock ]; do sleep 1; done
+if bitbake $BB_ARGS $CROSS_TARGETS_SET_BULLSEYE; then
+    echo "bullseye cross: PASSED"
+else
+    echo "bullseye cross: KFAIL"
+fi
+# In addition test SDK creation
+bitbake $BB_ARGS -c do_populate_sdk mc:qemuarm-stretch:isar-image-base
+while [ -e bitbake.sock ]; do sleep 1; done
+
+if [ -z "$FAST_BUILD" ]; then
+    # Cleanup and disable cross build
+    sudo rm -rf tmp
+    sed -i -e 's/ISAR_CROSS_COMPILE ?= "1"/ISAR_CROSS_COMPILE ?= "0"/g' conf/local.conf
+    bitbake $BB_ARGS $TARGETS_SET
+    while [ -e bitbake.sock ]; do sleep 1; done
+
+    if bitbake $BB_ARGS $TARGETS_SET_BULLSEYE; then
+        echo "bullseye: PASSED"
+    else
+	echo "bullseye: KFAIL"
+    fi
+    while [ -e bitbake.sock ]; do sleep 1; done
+fi
+
+eval $(bitbake -e | grep "^LAYERDIR_core=")
+eval $(bitbake -e | grep "^LAYERDIR_isar=")
+cp -a "${LAYERDIR_core}/classes/dpkg-base.bbclass" "${LAYERDIR_core}/classes/dpkg-base.bbclass.ci-backup"
+echo -e "do_fetch_append() {\n\n}" >> "${LAYERDIR_core}/classes/dpkg-base.bbclass"
+
+bitbake $BB_ARGS mc:qemuamd64-stretch:isar-image-base
+
+mv "${LAYERDIR_core}/classes/dpkg-base.bbclass.ci-backup" "${LAYERDIR_core}/classes/dpkg-base.bbclass"
+
+# Test wic --exclude-path
+cp -a "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks" "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks.ci-backup"
+mv ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img \
+    ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img.ci-backup
+sed -i -e 's/part \/ /part \/ --exclude-path usr /g' "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks"
+
+bitbake $BB_ARGS mc:qemuamd64-stretch:isar-image-base
+
+mv "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks.ci-backup" "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks"
+mv ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img.ci-backup \
+    ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img
+
+# Finalize with containerized images, since they remove some not-needed packages from the local.conf
+sed -i -e 's/\(IMAGE_INSTALL = .*\) example-module-${KERNEL_NAME}\(.*\)/\1\2/g' conf/local.conf
+sed -i -e 's/\(IMAGE_INSTALL = .*\) enable-fsck\(.*\)/\1\2/g' conf/local.conf
+bitbake $BB_ARGS $TARGETS_CONTAINERS
+while [ -e bitbake.sock ]; do sleep 1; done
+# and SDK container image creation
+echo 'SDK_FORMATS = "docker-archive"' >> conf/local.conf
+bitbake $BB_ARGS -c do_populate_sdk mc:container-amd64-stretch:isar-image-base
+while [ -e bitbake.sock ]; do sleep 1; done
+
-- 
2.25.1

[PATCH v8 16/17] vm_smoke_test: Migrate to Avocado

[Anton Mikanovich]

Call Avocado test cases instead of shell based.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 scripts/vm_smoke_test        | 110 ++++-------------------
 scripts/vm_smoke_test_legacy | 168 +++++++++++++++++++++++++++++++++++
 2 files changed, 185 insertions(+), 93 deletions(-)
 create mode 100755 scripts/vm_smoke_test_legacy

diff --git a/scripts/vm_smoke_test b/scripts/vm_smoke_test
index ce08213..647b417 100755
--- a/scripts/vm_smoke_test
+++ b/scripts/vm_smoke_test
@@ -3,9 +3,7 @@
 # This software is a part of ISAR.
 # Copyright (C) 2015-2018 ilbers GmbH
 
-CONSOLE_OUTPUT=/tmp/isar_console
-PID_FILE=/tmp/qemu.pid
-VERBOSE=1
+VERBOSE="--show=test"
 TIMEOUT=300
 
 # Error codes:
@@ -13,75 +11,12 @@ ES_OK=0
 ES_FAIL=1
 ES_BUG=3
 
-RET=$ES_OK
+RET=$ES_FAIL
 
-dump_boot_log() {
-    echo "Boot log:\n8<--"
-    cat $CONSOLE_OUTPUT
-    echo "\n8<--"
-}
-
-check_login_prompt() {
-    echo -n "Check login prompt: "
-
-    str=$(grep "isar login: " $CONSOLE_OUTPUT)
-
-    if [ -n "$str" ]; then
-        echo "PASSED"
-    else
-        echo "FAIL"
-        RET=$ES_FAIL
-        FAIL=1
-    fi
-}
-
-check_example_module() {
-    echo -n "Check example module: "
-
-    str=$(grep "Just an example" $CONSOLE_OUTPUT)
-
-    if [ -n "$str" ]; then
-        echo "PASSED"
-    else
-        echo "FAIL"
-        RET=$ES_FAIL
-        FAIL=1
-    fi
-}
-
-run_test () {
-    ARCH=$1
-    DISTRO=$2
-
-    echo "-------------------------------------------------"
-    echo "Testing Isar [$DISTRO] image for [$ARCH] machine:"
+# Get Avocado QEMU tests path
+VM_TEST_DIR="$(dirname "$0")/../testsuite/vm_boot_test"
 
-    start_vm -a $ARCH -d $DISTRO -o $CONSOLE_OUTPUT -p $PID_FILE > /dev/null 2>&1 &
-    sleep 5
-
-    if [ -z `ps -p $! -o pid=` ]; then
-        echo "QEMU start: FAILED"
-        RET=$ES_FAIL
-        echo "Command output:\n8<--"
-        start_vm -a $ARCH -d $DISTRO -o $CONSOLE_OUTPUT -p $PID_FILE
-        echo "\n8<--"
-    else
-        sleep $TIMEOUT
-        kill `cat $PID_FILE`
-
-        FAIL=0
-
-        check_login_prompt
-
-        check_example_module
-
-        [ $VERBOSE -eq 1 -o $FAIL -eq 1 ] && dump_boot_log
-
-        rm $CONSOLE_OUTPUT
-    fi
-
-    rm $PID_FILE
-}
+BUILD_DIR=$PWD
 
 show_help() {
     echo "This script tests the Isar images for default targets in QEMU."
@@ -91,10 +26,6 @@ show_help() {
     echo
     echo "Parameters:"
     echo "    -f,--fast             test reduced set of supported targets."
-    echo "    -o,--output FILE      specify file to store console output."
-    echo "                          The default is: /tmp/isar_console"
-    echo "    -p,--pid-file FILE    specify file to store QEMU process PID."
-    echo "                          The default is: /tmp/qemu.pid"
     echo "    -q, --quiet           do not display boot logs for all the targets."
     echo "                          If test failed for the specific configuration,"
     echo "                          the respective boot log will be printed anyway."
@@ -119,18 +50,18 @@ do
         exit 0
         ;;
     -o|--output)
-        CONSOLE_OUTPUT=$2
+        # Deprecated option
         shift
         ;;
     -p|--pid-file)
-        PID_FILE=$2
+        # Deprecated option
         shift
         ;;
     -f|--fast)
         FAST_BUILD="1"
         ;;
     -q|--quiet)
-        VERBOSE=0
+        VERBOSE=""
         ;;
     -t|--timeout)
         TIMEOUT=$2
@@ -145,24 +76,17 @@ do
     shift
 done
 
-# ARM machine
-run_test arm stretch
-run_test arm buster
-
-# AMD64 machine
-if [ -z "$FAST_BUILD" ]; then
-    run_test amd64 buster
-    run_test amd64 focal
+TAGS="full"
+if [ -n "$FAST_BUILD" ]; then
+    TAGS="fast"
 fi
-run_test amd64 stretch
 
-# i386 machine
-if [ -z "$FAST_BUILD" ]; then
-    run_test i386 stretch
-    run_test i386 buster
-fi
+# Provide working path
+export VIRTUAL_ENV="./"
 
-# ARM64 machine
-run_test arm64 stretch
+if avocado $VERBOSE run "$VM_TEST_DIR/vm_boot_test.py" -t $TAGS \
+    -p build_dir="$BUILD_DIR" -p time_to_wait=$TIMEOUT; then
+    RET=$ES_OK
+fi
 
 exit $RET
diff --git a/scripts/vm_smoke_test_legacy b/scripts/vm_smoke_test_legacy
new file mode 100755
index 0000000..ce08213
--- /dev/null
+++ b/scripts/vm_smoke_test_legacy
@@ -0,0 +1,168 @@
+#!/bin/sh
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2018 ilbers GmbH
+
+CONSOLE_OUTPUT=/tmp/isar_console
+PID_FILE=/tmp/qemu.pid
+VERBOSE=1
+TIMEOUT=300
+
+# Error codes:
+ES_OK=0
+ES_FAIL=1
+ES_BUG=3
+
+RET=$ES_OK
+
+dump_boot_log() {
+    echo "Boot log:\n8<--"
+    cat $CONSOLE_OUTPUT
+    echo "\n8<--"
+}
+
+check_login_prompt() {
+    echo -n "Check login prompt: "
+
+    str=$(grep "isar login: " $CONSOLE_OUTPUT)
+
+    if [ -n "$str" ]; then
+        echo "PASSED"
+    else
+        echo "FAIL"
+        RET=$ES_FAIL
+        FAIL=1
+    fi
+}
+
+check_example_module() {
+    echo -n "Check example module: "
+
+    str=$(grep "Just an example" $CONSOLE_OUTPUT)
+
+    if [ -n "$str" ]; then
+        echo "PASSED"
+    else
+        echo "FAIL"
+        RET=$ES_FAIL
+        FAIL=1
+    fi
+}
+
+run_test () {
+    ARCH=$1
+    DISTRO=$2
+
+    echo "-------------------------------------------------"
+    echo "Testing Isar [$DISTRO] image for [$ARCH] machine:"
+
+    start_vm -a $ARCH -d $DISTRO -o $CONSOLE_OUTPUT -p $PID_FILE > /dev/null 2>&1 &
+    sleep 5
+
+    if [ -z `ps -p $! -o pid=` ]; then
+        echo "QEMU start: FAILED"
+        RET=$ES_FAIL
+        echo "Command output:\n8<--"
+        start_vm -a $ARCH -d $DISTRO -o $CONSOLE_OUTPUT -p $PID_FILE
+        echo "\n8<--"
+    else
+        sleep $TIMEOUT
+        kill `cat $PID_FILE`
+
+        FAIL=0
+
+        check_login_prompt
+
+        check_example_module
+
+        [ $VERBOSE -eq 1 -o $FAIL -eq 1 ] && dump_boot_log
+
+        rm $CONSOLE_OUTPUT
+    fi
+
+    rm $PID_FILE
+}
+
+show_help() {
+    echo "This script tests the Isar images for default targets in QEMU."
+    echo
+    echo "Usage:"
+    echo "    $0 [params]"
+    echo
+    echo "Parameters:"
+    echo "    -f,--fast             test reduced set of supported targets."
+    echo "    -o,--output FILE      specify file to store console output."
+    echo "                          The default is: /tmp/isar_console"
+    echo "    -p,--pid-file FILE    specify file to store QEMU process PID."
+    echo "                          The default is: /tmp/qemu.pid"
+    echo "    -q, --quiet           do not display boot logs for all the targets."
+    echo "                          If test failed for the specific configuration,"
+    echo "                          the respective boot log will be printed anyway."
+    echo "    -t,--timeout SEC      specify time in seconds to wait before stop QEMU."
+    echo "                          The default is: 300"
+    echo "    -h, --help            display this message and exit."
+    echo
+    echo "Exit status:"
+    echo " 0  if OK,"
+    echo " 1  if test failed,"
+    echo " 3  if invalid parameters are passed."
+}
+
+# Parse command line to get user configuration
+while [ $# -gt 0 ]
+do
+    key="$1"
+
+    case $key in
+    -h|--help)
+        show_help
+        exit 0
+        ;;
+    -o|--output)
+        CONSOLE_OUTPUT=$2
+        shift
+        ;;
+    -p|--pid-file)
+        PID_FILE=$2
+        shift
+        ;;
+    -f|--fast)
+        FAST_BUILD="1"
+        ;;
+    -q|--quiet)
+        VERBOSE=0
+        ;;
+    -t|--timeout)
+        TIMEOUT=$2
+        shift
+        ;;
+    *)
+        echo "error: invalid parameter '$key', please try '--help' to get list of supported parameters"
+        exit $ES_BUG
+        ;;
+    esac
+
+    shift
+done
+
+# ARM machine
+run_test arm stretch
+run_test arm buster
+
+# AMD64 machine
+if [ -z "$FAST_BUILD" ]; then
+    run_test amd64 buster
+    run_test amd64 focal
+fi
+run_test amd64 stretch
+
+# i386 machine
+if [ -z "$FAST_BUILD" ]; then
+    run_test i386 stretch
+    run_test i386 buster
+fi
+
+# ARM64 machine
+run_test arm64 stretch
+
+exit $RET
-- 
2.25.1

[PATCH v8 17/17] ci_build: Install Avocado if needed

[Anton Mikanovich]

To keep compatibility will the build systems without avocado-framework
preinstalled add the code to install it from debian-isar repo.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 scripts/ci_build.sh | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index a406dcf..415cbc7 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -28,6 +28,23 @@ for prog in ${DEPENDENCIES} ; do
     fi
 done
 
+# Install Avocado if needed
+if [ ! -x "$(which avocado)" ] ; then
+    set +e
+    APT_LIST="/etc/apt/sources.list.d"
+    ISAR_REPO="deb http://deb.isar-build.org/debian-isar buster-isar main"
+    ISAR_KEY="http://deb.isar-build.org/debian-isar.key"
+    sudo -s <<EOSUDO
+    mkdir -p $APT_LIST
+    echo "$ISAR_REPO" > $APT_LIST/isar.list
+    apt-key adv --fetch-keys $ISAR_KEY
+    apt update
+    yes | apt install avocado
+EOSUDO
+    echo "Installed Avocado-framework"
+    set -e
+fi
+
 show_help() {
     echo "This script builds the default Isar images."
     echo
-- 
2.25.1

Re: [PATCH v8 00/17] Update Avocado testsuite

[Jan Kiszka]

On 02.06.21 10:09, Anton Mikanovich wrote:
> This patchset provides implementation of build and vm_start tests
> functionality in Python with help of Avocado framework.
> Scripts ci_build and vm_smoke_test are rewritten to call Avocado.
> ci_build and vm_smoke_test are left for compatibility and
> should be replaced by direct avocado calls later.
> Original ci_build and vm_smoke_test are renamed to *_legacy.
> Build and run logs are hidden now until an error occurs.
> 
> ---
> Changes since v7:
> - Remove code duplication.
> - Fix build_dir parsing.
> - Add missed protect files backup logic.
> Changes since v6:
> - Improve environment parsing.
> - Protect files backup logic.
> - Rebuild full log output option.
> - Protect ubuntu target with KFAIL.
> Changes since v5:
> - Rebase on current next.
> Changes since v4:
> - Add container tests.
> - Implement testcase skipping logic.
> - Separate and protect Bullseye tests.
> - Add checks for container dependencies.
> - Add Ilbers repo key.
> - Improve error handling.
> - Fix Ubuntu qemu image name.
> Changes since v3:
> - Add automatic bitbake init.
> - Fix line lengths.
> - Fix build without repro.
> - Fix build artifact paths.
> - Prepare for Gitlab build.
> - Change debug settings.
> - Move log splitting to start_vm.
> - Add deb package install.
> Changes since v2:
> - Make CI call Avocado tests.
> - Wait less on vm_start test.
> - Split Roman's vm_start commit.
> - Put logs to build folder.
> - Rearrange patches.
> - Split build test to classes.
> - Add test case tagging.
> - Get rid of avocado-framework-plugin-varianter-yaml-to-mux.
> - Get rid of python-subprocess32.
> - Improve logging.
> Changes since v1:
> - Merge start_vm rebuild patches.
> - Fix patch comments.
> 
> Anton Mikanovich (14):
>   start_vm.py: Fix ubuntu image name
>   vm_boot_test: Fix log file path in vm_boot_test
>   vm_boot_test: Remove external varianter
>   vm_boot_test: Improve QEMU images checking
>   build_test: Refactoring build tests cases
>   testsuite: Add Python generations for testsuite in gitignore
>   testsuite: Fix test suite prepare guide
>   gitlab-ci: Add Avocado build artifacts
>   gitlab-ci: Add debug flag
>   vm_boot_test: Add automatic bitbake init
>   build_test: Protect ubuntu target with KFAIL
>   ci_build: Migrate to Avocado
>   vm_smoke_test: Migrate to Avocado
>   ci_build: Install Avocado if needed
> 
> Roman Pletnev (3):
>   start_vm.py: Fix target name handling
>   start_vm.py: Add output and PID file vm_start.py options
>   start_vm.py: Add MIPS support
> 
>  .gitlab-ci.yml                         |   5 +-
>  scripts/ci_build.sh                    | 213 ++++----------------
>  scripts/ci_build_legacy.sh             | 267 +++++++++++++++++++++++++
>  scripts/vm_smoke_test                  | 110 ++--------
>  scripts/vm_smoke_test_legacy           | 168 ++++++++++++++++
>  testsuite/.gitignore                   |   1 +
>  testsuite/README.md                    |  21 +-
>  testsuite/build_test/build_test.py     | 217 ++++++++++++++++++--
>  testsuite/build_test/cibase.py         | 110 ++++++++++
>  testsuite/build_test/cibuilder.py      | 103 ++++++++++
>  testsuite/build_test/run.sh            |   3 -
>  testsuite/build_test/run_fast.sh       |   3 +
>  testsuite/build_test/run_full.sh       |   3 +
>  testsuite/build_test/variant.yaml      |  22 --
>  testsuite/start_vm.py                  |  27 ++-
>  testsuite/vm_boot_test/run.sh          |   3 -
>  testsuite/vm_boot_test/run_fast.sh     |   3 +
>  testsuite/vm_boot_test/run_full.sh     |   3 +
>  testsuite/vm_boot_test/variant.yaml    |  22 --
>  testsuite/vm_boot_test/vm_boot_test.py | 121 +++++++++--
>  20 files changed, 1043 insertions(+), 382 deletions(-)
>  create mode 100755 scripts/ci_build_legacy.sh
>  create mode 100755 scripts/vm_smoke_test_legacy
>  create mode 100644 testsuite/.gitignore
>  create mode 100644 testsuite/build_test/cibase.py
>  create mode 100644 testsuite/build_test/cibuilder.py
>  delete mode 100755 testsuite/build_test/run.sh
>  create mode 100755 testsuite/build_test/run_fast.sh
>  create mode 100755 testsuite/build_test/run_full.sh
>  delete mode 100644 testsuite/build_test/variant.yaml
>  delete mode 100755 testsuite/vm_boot_test/run.sh
>  create mode 100755 testsuite/vm_boot_test/run_fast.sh
>  create mode 100755 testsuite/vm_boot_test/run_full.sh
>  delete mode 100644 testsuite/vm_boot_test/variant.yaml
> 

I tried that revision in our CI, but all attempts are failing,
apparently due to connectivity issues. That would be something to blame
on our infrastructure - if current next wouldn't pass... Retrying once
again.


Then the log output is not yet optimally tuned. Avocado seems to spit
out lengthy unformatted error reports of the kind

TestFail: b'ERROR: mc:container-amd64-bullseye:isar-image-base-1.0-r0
do_rootfs_postprocess: Error executing a python function in
exec_python_func() autogenerated:\n\nThe stack trace of python calls
that resulted in this exception/failure was:\nFile: \'exec_python_func()
autogenerated\', lineno: 2, function: <module>\n     0001:\n ***
0002:do_rootfs_postprocess(d)\n     0003:\nFile:
\'/builds/Vs1BETDQ/0/ebsy/debian/isar/meta/classes/rootfs.bbclass\',
lineno: 248, function: do_rootfs_postprocess\n     0244:    if cmds is
None or not cmds.strip():\n     0245:        return\n     0246:    cmds
= cmds.split()\n     0247:    for i, cmd in enumerate(cmds):\n *** 0248:
       bb.build.exec_func(cmd, d)\n     0249:
progress_reporter.update(int(i / len(cmds) * 100))\n     0250:}\n
0251:addtask rootfs_postprocess before do_rootfs\n     0252:\nFile:
\'/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/build.py\', lineno:
251, function: exec_func\n     0247:    with
bb.utils.fileslocked(lockfiles):\n     0248:        if ispython:\n
0249:            exec_func_python(func, d, runfile, cwd=adir)\n
0250:        else:\n *** 0251:            exec_func_shell(func, d,
runfile, cwd=adir)\n     0252:\n     0253:    try:\n     0254:
curcwd = os.getcwd()\n     0255:    except:\nFile:
\'/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/build.py\', lineno:
452, function: exec_func_shell\n     0448:    with open(fifopath,
\'r+b\', buffering=0) as fifo:\n     0449:        try:\n     0450:
      bb.debug(2, "Executing shell function %s" % func)\n     0451:
       with open(os.devnull, \'r+\') as stdin, logfile:\n *** 0452:
           bb.process.run(cmd, shell=False, stdin=stdin, log=logfile,
extrafiles=[(fifo,readfifo)])\n     0453:        finally:\n     0454:
         os.unlink(fifopath)\n     0455:\n     0456:    bb.debug(2,
"Shell function %s finished" % func)\nFile:
\'/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/process.py\',
lineno: 182, function: run\n     0178:        if not stderr is None:\n
   0179:            stderr = stderr.decode("utf-8")\n     0180:\n
0181:    if pipe.returncode != 0:\n *** 0182:        raise
ExecutionError(cmd, pipe.returncode, stdout, stderr)\n     0183:
return stdout, stderr\nException: bb.process.ExecutionError: Execution
of
\'/builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/debian-bullseye-amd64/isar-image-base-container-amd64-container-img/1.0-r0/temp/run.cache_deb_src.5240\'
failed with exit code 100:\nReading package lists...\nNeed to get 1839
kB of source archives.\nGet:1 http://deb.debian.org/debian bullseye/main
fuse 2.9.9-5 (dsc) [2312 B]\nGet:2 htt

Those a likely duplicates of what the verbose log reporting already
listed, and that was formatted in readable manner. These duplicate dumps
are very unhandy and bloat the log for no value.


There is also some warning-like error about journalctl during startup:

+ avocado --show=test run
/builds/Vs1BETDQ/0/ebsy/debian/isar/testsuite/build_test/build_test.py
-t fast,-repro -p build_dir=./build -p quiet=1 -p cross=1
Commands configured by file:
/usr/lib/python3.7/dist-packages/avocado/etc/avocado/sysinfo/commands
Files configured by file:
/usr/lib/python3.7/dist-packages/avocado/etc/avocado/sysinfo/files
File
/usr/lib/python3.7/dist-packages/avocado/etc/avocado/sysinfo/fail_commands
does not exist.
File
/usr/lib/python3.7/dist-packages/avocado/etc/avocado/sysinfo/fail_files
does not exist.
Profilers configured by file:
/usr/lib/python3.7/dist-packages/avocado/etc/avocado/sysinfo/profilers
Profiler disabled
Journalctl collection failed: [Errno 2] No such file or directory:
'journalctl' (journalctl --quiet --lines 1 --output json): 'journalctl'
Not logging /sys/kernel/debug/sched_features (file does not exist)
Not logging /proc/pci (file does not exist)
Not logging /proc/slabinfo (lack of permissions)
Command line: /usr/bin/avocado --show=test run
/builds/Vs1BETDQ/0/ebsy/debian/isar/testsuite/build_test/build_test.py
-t fast,-repro -p build_dir=./build -p quiet=1 -p cross=1

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [PATCH v8 00/17] Update Avocado testsuite

[Jan Kiszka]

On 21.06.21 08:49, Jan Kiszka wrote:
> On 02.06.21 10:09, Anton Mikanovich wrote:
>> This patchset provides implementation of build and vm_start tests
>> functionality in Python with help of Avocado framework.
>> Scripts ci_build and vm_smoke_test are rewritten to call Avocado.
>> ci_build and vm_smoke_test are left for compatibility and
>> should be replaced by direct avocado calls later.
>> Original ci_build and vm_smoke_test are renamed to *_legacy.
>> Build and run logs are hidden now until an error occurs.
>>
>> ---
>> Changes since v7:
>> - Remove code duplication.
>> - Fix build_dir parsing.
>> - Add missed protect files backup logic.
>> Changes since v6:
>> - Improve environment parsing.
>> - Protect files backup logic.
>> - Rebuild full log output option.
>> - Protect ubuntu target with KFAIL.
>> Changes since v5:
>> - Rebase on current next.
>> Changes since v4:
>> - Add container tests.
>> - Implement testcase skipping logic.
>> - Separate and protect Bullseye tests.
>> - Add checks for container dependencies.
>> - Add Ilbers repo key.
>> - Improve error handling.
>> - Fix Ubuntu qemu image name.
>> Changes since v3:
>> - Add automatic bitbake init.
>> - Fix line lengths.
>> - Fix build without repro.
>> - Fix build artifact paths.
>> - Prepare for Gitlab build.
>> - Change debug settings.
>> - Move log splitting to start_vm.
>> - Add deb package install.
>> Changes since v2:
>> - Make CI call Avocado tests.
>> - Wait less on vm_start test.
>> - Split Roman's vm_start commit.
>> - Put logs to build folder.
>> - Rearrange patches.
>> - Split build test to classes.
>> - Add test case tagging.
>> - Get rid of avocado-framework-plugin-varianter-yaml-to-mux.
>> - Get rid of python-subprocess32.
>> - Improve logging.
>> Changes since v1:
>> - Merge start_vm rebuild patches.
>> - Fix patch comments.
>>
>> Anton Mikanovich (14):
>>   start_vm.py: Fix ubuntu image name
>>   vm_boot_test: Fix log file path in vm_boot_test
>>   vm_boot_test: Remove external varianter
>>   vm_boot_test: Improve QEMU images checking
>>   build_test: Refactoring build tests cases
>>   testsuite: Add Python generations for testsuite in gitignore
>>   testsuite: Fix test suite prepare guide
>>   gitlab-ci: Add Avocado build artifacts
>>   gitlab-ci: Add debug flag
>>   vm_boot_test: Add automatic bitbake init
>>   build_test: Protect ubuntu target with KFAIL
>>   ci_build: Migrate to Avocado
>>   vm_smoke_test: Migrate to Avocado
>>   ci_build: Install Avocado if needed
>>
>> Roman Pletnev (3):
>>   start_vm.py: Fix target name handling
>>   start_vm.py: Add output and PID file vm_start.py options
>>   start_vm.py: Add MIPS support
>>
>>  .gitlab-ci.yml                         |   5 +-
>>  scripts/ci_build.sh                    | 213 ++++----------------
>>  scripts/ci_build_legacy.sh             | 267 +++++++++++++++++++++++++
>>  scripts/vm_smoke_test                  | 110 ++--------
>>  scripts/vm_smoke_test_legacy           | 168 ++++++++++++++++
>>  testsuite/.gitignore                   |   1 +
>>  testsuite/README.md                    |  21 +-
>>  testsuite/build_test/build_test.py     | 217 ++++++++++++++++++--
>>  testsuite/build_test/cibase.py         | 110 ++++++++++
>>  testsuite/build_test/cibuilder.py      | 103 ++++++++++
>>  testsuite/build_test/run.sh            |   3 -
>>  testsuite/build_test/run_fast.sh       |   3 +
>>  testsuite/build_test/run_full.sh       |   3 +
>>  testsuite/build_test/variant.yaml      |  22 --
>>  testsuite/start_vm.py                  |  27 ++-
>>  testsuite/vm_boot_test/run.sh          |   3 -
>>  testsuite/vm_boot_test/run_fast.sh     |   3 +
>>  testsuite/vm_boot_test/run_full.sh     |   3 +
>>  testsuite/vm_boot_test/variant.yaml    |  22 --
>>  testsuite/vm_boot_test/vm_boot_test.py | 121 +++++++++--
>>  20 files changed, 1043 insertions(+), 382 deletions(-)
>>  create mode 100755 scripts/ci_build_legacy.sh
>>  create mode 100755 scripts/vm_smoke_test_legacy
>>  create mode 100644 testsuite/.gitignore
>>  create mode 100644 testsuite/build_test/cibase.py
>>  create mode 100644 testsuite/build_test/cibuilder.py
>>  delete mode 100755 testsuite/build_test/run.sh
>>  create mode 100755 testsuite/build_test/run_fast.sh
>>  create mode 100755 testsuite/build_test/run_full.sh
>>  delete mode 100644 testsuite/build_test/variant.yaml
>>  delete mode 100755 testsuite/vm_boot_test/run.sh
>>  create mode 100755 testsuite/vm_boot_test/run_fast.sh
>>  create mode 100755 testsuite/vm_boot_test/run_full.sh
>>  delete mode 100644 testsuite/vm_boot_test/variant.yaml
>>
> 
> I tried that revision in our CI, but all attempts are failing,
> apparently due to connectivity issues. That would be something to blame
> on our infrastructure - if current next wouldn't pass... Retrying once
> again.

Seems there to be some changing fetch patterns, and that upset our local
proxies once again. Isar is not to blaim for the proxy mess. I was
shortly confused that do_rootfs_postprocess is always failing with
fetches, but those are related to cache_deb_src pulling all related
sources. Wasn't that tested so far, and therefore current next didn't
trigger that for us?

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux