[PATCH v1 0/6] Update Avocado testsuite

[PATCH v1 0/6] Update Avocado testsuite

[Anton Mikanovich]

This patchset implements all ci_build functionality in python with
Avocado framework.
Also includes current fixes for vm_boot_test.

Anton Mikanovich (5):
  testsuite: Fix prepare guide for Debian 10
  testsuite: Fix format_qemu_cmdline input parameters
  testsuite: Add Python generations for testsuite in gitignore
  testsuite: Fix log file path in vm_boot_test
  ci-build: Add ci_build tests cases to Avocado

Roman Pletnev (1):
  start_vm.py: Fix target name handling

 testsuite/.gitignore                          |  1 +
 testsuite/README.md                           |  9 +--
 testsuite/build_test/build_repro_test.py      | 75 +++++++++++++++++++
 testsuite/build_test/build_test.py            | 58 ++++++++++----
 testsuite/build_test/build_wic_test.py        | 65 ++++++++++++++++
 testsuite/build_test/cibuilder.py             | 66 ++++++++++++++++
 testsuite/build_test/run.sh                   |  3 -
 testsuite/build_test/run_fast.sh              | 15 ++++
 testsuite/build_test/run_full.sh              | 18 +++++
 testsuite/build_test/variants.yaml            | 45 +++++++++++
 testsuite/build_test/variants_cross.yaml      | 22 ++++++
 testsuite/build_test/variants_repro.yaml      | 13 ++++
 testsuite/build_test/variants_sdk.yaml        |  5 ++
 .../build_test/variants_wic_exclude.yaml      |  4 +
 testsuite/start_vm.py                         | 20 +++--
 testsuite/vm_boot_test/vm_boot_test.py        | 10 ++-
 16 files changed, 398 insertions(+), 31 deletions(-)
 create mode 100644 testsuite/.gitignore
 create mode 100644 testsuite/build_test/build_repro_test.py
 create mode 100644 testsuite/build_test/build_wic_test.py
 create mode 100644 testsuite/build_test/cibuilder.py
 delete mode 100755 testsuite/build_test/run.sh
 create mode 100755 testsuite/build_test/run_fast.sh
 create mode 100755 testsuite/build_test/run_full.sh
 create mode 100644 testsuite/build_test/variants.yaml
 create mode 100644 testsuite/build_test/variants_cross.yaml
 create mode 100644 testsuite/build_test/variants_repro.yaml
 create mode 100644 testsuite/build_test/variants_sdk.yaml
 create mode 100644 testsuite/build_test/variants_wic_exclude.yaml

-- 
2.25.1

[PATCH v1 1/6] testsuite: Fix prepare guide for Debian 10

[Anton Mikanovich]

There is a version mismatch between avocado-framework and
avocado-framework-plugin-varianter-yaml-to-mux that require version
lock at 69.3 in Debian 10.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/README.md | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/testsuite/README.md b/testsuite/README.md
index 5e64223..153b827 100644
--- a/testsuite/README.md
+++ b/testsuite/README.md
@@ -8,12 +8,11 @@ Then you need to install varianter yaml-to-mux plugin by following these instruc
 
   https://github.com/avocado-framework/avocado/tree/master/optional_plugins
 
-## For Debian 9.x
+## For Debian (tested on Debian 10.x)
 
-        $ sudo apt-get install python-pip
-        $ pip install --user subprocess32
-        $ pip install --user avocado-framework
-        $ pip install --user avocado-framework-plugin-varianter-yaml-to-mux
+        $ sudo apt install python-subprocess32 python-pip
+        $ pip install --user avocado-framework==69.3
+        $ pip install --user avocado-framework-plugin-varianter-yaml-to-mux==69.3
 
 # Pre
 
-- 
2.25.1

[PATCH v1 2/6] start_vm.py: Fix target name handling

[Anton Mikanovich]

From: Roman Pletnev <rpletnev@ilbers.de>

This patch fix bb_output search (issue with wrong arch name being
selected for qemu). It also add mipsel to the list of supported
architectures, add options -o (output file) and -p (pid file).

Signed-off-by: Roman Pletnev <rpletnev@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/start_vm.py | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index a3e32ac..dce140e 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -17,11 +17,11 @@ def get_bitbake_env(arch, distro):
 def get_bitbake_var(output, var):
     ret = ''
     for line in output.splitlines():
-        if line.startswith(var):
+        if line.startswith(var + '='):
             ret = line.split('"')[1]
     return ret
 
-def format_qemu_cmdline(arch, build, distro):
+def format_qemu_cmdline(arch, build, distro, out, pid):
     bb_output = get_bitbake_env(arch, distro).decode()
 
     rootfs_image = ''
@@ -57,6 +57,11 @@ def format_qemu_cmdline(arch, build, distro):
     qemu_cpu = get_bitbake_var(bb_output, 'QEMU_CPU')
     qemu_disk_args = get_bitbake_var(bb_output, 'QEMU_DISK_ARGS')
 
+    if out:
+        extra_args.extend(['-serial', out])
+    if pid:
+        extra_args.extend(['-pidfile', pid])
+
     qemu_disk_args = qemu_disk_args.replace('##ROOTFS_IMAGE##', deploy_dir_image + '/' + rootfs_image).split()
 
     cmd = ['qemu-system-' + qemu_arch, '-m', '1024M']
@@ -72,17 +77,20 @@ def format_qemu_cmdline(arch, build, distro):
 
     return cmd
 
-def start_qemu(arch, build, distro):
-    cmdline = format_qemu_cmdline(arch, build, distro)
+def start_qemu(arch, build, distro, out, pid):
+    cmdline = format_qemu_cmdline(arch, build, distro, out, pid)
     cmdline.insert(1, '-nographic')
 
+    print(cmdline)
     p1 = subprocess.call(cmdline)
 
 if __name__ == "__main__":
     parser = argparse.ArgumentParser()
-    parser.add_argument('-a', '--arch', choices=['arm', 'arm64', 'amd64', 'i386'], help='set isar machine architecture.', default='arm')
+    parser.add_argument('-a', '--arch', choices=['arm', 'arm64', 'amd64', 'i386', 'mipsel'], help='set isar machine architecture.', default='arm')
     parser.add_argument('-b', '--build', help='set path to build directory.', default=os.getcwd())
     parser.add_argument('-d', '--distro', choices=['jessie', 'stretch'], help='set isar Debian distribution.', default='stretch')
+    parser.add_argument('-o', '--out', help='Route QEMU console output to specified file.')
+    parser.add_argument('-p', '--pid', help='Store QEMU pid to specified file.')
     args = parser.parse_args()
 
-    start_qemu(args.arch, args.build, args.distro)
+    start_qemu(args.arch, args.build, args.distro, args.out, args.pid)
-- 
2.25.1

[PATCH v1 3/6] testsuite: Fix format_qemu_cmdline input parameters

[Anton Mikanovich]

Test script of vm_boot_test need to be fixed after function parameters
change in start_vm.py.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/vm_boot_test/vm_boot_test.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index d4849c7..f1b7b1f 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -30,7 +30,8 @@ class VmBootTest(Test):
         if os.path.exists(output_file):
             os.remove(output_file)
 
-        cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro)
+        cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro,
+                                               None, None)
         cmdline.insert(1, '-nographic')
         cmdline.append('-serial')
         cmdline.append('file:' + output_file)
-- 
2.25.1

[PATCH v1 4/6] testsuite: Add Python generations for testsuite in gitignore

[Anton Mikanovich]

Ignore Python byte code auto generated files inside testsuite.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/.gitignore | 1 +
 1 file changed, 1 insertion(+)
 create mode 100644 testsuite/.gitignore

diff --git a/testsuite/.gitignore b/testsuite/.gitignore
new file mode 100644
index 0000000..0d20b64
--- /dev/null
+++ b/testsuite/.gitignore
@@ -0,0 +1 @@
+*.pyc
-- 
2.25.1

[PATCH v1 5/6] testsuite: Fix log file path in vm_boot_test

[Anton Mikanovich]

Make log filename to be random enough.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/vm_boot_test/vm_boot_test.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index f1b7b1f..defc589 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -4,6 +4,7 @@ import os
 import subprocess32
 import sys
 import time
+import tempfile
 
 from os.path import dirname
 sys.path.append(dirname(__file__) + '/..')
@@ -26,9 +27,9 @@ class VmBootTest(Test):
         self.log.info('Isar build folder is: ' + build_dir)
         self.log.info('===================================================')
 
-        output_file = '/tmp/vm_boot_test.log'
-        if os.path.exists(output_file):
-            os.remove(output_file)
+        fd, output_file = tempfile.mkstemp(suffix='_log.txt',
+                                           prefix='vm_start_' + distro + '_' +
+                                           arch + '_', text=True)
 
         cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro,
                                                None, None)
-- 
2.25.1

[PATCH v1 6/6] ci-build: Add ci_build tests cases to Avocado

[Anton Mikanovich]

All test cases were copied as-is without any changes.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/build_test/build_repro_test.py      | 75 +++++++++++++++++++
 testsuite/build_test/build_test.py            | 58 ++++++++++----
 testsuite/build_test/build_wic_test.py        | 65 ++++++++++++++++
 testsuite/build_test/cibuilder.py             | 66 ++++++++++++++++
 testsuite/build_test/run.sh                   |  3 -
 testsuite/build_test/run_fast.sh              | 15 ++++
 testsuite/build_test/run_full.sh              | 18 +++++
 testsuite/build_test/variants.yaml            | 45 +++++++++++
 testsuite/build_test/variants_cross.yaml      | 22 ++++++
 testsuite/build_test/variants_repro.yaml      | 13 ++++
 testsuite/build_test/variants_sdk.yaml        |  5 ++
 .../build_test/variants_wic_exclude.yaml      |  4 +
 12 files changed, 373 insertions(+), 16 deletions(-)
 create mode 100644 testsuite/build_test/build_repro_test.py
 create mode 100644 testsuite/build_test/build_wic_test.py
 create mode 100644 testsuite/build_test/cibuilder.py
 delete mode 100755 testsuite/build_test/run.sh
 create mode 100755 testsuite/build_test/run_fast.sh
 create mode 100755 testsuite/build_test/run_full.sh
 create mode 100644 testsuite/build_test/variants.yaml
 create mode 100644 testsuite/build_test/variants_cross.yaml
 create mode 100644 testsuite/build_test/variants_repro.yaml
 create mode 100644 testsuite/build_test/variants_sdk.yaml
 create mode 100644 testsuite/build_test/variants_wic_exclude.yaml

diff --git a/testsuite/build_test/build_repro_test.py b/testsuite/build_test/build_repro_test.py
new file mode 100644
index 0000000..32a0a51
--- /dev/null
+++ b/testsuite/build_test/build_repro_test.py
@@ -0,0 +1,75 @@
+#!/usr/bin/env python3
+
+import os
+import subprocess32
+import tempfile
+
+from avocado import Test
+
+import cibuilder
+
+class BuildTest(Test):
+
+    def test_repro(self):
+        isar_root = os.path.dirname(__file__) + '/../..'
+
+        build_dir = self.params.get('build_dir', default=isar_root + '/build')
+        machine = self.params.get('machine', default='qemuamd64')
+        distro = self.params.get('distro', default='buster')
+        image = self.params.get('image', default='isar-image-base')
+
+        bitbake_args = self.params.get('bitbake_args', default='-v')
+        bitbake_cmd = self.params.get('bitbake_cmd', default=None)
+
+        cross = self.params.get('cross', default=None)
+        debug = self.params.get('debug', default=None)
+        quiet = self.params.get('quiet', default=None)
+        signed = self.params.get('signed', default=None)
+        compat_arch = self.params.get('compat_arch', default=1)
+        debsrc_cache = self.params.get('debsrc_cache', default=1)
+
+        target = self.name.variant.rsplit('-', 1)[0]
+
+        self.log.info('===================================================')
+        self.log.info('Running repro Isar build test for: ' + target)
+        self.log.info('Isar build folder is: ' + build_dir)
+        self.log.info('===================================================')
+
+        cibuilder.init(isar_root, build_dir)
+        cibuilder.confprepare(build_dir, compat_arch, cross, debsrc_cache)
+
+        gpg_pub_key = os.path.dirname(__file__) + '/../base-apt/test_pub.key'
+        gpg_priv_key = os.path.dirname(__file__) + '/../base-apt/test_priv.key'
+
+        if signed:
+            with open(build_dir + '/conf/ci_build.conf', 'w') as file:
+                # Enable use of signed cached base repository
+                file.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
+
+        if debug:
+            bitbake_args += ' -D'
+        if quiet:
+            bitbake_args = ''
+
+        os.chdir(build_dir)
+
+        os.environ['GNUPGHOME'] = tempfile.mkdtemp()
+        result = subprocess32.run(['gpg', '--import', gpg_pub_key, gpg_priv_key])
+
+        if result.returncode:
+            self.fail('GPG import failed')
+
+        if cibuilder.bitbake(build_dir, 'mc:' + target + ':' + image, bitbake_cmd, bitbake_args).returncode:
+            self.fail('Repro test failed')
+
+        cibuilder.deletetmp(build_dir)
+        with open(build_dir + '/conf/ci_build.conf', 'a') as file:
+            file.write('ISAR_USE_CACHED_BASE_REPO = "1"\n')
+            file.write('BB_NO_NETWORK = "1"\n')
+
+        if cibuilder.bitbake(build_dir, 'mc:' + target + ':' + image, bitbake_cmd, bitbake_args).returncode:
+            self.fail('Repro cached test failed')
+
+        # Cleanup and disable use of signed cached base repository
+        cibuilder.deletetmp(build_dir)
+        cibuilder.confcleanup(build_dir)
diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
index 7a55c2f..7b1d9bc 100644
--- a/testsuite/build_test/build_test.py
+++ b/testsuite/build_test/build_test.py
@@ -1,29 +1,61 @@
 #!/usr/bin/env python3
 
 import os
-import subprocess32
-import sys
-from os.path import dirname
 
 from avocado import Test
 
+import cibuilder
+
 class BuildTest(Test):
 
     def test(self):
-        # TODO: add default values
-        build_dir = self.params.get('build_dir', default='.')
-        arch = self.params.get('arch', default='arm')
-        distro = self.params.get('distro', default='stretch')
+        isar_root = os.path.dirname(__file__) + '/../..'
+
+        build_dir = self.params.get('build_dir', default=isar_root + '/build')
+        machine = self.params.get('machine', default='qemuamd64')
+        distro = self.params.get('distro', default='buster')
+        image = self.params.get('image', default='isar-image-base')
+
+        bitbake_args = self.params.get('bitbake_args', default='-v')
+        bitbake_cmd = self.params.get('bitbake_cmd', default=None)
+
+        cross = self.params.get('cross', default=None)
+        debug = self.params.get('debug', default=None)
+        quiet = self.params.get('quiet', default=None)
+        compat_arch = self.params.get('compat_arch', default=1)
+        debsrc_cache = self.params.get('debsrc_cache', default=1)
+        rebuild = self.params.get('rebuild', default=0)
+
+        target = self.name.variant.rsplit('-', 1)[0]
 
         self.log.info('===================================================')
-        self.log.info('Running Isar build test for (' + distro + '-' + arch + ')')
+        self.log.info('Running Isar build test for: ' + target)
         self.log.info('Isar build folder is: ' + build_dir)
         self.log.info('===================================================')
 
-        #isar_root = dirname(__file__) + '/..'
-        os.chdir(build_dir)
-        cmdline = ['bitbake', 'mc:qemu' + arch + '-' + distro + ':isar-image-base']
-        p1 = subprocess32.run(cmdline)
+        cibuilder.init(isar_root, build_dir)
+        cibuilder.confprepare(build_dir, compat_arch, cross, debsrc_cache)
+
+        if debug:
+            bitbake_args += ' -D'
+        if quiet:
+            bitbake_args = ''
 
-        if p1.returncode:
+        self.log.info('Starting build of ' + target)
+
+        if cibuilder.bitbake(build_dir, 'mc:' + target + ':' + image, bitbake_cmd, bitbake_args).returncode:
             self.fail('Test failed')
+
+        if rebuild:
+            layerdir_core = cibuilder.getlayerdir('core')
+
+            cibuilder.backupfile(layerdir_core + '/classes/dpkg-base.bbclass')
+            with open(layerdir_core + '/classes/dpkg-base.bbclass', 'a') as file:
+                file.write('do_fetch_append() {\n\n}\n')
+
+            self.log.info('Starting rebuild of ' + target)
+
+            if cibuilder.bitbake(build_dir, 'mc:' + target + ':' + image, bitbake_cmd, bitbake_args).returncode:
+                self.fail('Rebuild test failed')
+
+            cibuilder.restorefile(layerdir_core + '/classes/dpkg-base.bbclass')
diff --git a/testsuite/build_test/build_wic_test.py b/testsuite/build_test/build_wic_test.py
new file mode 100644
index 0000000..b012cac
--- /dev/null
+++ b/testsuite/build_test/build_wic_test.py
@@ -0,0 +1,65 @@
+#!/usr/bin/env python3
+
+import os
+import re
+
+from avocado import Test
+
+import cibuilder
+
+class BuildTest(Test):
+
+    def test_wic(self):
+        isar_root = os.path.dirname(__file__) + '/../..'
+
+        build_dir = self.params.get('build_dir', default=isar_root + '/build')
+        machine = self.params.get('machine', default='qemuamd64')
+        distro = self.params.get('distro', default='buster')
+        image = self.params.get('image', default='isar-image-base')
+
+        bitbake_args = self.params.get('bitbake_args', default='-v')
+        bitbake_cmd = self.params.get('bitbake_cmd', default=None)
+
+        cross = self.params.get('cross', default=None)
+        debug = self.params.get('debug', default=None)
+        quiet = self.params.get('quiet', default=None)
+        compat_arch = self.params.get('compat_arch', default=1)
+        debsrc_cache = self.params.get('debsrc_cache', default=1)
+
+        target = self.name.variant.rsplit('-', 1)[0]
+
+        self.log.info('===================================================')
+        self.log.info('Running WIC exclude build test for: ' + target)
+        self.log.info('Isar build folder is: ' + build_dir)
+        self.log.info('===================================================')
+
+        cibuilder.init(isar_root, build_dir)
+        cibuilder.confprepare(build_dir, compat_arch, cross, debsrc_cache)
+
+        layerdir_isar = cibuilder.getlayerdir('isar')
+
+        wks_file = layerdir_isar + '/scripts/lib/wic/canned-wks/sdimage-efi.wks'
+        wic_img = build_dir + '/tmp/deploy/images/' + machine + '/' + image + '-debian-' + distro + '-' + machine + '.wic.img'
+
+        if not os.path.isfile(wic_img):
+            self.fail('Build was not performed before: ' + wic_img + ' not exist')
+
+        cibuilder.backupfile(wks_file)
+        cibuilder.backupmove(wic_img)
+
+        with open(layerdir_isar + '/scripts/lib/wic/canned-wks/sdimage-efi.wks', 'r') as file:
+            lines = file.readlines()
+        with open(layerdir_isar + '/scripts/lib/wic/canned-wks/sdimage-efi.wks', 'w') as file:
+            for line in lines:
+                file.write(re.sub(r'part \/ ', 'part \/ --exclude-path usr ', line))
+
+        if debug:
+            bitbake_args += ' -D'
+        if quiet:
+            bitbake_args = ''
+
+        if cibuilder.bitbake(build_dir, 'mc:' + target + ':' + image, bitbake_cmd, bitbake_args).returncode:
+            self.fail('WIC test failed')
+
+        cibuilder.restorefile(wks_file)
+        cibuilder.restorefile(wic_img)
diff --git a/testsuite/build_test/cibuilder.py b/testsuite/build_test/cibuilder.py
new file mode 100644
index 0000000..3d09d26
--- /dev/null
+++ b/testsuite/build_test/cibuilder.py
@@ -0,0 +1,66 @@
+#!/usr/bin/env python3
+
+import os
+import subprocess32
+import shutil
+
+from avocado import Test
+
+def init(isar_root, build_dir):
+    os.chdir(isar_root)
+    pipe = subprocess32.Popen("source isar-init-build-env %s 2>&1 >/dev/null; env" % build_dir, stdout=subprocess32.PIPE, shell=True, executable="/bin/bash")
+    output = pipe.communicate()[0].decode('utf-8')
+    env = dict((line.split("=", 1) for line in output.splitlines()))
+    os.environ.update(env)
+
+def confprepare(build_dir, compat_arch, cross, debsrc_cache):
+    with open(build_dir + '/conf/ci_build.conf', 'w') as file:
+        if compat_arch:
+            file.write('ISAR_ENABLE_COMPAT_ARCH_amd64 = "1"\n')
+            file.write('ISAR_ENABLE_COMPAT_ARCH_arm64 = "1"\n')
+            file.write('ISAR_ENABLE_COMPAT_ARCH_debian-stretch_amd64 = "0"\n')
+        if cross:
+            file.write('ISAR_CROSS_COMPILE = "1"\n')
+        if debsrc_cache:
+            file.write('BASE_REPO_FEATURES ?= "cache-deb-src"\n')
+
+    with open(build_dir + '/conf/local.conf', 'r+') as file:
+        for line in file:
+            if 'include ci_build.conf' in line:
+                break
+        else:
+            file.write('\ninclude ci_build.conf')
+
+def confcleanup(build_dir):
+    open(build_dir + '/conf/ci_build.conf', 'w').close()
+
+def deletetmp(build_dir):
+    subprocess32.run('sudo rm -rf ' + build_dir + '/tmp', shell=True, executable="/bin/bash")
+
+def bitbake(build_dir, target, cmd, args):
+    os.chdir(build_dir)
+    if not cmd and not args:
+        cmdline = ['bitbake', target]
+    elif not cmd:
+        cmdline = ['bitbake', args, target]
+    elif not args:
+        cmdline = ['bitbake', '-c', cmd, target]
+    else:
+        cmdline = ['bitbake', args, '-c', cmd, target]
+
+    return subprocess32.run(cmdline)
+
+def backupfile(path):
+    shutil.copy2(path, path + '.ci-backup')
+
+def backupmove(path):
+    shutil.move(path, path + '.ci-backup')
+
+def restorefile(path):
+    shutil.move(path + '.ci-backup', path)
+
+def getlayerdir(layer):
+    pipe = subprocess32.Popen('bitbake -e | grep "^LAYERDIR_.*="', stdout=subprocess32.PIPE, shell=True, executable="/bin/bash")
+    output = pipe.communicate()[0].decode('utf-8')
+    bitbake_env = dict((line.split("=", 1) for line in output.splitlines()))
+    return bitbake_env['LAYERDIR_' + layer].strip('"')
diff --git a/testsuite/build_test/run.sh b/testsuite/build_test/run.sh
deleted file mode 100755
index a8ea9cc..0000000
--- a/testsuite/build_test/run.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/bash
-
-avocado run build_test.py --mux-yaml test:variant.yaml --mux-inject build_dir:$BUILDDIR
diff --git a/testsuite/build_test/run_fast.sh b/testsuite/build_test/run_fast.sh
new file mode 100755
index 0000000..4c75cc0
--- /dev/null
+++ b/testsuite/build_test/run_fast.sh
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+export PATH=$PATH:~/.local/bin
+
+# Test cached base repository
+avocado run build_repro_test.py --mux-yaml variants_repro.yaml --mux-inject quiet:1 cross:1
+
+# Start cross build for the defined set of configurations
+avocado run build_test.py --mux-yaml variants_cross.yaml --mux-inject quiet:1 cross:1
+
+# In addition test SDK creation
+avocado run build_test.py --mux-yaml variants_sdk.yaml --mux-inject quiet:1 cross:1
+
+# Test wic --exclude-path
+avocado run build_wic_test.py --mux-yaml variants_wic_exclude.yaml --mux-inject quiet:1 cross:1
diff --git a/testsuite/build_test/run_full.sh b/testsuite/build_test/run_full.sh
new file mode 100755
index 0000000..59a87d4
--- /dev/null
+++ b/testsuite/build_test/run_full.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+
+export PATH=$PATH:~/.local/bin
+
+# Test cached base repository
+avocado run build_repro_test.py --mux-yaml variants_repro.yaml --mux-inject quiet:1
+
+# Start cross build for the defined set of configurations
+avocado run build_test.py --mux-yaml variants_cross.yaml --mux-inject quiet:1 cross:1
+
+# In addition test SDK creation
+avocado run build_test.py --mux-yaml variants_sdk.yaml --mux-inject quiet:1 cross:1
+
+# Start non-cross build for the defined set of configurations
+avocado run build_test.py --mux-yaml variants.yaml --mux-inject quiet:1
+
+# Test wic --exclude-path
+avocado run build_wic_test.py --mux-yaml variants_wic_exclude.yaml --mux-inject quiet:1
diff --git a/testsuite/build_test/variants.yaml b/testsuite/build_test/variants.yaml
new file mode 100644
index 0000000..6c60586
--- /dev/null
+++ b/testsuite/build_test/variants.yaml
@@ -0,0 +1,45 @@
+variants: !mux
+    qemuarm-stretch:
+        machine: "qemuarm"
+        distro:  "stretch"
+    qemuarm-buster:
+        machine: "qemuarm"
+    qemuarm64-stretch:
+        machine: "qemuarm64"
+        distro:  "stretch"
+    qemui386-stretch:
+        machine: "qemui386"
+        distro:  "stretch"
+    qemui386-buster:
+        machine: "qemui386"
+    qemuamd64-stretch:
+        machine: "qemuamd64"
+        distro:  "stretch"
+        rebuild: 1
+    qemuamd64-buster:
+        machine: "qemuamd64"
+    qemuamd64-buster-tgz:
+        machine: "qemuamd64"
+    qemumipsel-stretch:
+        machine: "qemumipsel"
+        distro:  "stretch"
+    qemumipsel-buster:
+        machine: "qemumipsel"
+    nand-ubi-demo-buster:
+        machine: "nand-ubi-demo"
+        image:   "isar-image-ubi"
+    rpi-stretch:
+        machine: "rpi"
+        distro:  "stretch"
+    qemuamd64-bullseye:
+        machine: "qemuamd64"
+        distro:  "bullseye"
+    qemuarm-bullseye:
+        machine: "qemuarm"
+        distro:  "bullseye"
+    qemui386-bullseye:
+        machine: "qemui386"
+        distro:  "bullseye"
+    qemumipsel-bullseye:
+        machine: "qemumipsel"
+        distro:  "bullseye"
diff --git a/testsuite/build_test/variants_cross.yaml b/testsuite/build_test/variants_cross.yaml
new file mode 100644
index 0000000..e2732f3
--- /dev/null
+++ b/testsuite/build_test/variants_cross.yaml
@@ -0,0 +1,22 @@
+variants: !mux
+    qemuarm-stretch:
+        machine: "qemuarm"
+        distro:  "stretch"
+    qemuarm-buster:
+        machine: "qemuarm"
+    qemuarm64-stretch:
+        machine: "qemuarm64"
+        distro:  "stretch"
+    qemuamd64-stretch:
+        machine: "qemuamd64"
+        distro:  "stretch"
+    de0-nano-soc-buster:
+        machine: "de0-nano-soc"
+    stm32mp15x-buster:
+        machine: "stm32mp15x"
+    rpi-stretch:
+        machine: "rpi"
+        distro:  "stretch"
+    qemuarm-bullseye:
+        machine: "qemuarm"
+        distro:  "bullseye"
diff --git a/testsuite/build_test/variants_repro.yaml b/testsuite/build_test/variants_repro.yaml
new file mode 100644
index 0000000..652f7c0
--- /dev/null
+++ b/testsuite/build_test/variants_repro.yaml
@@ -0,0 +1,13 @@
+variants: !mux
+    de0-nano-soc-buster:
+        machine: "de0-nano-soc"
+        signed:  1
+    qemuarm64-stretch:
+        machine: "qemuarm64"
+        distro:  "stretch"
+        signed:  1
+    qemuamd64-stretch:
+        machine: "qemuamd64"
+        distro:  "stretch"
+    qemuarm-stretch:
+        machine: "qemuarm"
diff --git a/testsuite/build_test/variants_sdk.yaml b/testsuite/build_test/variants_sdk.yaml
new file mode 100644
index 0000000..a38b4ff
--- /dev/null
+++ b/testsuite/build_test/variants_sdk.yaml
@@ -0,0 +1,5 @@
+variants: !mux
+    qemuarm-stretch:
+        machine:     "qemuarm"
+        distro:      "stretch"
+        bitbake_cmd: "do_populate_sdk"
diff --git a/testsuite/build_test/variants_wic_exclude.yaml b/testsuite/build_test/variants_wic_exclude.yaml
new file mode 100644
index 0000000..392036f
--- /dev/null
+++ b/testsuite/build_test/variants_wic_exclude.yaml
@@ -0,0 +1,4 @@
+variants: !mux
+    qemuamd64-stretch:
+        machine: "qemuamd64"
+        distro:  "stretch"
-- 
2.25.1

[PATCH v2 0/5] Update Avocado testsuite

[Anton Mikanovich]

This patchset implements all ci_build functionality in python with
Avocado framework.
Also includes current fixes for vm_boot_test.
---
Changes since v1:
- Merge start_vm rebuild patches.
- Fix patch comments.

Anton Mikanovich (4):
  testsuite: Fix prepare guide for Debian 10
  testsuite: Add Python generations for testsuite in gitignore
  testsuite: Fix log file path in vm_boot_test
  ci-build: Add ci_build tests cases to Avocado

Roman Pletnev (1):
  start_vm.py: Fix target name handling

 testsuite/.gitignore                          |  1 +
 testsuite/README.md                           |  9 +--
 testsuite/build_test/build_repro_test.py      | 75 +++++++++++++++++++
 testsuite/build_test/build_test.py            | 58 ++++++++++----
 testsuite/build_test/build_wic_test.py        | 65 ++++++++++++++++
 testsuite/build_test/cibuilder.py             | 66 ++++++++++++++++
 testsuite/build_test/run.sh                   |  3 -
 testsuite/build_test/run_fast.sh              | 15 ++++
 testsuite/build_test/run_full.sh              | 18 +++++
 testsuite/build_test/variants.yaml            | 45 +++++++++++
 testsuite/build_test/variants_cross.yaml      | 22 ++++++
 testsuite/build_test/variants_repro.yaml      | 13 ++++
 testsuite/build_test/variants_sdk.yaml        |  5 ++
 .../build_test/variants_wic_exclude.yaml      |  4 +
 testsuite/start_vm.py                         | 20 +++--
 testsuite/vm_boot_test/vm_boot_test.py        | 10 ++-
 16 files changed, 398 insertions(+), 31 deletions(-)
 create mode 100644 testsuite/.gitignore
 create mode 100644 testsuite/build_test/build_repro_test.py
 create mode 100644 testsuite/build_test/build_wic_test.py
 create mode 100644 testsuite/build_test/cibuilder.py
 delete mode 100755 testsuite/build_test/run.sh
 create mode 100755 testsuite/build_test/run_fast.sh
 create mode 100755 testsuite/build_test/run_full.sh
 create mode 100644 testsuite/build_test/variants.yaml
 create mode 100644 testsuite/build_test/variants_cross.yaml
 create mode 100644 testsuite/build_test/variants_repro.yaml
 create mode 100644 testsuite/build_test/variants_sdk.yaml
 create mode 100644 testsuite/build_test/variants_wic_exclude.yaml

-- 
2.25.1

[PATCH v2 1/5] testsuite: Fix prepare guide for Debian 10

[Anton Mikanovich]

There is a version mismatch between avocado-framework and
avocado-framework-plugin-varianter-yaml-to-mux that require version
lock at 69.3 in Debian 10.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/README.md | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/testsuite/README.md b/testsuite/README.md
index 5e64223..153b827 100644
--- a/testsuite/README.md
+++ b/testsuite/README.md
@@ -8,12 +8,11 @@ Then you need to install varianter yaml-to-mux plugin by following these instruc
 
   https://github.com/avocado-framework/avocado/tree/master/optional_plugins
 
-## For Debian 9.x
+## For Debian (tested on Debian 10.x)
 
-        $ sudo apt-get install python-pip
-        $ pip install --user subprocess32
-        $ pip install --user avocado-framework
-        $ pip install --user avocado-framework-plugin-varianter-yaml-to-mux
+        $ sudo apt install python-subprocess32 python-pip
+        $ pip install --user avocado-framework==69.3
+        $ pip install --user avocado-framework-plugin-varianter-yaml-to-mux==69.3
 
 # Pre
 
-- 
2.25.1

[PATCH v2 2/5] start_vm.py: Fix target name handling

[Anton Mikanovich]

From: Roman Pletnev <rpletnev@ilbers.de>

This patch fix bb_output search (issue with wrong arch name being
selected for qemu). It also add mipsel to the list of supported
architectures, add options -o (output file) and -p (pid file).

Signed-off-by: Roman Pletnev <rpletnev@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/start_vm.py                  | 20 ++++++++++++++------
 testsuite/vm_boot_test/vm_boot_test.py |  3 ++-
 2 files changed, 16 insertions(+), 7 deletions(-)

diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index a3e32ac..dce140e 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -17,11 +17,11 @@ def get_bitbake_env(arch, distro):
 def get_bitbake_var(output, var):
     ret = ''
     for line in output.splitlines():
-        if line.startswith(var):
+        if line.startswith(var + '='):
             ret = line.split('"')[1]
     return ret
 
-def format_qemu_cmdline(arch, build, distro):
+def format_qemu_cmdline(arch, build, distro, out, pid):
     bb_output = get_bitbake_env(arch, distro).decode()
 
     rootfs_image = ''
@@ -57,6 +57,11 @@ def format_qemu_cmdline(arch, build, distro):
     qemu_cpu = get_bitbake_var(bb_output, 'QEMU_CPU')
     qemu_disk_args = get_bitbake_var(bb_output, 'QEMU_DISK_ARGS')
 
+    if out:
+        extra_args.extend(['-serial', out])
+    if pid:
+        extra_args.extend(['-pidfile', pid])
+
     qemu_disk_args = qemu_disk_args.replace('##ROOTFS_IMAGE##', deploy_dir_image + '/' + rootfs_image).split()
 
     cmd = ['qemu-system-' + qemu_arch, '-m', '1024M']
@@ -72,17 +77,20 @@ def format_qemu_cmdline(arch, build, distro):
 
     return cmd
 
-def start_qemu(arch, build, distro):
-    cmdline = format_qemu_cmdline(arch, build, distro)
+def start_qemu(arch, build, distro, out, pid):
+    cmdline = format_qemu_cmdline(arch, build, distro, out, pid)
     cmdline.insert(1, '-nographic')
 
+    print(cmdline)
     p1 = subprocess.call(cmdline)
 
 if __name__ == "__main__":
     parser = argparse.ArgumentParser()
-    parser.add_argument('-a', '--arch', choices=['arm', 'arm64', 'amd64', 'i386'], help='set isar machine architecture.', default='arm')
+    parser.add_argument('-a', '--arch', choices=['arm', 'arm64', 'amd64', 'i386', 'mipsel'], help='set isar machine architecture.', default='arm')
     parser.add_argument('-b', '--build', help='set path to build directory.', default=os.getcwd())
     parser.add_argument('-d', '--distro', choices=['jessie', 'stretch'], help='set isar Debian distribution.', default='stretch')
+    parser.add_argument('-o', '--out', help='Route QEMU console output to specified file.')
+    parser.add_argument('-p', '--pid', help='Store QEMU pid to specified file.')
     args = parser.parse_args()
 
-    start_qemu(args.arch, args.build, args.distro)
+    start_qemu(args.arch, args.build, args.distro, args.out, args.pid)
diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index d4849c7..f1b7b1f 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -30,7 +30,8 @@ class VmBootTest(Test):
         if os.path.exists(output_file):
             os.remove(output_file)
 
-        cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro)
+        cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro,
+                                               None, None)
         cmdline.insert(1, '-nographic')
         cmdline.append('-serial')
         cmdline.append('file:' + output_file)
-- 
2.25.1

[PATCH v2 3/5] testsuite: Add Python generations for testsuite in gitignore

[Anton Mikanovich]

Ignore Python byte code auto generated files inside testsuite.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/.gitignore | 1 +
 1 file changed, 1 insertion(+)
 create mode 100644 testsuite/.gitignore

diff --git a/testsuite/.gitignore b/testsuite/.gitignore
new file mode 100644
index 0000000..0d20b64
--- /dev/null
+++ b/testsuite/.gitignore
@@ -0,0 +1 @@
+*.pyc
-- 
2.25.1

[PATCH v2 4/5] testsuite: Fix log file path in vm_boot_test

[Anton Mikanovich]

Make log filename random.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/vm_boot_test/vm_boot_test.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index f1b7b1f..defc589 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -4,6 +4,7 @@ import os
 import subprocess32
 import sys
 import time
+import tempfile
 
 from os.path import dirname
 sys.path.append(dirname(__file__) + '/..')
@@ -26,9 +27,9 @@ class VmBootTest(Test):
         self.log.info('Isar build folder is: ' + build_dir)
         self.log.info('===================================================')
 
-        output_file = '/tmp/vm_boot_test.log'
-        if os.path.exists(output_file):
-            os.remove(output_file)
+        fd, output_file = tempfile.mkstemp(suffix='_log.txt',
+                                           prefix='vm_start_' + distro + '_' +
+                                           arch + '_', text=True)
 
         cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro,
                                                None, None)
-- 
2.25.1

[PATCH v2 5/5] ci-build: Add ci_build tests cases to Avocado

[Anton Mikanovich]

All test cases were copied as-is without any changes.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/build_test/build_repro_test.py      | 75 +++++++++++++++++++
 testsuite/build_test/build_test.py            | 58 ++++++++++----
 testsuite/build_test/build_wic_test.py        | 65 ++++++++++++++++
 testsuite/build_test/cibuilder.py             | 66 ++++++++++++++++
 testsuite/build_test/run.sh                   |  3 -
 testsuite/build_test/run_fast.sh              | 15 ++++
 testsuite/build_test/run_full.sh              | 18 +++++
 testsuite/build_test/variants.yaml            | 45 +++++++++++
 testsuite/build_test/variants_cross.yaml      | 22 ++++++
 testsuite/build_test/variants_repro.yaml      | 13 ++++
 testsuite/build_test/variants_sdk.yaml        |  5 ++
 .../build_test/variants_wic_exclude.yaml      |  4 +
 12 files changed, 373 insertions(+), 16 deletions(-)
 create mode 100644 testsuite/build_test/build_repro_test.py
 create mode 100644 testsuite/build_test/build_wic_test.py
 create mode 100644 testsuite/build_test/cibuilder.py
 delete mode 100755 testsuite/build_test/run.sh
 create mode 100755 testsuite/build_test/run_fast.sh
 create mode 100755 testsuite/build_test/run_full.sh
 create mode 100644 testsuite/build_test/variants.yaml
 create mode 100644 testsuite/build_test/variants_cross.yaml
 create mode 100644 testsuite/build_test/variants_repro.yaml
 create mode 100644 testsuite/build_test/variants_sdk.yaml
 create mode 100644 testsuite/build_test/variants_wic_exclude.yaml

diff --git a/testsuite/build_test/build_repro_test.py b/testsuite/build_test/build_repro_test.py
new file mode 100644
index 0000000..32a0a51
--- /dev/null
+++ b/testsuite/build_test/build_repro_test.py
@@ -0,0 +1,75 @@
+#!/usr/bin/env python3
+
+import os
+import subprocess32
+import tempfile
+
+from avocado import Test
+
+import cibuilder
+
+class BuildTest(Test):
+
+    def test_repro(self):
+        isar_root = os.path.dirname(__file__) + '/../..'
+
+        build_dir = self.params.get('build_dir', default=isar_root + '/build')
+        machine = self.params.get('machine', default='qemuamd64')
+        distro = self.params.get('distro', default='buster')
+        image = self.params.get('image', default='isar-image-base')
+
+        bitbake_args = self.params.get('bitbake_args', default='-v')
+        bitbake_cmd = self.params.get('bitbake_cmd', default=None)
+
+        cross = self.params.get('cross', default=None)
+        debug = self.params.get('debug', default=None)
+        quiet = self.params.get('quiet', default=None)
+        signed = self.params.get('signed', default=None)
+        compat_arch = self.params.get('compat_arch', default=1)
+        debsrc_cache = self.params.get('debsrc_cache', default=1)
+
+        target = self.name.variant.rsplit('-', 1)[0]
+
+        self.log.info('===================================================')
+        self.log.info('Running repro Isar build test for: ' + target)
+        self.log.info('Isar build folder is: ' + build_dir)
+        self.log.info('===================================================')
+
+        cibuilder.init(isar_root, build_dir)
+        cibuilder.confprepare(build_dir, compat_arch, cross, debsrc_cache)
+
+        gpg_pub_key = os.path.dirname(__file__) + '/../base-apt/test_pub.key'
+        gpg_priv_key = os.path.dirname(__file__) + '/../base-apt/test_priv.key'
+
+        if signed:
+            with open(build_dir + '/conf/ci_build.conf', 'w') as file:
+                # Enable use of signed cached base repository
+                file.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
+
+        if debug:
+            bitbake_args += ' -D'
+        if quiet:
+            bitbake_args = ''
+
+        os.chdir(build_dir)
+
+        os.environ['GNUPGHOME'] = tempfile.mkdtemp()
+        result = subprocess32.run(['gpg', '--import', gpg_pub_key, gpg_priv_key])
+
+        if result.returncode:
+            self.fail('GPG import failed')
+
+        if cibuilder.bitbake(build_dir, 'mc:' + target + ':' + image, bitbake_cmd, bitbake_args).returncode:
+            self.fail('Repro test failed')
+
+        cibuilder.deletetmp(build_dir)
+        with open(build_dir + '/conf/ci_build.conf', 'a') as file:
+            file.write('ISAR_USE_CACHED_BASE_REPO = "1"\n')
+            file.write('BB_NO_NETWORK = "1"\n')
+
+        if cibuilder.bitbake(build_dir, 'mc:' + target + ':' + image, bitbake_cmd, bitbake_args).returncode:
+            self.fail('Repro cached test failed')
+
+        # Cleanup and disable use of signed cached base repository
+        cibuilder.deletetmp(build_dir)
+        cibuilder.confcleanup(build_dir)
diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
index 7a55c2f..7b1d9bc 100644
--- a/testsuite/build_test/build_test.py
+++ b/testsuite/build_test/build_test.py
@@ -1,29 +1,61 @@
 #!/usr/bin/env python3
 
 import os
-import subprocess32
-import sys
-from os.path import dirname
 
 from avocado import Test
 
+import cibuilder
+
 class BuildTest(Test):
 
     def test(self):
-        # TODO: add default values
-        build_dir = self.params.get('build_dir', default='.')
-        arch = self.params.get('arch', default='arm')
-        distro = self.params.get('distro', default='stretch')
+        isar_root = os.path.dirname(__file__) + '/../..'
+
+        build_dir = self.params.get('build_dir', default=isar_root + '/build')
+        machine = self.params.get('machine', default='qemuamd64')
+        distro = self.params.get('distro', default='buster')
+        image = self.params.get('image', default='isar-image-base')
+
+        bitbake_args = self.params.get('bitbake_args', default='-v')
+        bitbake_cmd = self.params.get('bitbake_cmd', default=None)
+
+        cross = self.params.get('cross', default=None)
+        debug = self.params.get('debug', default=None)
+        quiet = self.params.get('quiet', default=None)
+        compat_arch = self.params.get('compat_arch', default=1)
+        debsrc_cache = self.params.get('debsrc_cache', default=1)
+        rebuild = self.params.get('rebuild', default=0)
+
+        target = self.name.variant.rsplit('-', 1)[0]
 
         self.log.info('===================================================')
-        self.log.info('Running Isar build test for (' + distro + '-' + arch + ')')
+        self.log.info('Running Isar build test for: ' + target)
         self.log.info('Isar build folder is: ' + build_dir)
         self.log.info('===================================================')
 
-        #isar_root = dirname(__file__) + '/..'
-        os.chdir(build_dir)
-        cmdline = ['bitbake', 'mc:qemu' + arch + '-' + distro + ':isar-image-base']
-        p1 = subprocess32.run(cmdline)
+        cibuilder.init(isar_root, build_dir)
+        cibuilder.confprepare(build_dir, compat_arch, cross, debsrc_cache)
+
+        if debug:
+            bitbake_args += ' -D'
+        if quiet:
+            bitbake_args = ''
 
-        if p1.returncode:
+        self.log.info('Starting build of ' + target)
+
+        if cibuilder.bitbake(build_dir, 'mc:' + target + ':' + image, bitbake_cmd, bitbake_args).returncode:
             self.fail('Test failed')
+
+        if rebuild:
+            layerdir_core = cibuilder.getlayerdir('core')
+
+            cibuilder.backupfile(layerdir_core + '/classes/dpkg-base.bbclass')
+            with open(layerdir_core + '/classes/dpkg-base.bbclass', 'a') as file:
+                file.write('do_fetch_append() {\n\n}\n')
+
+            self.log.info('Starting rebuild of ' + target)
+
+            if cibuilder.bitbake(build_dir, 'mc:' + target + ':' + image, bitbake_cmd, bitbake_args).returncode:
+                self.fail('Rebuild test failed')
+
+            cibuilder.restorefile(layerdir_core + '/classes/dpkg-base.bbclass')
diff --git a/testsuite/build_test/build_wic_test.py b/testsuite/build_test/build_wic_test.py
new file mode 100644
index 0000000..b012cac
--- /dev/null
+++ b/testsuite/build_test/build_wic_test.py
@@ -0,0 +1,65 @@
+#!/usr/bin/env python3
+
+import os
+import re
+
+from avocado import Test
+
+import cibuilder
+
+class BuildTest(Test):
+
+    def test_wic(self):
+        isar_root = os.path.dirname(__file__) + '/../..'
+
+        build_dir = self.params.get('build_dir', default=isar_root + '/build')
+        machine = self.params.get('machine', default='qemuamd64')
+        distro = self.params.get('distro', default='buster')
+        image = self.params.get('image', default='isar-image-base')
+
+        bitbake_args = self.params.get('bitbake_args', default='-v')
+        bitbake_cmd = self.params.get('bitbake_cmd', default=None)
+
+        cross = self.params.get('cross', default=None)
+        debug = self.params.get('debug', default=None)
+        quiet = self.params.get('quiet', default=None)
+        compat_arch = self.params.get('compat_arch', default=1)
+        debsrc_cache = self.params.get('debsrc_cache', default=1)
+
+        target = self.name.variant.rsplit('-', 1)[0]
+
+        self.log.info('===================================================')
+        self.log.info('Running WIC exclude build test for: ' + target)
+        self.log.info('Isar build folder is: ' + build_dir)
+        self.log.info('===================================================')
+
+        cibuilder.init(isar_root, build_dir)
+        cibuilder.confprepare(build_dir, compat_arch, cross, debsrc_cache)
+
+        layerdir_isar = cibuilder.getlayerdir('isar')
+
+        wks_file = layerdir_isar + '/scripts/lib/wic/canned-wks/sdimage-efi.wks'
+        wic_img = build_dir + '/tmp/deploy/images/' + machine + '/' + image + '-debian-' + distro + '-' + machine + '.wic.img'
+
+        if not os.path.isfile(wic_img):
+            self.fail('Build was not performed before: ' + wic_img + ' not exist')
+
+        cibuilder.backupfile(wks_file)
+        cibuilder.backupmove(wic_img)
+
+        with open(layerdir_isar + '/scripts/lib/wic/canned-wks/sdimage-efi.wks', 'r') as file:
+            lines = file.readlines()
+        with open(layerdir_isar + '/scripts/lib/wic/canned-wks/sdimage-efi.wks', 'w') as file:
+            for line in lines:
+                file.write(re.sub(r'part \/ ', 'part \/ --exclude-path usr ', line))
+
+        if debug:
+            bitbake_args += ' -D'
+        if quiet:
+            bitbake_args = ''
+
+        if cibuilder.bitbake(build_dir, 'mc:' + target + ':' + image, bitbake_cmd, bitbake_args).returncode:
+            self.fail('WIC test failed')
+
+        cibuilder.restorefile(wks_file)
+        cibuilder.restorefile(wic_img)
diff --git a/testsuite/build_test/cibuilder.py b/testsuite/build_test/cibuilder.py
new file mode 100644
index 0000000..3d09d26
--- /dev/null
+++ b/testsuite/build_test/cibuilder.py
@@ -0,0 +1,66 @@
+#!/usr/bin/env python3
+
+import os
+import subprocess32
+import shutil
+
+from avocado import Test
+
+def init(isar_root, build_dir):
+    os.chdir(isar_root)
+    pipe = subprocess32.Popen("source isar-init-build-env %s 2>&1 >/dev/null; env" % build_dir, stdout=subprocess32.PIPE, shell=True, executable="/bin/bash")
+    output = pipe.communicate()[0].decode('utf-8')
+    env = dict((line.split("=", 1) for line in output.splitlines()))
+    os.environ.update(env)
+
+def confprepare(build_dir, compat_arch, cross, debsrc_cache):
+    with open(build_dir + '/conf/ci_build.conf', 'w') as file:
+        if compat_arch:
+            file.write('ISAR_ENABLE_COMPAT_ARCH_amd64 = "1"\n')
+            file.write('ISAR_ENABLE_COMPAT_ARCH_arm64 = "1"\n')
+            file.write('ISAR_ENABLE_COMPAT_ARCH_debian-stretch_amd64 = "0"\n')
+        if cross:
+            file.write('ISAR_CROSS_COMPILE = "1"\n')
+        if debsrc_cache:
+            file.write('BASE_REPO_FEATURES ?= "cache-deb-src"\n')
+
+    with open(build_dir + '/conf/local.conf', 'r+') as file:
+        for line in file:
+            if 'include ci_build.conf' in line:
+                break
+        else:
+            file.write('\ninclude ci_build.conf')
+
+def confcleanup(build_dir):
+    open(build_dir + '/conf/ci_build.conf', 'w').close()
+
+def deletetmp(build_dir):
+    subprocess32.run('sudo rm -rf ' + build_dir + '/tmp', shell=True, executable="/bin/bash")
+
+def bitbake(build_dir, target, cmd, args):
+    os.chdir(build_dir)
+    if not cmd and not args:
+        cmdline = ['bitbake', target]
+    elif not cmd:
+        cmdline = ['bitbake', args, target]
+    elif not args:
+        cmdline = ['bitbake', '-c', cmd, target]
+    else:
+        cmdline = ['bitbake', args, '-c', cmd, target]
+
+    return subprocess32.run(cmdline)
+
+def backupfile(path):
+    shutil.copy2(path, path + '.ci-backup')
+
+def backupmove(path):
+    shutil.move(path, path + '.ci-backup')
+
+def restorefile(path):
+    shutil.move(path + '.ci-backup', path)
+
+def getlayerdir(layer):
+    pipe = subprocess32.Popen('bitbake -e | grep "^LAYERDIR_.*="', stdout=subprocess32.PIPE, shell=True, executable="/bin/bash")
+    output = pipe.communicate()[0].decode('utf-8')
+    bitbake_env = dict((line.split("=", 1) for line in output.splitlines()))
+    return bitbake_env['LAYERDIR_' + layer].strip('"')
diff --git a/testsuite/build_test/run.sh b/testsuite/build_test/run.sh
deleted file mode 100755
index a8ea9cc..0000000
--- a/testsuite/build_test/run.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/bash
-
-avocado run build_test.py --mux-yaml test:variant.yaml --mux-inject build_dir:$BUILDDIR
diff --git a/testsuite/build_test/run_fast.sh b/testsuite/build_test/run_fast.sh
new file mode 100755
index 0000000..4c75cc0
--- /dev/null
+++ b/testsuite/build_test/run_fast.sh
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+export PATH=$PATH:~/.local/bin
+
+# Test cached base repository
+avocado run build_repro_test.py --mux-yaml variants_repro.yaml --mux-inject quiet:1 cross:1
+
+# Start cross build for the defined set of configurations
+avocado run build_test.py --mux-yaml variants_cross.yaml --mux-inject quiet:1 cross:1
+
+# In addition test SDK creation
+avocado run build_test.py --mux-yaml variants_sdk.yaml --mux-inject quiet:1 cross:1
+
+# Test wic --exclude-path
+avocado run build_wic_test.py --mux-yaml variants_wic_exclude.yaml --mux-inject quiet:1 cross:1
diff --git a/testsuite/build_test/run_full.sh b/testsuite/build_test/run_full.sh
new file mode 100755
index 0000000..59a87d4
--- /dev/null
+++ b/testsuite/build_test/run_full.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+
+export PATH=$PATH:~/.local/bin
+
+# Test cached base repository
+avocado run build_repro_test.py --mux-yaml variants_repro.yaml --mux-inject quiet:1
+
+# Start cross build for the defined set of configurations
+avocado run build_test.py --mux-yaml variants_cross.yaml --mux-inject quiet:1 cross:1
+
+# In addition test SDK creation
+avocado run build_test.py --mux-yaml variants_sdk.yaml --mux-inject quiet:1 cross:1
+
+# Start non-cross build for the defined set of configurations
+avocado run build_test.py --mux-yaml variants.yaml --mux-inject quiet:1
+
+# Test wic --exclude-path
+avocado run build_wic_test.py --mux-yaml variants_wic_exclude.yaml --mux-inject quiet:1
diff --git a/testsuite/build_test/variants.yaml b/testsuite/build_test/variants.yaml
new file mode 100644
index 0000000..6c60586
--- /dev/null
+++ b/testsuite/build_test/variants.yaml
@@ -0,0 +1,45 @@
+variants: !mux
+    qemuarm-stretch:
+        machine: "qemuarm"
+        distro:  "stretch"
+    qemuarm-buster:
+        machine: "qemuarm"
+    qemuarm64-stretch:
+        machine: "qemuarm64"
+        distro:  "stretch"
+    qemui386-stretch:
+        machine: "qemui386"
+        distro:  "stretch"
+    qemui386-buster:
+        machine: "qemui386"
+    qemuamd64-stretch:
+        machine: "qemuamd64"
+        distro:  "stretch"
+        rebuild: 1
+    qemuamd64-buster:
+        machine: "qemuamd64"
+    qemuamd64-buster-tgz:
+        machine: "qemuamd64"
+    qemumipsel-stretch:
+        machine: "qemumipsel"
+        distro:  "stretch"
+    qemumipsel-buster:
+        machine: "qemumipsel"
+    nand-ubi-demo-buster:
+        machine: "nand-ubi-demo"
+        image:   "isar-image-ubi"
+    rpi-stretch:
+        machine: "rpi"
+        distro:  "stretch"
+    qemuamd64-bullseye:
+        machine: "qemuamd64"
+        distro:  "bullseye"
+    qemuarm-bullseye:
+        machine: "qemuarm"
+        distro:  "bullseye"
+    qemui386-bullseye:
+        machine: "qemui386"
+        distro:  "bullseye"
+    qemumipsel-bullseye:
+        machine: "qemumipsel"
+        distro:  "bullseye"
diff --git a/testsuite/build_test/variants_cross.yaml b/testsuite/build_test/variants_cross.yaml
new file mode 100644
index 0000000..e2732f3
--- /dev/null
+++ b/testsuite/build_test/variants_cross.yaml
@@ -0,0 +1,22 @@
+variants: !mux
+    qemuarm-stretch:
+        machine: "qemuarm"
+        distro:  "stretch"
+    qemuarm-buster:
+        machine: "qemuarm"
+    qemuarm64-stretch:
+        machine: "qemuarm64"
+        distro:  "stretch"
+    qemuamd64-stretch:
+        machine: "qemuamd64"
+        distro:  "stretch"
+    de0-nano-soc-buster:
+        machine: "de0-nano-soc"
+    stm32mp15x-buster:
+        machine: "stm32mp15x"
+    rpi-stretch:
+        machine: "rpi"
+        distro:  "stretch"
+    qemuarm-bullseye:
+        machine: "qemuarm"
+        distro:  "bullseye"
diff --git a/testsuite/build_test/variants_repro.yaml b/testsuite/build_test/variants_repro.yaml
new file mode 100644
index 0000000..652f7c0
--- /dev/null
+++ b/testsuite/build_test/variants_repro.yaml
@@ -0,0 +1,13 @@
+variants: !mux
+    de0-nano-soc-buster:
+        machine: "de0-nano-soc"
+        signed:  1
+    qemuarm64-stretch:
+        machine: "qemuarm64"
+        distro:  "stretch"
+        signed:  1
+    qemuamd64-stretch:
+        machine: "qemuamd64"
+        distro:  "stretch"
+    qemuarm-stretch:
+        machine: "qemuarm"
diff --git a/testsuite/build_test/variants_sdk.yaml b/testsuite/build_test/variants_sdk.yaml
new file mode 100644
index 0000000..a38b4ff
--- /dev/null
+++ b/testsuite/build_test/variants_sdk.yaml
@@ -0,0 +1,5 @@
+variants: !mux
+    qemuarm-stretch:
+        machine:     "qemuarm"
+        distro:      "stretch"
+        bitbake_cmd: "do_populate_sdk"
diff --git a/testsuite/build_test/variants_wic_exclude.yaml b/testsuite/build_test/variants_wic_exclude.yaml
new file mode 100644
index 0000000..392036f
--- /dev/null
+++ b/testsuite/build_test/variants_wic_exclude.yaml
@@ -0,0 +1,4 @@
+variants: !mux
+    qemuamd64-stretch:
+        machine: "qemuamd64"
+        distro:  "stretch"
-- 
2.25.1

Re: [PATCH v2 0/5] Update Avocado testsuite

[Jan Kiszka]

On 15.02.21 10:26, Anton Mikanovich wrote:
> This patchset implements all ci_build functionality in python with
> Avocado framework.

...but it does not plug into scripts/ci_build.sh, does it? Who is using
this then? And are we duplicating tests ATM?

Jan

> Also includes current fixes for vm_boot_test.
> ---
> Changes since v1:
> - Merge start_vm rebuild patches.
> - Fix patch comments.
> 
> Anton Mikanovich (4):
>   testsuite: Fix prepare guide for Debian 10
>   testsuite: Add Python generations for testsuite in gitignore
>   testsuite: Fix log file path in vm_boot_test
>   ci-build: Add ci_build tests cases to Avocado
> 
> Roman Pletnev (1):
>   start_vm.py: Fix target name handling
> 
>  testsuite/.gitignore                          |  1 +
>  testsuite/README.md                           |  9 +--
>  testsuite/build_test/build_repro_test.py      | 75 +++++++++++++++++++
>  testsuite/build_test/build_test.py            | 58 ++++++++++----
>  testsuite/build_test/build_wic_test.py        | 65 ++++++++++++++++
>  testsuite/build_test/cibuilder.py             | 66 ++++++++++++++++
>  testsuite/build_test/run.sh                   |  3 -
>  testsuite/build_test/run_fast.sh              | 15 ++++
>  testsuite/build_test/run_full.sh              | 18 +++++
>  testsuite/build_test/variants.yaml            | 45 +++++++++++
>  testsuite/build_test/variants_cross.yaml      | 22 ++++++
>  testsuite/build_test/variants_repro.yaml      | 13 ++++
>  testsuite/build_test/variants_sdk.yaml        |  5 ++
>  .../build_test/variants_wic_exclude.yaml      |  4 +
>  testsuite/start_vm.py                         | 20 +++--
>  testsuite/vm_boot_test/vm_boot_test.py        | 10 ++-
>  16 files changed, 398 insertions(+), 31 deletions(-)
>  create mode 100644 testsuite/.gitignore
>  create mode 100644 testsuite/build_test/build_repro_test.py
>  create mode 100644 testsuite/build_test/build_wic_test.py
>  create mode 100644 testsuite/build_test/cibuilder.py
>  delete mode 100755 testsuite/build_test/run.sh
>  create mode 100755 testsuite/build_test/run_fast.sh
>  create mode 100755 testsuite/build_test/run_full.sh
>  create mode 100644 testsuite/build_test/variants.yaml
>  create mode 100644 testsuite/build_test/variants_cross.yaml
>  create mode 100644 testsuite/build_test/variants_repro.yaml
>  create mode 100644 testsuite/build_test/variants_sdk.yaml
>  create mode 100644 testsuite/build_test/variants_wic_exclude.yaml
> 

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [PATCH v2 0/5] Update Avocado testsuite

[Baurzhan Ismagulov]

On Mon, Feb 15, 2021 at 11:42:03AM +0100, Jan Kiszka wrote:
> ...but it does not plug into scripts/ci_build.sh, does it? Who is using
> this then? And are we duplicating tests ATM?

Yes, as mentioned in the other thread, it doesn't yet. The idea is to give
users an opportunity to test it before making it the default. If the consensus
is we should do that earlier than later, we could do that.

With kind regards,
Baurzhan.

Re: [PATCH v2 0/5] Update Avocado testsuite

[Jan Kiszka]

On 15.02.21 12:41, Baurzhan Ismagulov wrote:
> On Mon, Feb 15, 2021 at 11:42:03AM +0100, Jan Kiszka wrote:
>> ...but it does not plug into scripts/ci_build.sh, does it? Who is using
>> this then? And are we duplicating tests ATM?
> 
> Yes, as mentioned in the other thread, it doesn't yet. The idea is to give
> users an opportunity to test it before making it the default. If the consensus
> is we should do that earlier than later, we could do that.

I likely won't run it until it's integrated in CI. So far, it seems to
require too many manual steps. At least create a branch where this
replacement already happens so that people can inspect. Bonus for
updating .gitlab-ci.yml as well.

Jan