* [PATCH v14 00/16] Update Avocado testsuite
@ 2021-09-30 16:45 Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 01/16] start_vm.py: Fix target name handling Anton Mikanovich
` (16 more replies)
0 siblings, 17 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
This patchset provides implementation of build and vm_start tests
functionality in Python with help of Avocado framework.
Scripts ci_build and vm_smoke_test are rewritten to call Avocado.
Build and run logs are hidden by default now until an error occurs.
Resending v14 because v13 was just a copy of v12 by mistake.
---
Changes since v13:
- Remove legacy scripts.
Changes since v12:
- No changes, wrong patchset.
Changes since v11:
- Rebase on next.
- Init environment in vm_smoke_test.
Changes since v10:
- Rebase on next.
- Add DISTRO_APT_PREMIRRORS usage.
- Remove avocado deb package install.
- Set default runner.
Changes since v9:
- Rebase on next.
- Fix avocado artifacts path.
Changes since v8:
- Remove duplication in logs.
- Disable sysinfo plugin.
- Show full errors in default mode.
Changes since v7:
- Remove code duplication.
- Fix build_dir parsing.
- Add missed protect files backup logic.
Changes since v6:
- Improve environment parsing.
- Protect files backup logic.
- Rebuild full log output option.
- Protect ubuntu target with KFAIL.
Changes since v5:
- Rebase on current next.
Changes since v4:
- Add container tests.
- Implement testcase skipping logic.
- Separate and protect Bullseye tests.
- Add checks for container dependencies.
- Add Ilbers repo key.
- Improve error handling.
- Fix Ubuntu qemu image name.
Changes since v3:
- Add automatic bitbake init.
- Fix line lengths.
- Fix build without repro.
- Fix build artifact paths.
- Prepare for Gitlab build.
- Change debug settings.
- Move log splitting to start_vm.
- Add deb package install.
Changes since v2:
- Make CI call Avocado tests.
- Wait less on vm_start test.
- Split Roman's vm_start commit.
- Put logs to build folder.
- Rearrange patches.
- Split build test to classes.
- Add test case tagging.
- Get rid of avocado-framework-plugin-varianter-yaml-to-mux.
- Get rid of python-subprocess32.
- Improve logging.
Changes since v1:
- Merge start_vm rebuild patches.
- Fix patch comments.
Anton Mikanovich (13):
start_vm.py: Fix ubuntu image name
vm_boot_test: Fix log file path in vm_boot_test
vm_boot_test: Remove external varianter
vm_boot_test: Improve QEMU images checking
build_test: Refactoring build tests cases
testsuite: Add Python generations for testsuite in gitignore
testsuite: Fix test suite prepare guide
gitlab-ci: Add Avocado build artifacts
gitlab-ci: Add debug flag
vm_boot_test: Add automatic bitbake init
build_test: Protect ubuntu target with KFAIL
ci_build: Migrate to Avocado
vm_smoke_test: Migrate to Avocado
Roman Pletnev (3):
start_vm.py: Fix target name handling
start_vm.py: Add output and PID file vm_start.py options
start_vm.py: Add MIPS support
.gitlab-ci.yml | 5 +-
scripts/ci_build.sh | 208 ++++---------------------------
scripts/vm_smoke_test | 120 +++++-------------
testsuite/.gitignore | 1 +
testsuite/README.md | 21 +---
testsuite/build_test/build_test.py | 219 ++++++++++++++++++++++++++++++---
testsuite/build_test/cibase.py | 116 +++++++++++++++++
testsuite/build_test/cibuilder.py | 132 ++++++++++++++++++++
testsuite/build_test/run.sh | 3 -
testsuite/build_test/run_fast.sh | 3 +
testsuite/build_test/run_full.sh | 3 +
testsuite/build_test/variant.yaml | 22 ----
testsuite/start_vm.py | 27 ++--
testsuite/vm_boot_test/run.sh | 3 -
testsuite/vm_boot_test/run_fast.sh | 3 +
testsuite/vm_boot_test/run_full.sh | 3 +
testsuite/vm_boot_test/variant.yaml | 22 ----
testsuite/vm_boot_test/vm_boot_test.py | 121 +++++++++++++++---
18 files changed, 641 insertions(+), 391 deletions(-)
create mode 100644 testsuite/.gitignore
create mode 100644 testsuite/build_test/cibase.py
create mode 100644 testsuite/build_test/cibuilder.py
delete mode 100755 testsuite/build_test/run.sh
create mode 100755 testsuite/build_test/run_fast.sh
create mode 100755 testsuite/build_test/run_full.sh
delete mode 100644 testsuite/build_test/variant.yaml
delete mode 100755 testsuite/vm_boot_test/run.sh
create mode 100755 testsuite/vm_boot_test/run_fast.sh
create mode 100755 testsuite/vm_boot_test/run_full.sh
delete mode 100644 testsuite/vm_boot_test/variant.yaml
--
2.7.4
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v14 01/16] start_vm.py: Fix target name handling
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
@ 2021-09-30 16:45 ` Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 02/16] start_vm.py: Add output and PID file vm_start.py options Anton Mikanovich
` (15 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw)
To: isar-users; +Cc: Roman Pletnev, Anton Mikanovich
From: Roman Pletnev <rpletnev@ilbers.de>
This patch fix bb_output search (issue with wrong arch name being
selected for qemu) which caused qemu-system-riscv64 to be used for all
the targets:
>ERROR: [Errno 2] No such file or directory: u'qemu-system-riscv64' (1.05 s)
Signed-off-by: Roman Pletnev <rpletnev@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/start_vm.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index a3e32ac..c10db67 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -17,7 +17,7 @@ def get_bitbake_env(arch, distro):
def get_bitbake_var(output, var):
ret = ''
for line in output.splitlines():
- if line.startswith(var):
+ if line.startswith(var + '='):
ret = line.split('"')[1]
return ret
--
2.7.4
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v14 02/16] start_vm.py: Add output and PID file vm_start.py options
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 01/16] start_vm.py: Fix target name handling Anton Mikanovich
@ 2021-09-30 16:45 ` Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 03/16] start_vm.py: Add MIPS support Anton Mikanovich
` (14 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw)
To: isar-users; +Cc: Roman Pletnev, Anton Mikanovich
From: Roman Pletnev <rpletnev@ilbers.de>
This patch adds options -o (output file) and -p (pid file).
Implement virtual machine log output both to the file and stdout.
Signed-off-by: Roman Pletnev <rpletnev@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/start_vm.py | 18 ++++++++++++++----
testsuite/vm_boot_test/vm_boot_test.py | 5 ++---
2 files changed, 16 insertions(+), 7 deletions(-)
diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index c10db67..3736e51 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -21,7 +21,7 @@ def get_bitbake_var(output, var):
ret = line.split('"')[1]
return ret
-def format_qemu_cmdline(arch, build, distro):
+def format_qemu_cmdline(arch, build, distro, out, pid):
bb_output = get_bitbake_env(arch, distro).decode()
rootfs_image = ''
@@ -57,6 +57,13 @@ def format_qemu_cmdline(arch, build, distro):
qemu_cpu = get_bitbake_var(bb_output, 'QEMU_CPU')
qemu_disk_args = get_bitbake_var(bb_output, 'QEMU_DISK_ARGS')
+ if out:
+ extra_args.extend(['-chardev','stdio,id=ch0,logfile=' + out])
+ extra_args.extend(['-serial','chardev:ch0'])
+ extra_args.extend(['-monitor','none'])
+ if pid:
+ extra_args.extend(['-pidfile', pid])
+
qemu_disk_args = qemu_disk_args.replace('##ROOTFS_IMAGE##', deploy_dir_image + '/' + rootfs_image).split()
cmd = ['qemu-system-' + qemu_arch, '-m', '1024M']
@@ -72,10 +79,11 @@ def format_qemu_cmdline(arch, build, distro):
return cmd
-def start_qemu(arch, build, distro):
- cmdline = format_qemu_cmdline(arch, build, distro)
+def start_qemu(arch, build, distro, out, pid):
+ cmdline = format_qemu_cmdline(arch, build, distro, out, pid)
cmdline.insert(1, '-nographic')
+ print(cmdline)
p1 = subprocess.call(cmdline)
if __name__ == "__main__":
@@ -83,6 +91,8 @@ if __name__ == "__main__":
parser.add_argument('-a', '--arch', choices=['arm', 'arm64', 'amd64', 'i386'], help='set isar machine architecture.', default='arm')
parser.add_argument('-b', '--build', help='set path to build directory.', default=os.getcwd())
parser.add_argument('-d', '--distro', choices=['jessie', 'stretch'], help='set isar Debian distribution.', default='stretch')
+ parser.add_argument('-o', '--out', help='Route QEMU console output to specified file.')
+ parser.add_argument('-p', '--pid', help='Store QEMU pid to specified file.')
args = parser.parse_args()
- start_qemu(args.arch, args.build, args.distro)
+ start_qemu(args.arch, args.build, args.distro, args.out, args.pid)
diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index d4849c7..96e02f7 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -30,10 +30,9 @@ class VmBootTest(Test):
if os.path.exists(output_file):
os.remove(output_file)
- cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro)
+ cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro,
+ output_file, None)
cmdline.insert(1, '-nographic')
- cmdline.append('-serial')
- cmdline.append('file:' + output_file)
self.log.info('QEMU boot line: ' + str(cmdline))
--
2.7.4
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v14 03/16] start_vm.py: Add MIPS support
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 01/16] start_vm.py: Fix target name handling Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 02/16] start_vm.py: Add output and PID file vm_start.py options Anton Mikanovich
@ 2021-09-30 16:45 ` Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 04/16] start_vm.py: Fix ubuntu image name Anton Mikanovich
` (13 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw)
To: isar-users; +Cc: Roman Pletnev, Anton Mikanovich
From: Roman Pletnev <rpletnev@ilbers.de>
This patch adds mipsel to the list of supported architectures.
Signed-off-by: Roman Pletnev <rpletnev@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/start_vm.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index 3736e51..face747 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -88,7 +88,7 @@ def start_qemu(arch, build, distro, out, pid):
if __name__ == "__main__":
parser = argparse.ArgumentParser()
- parser.add_argument('-a', '--arch', choices=['arm', 'arm64', 'amd64', 'i386'], help='set isar machine architecture.', default='arm')
+ parser.add_argument('-a', '--arch', choices=['arm', 'arm64', 'amd64', 'i386', 'mipsel'], help='set isar machine architecture.', default='arm')
parser.add_argument('-b', '--build', help='set path to build directory.', default=os.getcwd())
parser.add_argument('-d', '--distro', choices=['jessie', 'stretch'], help='set isar Debian distribution.', default='stretch')
parser.add_argument('-o', '--out', help='Route QEMU console output to specified file.')
--
2.7.4
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v14 04/16] start_vm.py: Fix ubuntu image name
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
` (2 preceding siblings ...)
2021-09-30 16:45 ` [PATCH v14 03/16] start_vm.py: Add MIPS support Anton Mikanovich
@ 2021-09-30 16:45 ` Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 05/16] vm_boot_test: Fix log file path in vm_boot_test Anton Mikanovich
` (12 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
From: Anton Mikanovich <amikan@ilbers.de>
Choose 'ubuntu' base name for 'focal' target, not 'debian'.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/start_vm.py | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index face747..8f0ccd6 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -30,8 +30,9 @@ def format_qemu_cmdline(arch, build, distro, out, pid):
image_type = get_bitbake_var(bb_output, 'IMAGE_TYPE')
deploy_dir_image = get_bitbake_var(bb_output, 'DEPLOY_DIR_IMAGE')
+ base = 'ubuntu' if distro in ['focal', 'bionic'] else 'debian'
if image_type == 'ext4-img':
- rootfs_image = 'isar-image-base-debian-' + distro + '-qemu' + arch + '.ext4.img'
+ rootfs_image = 'isar-image-base-' + base + '-' + distro + '-qemu' + arch + '.ext4.img'
kernel_image = deploy_dir_image + '/' + get_bitbake_var(bb_output, 'KERNEL_IMAGE')
initrd_image = get_bitbake_var(bb_output, 'INITRD_IMAGE')
@@ -47,7 +48,7 @@ def format_qemu_cmdline(arch, build, distro, out, pid):
extra_args = ['-kernel', kernel_image, '-initrd', initrd_image]
extra_args.extend(kargs)
elif image_type == 'wic-img':
- rootfs_image = 'isar-image-base-debian-' + distro + '-qemu' + arch + '.wic.img'
+ rootfs_image = 'isar-image-base-' + base + '-' + distro + '-qemu' + arch + '.wic.img'
extra_args = ['-snapshot']
else:
raise ValueError('Invalid image type: ' + str(image_type))
--
2.7.4
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v14 05/16] vm_boot_test: Fix log file path in vm_boot_test
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
` (3 preceding siblings ...)
2021-09-30 16:45 ` [PATCH v14 04/16] start_vm.py: Fix ubuntu image name Anton Mikanovich
@ 2021-09-30 16:45 ` Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 06/16] vm_boot_test: Remove external varianter Anton Mikanovich
` (11 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
From: Anton Mikanovich <amikan@ilbers.de>
Make log filename random and store it in build folder.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/vm_boot_test/vm_boot_test.py | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index 96e02f7..64dfda6 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -4,6 +4,7 @@ import os
import subprocess32
import sys
import time
+import tempfile
from os.path import dirname
sys.path.append(dirname(__file__) + '/..')
@@ -26,9 +27,9 @@ class VmBootTest(Test):
self.log.info('Isar build folder is: ' + build_dir)
self.log.info('===================================================')
- output_file = '/tmp/vm_boot_test.log'
- if os.path.exists(output_file):
- os.remove(output_file)
+ fd, output_file = tempfile.mkstemp(suffix='_log.txt',
+ prefix='vm_start_' + distro + '_' +
+ arch + '_', dir=build_dir, text=True)
cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro,
output_file, None)
--
2.7.4
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v14 06/16] vm_boot_test: Remove external varianter
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
` (4 preceding siblings ...)
2021-09-30 16:45 ` [PATCH v14 05/16] vm_boot_test: Fix log file path in vm_boot_test Anton Mikanovich
@ 2021-09-30 16:45 ` Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 07/16] vm_boot_test: Improve QEMU images checking Anton Mikanovich
` (10 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
From: Anton Mikanovich <amikan@ilbers.de>
To get rid of avocado-framework-plugin-varianter-yaml-to-mux pip
dependency we should not use yaml-to-mux parameters.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/vm_boot_test/run.sh | 3 ---
testsuite/vm_boot_test/run_fast.sh | 3 +++
testsuite/vm_boot_test/run_full.sh | 3 +++
testsuite/vm_boot_test/variant.yaml | 22 -----------------
testsuite/vm_boot_test/vm_boot_test.py | 45 ++++++++++++++++++++++++++++++----
5 files changed, 46 insertions(+), 30 deletions(-)
delete mode 100755 testsuite/vm_boot_test/run.sh
create mode 100755 testsuite/vm_boot_test/run_fast.sh
create mode 100755 testsuite/vm_boot_test/run_full.sh
delete mode 100644 testsuite/vm_boot_test/variant.yaml
diff --git a/testsuite/vm_boot_test/run.sh b/testsuite/vm_boot_test/run.sh
deleted file mode 100755
index 9fdda95..0000000
--- a/testsuite/vm_boot_test/run.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/bash
-
-avocado run vm_boot_test.py --mux-yaml test:variant.yaml --mux-inject build_dir:$BUILDDIR time_to_wait:300
diff --git a/testsuite/vm_boot_test/run_fast.sh b/testsuite/vm_boot_test/run_fast.sh
new file mode 100755
index 0000000..0fc77b3
--- /dev/null
+++ b/testsuite/vm_boot_test/run_fast.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+avocado run vm_boot_test.py -t fast -p build_dir="$BUILDDIR" -p time_to_wait=300
diff --git a/testsuite/vm_boot_test/run_full.sh b/testsuite/vm_boot_test/run_full.sh
new file mode 100755
index 0000000..a561a36
--- /dev/null
+++ b/testsuite/vm_boot_test/run_full.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+avocado run vm_boot_test.py -t full -p build_dir="$BUILDDIR" -p time_to_wait=300
diff --git a/testsuite/vm_boot_test/variant.yaml b/testsuite/vm_boot_test/variant.yaml
deleted file mode 100644
index 9ddc634..0000000
--- a/testsuite/vm_boot_test/variant.yaml
+++ /dev/null
@@ -1,22 +0,0 @@
-variants: !mux
- stretch-amd64:
- distro: "stretch"
- arch: "amd64"
- stretch-i386:
- distro: "stretch"
- arch: "i386"
- stretch-arm:
- distro: "stretch"
- arch: "arm"
- stretch-arm64:
- distro: "stretch"
- arch: "arm64"
- buster-amd64:
- distro: "buster"
- arch: "amd64"
- buster-i386:
- distro: "buster"
- arch: "i386"
- buster-arm:
- distro: "buster"
- arch: "arm"
diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index 64dfda6..eee14b2 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -13,13 +13,10 @@ import start_vm
from avocado import Test
-class VmBootTest(Test):
+class VmBase(Test):
- def test(self):
- # TODO: add default values
+ def vm_start(self, arch='amd64', distro='buster'):
build_dir = self.params.get('build_dir', default='.')
- arch = self.params.get('arch', default='arm')
- distro = self.params.get('distro', default='stretch')
time_to_wait = self.params.get('time_to_wait', default=60)
self.log.info('===================================================')
@@ -49,3 +46,41 @@ class VmBootTest(Test):
return
self.fail('Test failed')
+
+class VmBootTestFast(VmBase):
+
+ """
+ Test QEMU image start (fast)
+
+ :avocado: tags=fast,full
+ """
+ def test_arm_stretch(self):
+ self.vm_start('arm','stretch')
+
+ def test_arm_buster(self):
+ self.vm_start('arm','buster')
+
+ def test_arm64_stretch(self):
+ self.vm_start('arm64','stretch')
+
+ def test_amd64_stretch(self):
+ self.vm_start('amd64','stretch')
+
+class VmBootTestFull(VmBase):
+
+ """
+ Test QEMU image start (full)
+
+ :avocado: tags=full
+ """
+ def test_i386_stretch(self):
+ self.vm_start('i386','stretch')
+
+ def test_i386_buster(self):
+ self.vm_start('i386','buster')
+
+ def test_amd64_buster(self):
+ self.vm_start('amd64','buster')
+
+ def test_amd64_focal(self):
+ self.vm_start('amd64','focal')
--
2.7.4
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v14 07/16] vm_boot_test: Improve QEMU images checking
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
` (5 preceding siblings ...)
2021-09-30 16:45 ` [PATCH v14 06/16] vm_boot_test: Remove external varianter Anton Mikanovich
@ 2021-09-30 16:45 ` Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 08/16] build_test: Refactoring build tests cases Anton Mikanovich
` (9 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
From: Anton Mikanovich <amikan@ilbers.de>
Makes QEMU start test to analyze boot log in real-time. It helps test
cases to finish as soon as booting succeeds and do not wasting time on
waiting.
Get rid of python-subprocess32 backport package.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/vm_boot_test/vm_boot_test.py | 53 +++++++++++++++++++++++++++-------
1 file changed, 42 insertions(+), 11 deletions(-)
diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index eee14b2..6e580d0 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -1,7 +1,8 @@
#!/usr/bin/env python3
import os
-import subprocess32
+import select
+import subprocess
import sys
import time
import tempfile
@@ -13,6 +14,9 @@ import start_vm
from avocado import Test
+class CanBeFinished(Exception):
+ pass
+
class VmBase(Test):
def vm_start(self, arch='amd64', distro='buster'):
@@ -34,18 +38,45 @@ class VmBase(Test):
self.log.info('QEMU boot line: ' + str(cmdline))
- devnull = open(os.devnull, 'w')
-
- p1 = subprocess32.Popen(cmdline, stdout=devnull, stderr=devnull)
- time.sleep(int(time_to_wait))
- p1.kill()
- p1.wait()
+ login_prompt = b'isar login:'
+ service_prompt = b'Just an example'
+
+ timeout = time.time() + int(time_to_wait)
+
+ p1 = subprocess.Popen(cmdline, stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE)
+ try:
+ poller = select.poll()
+ poller.register(p1.stdout, select.POLLIN)
+ poller.register(p1.stderr, select.POLLIN)
+ while time.time() < timeout and p1.poll() is None:
+ events = poller.poll(1000 * (timeout - time.time()))
+ for fd, event in events:
+ if fd == p1.stdout.fileno():
+ # Wait for the complete string if it is read in chunks
+ # like "i", "sar", " login:"
+ time.sleep(0.01)
+ data = os.read(fd, 1024)
+ if login_prompt in data:
+ raise CanBeFinished
+ if fd == p1.stderr.fileno():
+ self.log.error(p1.stderr.readline())
+ except CanBeFinished:
+ self.log.debug('Got login prompt')
+ finally:
+ if p1.poll() is None:
+ p1.kill()
+ p1.wait()
if os.path.exists(output_file):
- if 'isar login:' in open(output_file).read():
- return
-
- self.fail('Test failed')
+ with open(output_file, "rb") as f1:
+ data = f1.read()
+ if service_prompt in data and login_prompt in data:
+ return
+ else:
+ self.log.error(data)
+
+ self.fail('Log ' + output_file)
class VmBootTestFast(VmBase):
--
2.7.4
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v14 08/16] build_test: Refactoring build tests cases
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
` (6 preceding siblings ...)
2021-09-30 16:45 ` [PATCH v14 07/16] vm_boot_test: Improve QEMU images checking Anton Mikanovich
@ 2021-09-30 16:45 ` Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 09/16] testsuite: Add Python generations for testsuite in gitignore Anton Mikanovich
` (8 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
From: Anton Mikanovich <amikan@ilbers.de>
Split build test into classes and separate cases. Implement test cases
tagging for external configuring. Also remove the usage of external
varianter plugin.
In this step, all ci_build test cases were copied as-is without
functional changes.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/build_test/build_test.py | 210 +++++++++++++++++++++++++++++++++----
testsuite/build_test/cibase.py | 116 ++++++++++++++++++++
testsuite/build_test/cibuilder.py | 132 +++++++++++++++++++++++
testsuite/build_test/run.sh | 3 -
testsuite/build_test/run_fast.sh | 3 +
testsuite/build_test/run_full.sh | 3 +
testsuite/build_test/variant.yaml | 22 ----
7 files changed, 444 insertions(+), 45 deletions(-)
create mode 100644 testsuite/build_test/cibase.py
create mode 100644 testsuite/build_test/cibuilder.py
delete mode 100755 testsuite/build_test/run.sh
create mode 100755 testsuite/build_test/run_fast.sh
create mode 100755 testsuite/build_test/run_full.sh
delete mode 100644 testsuite/build_test/variant.yaml
diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
index 7a55c2f..4aca56e 100644
--- a/testsuite/build_test/build_test.py
+++ b/testsuite/build_test/build_test.py
@@ -1,29 +1,199 @@
#!/usr/bin/env python3
import os
-import subprocess32
-import sys
-from os.path import dirname
-from avocado import Test
+from avocado import skipUnless
+from avocado.utils import path
+from cibase import CIBaseTest
-class BuildTest(Test):
+UMOCI_AVAILABLE = True
+SKOPEO_AVAILABLE = True
+try:
+ path.find_command('umoci')
+except path.CmdNotFoundError:
+ UMOCI_AVAILABLE = False
+try:
+ path.find_command('skopeo')
+except path.CmdNotFoundError:
+ SKOPEO_AVAILABLE = False
- def test(self):
- # TODO: add default values
- build_dir = self.params.get('build_dir', default='.')
- arch = self.params.get('arch', default='arm')
- distro = self.params.get('distro', default='stretch')
+class ReproTest(CIBaseTest):
- self.log.info('===================================================')
- self.log.info('Running Isar build test for (' + distro + '-' + arch + ')')
- self.log.info('Isar build folder is: ' + build_dir)
- self.log.info('===================================================')
+ """
+ Test cached base repository
- #isar_root = dirname(__file__) + '/..'
- os.chdir(build_dir)
- cmdline = ['bitbake', 'mc:qemu' + arch + '-' + distro + ':isar-image-base']
- p1 = subprocess32.run(cmdline)
+ :avocado: tags=repro,fast,full
+ """
+ def test_repro_signed(self):
+ targets = [
+ 'mc:de0-nano-soc-buster:isar-image-base',
+ 'mc:qemuarm64-stretch:isar-image-base'
+ ]
- if p1.returncode:
- self.fail('Test failed')
+ self.perform_repro_test(targets, 1)
+
+ def test_repro_unsigned(self):
+ targets = [
+ 'mc:qemuamd64-stretch:isar-image-base',
+ 'mc:qemuarm-stretch:isar-image-base'
+ ]
+
+ self.perform_repro_test(targets, 0)
+
+class CrossTest(CIBaseTest):
+
+ """
+ Start cross build for the defined set of configurations
+
+ :avocado: tags=cross,fast,full
+ """
+ def test_cross(self):
+ targets = [
+ 'mc:qemuarm-stretch:isar-image-base',
+ 'mc:qemuarm-buster:isar-image-base',
+ 'mc:qemuarm64-stretch:isar-image-base',
+ 'mc:qemuamd64-stretch:isar-image-base',
+ 'mc:de0-nano-soc-buster:isar-image-base',
+ 'mc:stm32mp15x-buster:isar-image-base',
+ 'mc:rpi-stretch:isar-image-base',
+ 'mc:qemuarm64-focal:isar-image-base'
+ ]
+
+ self.perform_build_test(targets, 1, None)
+
+ def test_cross_bullseye(self):
+ targets = [
+ 'mc:qemuarm-bullseye:isar-image-base'
+ ]
+
+ try:
+ self.perform_build_test(targets, 1, None)
+ except:
+ self.cancel('KFAIL')
+
+class SdkTest(CIBaseTest):
+
+ """
+ In addition test SDK creation
+
+ :avocado: tags=sdk,fast,full
+ """
+ def test_sdk(self):
+ targets = ['mc:qemuarm-stretch:isar-image-base']
+
+ self.perform_build_test(targets, 1, 'do_populate_sdk')
+
+class NoCrossTest(CIBaseTest):
+
+ """
+ Start non-cross build for the defined set of configurations
+
+ :avocado: tags=nocross,full
+ """
+ def test_nocross(self):
+ targets = [
+ 'mc:qemuarm-stretch:isar-image-base',
+ 'mc:qemuarm-buster:isar-image-base',
+ 'mc:qemuarm64-stretch:isar-image-base',
+ 'mc:qemui386-stretch:isar-image-base',
+ 'mc:qemui386-buster:isar-image-base',
+ 'mc:qemuamd64-stretch:isar-image-base',
+ 'mc:qemuamd64-buster:isar-image-base',
+ 'mc:qemuamd64-buster-tgz:isar-image-base',
+ 'mc:qemuamd64-buster-cpiogz:isar-image-base',
+ 'mc:qemuamd64-buster:isar-initramfs',
+ 'mc:qemumipsel-stretch:isar-image-base',
+ 'mc:qemumipsel-buster:isar-image-base',
+ 'mc:nand-ubi-demo-buster:isar-image-ubi',
+ 'mc:rpi-stretch:isar-image-base',
+ 'mc:qemuamd64-focal:isar-image-base',
+ 'mc:virtualbox-ova-buster:isar-image-base'
+ ]
+
+ # Cleanup after cross build
+ self.deletetmp(self.params.get('build_dir',
+ default=os.path.dirname(__file__) + '/../../build'))
+
+ self.perform_build_test(targets, 0, None)
+
+ def test_nocross_bullseye(self):
+ targets = [
+ 'mc:qemuamd64-bullseye:isar-image-base',
+ 'mc:qemuarm-bullseye:isar-image-base',
+ 'mc:qemui386-bullseye:isar-image-base',
+ 'mc:qemumipsel-bullseye:isar-image-base'
+ ]
+
+ try:
+ self.perform_build_test(targets, 0, None)
+ except:
+ self.cancel('KFAIL')
+
+class RebuildTest(CIBaseTest):
+
+ """
+ Test image rebuild
+
+ :avocado: tags=rebuild,fast,full
+ """
+ def test_rebuild(self):
+ is_cross_build = int(self.params.get('cross', default=0))
+
+ layerdir_core = self.getlayerdir('core')
+
+ dpkgbase_file = layerdir_core + '/classes/dpkg-base.bbclass'
+
+ self.backupfile(dpkgbase_file)
+ with open(dpkgbase_file, 'a') as file:
+ file.write('do_fetch_append() {\n\n}')
+
+ try:
+ self.perform_build_test('mc:qemuamd64-stretch:isar-image-base',
+ is_cross_build, None)
+ finally:
+ self.restorefile(dpkgbase_file)
+
+class WicTest(CIBaseTest):
+
+ """
+ Test wic --exclude-path
+
+ :avocado: tags=wic,fast,full
+ """
+ def test_wic_exclude(self):
+ # TODO: remove hardcoded filenames
+ wks_path = '/scripts/lib/wic/canned-wks/sdimage-efi.wks'
+ wic_path = '/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img'
+
+ self.perform_wic_test('mc:qemuamd64-stretch:isar-image-base',
+ wks_path, wic_path)
+
+class ContainerImageTest(CIBaseTest):
+
+ """
+ Test containerized images creation
+
+ :avocado: tags=containerbuild,fast,full,container
+ """
+ @skipUnless(UMOCI_AVAILABLE and SKOPEO_AVAILABLE, 'umoci/skopeo not found')
+ def test_nocross(self):
+ targets = [
+ 'mc:container-amd64-stretch:isar-image-base',
+ 'mc:container-amd64-buster:isar-image-base',
+ 'mc:container-amd64-bullseye:isar-image-base'
+ ]
+
+ self.perform_container_test(targets, None)
+
+class ContainerSdkTest(CIBaseTest):
+
+ """
+ Test SDK container image creation
+
+ :avocado: tags=containersdk,fast,full,container
+ """
+ @skipUnless(UMOCI_AVAILABLE and SKOPEO_AVAILABLE, 'umoci/skopeo not found')
+ def test_container_sdk(self):
+ targets = ['mc:container-amd64-stretch:isar-image-base']
+
+ self.perform_container_test(targets, 'do_populate_sdk')
diff --git a/testsuite/build_test/cibase.py b/testsuite/build_test/cibase.py
new file mode 100644
index 0000000..78d7bdb
--- /dev/null
+++ b/testsuite/build_test/cibase.py
@@ -0,0 +1,116 @@
+#!/usr/bin/env python3
+
+import os
+import re
+import tempfile
+
+from cibuilder import CIBuilder
+from avocado.utils import process
+
+isar_root = os.path.dirname(__file__) + '/../..'
+
+class CIBaseTest(CIBuilder):
+
+ def prep(self, testname, targets, cross, debsrc_cache):
+ build_dir = self.params.get('build_dir', default=isar_root + '/build')
+ build_dir = os.path.realpath(build_dir)
+ quiet = int(self.params.get('quiet', default=0))
+ bitbake_args = '-v'
+
+ if quiet:
+ bitbake_args = ''
+
+ self.log.info('===================================================')
+ self.log.info('Running ' + testname + ' test for:')
+ self.log.info(targets)
+ self.log.info('Isar build folder is: ' + build_dir)
+ self.log.info('===================================================')
+
+ self.init(build_dir)
+ self.confprepare(build_dir, 1, cross, debsrc_cache)
+
+ return build_dir, bitbake_args;
+
+ def perform_build_test(self, targets, cross, bitbake_cmd):
+ build_dir, bb_args = self.prep('Isar build', targets, cross, 1)
+
+ self.log.info('Starting build...')
+
+ self.bitbake(build_dir, targets, bitbake_cmd, bb_args)
+
+ def perform_repro_test(self, targets, signed):
+ cross = int(self.params.get('cross', default=0))
+ build_dir, bb_args = self.prep('repro Isar build', targets, cross, 0)
+
+ gpg_pub_key = os.path.dirname(__file__) + '/../base-apt/test_pub.key'
+ gpg_priv_key = os.path.dirname(__file__) + '/../base-apt/test_priv.key'
+
+ if signed:
+ with open(build_dir + '/conf/ci_build.conf', 'a') as file:
+ # Enable use of signed cached base repository
+ file.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
+
+ os.chdir(build_dir)
+
+ os.environ['GNUPGHOME'] = tempfile.mkdtemp()
+ result = process.run('gpg --import %s %s' % (gpg_pub_key, gpg_priv_key))
+
+ if result.exit_status:
+ self.fail('GPG import failed')
+
+ self.bitbake(build_dir, targets, None, bb_args)
+
+ self.deletetmp(build_dir)
+ with open(build_dir + '/conf/ci_build.conf', 'a') as file:
+ file.write('ISAR_USE_CACHED_BASE_REPO = "1"\n')
+ file.write('BB_NO_NETWORK = "1"\n')
+
+ self.bitbake(build_dir, targets, None, bb_args)
+
+ # Disable use of cached base repository
+ self.confcleanup(build_dir)
+
+ if not signed:
+ # Try to build with changed configuration with no cleanup
+ self.bitbake(build_dir, targets, None, bb_args)
+
+ # Cleanup
+ self.deletetmp(build_dir)
+
+ def perform_wic_test(self, targets, wks_path, wic_path):
+ cross = int(self.params.get('cross', default=0))
+ build_dir, bb_args = self.prep('WIC exclude build', targets, cross, 1)
+
+ layerdir_isar = self.getlayerdir('isar')
+
+ wks_file = layerdir_isar + wks_path
+ wic_img = build_dir + wic_path
+
+ if not os.path.isfile(wic_img):
+ self.fail('No build started before: ' + wic_img + ' not exist')
+
+ self.backupfile(wks_file)
+ self.backupmove(wic_img)
+
+ with open(wks_file, 'r') as file:
+ lines = file.readlines()
+ with open(wks_file, 'w') as file:
+ for line in lines:
+ file.write(re.sub(r'part \/ ', 'part \/ --exclude-path usr ',
+ line))
+
+ try:
+ self.bitbake(build_dir, targets, None, bb_args)
+ finally:
+ self.restorefile(wks_file)
+
+ self.restorefile(wic_img)
+
+ def perform_container_test(self, targets, bitbake_cmd):
+ cross = int(self.params.get('cross', default=0))
+ build_dir, bb_args = self.prep('Isar Container', targets, cross, 1)
+
+ self.containerprep(build_dir)
+
+ self.bitbake(build_dir, targets, bitbake_cmd, bb_args)
+
diff --git a/testsuite/build_test/cibuilder.py b/testsuite/build_test/cibuilder.py
new file mode 100644
index 0000000..baa7185
--- /dev/null
+++ b/testsuite/build_test/cibuilder.py
@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+
+import logging
+import os
+import re
+import select
+import shutil
+import subprocess
+
+from avocado import Test
+from avocado.utils import path
+from avocado.utils import process
+
+isar_root = os.path.dirname(__file__) + '/../..'
+backup_prefix = '.ci-backup'
+
+app_log = logging.getLogger("avocado.app")
+
+class CIBuilder(Test):
+ def setUp(self):
+ super(CIBuilder, self).setUp()
+ job_log = os.path.join(os.path.dirname(self.logdir), '..', 'job.log')
+ self._file_handler = logging.FileHandler(filename=job_log)
+ self._file_handler.setLevel(logging.ERROR)
+ fmt = ('%(asctime)s %(module)-16.16s L%(lineno)-.4d %('
+ 'levelname)-5.5s| %(message)s')
+ formatter = logging.Formatter(fmt=fmt)
+ self._file_handler.setFormatter(formatter)
+ app_log.addHandler(self._file_handler)
+
+ def init(self, build_dir):
+ os.chdir(isar_root)
+ path.usable_rw_dir(build_dir)
+ output = process.getoutput('/bin/bash -c "source isar-init-build-env \
+ %s 2>&1 >/dev/null; env"' % build_dir)
+ env = dict(((x.split('=', 1) + [''])[:2] \
+ for x in output.splitlines() if x != ''))
+ os.environ.update(env)
+
+ def confprepare(self, build_dir, compat_arch, cross, debsrc_cache):
+ with open(build_dir + '/conf/ci_build.conf', 'w') as f:
+ if compat_arch:
+ f.write('ISAR_ENABLE_COMPAT_ARCH_amd64 = "1"\n')
+ f.write('ISAR_ENABLE_COMPAT_ARCH_arm64 = "1"\n')
+ f.write('ISAR_ENABLE_COMPAT_ARCH_debian-stretch_amd64 = "0"\n')
+ f.write('IMAGE_INSTALL += "kselftest"\n')
+ if cross:
+ f.write('ISAR_CROSS_COMPILE = "1"\n')
+ if debsrc_cache:
+ f.write('BASE_REPO_FEATURES = "cache-deb-src"\n')
+ distro_apt_premir = os.getenv('DISTRO_APT_PREMIRRORS')
+ if distro_apt_premir:
+ f.write('DISTRO_APT_PREMIRRORS = "%s"\n' % distro_apt_premir)
+
+ with open(build_dir + '/conf/local.conf', 'r+') as f:
+ for line in f:
+ if 'include ci_build.conf' in line:
+ break
+ else:
+ f.write('\ninclude ci_build.conf')
+
+ def containerprep(self, build_dir):
+ with open(build_dir + '/conf/ci_build.conf', 'a') as f:
+ f.write('SDK_FORMATS = "docker-archive"\n')
+ f.write('IMAGE_INSTALL_remove = "example-module-${KERNEL_NAME} enable-fsck"\n')
+
+ def confcleanup(self, build_dir):
+ open(build_dir + '/conf/ci_build.conf', 'w').close()
+
+ def deletetmp(self, build_dir):
+ process.run('rm -rf ' + build_dir + '/tmp', sudo=True)
+
+ def bitbake(self, build_dir, target, cmd, args):
+ os.chdir(build_dir)
+ cmdline = ['bitbake']
+ if args:
+ cmdline.append(args)
+ if cmd:
+ cmdline.append('-c')
+ cmdline.append(cmd)
+ if isinstance(target, list):
+ cmdline.extend(target)
+ else:
+ cmdline.append(target)
+
+ with subprocess.Popen(" ".join(cmdline), stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE, universal_newlines=True,
+ shell=True) as p1:
+ poller = select.poll()
+ poller.register(p1.stdout, select.POLLIN)
+ poller.register(p1.stderr, select.POLLIN)
+ while p1.poll() is None:
+ events = poller.poll(1000)
+ for fd, event in events:
+ if fd == p1.stdout.fileno():
+ self.log.info(p1.stdout.readline().rstrip())
+ if fd == p1.stderr.fileno():
+ app_log.error(p1.stderr.readline().rstrip())
+ p1.wait()
+ if p1.returncode:
+ self.fail('Bitbake failed')
+
+ def backupfile(self, path):
+ try:
+ shutil.copy2(path, path + backup_prefix)
+ except FileNotFoundError:
+ self.log.warn(path + ' not exist')
+
+ def backupmove(self, path):
+ try:
+ shutil.move(path, path + backup_prefix)
+ except FileNotFoundError:
+ self.log.warn(path + ' not exist')
+
+ def restorefile(self, path):
+ try:
+ shutil.move(path + backup_prefix, path)
+ except FileNotFoundError:
+ self.log.warn(path + backup_prefix + ' not exist')
+
+ def getlayerdir(self, layer):
+ try:
+ path.find_command('bitbake')
+ except path.CmdNotFoundError:
+ build_dir = self.params.get('build_dir',
+ default=isar_root + '/build')
+ self.init(build_dir)
+ output = process.getoutput('bitbake -e | grep "^LAYERDIR_.*="')
+ env = dict(((x.split('=', 1) + [''])[:2] \
+ for x in output.splitlines() if x != ''))
+
+ return env['LAYERDIR_' + layer].strip('"')
diff --git a/testsuite/build_test/run.sh b/testsuite/build_test/run.sh
deleted file mode 100755
index a8ea9cc..0000000
--- a/testsuite/build_test/run.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/bash
-
-avocado run build_test.py --mux-yaml test:variant.yaml --mux-inject build_dir:$BUILDDIR
diff --git a/testsuite/build_test/run_fast.sh b/testsuite/build_test/run_fast.sh
new file mode 100755
index 0000000..4d1bf44
--- /dev/null
+++ b/testsuite/build_test/run_fast.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+avocado run build_test.py -t fast -p quiet=1 -p cross=1
diff --git a/testsuite/build_test/run_full.sh b/testsuite/build_test/run_full.sh
new file mode 100755
index 0000000..af5ec59
--- /dev/null
+++ b/testsuite/build_test/run_full.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+avocado run build_test.py -t full -p quiet=1
diff --git a/testsuite/build_test/variant.yaml b/testsuite/build_test/variant.yaml
deleted file mode 100644
index 9ddc634..0000000
--- a/testsuite/build_test/variant.yaml
+++ /dev/null
@@ -1,22 +0,0 @@
-variants: !mux
- stretch-amd64:
- distro: "stretch"
- arch: "amd64"
- stretch-i386:
- distro: "stretch"
- arch: "i386"
- stretch-arm:
- distro: "stretch"
- arch: "arm"
- stretch-arm64:
- distro: "stretch"
- arch: "arm64"
- buster-amd64:
- distro: "buster"
- arch: "amd64"
- buster-i386:
- distro: "buster"
- arch: "i386"
- buster-arm:
- distro: "buster"
- arch: "arm"
--
2.7.4
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v14 09/16] testsuite: Add Python generations for testsuite in gitignore
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
` (7 preceding siblings ...)
2021-09-30 16:45 ` [PATCH v14 08/16] build_test: Refactoring build tests cases Anton Mikanovich
@ 2021-09-30 16:45 ` Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 10/16] testsuite: Fix test suite prepare guide Anton Mikanovich
` (7 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
From: Anton Mikanovich <amikan@ilbers.de>
Ignore Python byte code auto generated files inside testsuite.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/.gitignore | 1 +
1 file changed, 1 insertion(+)
create mode 100644 testsuite/.gitignore
diff --git a/testsuite/.gitignore b/testsuite/.gitignore
new file mode 100644
index 0000000..0d20b64
--- /dev/null
+++ b/testsuite/.gitignore
@@ -0,0 +1 @@
+*.pyc
--
2.7.4
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v14 10/16] testsuite: Fix test suite prepare guide
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
` (8 preceding siblings ...)
2021-09-30 16:45 ` [PATCH v14 09/16] testsuite: Add Python generations for testsuite in gitignore Anton Mikanovich
@ 2021-09-30 16:45 ` Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 11/16] gitlab-ci: Add Avocado build artifacts Anton Mikanovich
` (6 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
From: Anton Mikanovich <amikan@ilbers.de>
Remove deprecated yaml-to-mux plugin and yml variants from the doc.
Also remove subprocess32 backport installation which is not used.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/README.md | 21 +++------------------
1 file changed, 3 insertions(+), 18 deletions(-)
diff --git a/testsuite/README.md b/testsuite/README.md
index 5e64223..fcbea1e 100644
--- a/testsuite/README.md
+++ b/testsuite/README.md
@@ -4,29 +4,14 @@ The framework could be installed by using standard HOWTO:
https://github.com/avocado-framework/avocado#installing-with-standard-python-tools
-Then you need to install varianter yaml-to-mux plugin by following these instructions:
-
- https://github.com/avocado-framework/avocado/tree/master/optional_plugins
-
-## For Debian 9.x
-
- $ sudo apt-get install python-pip
- $ pip install --user subprocess32
- $ pip install --user avocado-framework
- $ pip install --user avocado-framework-plugin-varianter-yaml-to-mux
-
-# Pre
-
- $ export PATH=$PATH:~/.local/bin
- $ cd isar
- $ source isar-init-build-env
+## For Debian (tested on Debian 10.x)
+ $ sudo dpkg -i avocado_91.0_all.deb
# Run test
Each testsuite directory contains:
- - run.sh - script to start tests
- - variants.yaml - set of input data
+ - run_*.sh - script to start tests
- *.py - test case
# Other
--
2.7.4
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v14 11/16] gitlab-ci: Add Avocado build artifacts
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
` (9 preceding siblings ...)
2021-09-30 16:45 ` [PATCH v14 10/16] testsuite: Fix test suite prepare guide Anton Mikanovich
@ 2021-09-30 16:45 ` Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 12/16] gitlab-ci: Add debug flag Anton Mikanovich
` (5 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
From: Anton Mikanovich <amikan@ilbers.de>
Save build logs in case of Avocado CI build.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
.gitlab-ci.yml | 1 +
1 file changed, 1 insertion(+)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index c91a997..0760161 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -15,6 +15,7 @@ variables:
name: "logs-$CI_JOB_ID"
paths:
- build/tmp/work/*/*/*/temp
+ - build/job-results
when: on_failure
expire_in: 1 week
--
2.7.4
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v14 12/16] gitlab-ci: Add debug flag
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
` (10 preceding siblings ...)
2021-09-30 16:45 ` [PATCH v14 11/16] gitlab-ci: Add Avocado build artifacts Anton Mikanovich
@ 2021-09-30 16:45 ` Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 13/16] vm_boot_test: Add automatic bitbake init Anton Mikanovich
` (4 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
From: Anton Mikanovich <amikan@ilbers.de>
Gitlab users expect full build log output witch was enabled by default.
Debug flag was ignored and never used, now it becomes functional.
To keed compatibility add '-d' flag to ci_build.sh call.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
.gitlab-ci.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 0760161..fad4210 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -24,7 +24,7 @@ fast-ci:
except:
- schedules
script:
- - scripts/ci_build.sh -q -f
+ - scripts/ci_build.sh -q -f -d
full-ci:
<<: *common-build
@@ -34,7 +34,7 @@ full-ci:
- PREVIOUS_SHA="$(cat .CI_COMMIT_SHA || true)"
- if [ "$CI_COMMIT_SHA" != "$PREVIOUS_SHA" ]; then
echo "$CI_COMMIT_SHA" > .CI_COMMIT_SHA;
- scripts/ci_build.sh -q -c -r;
+ scripts/ci_build.sh -q -c -r -d;
fi
cache:
key: "$CI_COMMIT_REF_SLUG"
--
2.7.4
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v14 13/16] vm_boot_test: Add automatic bitbake init
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
` (11 preceding siblings ...)
2021-09-30 16:45 ` [PATCH v14 12/16] gitlab-ci: Add debug flag Anton Mikanovich
@ 2021-09-30 16:45 ` Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 14/16] build_test: Protect ubuntu target with KFAIL Anton Mikanovich
` (3 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
From: Anton Mikanovich <amikan@ilbers.de>
Implement source of isar-init-build-env in case there is no bitbake
command found.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/vm_boot_test/vm_boot_test.py | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index 6e580d0..01623c5 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -13,12 +13,25 @@ sys.path.append(dirname(__file__) + '/..')
import start_vm
from avocado import Test
+from avocado.utils import process
+from avocado.utils import path
class CanBeFinished(Exception):
pass
class VmBase(Test):
+ def check_bitbake(self, build_dir):
+ try:
+ path.find_command('bitbake')
+ except path.CmdNotFoundError:
+ out = process.getoutput('/bin/bash -c "cd ../.. && \
+ source isar-init-build-env \
+ %s 2>&1 >/dev/null; env"' % build_dir)
+ env = dict(((x.split('=', 1) + [''])[:2] \
+ for x in output.splitlines() if x != ''))
+ os.environ.update(env)
+
def vm_start(self, arch='amd64', distro='buster'):
build_dir = self.params.get('build_dir', default='.')
time_to_wait = self.params.get('time_to_wait', default=60)
@@ -28,6 +41,8 @@ class VmBase(Test):
self.log.info('Isar build folder is: ' + build_dir)
self.log.info('===================================================')
+ self.check_bitbake(build_dir)
+
fd, output_file = tempfile.mkstemp(suffix='_log.txt',
prefix='vm_start_' + distro + '_' +
arch + '_', dir=build_dir, text=True)
--
2.7.4
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v14 14/16] build_test: Protect ubuntu target with KFAIL
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
` (12 preceding siblings ...)
2021-09-30 16:45 ` [PATCH v14 13/16] vm_boot_test: Add automatic bitbake init Anton Mikanovich
@ 2021-09-30 16:45 ` Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 15/16] ci_build: Migrate to Avocado Anton Mikanovich
` (2 subsequent siblings)
16 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
From: Anton Mikanovich <amikan@ilbers.de>
We experience the same QEMU-related semop issue on qemuarm64-focal:
| ERROR: mc:qemuarm64-focal:kselftest-5.6.13-r0 do_install_builddeps: Execution of '/build/tmp/work/ubuntu-focal-arm64/kselftest/5.6.13-r0/temp/run.do_install_builddeps.33809' failed with exit code 1:
| /bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
| hostname: Temporary failure in name resolution
| semop(1): encountered an error: Function not implemented
| Error in the build process: exit status 1
| dpkg: error: cannot access archive 'kselftest-build-deps_5.6.13_arm64.deb': No such file or directory
So we need to protect with KFAIL not only bullseye targets, but Ubuntu
Focal targets also.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/build_test/build_test.py | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
index 4aca56e..d39c10c 100644
--- a/testsuite/build_test/build_test.py
+++ b/testsuite/build_test/build_test.py
@@ -55,12 +55,21 @@ class CrossTest(CIBaseTest):
'mc:qemuamd64-stretch:isar-image-base',
'mc:de0-nano-soc-buster:isar-image-base',
'mc:stm32mp15x-buster:isar-image-base',
- 'mc:rpi-stretch:isar-image-base',
- 'mc:qemuarm64-focal:isar-image-base'
+ 'mc:rpi-stretch:isar-image-base'
]
self.perform_build_test(targets, 1, None)
+ def test_cross_ubuntu(self):
+ targets = [
+ 'mc:qemuarm64-focal:isar-image-base'
+ ]
+
+ try:
+ self.perform_build_test(targets, 1, None)
+ except:
+ self.cancel('KFAIL')
+
def test_cross_bullseye(self):
targets = [
'mc:qemuarm-bullseye:isar-image-base'
--
2.7.4
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v14 15/16] ci_build: Migrate to Avocado
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
` (13 preceding siblings ...)
2021-09-30 16:45 ` [PATCH v14 14/16] build_test: Protect ubuntu target with KFAIL Anton Mikanovich
@ 2021-09-30 16:45 ` Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 16/16] vm_smoke_test: " Anton Mikanovich
2021-10-14 8:54 ` [PATCH v14 00/16] Update Avocado testsuite Jan Kiszka
16 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
From: Anton Mikanovich <amikan@ilbers.de>
Call Avocado test cases instead of shell based.
Build artifacts are placed in build directory.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
scripts/ci_build.sh | 208 +++++++---------------------------------------------
1 file changed, 27 insertions(+), 181 deletions(-)
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 48a51f2..4534957 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -14,6 +14,9 @@ export PATH=$PATH:/sbin
# Go to Isar root
cd "$(dirname "$0")/.."
+# Get Avocado build tests path
+BUILD_TEST_DIR="$(pwd)/testsuite/build_test"
+
# Start build in Isar tree by default
BUILD_DIR=./build
@@ -21,70 +24,10 @@ BUILD_DIR=./build
DEPENDENCIES="umoci skopeo"
for prog in ${DEPENDENCIES} ; do
if [ ! -x "$(which $prog)" ] ; then
- echo "missing $prog in PATH, exiting" >&2
- exit 1
+ echo "missing $prog in PATH" >&2
fi
done
-BB_ARGS="-v"
-
-TARGETS_SET="\
- mc:qemuarm-stretch:isar-image-base \
- mc:qemuarm-buster:isar-image-base \
- mc:qemuarm64-stretch:isar-image-base \
- mc:qemui386-stretch:isar-image-base \
- mc:qemui386-buster:isar-image-base \
- mc:qemuamd64-stretch:isar-image-base \
- mc:qemuamd64-buster:isar-image-base \
- mc:qemuamd64-buster-tgz:isar-image-base \
- mc:qemuamd64-buster-cpiogz:isar-image-base \
- mc:qemuamd64-buster:isar-initramfs \
- mc:qemumipsel-stretch:isar-image-base \
- mc:qemumipsel-buster:isar-image-base \
- mc:nand-ubi-demo-buster:isar-image-ubi \
- mc:rpi-stretch:isar-image-base \
- mc:qemuamd64-focal:isar-image-base \
- mc:virtualbox-ova-buster:isar-image-base \
- "
- # qemu-user-static of <= buster too old to build that
- # mc:qemuarm64-buster:isar-image-base
- # mc:qemuarm64-bullseye:isar-image-base
-
-TARGETS_SET_BULLSEYE="\
- mc:qemuamd64-bullseye:isar-image-base \
- mc:qemuarm-bullseye:isar-image-base \
- mc:qemui386-bullseye:isar-image-base \
- mc:qemumipsel-bullseye:isar-image-base \
-"
-
-TARGETS_CONTAINERS="\
- mc:container-amd64-stretch:isar-image-base \
- mc:container-amd64-buster:isar-image-base \
- mc:container-amd64-bullseye:isar-image-base \
-"
-
-CROSS_TARGETS_SET="\
- mc:qemuarm-stretch:isar-image-base \
- mc:qemuarm-buster:isar-image-base \
- mc:qemuarm64-stretch:isar-image-base \
- mc:qemuamd64-stretch:isar-image-base \
- mc:de0-nano-soc-buster:isar-image-base \
- mc:stm32mp15x-buster:isar-image-base \
- mc:rpi-stretch:isar-image-base \
- mc:qemuarm64-focal:isar-image-base"
-
-CROSS_TARGETS_SET_BULLSEYE="\
- mc:qemuarm-bullseye:isar-image-base \
-"
-
-REPRO_TARGETS_SET_SIGNED="\
- mc:de0-nano-soc-buster:isar-image-base \
- mc:qemuarm64-stretch:isar-image-base"
-
-REPRO_TARGETS_SET="\
- mc:qemuamd64-stretch:isar-image-base \
- mc:qemuarm-buster:isar-image-base"
-
show_help() {
echo "This script builds the default Isar images."
echo
@@ -106,6 +49,10 @@ show_help() {
echo " 3 if invalid parameters are passed."
}
+TAGS="full"
+CROSS_BUILD="0"
+QUIET="0"
+
# Parse command line to get user configuration
while [ $# -gt 0 ]
do
@@ -124,16 +71,16 @@ do
CROSS_BUILD="1"
;;
-d|--debug)
- BB_ARGS="$BB_ARGS -D"
+ VERBOSE="--show=app,test"
;;
-f|--fast)
# Start build for the reduced set of configurations
# Enforce cross-compilation to speed up the build
- FAST_BUILD="1"
+ TAGS="fast"
CROSS_BUILD="1"
;;
-q|--quiet)
- BB_ARGS=""
+ QUIET="1"
;;
-r|--repro)
REPRO_BUILD="1"
@@ -152,125 +99,24 @@ do
shift
done
+if [ -z "$REPRO_BUILD" ]; then
+ TAGS="$TAGS,-repro"
+fi
+
# the real stuff starts here, trace commands from now on
set -x
-# Setup build folder for the current build
-if [ ! -d "$BUILD_DIR" ]; then
- mkdir -p "$BUILD_DIR"
-fi
-source isar-init-build-env "$BUILD_DIR"
-
-cat >>conf/local.conf <<EOF
-ISAR_ENABLE_COMPAT_ARCH_amd64 = "1"
-ISAR_ENABLE_COMPAT_ARCH_arm64 = "1"
-ISAR_ENABLE_COMPAT_ARCH_debian-stretch_amd64 = "0"
-IMAGE_INSTALL += "kselftest"
+# Provide working path
+mkdir -p .config/avocado
+cat <<EOF > .config/avocado/avocado.conf
+[datadir.paths]
+base_dir = $BUILD_DIR/
+test_dir = $BUILD_DIR/tests
+data_dir = $BUILD_DIR/data
+logs_dir = $BUILD_DIR/job-results
EOF
+export VIRTUAL_ENV="./"
-if [ -n "$DISTRO_APT_PREMIRRORS" ]; then
- echo "DISTRO_APT_PREMIRRORS = \"$DISTRO_APT_PREMIRRORS\"" >> conf/local.conf
-fi
-
-if [ -n "$CROSS_BUILD" ]; then
- sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
-fi
-
-if [ -n "$REPRO_BUILD" ]; then
- ISAR_TESTSUITE_GPG_PUB_KEY_FILE="$TESTSUITEDIR/base-apt/test_pub.key"
- ISAR_TESTSUITE_GPG_PRIV_KEY_FILE="$TESTSUITEDIR/base-apt/test_priv.key"
- export GNUPGHOME=$(mktemp -d)
- gpg --import $ISAR_TESTSUITE_GPG_PUB_KEY_FILE $ISAR_TESTSUITE_GPG_PRIV_KEY_FILE
-
- # Enable use of signed cached base repository
- echo BASE_REPO_KEY=\"file://$ISAR_TESTSUITE_GPG_PUB_KEY_FILE\" >> conf/local.conf
- bitbake $BB_ARGS $REPRO_TARGETS_SET_SIGNED
- while [ -e bitbake.sock ]; do sleep 1; done
- sudo rm -rf tmp
- sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
- sed -i -e 's/^#BB_NO_NETWORK/BB_NO_NETWORK/g' conf/local.conf
- bitbake $BB_ARGS $REPRO_TARGETS_SET_SIGNED
- while [ -e bitbake.sock ]; do sleep 1; done
- # Cleanup and disable use of signed cached base repository
- sudo rm -rf tmp
- sed -i -e 's/ISAR_USE_CACHED_BASE_REPO ?= "1"/#ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
- sed -i -e 's/^BB_NO_NETWORK/#BB_NO_NETWORK/g' conf/local.conf
- sed -i -e 's/^BASE_REPO_KEY/#BASE_REPO_KEY/g' conf/local.conf
- bitbake $BB_ARGS $REPRO_TARGETS_SET
- while [ -e bitbake.sock ]; do sleep 1; done
- # Enable use of unsigned cached base repository
- sudo rm -rf tmp
- sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
- sed -i -e 's/^#BB_NO_NETWORK/BB_NO_NETWORK/g' conf/local.conf
- bitbake $BB_ARGS $REPRO_TARGETS_SET
- while [ -e bitbake.sock ]; do sleep 1; done
- # Disable use of unsigned cached base repository
- sed -i -e 's/ISAR_USE_CACHED_BASE_REPO ?= "1"/#ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
- sed -i -e 's/^BB_NO_NETWORK/#BB_NO_NETWORK/g' conf/local.conf
- # Try to build with changed configuration with no cleanup
- bitbake $BB_ARGS $REPRO_TARGETS_SET
- while [ -e bitbake.sock ]; do sleep 1; done
- # Cleanup
- sudo rm -rf tmp
-fi
-
-sed -i -e 's/^#BASE_REPO_FEATURES ?= "cache-deb-src"/BASE_REPO_FEATURES ?= "cache-deb-src"/g' conf/local.conf
-# Start cross build for the defined set of configurations
-sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
-bitbake $BB_ARGS $CROSS_TARGETS_SET
-while [ -e bitbake.sock ]; do sleep 1; done
-if bitbake $BB_ARGS $CROSS_TARGETS_SET_BULLSEYE; then
- echo "bullseye cross: PASSED"
-else
- echo "bullseye cross: KFAIL"
-fi
-# In addition test SDK creation
-bitbake $BB_ARGS -c do_populate_sdk mc:qemuarm-stretch:isar-image-base
-while [ -e bitbake.sock ]; do sleep 1; done
-
-if [ -z "$FAST_BUILD" ]; then
- # Cleanup and disable cross build
- sudo rm -rf tmp
- sed -i -e 's/ISAR_CROSS_COMPILE ?= "1"/ISAR_CROSS_COMPILE ?= "0"/g' conf/local.conf
- bitbake $BB_ARGS $TARGETS_SET
- while [ -e bitbake.sock ]; do sleep 1; done
-
- if bitbake $BB_ARGS $TARGETS_SET_BULLSEYE; then
- echo "bullseye: PASSED"
- else
- echo "bullseye: KFAIL"
- fi
- while [ -e bitbake.sock ]; do sleep 1; done
-fi
-
-eval $(bitbake -e | grep "^LAYERDIR_core=")
-eval $(bitbake -e | grep "^LAYERDIR_isar=")
-cp -a "${LAYERDIR_core}/classes/dpkg-base.bbclass" "${LAYERDIR_core}/classes/dpkg-base.bbclass.ci-backup"
-echo -e "do_fetch_append() {\n\n}" >> "${LAYERDIR_core}/classes/dpkg-base.bbclass"
-
-bitbake $BB_ARGS mc:qemuamd64-stretch:isar-image-base
-
-mv "${LAYERDIR_core}/classes/dpkg-base.bbclass.ci-backup" "${LAYERDIR_core}/classes/dpkg-base.bbclass"
-
-# Test wic --exclude-path
-cp -a "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks" "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks.ci-backup"
-mv ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img \
- ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img.ci-backup
-sed -i -e 's/part \/ /part \/ --exclude-path usr /g' "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks"
-
-bitbake $BB_ARGS mc:qemuamd64-stretch:isar-image-base
-
-mv "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks.ci-backup" "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks"
-mv ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img.ci-backup \
- ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img
-
-# Finalize with containerized images, since they remove some not-needed packages from the local.conf
-sed -i -e 's/\(IMAGE_INSTALL = .*\) example-module-${KERNEL_NAME}\(.*\)/\1\2/g' conf/local.conf
-sed -i -e 's/\(IMAGE_INSTALL = .*\) enable-fsck\(.*\)/\1\2/g' conf/local.conf
-bitbake $BB_ARGS $TARGETS_CONTAINERS
-while [ -e bitbake.sock ]; do sleep 1; done
-# and SDK container image creation
-echo 'SDK_FORMATS = "docker-archive"' >> conf/local.conf
-bitbake $BB_ARGS -c do_populate_sdk mc:container-amd64-stretch:isar-image-base
-while [ -e bitbake.sock ]; do sleep 1; done
-
+avocado $VERBOSE run "$BUILD_TEST_DIR/build_test.py" \
+ -t $TAGS --test-runner=runner --disable-sysinfo \
+ -p build_dir="$BUILD_DIR" -p quiet=$QUIET -p cross=$CROSS_BUILD
--
2.7.4
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v14 16/16] vm_smoke_test: Migrate to Avocado
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
` (14 preceding siblings ...)
2021-09-30 16:45 ` [PATCH v14 15/16] ci_build: Migrate to Avocado Anton Mikanovich
@ 2021-09-30 16:45 ` Anton Mikanovich
2021-10-14 8:54 ` [PATCH v14 00/16] Update Avocado testsuite Jan Kiszka
16 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
From: Anton Mikanovich <amikan@ilbers.de>
Call Avocado test cases instead of shell based.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
scripts/vm_smoke_test | 120 ++++++++++++--------------------------------------
1 file changed, 28 insertions(+), 92 deletions(-)
diff --git a/scripts/vm_smoke_test b/scripts/vm_smoke_test
index ce08213..6416250 100755
--- a/scripts/vm_smoke_test
+++ b/scripts/vm_smoke_test
@@ -3,9 +3,7 @@
# This software is a part of ISAR.
# Copyright (C) 2015-2018 ilbers GmbH
-CONSOLE_OUTPUT=/tmp/isar_console
-PID_FILE=/tmp/qemu.pid
-VERBOSE=1
+VERBOSE="--show=test"
TIMEOUT=300
# Error codes:
@@ -13,75 +11,15 @@ ES_OK=0
ES_FAIL=1
ES_BUG=3
-RET=$ES_OK
+RET=$ES_FAIL
-dump_boot_log() {
- echo "Boot log:\n8<--"
- cat $CONSOLE_OUTPUT
- echo "\n8<--"
-}
-
-check_login_prompt() {
- echo -n "Check login prompt: "
-
- str=$(grep "isar login: " $CONSOLE_OUTPUT)
-
- if [ -n "$str" ]; then
- echo "PASSED"
- else
- echo "FAIL"
- RET=$ES_FAIL
- FAIL=1
- fi
-}
-
-check_example_module() {
- echo -n "Check example module: "
-
- str=$(grep "Just an example" $CONSOLE_OUTPUT)
-
- if [ -n "$str" ]; then
- echo "PASSED"
- else
- echo "FAIL"
- RET=$ES_FAIL
- FAIL=1
- fi
-}
-
-run_test () {
- ARCH=$1
- DISTRO=$2
+# Get Avocado QEMU tests path
+VM_TEST_DIR="$(dirname "$0")/../testsuite/vm_boot_test"
- echo "-------------------------------------------------"
- echo "Testing Isar [$DISTRO] image for [$ARCH] machine:"
+# Go to Isar root
+cd "$(dirname "$0")/.."
- start_vm -a $ARCH -d $DISTRO -o $CONSOLE_OUTPUT -p $PID_FILE > /dev/null 2>&1 &
- sleep 5
-
- if [ -z `ps -p $! -o pid=` ]; then
- echo "QEMU start: FAILED"
- RET=$ES_FAIL
- echo "Command output:\n8<--"
- start_vm -a $ARCH -d $DISTRO -o $CONSOLE_OUTPUT -p $PID_FILE
- echo "\n8<--"
- else
- sleep $TIMEOUT
- kill `cat $PID_FILE`
-
- FAIL=0
-
- check_login_prompt
-
- check_example_module
-
- [ $VERBOSE -eq 1 -o $FAIL -eq 1 ] && dump_boot_log
-
- rm $CONSOLE_OUTPUT
- fi
-
- rm $PID_FILE
-}
+BUILD_DIR=./build
show_help() {
echo "This script tests the Isar images for default targets in QEMU."
@@ -91,10 +29,6 @@ show_help() {
echo
echo "Parameters:"
echo " -f,--fast test reduced set of supported targets."
- echo " -o,--output FILE specify file to store console output."
- echo " The default is: /tmp/isar_console"
- echo " -p,--pid-file FILE specify file to store QEMU process PID."
- echo " The default is: /tmp/qemu.pid"
echo " -q, --quiet do not display boot logs for all the targets."
echo " If test failed for the specific configuration,"
echo " the respective boot log will be printed anyway."
@@ -119,18 +53,18 @@ do
exit 0
;;
-o|--output)
- CONSOLE_OUTPUT=$2
+ # Deprecated option
shift
;;
-p|--pid-file)
- PID_FILE=$2
+ # Deprecated option
shift
;;
-f|--fast)
FAST_BUILD="1"
;;
-q|--quiet)
- VERBOSE=0
+ VERBOSE=""
;;
-t|--timeout)
TIMEOUT=$2
@@ -145,24 +79,26 @@ do
shift
done
-# ARM machine
-run_test arm stretch
-run_test arm buster
-
-# AMD64 machine
-if [ -z "$FAST_BUILD" ]; then
- run_test amd64 buster
- run_test amd64 focal
+TAGS="full"
+if [ -n "$FAST_BUILD" ]; then
+ TAGS="fast"
fi
-run_test amd64 stretch
-# i386 machine
-if [ -z "$FAST_BUILD" ]; then
- run_test i386 stretch
- run_test i386 buster
+# Provide working path
+mkdir -p .config/avocado
+cat <<EOF > .config/avocado/avocado.conf
+[datadir.paths]
+base_dir = $BUILD_DIR/
+test_dir = $BUILD_DIR/tests
+data_dir = $BUILD_DIR/data
+logs_dir = $BUILD_DIR/job-results
+EOF
+export VIRTUAL_ENV="./"
+
+if avocado $VERBOSE run "$VM_TEST_DIR/vm_boot_test.py" -t $TAGS \
+ --test-runner=runner --disable-sysinfo \
+ -p build_dir="$BUILD_DIR" -p time_to_wait=$TIMEOUT; then
+ RET=$ES_OK
fi
-# ARM64 machine
-run_test arm64 stretch
-
exit $RET
--
2.7.4
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v14 00/16] Update Avocado testsuite
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
` (15 preceding siblings ...)
2021-09-30 16:45 ` [PATCH v14 16/16] vm_smoke_test: " Anton Mikanovich
@ 2021-10-14 8:54 ` Jan Kiszka
2021-10-14 10:25 ` Baurzhan Ismagulov
16 siblings, 1 reply; 25+ messages in thread
From: Jan Kiszka @ 2021-10-14 8:54 UTC (permalink / raw)
To: Anton Mikanovich, isar-users; +Cc: Schmidt, Adriaan
On 30.09.21 18:45, Anton Mikanovich wrote:
> This patchset provides implementation of build and vm_start tests
> functionality in Python with help of Avocado framework.
> Scripts ci_build and vm_smoke_test are rewritten to call Avocado.
> Build and run logs are hidden by default now until an error occurs.
>
> Resending v14 because v13 was just a copy of v12 by mistake.
>
> ---
> Changes since v13:
> - Remove legacy scripts.
> Changes since v12:
> - No changes, wrong patchset.
> Changes since v11:
> - Rebase on next.
> - Init environment in vm_smoke_test.
> Changes since v10:
> - Rebase on next.
> - Add DISTRO_APT_PREMIRRORS usage.
> - Remove avocado deb package install.
> - Set default runner.
> Changes since v9:
> - Rebase on next.
> - Fix avocado artifacts path.
> Changes since v8:
> - Remove duplication in logs.
> - Disable sysinfo plugin.
> - Show full errors in default mode.
> Changes since v7:
> - Remove code duplication.
> - Fix build_dir parsing.
> - Add missed protect files backup logic.
> Changes since v6:
> - Improve environment parsing.
> - Protect files backup logic.
> - Rebuild full log output option.
> - Protect ubuntu target with KFAIL.
> Changes since v5:
> - Rebase on current next.
> Changes since v4:
> - Add container tests.
> - Implement testcase skipping logic.
> - Separate and protect Bullseye tests.
> - Add checks for container dependencies.
> - Add Ilbers repo key.
> - Improve error handling.
> - Fix Ubuntu qemu image name.
> Changes since v3:
> - Add automatic bitbake init.
> - Fix line lengths.
> - Fix build without repro.
> - Fix build artifact paths.
> - Prepare for Gitlab build.
> - Change debug settings.
> - Move log splitting to start_vm.
> - Add deb package install.
> Changes since v2:
> - Make CI call Avocado tests.
> - Wait less on vm_start test.
> - Split Roman's vm_start commit.
> - Put logs to build folder.
> - Rearrange patches.
> - Split build test to classes.
> - Add test case tagging.
> - Get rid of avocado-framework-plugin-varianter-yaml-to-mux.
> - Get rid of python-subprocess32.
> - Improve logging.
> Changes since v1:
> - Merge start_vm rebuild patches.
> - Fix patch comments.
>
> Anton Mikanovich (13):
> start_vm.py: Fix ubuntu image name
> vm_boot_test: Fix log file path in vm_boot_test
> vm_boot_test: Remove external varianter
> vm_boot_test: Improve QEMU images checking
> build_test: Refactoring build tests cases
> testsuite: Add Python generations for testsuite in gitignore
> testsuite: Fix test suite prepare guide
> gitlab-ci: Add Avocado build artifacts
> gitlab-ci: Add debug flag
> vm_boot_test: Add automatic bitbake init
> build_test: Protect ubuntu target with KFAIL
> ci_build: Migrate to Avocado
> vm_smoke_test: Migrate to Avocado
>
> Roman Pletnev (3):
> start_vm.py: Fix target name handling
> start_vm.py: Add output and PID file vm_start.py options
> start_vm.py: Add MIPS support
>
> .gitlab-ci.yml | 5 +-
> scripts/ci_build.sh | 208 ++++---------------------------
> scripts/vm_smoke_test | 120 +++++-------------
> testsuite/.gitignore | 1 +
> testsuite/README.md | 21 +---
> testsuite/build_test/build_test.py | 219 ++++++++++++++++++++++++++++++---
> testsuite/build_test/cibase.py | 116 +++++++++++++++++
> testsuite/build_test/cibuilder.py | 132 ++++++++++++++++++++
> testsuite/build_test/run.sh | 3 -
> testsuite/build_test/run_fast.sh | 3 +
> testsuite/build_test/run_full.sh | 3 +
> testsuite/build_test/variant.yaml | 22 ----
> testsuite/start_vm.py | 27 ++--
> testsuite/vm_boot_test/run.sh | 3 -
> testsuite/vm_boot_test/run_fast.sh | 3 +
> testsuite/vm_boot_test/run_full.sh | 3 +
> testsuite/vm_boot_test/variant.yaml | 22 ----
> testsuite/vm_boot_test/vm_boot_test.py | 121 +++++++++++++++---
> 18 files changed, 641 insertions(+), 391 deletions(-)
> create mode 100644 testsuite/.gitignore
> create mode 100644 testsuite/build_test/cibase.py
> create mode 100644 testsuite/build_test/cibuilder.py
> delete mode 100755 testsuite/build_test/run.sh
> create mode 100755 testsuite/build_test/run_fast.sh
> create mode 100755 testsuite/build_test/run_full.sh
> delete mode 100644 testsuite/build_test/variant.yaml
> delete mode 100755 testsuite/vm_boot_test/run.sh
> create mode 100755 testsuite/vm_boot_test/run_fast.sh
> create mode 100755 testsuite/vm_boot_test/run_full.sh
> delete mode 100644 testsuite/vm_boot_test/variant.yaml
>
Didn't you once installed avocado for execution in CI? This is no longer
happening - unless I missed something - and, thus, broke gitlab-ci.
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v14 00/16] Update Avocado testsuite
2021-10-14 8:54 ` [PATCH v14 00/16] Update Avocado testsuite Jan Kiszka
@ 2021-10-14 10:25 ` Baurzhan Ismagulov
2021-10-14 15:51 ` Jan Kiszka
0 siblings, 1 reply; 25+ messages in thread
From: Baurzhan Ismagulov @ 2021-10-14 10:25 UTC (permalink / raw)
To: isar-users
On Thu, Oct 14, 2021 at 10:54:37AM +0200, Jan Kiszka wrote:
> Didn't you once installed avocado for execution in CI? This is no longer
> happening - unless I missed something - and, thus, broke gitlab-ci.
Yes, this happened in the v10 discussion:
https://groups.google.com/g/isar-users/c/eSL1-OLaErE/m/6J3EfD9YCQAJ
In a nutshell, the execution environment can have the control how the necessary
tools are provided (Debian packages or pip), just as it is today with the other
packages required by Isar.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v14 00/16] Update Avocado testsuite
2021-10-14 10:25 ` Baurzhan Ismagulov
@ 2021-10-14 15:51 ` Jan Kiszka
2021-10-19 8:34 ` Schmidt, Adriaan
` (2 more replies)
0 siblings, 3 replies; 25+ messages in thread
From: Jan Kiszka @ 2021-10-14 15:51 UTC (permalink / raw)
To: isar-users
On 14.10.21 12:25, Baurzhan Ismagulov wrote:
> On Thu, Oct 14, 2021 at 10:54:37AM +0200, Jan Kiszka wrote:
>> Didn't you once installed avocado for execution in CI? This is no longer
>> happening - unless I missed something - and, thus, broke gitlab-ci.
>
> Yes, this happened in the v10 discussion:
>
> https://groups.google.com/g/isar-users/c/eSL1-OLaErE/m/6J3EfD9YCQAJ
>
> In a nutshell, the execution environment can have the control how the necessary
> tools are provided (Debian packages or pip), just as it is today with the other
> packages required by Isar.
You broke gitlab-ci, please fix. This can only reasonably be done in
Isar, not outside (avokado is not build dependency, this is not part of
kas-isar).
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 25+ messages in thread
* RE: [PATCH v14 00/16] Update Avocado testsuite
2021-10-14 15:51 ` Jan Kiszka
@ 2021-10-19 8:34 ` Schmidt, Adriaan
2021-10-22 6:54 ` Anton Mikanovich
2021-10-19 9:54 ` Henning Schild
2021-10-21 9:48 ` Baurzhan Ismagulov
2 siblings, 1 reply; 25+ messages in thread
From: Schmidt, Adriaan @ 2021-10-19 8:34 UTC (permalink / raw)
To: jan.kiszka, isar-users
> -----Original Message-----
> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf Of
> Jan Kiszka
> Sent: Donnerstag, 14. Oktober 2021 17:51
> To: isar-users@googlegroups.com
> Subject: Re: [PATCH v14 00/16] Update Avocado testsuite
>
> On 14.10.21 12:25, Baurzhan Ismagulov wrote:
> > On Thu, Oct 14, 2021 at 10:54:37AM +0200, Jan Kiszka wrote:
> >> Didn't you once installed avocado for execution in CI? This is no longer
> >> happening - unless I missed something - and, thus, broke gitlab-ci.
> >
> > Yes, this happened in the v10 discussion:
> >
> > https://groups.google.com/g/isar-users/c/eSL1-OLaErE/m/6J3EfD9YCQAJ
> >
> > In a nutshell, the execution environment can have the control how the
> necessary
> > tools are provided (Debian packages or pip), just as it is today with the
> other
> > packages required by Isar.
>
> You broke gitlab-ci, please fix. This can only reasonably be done in
> Isar, not outside (avokado is not build dependency, this is not part of
> kas-isar).
Even when installing avocado manually (pip3 install avocado-framework), the tests (or rather the collection of results?) fail in GitLab CI using the kas-isar:latest container:
Traceback (most recent call last):
File "/builder/.local/lib/python3.7/site-packages/avocado/core/test.py", line 825, in _run_avocado
genio.write_file(whiteboard_file, self.whiteboard)
File "/builder/.local/lib/python3.7/site-packages/avocado/utils/genio.py", line 117, in write_file
with open(filename, 'w') as file_obj:
FileNotFoundError: [Errno 2] No such file or directory: './build/job-results/job-2021-10-15T14.02-5ab2e09/test-results/1-_builds_cEV_-yzV_0_ebsy_debian_isar_testsuite_build_test_build_test.py_CrossTest.test_cross/whiteboard'
ERROR 1-/builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross -> FileNotFoundError: [Errno 2] No such file or directory: './build/job-results/job-2021-10-15T14.02-5ab2e09/test-results/1-_builds_cEV_-yzV_0_ebsy_debian_isar_testsuite_build_test_build_test.py_CrossTest.test_cross/whiteboard'
ERROR: [Errno 2] No such file or directory: './build/job-results/job-2021-10-15T14.02-5ab2e09/test-results/1-_builds_cEV_-yzV_0_ebsy_debian_isar_testsuite_build_test_build_test.py_CrossTest.test_cross/whiteboard' (5528.48 s)
INIT 2-/builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross_ubuntu
PARAMS (key=timeout, path=*, default=None) => None
Test metadata:
filename: /builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py
teststmpdir: /var/tmp/avocado_haez6g83
(2/8) /builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross_ubuntu: START 2-/builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross_ubuntu
DATA (filename=output.expected) => NOT FOUND (data sources: variant, test, file)
Adriaan
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v14 00/16] Update Avocado testsuite
2021-10-14 15:51 ` Jan Kiszka
2021-10-19 8:34 ` Schmidt, Adriaan
@ 2021-10-19 9:54 ` Henning Schild
2021-10-21 9:48 ` Baurzhan Ismagulov
2 siblings, 0 replies; 25+ messages in thread
From: Henning Schild @ 2021-10-19 9:54 UTC (permalink / raw)
To: Jan Kiszka; +Cc: isar-users
I am on that ... have patches practically ready. Did expect ilbers to
break that because my remarks have been ignored earlier.
My patch will use a virtualenv and install avocado conditionally if not
found. Will send them anytime soon.
Henning
Am Thu, 14 Oct 2021 17:51:22 +0200
schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> On 14.10.21 12:25, Baurzhan Ismagulov wrote:
> > On Thu, Oct 14, 2021 at 10:54:37AM +0200, Jan Kiszka wrote:
> >> Didn't you once installed avocado for execution in CI? This is no
> >> longer happening - unless I missed something - and, thus, broke
> >> gitlab-ci.
> >
> > Yes, this happened in the v10 discussion:
> >
> > https://groups.google.com/g/isar-users/c/eSL1-OLaErE/m/6J3EfD9YCQAJ
> >
> > In a nutshell, the execution environment can have the control how
> > the necessary tools are provided (Debian packages or pip), just as
> > it is today with the other packages required by Isar.
>
> You broke gitlab-ci, please fix. This can only reasonably be done in
> Isar, not outside (avokado is not build dependency, this is not part
> of kas-isar).
>
> Jan
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v14 00/16] Update Avocado testsuite
2021-10-14 15:51 ` Jan Kiszka
2021-10-19 8:34 ` Schmidt, Adriaan
2021-10-19 9:54 ` Henning Schild
@ 2021-10-21 9:48 ` Baurzhan Ismagulov
2 siblings, 0 replies; 25+ messages in thread
From: Baurzhan Ismagulov @ 2021-10-21 9:48 UTC (permalink / raw)
To: isar-users
On Thu, Oct 14, 2021 at 05:51:22PM +0200, Jan Kiszka wrote:
> > In a nutshell, the execution environment can have the control how the necessary
> > tools are provided (Debian packages or pip), just as it is today with the other
> > packages required by Isar.
>
> You broke gitlab-ci, please fix. This can only reasonably be done in
> Isar, not outside (avokado is not build dependency, this is not part of
> kas-isar).
We can do this to unblock the situation. However, I think that drawing the line
between build dependency and test tooling is a disservice for kas users. One of
the goals for migrating to a test framework is to split the previously
monolithic testsuite into individual testcases and the ability to run them
manually.
People like kas because it hides the environment preparation issues while also
providing the necessary customization options. If a user spawns a kas shell to
make his changes, he could use e.g. avocado run
testsuite/build_test/build_test.py -t sdk to test only the area he is
interested in. The same for debugging failing CI cases without having to run
the full testsuite, which meanwhile takes several hours.
That is why I think providing the test tooling in the default image is actually
a good thing for the developers.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v14 00/16] Update Avocado testsuite
2021-10-19 8:34 ` Schmidt, Adriaan
@ 2021-10-22 6:54 ` Anton Mikanovich
0 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-10-22 6:54 UTC (permalink / raw)
To: Schmidt, Adriaan, jan.kiszka, isar-users
On 19.10.21 11:34, Schmidt, Adriaan wrote:
> Even when installing avocado manually (pip3 install avocado-framework), the tests (or rather the collection of results?) fail in GitLab CI using the kas-isar:latest container:
>
> Traceback (most recent call last):
> File "/builder/.local/lib/python3.7/site-packages/avocado/core/test.py", line 825, in _run_avocado
> genio.write_file(whiteboard_file, self.whiteboard)
> File "/builder/.local/lib/python3.7/site-packages/avocado/utils/genio.py", line 117, in write_file
> with open(filename, 'w') as file_obj:
> FileNotFoundError: [Errno 2] No such file or directory: './build/job-results/job-2021-10-15T14.02-5ab2e09/test-results/1-_builds_cEV_-yzV_0_ebsy_debian_isar_testsuite_build_test_build_test.py_CrossTest.test_cross/whiteboard'
> ERROR 1-/builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross -> FileNotFoundError: [Errno 2] No such file or directory: './build/job-results/job-2021-10-15T14.02-5ab2e09/test-results/1-_builds_cEV_-yzV_0_ebsy_debian_isar_testsuite_build_test_build_test.py_CrossTest.test_cross/whiteboard'
> ERROR: [Errno 2] No such file or directory: './build/job-results/job-2021-10-15T14.02-5ab2e09/test-results/1-_builds_cEV_-yzV_0_ebsy_debian_isar_testsuite_build_test_build_test.py_CrossTest.test_cross/whiteboard' (5528.48 s)
> INIT 2-/builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross_ubuntu
> PARAMS (key=timeout, path=*, default=None) => None
> Test metadata:
> filename: /builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py
> teststmpdir: /var/tmp/avocado_haez6g83
> (2/8) /builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross_ubuntu: START 2-/builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross_ubuntu
> DATA (filename=output.expected) => NOT FOUND (data sources: variant, test, file)
>
> Adriaan
This was caused by logdir management changes between Avocado 88.0 and
89.0, so we couldn't use absolute paths in config file.
Fix provided in `[PATCH] CI: Fix paths for the latest Avocado version`
--
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v14 00/16] Update Avocado testsuite
[not found] <1633020197-26007-1-git-send-email-anton.mikanovich@promwad.com>
@ 2021-10-13 14:06 ` Anton Mikanovich
0 siblings, 0 replies; 25+ messages in thread
From: Anton Mikanovich @ 2021-10-13 14:06 UTC (permalink / raw)
To: isar-users; +Cc: Henning Schild
On 30.09.21 19:43, Anton Mikanovich wrote:
> This patchset provides implementation of build and vm_start tests
> functionality in Python with help of Avocado framework.
> Scripts ci_build and vm_smoke_test are rewritten to call Avocado.
> Build and run logs are hidden by default now until an error occurs.
>
> Resending v14 because v13 was just a copy of v12 by mistake.
Applied to next.
--
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2021-10-22 6:55 UTC | newest]
Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-30 16:45 [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 01/16] start_vm.py: Fix target name handling Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 02/16] start_vm.py: Add output and PID file vm_start.py options Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 03/16] start_vm.py: Add MIPS support Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 04/16] start_vm.py: Fix ubuntu image name Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 05/16] vm_boot_test: Fix log file path in vm_boot_test Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 06/16] vm_boot_test: Remove external varianter Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 07/16] vm_boot_test: Improve QEMU images checking Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 08/16] build_test: Refactoring build tests cases Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 09/16] testsuite: Add Python generations for testsuite in gitignore Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 10/16] testsuite: Fix test suite prepare guide Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 11/16] gitlab-ci: Add Avocado build artifacts Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 12/16] gitlab-ci: Add debug flag Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 13/16] vm_boot_test: Add automatic bitbake init Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 14/16] build_test: Protect ubuntu target with KFAIL Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 15/16] ci_build: Migrate to Avocado Anton Mikanovich
2021-09-30 16:45 ` [PATCH v14 16/16] vm_smoke_test: " Anton Mikanovich
2021-10-14 8:54 ` [PATCH v14 00/16] Update Avocado testsuite Jan Kiszka
2021-10-14 10:25 ` Baurzhan Ismagulov
2021-10-14 15:51 ` Jan Kiszka
2021-10-19 8:34 ` Schmidt, Adriaan
2021-10-22 6:54 ` Anton Mikanovich
2021-10-19 9:54 ` Henning Schild
2021-10-21 9:48 ` Baurzhan Ismagulov
[not found] <1633020197-26007-1-git-send-email-anton.mikanovich@promwad.com>
2021-10-13 14:06 ` Anton Mikanovich
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox