* [PATCH v12 00/16] Update Avocado testsuite
@ 2021-09-17 17:00 Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 01/16] start_vm.py: Fix target name handling Anton Mikanovich
` (16 more replies)
0 siblings, 17 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-17 17:00 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
This patchset provides implementation of build and vm_start tests
functionality in Python with help of Avocado framework.
Scripts ci_build and vm_smoke_test are rewritten to call Avocado.
ci_build and vm_smoke_test are left for compatibility and
should be replaced by direct avocado calls later.
Original ci_build and vm_smoke_test are renamed to *_legacy.
Build and run logs are hidden by default now until an error occurs.
---
Changes since v11:
- Rebase on next.
- Init environment in vm_smoke_test.
Changes since v10:
- Rebase on next.
- Add DISTRO_APT_PREMIRRORS usage.
- Remove avocado deb package install.
- Set default runner.
Changes since v9:
- Rebase on next.
- Fix avocado artifacts path.
Changes since v8:
- Remove duplication in logs.
- Disable sysinfo plugin.
- Show full errors in default mode.
Changes since v7:
- Remove code duplication.
- Fix build_dir parsing.
- Add missed protect files backup logic.
Changes since v6:
- Improve environment parsing.
- Protect files backup logic.
- Rebuild full log output option.
- Protect ubuntu target with KFAIL.
Changes since v5:
- Rebase on current next.
Changes since v4:
- Add container tests.
- Implement testcase skipping logic.
- Separate and protect Bullseye tests.
- Add checks for container dependencies.
- Add Ilbers repo key.
- Improve error handling.
- Fix Ubuntu qemu image name.
Changes since v3:
- Add automatic bitbake init.
- Fix line lengths.
- Fix build without repro.
- Fix build artifact paths.
- Prepare for Gitlab build.
- Change debug settings.
- Move log splitting to start_vm.
- Add deb package install.
Changes since v2:
- Make CI call Avocado tests.
- Wait less on vm_start test.
- Split Roman's vm_start commit.
- Put logs to build folder.
- Rearrange patches.
- Split build test to classes.
- Add test case tagging.
- Get rid of avocado-framework-plugin-varianter-yaml-to-mux.
- Get rid of python-subprocess32.
- Improve logging.
Changes since v1:
- Merge start_vm rebuild patches.
- Fix patch comments.
Anton Mikanovich (13):
start_vm.py: Fix ubuntu image name
vm_boot_test: Fix log file path in vm_boot_test
vm_boot_test: Remove external varianter
vm_boot_test: Improve QEMU images checking
build_test: Refactoring build tests cases
testsuite: Add Python generations for testsuite in gitignore
testsuite: Fix test suite prepare guide
gitlab-ci: Add Avocado build artifacts
gitlab-ci: Add debug flag
vm_boot_test: Add automatic bitbake init
build_test: Protect ubuntu target with KFAIL
ci_build: Migrate to Avocado
vm_smoke_test: Migrate to Avocado
Roman Pletnev (3):
start_vm.py: Fix target name handling
start_vm.py: Add output and PID file vm_start.py options
start_vm.py: Add MIPS support
.gitlab-ci.yml | 5 +-
scripts/ci_build.sh | 208 +++----------------
scripts/ci_build_legacy.sh | 275 +++++++++++++++++++++++++
scripts/vm_smoke_test | 120 +++--------
scripts/vm_smoke_test_legacy | 168 +++++++++++++++
testsuite/.gitignore | 1 +
testsuite/README.md | 21 +-
testsuite/build_test/build_test.py | 219 ++++++++++++++++++--
testsuite/build_test/cibase.py | 116 +++++++++++
testsuite/build_test/cibuilder.py | 132 ++++++++++++
testsuite/build_test/run.sh | 3 -
testsuite/build_test/run_fast.sh | 3 +
testsuite/build_test/run_full.sh | 3 +
testsuite/build_test/variant.yaml | 22 --
testsuite/start_vm.py | 27 ++-
testsuite/vm_boot_test/run.sh | 3 -
testsuite/vm_boot_test/run_fast.sh | 3 +
testsuite/vm_boot_test/run_full.sh | 3 +
testsuite/vm_boot_test/variant.yaml | 22 --
testsuite/vm_boot_test/vm_boot_test.py | 121 +++++++++--
20 files changed, 1084 insertions(+), 391 deletions(-)
create mode 100755 scripts/ci_build_legacy.sh
create mode 100755 scripts/vm_smoke_test_legacy
create mode 100644 testsuite/.gitignore
create mode 100644 testsuite/build_test/cibase.py
create mode 100644 testsuite/build_test/cibuilder.py
delete mode 100755 testsuite/build_test/run.sh
create mode 100755 testsuite/build_test/run_fast.sh
create mode 100755 testsuite/build_test/run_full.sh
delete mode 100644 testsuite/build_test/variant.yaml
delete mode 100755 testsuite/vm_boot_test/run.sh
create mode 100755 testsuite/vm_boot_test/run_fast.sh
create mode 100755 testsuite/vm_boot_test/run_full.sh
delete mode 100644 testsuite/vm_boot_test/variant.yaml
--
2.25.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v12 01/16] start_vm.py: Fix target name handling
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
@ 2021-09-17 17:00 ` Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 02/16] start_vm.py: Add output and PID file vm_start.py options Anton Mikanovich
` (15 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-17 17:00 UTC (permalink / raw)
To: isar-users; +Cc: Roman Pletnev, Anton Mikanovich
From: Roman Pletnev <rpletnev@ilbers.de>
This patch fix bb_output search (issue with wrong arch name being
selected for qemu) which caused qemu-system-riscv64 to be used for all
the targets:
>ERROR: [Errno 2] No such file or directory: u'qemu-system-riscv64' (1.05 s)
Signed-off-by: Roman Pletnev <rpletnev@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/start_vm.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index a3e32ac..c10db67 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -17,7 +17,7 @@ def get_bitbake_env(arch, distro):
def get_bitbake_var(output, var):
ret = ''
for line in output.splitlines():
- if line.startswith(var):
+ if line.startswith(var + '='):
ret = line.split('"')[1]
return ret
--
2.25.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v12 02/16] start_vm.py: Add output and PID file vm_start.py options
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 01/16] start_vm.py: Fix target name handling Anton Mikanovich
@ 2021-09-17 17:00 ` Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 03/16] start_vm.py: Add MIPS support Anton Mikanovich
` (14 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-17 17:00 UTC (permalink / raw)
To: isar-users; +Cc: Roman Pletnev, Anton Mikanovich
From: Roman Pletnev <rpletnev@ilbers.de>
This patch adds options -o (output file) and -p (pid file).
Implement virtual machine log output both to the file and stdout.
Signed-off-by: Roman Pletnev <rpletnev@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/start_vm.py | 18 ++++++++++++++----
testsuite/vm_boot_test/vm_boot_test.py | 5 ++---
2 files changed, 16 insertions(+), 7 deletions(-)
diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index c10db67..3736e51 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -21,7 +21,7 @@ def get_bitbake_var(output, var):
ret = line.split('"')[1]
return ret
-def format_qemu_cmdline(arch, build, distro):
+def format_qemu_cmdline(arch, build, distro, out, pid):
bb_output = get_bitbake_env(arch, distro).decode()
rootfs_image = ''
@@ -57,6 +57,13 @@ def format_qemu_cmdline(arch, build, distro):
qemu_cpu = get_bitbake_var(bb_output, 'QEMU_CPU')
qemu_disk_args = get_bitbake_var(bb_output, 'QEMU_DISK_ARGS')
+ if out:
+ extra_args.extend(['-chardev','stdio,id=ch0,logfile=' + out])
+ extra_args.extend(['-serial','chardev:ch0'])
+ extra_args.extend(['-monitor','none'])
+ if pid:
+ extra_args.extend(['-pidfile', pid])
+
qemu_disk_args = qemu_disk_args.replace('##ROOTFS_IMAGE##', deploy_dir_image + '/' + rootfs_image).split()
cmd = ['qemu-system-' + qemu_arch, '-m', '1024M']
@@ -72,10 +79,11 @@ def format_qemu_cmdline(arch, build, distro):
return cmd
-def start_qemu(arch, build, distro):
- cmdline = format_qemu_cmdline(arch, build, distro)
+def start_qemu(arch, build, distro, out, pid):
+ cmdline = format_qemu_cmdline(arch, build, distro, out, pid)
cmdline.insert(1, '-nographic')
+ print(cmdline)
p1 = subprocess.call(cmdline)
if __name__ == "__main__":
@@ -83,6 +91,8 @@ if __name__ == "__main__":
parser.add_argument('-a', '--arch', choices=['arm', 'arm64', 'amd64', 'i386'], help='set isar machine architecture.', default='arm')
parser.add_argument('-b', '--build', help='set path to build directory.', default=os.getcwd())
parser.add_argument('-d', '--distro', choices=['jessie', 'stretch'], help='set isar Debian distribution.', default='stretch')
+ parser.add_argument('-o', '--out', help='Route QEMU console output to specified file.')
+ parser.add_argument('-p', '--pid', help='Store QEMU pid to specified file.')
args = parser.parse_args()
- start_qemu(args.arch, args.build, args.distro)
+ start_qemu(args.arch, args.build, args.distro, args.out, args.pid)
diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index d4849c7..96e02f7 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -30,10 +30,9 @@ class VmBootTest(Test):
if os.path.exists(output_file):
os.remove(output_file)
- cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro)
+ cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro,
+ output_file, None)
cmdline.insert(1, '-nographic')
- cmdline.append('-serial')
- cmdline.append('file:' + output_file)
self.log.info('QEMU boot line: ' + str(cmdline))
--
2.25.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v12 03/16] start_vm.py: Add MIPS support
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 01/16] start_vm.py: Fix target name handling Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 02/16] start_vm.py: Add output and PID file vm_start.py options Anton Mikanovich
@ 2021-09-17 17:00 ` Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 04/16] start_vm.py: Fix ubuntu image name Anton Mikanovich
` (13 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-17 17:00 UTC (permalink / raw)
To: isar-users; +Cc: Roman Pletnev, Anton Mikanovich
From: Roman Pletnev <rpletnev@ilbers.de>
This patch adds mipsel to the list of supported architectures.
Signed-off-by: Roman Pletnev <rpletnev@ilbers.de>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/start_vm.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index 3736e51..face747 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -88,7 +88,7 @@ def start_qemu(arch, build, distro, out, pid):
if __name__ == "__main__":
parser = argparse.ArgumentParser()
- parser.add_argument('-a', '--arch', choices=['arm', 'arm64', 'amd64', 'i386'], help='set isar machine architecture.', default='arm')
+ parser.add_argument('-a', '--arch', choices=['arm', 'arm64', 'amd64', 'i386', 'mipsel'], help='set isar machine architecture.', default='arm')
parser.add_argument('-b', '--build', help='set path to build directory.', default=os.getcwd())
parser.add_argument('-d', '--distro', choices=['jessie', 'stretch'], help='set isar Debian distribution.', default='stretch')
parser.add_argument('-o', '--out', help='Route QEMU console output to specified file.')
--
2.25.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v12 04/16] start_vm.py: Fix ubuntu image name
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
` (2 preceding siblings ...)
2021-09-17 17:00 ` [PATCH v12 03/16] start_vm.py: Add MIPS support Anton Mikanovich
@ 2021-09-17 17:00 ` Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 05/16] vm_boot_test: Fix log file path in vm_boot_test Anton Mikanovich
` (12 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-17 17:00 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
Choose 'ubuntu' base name for 'focal' target, not 'debian'.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/start_vm.py | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index face747..8f0ccd6 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -30,8 +30,9 @@ def format_qemu_cmdline(arch, build, distro, out, pid):
image_type = get_bitbake_var(bb_output, 'IMAGE_TYPE')
deploy_dir_image = get_bitbake_var(bb_output, 'DEPLOY_DIR_IMAGE')
+ base = 'ubuntu' if distro in ['focal', 'bionic'] else 'debian'
if image_type == 'ext4-img':
- rootfs_image = 'isar-image-base-debian-' + distro + '-qemu' + arch + '.ext4.img'
+ rootfs_image = 'isar-image-base-' + base + '-' + distro + '-qemu' + arch + '.ext4.img'
kernel_image = deploy_dir_image + '/' + get_bitbake_var(bb_output, 'KERNEL_IMAGE')
initrd_image = get_bitbake_var(bb_output, 'INITRD_IMAGE')
@@ -47,7 +48,7 @@ def format_qemu_cmdline(arch, build, distro, out, pid):
extra_args = ['-kernel', kernel_image, '-initrd', initrd_image]
extra_args.extend(kargs)
elif image_type == 'wic-img':
- rootfs_image = 'isar-image-base-debian-' + distro + '-qemu' + arch + '.wic.img'
+ rootfs_image = 'isar-image-base-' + base + '-' + distro + '-qemu' + arch + '.wic.img'
extra_args = ['-snapshot']
else:
raise ValueError('Invalid image type: ' + str(image_type))
--
2.25.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v12 05/16] vm_boot_test: Fix log file path in vm_boot_test
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
` (3 preceding siblings ...)
2021-09-17 17:00 ` [PATCH v12 04/16] start_vm.py: Fix ubuntu image name Anton Mikanovich
@ 2021-09-17 17:00 ` Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 06/16] vm_boot_test: Remove external varianter Anton Mikanovich
` (11 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-17 17:00 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
Make log filename random and store it in build folder.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/vm_boot_test/vm_boot_test.py | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index 96e02f7..64dfda6 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -4,6 +4,7 @@ import os
import subprocess32
import sys
import time
+import tempfile
from os.path import dirname
sys.path.append(dirname(__file__) + '/..')
@@ -26,9 +27,9 @@ class VmBootTest(Test):
self.log.info('Isar build folder is: ' + build_dir)
self.log.info('===================================================')
- output_file = '/tmp/vm_boot_test.log'
- if os.path.exists(output_file):
- os.remove(output_file)
+ fd, output_file = tempfile.mkstemp(suffix='_log.txt',
+ prefix='vm_start_' + distro + '_' +
+ arch + '_', dir=build_dir, text=True)
cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro,
output_file, None)
--
2.25.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v12 06/16] vm_boot_test: Remove external varianter
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
` (4 preceding siblings ...)
2021-09-17 17:00 ` [PATCH v12 05/16] vm_boot_test: Fix log file path in vm_boot_test Anton Mikanovich
@ 2021-09-17 17:00 ` Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 07/16] vm_boot_test: Improve QEMU images checking Anton Mikanovich
` (10 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-17 17:00 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
To get rid of avocado-framework-plugin-varianter-yaml-to-mux pip
dependency we should not use yaml-to-mux parameters.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/vm_boot_test/run.sh | 3 --
testsuite/vm_boot_test/run_fast.sh | 3 ++
testsuite/vm_boot_test/run_full.sh | 3 ++
testsuite/vm_boot_test/variant.yaml | 22 -------------
testsuite/vm_boot_test/vm_boot_test.py | 45 +++++++++++++++++++++++---
5 files changed, 46 insertions(+), 30 deletions(-)
delete mode 100755 testsuite/vm_boot_test/run.sh
create mode 100755 testsuite/vm_boot_test/run_fast.sh
create mode 100755 testsuite/vm_boot_test/run_full.sh
delete mode 100644 testsuite/vm_boot_test/variant.yaml
diff --git a/testsuite/vm_boot_test/run.sh b/testsuite/vm_boot_test/run.sh
deleted file mode 100755
index 9fdda95..0000000
--- a/testsuite/vm_boot_test/run.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/bash
-
-avocado run vm_boot_test.py --mux-yaml test:variant.yaml --mux-inject build_dir:$BUILDDIR time_to_wait:300
diff --git a/testsuite/vm_boot_test/run_fast.sh b/testsuite/vm_boot_test/run_fast.sh
new file mode 100755
index 0000000..0fc77b3
--- /dev/null
+++ b/testsuite/vm_boot_test/run_fast.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+avocado run vm_boot_test.py -t fast -p build_dir="$BUILDDIR" -p time_to_wait=300
diff --git a/testsuite/vm_boot_test/run_full.sh b/testsuite/vm_boot_test/run_full.sh
new file mode 100755
index 0000000..a561a36
--- /dev/null
+++ b/testsuite/vm_boot_test/run_full.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+avocado run vm_boot_test.py -t full -p build_dir="$BUILDDIR" -p time_to_wait=300
diff --git a/testsuite/vm_boot_test/variant.yaml b/testsuite/vm_boot_test/variant.yaml
deleted file mode 100644
index 9ddc634..0000000
--- a/testsuite/vm_boot_test/variant.yaml
+++ /dev/null
@@ -1,22 +0,0 @@
-variants: !mux
- stretch-amd64:
- distro: "stretch"
- arch: "amd64"
- stretch-i386:
- distro: "stretch"
- arch: "i386"
- stretch-arm:
- distro: "stretch"
- arch: "arm"
- stretch-arm64:
- distro: "stretch"
- arch: "arm64"
- buster-amd64:
- distro: "buster"
- arch: "amd64"
- buster-i386:
- distro: "buster"
- arch: "i386"
- buster-arm:
- distro: "buster"
- arch: "arm"
diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index 64dfda6..eee14b2 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -13,13 +13,10 @@ import start_vm
from avocado import Test
-class VmBootTest(Test):
+class VmBase(Test):
- def test(self):
- # TODO: add default values
+ def vm_start(self, arch='amd64', distro='buster'):
build_dir = self.params.get('build_dir', default='.')
- arch = self.params.get('arch', default='arm')
- distro = self.params.get('distro', default='stretch')
time_to_wait = self.params.get('time_to_wait', default=60)
self.log.info('===================================================')
@@ -49,3 +46,41 @@ class VmBootTest(Test):
return
self.fail('Test failed')
+
+class VmBootTestFast(VmBase):
+
+ """
+ Test QEMU image start (fast)
+
+ :avocado: tags=fast,full
+ """
+ def test_arm_stretch(self):
+ self.vm_start('arm','stretch')
+
+ def test_arm_buster(self):
+ self.vm_start('arm','buster')
+
+ def test_arm64_stretch(self):
+ self.vm_start('arm64','stretch')
+
+ def test_amd64_stretch(self):
+ self.vm_start('amd64','stretch')
+
+class VmBootTestFull(VmBase):
+
+ """
+ Test QEMU image start (full)
+
+ :avocado: tags=full
+ """
+ def test_i386_stretch(self):
+ self.vm_start('i386','stretch')
+
+ def test_i386_buster(self):
+ self.vm_start('i386','buster')
+
+ def test_amd64_buster(self):
+ self.vm_start('amd64','buster')
+
+ def test_amd64_focal(self):
+ self.vm_start('amd64','focal')
--
2.25.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v12 07/16] vm_boot_test: Improve QEMU images checking
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
` (5 preceding siblings ...)
2021-09-17 17:00 ` [PATCH v12 06/16] vm_boot_test: Remove external varianter Anton Mikanovich
@ 2021-09-17 17:00 ` Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 08/16] build_test: Refactoring build tests cases Anton Mikanovich
` (9 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-17 17:00 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
Makes QEMU start test to analyze boot log in real-time. It helps test
cases to finish as soon as booting succeeds and do not wasting time on
waiting.
Get rid of python-subprocess32 backport package.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/vm_boot_test/vm_boot_test.py | 53 ++++++++++++++++++++------
1 file changed, 42 insertions(+), 11 deletions(-)
diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index eee14b2..6e580d0 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -1,7 +1,8 @@
#!/usr/bin/env python3
import os
-import subprocess32
+import select
+import subprocess
import sys
import time
import tempfile
@@ -13,6 +14,9 @@ import start_vm
from avocado import Test
+class CanBeFinished(Exception):
+ pass
+
class VmBase(Test):
def vm_start(self, arch='amd64', distro='buster'):
@@ -34,18 +38,45 @@ class VmBase(Test):
self.log.info('QEMU boot line: ' + str(cmdline))
- devnull = open(os.devnull, 'w')
-
- p1 = subprocess32.Popen(cmdline, stdout=devnull, stderr=devnull)
- time.sleep(int(time_to_wait))
- p1.kill()
- p1.wait()
+ login_prompt = b'isar login:'
+ service_prompt = b'Just an example'
+
+ timeout = time.time() + int(time_to_wait)
+
+ p1 = subprocess.Popen(cmdline, stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE)
+ try:
+ poller = select.poll()
+ poller.register(p1.stdout, select.POLLIN)
+ poller.register(p1.stderr, select.POLLIN)
+ while time.time() < timeout and p1.poll() is None:
+ events = poller.poll(1000 * (timeout - time.time()))
+ for fd, event in events:
+ if fd == p1.stdout.fileno():
+ # Wait for the complete string if it is read in chunks
+ # like "i", "sar", " login:"
+ time.sleep(0.01)
+ data = os.read(fd, 1024)
+ if login_prompt in data:
+ raise CanBeFinished
+ if fd == p1.stderr.fileno():
+ self.log.error(p1.stderr.readline())
+ except CanBeFinished:
+ self.log.debug('Got login prompt')
+ finally:
+ if p1.poll() is None:
+ p1.kill()
+ p1.wait()
if os.path.exists(output_file):
- if 'isar login:' in open(output_file).read():
- return
-
- self.fail('Test failed')
+ with open(output_file, "rb") as f1:
+ data = f1.read()
+ if service_prompt in data and login_prompt in data:
+ return
+ else:
+ self.log.error(data)
+
+ self.fail('Log ' + output_file)
class VmBootTestFast(VmBase):
--
2.25.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v12 08/16] build_test: Refactoring build tests cases
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
` (6 preceding siblings ...)
2021-09-17 17:00 ` [PATCH v12 07/16] vm_boot_test: Improve QEMU images checking Anton Mikanovich
@ 2021-09-17 17:00 ` Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 09/16] testsuite: Add Python generations for testsuite in gitignore Anton Mikanovich
` (8 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-17 17:00 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
Split build test into classes and separate cases. Implement test cases
tagging for external configuring. Also remove the usage of external
varianter plugin.
In this step, all ci_build test cases were copied as-is without
functional changes.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/build_test/build_test.py | 208 ++++++++++++++++++++++++++---
testsuite/build_test/cibase.py | 110 +++++++++++++++
testsuite/build_test/cibuilder.py | 130 ++++++++++++++++++
testsuite/build_test/run.sh | 3 -
testsuite/build_test/run_fast.sh | 3 +
testsuite/build_test/run_full.sh | 3 +
testsuite/build_test/variant.yaml | 22 ---
7 files changed, 434 insertions(+), 45 deletions(-)
create mode 100644 testsuite/build_test/cibase.py
create mode 100644 testsuite/build_test/cibuilder.py
delete mode 100755 testsuite/build_test/run.sh
create mode 100755 testsuite/build_test/run_fast.sh
create mode 100755 testsuite/build_test/run_full.sh
delete mode 100644 testsuite/build_test/variant.yaml
diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
index 7a55c2f..9038ec5 100644
--- a/testsuite/build_test/build_test.py
+++ b/testsuite/build_test/build_test.py
@@ -1,29 +1,197 @@
#!/usr/bin/env python3
import os
-import subprocess32
-import sys
-from os.path import dirname
-from avocado import Test
+from avocado import skipUnless
+from avocado.utils import path
+from cibase import CIBaseTest
-class BuildTest(Test):
+UMOCI_AVAILABLE = True
+SKOPEO_AVAILABLE = True
+try:
+ path.find_command('umoci')
+except path.CmdNotFoundError:
+ UMOCI_AVAILABLE = False
+try:
+ path.find_command('skopeo')
+except path.CmdNotFoundError:
+ SKOPEO_AVAILABLE = False
- def test(self):
- # TODO: add default values
- build_dir = self.params.get('build_dir', default='.')
- arch = self.params.get('arch', default='arm')
- distro = self.params.get('distro', default='stretch')
+class ReproTest(CIBaseTest):
- self.log.info('===================================================')
- self.log.info('Running Isar build test for (' + distro + '-' + arch + ')')
- self.log.info('Isar build folder is: ' + build_dir)
- self.log.info('===================================================')
+ """
+ Test cached base repository
- #isar_root = dirname(__file__) + '/..'
- os.chdir(build_dir)
- cmdline = ['bitbake', 'mc:qemu' + arch + '-' + distro + ':isar-image-base']
- p1 = subprocess32.run(cmdline)
+ :avocado: tags=repro,fast,full
+ """
+ def test_repro_signed(self):
+ targets = [
+ 'mc:de0-nano-soc-buster:isar-image-base',
+ 'mc:qemuarm64-stretch:isar-image-base'
+ ]
- if p1.returncode:
- self.fail('Test failed')
+ self.perform_repro_test(targets, 1)
+
+ def test_repro_unsigned(self):
+ targets = [
+ 'mc:qemuamd64-stretch:isar-image-base',
+ 'mc:qemuarm-stretch:isar-image-base'
+ ]
+
+ self.perform_repro_test(targets, 0)
+
+class CrossTest(CIBaseTest):
+
+ """
+ Start cross build for the defined set of configurations
+
+ :avocado: tags=cross,fast,full
+ """
+ def test_cross(self):
+ targets = [
+ 'mc:qemuarm-stretch:isar-image-base',
+ 'mc:qemuarm-buster:isar-image-base',
+ 'mc:qemuarm64-stretch:isar-image-base',
+ 'mc:qemuamd64-stretch:isar-image-base',
+ 'mc:de0-nano-soc-buster:isar-image-base',
+ 'mc:stm32mp15x-buster:isar-image-base',
+ 'mc:rpi-stretch:isar-image-base',
+ 'mc:qemuarm64-focal:isar-image-base'
+ ]
+
+ self.perform_build_test(targets, 1, None)
+
+ def test_cross_bullseye(self):
+ targets = [
+ 'mc:qemuarm-bullseye:isar-image-base'
+ ]
+
+ try:
+ self.perform_build_test(targets, 1, None)
+ except:
+ self.cancel('KFAIL')
+
+class SdkTest(CIBaseTest):
+
+ """
+ In addition test SDK creation
+
+ :avocado: tags=sdk,fast,full
+ """
+ def test_sdk(self):
+ targets = ['mc:qemuarm-stretch:isar-image-base']
+
+ self.perform_build_test(targets, 1, 'do_populate_sdk')
+
+class NoCrossTest(CIBaseTest):
+
+ """
+ Start non-cross build for the defined set of configurations
+
+ :avocado: tags=nocross,full
+ """
+ def test_nocross(self):
+ targets = [
+ 'mc:qemuarm-stretch:isar-image-base',
+ 'mc:qemuarm-buster:isar-image-base',
+ 'mc:qemuarm64-stretch:isar-image-base',
+ 'mc:qemui386-stretch:isar-image-base',
+ 'mc:qemui386-buster:isar-image-base',
+ 'mc:qemuamd64-stretch:isar-image-base',
+ 'mc:qemuamd64-buster:isar-image-base',
+ 'mc:qemuamd64-buster-tgz:isar-image-base',
+ 'mc:qemuamd64-buster:isar-initramfs',
+ 'mc:qemumipsel-stretch:isar-image-base',
+ 'mc:qemumipsel-buster:isar-image-base',
+ 'mc:nand-ubi-demo-buster:isar-image-ubi',
+ 'mc:rpi-stretch:isar-image-base',
+ 'mc:qemuamd64-focal:isar-image-base'
+ ]
+
+ # Cleanup after cross build
+ self.deletetmp(self.params.get('build_dir',
+ default=os.path.dirname(__file__) + '/../../build'))
+
+ self.perform_build_test(targets, 0, None)
+
+ def test_nocross_bullseye(self):
+ targets = [
+ 'mc:qemuamd64-bullseye:isar-image-base',
+ 'mc:qemuarm-bullseye:isar-image-base',
+ 'mc:qemui386-bullseye:isar-image-base',
+ 'mc:qemumipsel-bullseye:isar-image-base'
+ ]
+
+ try:
+ self.perform_build_test(targets, 0, None)
+ except:
+ self.cancel('KFAIL')
+
+class RebuildTest(CIBaseTest):
+
+ """
+ Test image rebuild
+
+ :avocado: tags=rebuild,fast,full
+ """
+ def test_rebuild(self):
+ is_cross_build = int(self.params.get('cross', default=0))
+
+ layerdir_core = self.getlayerdir('core')
+
+ dpkgbase_file = layerdir_core + '/classes/dpkg-base.bbclass'
+
+ self.backupfile(dpkgbase_file)
+ with open(dpkgbase_file, 'a') as file:
+ file.write('do_fetch_append() {\n\n}')
+
+ try:
+ self.perform_build_test('mc:qemuamd64-stretch:isar-image-base',
+ is_cross_build, None)
+ finally:
+ self.restorefile(dpkgbase_file)
+
+class WicTest(CIBaseTest):
+
+ """
+ Test wic --exclude-path
+
+ :avocado: tags=wic,fast,full
+ """
+ def test_wic_exclude(self):
+ # TODO: remove hardcoded filenames
+ wks_path = '/scripts/lib/wic/canned-wks/sdimage-efi.wks'
+ wic_path = '/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img'
+
+ self.perform_wic_test('mc:qemuamd64-stretch:isar-image-base',
+ wks_path, wic_path)
+
+class ContainerImageTest(CIBaseTest):
+
+ """
+ Test containerized images creation
+
+ :avocado: tags=containerbuild,fast,full,container
+ """
+ @skipUnless(UMOCI_AVAILABLE and SKOPEO_AVAILABLE, 'umoci/skopeo not found')
+ def test_nocross(self):
+ targets = [
+ 'mc:container-amd64-stretch:isar-image-base',
+ 'mc:container-amd64-buster:isar-image-base',
+ 'mc:container-amd64-bullseye:isar-image-base'
+ ]
+
+ self.perform_container_test(targets, None)
+
+class ContainerSdkTest(CIBaseTest):
+
+ """
+ Test SDK container image creation
+
+ :avocado: tags=containersdk,fast,full,container
+ """
+ @skipUnless(UMOCI_AVAILABLE and SKOPEO_AVAILABLE, 'umoci/skopeo not found')
+ def test_container_sdk(self):
+ targets = ['mc:container-amd64-stretch:isar-image-base']
+
+ self.perform_container_test(targets, 'do_populate_sdk')
diff --git a/testsuite/build_test/cibase.py b/testsuite/build_test/cibase.py
new file mode 100644
index 0000000..1a89f43
--- /dev/null
+++ b/testsuite/build_test/cibase.py
@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+
+import os
+import re
+import tempfile
+
+from cibuilder import CIBuilder
+from avocado.utils import process
+
+isar_root = os.path.dirname(__file__) + '/../..'
+
+class CIBaseTest(CIBuilder):
+
+ def prep(self, testname, targets, cross, debsrc_cache):
+ build_dir = self.params.get('build_dir', default=isar_root + '/build')
+ build_dir = os.path.realpath(build_dir)
+ quiet = int(self.params.get('quiet', default=0))
+ bitbake_args = '-v'
+
+ if quiet:
+ bitbake_args = ''
+
+ self.log.info('===================================================')
+ self.log.info('Running ' + testname + ' test for:')
+ self.log.info(targets)
+ self.log.info('Isar build folder is: ' + build_dir)
+ self.log.info('===================================================')
+
+ self.init(build_dir)
+ self.confprepare(build_dir, 1, cross, debsrc_cache)
+
+ return build_dir, bitbake_args;
+
+ def perform_build_test(self, targets, cross, bitbake_cmd):
+ build_dir, bb_args = self.prep('Isar build', targets, cross, 1)
+
+ self.log.info('Starting build...')
+
+ self.bitbake(build_dir, targets, bitbake_cmd, bb_args)
+
+ def perform_repro_test(self, targets, signed):
+ cross = int(self.params.get('cross', default=0))
+ build_dir, bb_args = self.prep('repro Isar build', targets, cross, 0)
+
+ gpg_pub_key = os.path.dirname(__file__) + '/../base-apt/test_pub.key'
+ gpg_priv_key = os.path.dirname(__file__) + '/../base-apt/test_priv.key'
+
+ if signed:
+ with open(build_dir + '/conf/ci_build.conf', 'a') as file:
+ # Enable use of signed cached base repository
+ file.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
+
+ os.chdir(build_dir)
+
+ os.environ['GNUPGHOME'] = tempfile.mkdtemp()
+ result = process.run('gpg --import %s %s' % (gpg_pub_key, gpg_priv_key))
+
+ if result.exit_status:
+ self.fail('GPG import failed')
+
+ self.bitbake(build_dir, targets, None, bb_args)
+
+ self.deletetmp(build_dir)
+ with open(build_dir + '/conf/ci_build.conf', 'a') as file:
+ file.write('ISAR_USE_CACHED_BASE_REPO = "1"\n')
+ file.write('BB_NO_NETWORK = "1"\n')
+
+ self.bitbake(build_dir, targets, None, bb_args)
+
+ # Cleanup and disable use of signed cached base repository
+ self.deletetmp(build_dir)
+ self.confcleanup(build_dir)
+
+ def perform_wic_test(self, targets, wks_path, wic_path):
+ cross = int(self.params.get('cross', default=0))
+ build_dir, bb_args = self.prep('WIC exclude build', targets, cross, 1)
+
+ layerdir_isar = self.getlayerdir('isar')
+
+ wks_file = layerdir_isar + wks_path
+ wic_img = build_dir + wic_path
+
+ if not os.path.isfile(wic_img):
+ self.fail('No build started before: ' + wic_img + ' not exist')
+
+ self.backupfile(wks_file)
+ self.backupmove(wic_img)
+
+ with open(wks_file, 'r') as file:
+ lines = file.readlines()
+ with open(wks_file, 'w') as file:
+ for line in lines:
+ file.write(re.sub(r'part \/ ', 'part \/ --exclude-path usr ',
+ line))
+
+ try:
+ self.bitbake(build_dir, targets, None, bb_args)
+ finally:
+ self.restorefile(wks_file)
+
+ self.restorefile(wic_img)
+
+ def perform_container_test(self, targets, bitbake_cmd):
+ cross = int(self.params.get('cross', default=0))
+ build_dir, bb_args = self.prep('Isar Container', targets, cross, 1)
+
+ self.containerprep(build_dir)
+
+ self.bitbake(build_dir, targets, bitbake_cmd, bb_args)
+
diff --git a/testsuite/build_test/cibuilder.py b/testsuite/build_test/cibuilder.py
new file mode 100644
index 0000000..187363f
--- /dev/null
+++ b/testsuite/build_test/cibuilder.py
@@ -0,0 +1,130 @@
+#!/usr/bin/env python3
+
+import logging
+import os
+import re
+import select
+import shutil
+import subprocess
+
+from avocado import Test
+from avocado.utils import path
+from avocado.utils import process
+
+isar_root = os.path.dirname(__file__) + '/../..'
+backup_prefix = '.ci-backup'
+
+app_log = logging.getLogger("avocado.app")
+
+class CIBuilder(Test):
+ def setUp(self):
+ super(CIBuilder, self).setUp()
+ job_log = os.path.join(os.path.dirname(self.logdir), '..', 'job.log')
+ self._file_handler = logging.FileHandler(filename=job_log)
+ self._file_handler.setLevel(logging.ERROR)
+ fmt = ('%(asctime)s %(module)-16.16s L%(lineno)-.4d %('
+ 'levelname)-5.5s| %(message)s')
+ formatter = logging.Formatter(fmt=fmt)
+ self._file_handler.setFormatter(formatter)
+ app_log.addHandler(self._file_handler)
+
+ def init(self, build_dir):
+ os.chdir(isar_root)
+ path.usable_rw_dir(build_dir)
+ output = process.getoutput('/bin/bash -c "source isar-init-build-env \
+ %s 2>&1 >/dev/null; env"' % build_dir)
+ env = dict(((x.split('=', 1) + [''])[:2] \
+ for x in output.splitlines() if x != ''))
+ os.environ.update(env)
+
+ def confprepare(self, build_dir, compat_arch, cross, debsrc_cache):
+ with open(build_dir + '/conf/ci_build.conf', 'w') as f:
+ if compat_arch:
+ f.write('ISAR_ENABLE_COMPAT_ARCH_amd64 = "1"\n')
+ f.write('ISAR_ENABLE_COMPAT_ARCH_arm64 = "1"\n')
+ f.write('ISAR_ENABLE_COMPAT_ARCH_debian-stretch_amd64 = "0"\n')
+ f.write('IMAGE_INSTALL += "kselftest"\n')
+ if cross:
+ f.write('ISAR_CROSS_COMPILE = "1"\n')
+ if debsrc_cache:
+ f.write('BASE_REPO_FEATURES = "cache-deb-src"\n')
+
+ with open(build_dir + '/conf/local.conf', 'r+') as f:
+ for line in f:
+ if 'include ci_build.conf' in line:
+ break
+ else:
+ f.write('\ninclude ci_build.conf')
+
+ def containerprep(self, build_dir):
+ with open(build_dir + '/conf/ci_build.conf', 'a') as f:
+ f.write('SDK_FORMATS = "docker-archive"\n')
+ f.write('IMAGE_INSTALL_remove = "example-module-${KERNEL_NAME} enable-fsck"\n')
+
+ def confcleanup(self, build_dir):
+ open(build_dir + '/conf/ci_build.conf', 'w').close()
+
+ def deletetmp(self, build_dir):
+ process.run('rm -rf ' + build_dir + '/tmp', sudo=True)
+
+ def bitbake(self, build_dir, target, cmd, args):
+ os.chdir(build_dir)
+ cmdline = ['bitbake']
+ if args:
+ cmdline.append(args)
+ if cmd:
+ cmdline.append('-c')
+ cmdline.append(cmd)
+ if isinstance(target, list):
+ cmdline.extend(target)
+ else:
+ cmdline.append(target)
+
+ with subprocess.Popen(" ".join(cmdline), stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE, universal_newlines=True,
+ shell=True) as p1:
+ poller = select.poll()
+ poller.register(p1.stdout, select.POLLIN)
+ poller.register(p1.stderr, select.POLLIN)
+ while p1.poll() is None:
+ events = poller.poll(1000)
+ for fd, event in events:
+ if fd == p1.stdout.fileno():
+ self.log.info(p1.stdout.readline().rstrip())
+ if fd == p1.stderr.fileno():
+ app_log.error(p1.stderr.readline().rstrip())
+ p1.wait()
+ if p1.returncode:
+ self.fail('Bitbake failed')
+
+ def backupfile(self, path):
+ try:
+ shutil.copy2(path, path + backup_prefix)
+ except FileNotFoundError:
+ self.log.warn(path + ' not exist')
+
+ def backupmove(self, path):
+ try:
+ shutil.move(path, path + backup_prefix)
+ except FileNotFoundError:
+ self.log.warn(path + ' not exist')
+
+ def restorefile(self, path):
+ try:
+ shutil.move(path + backup_prefix, path)
+ except FileNotFoundError:
+ self.log.warn(path + backup_prefix + ' not exist')
+
+ def getlayerdir(self, layer):
+ try:
+ path.find_command('bitbake')
+ except path.CmdNotFoundError:
+ build_dir = self.params.get('build_dir',
+ default=isar_root + '/build')
+ self.init(build_dir)
+ output = process.getoutput('bitbake -e | grep "^LAYERDIR_.*="')
+ env = dict(((x.split('=', 1) + [''])[:2] \
+ for x in output.splitlines() if x != ''))
+
+ return env['LAYERDIR_' + layer].strip('"')
+
diff --git a/testsuite/build_test/run.sh b/testsuite/build_test/run.sh
deleted file mode 100755
index a8ea9cc..0000000
--- a/testsuite/build_test/run.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/bin/bash
-
-avocado run build_test.py --mux-yaml test:variant.yaml --mux-inject build_dir:$BUILDDIR
diff --git a/testsuite/build_test/run_fast.sh b/testsuite/build_test/run_fast.sh
new file mode 100755
index 0000000..4d1bf44
--- /dev/null
+++ b/testsuite/build_test/run_fast.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+avocado run build_test.py -t fast -p quiet=1 -p cross=1
diff --git a/testsuite/build_test/run_full.sh b/testsuite/build_test/run_full.sh
new file mode 100755
index 0000000..af5ec59
--- /dev/null
+++ b/testsuite/build_test/run_full.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+avocado run build_test.py -t full -p quiet=1
diff --git a/testsuite/build_test/variant.yaml b/testsuite/build_test/variant.yaml
deleted file mode 100644
index 9ddc634..0000000
--- a/testsuite/build_test/variant.yaml
+++ /dev/null
@@ -1,22 +0,0 @@
-variants: !mux
- stretch-amd64:
- distro: "stretch"
- arch: "amd64"
- stretch-i386:
- distro: "stretch"
- arch: "i386"
- stretch-arm:
- distro: "stretch"
- arch: "arm"
- stretch-arm64:
- distro: "stretch"
- arch: "arm64"
- buster-amd64:
- distro: "buster"
- arch: "amd64"
- buster-i386:
- distro: "buster"
- arch: "i386"
- buster-arm:
- distro: "buster"
- arch: "arm"
--
2.25.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v12 09/16] testsuite: Add Python generations for testsuite in gitignore
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
` (7 preceding siblings ...)
2021-09-17 17:00 ` [PATCH v12 08/16] build_test: Refactoring build tests cases Anton Mikanovich
@ 2021-09-17 17:00 ` Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 10/16] testsuite: Fix test suite prepare guide Anton Mikanovich
` (7 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-17 17:00 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
Ignore Python byte code auto generated files inside testsuite.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/.gitignore | 1 +
1 file changed, 1 insertion(+)
create mode 100644 testsuite/.gitignore
diff --git a/testsuite/.gitignore b/testsuite/.gitignore
new file mode 100644
index 0000000..0d20b64
--- /dev/null
+++ b/testsuite/.gitignore
@@ -0,0 +1 @@
+*.pyc
--
2.25.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v12 10/16] testsuite: Fix test suite prepare guide
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
` (8 preceding siblings ...)
2021-09-17 17:00 ` [PATCH v12 09/16] testsuite: Add Python generations for testsuite in gitignore Anton Mikanovich
@ 2021-09-17 17:00 ` Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 11/16] gitlab-ci: Add Avocado build artifacts Anton Mikanovich
` (6 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-17 17:00 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
Remove deprecated yaml-to-mux plugin and yml variants from the doc.
Also remove subprocess32 backport installation which is not used.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/README.md | 21 +++------------------
1 file changed, 3 insertions(+), 18 deletions(-)
diff --git a/testsuite/README.md b/testsuite/README.md
index 5e64223..fcbea1e 100644
--- a/testsuite/README.md
+++ b/testsuite/README.md
@@ -4,29 +4,14 @@ The framework could be installed by using standard HOWTO:
https://github.com/avocado-framework/avocado#installing-with-standard-python-tools
-Then you need to install varianter yaml-to-mux plugin by following these instructions:
-
- https://github.com/avocado-framework/avocado/tree/master/optional_plugins
-
-## For Debian 9.x
-
- $ sudo apt-get install python-pip
- $ pip install --user subprocess32
- $ pip install --user avocado-framework
- $ pip install --user avocado-framework-plugin-varianter-yaml-to-mux
-
-# Pre
-
- $ export PATH=$PATH:~/.local/bin
- $ cd isar
- $ source isar-init-build-env
+## For Debian (tested on Debian 10.x)
+ $ sudo dpkg -i avocado_91.0_all.deb
# Run test
Each testsuite directory contains:
- - run.sh - script to start tests
- - variants.yaml - set of input data
+ - run_*.sh - script to start tests
- *.py - test case
# Other
--
2.25.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v12 11/16] gitlab-ci: Add Avocado build artifacts
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
` (9 preceding siblings ...)
2021-09-17 17:00 ` [PATCH v12 10/16] testsuite: Fix test suite prepare guide Anton Mikanovich
@ 2021-09-17 17:00 ` Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 12/16] gitlab-ci: Add debug flag Anton Mikanovich
` (5 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-17 17:00 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
Save build logs in case of Avocado CI build.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
.gitlab-ci.yml | 1 +
1 file changed, 1 insertion(+)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index c91a997..0760161 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -15,6 +15,7 @@ variables:
name: "logs-$CI_JOB_ID"
paths:
- build/tmp/work/*/*/*/temp
+ - build/job-results
when: on_failure
expire_in: 1 week
--
2.25.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v12 12/16] gitlab-ci: Add debug flag
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
` (10 preceding siblings ...)
2021-09-17 17:00 ` [PATCH v12 11/16] gitlab-ci: Add Avocado build artifacts Anton Mikanovich
@ 2021-09-17 17:00 ` Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 13/16] vm_boot_test: Add automatic bitbake init Anton Mikanovich
` (4 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-17 17:00 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
Gitlab users expect full build log output witch was enabled by default.
Debug flag was ignored and never used, now it becomes functional.
To keed compatibility add '-d' flag to ci_build.sh call.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
.gitlab-ci.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 0760161..fad4210 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -24,7 +24,7 @@ fast-ci:
except:
- schedules
script:
- - scripts/ci_build.sh -q -f
+ - scripts/ci_build.sh -q -f -d
full-ci:
<<: *common-build
@@ -34,7 +34,7 @@ full-ci:
- PREVIOUS_SHA="$(cat .CI_COMMIT_SHA || true)"
- if [ "$CI_COMMIT_SHA" != "$PREVIOUS_SHA" ]; then
echo "$CI_COMMIT_SHA" > .CI_COMMIT_SHA;
- scripts/ci_build.sh -q -c -r;
+ scripts/ci_build.sh -q -c -r -d;
fi
cache:
key: "$CI_COMMIT_REF_SLUG"
--
2.25.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v12 13/16] vm_boot_test: Add automatic bitbake init
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
` (11 preceding siblings ...)
2021-09-17 17:00 ` [PATCH v12 12/16] gitlab-ci: Add debug flag Anton Mikanovich
@ 2021-09-17 17:00 ` Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 14/16] build_test: Protect ubuntu target with KFAIL Anton Mikanovich
` (3 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-17 17:00 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
Implement source of isar-init-build-env in case there is no bitbake
command found.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/vm_boot_test/vm_boot_test.py | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py
index 6e580d0..01623c5 100644
--- a/testsuite/vm_boot_test/vm_boot_test.py
+++ b/testsuite/vm_boot_test/vm_boot_test.py
@@ -13,12 +13,25 @@ sys.path.append(dirname(__file__) + '/..')
import start_vm
from avocado import Test
+from avocado.utils import process
+from avocado.utils import path
class CanBeFinished(Exception):
pass
class VmBase(Test):
+ def check_bitbake(self, build_dir):
+ try:
+ path.find_command('bitbake')
+ except path.CmdNotFoundError:
+ out = process.getoutput('/bin/bash -c "cd ../.. && \
+ source isar-init-build-env \
+ %s 2>&1 >/dev/null; env"' % build_dir)
+ env = dict(((x.split('=', 1) + [''])[:2] \
+ for x in output.splitlines() if x != ''))
+ os.environ.update(env)
+
def vm_start(self, arch='amd64', distro='buster'):
build_dir = self.params.get('build_dir', default='.')
time_to_wait = self.params.get('time_to_wait', default=60)
@@ -28,6 +41,8 @@ class VmBase(Test):
self.log.info('Isar build folder is: ' + build_dir)
self.log.info('===================================================')
+ self.check_bitbake(build_dir)
+
fd, output_file = tempfile.mkstemp(suffix='_log.txt',
prefix='vm_start_' + distro + '_' +
arch + '_', dir=build_dir, text=True)
--
2.25.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v12 14/16] build_test: Protect ubuntu target with KFAIL
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
` (12 preceding siblings ...)
2021-09-17 17:00 ` [PATCH v12 13/16] vm_boot_test: Add automatic bitbake init Anton Mikanovich
@ 2021-09-17 17:00 ` Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 15/16] ci_build: Migrate to Avocado Anton Mikanovich
` (2 subsequent siblings)
16 siblings, 0 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-17 17:00 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
We experience the same QEMU-related semop issue on qemuarm64-focal:
| ERROR: mc:qemuarm64-focal:kselftest-5.6.13-r0 do_install_builddeps: Execution of '/build/tmp/work/ubuntu-focal-arm64/kselftest/5.6.13-r0/temp/run.do_install_builddeps.33809' failed with exit code 1:
| /bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
| hostname: Temporary failure in name resolution
| semop(1): encountered an error: Function not implemented
| Error in the build process: exit status 1
| dpkg: error: cannot access archive 'kselftest-build-deps_5.6.13_arm64.deb': No such file or directory
So we need to protect with KFAIL not only bullseye targets, but Ubuntu
Focal targets also.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/build_test/build_test.py | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
index 9038ec5..63ea301 100644
--- a/testsuite/build_test/build_test.py
+++ b/testsuite/build_test/build_test.py
@@ -55,12 +55,21 @@ class CrossTest(CIBaseTest):
'mc:qemuamd64-stretch:isar-image-base',
'mc:de0-nano-soc-buster:isar-image-base',
'mc:stm32mp15x-buster:isar-image-base',
- 'mc:rpi-stretch:isar-image-base',
- 'mc:qemuarm64-focal:isar-image-base'
+ 'mc:rpi-stretch:isar-image-base'
]
self.perform_build_test(targets, 1, None)
+ def test_cross_ubuntu(self):
+ targets = [
+ 'mc:qemuarm64-focal:isar-image-base'
+ ]
+
+ try:
+ self.perform_build_test(targets, 1, None)
+ except:
+ self.cancel('KFAIL')
+
def test_cross_bullseye(self):
targets = [
'mc:qemuarm-bullseye:isar-image-base'
--
2.25.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v12 15/16] ci_build: Migrate to Avocado
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
` (13 preceding siblings ...)
2021-09-17 17:00 ` [PATCH v12 14/16] build_test: Protect ubuntu target with KFAIL Anton Mikanovich
@ 2021-09-17 17:00 ` Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 16/16] vm_smoke_test: " Anton Mikanovich
2021-09-24 12:18 ` [PATCH v12 00/16] Update Avocado testsuite Henning Schild
16 siblings, 0 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-17 17:00 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
Call Avocado test cases instead of shell based.
Build artifacts are placed in build directory.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
scripts/ci_build.sh | 208 +++-------------------
scripts/ci_build_legacy.sh | 275 +++++++++++++++++++++++++++++
testsuite/build_test/build_test.py | 4 +-
testsuite/build_test/cibase.py | 10 +-
testsuite/build_test/cibuilder.py | 4 +-
5 files changed, 316 insertions(+), 185 deletions(-)
create mode 100755 scripts/ci_build_legacy.sh
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 48a51f2..4534957 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -14,6 +14,9 @@ export PATH=$PATH:/sbin
# Go to Isar root
cd "$(dirname "$0")/.."
+# Get Avocado build tests path
+BUILD_TEST_DIR="$(pwd)/testsuite/build_test"
+
# Start build in Isar tree by default
BUILD_DIR=./build
@@ -21,70 +24,10 @@ BUILD_DIR=./build
DEPENDENCIES="umoci skopeo"
for prog in ${DEPENDENCIES} ; do
if [ ! -x "$(which $prog)" ] ; then
- echo "missing $prog in PATH, exiting" >&2
- exit 1
+ echo "missing $prog in PATH" >&2
fi
done
-BB_ARGS="-v"
-
-TARGETS_SET="\
- mc:qemuarm-stretch:isar-image-base \
- mc:qemuarm-buster:isar-image-base \
- mc:qemuarm64-stretch:isar-image-base \
- mc:qemui386-stretch:isar-image-base \
- mc:qemui386-buster:isar-image-base \
- mc:qemuamd64-stretch:isar-image-base \
- mc:qemuamd64-buster:isar-image-base \
- mc:qemuamd64-buster-tgz:isar-image-base \
- mc:qemuamd64-buster-cpiogz:isar-image-base \
- mc:qemuamd64-buster:isar-initramfs \
- mc:qemumipsel-stretch:isar-image-base \
- mc:qemumipsel-buster:isar-image-base \
- mc:nand-ubi-demo-buster:isar-image-ubi \
- mc:rpi-stretch:isar-image-base \
- mc:qemuamd64-focal:isar-image-base \
- mc:virtualbox-ova-buster:isar-image-base \
- "
- # qemu-user-static of <= buster too old to build that
- # mc:qemuarm64-buster:isar-image-base
- # mc:qemuarm64-bullseye:isar-image-base
-
-TARGETS_SET_BULLSEYE="\
- mc:qemuamd64-bullseye:isar-image-base \
- mc:qemuarm-bullseye:isar-image-base \
- mc:qemui386-bullseye:isar-image-base \
- mc:qemumipsel-bullseye:isar-image-base \
-"
-
-TARGETS_CONTAINERS="\
- mc:container-amd64-stretch:isar-image-base \
- mc:container-amd64-buster:isar-image-base \
- mc:container-amd64-bullseye:isar-image-base \
-"
-
-CROSS_TARGETS_SET="\
- mc:qemuarm-stretch:isar-image-base \
- mc:qemuarm-buster:isar-image-base \
- mc:qemuarm64-stretch:isar-image-base \
- mc:qemuamd64-stretch:isar-image-base \
- mc:de0-nano-soc-buster:isar-image-base \
- mc:stm32mp15x-buster:isar-image-base \
- mc:rpi-stretch:isar-image-base \
- mc:qemuarm64-focal:isar-image-base"
-
-CROSS_TARGETS_SET_BULLSEYE="\
- mc:qemuarm-bullseye:isar-image-base \
-"
-
-REPRO_TARGETS_SET_SIGNED="\
- mc:de0-nano-soc-buster:isar-image-base \
- mc:qemuarm64-stretch:isar-image-base"
-
-REPRO_TARGETS_SET="\
- mc:qemuamd64-stretch:isar-image-base \
- mc:qemuarm-buster:isar-image-base"
-
show_help() {
echo "This script builds the default Isar images."
echo
@@ -106,6 +49,10 @@ show_help() {
echo " 3 if invalid parameters are passed."
}
+TAGS="full"
+CROSS_BUILD="0"
+QUIET="0"
+
# Parse command line to get user configuration
while [ $# -gt 0 ]
do
@@ -124,16 +71,16 @@ do
CROSS_BUILD="1"
;;
-d|--debug)
- BB_ARGS="$BB_ARGS -D"
+ VERBOSE="--show=app,test"
;;
-f|--fast)
# Start build for the reduced set of configurations
# Enforce cross-compilation to speed up the build
- FAST_BUILD="1"
+ TAGS="fast"
CROSS_BUILD="1"
;;
-q|--quiet)
- BB_ARGS=""
+ QUIET="1"
;;
-r|--repro)
REPRO_BUILD="1"
@@ -152,125 +99,24 @@ do
shift
done
+if [ -z "$REPRO_BUILD" ]; then
+ TAGS="$TAGS,-repro"
+fi
+
# the real stuff starts here, trace commands from now on
set -x
-# Setup build folder for the current build
-if [ ! -d "$BUILD_DIR" ]; then
- mkdir -p "$BUILD_DIR"
-fi
-source isar-init-build-env "$BUILD_DIR"
-
-cat >>conf/local.conf <<EOF
-ISAR_ENABLE_COMPAT_ARCH_amd64 = "1"
-ISAR_ENABLE_COMPAT_ARCH_arm64 = "1"
-ISAR_ENABLE_COMPAT_ARCH_debian-stretch_amd64 = "0"
-IMAGE_INSTALL += "kselftest"
+# Provide working path
+mkdir -p .config/avocado
+cat <<EOF > .config/avocado/avocado.conf
+[datadir.paths]
+base_dir = $BUILD_DIR/
+test_dir = $BUILD_DIR/tests
+data_dir = $BUILD_DIR/data
+logs_dir = $BUILD_DIR/job-results
EOF
+export VIRTUAL_ENV="./"
-if [ -n "$DISTRO_APT_PREMIRRORS" ]; then
- echo "DISTRO_APT_PREMIRRORS = \"$DISTRO_APT_PREMIRRORS\"" >> conf/local.conf
-fi
-
-if [ -n "$CROSS_BUILD" ]; then
- sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
-fi
-
-if [ -n "$REPRO_BUILD" ]; then
- ISAR_TESTSUITE_GPG_PUB_KEY_FILE="$TESTSUITEDIR/base-apt/test_pub.key"
- ISAR_TESTSUITE_GPG_PRIV_KEY_FILE="$TESTSUITEDIR/base-apt/test_priv.key"
- export GNUPGHOME=$(mktemp -d)
- gpg --import $ISAR_TESTSUITE_GPG_PUB_KEY_FILE $ISAR_TESTSUITE_GPG_PRIV_KEY_FILE
-
- # Enable use of signed cached base repository
- echo BASE_REPO_KEY=\"file://$ISAR_TESTSUITE_GPG_PUB_KEY_FILE\" >> conf/local.conf
- bitbake $BB_ARGS $REPRO_TARGETS_SET_SIGNED
- while [ -e bitbake.sock ]; do sleep 1; done
- sudo rm -rf tmp
- sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
- sed -i -e 's/^#BB_NO_NETWORK/BB_NO_NETWORK/g' conf/local.conf
- bitbake $BB_ARGS $REPRO_TARGETS_SET_SIGNED
- while [ -e bitbake.sock ]; do sleep 1; done
- # Cleanup and disable use of signed cached base repository
- sudo rm -rf tmp
- sed -i -e 's/ISAR_USE_CACHED_BASE_REPO ?= "1"/#ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
- sed -i -e 's/^BB_NO_NETWORK/#BB_NO_NETWORK/g' conf/local.conf
- sed -i -e 's/^BASE_REPO_KEY/#BASE_REPO_KEY/g' conf/local.conf
- bitbake $BB_ARGS $REPRO_TARGETS_SET
- while [ -e bitbake.sock ]; do sleep 1; done
- # Enable use of unsigned cached base repository
- sudo rm -rf tmp
- sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
- sed -i -e 's/^#BB_NO_NETWORK/BB_NO_NETWORK/g' conf/local.conf
- bitbake $BB_ARGS $REPRO_TARGETS_SET
- while [ -e bitbake.sock ]; do sleep 1; done
- # Disable use of unsigned cached base repository
- sed -i -e 's/ISAR_USE_CACHED_BASE_REPO ?= "1"/#ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
- sed -i -e 's/^BB_NO_NETWORK/#BB_NO_NETWORK/g' conf/local.conf
- # Try to build with changed configuration with no cleanup
- bitbake $BB_ARGS $REPRO_TARGETS_SET
- while [ -e bitbake.sock ]; do sleep 1; done
- # Cleanup
- sudo rm -rf tmp
-fi
-
-sed -i -e 's/^#BASE_REPO_FEATURES ?= "cache-deb-src"/BASE_REPO_FEATURES ?= "cache-deb-src"/g' conf/local.conf
-# Start cross build for the defined set of configurations
-sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
-bitbake $BB_ARGS $CROSS_TARGETS_SET
-while [ -e bitbake.sock ]; do sleep 1; done
-if bitbake $BB_ARGS $CROSS_TARGETS_SET_BULLSEYE; then
- echo "bullseye cross: PASSED"
-else
- echo "bullseye cross: KFAIL"
-fi
-# In addition test SDK creation
-bitbake $BB_ARGS -c do_populate_sdk mc:qemuarm-stretch:isar-image-base
-while [ -e bitbake.sock ]; do sleep 1; done
-
-if [ -z "$FAST_BUILD" ]; then
- # Cleanup and disable cross build
- sudo rm -rf tmp
- sed -i -e 's/ISAR_CROSS_COMPILE ?= "1"/ISAR_CROSS_COMPILE ?= "0"/g' conf/local.conf
- bitbake $BB_ARGS $TARGETS_SET
- while [ -e bitbake.sock ]; do sleep 1; done
-
- if bitbake $BB_ARGS $TARGETS_SET_BULLSEYE; then
- echo "bullseye: PASSED"
- else
- echo "bullseye: KFAIL"
- fi
- while [ -e bitbake.sock ]; do sleep 1; done
-fi
-
-eval $(bitbake -e | grep "^LAYERDIR_core=")
-eval $(bitbake -e | grep "^LAYERDIR_isar=")
-cp -a "${LAYERDIR_core}/classes/dpkg-base.bbclass" "${LAYERDIR_core}/classes/dpkg-base.bbclass.ci-backup"
-echo -e "do_fetch_append() {\n\n}" >> "${LAYERDIR_core}/classes/dpkg-base.bbclass"
-
-bitbake $BB_ARGS mc:qemuamd64-stretch:isar-image-base
-
-mv "${LAYERDIR_core}/classes/dpkg-base.bbclass.ci-backup" "${LAYERDIR_core}/classes/dpkg-base.bbclass"
-
-# Test wic --exclude-path
-cp -a "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks" "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks.ci-backup"
-mv ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img \
- ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img.ci-backup
-sed -i -e 's/part \/ /part \/ --exclude-path usr /g' "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks"
-
-bitbake $BB_ARGS mc:qemuamd64-stretch:isar-image-base
-
-mv "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks.ci-backup" "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks"
-mv ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img.ci-backup \
- ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img
-
-# Finalize with containerized images, since they remove some not-needed packages from the local.conf
-sed -i -e 's/\(IMAGE_INSTALL = .*\) example-module-${KERNEL_NAME}\(.*\)/\1\2/g' conf/local.conf
-sed -i -e 's/\(IMAGE_INSTALL = .*\) enable-fsck\(.*\)/\1\2/g' conf/local.conf
-bitbake $BB_ARGS $TARGETS_CONTAINERS
-while [ -e bitbake.sock ]; do sleep 1; done
-# and SDK container image creation
-echo 'SDK_FORMATS = "docker-archive"' >> conf/local.conf
-bitbake $BB_ARGS -c do_populate_sdk mc:container-amd64-stretch:isar-image-base
-while [ -e bitbake.sock ]; do sleep 1; done
-
+avocado $VERBOSE run "$BUILD_TEST_DIR/build_test.py" \
+ -t $TAGS --test-runner=runner --disable-sysinfo \
+ -p build_dir="$BUILD_DIR" -p quiet=$QUIET -p cross=$CROSS_BUILD
diff --git a/scripts/ci_build_legacy.sh b/scripts/ci_build_legacy.sh
new file mode 100755
index 0000000..d8418a9
--- /dev/null
+++ b/scripts/ci_build_legacy.sh
@@ -0,0 +1,275 @@
+#!/usr/bin/env bash
+# Script for CI system build
+#
+# Alexander Smirnov <asmirnov@ilbers.de>
+# Copyright (c) 2016-2018 ilbers GmbH
+
+set -e
+
+ES_BUG=3
+
+# Export $PATH to use 'parted' tool
+export PATH=$PATH:/sbin
+
+# Go to Isar root
+cd "$(dirname "$0")/.."
+
+# Start build in Isar tree by default
+BUILD_DIR=./build
+
+# Check dependencies
+DEPENDENCIES="umoci skopeo"
+for prog in ${DEPENDENCIES} ; do
+ if [ ! -x "$(which $prog)" ] ; then
+ echo "missing $prog in PATH, exiting" >&2
+ exit 1
+ fi
+done
+
+BB_ARGS="-v"
+
+TARGETS_SET="\
+ mc:qemuarm-stretch:isar-image-base \
+ mc:qemuarm-buster:isar-image-base \
+ mc:qemuarm64-stretch:isar-image-base \
+ mc:qemui386-stretch:isar-image-base \
+ mc:qemui386-buster:isar-image-base \
+ mc:qemuamd64-stretch:isar-image-base \
+ mc:qemuamd64-buster:isar-image-base \
+ mc:qemuamd64-buster-tgz:isar-image-base \
+ mc:qemuamd64-buster-cpiogz:isar-image-base \
+ mc:qemuamd64-buster:isar-initramfs \
+ mc:qemumipsel-stretch:isar-image-base \
+ mc:qemumipsel-buster:isar-image-base \
+ mc:nand-ubi-demo-buster:isar-image-ubi \
+ mc:rpi-stretch:isar-image-base \
+ mc:qemuamd64-focal:isar-image-base \
+ mc:virtualbox-ova-buster:isar-image-base \
+ "
+ # qemu-user-static of <= buster too old to build that
+ # mc:qemuarm64-buster:isar-image-base
+ # mc:qemuarm64-bullseye:isar-image-base
+
+TARGETS_SET_BULLSEYE="\
+ mc:qemuamd64-bullseye:isar-image-base \
+ mc:qemuarm-bullseye:isar-image-base \
+ mc:qemui386-bullseye:isar-image-base \
+ mc:qemumipsel-bullseye:isar-image-base \
+"
+
+TARGETS_CONTAINERS="\
+ mc:container-amd64-stretch:isar-image-base \
+ mc:container-amd64-buster:isar-image-base \
+ mc:container-amd64-bullseye:isar-image-base \
+"
+
+CROSS_TARGETS_SET="\
+ mc:qemuarm-stretch:isar-image-base \
+ mc:qemuarm-buster:isar-image-base \
+ mc:qemuarm64-stretch:isar-image-base \
+ mc:qemuamd64-stretch:isar-image-base \
+ mc:de0-nano-soc-buster:isar-image-base \
+ mc:stm32mp15x-buster:isar-image-base \
+ mc:rpi-stretch:isar-image-base \
+ mc:qemuarm64-focal:isar-image-base"
+
+CROSS_TARGETS_SET_BULLSEYE="\
+ mc:qemuarm-bullseye:isar-image-base \
+"
+
+REPRO_TARGETS_SET_SIGNED="\
+ mc:de0-nano-soc-buster:isar-image-base \
+ mc:qemuarm64-stretch:isar-image-base"
+
+REPRO_TARGETS_SET="\
+ mc:qemuamd64-stretch:isar-image-base \
+ mc:qemuarm-buster:isar-image-base"
+
+show_help() {
+ echo "This script builds the default Isar images."
+ echo
+ echo "Usage:"
+ echo " $0 [params]"
+ echo
+ echo "Parameters:"
+ echo " -b, --build BUILD_DIR set path to build directory. If not set,"
+ echo " the build will be started in current path."
+ echo " -c, --cross enable cross-compilation."
+ echo " -d, --debug enable debug bitbake output."
+ echo " -f, --fast cross build reduced set of configurations."
+ echo " -q, --quiet suppress verbose bitbake output."
+ echo " -r, --repro enable use of cached base repository."
+ echo " --help display this message and exit."
+ echo
+ echo "Exit status:"
+ echo " 0 if OK,"
+ echo " 3 if invalid parameters are passed."
+}
+
+# Parse command line to get user configuration
+while [ $# -gt 0 ]
+do
+ key="$1"
+
+ case $key in
+ -h|--help)
+ show_help
+ exit 0
+ ;;
+ -b|--build)
+ BUILD_DIR="$2"
+ shift
+ ;;
+ -c|--cross)
+ CROSS_BUILD="1"
+ ;;
+ -d|--debug)
+ BB_ARGS="$BB_ARGS -D"
+ ;;
+ -f|--fast)
+ # Start build for the reduced set of configurations
+ # Enforce cross-compilation to speed up the build
+ FAST_BUILD="1"
+ CROSS_BUILD="1"
+ ;;
+ -q|--quiet)
+ BB_ARGS=""
+ ;;
+ -r|--repro)
+ REPRO_BUILD="1"
+ # This switch is deprecated, just here to not cause failing CI on
+ # legacy configs
+ case "$2" in
+ -s|--sign) shift ;;
+ esac
+ ;;
+ *)
+ echo "error: invalid parameter '$key', please try '--help' to get list of supported parameters"
+ exit $ES_BUG
+ ;;
+ esac
+
+ shift
+done
+
+# the real stuff starts here, trace commands from now on
+set -x
+
+# Setup build folder for the current build
+if [ ! -d "$BUILD_DIR" ]; then
+ mkdir -p "$BUILD_DIR"
+fi
+source isar-init-build-env "$BUILD_DIR"
+
+cat >>conf/local.conf <<EOF
+ISAR_ENABLE_COMPAT_ARCH_amd64 = "1"
+ISAR_ENABLE_COMPAT_ARCH_arm64 = "1"
+ISAR_ENABLE_COMPAT_ARCH_debian-stretch_amd64 = "0"
+IMAGE_INSTALL += "kselftest"
+EOF
+
+if [ -n "$DISTRO_APT_PREMIRRORS" ]; then
+ echo "DISTRO_APT_PREMIRRORS = \"$DISTRO_APT_PREMIRRORS\"" >> conf/local.conf
+fi
+
+if [ -n "$CROSS_BUILD" ]; then
+ sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
+fi
+
+if [ -n "$REPRO_BUILD" ]; then
+ ISAR_TESTSUITE_GPG_PUB_KEY_FILE="$TESTSUITEDIR/base-apt/test_pub.key"
+ ISAR_TESTSUITE_GPG_PRIV_KEY_FILE="$TESTSUITEDIR/base-apt/test_priv.key"
+ export GNUPGHOME=$(mktemp -d)
+ gpg --import $ISAR_TESTSUITE_GPG_PUB_KEY_FILE $ISAR_TESTSUITE_GPG_PRIV_KEY_FILE
+
+ # Enable use of signed cached base repository
+ echo BASE_REPO_KEY=\"file://$ISAR_TESTSUITE_GPG_PUB_KEY_FILE\" >> conf/local.conf
+ bitbake $BB_ARGS $REPRO_TARGETS_SET_SIGNED
+ while [ -e bitbake.sock ]; do sleep 1; done
+ sudo rm -rf tmp
+ sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
+ sed -i -e 's/^#BB_NO_NETWORK/BB_NO_NETWORK/g' conf/local.conf
+ bitbake $BB_ARGS $REPRO_TARGETS_SET_SIGNED
+ while [ -e bitbake.sock ]; do sleep 1; done
+ # Cleanup and disable use of signed cached base repository
+ sudo rm -rf tmp
+ sed -i -e 's/ISAR_USE_CACHED_BASE_REPO ?= "1"/#ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
+ sed -i -e 's/^BB_NO_NETWORK/#BB_NO_NETWORK/g' conf/local.conf
+ sed -i -e 's/^BASE_REPO_KEY/#BASE_REPO_KEY/g' conf/local.conf
+ bitbake $BB_ARGS $REPRO_TARGETS_SET
+ while [ -e bitbake.sock ]; do sleep 1; done
+ # Enable use of unsigned cached base repository
+ sudo rm -rf tmp
+ sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
+ sed -i -e 's/^#BB_NO_NETWORK/BB_NO_NETWORK/g' conf/local.conf
+ bitbake $BB_ARGS $REPRO_TARGETS_SET
+ while [ -e bitbake.sock ]; do sleep 1; done
+ # Disable use of unsigned cached base repository
+ sed -i -e 's/ISAR_USE_CACHED_BASE_REPO ?= "1"/#ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
+ sed -i -e 's/^BB_NO_NETWORK/#BB_NO_NETWORK/g' conf/local.conf
+ # Try to build with changed configuration with no cleanup
+ bitbake $BB_ARGS $REPRO_TARGETS_SET
+ while [ -e bitbake.sock ]; do sleep 1; done
+ # Cleanup
+ sudo rm -rf tmp
+fi
+
+sed -i -e 's/^#BASE_REPO_FEATURES ?= "cache-deb-src"/BASE_REPO_FEATURES ?= "cache-deb-src"/g' conf/local.conf
+# Start cross build for the defined set of configurations
+sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
+bitbake $BB_ARGS $CROSS_TARGETS_SET
+while [ -e bitbake.sock ]; do sleep 1; done
+if bitbake $BB_ARGS $CROSS_TARGETS_SET_BULLSEYE; then
+ echo "bullseye cross: PASSED"
+else
+ echo "bullseye cross: KFAIL"
+fi
+# In addition test SDK creation
+bitbake $BB_ARGS -c do_populate_sdk mc:qemuarm-stretch:isar-image-base
+while [ -e bitbake.sock ]; do sleep 1; done
+
+if [ -z "$FAST_BUILD" ]; then
+ # Cleanup and disable cross build
+ sudo rm -rf tmp
+ sed -i -e 's/ISAR_CROSS_COMPILE ?= "1"/ISAR_CROSS_COMPILE ?= "0"/g' conf/local.conf
+ bitbake $BB_ARGS $TARGETS_SET
+ while [ -e bitbake.sock ]; do sleep 1; done
+
+ if bitbake $BB_ARGS $TARGETS_SET_BULLSEYE; then
+ echo "bullseye: PASSED"
+ else
+ echo "bullseye: KFAIL"
+ fi
+ while [ -e bitbake.sock ]; do sleep 1; done
+fi
+
+eval $(bitbake -e | grep "^LAYERDIR_core=")
+eval $(bitbake -e | grep "^LAYERDIR_isar=")
+cp -a "${LAYERDIR_core}/classes/dpkg-base.bbclass" "${LAYERDIR_core}/classes/dpkg-base.bbclass.ci-backup"
+echo -e "do_fetch_append() {\n\n}" >> "${LAYERDIR_core}/classes/dpkg-base.bbclass"
+
+bitbake $BB_ARGS mc:qemuamd64-stretch:isar-image-base
+
+mv "${LAYERDIR_core}/classes/dpkg-base.bbclass.ci-backup" "${LAYERDIR_core}/classes/dpkg-base.bbclass"
+
+# Test wic --exclude-path
+cp -a "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks" "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks.ci-backup"
+mv ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img \
+ ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img.ci-backup
+sed -i -e 's/part \/ /part \/ --exclude-path usr /g' "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks"
+
+bitbake $BB_ARGS mc:qemuamd64-stretch:isar-image-base
+
+mv "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks.ci-backup" "${LAYERDIR_isar}/scripts/lib/wic/canned-wks/sdimage-efi.wks"
+mv ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img.ci-backup \
+ ${BUILDDIR}/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img
+
+# Finalize with containerized images, since they remove some not-needed packages from the local.conf
+sed -i -e 's/\(IMAGE_INSTALL = .*\) example-module-${KERNEL_NAME}\(.*\)/\1\2/g' conf/local.conf
+sed -i -e 's/\(IMAGE_INSTALL = .*\) enable-fsck\(.*\)/\1\2/g' conf/local.conf
+bitbake $BB_ARGS $TARGETS_CONTAINERS
+while [ -e bitbake.sock ]; do sleep 1; done
+# and SDK container image creation
+echo 'SDK_FORMATS = "docker-archive"' >> conf/local.conf
+bitbake $BB_ARGS -c do_populate_sdk mc:container-amd64-stretch:isar-image-base
+while [ -e bitbake.sock ]; do sleep 1; done
diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
index 63ea301..d39c10c 100644
--- a/testsuite/build_test/build_test.py
+++ b/testsuite/build_test/build_test.py
@@ -109,12 +109,14 @@ class NoCrossTest(CIBaseTest):
'mc:qemuamd64-stretch:isar-image-base',
'mc:qemuamd64-buster:isar-image-base',
'mc:qemuamd64-buster-tgz:isar-image-base',
+ 'mc:qemuamd64-buster-cpiogz:isar-image-base',
'mc:qemuamd64-buster:isar-initramfs',
'mc:qemumipsel-stretch:isar-image-base',
'mc:qemumipsel-buster:isar-image-base',
'mc:nand-ubi-demo-buster:isar-image-ubi',
'mc:rpi-stretch:isar-image-base',
- 'mc:qemuamd64-focal:isar-image-base'
+ 'mc:qemuamd64-focal:isar-image-base',
+ 'mc:virtualbox-ova-buster:isar-image-base'
]
# Cleanup after cross build
diff --git a/testsuite/build_test/cibase.py b/testsuite/build_test/cibase.py
index 1a89f43..78d7bdb 100644
--- a/testsuite/build_test/cibase.py
+++ b/testsuite/build_test/cibase.py
@@ -67,10 +67,16 @@ class CIBaseTest(CIBuilder):
self.bitbake(build_dir, targets, None, bb_args)
- # Cleanup and disable use of signed cached base repository
- self.deletetmp(build_dir)
+ # Disable use of cached base repository
self.confcleanup(build_dir)
+ if not signed:
+ # Try to build with changed configuration with no cleanup
+ self.bitbake(build_dir, targets, None, bb_args)
+
+ # Cleanup
+ self.deletetmp(build_dir)
+
def perform_wic_test(self, targets, wks_path, wic_path):
cross = int(self.params.get('cross', default=0))
build_dir, bb_args = self.prep('WIC exclude build', targets, cross, 1)
diff --git a/testsuite/build_test/cibuilder.py b/testsuite/build_test/cibuilder.py
index 187363f..baa7185 100644
--- a/testsuite/build_test/cibuilder.py
+++ b/testsuite/build_test/cibuilder.py
@@ -48,6 +48,9 @@ class CIBuilder(Test):
f.write('ISAR_CROSS_COMPILE = "1"\n')
if debsrc_cache:
f.write('BASE_REPO_FEATURES = "cache-deb-src"\n')
+ distro_apt_premir = os.getenv('DISTRO_APT_PREMIRRORS')
+ if distro_apt_premir:
+ f.write('DISTRO_APT_PREMIRRORS = "%s"\n' % distro_apt_premir)
with open(build_dir + '/conf/local.conf', 'r+') as f:
for line in f:
@@ -127,4 +130,3 @@ class CIBuilder(Test):
for x in output.splitlines() if x != ''))
return env['LAYERDIR_' + layer].strip('"')
-
--
2.25.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v12 16/16] vm_smoke_test: Migrate to Avocado
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
` (14 preceding siblings ...)
2021-09-17 17:00 ` [PATCH v12 15/16] ci_build: Migrate to Avocado Anton Mikanovich
@ 2021-09-17 17:00 ` Anton Mikanovich
2021-09-24 12:18 ` [PATCH v12 00/16] Update Avocado testsuite Henning Schild
16 siblings, 0 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-17 17:00 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
Call Avocado test cases instead of shell based.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
scripts/vm_smoke_test | 120 ++++++-------------------
scripts/vm_smoke_test_legacy | 168 +++++++++++++++++++++++++++++++++++
2 files changed, 196 insertions(+), 92 deletions(-)
create mode 100755 scripts/vm_smoke_test_legacy
diff --git a/scripts/vm_smoke_test b/scripts/vm_smoke_test
index ce08213..6416250 100755
--- a/scripts/vm_smoke_test
+++ b/scripts/vm_smoke_test
@@ -3,9 +3,7 @@
# This software is a part of ISAR.
# Copyright (C) 2015-2018 ilbers GmbH
-CONSOLE_OUTPUT=/tmp/isar_console
-PID_FILE=/tmp/qemu.pid
-VERBOSE=1
+VERBOSE="--show=test"
TIMEOUT=300
# Error codes:
@@ -13,75 +11,15 @@ ES_OK=0
ES_FAIL=1
ES_BUG=3
-RET=$ES_OK
+RET=$ES_FAIL
-dump_boot_log() {
- echo "Boot log:\n8<--"
- cat $CONSOLE_OUTPUT
- echo "\n8<--"
-}
-
-check_login_prompt() {
- echo -n "Check login prompt: "
-
- str=$(grep "isar login: " $CONSOLE_OUTPUT)
-
- if [ -n "$str" ]; then
- echo "PASSED"
- else
- echo "FAIL"
- RET=$ES_FAIL
- FAIL=1
- fi
-}
-
-check_example_module() {
- echo -n "Check example module: "
-
- str=$(grep "Just an example" $CONSOLE_OUTPUT)
-
- if [ -n "$str" ]; then
- echo "PASSED"
- else
- echo "FAIL"
- RET=$ES_FAIL
- FAIL=1
- fi
-}
-
-run_test () {
- ARCH=$1
- DISTRO=$2
+# Get Avocado QEMU tests path
+VM_TEST_DIR="$(dirname "$0")/../testsuite/vm_boot_test"
- echo "-------------------------------------------------"
- echo "Testing Isar [$DISTRO] image for [$ARCH] machine:"
+# Go to Isar root
+cd "$(dirname "$0")/.."
- start_vm -a $ARCH -d $DISTRO -o $CONSOLE_OUTPUT -p $PID_FILE > /dev/null 2>&1 &
- sleep 5
-
- if [ -z `ps -p $! -o pid=` ]; then
- echo "QEMU start: FAILED"
- RET=$ES_FAIL
- echo "Command output:\n8<--"
- start_vm -a $ARCH -d $DISTRO -o $CONSOLE_OUTPUT -p $PID_FILE
- echo "\n8<--"
- else
- sleep $TIMEOUT
- kill `cat $PID_FILE`
-
- FAIL=0
-
- check_login_prompt
-
- check_example_module
-
- [ $VERBOSE -eq 1 -o $FAIL -eq 1 ] && dump_boot_log
-
- rm $CONSOLE_OUTPUT
- fi
-
- rm $PID_FILE
-}
+BUILD_DIR=./build
show_help() {
echo "This script tests the Isar images for default targets in QEMU."
@@ -91,10 +29,6 @@ show_help() {
echo
echo "Parameters:"
echo " -f,--fast test reduced set of supported targets."
- echo " -o,--output FILE specify file to store console output."
- echo " The default is: /tmp/isar_console"
- echo " -p,--pid-file FILE specify file to store QEMU process PID."
- echo " The default is: /tmp/qemu.pid"
echo " -q, --quiet do not display boot logs for all the targets."
echo " If test failed for the specific configuration,"
echo " the respective boot log will be printed anyway."
@@ -119,18 +53,18 @@ do
exit 0
;;
-o|--output)
- CONSOLE_OUTPUT=$2
+ # Deprecated option
shift
;;
-p|--pid-file)
- PID_FILE=$2
+ # Deprecated option
shift
;;
-f|--fast)
FAST_BUILD="1"
;;
-q|--quiet)
- VERBOSE=0
+ VERBOSE=""
;;
-t|--timeout)
TIMEOUT=$2
@@ -145,24 +79,26 @@ do
shift
done
-# ARM machine
-run_test arm stretch
-run_test arm buster
-
-# AMD64 machine
-if [ -z "$FAST_BUILD" ]; then
- run_test amd64 buster
- run_test amd64 focal
+TAGS="full"
+if [ -n "$FAST_BUILD" ]; then
+ TAGS="fast"
fi
-run_test amd64 stretch
-# i386 machine
-if [ -z "$FAST_BUILD" ]; then
- run_test i386 stretch
- run_test i386 buster
+# Provide working path
+mkdir -p .config/avocado
+cat <<EOF > .config/avocado/avocado.conf
+[datadir.paths]
+base_dir = $BUILD_DIR/
+test_dir = $BUILD_DIR/tests
+data_dir = $BUILD_DIR/data
+logs_dir = $BUILD_DIR/job-results
+EOF
+export VIRTUAL_ENV="./"
+
+if avocado $VERBOSE run "$VM_TEST_DIR/vm_boot_test.py" -t $TAGS \
+ --test-runner=runner --disable-sysinfo \
+ -p build_dir="$BUILD_DIR" -p time_to_wait=$TIMEOUT; then
+ RET=$ES_OK
fi
-# ARM64 machine
-run_test arm64 stretch
-
exit $RET
diff --git a/scripts/vm_smoke_test_legacy b/scripts/vm_smoke_test_legacy
new file mode 100755
index 0000000..ce08213
--- /dev/null
+++ b/scripts/vm_smoke_test_legacy
@@ -0,0 +1,168 @@
+#!/bin/sh
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2018 ilbers GmbH
+
+CONSOLE_OUTPUT=/tmp/isar_console
+PID_FILE=/tmp/qemu.pid
+VERBOSE=1
+TIMEOUT=300
+
+# Error codes:
+ES_OK=0
+ES_FAIL=1
+ES_BUG=3
+
+RET=$ES_OK
+
+dump_boot_log() {
+ echo "Boot log:\n8<--"
+ cat $CONSOLE_OUTPUT
+ echo "\n8<--"
+}
+
+check_login_prompt() {
+ echo -n "Check login prompt: "
+
+ str=$(grep "isar login: " $CONSOLE_OUTPUT)
+
+ if [ -n "$str" ]; then
+ echo "PASSED"
+ else
+ echo "FAIL"
+ RET=$ES_FAIL
+ FAIL=1
+ fi
+}
+
+check_example_module() {
+ echo -n "Check example module: "
+
+ str=$(grep "Just an example" $CONSOLE_OUTPUT)
+
+ if [ -n "$str" ]; then
+ echo "PASSED"
+ else
+ echo "FAIL"
+ RET=$ES_FAIL
+ FAIL=1
+ fi
+}
+
+run_test () {
+ ARCH=$1
+ DISTRO=$2
+
+ echo "-------------------------------------------------"
+ echo "Testing Isar [$DISTRO] image for [$ARCH] machine:"
+
+ start_vm -a $ARCH -d $DISTRO -o $CONSOLE_OUTPUT -p $PID_FILE > /dev/null 2>&1 &
+ sleep 5
+
+ if [ -z `ps -p $! -o pid=` ]; then
+ echo "QEMU start: FAILED"
+ RET=$ES_FAIL
+ echo "Command output:\n8<--"
+ start_vm -a $ARCH -d $DISTRO -o $CONSOLE_OUTPUT -p $PID_FILE
+ echo "\n8<--"
+ else
+ sleep $TIMEOUT
+ kill `cat $PID_FILE`
+
+ FAIL=0
+
+ check_login_prompt
+
+ check_example_module
+
+ [ $VERBOSE -eq 1 -o $FAIL -eq 1 ] && dump_boot_log
+
+ rm $CONSOLE_OUTPUT
+ fi
+
+ rm $PID_FILE
+}
+
+show_help() {
+ echo "This script tests the Isar images for default targets in QEMU."
+ echo
+ echo "Usage:"
+ echo " $0 [params]"
+ echo
+ echo "Parameters:"
+ echo " -f,--fast test reduced set of supported targets."
+ echo " -o,--output FILE specify file to store console output."
+ echo " The default is: /tmp/isar_console"
+ echo " -p,--pid-file FILE specify file to store QEMU process PID."
+ echo " The default is: /tmp/qemu.pid"
+ echo " -q, --quiet do not display boot logs for all the targets."
+ echo " If test failed for the specific configuration,"
+ echo " the respective boot log will be printed anyway."
+ echo " -t,--timeout SEC specify time in seconds to wait before stop QEMU."
+ echo " The default is: 300"
+ echo " -h, --help display this message and exit."
+ echo
+ echo "Exit status:"
+ echo " 0 if OK,"
+ echo " 1 if test failed,"
+ echo " 3 if invalid parameters are passed."
+}
+
+# Parse command line to get user configuration
+while [ $# -gt 0 ]
+do
+ key="$1"
+
+ case $key in
+ -h|--help)
+ show_help
+ exit 0
+ ;;
+ -o|--output)
+ CONSOLE_OUTPUT=$2
+ shift
+ ;;
+ -p|--pid-file)
+ PID_FILE=$2
+ shift
+ ;;
+ -f|--fast)
+ FAST_BUILD="1"
+ ;;
+ -q|--quiet)
+ VERBOSE=0
+ ;;
+ -t|--timeout)
+ TIMEOUT=$2
+ shift
+ ;;
+ *)
+ echo "error: invalid parameter '$key', please try '--help' to get list of supported parameters"
+ exit $ES_BUG
+ ;;
+ esac
+
+ shift
+done
+
+# ARM machine
+run_test arm stretch
+run_test arm buster
+
+# AMD64 machine
+if [ -z "$FAST_BUILD" ]; then
+ run_test amd64 buster
+ run_test amd64 focal
+fi
+run_test amd64 stretch
+
+# i386 machine
+if [ -z "$FAST_BUILD" ]; then
+ run_test i386 stretch
+ run_test i386 buster
+fi
+
+# ARM64 machine
+run_test arm64 stretch
+
+exit $RET
--
2.25.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v12 00/16] Update Avocado testsuite
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
` (15 preceding siblings ...)
2021-09-17 17:00 ` [PATCH v12 16/16] vm_smoke_test: " Anton Mikanovich
@ 2021-09-24 12:18 ` Henning Schild
2021-09-27 11:31 ` Anton Mikanovich
16 siblings, 1 reply; 20+ messages in thread
From: Henning Schild @ 2021-09-24 12:18 UTC (permalink / raw)
To: Anton Mikanovich; +Cc: isar-users
Am Fri, 17 Sep 2021 20:00:17 +0300
schrieb Anton Mikanovich <amikan@ilbers.de>:
> This patchset provides implementation of build and vm_start tests
> functionality in Python with help of Avocado framework.
> Scripts ci_build and vm_smoke_test are rewritten to call Avocado.
> ci_build and vm_smoke_test are left for compatibility and
> should be replaced by direct avocado calls later.
What to stay compatible with? I think the old one should be dropped
right when the new one comes. Or do we maintain both wile we have both?
I am all for dropping the old and merging this rather soon. Because any
change made is likely to add tests ... and will have merge conflicts
with this not being in the tree yet.
Henning
> Original ci_build and vm_smoke_test are renamed to *_legacy.
> Build and run logs are hidden by default now until an error occurs.
>
> ---
> Changes since v11:
> - Rebase on next.
> - Init environment in vm_smoke_test.
> Changes since v10:
> - Rebase on next.
> - Add DISTRO_APT_PREMIRRORS usage.
> - Remove avocado deb package install.
> - Set default runner.
> Changes since v9:
> - Rebase on next.
> - Fix avocado artifacts path.
> Changes since v8:
> - Remove duplication in logs.
> - Disable sysinfo plugin.
> - Show full errors in default mode.
> Changes since v7:
> - Remove code duplication.
> - Fix build_dir parsing.
> - Add missed protect files backup logic.
> Changes since v6:
> - Improve environment parsing.
> - Protect files backup logic.
> - Rebuild full log output option.
> - Protect ubuntu target with KFAIL.
> Changes since v5:
> - Rebase on current next.
> Changes since v4:
> - Add container tests.
> - Implement testcase skipping logic.
> - Separate and protect Bullseye tests.
> - Add checks for container dependencies.
> - Add Ilbers repo key.
> - Improve error handling.
> - Fix Ubuntu qemu image name.
> Changes since v3:
> - Add automatic bitbake init.
> - Fix line lengths.
> - Fix build without repro.
> - Fix build artifact paths.
> - Prepare for Gitlab build.
> - Change debug settings.
> - Move log splitting to start_vm.
> - Add deb package install.
> Changes since v2:
> - Make CI call Avocado tests.
> - Wait less on vm_start test.
> - Split Roman's vm_start commit.
> - Put logs to build folder.
> - Rearrange patches.
> - Split build test to classes.
> - Add test case tagging.
> - Get rid of avocado-framework-plugin-varianter-yaml-to-mux.
> - Get rid of python-subprocess32.
> - Improve logging.
> Changes since v1:
> - Merge start_vm rebuild patches.
> - Fix patch comments.
>
> Anton Mikanovich (13):
> start_vm.py: Fix ubuntu image name
> vm_boot_test: Fix log file path in vm_boot_test
> vm_boot_test: Remove external varianter
> vm_boot_test: Improve QEMU images checking
> build_test: Refactoring build tests cases
> testsuite: Add Python generations for testsuite in gitignore
> testsuite: Fix test suite prepare guide
> gitlab-ci: Add Avocado build artifacts
> gitlab-ci: Add debug flag
> vm_boot_test: Add automatic bitbake init
> build_test: Protect ubuntu target with KFAIL
> ci_build: Migrate to Avocado
> vm_smoke_test: Migrate to Avocado
>
> Roman Pletnev (3):
> start_vm.py: Fix target name handling
> start_vm.py: Add output and PID file vm_start.py options
> start_vm.py: Add MIPS support
>
> .gitlab-ci.yml | 5 +-
> scripts/ci_build.sh | 208 +++----------------
> scripts/ci_build_legacy.sh | 275
> +++++++++++++++++++++++++ scripts/vm_smoke_test |
> 120 +++-------- scripts/vm_smoke_test_legacy | 168
> +++++++++++++++ testsuite/.gitignore | 1 +
> testsuite/README.md | 21 +-
> testsuite/build_test/build_test.py | 219 ++++++++++++++++++--
> testsuite/build_test/cibase.py | 116 +++++++++++
> testsuite/build_test/cibuilder.py | 132 ++++++++++++
> testsuite/build_test/run.sh | 3 -
> testsuite/build_test/run_fast.sh | 3 +
> testsuite/build_test/run_full.sh | 3 +
> testsuite/build_test/variant.yaml | 22 --
> testsuite/start_vm.py | 27 ++-
> testsuite/vm_boot_test/run.sh | 3 -
> testsuite/vm_boot_test/run_fast.sh | 3 +
> testsuite/vm_boot_test/run_full.sh | 3 +
> testsuite/vm_boot_test/variant.yaml | 22 --
> testsuite/vm_boot_test/vm_boot_test.py | 121 +++++++++--
> 20 files changed, 1084 insertions(+), 391 deletions(-)
> create mode 100755 scripts/ci_build_legacy.sh
> create mode 100755 scripts/vm_smoke_test_legacy
> create mode 100644 testsuite/.gitignore
> create mode 100644 testsuite/build_test/cibase.py
> create mode 100644 testsuite/build_test/cibuilder.py
> delete mode 100755 testsuite/build_test/run.sh
> create mode 100755 testsuite/build_test/run_fast.sh
> create mode 100755 testsuite/build_test/run_full.sh
> delete mode 100644 testsuite/build_test/variant.yaml
> delete mode 100755 testsuite/vm_boot_test/run.sh
> create mode 100755 testsuite/vm_boot_test/run_fast.sh
> create mode 100755 testsuite/vm_boot_test/run_full.sh
> delete mode 100644 testsuite/vm_boot_test/variant.yaml
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v12 00/16] Update Avocado testsuite
2021-09-24 12:18 ` [PATCH v12 00/16] Update Avocado testsuite Henning Schild
@ 2021-09-27 11:31 ` Anton Mikanovich
2021-09-30 16:48 ` Anton Mikanovich
0 siblings, 1 reply; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-27 11:31 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On 24.09.21 15:18, Henning Schild wrote:
> Am Fri, 17 Sep 2021 20:00:17 +0300
> schrieb Anton Mikanovich <amikan@ilbers.de>:
>
>> This patchset provides implementation of build and vm_start tests
>> functionality in Python with help of Avocado framework.
>> Scripts ci_build and vm_smoke_test are rewritten to call Avocado.
>> ci_build and vm_smoke_test are left for compatibility and
>> should be replaced by direct avocado calls later.
> What to stay compatible with? I think the old one should be dropped
> right when the new one comes. Or do we maintain both wile we have both?
> I am all for dropping the old and merging this rather soon. Because any
> change made is likely to add tests ... and will have merge conflicts
> with this not being in the tree yet.
>
> Henning
I'm ok to remove old shell code, nobody would like to maintain both.
Just send v13 without it and hope to merge the patchset in next days.
--
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v12 00/16] Update Avocado testsuite
2021-09-27 11:31 ` Anton Mikanovich
@ 2021-09-30 16:48 ` Anton Mikanovich
0 siblings, 0 replies; 20+ messages in thread
From: Anton Mikanovich @ 2021-09-30 16:48 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On 27.09.21 14:31, Anton Mikanovich wrote:
> On 24.09.21 15:18, Henning Schild wrote:
>> Am Fri, 17 Sep 2021 20:00:17 +0300
>> schrieb Anton Mikanovich <amikan@ilbers.de>:
>>
>>> This patchset provides implementation of build and vm_start tests
>>> functionality in Python with help of Avocado framework.
>>> Scripts ci_build and vm_smoke_test are rewritten to call Avocado.
>>> ci_build and vm_smoke_test are left for compatibility and
>>> should be replaced by direct avocado calls later.
>> What to stay compatible with? I think the old one should be dropped
>> right when the new one comes. Or do we maintain both wile we have both?
>> I am all for dropping the old and merging this rather soon. Because any
>> change made is likely to add tests ... and will have merge conflicts
>> with this not being in the tree yet.
>>
>> Henning
>
> I'm ok to remove old shell code, nobody would like to maintain both.
> Just send v13 without it and hope to merge the patchset in next days.
>
Sorry, v13 was just the same as v12.
I've resent correct patchset as v14.
Please approve it is ok to merge.
--
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2021-09-30 16:49 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-17 17:00 [PATCH v12 00/16] Update Avocado testsuite Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 01/16] start_vm.py: Fix target name handling Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 02/16] start_vm.py: Add output and PID file vm_start.py options Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 03/16] start_vm.py: Add MIPS support Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 04/16] start_vm.py: Fix ubuntu image name Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 05/16] vm_boot_test: Fix log file path in vm_boot_test Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 06/16] vm_boot_test: Remove external varianter Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 07/16] vm_boot_test: Improve QEMU images checking Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 08/16] build_test: Refactoring build tests cases Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 09/16] testsuite: Add Python generations for testsuite in gitignore Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 10/16] testsuite: Fix test suite prepare guide Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 11/16] gitlab-ci: Add Avocado build artifacts Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 12/16] gitlab-ci: Add debug flag Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 13/16] vm_boot_test: Add automatic bitbake init Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 14/16] build_test: Protect ubuntu target with KFAIL Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 15/16] ci_build: Migrate to Avocado Anton Mikanovich
2021-09-17 17:00 ` [PATCH v12 16/16] vm_smoke_test: " Anton Mikanovich
2021-09-24 12:18 ` [PATCH v12 00/16] Update Avocado testsuite Henning Schild
2021-09-27 11:31 ` Anton Mikanovich
2021-09-30 16:48 ` Anton Mikanovich
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox