* [PATCH 0/8] Testsuite improvements for SSH-based tests
@ 2023-08-18 7:06 Uladzimir Bely
2023-08-18 7:06 ` [PATCH 1/8] cibuilder.py: Support custom arguments passing to CI scripts Uladzimir Bely
` (8 more replies)
0 siblings, 9 replies; 10+ messages in thread
From: Uladzimir Bely @ 2023-08-18 7:06 UTC (permalink / raw)
To: isar-users
This patchset expands testsuite capabilities in the field of
SSH-based tests:
1. Remote scripts can now accept additional arguments.
2. Return code, stdout and stderr from remote scripts and commands
can now be handled in the tests.
3. Several tests can now reuse the same qemu process and don't need
to start/kill it in each test.
Uladzimir Bely (8):
cibuilder.py: Support custom arguments passing to CI scripts
meta-isar: Add more extra space to qemu ext4 images
cibuilder.py: Split vm_start function to smaller subfunctions
cibuilder.py: Simplify remote_run command
cibuilder.py: Reuse the same qemu machine in ssh-based tests
citest.py: Adapt tests to qemu reuse
cibuilder.py: enable output from remote scripts
testsuite: Switch to remote scripts with arguments
meta-isar/conf/machine/qemuarm.conf | 1 +
meta-isar/conf/machine/qemuarm64.conf | 1 +
meta-isar/conf/machine/qemumipsel.conf | 1 +
meta-isar/conf/machine/qemuriscv64.conf | 1 +
testsuite/README.md | 2 +-
testsuite/cibuilder.py | 339 +++++++++++++++--------
testsuite/citest.py | 136 +++++----
testsuite/scripts/test_example_module.sh | 5 -
testsuite/scripts/test_getty_target.sh | 7 -
testsuite/scripts/test_kernel_module.sh | 5 +
testsuite/scripts/test_systemd_unit.sh | 23 ++
11 files changed, 335 insertions(+), 186 deletions(-)
delete mode 100755 testsuite/scripts/test_example_module.sh
delete mode 100755 testsuite/scripts/test_getty_target.sh
create mode 100755 testsuite/scripts/test_kernel_module.sh
create mode 100755 testsuite/scripts/test_systemd_unit.sh
--
2.20.1
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 1/8] cibuilder.py: Support custom arguments passing to CI scripts
2023-08-18 7:06 [PATCH 0/8] Testsuite improvements for SSH-based tests Uladzimir Bely
@ 2023-08-18 7:06 ` Uladzimir Bely
2023-08-18 7:07 ` [PATCH 2/8] meta-isar: Add more extra space to qemu ext4 images Uladzimir Bely
` (7 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Uladzimir Bely @ 2023-08-18 7:06 UTC (permalink / raw)
To: isar-users
With this change, it becomes possible to add custom arguments
to the scripts we want to run remotely on the target.
This means that remote script execution process is now consist
of two parts: delivering the script on board and running it
with custom arguments.
An interface of the call doesn't change just allowing to add
space-separated arguments:
```
self.vm_start(script='test_script arg1 arg2 .. argN')
```
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
testsuite/cibuilder.py | 16 ++++++++++++++--
1 file changed, 14 insertions(+), 2 deletions(-)
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index a03ff050..80b3aa88 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -246,13 +246,25 @@ class CIBuilder(Test):
def run_script(self, script, cmd_prefix):
script_dir = self.params.get('test_script_dir',
default=os.path.abspath(os.path.dirname(__file__))) + '/scripts/'
- script_path = script_dir + script
+ script_path = script_dir + script.split()[0]
+ script_args = ' '.join(script.split()[1:])
+
+ self.log.debug("script_path: '%s'" % (script_path))
+ self.log.debug("script args: '%s'" % (script_args))
+
if not os.path.exists(script_path):
self.log.error('Script not found: ' + script_path)
return 2
- rc = subprocess.call('cat ' + script_path + ' | ' + str(cmd_prefix), shell=True,
+ # Copy the script to the target
+ rc = subprocess.call('cat %s | %s install -m 755 /dev/stdin ./ci.sh' % (script_path, cmd_prefix), shell=True,
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ time.sleep(1)
+
+ # Run the script remotely with the arguments
+ if rc == 0:
+ rc = subprocess.call('%s ./ci.sh %s' % (cmd_prefix, script_args), shell=True,
+ stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
return rc
--
2.20.1
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 2/8] meta-isar: Add more extra space to qemu ext4 images
2023-08-18 7:06 [PATCH 0/8] Testsuite improvements for SSH-based tests Uladzimir Bely
2023-08-18 7:06 ` [PATCH 1/8] cibuilder.py: Support custom arguments passing to CI scripts Uladzimir Bely
@ 2023-08-18 7:07 ` Uladzimir Bely
2023-08-18 7:07 ` [PATCH 3/8] cibuilder.py: Split vm_start function to smaller subfunctions Uladzimir Bely
` (6 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Uladzimir Bely @ 2023-08-18 7:07 UTC (permalink / raw)
To: isar-users
Some qemu machines use IMAGE_FSTYPES="ext4". Default 64 MiB of extra
space passed to mke2fs seems to be not enough to deploy any files
on the target, because when the machine completes booting, `df` shows
0B available on rootfs.
This causes some SSH-bases tests fail with "no space left" error,
since we need test script to be first deployed on board and then
executed (with possible additional arguments).
Changing the default value of `ROOTFS_EXTRA` variable from 64 to 128
resolves the problem.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta-isar/conf/machine/qemuarm.conf | 1 +
meta-isar/conf/machine/qemuarm64.conf | 1 +
meta-isar/conf/machine/qemumipsel.conf | 1 +
meta-isar/conf/machine/qemuriscv64.conf | 1 +
4 files changed, 4 insertions(+)
diff --git a/meta-isar/conf/machine/qemuarm.conf b/meta-isar/conf/machine/qemuarm.conf
index 6426a4a7..b70d6af0 100644
--- a/meta-isar/conf/machine/qemuarm.conf
+++ b/meta-isar/conf/machine/qemuarm.conf
@@ -6,6 +6,7 @@ DISTRO_ARCH ?= "armhf"
KERNEL_NAME ?= "armmp"
IMAGE_FSTYPES ?= "ext4"
+ROOTFS_EXTRA ?= "128"
QEMU_ROOTFS_DEV ?= "vda"
diff --git a/meta-isar/conf/machine/qemuarm64.conf b/meta-isar/conf/machine/qemuarm64.conf
index f3b379c8..9706bbe0 100644
--- a/meta-isar/conf/machine/qemuarm64.conf
+++ b/meta-isar/conf/machine/qemuarm64.conf
@@ -8,6 +8,7 @@ KERNEL_NAME:ubuntu-focal ?= "generic"
KERNEL_NAME:ubuntu-jammy ?= "generic"
IMAGE_FSTYPES ?= "ext4"
+ROOTFS_EXTRA ?= "128"
QEMU_ROOTFS_DEV ?= "vda"
diff --git a/meta-isar/conf/machine/qemumipsel.conf b/meta-isar/conf/machine/qemumipsel.conf
index 40f5e744..4c27ff2f 100644
--- a/meta-isar/conf/machine/qemumipsel.conf
+++ b/meta-isar/conf/machine/qemumipsel.conf
@@ -6,6 +6,7 @@ DISTRO_ARCH ?= "mipsel"
KERNEL_NAME ?= "4kc-malta"
IMAGE_FSTYPES ?= "ext4"
+ROOTFS_EXTRA ?= "128"
QEMU_ROOTFS_DEV ?= "sda"
diff --git a/meta-isar/conf/machine/qemuriscv64.conf b/meta-isar/conf/machine/qemuriscv64.conf
index f7a7c8d5..15711054 100644
--- a/meta-isar/conf/machine/qemuriscv64.conf
+++ b/meta-isar/conf/machine/qemuriscv64.conf
@@ -9,6 +9,7 @@ DISTRO_ARCH ?= "riscv64"
KERNEL_NAME ?= "riscv64"
IMAGE_FSTYPES ?= "ext4"
+ROOTFS_EXTRA ?= "128"
QEMU_ROOTFS_DEV ?= "vda"
--
2.20.1
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 3/8] cibuilder.py: Split vm_start function to smaller subfunctions
2023-08-18 7:06 [PATCH 0/8] Testsuite improvements for SSH-based tests Uladzimir Bely
2023-08-18 7:06 ` [PATCH 1/8] cibuilder.py: Support custom arguments passing to CI scripts Uladzimir Bely
2023-08-18 7:07 ` [PATCH 2/8] meta-isar: Add more extra space to qemu ext4 images Uladzimir Bely
@ 2023-08-18 7:07 ` Uladzimir Bely
2023-08-18 7:07 ` [PATCH 4/8] cibuilder.py: Simplify remote_run command Uladzimir Bely
` (5 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Uladzimir Bely @ 2023-08-18 7:07 UTC (permalink / raw)
To: isar-users
vm_start is a pretty large function that does the following:
- starts machine in qemu;
- parses machine output (e.g., serial console);
- runs remote command over ssh, if requested;
- kills qemu process.
Splitting it to smaller sub-functions improves code readability and
potentially allows to reuse these sub-functions results in different
tests tests.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
testsuite/cibuilder.py | 186 +++++++++++++++++++++++------------------
1 file changed, 103 insertions(+), 83 deletions(-)
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 80b3aa88..8f28d05a 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -346,47 +346,70 @@ class CIBuilder(Test):
self.fail('No command to run specified')
- def vm_start(self, arch='amd64', distro='buster',
- enforce_pcbios=False, skip_modulecheck=False,
- image='isar-image-base', cmd=None, script=None):
- time_to_wait = self.params.get('time_to_wait', default=DEF_VM_TO_SEC)
-
- self.log.info('===================================================')
- self.log.info('Running Isar VM boot test for (' + distro + '-' + arch + ')')
- self.log.info('Remote command is ' + str(cmd))
- self.log.info('Remote script is ' + str(script))
- self.log.info('Isar build folder is: ' + self.build_dir)
- self.log.info('===================================================')
-
- self.check_init()
-
+ def vm_turn_on(self, arch='amd64', distro='buster', image='isar-image-base',
+ enforce_pcbios=False):
logdir = '%s/vm_start' % self.build_dir
if not os.path.exists(logdir):
os.mkdir(logdir)
prefix = '%s-vm_start_%s_%s_' % (time.strftime('%Y%m%d-%H%M%S'),
distro, arch)
- fd, output_file = tempfile.mkstemp(suffix='_log.txt', prefix=prefix,
+ fd, boot_log = tempfile.mkstemp(suffix='_log.txt', prefix=prefix,
dir=logdir, text=True)
- os.chmod(output_file, 0o644)
+ os.chmod(boot_log, 0o644)
latest_link = '%s/vm_start_%s_%s_latest.txt' % (logdir, distro, arch)
if os.path.exists(latest_link):
os.unlink(latest_link)
- os.symlink(os.path.basename(output_file), latest_link)
+ os.symlink(os.path.basename(boot_log), latest_link)
cmdline = start_vm.format_qemu_cmdline(arch, self.build_dir, distro, image,
- output_file, None, enforce_pcbios)
+ boot_log, None, enforce_pcbios)
cmdline.insert(1, '-nographic')
self.log.info('QEMU boot line:\n' + ' '.join(cmdline))
+ self.log.info('QEMU boot log:\n' + boot_log)
+
+ p1 = subprocess.Popen('exec ' + ' '.join(cmdline), shell=True,
+ stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ universal_newlines=True)
+ self.log.debug("Started VM with pid %s" % (p1.pid))
- login_prompt = b'isar login:'
+ return p1, cmdline, boot_log
+
+
+ def vm_wait_boot(self, p1, timeout):
+ login_prompt = b' login:'
+
+ poller = select.poll()
+ poller.register(p1.stdout, select.POLLIN)
+ poller.register(p1.stderr, select.POLLIN)
+
+ while time.time() < timeout and p1.poll() is None:
+ events = poller.poll(1000 * (timeout - time.time()))
+ for fd, event in events:
+ if event != select.POLLIN:
+ continue
+ if fd == p1.stdout.fileno():
+ # Wait for the complete string if it is read in chunks
+ # like "i", "sar", " login:"
+ time.sleep(0.01)
+ data = os.read(fd, 1024)
+ if login_prompt in data:
+ self.log.debug('Got login prompt')
+ return 0
+ if fd == p1.stderr.fileno():
+ app_log.error(p1.stderr.readline().rstrip())
+
+ self.log.error("Didn't get login prompt")
+ return 1
+
+
+ def vm_parse_output(self, boot_log, bb_output, skip_modulecheck):
# the printk of recipes-kernel/example-module
module_output = b'Just an example'
resize_output = None
-
- bb_output = start_vm.get_bitbake_env(arch, distro, image).decode()
image_fstypes = start_vm.get_bitbake_var(bb_output, 'IMAGE_FSTYPES')
wks_file = start_vm.get_bitbake_var(bb_output, 'WKS_FILE')
+
# only the first type will be tested in start_vm.py
if image_fstypes.split()[0] == 'wic':
if wks_file:
@@ -399,72 +422,69 @@ class CIBuilder(Test):
resize_output = b'resized filesystem to'
if "sdimage-efi-btrfs" in wks_file:
resize_output = b': resize device '
+ rc = 0
+ if os.path.exists(boot_log) and os.path.getsize(boot_log) > 0:
+ with open(boot_log, "rb") as f1:
+ data = f1.read()
+ if (module_output in data or skip_modulecheck):
+ if resize_output and not resize_output in data:
+ rc = 1
+ self.log.error("No resize output while expected")
+ else:
+ rc = 2
+ self.log.error("No example module output while expected")
+ return rc
- timeout = time.time() + int(time_to_wait)
- p1 = subprocess.Popen('exec ' + ' '.join(cmdline), shell=True,
- stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
- universal_newlines=True)
+ def vm_turn_off(self, p1):
+ if p1.poll() is None:
+ p1.kill()
+ p1.wait()
- if cmd is not None or script is not None:
- try:
- user='ci'
- host='localhost'
- port = 22
- for arg in cmdline:
- match = re.match(r".*hostfwd=tcp::(\d*).*", arg)
- if match:
- port = match.group(1)
- break
-
- rc = self.remote_run(user, host, port, cmd, script, p1, timeout)
-
- finally:
- if p1.poll() is None:
- self.log.debug('Killing qemu...')
- p1.kill()
- p1.wait()
+ self.log.debug("Stopped VM with pid %s" % (p1.pid))
- if rc != 0:
- self.fail('Log ' + output_file)
- return
+ def vm_start(self, arch='amd64', distro='buster',
+ enforce_pcbios=False, skip_modulecheck=False,
+ image='isar-image-base', cmd=None, script=None):
+ time_to_wait = self.params.get('time_to_wait', default=DEF_VM_TO_SEC)
- try:
- poller = select.poll()
- poller.register(p1.stdout, select.POLLIN)
- poller.register(p1.stderr, select.POLLIN)
- while time.time() < timeout and p1.poll() is None:
- events = poller.poll(1000 * (timeout - time.time()))
- for fd, event in events:
- if event != select.POLLIN:
- continue
- if fd == p1.stdout.fileno():
- # Wait for the complete string if it is read in chunks
- # like "i", "sar", " login:"
- time.sleep(0.01)
- data = os.read(fd, 1024)
- if login_prompt in data:
- raise CanBeFinished
- if fd == p1.stderr.fileno():
- app_log.error(p1.stderr.readline().rstrip())
- except CanBeFinished:
- self.log.debug('Got login prompt')
- finally:
- if p1.poll() is None:
- p1.kill()
- p1.wait()
+ self.log.info('===================================================')
+ self.log.info('Running Isar VM boot test for (' + distro + '-' + arch + ')')
+ self.log.info('Remote command is ' + str(cmd))
+ self.log.info('Remote script is ' + str(script))
+ self.log.info('Isar build folder is: ' + self.build_dir)
+ self.log.info('===================================================')
- if os.path.exists(output_file) and os.path.getsize(output_file) > 0:
- with open(output_file, "rb") as f1:
- data = f1.read()
- if (module_output in data or skip_modulecheck) \
- and login_prompt in data:
- if resize_output:
- if resize_output in data:
- return
- else:
- return
- app_log.error(data.decode(errors='replace'))
-
- self.fail('Log ' + output_file)
+ self.check_init()
+
+ timeout = time.time() + int(time_to_wait)
+
+ p1, cmdline, boot_log = self.vm_turn_on(arch, distro, image, enforce_pcbios)
+
+ rc = self.vm_wait_boot(p1, timeout)
+ if rc != 0:
+ self.vm_turn_off(p1)
+ self.fail('Failed to boot qemu machine')
+
+ if cmd is not None or script is not None:
+ user='ci'
+ host='localhost'
+ port = 22
+ for arg in cmdline:
+ match = re.match(r".*hostfwd=tcp::(\d*).*", arg)
+ if match:
+ port = match.group(1)
+ break
+ rc = self.remote_run(user, host, port, cmd, script, p1, timeout)
+ if rc != 0:
+ self.vm_turn_off(p1)
+ self.fail('Failed to run test over ssh')
+ else:
+ bb_output = start_vm.get_bitbake_env(arch, distro, image).decode()
+ rc = self.vm_parse_output(boot_log, bb_output, skip_modulecheck)
+ if rc != 0:
+ self.vm_turn_off(p1)
+ self.fail('Failed to parse output')
+
+ self.vm_turn_off(p1)
--
2.20.1
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 4/8] cibuilder.py: Simplify remote_run command
2023-08-18 7:06 [PATCH 0/8] Testsuite improvements for SSH-based tests Uladzimir Bely
` (2 preceding siblings ...)
2023-08-18 7:07 ` [PATCH 3/8] cibuilder.py: Split vm_start function to smaller subfunctions Uladzimir Bely
@ 2023-08-18 7:07 ` Uladzimir Bely
2023-08-18 7:07 ` [PATCH 5/8] cibuilder.py: Reuse the same qemu machine in ssh-based tests Uladzimir Bely
` (4 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Uladzimir Bely @ 2023-08-18 7:07 UTC (permalink / raw)
To: isar-users
Since we now parse qemu boot log in a separate small function even
in case of executing SSH-based test, we don't need to pass qemu-related
`proc` object to the ssh ping routine `wait_connection`.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
testsuite/cibuilder.py | 16 ++++++----------
1 file changed, 6 insertions(+), 10 deletions(-)
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 8f28d05a..c0bfb75c 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -268,10 +268,7 @@ class CIBuilder(Test):
return rc
- def wait_connection(self, proc, cmd_prefix, timeout):
- if proc is None:
- return 0
-
+ def wait_connection(self, cmd_prefix, timeout):
self.log.debug('Waiting for SSH server ready...')
rc = None
@@ -279,9 +276,6 @@ class CIBuilder(Test):
# Use 3 good SSH ping attempts to consider SSH connection is stable
while time.time() < timeout and goodcnt < 3:
goodcnt += 1
- if proc.poll() is not None:
- self.log.error('Machine is not running')
- return rc
rc = self.exec_cmd('/bin/true', cmd_prefix)
time.sleep(1)
@@ -305,11 +299,13 @@ class CIBuilder(Test):
return priv_key
- def remote_run(self, user, host, port, cmd, script, proc=None, timeout=0):
+ def remote_run(self, user, host, port, cmd, script, timeout=0):
priv_key = self.prepare_priv_key()
cmd_prefix = self.get_ssh_cmd_prefix(user, host, port, priv_key)
- rc = self.wait_connection(proc, cmd_prefix, timeout)
+ rc = 0
+ if timeout:
+ rc = self.wait_connection(cmd_prefix, timeout)
if rc == 0:
if cmd is not None:
@@ -476,7 +472,7 @@ class CIBuilder(Test):
if match:
port = match.group(1)
break
- rc = self.remote_run(user, host, port, cmd, script, p1, timeout)
+ rc = self.remote_run(user, host, port, cmd, script, timeout)
if rc != 0:
self.vm_turn_off(p1)
self.fail('Failed to run test over ssh')
--
2.20.1
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 5/8] cibuilder.py: Reuse the same qemu machine in ssh-based tests
2023-08-18 7:06 [PATCH 0/8] Testsuite improvements for SSH-based tests Uladzimir Bely
` (3 preceding siblings ...)
2023-08-18 7:07 ` [PATCH 4/8] cibuilder.py: Simplify remote_run command Uladzimir Bely
@ 2023-08-18 7:07 ` Uladzimir Bely
2023-08-18 7:07 ` [PATCH 6/8] citest.py: Adapt tests to qemu reuse Uladzimir Bely
` (3 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Uladzimir Bely @ 2023-08-18 7:07 UTC (permalink / raw)
To: isar-users
Every ssh-based test starts new qemu machine, runs remote command
and closes the machine.
This patch allows not to close (kill) the machine started by first
test and reuse it to run remote commands in the following tests and
save some time.
Since qemu machine now remains running by default, the last test for
this particular machine should care about closing it by passing
additonal `stop_vm=True` parameter to `vm_start()` function.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
testsuite/cibuilder.py | 75 ++++++++++++++++++++++++++++++++++--------
1 file changed, 61 insertions(+), 14 deletions(-)
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index c0bfb75c..73c8ee2c 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -2,9 +2,11 @@
import logging
import os
+import pickle
import re
import select
import shutil
+import signal
import subprocess
import time
import tempfile
@@ -52,6 +54,15 @@ class CIBuilder(Test):
for x in output.splitlines() if x != ''))
os.environ.update(env)
+ self.vm_dict = {}
+ self.vm_dict_file = '%s/vm_dict_file' % self.build_dir
+
+ if os.path.isfile(self.vm_dict_file):
+ with open(self.vm_dict_file, "rb") as f:
+ data = f.read()
+ if data:
+ self.vm_dict = pickle.loads(data)
+
def check_init(self):
if not hasattr(self, 'build_dir'):
self.error("Broken test implementation: need to call init().")
@@ -432,17 +443,26 @@ class CIBuilder(Test):
return rc
- def vm_turn_off(self, p1):
- if p1.poll() is None:
- p1.kill()
- p1.wait()
+ def vm_dump_dict(self, vm):
+ f = open(self.vm_dict_file, "wb")
+ pickle.dump(self.vm_dict, f)
+ f.close()
+
- self.log.debug("Stopped VM with pid %s" % (p1.pid))
+ def vm_turn_off(self, vm):
+ pid = self.vm_dict[vm][0]
+ os.kill(pid, signal.SIGKILL)
+
+ del(self.vm_dict[vm])
+ self.vm_dump_dict(vm)
+
+ self.log.debug("Stopped VM with pid %s" % (pid))
def vm_start(self, arch='amd64', distro='buster',
enforce_pcbios=False, skip_modulecheck=False,
- image='isar-image-base', cmd=None, script=None):
+ image='isar-image-base', cmd=None, script=None,
+ stop_vm=False):
time_to_wait = self.params.get('time_to_wait', default=DEF_VM_TO_SEC)
self.log.info('===================================================')
@@ -456,12 +476,36 @@ class CIBuilder(Test):
timeout = time.time() + int(time_to_wait)
- p1, cmdline, boot_log = self.vm_turn_on(arch, distro, image, enforce_pcbios)
+ vm = "%s_%s_%s_%d" % (arch, distro, image, enforce_pcbios)
+
+ p1 = None
+ pid = None
+ cmdline = ""
+ boot_log = ""
- rc = self.vm_wait_boot(p1, timeout)
- if rc != 0:
- self.vm_turn_off(p1)
- self.fail('Failed to boot qemu machine')
+ run_qemu = True
+
+ if vm in self.vm_dict:
+ pid, cmdline, boot_log = self.vm_dict[vm]
+
+ # Check that corresponding process exists
+ proc = subprocess.run("ps -o cmd= %d" % (pid), shell=True, text=True,
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+ if cmdline[0] in proc.stdout:
+ self.log.debug("Found '%s' process with pid '%d', use it" % (cmdline[0], pid))
+ run_qemu = False
+
+ if run_qemu:
+ self.log.debug("No qemu-system process for `%s` found, run new VM" % (vm))
+
+ p1, cmdline, boot_log = self.vm_turn_on(arch, distro, image, enforce_pcbios)
+ self.vm_dict[vm] = p1.pid, cmdline, boot_log
+ self.vm_dump_dict(vm)
+
+ rc = self.vm_wait_boot(p1, timeout)
+ if rc != 0:
+ self.vm_turn_off(vm)
+ self.fail('Failed to boot qemu machine')
if cmd is not None or script is not None:
user='ci'
@@ -474,13 +518,16 @@ class CIBuilder(Test):
break
rc = self.remote_run(user, host, port, cmd, script, timeout)
if rc != 0:
- self.vm_turn_off(p1)
+ if stop_vm:
+ self.vm_turn_off(vm)
self.fail('Failed to run test over ssh')
else:
bb_output = start_vm.get_bitbake_env(arch, distro, image).decode()
rc = self.vm_parse_output(boot_log, bb_output, skip_modulecheck)
if rc != 0:
- self.vm_turn_off(p1)
+ if stop_vm:
+ self.vm_turn_off(vm)
self.fail('Failed to parse output')
- self.vm_turn_off(p1)
+ if stop_vm:
+ self.vm_turn_off(vm)
--
2.20.1
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 6/8] citest.py: Adapt tests to qemu reuse
2023-08-18 7:06 [PATCH 0/8] Testsuite improvements for SSH-based tests Uladzimir Bely
` (4 preceding siblings ...)
2023-08-18 7:07 ` [PATCH 5/8] cibuilder.py: Reuse the same qemu machine in ssh-based tests Uladzimir Bely
@ 2023-08-18 7:07 ` Uladzimir Bely
2023-08-18 7:07 ` [PATCH 7/8] cibuilder.py: enable output from remote scripts Uladzimir Bely
` (2 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Uladzimir Bely @ 2023-08-18 7:07 UTC (permalink / raw)
To: isar-users
Reusing the same qemu machine in several tests requires stopping
the VM by the last one.
Adapt testsuite top level to fit this requirement.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
testsuite/citest.py | 136 +++++++++++++++++++++++++-------------------
1 file changed, 79 insertions(+), 57 deletions(-)
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 461b26c1..0cfb46ff 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -61,16 +61,18 @@ class DevTest(CIBaseTest):
def test_dev_run_amd64_bullseye(self):
self.init()
- self.vm_start('amd64', 'bullseye', \
- image='isar-image-ci')
+ self.vm_start('amd64', 'bullseye', image='isar-image-ci',
+ stop_vm=True)
def test_dev_run_arm64_bullseye(self):
self.init()
- self.vm_start('arm64', 'bullseye')
+ self.vm_start('arm64', 'bullseye',
+ stop_vm=True)
def test_dev_run_arm_bullseye(self):
self.init()
- self.vm_start('arm', 'bullseye', skip_modulecheck=True)
+ self.vm_start('arm', 'bullseye', skip_modulecheck=True,
+ stop_vm=True)
class ReproTest(CIBaseTest):
@@ -300,7 +302,9 @@ class SingleTest(CIBaseTest):
machine = self.params.get('machine', default='qemuamd64')
distro = self.params.get('distro', default='bullseye')
- self.vm_start(machine.removeprefix('qemu'), distro)
+ self.vm_start(machine.removeprefix('qemu'), distro,
+ stop_vm=True)
+
class VmBootTestFast(CIBaseTest):
@@ -309,50 +313,54 @@ class VmBootTestFast(CIBaseTest):
:avocado: tags=startvm,fast
"""
+
def test_arm_bullseye(self):
self.init()
- self.vm_start('arm','bullseye', \
- image='isar-image-ci')
+ self.vm_start('arm','bullseye', image='isar-image-ci')
def test_arm_bullseye_example_module(self):
self.init()
- self.vm_start('arm','bullseye', \
- image='isar-image-ci', cmd='lsmod | grep example_module')
+ self.vm_start('arm','bullseye', image='isar-image-ci',
+ cmd='lsmod | grep example_module')
def test_arm_bullseye_getty_target(self):
self.init()
- self.vm_start('arm','bullseye', \
- image='isar-image-ci', script='test_getty_target.sh')
+ self.vm_start('arm','bullseye', image='isar-image-ci',
+ script='test_getty_target.sh',
+ stop_vm=True)
+
def test_arm_buster(self):
self.init()
- self.vm_start('arm','buster', \
- image='isar-image-ci')
+ self.vm_start('arm','buster', image='isar-image-ci')
def test_arm_buster_getty_target(self):
self.init()
- self.vm_start('arm','buster', \
- image='isar-image-ci', cmd='systemctl is-active getty.target')
+ self.vm_start('arm','buster', image='isar-image-ci',
+ cmd='systemctl is-active getty.target')
def test_arm_buster_example_module(self):
self.init()
- self.vm_start('arm','buster', \
- image='isar-image-ci', script='test_example_module.sh')
+ self.vm_start('arm','buster', image='isar-image-ci',
+ script='test_example_module.sh',
+ stop_vm=True)
+
def test_arm_bookworm(self):
self.init()
- self.vm_start('arm','bookworm', \
- image='isar-image-ci')
+ self.vm_start('arm','bookworm', image='isar-image-ci')
def test_arm_bookworm_example_module(self):
self.init()
- self.vm_start('arm','bookworm', \
- image='isar-image-ci', cmd='lsmod | grep example_module')
+ self.vm_start('arm','bookworm', image='isar-image-ci',
+ cmd='lsmod | grep example_module')
def test_arm_bookworm_getty_target(self):
self.init()
- self.vm_start('arm','bookworm', \
- image='isar-image-ci', script='test_getty_target.sh')
+ self.vm_start('arm','bookworm', image='isar-image-ci',
+ script='test_getty_target.sh',
+ stop_vm=True)
+
class VmBootTestFull(CIBaseTest):
@@ -361,92 +369,106 @@ class VmBootTestFull(CIBaseTest):
:avocado: tags=startvm,full
"""
+
def test_arm_bullseye(self):
self.init()
- self.vm_start('arm','bullseye')
+ self.vm_start('arm','bullseye',
+ stop_vm=True)
+
def test_arm_buster(self):
self.init()
- self.vm_start('arm','buster', \
- image='isar-image-ci')
+ self.vm_start('arm','buster', image='isar-image-ci')
def test_arm_buster_example_module(self):
self.init()
- self.vm_start('arm','buster', \
- image='isar-image-ci', cmd='lsmod | grep example_module')
+ self.vm_start('arm','buster', image='isar-image-ci',
+ cmd='lsmod | grep example_module')
def test_arm_buster_getty_target(self):
self.init()
- self.vm_start('arm','buster', \
- image='isar-image-ci', script='test_getty_target.sh')
+ self.vm_start('arm','buster', image='isar-image-ci',
+ script='test_getty_target.sh',
+ stop_vm=True)
+
def test_arm64_bullseye(self):
self.init()
- self.vm_start('arm64','bullseye', \
- image='isar-image-ci')
+ self.vm_start('arm64','bullseye', image='isar-image-ci')
def test_arm64_bullseye_getty_target(self):
self.init()
- self.vm_start('arm64','bullseye', \
- image='isar-image-ci', cmd='systemctl is-active getty.target')
+ self.vm_start('arm64','bullseye', image='isar-image-ci',
+ cmd='systemctl is-active getty.target')
def test_arm64_bullseye_example_module(self):
self.init()
- self.vm_start('arm64','bullseye', \
- image='isar-image-ci', script='test_example_module.sh')
+ self.vm_start('arm64','bullseye', image='isar-image-ci',
+ script='test_example_module.sh',
+ stop_vm=True)
+
def test_i386_buster(self):
self.init()
- self.vm_start('i386','buster')
+ self.vm_start('i386','buster',
+ stop_vm=True)
+
def test_amd64_buster(self):
self.init()
# test efi boot
- self.vm_start('amd64','buster', \
- image='isar-image-ci')
+ self.vm_start('amd64','buster', image='isar-image-ci',
+ stop_vm=True)
# test pcbios boot
- self.vm_start('amd64', 'buster', True, \
- image='isar-image-ci')
+ self.vm_start('amd64', 'buster', True, image='isar-image-ci',
+ stop_vm=True)
+
def test_amd64_focal(self):
self.init()
- self.vm_start('amd64','focal', \
- image='isar-image-ci')
+ self.vm_start('amd64','focal', image='isar-image-ci')
def test_amd64_focal_example_module(self):
self.init()
- self.vm_start('amd64','focal', \
- image='isar-image-ci', cmd='lsmod | grep example_module')
+ self.vm_start('amd64','focal', image='isar-image-ci',
+ cmd='lsmod | grep example_module')
def test_amd64_focal_getty_target(self):
self.init()
- self.vm_start('amd64','focal', \
- image='isar-image-ci', script='test_getty_target.sh')
+ self.vm_start('amd64','focal', image='isar-image-ci',
+ script='test_getty_target.sh',
+ stop_vm=True)
+
def test_amd64_bookworm(self):
self.init()
- self.vm_start('amd64', 'bookworm', image='isar-image-ci')
+ self.vm_start('amd64', 'bookworm', image='isar-image-ci',
+ stop_vm=True)
+
def test_arm_bookworm(self):
self.init()
- self.vm_start('arm','bookworm', \
- image='isar-image-ci')
+ self.vm_start('arm','bookworm', image='isar-image-ci',
+ stop_vm=True)
+
def test_i386_bookworm(self):
self.init()
- self.vm_start('i386','bookworm')
+ self.vm_start('i386','bookworm',
+ stop_vm=True)
+
def test_mipsel_bookworm(self):
self.init()
- self.vm_start('mipsel','bookworm', \
- image='isar-image-ci')
+ self.vm_start('mipsel','bookworm', image='isar-image-ci')
def test_mipsel_bookworm_getty_target(self):
self.init()
- self.vm_start('mipsel','bookworm', \
- image='isar-image-ci', cmd='systemctl is-active getty.target')
+ self.vm_start('mipsel','bookworm', image='isar-image-ci',
+ cmd='systemctl is-active getty.target')
def test_mipsel_bookworm_example_module(self):
self.init()
- self.vm_start('mipsel','bookworm', \
- image='isar-image-ci', script='test_example_module.sh')
+ self.vm_start('mipsel','bookworm', image='isar-image-ci',
+ script='test_example_module.sh',
+ stop_vm=True)
--
2.20.1
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 7/8] cibuilder.py: enable output from remote scripts
2023-08-18 7:06 [PATCH 0/8] Testsuite improvements for SSH-based tests Uladzimir Bely
` (5 preceding siblings ...)
2023-08-18 7:07 ` [PATCH 6/8] citest.py: Adapt tests to qemu reuse Uladzimir Bely
@ 2023-08-18 7:07 ` Uladzimir Bely
2023-08-18 7:07 ` [PATCH 8/8] testsuite: Switch to remote scripts with arguments Uladzimir Bely
2023-08-24 15:34 ` [PATCH 0/8] Testsuite improvements for SSH-based tests Uladzimir Bely
8 siblings, 0 replies; 10+ messages in thread
From: Uladzimir Bely @ 2023-08-18 7:07 UTC (permalink / raw)
To: isar-users
With this patch, `ssh_start()` and `vm_start()` functions become able
to return stdout and stderr.
Function `remote_send_file()` can be used to deliver files on remote
machine.
Function `remote_run()` is allowed to just run remote commands and it
doesn't fail now when non-zero returncode. This allows to handle any
returncode in a testcase, e.g.:
```
def test_custom(self):
self.init()
self.vm_start('arm64', 'bullseye', image='isar-image-ci')
ret = self.remote_run(cmd='<custom_command>`)
if ret[0] != 0:
self.log.error(ret[2])
self.fail("Failed to run <custom_command>")
```
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
testsuite/cibuilder.py | 128 +++++++++++++++++++++++++----------------
1 file changed, 80 insertions(+), 48 deletions(-)
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 73c8ee2c..12f0e193 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -243,16 +243,25 @@ class CIBuilder(Test):
'-p %s -o IdentityFile=%s %s@%s ' \
% (port, priv_key, user, host)
- self.log.debug('Connect command:\n' + cmd_prefix)
-
return cmd_prefix
def exec_cmd(self, cmd, cmd_prefix):
- rc = subprocess.call('exec ' + str(cmd_prefix) + ' "' + str(cmd) + '"', shell=True,
- stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
- return rc
+ proc = subprocess.run('exec ' + str(cmd_prefix) + ' "' + str(cmd) + '"', shell=True,
+ stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+ return proc.returncode, proc.stdout, proc.stderr
+
+
+ def remote_send_file(self, src, dest, mode):
+ priv_key = self.prepare_priv_key()
+ cmd_prefix = self.get_ssh_cmd_prefix(self.ssh_user, self.ssh_host, self.ssh_port, priv_key)
+
+ proc = subprocess.run('cat %s | %s install -m %s /dev/stdin %s' %
+ (src, cmd_prefix, mode, dest), shell=True,
+ stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ return proc.returncode, proc.stdout, proc.stderr
def run_script(self, script, cmd_prefix):
script_dir = self.params.get('test_script_dir',
@@ -260,44 +269,45 @@ class CIBuilder(Test):
script_path = script_dir + script.split()[0]
script_args = ' '.join(script.split()[1:])
- self.log.debug("script_path: '%s'" % (script_path))
- self.log.debug("script args: '%s'" % (script_args))
-
if not os.path.exists(script_path):
self.log.error('Script not found: ' + script_path)
- return 2
+ return (2, '', 'Script not found: ' + script_path)
+
+ rc, stdout, stderr = self.remote_send_file(script_path, "./ci.sh", "755")
+
+ if rc != 0:
+ self.log.error('Failed to deploy the script on target')
+ return (rc, stdout, stderr)
- # Copy the script to the target
- rc = subprocess.call('cat %s | %s install -m 755 /dev/stdin ./ci.sh' % (script_path, cmd_prefix), shell=True,
- stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
time.sleep(1)
- # Run the script remotely with the arguments
- if rc == 0:
- rc = subprocess.call('%s ./ci.sh %s' % (cmd_prefix, script_args), shell=True,
- stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
- return rc
+ proc = subprocess.run('%s ./ci.sh %s' % (cmd_prefix, script_args), shell=True,
+ stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ return (proc.returncode, proc.stdout, proc.stderr)
def wait_connection(self, cmd_prefix, timeout):
- self.log.debug('Waiting for SSH server ready...')
+ self.log.info('Waiting for SSH server ready...')
rc = None
+ stdout = ""
+ stderr = ""
+
goodcnt = 0
# Use 3 good SSH ping attempts to consider SSH connection is stable
while time.time() < timeout and goodcnt < 3:
goodcnt += 1
- rc = self.exec_cmd('/bin/true', cmd_prefix)
+ rc, stdout, stderr = self.exec_cmd('/bin/true', cmd_prefix)
time.sleep(1)
if rc != 0:
goodcnt = 0
time_left = timeout - time.time()
- self.log.debug('SSH ping result: %d, left: %.fs' % (rc, time_left))
+ self.log.info('SSH ping result: %d, left: %.fs' % (rc, time_left))
- return rc
+ return rc, stdout, stderr
def prepare_priv_key(self):
@@ -310,45 +320,57 @@ class CIBuilder(Test):
return priv_key
- def remote_run(self, user, host, port, cmd, script, timeout=0):
+ def remote_run(self, cmd=None, script=None, timeout=0):
+ if cmd:
+ self.log.info('Remote command is `%s`' % (cmd))
+ if script:
+ self.log.info('Remote script is `%s`' % (script))
+
priv_key = self.prepare_priv_key()
- cmd_prefix = self.get_ssh_cmd_prefix(user, host, port, priv_key)
+ cmd_prefix = self.get_ssh_cmd_prefix(self.ssh_user, self.ssh_host, self.ssh_port, priv_key)
- rc = 0
- if timeout:
- rc = self.wait_connection(cmd_prefix, timeout)
+ rc = None
+ stdout = ""
+ stderr = ""
+
+ if timeout != 0:
+ rc, stdout, stderr = self.wait_connection(cmd_prefix, timeout)
- if rc == 0:
+ if rc == 0 or timeout == 0:
if cmd is not None:
- rc = self.exec_cmd(cmd, cmd_prefix)
- self.log.debug('`' + cmd + '` returned ' + str(rc))
+ rc, stdout, stderr = self.exec_cmd(cmd, cmd_prefix)
+ self.log.info('`' + cmd + '` returned ' + str(rc))
elif script is not None:
- rc = self.run_script(script, cmd_prefix)
- self.log.debug('`' + script + '` returned ' + str(rc))
- else:
- return None
+ rc, stdout, stderr = self.run_script(script, cmd_prefix)
+ self.log.info('`' + script + '` returned ' + str(rc))
- return rc
+ return rc, stdout, stderr
def ssh_start(self, user='ci', host='localhost', port=22,
cmd=None, script=None):
self.log.info('===================================================')
self.log.info('Running Isar SSH test for `%s@%s:%s`' % (user, host, port))
- self.log.info('Remote command is `%s`' % (cmd))
- self.log.info('Remote script is `%s`' % (script))
self.log.info('Isar build folder is: ' + self.build_dir)
self.log.info('===================================================')
self.check_init()
+ self.ssh_user = user
+ self.ssh_host = host
+ self.ssh_port = port
+
+ priv_key = self.prepare_priv_key()
+ cmd_prefix = self.get_ssh_cmd_prefix(self.ssh_user, self.ssh_host, self.ssh_port, priv_key)
+ self.log.info('Connect command:\n' + cmd_prefix)
+
if cmd is not None or script is not None:
- rc = self.remote_run(user, host, port, cmd, script)
+ rc, stdout, stderr = self.remote_run(cmd, script)
if rc != 0:
self.fail('Failed with rc=%s' % rc)
- return
+ return stdout, stderr
self.fail('No command to run specified')
@@ -378,7 +400,7 @@ class CIBuilder(Test):
p1 = subprocess.Popen('exec ' + ' '.join(cmdline), shell=True,
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
universal_newlines=True)
- self.log.debug("Started VM with pid %s" % (p1.pid))
+ self.log.info("Started VM with pid %s" % (p1.pid))
return p1, cmdline, boot_log
@@ -401,7 +423,7 @@ class CIBuilder(Test):
time.sleep(0.01)
data = os.read(fd, 1024)
if login_prompt in data:
- self.log.debug('Got login prompt')
+ self.log.info('Got login prompt')
return 0
if fd == p1.stderr.fileno():
app_log.error(p1.stderr.readline().rstrip())
@@ -456,7 +478,7 @@ class CIBuilder(Test):
del(self.vm_dict[vm])
self.vm_dump_dict(vm)
- self.log.debug("Stopped VM with pid %s" % (pid))
+ self.log.info("Stopped VM with pid %s" % (pid))
def vm_start(self, arch='amd64', distro='buster',
@@ -485,6 +507,9 @@ class CIBuilder(Test):
run_qemu = True
+ stdout = ""
+ stderr = ""
+
if vm in self.vm_dict:
pid, cmdline, boot_log = self.vm_dict[vm]
@@ -492,11 +517,11 @@ class CIBuilder(Test):
proc = subprocess.run("ps -o cmd= %d" % (pid), shell=True, text=True,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
if cmdline[0] in proc.stdout:
- self.log.debug("Found '%s' process with pid '%d', use it" % (cmdline[0], pid))
+ self.log.info("Found '%s' process with pid '%d', use it" % (cmdline[0], pid))
run_qemu = False
if run_qemu:
- self.log.debug("No qemu-system process for `%s` found, run new VM" % (vm))
+ self.log.info("No qemu-system process for `%s` found, run new VM" % (vm))
p1, cmdline, boot_log = self.vm_turn_on(arch, distro, image, enforce_pcbios)
self.vm_dict[vm] = p1.pid, cmdline, boot_log
@@ -508,15 +533,20 @@ class CIBuilder(Test):
self.fail('Failed to boot qemu machine')
if cmd is not None or script is not None:
- user='ci'
- host='localhost'
- port = 22
+ self.ssh_user='ci'
+ self.ssh_host='localhost'
+ self.ssh_port = 22
for arg in cmdline:
match = re.match(r".*hostfwd=tcp::(\d*).*", arg)
if match:
- port = match.group(1)
+ self.ssh_port = match.group(1)
break
- rc = self.remote_run(user, host, port, cmd, script, timeout)
+
+ priv_key = self.prepare_priv_key()
+ cmd_prefix = self.get_ssh_cmd_prefix(self.ssh_user, self.ssh_host, self.ssh_port, priv_key)
+ self.log.info('Connect command:\n' + cmd_prefix)
+
+ rc, stdout, stderr = self.remote_run(cmd, script, timeout)
if rc != 0:
if stop_vm:
self.vm_turn_off(vm)
@@ -531,3 +561,5 @@ class CIBuilder(Test):
if stop_vm:
self.vm_turn_off(vm)
+
+ return stdout, stderr
--
2.20.1
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 8/8] testsuite: Switch to remote scripts with arguments
2023-08-18 7:06 [PATCH 0/8] Testsuite improvements for SSH-based tests Uladzimir Bely
` (6 preceding siblings ...)
2023-08-18 7:07 ` [PATCH 7/8] cibuilder.py: enable output from remote scripts Uladzimir Bely
@ 2023-08-18 7:07 ` Uladzimir Bely
2023-08-24 15:34 ` [PATCH 0/8] Testsuite improvements for SSH-based tests Uladzimir Bely
8 siblings, 0 replies; 10+ messages in thread
From: Uladzimir Bely @ 2023-08-18 7:07 UTC (permalink / raw)
To: isar-users
Scripts `test_getty_target.sh` and `test_example_module.sh` are limited
to the only function.
Rename and refactor (add custom arguments support) them in order
to support any systemd unit / kernel module to check.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
testsuite/README.md | 2 +-
testsuite/citest.py | 14 +++++++-------
testsuite/scripts/test_example_module.sh | 5 -----
testsuite/scripts/test_getty_target.sh | 7 -------
testsuite/scripts/test_kernel_module.sh | 5 +++++
testsuite/scripts/test_systemd_unit.sh | 23 +++++++++++++++++++++++
6 files changed, 36 insertions(+), 20 deletions(-)
delete mode 100755 testsuite/scripts/test_example_module.sh
delete mode 100755 testsuite/scripts/test_getty_target.sh
create mode 100755 testsuite/scripts/test_kernel_module.sh
create mode 100755 testsuite/scripts/test_systemd_unit.sh
diff --git a/testsuite/README.md b/testsuite/README.md
index b58a1013..913767fc 100644
--- a/testsuite/README.md
+++ b/testsuite/README.md
@@ -87,7 +87,7 @@ can be executed instead of command:
self.init()
self.vm_start('amd64','bookworm', \
image='isar-image-ci',
- script='test_getty_target.sh')
+ script='test_systemd_unit.sh getty.target 10')
```
The default location of custom scripts is `isar/testsuite/`. It can be changed
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 0cfb46ff..90b8955f 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -326,7 +326,7 @@ class VmBootTestFast(CIBaseTest):
def test_arm_bullseye_getty_target(self):
self.init()
self.vm_start('arm','bullseye', image='isar-image-ci',
- script='test_getty_target.sh',
+ script='test_systemd_unit.sh getty.target 10',
stop_vm=True)
@@ -342,7 +342,7 @@ class VmBootTestFast(CIBaseTest):
def test_arm_buster_example_module(self):
self.init()
self.vm_start('arm','buster', image='isar-image-ci',
- script='test_example_module.sh',
+ script='test_kernel_module.sh example_module',
stop_vm=True)
@@ -358,7 +358,7 @@ class VmBootTestFast(CIBaseTest):
def test_arm_bookworm_getty_target(self):
self.init()
self.vm_start('arm','bookworm', image='isar-image-ci',
- script='test_getty_target.sh',
+ script='test_systemd_unit.sh getty.target 10',
stop_vm=True)
@@ -388,7 +388,7 @@ class VmBootTestFull(CIBaseTest):
def test_arm_buster_getty_target(self):
self.init()
self.vm_start('arm','buster', image='isar-image-ci',
- script='test_getty_target.sh',
+ script='test_systemd_unit.sh getty.target 10',
stop_vm=True)
@@ -404,7 +404,7 @@ class VmBootTestFull(CIBaseTest):
def test_arm64_bullseye_example_module(self):
self.init()
self.vm_start('arm64','bullseye', image='isar-image-ci',
- script='test_example_module.sh',
+ script='test_kernel_module.sh example_module',
stop_vm=True)
@@ -436,7 +436,7 @@ class VmBootTestFull(CIBaseTest):
def test_amd64_focal_getty_target(self):
self.init()
self.vm_start('amd64','focal', image='isar-image-ci',
- script='test_getty_target.sh',
+ script='test_systemd_unit.sh getty.target 10',
stop_vm=True)
@@ -470,5 +470,5 @@ class VmBootTestFull(CIBaseTest):
def test_mipsel_bookworm_example_module(self):
self.init()
self.vm_start('mipsel','bookworm', image='isar-image-ci',
- script='test_example_module.sh',
+ script='test_kernel_module.sh example_module',
stop_vm=True)
diff --git a/testsuite/scripts/test_example_module.sh b/testsuite/scripts/test_example_module.sh
deleted file mode 100755
index 175030d1..00000000
--- a/testsuite/scripts/test_example_module.sh
+++ /dev/null
@@ -1,5 +0,0 @@
-#!/bin/sh
-
-set -e
-
-lsmod | grep '^example_module '
diff --git a/testsuite/scripts/test_getty_target.sh b/testsuite/scripts/test_getty_target.sh
deleted file mode 100755
index 38c0fc1e..00000000
--- a/testsuite/scripts/test_getty_target.sh
+++ /dev/null
@@ -1,7 +0,0 @@
-#!/bin/sh
-
-set -e
-
-sleep 10
-
-systemctl is-active getty.target
diff --git a/testsuite/scripts/test_kernel_module.sh b/testsuite/scripts/test_kernel_module.sh
new file mode 100755
index 00000000..1e43e5cb
--- /dev/null
+++ b/testsuite/scripts/test_kernel_module.sh
@@ -0,0 +1,5 @@
+#!/bin/sh
+
+set -e
+
+lsmod | grep "^${1} "
diff --git a/testsuite/scripts/test_systemd_unit.sh b/testsuite/scripts/test_systemd_unit.sh
new file mode 100755
index 00000000..c97d7a7c
--- /dev/null
+++ b/testsuite/scripts/test_systemd_unit.sh
@@ -0,0 +1,23 @@
+#!/bin/sh
+
+# Checks system unit until it gets active or retry count reaches the limit
+
+set -e
+
+unit=$1
+cnt=$2
+
+ret=1
+
+while [ "$cnt" -gt 0 ]; do
+ if systemctl is-active "${unit}"; then
+ exit 0
+ else
+ ret=$?
+ fi
+
+ cnt=$((cnt - 1))
+ sleep 1
+done
+
+exit $ret
--
2.20.1
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 0/8] Testsuite improvements for SSH-based tests
2023-08-18 7:06 [PATCH 0/8] Testsuite improvements for SSH-based tests Uladzimir Bely
` (7 preceding siblings ...)
2023-08-18 7:07 ` [PATCH 8/8] testsuite: Switch to remote scripts with arguments Uladzimir Bely
@ 2023-08-24 15:34 ` Uladzimir Bely
8 siblings, 0 replies; 10+ messages in thread
From: Uladzimir Bely @ 2023-08-24 15:34 UTC (permalink / raw)
To: isar-users
On Fri, 2023-08-18 at 09:06 +0200, Uladzimir Bely wrote:
> This patchset expands testsuite capabilities in the field of
> SSH-based tests:
>
> 1. Remote scripts can now accept additional arguments.
>
> 2. Return code, stdout and stderr from remote scripts and commands
> can now be handled in the tests.
>
> 3. Several tests can now reuse the same qemu process and don't need
> to start/kill it in each test.
>
> Uladzimir Bely (8):
> cibuilder.py: Support custom arguments passing to CI scripts
> meta-isar: Add more extra space to qemu ext4 images
> cibuilder.py: Split vm_start function to smaller subfunctions
> cibuilder.py: Simplify remote_run command
> cibuilder.py: Reuse the same qemu machine in ssh-based tests
> citest.py: Adapt tests to qemu reuse
> cibuilder.py: enable output from remote scripts
> testsuite: Switch to remote scripts with arguments
>
> meta-isar/conf/machine/qemuarm.conf | 1 +
> meta-isar/conf/machine/qemuarm64.conf | 1 +
> meta-isar/conf/machine/qemumipsel.conf | 1 +
> meta-isar/conf/machine/qemuriscv64.conf | 1 +
> testsuite/README.md | 2 +-
> testsuite/cibuilder.py | 339 +++++++++++++++------
> --
> testsuite/citest.py | 136 +++++----
> testsuite/scripts/test_example_module.sh | 5 -
> testsuite/scripts/test_getty_target.sh | 7 -
> testsuite/scripts/test_kernel_module.sh | 5 +
> testsuite/scripts/test_systemd_unit.sh | 23 ++
> 11 files changed, 335 insertions(+), 186 deletions(-)
> delete mode 100755 testsuite/scripts/test_example_module.sh
> delete mode 100755 testsuite/scripts/test_getty_target.sh
> create mode 100755 testsuite/scripts/test_kernel_module.sh
> create mode 100755 testsuite/scripts/test_systemd_unit.sh
>
> --
> 2.20.1
>
Applied to next.
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2023-08-24 15:35 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-08-18 7:06 [PATCH 0/8] Testsuite improvements for SSH-based tests Uladzimir Bely
2023-08-18 7:06 ` [PATCH 1/8] cibuilder.py: Support custom arguments passing to CI scripts Uladzimir Bely
2023-08-18 7:07 ` [PATCH 2/8] meta-isar: Add more extra space to qemu ext4 images Uladzimir Bely
2023-08-18 7:07 ` [PATCH 3/8] cibuilder.py: Split vm_start function to smaller subfunctions Uladzimir Bely
2023-08-18 7:07 ` [PATCH 4/8] cibuilder.py: Simplify remote_run command Uladzimir Bely
2023-08-18 7:07 ` [PATCH 5/8] cibuilder.py: Reuse the same qemu machine in ssh-based tests Uladzimir Bely
2023-08-18 7:07 ` [PATCH 6/8] citest.py: Adapt tests to qemu reuse Uladzimir Bely
2023-08-18 7:07 ` [PATCH 7/8] cibuilder.py: enable output from remote scripts Uladzimir Bely
2023-08-18 7:07 ` [PATCH 8/8] testsuite: Switch to remote scripts with arguments Uladzimir Bely
2023-08-24 15:34 ` [PATCH 0/8] Testsuite improvements for SSH-based tests Uladzimir Bely
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox