[PATCH v2 0/9] Support running custom commands in VM

[PATCH v2 0/9] Support running custom commands in VM

[Uladzimir Bely]

VM run tests implementation in Isar testsuite allows only parsing
virtual machine boot logs, so the functionality is limited.

This patchset additionally allows to run custom commands on VM
that runs in qemu that allows to implement any custom tests.

The concept is the following.

For running custom tests SSH is used. So:

 - SSH server on machine should be configured, so `sshd-regen-keys`
package needs to be installed in the image.

 - Ethernet interface on machine should be automatically brought up
to make host-to-vm port forwarding working.

 - User in VM we use to connect via SSH should have public SSH key
in their `~/.ssh/` directory preinstalled.

 - Since base isar image don't have this included, custom image name
for vm run tests should be supported. This also allows to make reusing
Isar testsuite in downstream simpler.

Changes since v1:
 - Patches "meta-isar: Fix PCI bus initialization in qemuarm machine"
and "testsuite: Use random free port for qemu SSH forwarding" are
excluded from patchset since they are already merged.
 - User `ci` creation/configuration is moved to a separate package so
now it can be simply removed from target using apt. No new user flags
are required anymore.
 - In addition to simple commands (cmd="<command>"), running custom
local scripts (script="<script>.sh") on remote machine is implemented.
 - Appropriate documentation was added.
 - patchset rebased on top of "Add developers test" patchset which
is supposed to be merged earlier.

Notes:
 - Mixing positional and named arguments, e.g. 
|    self.vm_start('arm','bookworm',
|        image='isar-image-ci', script='test_getty_target.sh')
is still here, since it's easy to read.

Uladzimir Bely (9):
  testsuite: Fix failing hostname service in qemu guest
  testsuite: Add SSH key pair for using in CI
  meta-isar: Add a recipe that configures ci user
  meta-isar: Use a separate image recipe in CI
  testsuite: Allow custom image names in start_vm.py
  testsuite: Support running custom commands in VM
  testsuite: Support running custom scripts in VM
  testsuite: Run custom commands and scripts on some qemu targets
  testsuite: Update testsuite qemu-related documentation

 meta-isar/recipes-ci/images/isar-image-ci.bb  |   9 ++
 .../isar-ci-ssh-setup/files/postinst          |  18 +++
 .../isar-ci-ssh-setup_0.1.bb                  |  22 ++++
 testsuite/README.md                           |  41 ++++++-
 testsuite/cibuilder.py                        | 105 +++++++++++++++++-
 testsuite/citest.py                           | 105 +++++++++++++++---
 testsuite/keys/ssh/id_rsa                     |  38 +++++++
 testsuite/keys/ssh/id_rsa.pub                 |   1 +
 testsuite/start_vm.py                         |  23 ++--
 testsuite/test_example_module.sh              |   5 +
 testsuite/test_getty_target.sh                |   7 ++
 11 files changed, 341 insertions(+), 33 deletions(-)
 create mode 100644 meta-isar/recipes-ci/images/isar-image-ci.bb
 create mode 100644 meta-isar/recipes-ci/isar-ci-ssh-setup/files/postinst
 create mode 100644 meta-isar/recipes-ci/isar-ci-ssh-setup/isar-ci-ssh-setup_0.1.bb
 create mode 100644 testsuite/keys/ssh/id_rsa
 create mode 100644 testsuite/keys/ssh/id_rsa.pub
 create mode 100644 testsuite/test_example_module.sh
 create mode 100644 testsuite/test_getty_target.sh

-- 
2.20.1

[PATCH v2 1/9] testsuite: Fix failing hostname service in qemu guest

[Uladzimir Bely]

Hostname service in qemu may fail due to namespace issues. It happens
only when qemu subprocess run with `shell=False` (default).

```
systemctl status systemd-hostnamed.service
...
Failed to set up mount namespacing: /run/systemd/unit-root/dev:
     Read-only file system
Failed at step NAMESPACE spawning /lib/systemd/systemd-hostnamed:
    Read-only file system
Main process exited, code=exited, status=226/NAMESPACE
...
```
The issue was caught in a downstream, not Isar image itself.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/cibuilder.py | 2 +-
 testsuite/start_vm.py  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 72522f4d..5d82ec4b 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -262,7 +262,7 @@ class CIBuilder(Test):
 
         timeout = time.time() + int(time_to_wait)
 
-        p1 = subprocess.Popen(cmdline,
+        p1 = subprocess.Popen('exec ' + ' '.join(cmdline), shell=True,
                               stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
                               universal_newlines=True)
         try:
diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index 82ecc17d..c3726f14 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -96,7 +96,7 @@ def start_qemu(arch, build, distro, out, pid, enforce_pcbios):
     cmdline.insert(1, '-nographic')
 
     print(cmdline)
-    p1 = subprocess.call(cmdline)
+    p1 = subprocess.call('exec ' + ' '.join(cmdline), shell=True)
 
 if __name__ == "__main__":
     parser = argparse.ArgumentParser()
-- 
2.20.1

[PATCH v2 2/9] testsuite: Add SSH key pair for using in CI

[Uladzimir Bely]

These SSH keys are generated with empty passphrase and supposed
to be used for non-interactive SSH access to qemu machines
testsuite runs.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/keys/ssh/id_rsa     | 38 +++++++++++++++++++++++++++++++++++
 testsuite/keys/ssh/id_rsa.pub |  1 +
 2 files changed, 39 insertions(+)
 create mode 100644 testsuite/keys/ssh/id_rsa
 create mode 100644 testsuite/keys/ssh/id_rsa.pub

diff --git a/testsuite/keys/ssh/id_rsa b/testsuite/keys/ssh/id_rsa
new file mode 100644
index 00000000..7556b63b
--- /dev/null
+++ b/testsuite/keys/ssh/id_rsa
@@ -0,0 +1,38 @@
+-----BEGIN OPENSSH PRIVATE KEY-----
+b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn
+NhAAAAAwEAAQAAAYEAyM3aszWLWO/0Ztqi+VNusHpsUbffmUk17BoKmIqv51dK4TyNjkpq
+XyvjvOv98lXMzEfqnwAH3oL0H1ZCaS0qnVuqQ6bcL3NXe3beh2H6aMBbvwlY+rwXlcFuxH
+wGby3U8asH6dx+ZTu0skOaJKRNgCAOPGM3CVft2YPJWoaxBLtLsp/t2xcncL2hhgCtDa0y
+TVTQn/3vW/napXpjvNNtZdoDNBpto6lXCDYUREj0e3q3AD4svJIn9yLoJ6q8kMZL+ry+Cd
+FKLrwsEEj+J8dsATbAlVV0a5tV1aGvMwd4vQBDPtTyMTQel8/LGLC96pw1K7nH4c/kVyyO
+80O2eVpp95lFqb/mlE/lUM/KRPOGkmsG5gcMKEwAfmVEK5ZuIS3zt5Jp0AWWOi77qhL09e
+OhveLFXfb2Q+57VYXQvrqdgOasm1fNPQjscNDK+a3CssYnLz+mb3Fc7lYk5NTqCTvms+UP
+Yn9iYegiV3anxM5BximXoYdRIsvb8Pudtpo+9+FDAAAFkL99Ho+/fR6PAAAAB3NzaC1yc2
+EAAAGBAMjN2rM1i1jv9GbaovlTbrB6bFG335lJNewaCpiKr+dXSuE8jY5Kal8r47zr/fJV
+zMxH6p8AB96C9B9WQmktKp1bqkOm3C9zV3t23odh+mjAW78JWPq8F5XBbsR8Bm8t1PGrB+
+ncfmU7tLJDmiSkTYAgDjxjNwlX7dmDyVqGsQS7S7Kf7dsXJ3C9oYYArQ2tMk1U0J/971v5
+2qV6Y7zTbWXaAzQabaOpVwg2FERI9Ht6twA+LLySJ/ci6CeqvJDGS/q8vgnRSi68LBBI/i
+fHbAE2wJVVdGubVdWhrzMHeL0AQz7U8jE0HpfPyxiwveqcNSu5x+HP5FcsjvNDtnlaafeZ
+Ram/5pRP5VDPykTzhpJrBuYHDChMAH5lRCuWbiEt87eSadAFljou+6oS9PXjob3ixV329k
+Pue1WF0L66nYDmrJtXzT0I7HDQyvmtwrLGJy8/pm9xXO5WJOTU6gk75rPlD2J/YmHoIld2
+p8TOQcYpl6GHUSLL2/D7nbaaPvfhQwAAAAMBAAEAAAGBALQibsmLBAlxoLbP9Zlirg5klF
+0ubMiOs2+s4Mp8x1XpqgOqFXaK7SeZMMBel73YGFM9RI0EMwr6QNMHx6WgXfUxsOjle+qZ
+NuZL+U5Lp55mySbz1L2BFEtEbgCXHhKIc9mjywKQyVY5HF06ZcHvHpMMrpifjFS2a2lQXs
+X1xEDxX/PKok6DU99ATfDvFE8Liu9yYYDVBO9mkS0XQPe4VxAiZfNMrgQ+OhPkp4OmSqOm
+rxnprKoxFucNimNjN6tMJm2WLNTtFydVcZIjjpGirPSntlJUT2jVB+CzpJlIj+5yhMOsga
+Ej6brkBB1384Y7gEhKhcgvejhaVHPR9Q3RVykOclKh8SWr44k9uRrOW4vxkMmIYh/13EvO
+SXac4V1p19ebmU5BShMK4OfIJZoouVxmlgw5CfXdSmeWFDl4y9CjSbwMrFQmTc2nsDZC8n
+SE3AyO8UUHGPCYxvsg28sZFLx/rn7E6Iq3l7qAeafFiJ5fBWzQKWDER0uNTkH4uZ9iUQAA
+AMBSBqUDUhJGJ/yFDiBYfOAqBahU4iPAVSl/OuNcxB3yvEe0zER2ycVonFfTINSRBYaw89
+jF78xu7EaXnKbo5JJS5ZIdpT7f5yPVvETduqMZ1D1iqtjc8IlnwRIKc5h37Sk0m8E5PlQR
+hwVsX9NO7GixI/lfqQjUzmG6pagoAW9Xsd9lN4YJ3Oy0QZhSg+a5rrKtMNJlS6XpxKjiEL
+CqQHPy/9vw3fv4vOyGZ4iz1aBMfivBQS0V1CazUOgLwg1+oIEAAADBAPxY7KCJYXwe3+Hh
+q2AOVtB69nCQqwUrl7PINluH5Xp0vMzHepRpwbpuxnsotJq0CMsJXeiQ6FYTIfxHO4Da/5
+seFbmHeK9paNcWZCRJJK4Tjyf3NBaF+GEgMapiVxlaH2m0VkF4I1RzQi83A239bTtoORvl
+mID7odK0Z3NCV1IOPVri/qLoDi+cmVbdRobcIVPQmgSiK+ajBCypGM6XxyjFNqm5XoDPnY
+cVj7EkGc/MdLz2Aho4B7f5vfPQcsKxGwAAAMEAy7Xvm89N+ZxHcO4Air9Ej37hWRw7qvtO
+nfPEWhYaEWmK3mb0DwIzLBKsaiqgKlq6FGnu5cWhL2W4VsS0jerh4jhHtoyrrRkzfIBOiZ
+d6dmsjRVXOywTTU8nv6emU79956vJMc8HchDdu+R6NGpN4FuzQdkW/vY+mlRdjMR8HgPXe
+ZqNoARglgY0aT0Ugt94otCO/nCXFNj/j5t06sob3gaowxBSD22HYWBwROevzEvprHIYHrH
+zR7bwCJ7gbhLr5AAAAFGJ1aWxkZXJAOGYwY2I2MzU5NDU0AQIDBAUG
+-----END OPENSSH PRIVATE KEY-----
diff --git a/testsuite/keys/ssh/id_rsa.pub b/testsuite/keys/ssh/id_rsa.pub
new file mode 100644
index 00000000..bde1bf98
--- /dev/null
+++ b/testsuite/keys/ssh/id_rsa.pub
@@ -0,0 +1 @@
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIzdqzNYtY7/Rm2qL5U26wemxRt9+ZSTXsGgqYiq/nV0rhPI2OSmpfK+O86/3yVczMR+qfAAfegvQfVkJpLSqdW6pDptwvc1d7dt6HYfpowFu/CVj6vBeVwW7EfAZvLdTxqwfp3H5lO7SyQ5okpE2AIA48YzcJV+3Zg8lahrEEu0uyn+3bFydwvaGGAK0NrTJNVNCf/e9b+dqlemO8021l2gM0Gm2jqVcINhRESPR7ercAPiy8kif3IugnqryQxkv6vL4J0UouvCwQSP4nx2wBNsCVVXRrm1XVoa8zB3i9AEM+1PIxNB6Xz8sYsL3qnDUrucfhz+RXLI7zQ7Z5Wmn3mUWpv+aUT+VQz8pE84aSawbmBwwoTAB+ZUQrlm4hLfO3kmnQBZY6LvuqEvT146G94sVd9vZD7ntVhdC+up2A5qybV809COxw0Mr5rcKyxicvP6ZvcVzuViTk1OoJO+az5Q9if2Jh6CJXdqfEzkHGKZehh1Eiy9vw+522mj734UM= ci
-- 
2.20.1

[PATCH v2 3/9] meta-isar: Add a recipe that configures ci user

[Uladzimir Bely]

This creates `ci` user on a target and configures it for
non-interactive access to the image via SSH.

Non-interactive access is provided by placing testsuite's public part
of ssh key to `authorized_keys`.

This allows non-interactive SSH access to the machine with executing
custom commands on the guest VM.

Also, NetworkManager is installed to the image to make ethernet on
target auto configured and thus port forwarding from host to qemu
machine would work.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 .../isar-ci-ssh-setup/files/postinst          | 18 +++++++++++++++
 .../isar-ci-ssh-setup_0.1.bb                  | 22 +++++++++++++++++++
 2 files changed, 40 insertions(+)
 create mode 100644 meta-isar/recipes-ci/isar-ci-ssh-setup/files/postinst
 create mode 100644 meta-isar/recipes-ci/isar-ci-ssh-setup/isar-ci-ssh-setup_0.1.bb

diff --git a/meta-isar/recipes-ci/isar-ci-ssh-setup/files/postinst b/meta-isar/recipes-ci/isar-ci-ssh-setup/files/postinst
new file mode 100644
index 00000000..37d1bcb0
--- /dev/null
+++ b/meta-isar/recipes-ci/isar-ci-ssh-setup/files/postinst
@@ -0,0 +1,18 @@
+#!/bin/sh
+
+set -e
+
+if ! getent group ci >/dev/null; then
+    groupadd --system ci
+fi
+
+if ! getent passwd ci >/dev/null; then
+    useradd --system --gid ci --create-home \
+        --home-dir /var/lib/isar-ci --no-user-group \
+        --comment "Isar CI user" \
+        ci
+fi
+
+# since the homedir was part of the package, useradd did not include skel
+cp -RTn /etc/skel ~ci
+chown -R ci:ci ~ci
diff --git a/meta-isar/recipes-ci/isar-ci-ssh-setup/isar-ci-ssh-setup_0.1.bb b/meta-isar/recipes-ci/isar-ci-ssh-setup/isar-ci-ssh-setup_0.1.bb
new file mode 100644
index 00000000..74fecf92
--- /dev/null
+++ b/meta-isar/recipes-ci/isar-ci-ssh-setup/isar-ci-ssh-setup_0.1.bb
@@ -0,0 +1,22 @@
+# This software is a part of ISAR.
+
+DESCRIPTION = "Setup user with non-interactive SSH access"
+MAINTAINER = "Uladzimir Bely <uladzimir.bely@ilbers.de>"
+
+SRC_URI = " \
+    file://postinst \
+"
+
+DEPENDS += "sshd-regen-keys"
+DEBIAN_DEPENDS = "adduser, apt (>= 0.4.2), network-manager, sshd-regen-keys"
+
+inherit dpkg-raw
+
+do_install() {
+    # Install authorized SSH keys
+    install -v -d ${D}/var/lib/isar-ci/.ssh/
+    install -v -m 644 ${TESTSUITEDIR}/keys/ssh/id_rsa.pub ${D}/var/lib/isar-ci/.ssh/authorized_keys
+
+    # Manage all interfaces (including ethernet) by NetworkManager
+    install -D -m 644 /dev/null ${D}/etc/NetworkManager/conf.d/10-globally-managed-devices.conf
+}
-- 
2.20.1

[PATCH v2 4/9] meta-isar: Use a separate image recipe in CI

[Uladzimir Bely]

In order not to spoil `isar-image-base` or `local.conf.sample`,
put CI-specific configuration to a separate image recipe.

This includes SSH configuration for separate `ci` user that is done
in `isar-ci-ssh-setup` recipe.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta-isar/recipes-ci/images/isar-image-ci.bb | 9 +++++++++
 1 file changed, 9 insertions(+)
 create mode 100644 meta-isar/recipes-ci/images/isar-image-ci.bb

diff --git a/meta-isar/recipes-ci/images/isar-image-ci.bb b/meta-isar/recipes-ci/images/isar-image-ci.bb
new file mode 100644
index 00000000..77b1ef6e
--- /dev/null
+++ b/meta-isar/recipes-ci/images/isar-image-ci.bb
@@ -0,0 +1,9 @@
+# CI root filesystem for target installation
+#
+# This software is a part of ISAR.
+# Copyright (C) 2023 ilbers GmbH
+
+require recipes-core/images/isar-image-base.bb
+
+# Setup SSH server on board
+IMAGE_INSTALL += "isar-ci-ssh-setup"
-- 
2.20.1

[PATCH v2 5/9] testsuite: Allow custom image names in start_vm.py

[Uladzimir Bely]

This removes hardcoded `isar-image-base` image name used in
`testsuite/start_vm.py`.

Now, user can pass custom value with `-i` or `--image` option
(e.g. `isar-image-debug` or any other value used in their downstream.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/cibuilder.py |  9 +++++----
 testsuite/start_vm.py  | 21 +++++++++++----------
 2 files changed, 16 insertions(+), 14 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 5d82ec4b..247b998b 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -209,8 +209,9 @@ class CIBuilder(Test):
 
         return env['LAYERDIR_' + layer].strip('"')
 
-    def vm_start(self, arch='amd64', distro='buster', enforce_pcbios=False,
-                 skip_modulecheck=False):
+    def vm_start(self, arch='amd64', distro='buster',
+                 enforce_pcbios=False, skip_modulecheck=False,
+                 image='isar-image-base'):
         time_to_wait = self.params.get('time_to_wait', default=60)
 
         self.log.info('===================================================')
@@ -233,7 +234,7 @@ class CIBuilder(Test):
             os.unlink(latest_link)
         os.symlink(os.path.basename(output_file), latest_link)
 
-        cmdline = start_vm.format_qemu_cmdline(arch, self.build_dir, distro,
+        cmdline = start_vm.format_qemu_cmdline(arch, self.build_dir, distro, image,
                                                output_file, None, enforce_pcbios)
         cmdline.insert(1, '-nographic')
 
@@ -244,7 +245,7 @@ class CIBuilder(Test):
         module_output = b'Just an example'
         resize_output = None
 
-        bb_output = start_vm.get_bitbake_env(arch, distro).decode()
+        bb_output = start_vm.get_bitbake_env(arch, distro, image).decode()
         image_fstypes = start_vm.get_bitbake_var(bb_output, 'IMAGE_FSTYPES')
         wks_file = start_vm.get_bitbake_var(bb_output, 'WKS_FILE')
         # only the first type will be tested in start_vm.py
diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index c3726f14..d151232e 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -10,8 +10,8 @@ import subprocess
 import sys
 import time
 
-def get_bitbake_env(arch, distro):
-    multiconfig = 'mc:qemu' + arch + '-' + distro + ':isar-image-base'
+def get_bitbake_env(arch, distro, image):
+    multiconfig = 'mc:qemu' + arch + '-' + distro + ':' + image
     output = subprocess.check_output(['bitbake', '-e', str(multiconfig)])
     return output
 
@@ -22,18 +22,19 @@ def get_bitbake_var(output, var):
             ret = line.split('"')[1]
     return ret
 
-def format_qemu_cmdline(arch, build, distro, out, pid, enforce_pcbios=False):
-    bb_output = get_bitbake_env(arch, distro).decode()
+def format_qemu_cmdline(arch, build, distro, image, out, pid, enforce_pcbios=False):
+    bb_output = get_bitbake_env(arch, distro, image).decode()
 
-    rootfs_image = ''
     extra_args = ''
     cpu = ['']
 
     image_type = get_bitbake_var(bb_output, 'IMAGE_FSTYPES').split()[0]
     deploy_dir_image = get_bitbake_var(bb_output, 'DEPLOY_DIR_IMAGE')
     base = 'ubuntu' if distro in ['focal', 'bionic'] else 'debian'
+
+    rootfs_image = image + '-' + base + '-' + distro + '-qemu' + arch + '.' + image_type
+
     if image_type == 'ext4':
-        rootfs_image = 'isar-image-base-' + base + '-' + distro + '-qemu' + arch + '.ext4'
         kernel_image = deploy_dir_image + '/' + get_bitbake_var(bb_output, 'KERNEL_IMAGE')
         initrd_image = get_bitbake_var(bb_output, 'INITRD_IMAGE')
 
@@ -49,7 +50,6 @@ def format_qemu_cmdline(arch, build, distro, out, pid, enforce_pcbios=False):
         extra_args = ['-kernel', kernel_image, '-initrd', initrd_image]
         extra_args.extend(kargs)
     elif image_type == 'wic':
-        rootfs_image = 'isar-image-base-' + base + '-' + distro + '-qemu' + arch + '.wic'
         extra_args = ['-snapshot']
     else:
         raise ValueError('Invalid image type: ' + str(image_type))
@@ -91,8 +91,8 @@ def format_qemu_cmdline(arch, build, distro, out, pid, enforce_pcbios=False):
 
     return cmd
 
-def start_qemu(arch, build, distro, out, pid, enforce_pcbios):
-    cmdline = format_qemu_cmdline(arch, build, distro, out, pid, enforce_pcbios)
+def start_qemu(arch, build, distro, image, out, pid, enforce_pcbios):
+    cmdline = format_qemu_cmdline(arch, build, distro, image, out, pid, enforce_pcbios)
     cmdline.insert(1, '-nographic')
 
     print(cmdline)
@@ -103,9 +103,10 @@ if __name__ == "__main__":
     parser.add_argument('-a', '--arch', choices=['arm', 'arm64', 'amd64', 'i386', 'mipsel'], help='set isar machine architecture.', default='arm')
     parser.add_argument('-b', '--build', help='set path to build directory.', default=os.getcwd())
     parser.add_argument('-d', '--distro', choices=['buster', 'bullseye', 'bookworm'], help='set isar Debian distribution.', default='bookworm')
+    parser.add_argument('-i', '--image', help='set image name.', default='isar-image-base')
     parser.add_argument('-o', '--out', help='Route QEMU console output to specified file.')
     parser.add_argument('-p', '--pid', help='Store QEMU pid to specified file.')
     parser.add_argument('--pcbios', action="store_true", help='remove any bios options to enforce use of pc bios')
     args = parser.parse_args()
 
-    start_qemu(args.arch, args.build, args.distro, args.out, args.pid, args.pcbios)
+    start_qemu(args.arch, args.build, args.distro, args.image, args.out, args.pid, args.pcbios)
-- 
2.20.1

[PATCH v2 6/9] testsuite: Support running custom commands in VM

[Uladzimir Bely]

By specifying `cmd='<command>'` it's now possible to run VM and
execute the command over SSH connection.

If this is not specified, tests uses previous approach, with
parsing output log and searching some specific fragments.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/cibuilder.py | 81 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 79 insertions(+), 2 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 247b998b..9b64af90 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -2,6 +2,7 @@
 
 import logging
 import os
+import re
 import select
 import shutil
 import subprocess
@@ -209,13 +210,54 @@ class CIBuilder(Test):
 
         return env['LAYERDIR_' + layer].strip('"')
 
+    def get_ssh_cmd_prefix(self, port, priv_key):
+        port_args = ''
+        if port:
+            port_args = ' -p ' + str(port)
+
+        cmd_prefix = 'ssh' + port_args + \
+                     ' -o ConnectTimeout=5 -o IdentityFile=' + priv_key + \
+                     ' -o StrictHostKeyChecking=no ci@localhost '
+
+        return cmd_prefix
+
+
+    def exec_cmd(self, cmd, cmd_prefix):
+        rc = subprocess.call('exec ' + str(cmd_prefix) + ' "' + str(cmd) + '"', shell=True,
+                             stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        return rc
+
+
+    def wait_connection(self, proc, cmd_prefix, timeout):
+        self.log.debug('Waiting for SSH server ready...')
+
+        rc = None
+        while time.time() < timeout:
+            if proc.poll() is not None:
+                self.log.error('Machine is not running')
+                return rc
+
+            rc = self.exec_cmd('/bin/true', cmd_prefix)
+            time.sleep(1)
+
+            if rc == 0:
+                self.log.debug('SSH server is ready')
+                break
+
+        if rc != 0:
+            self.log.error('SSH server is not ready')
+
+        return rc
+
+
     def vm_start(self, arch='amd64', distro='buster',
                  enforce_pcbios=False, skip_modulecheck=False,
-                 image='isar-image-base'):
+                 image='isar-image-base', cmd=None):
         time_to_wait = self.params.get('time_to_wait', default=60)
 
         self.log.info('===================================================')
         self.log.info('Running Isar VM boot test for (' + distro + '-' + arch + ')')
+        self.log.info('Remote command is ' + str(cmd))
         self.log.info('Isar build folder is: ' + self.build_dir)
         self.log.info('===================================================')
 
@@ -238,7 +280,7 @@ class CIBuilder(Test):
                                                output_file, None, enforce_pcbios)
         cmdline.insert(1, '-nographic')
 
-        self.log.info('QEMU boot line: ' + str(cmdline))
+        self.log.info('QEMU boot line:\n' + ' '.join(cmdline))
 
         login_prompt = b'isar login:'
         # the printk of recipes-kernel/example-module
@@ -266,6 +308,41 @@ class CIBuilder(Test):
         p1 = subprocess.Popen('exec ' + ' '.join(cmdline), shell=True,
                               stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
                               universal_newlines=True)
+
+        if cmd is not None:
+            rc = None
+            try:
+                port = None
+                for arg in cmdline:
+                    match = re.match(r".*hostfwd=tcp::(\d*).*", arg)
+                    if match:
+                        port = match.group(1)
+                        break
+
+                # copy private key to build directory
+                priv_key = '%s/ci_priv_key' % self.build_dir
+                if not os.path.exists(priv_key):
+                    shutil.copy(os.path.dirname(__file__) + '/keys/ssh/id_rsa', priv_key)
+                    os.chmod(priv_key, 0o400)
+
+                cmd_prefix = self.get_ssh_cmd_prefix(port, priv_key)
+                self.log.debug('Connect command:\n' + cmd_prefix)
+                rc = self.wait_connection(p1, cmd_prefix, timeout)
+
+                if rc == 0:
+                    rc = self.exec_cmd(cmd, cmd_prefix)
+                    self.log.debug('`' + cmd + '` returned ' + str(rc))
+            finally:
+                if p1.poll() is None:
+                    self.log.debug('Killing qemu...')
+                    p1.kill()
+                p1.wait()
+
+            if rc != 0:
+                self.fail('Log ' + output_file)
+
+            return
+
         try:
             poller = select.poll()
             poller.register(p1.stdout, select.POLLIN)
-- 
2.20.1

[PATCH v2 7/9] testsuite: Support running custom scripts in VM

[Uladzimir Bely]

By specifying `script='<file.sh>'` it's now possible to run VM and
execute local script over SSH connection.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/cibuilder.py | 23 +++++++++++++++++++----
 1 file changed, 19 insertions(+), 4 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 9b64af90..5eebd5aa 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -228,6 +228,15 @@ class CIBuilder(Test):
         return rc
 
 
+    def run_script(self, script, cmd_prefix):
+        script_dir = self.params.get('test_script_dir',
+                                     default=os.path.abspath(os.path.dirname(__file__))) + '/'
+        script_path = script_dir + script
+        rc = subprocess.call('cat ' + script_path + ' | ' + str(cmd_prefix), shell=True,
+                             stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        return rc
+
+
     def wait_connection(self, proc, cmd_prefix, timeout):
         self.log.debug('Waiting for SSH server ready...')
 
@@ -252,12 +261,13 @@ class CIBuilder(Test):
 
     def vm_start(self, arch='amd64', distro='buster',
                  enforce_pcbios=False, skip_modulecheck=False,
-                 image='isar-image-base', cmd=None):
+                 image='isar-image-base', cmd=None, script=None):
         time_to_wait = self.params.get('time_to_wait', default=60)
 
         self.log.info('===================================================')
         self.log.info('Running Isar VM boot test for (' + distro + '-' + arch + ')')
         self.log.info('Remote command is ' + str(cmd))
+        self.log.info('Remote script is ' + str(script))
         self.log.info('Isar build folder is: ' + self.build_dir)
         self.log.info('===================================================')
 
@@ -309,7 +319,7 @@ class CIBuilder(Test):
                               stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
                               universal_newlines=True)
 
-        if cmd is not None:
+        if cmd is not None or script is not None:
             rc = None
             try:
                 port = None
@@ -330,8 +340,13 @@ class CIBuilder(Test):
                 rc = self.wait_connection(p1, cmd_prefix, timeout)
 
                 if rc == 0:
-                    rc = self.exec_cmd(cmd, cmd_prefix)
-                    self.log.debug('`' + cmd + '` returned ' + str(rc))
+                    if cmd is not None:
+                        rc = self.exec_cmd(cmd, cmd_prefix)
+                        self.log.debug('`' + cmd + '` returned ' + str(rc))
+                    elif script is not None:
+                        rc = self.run_script(script, cmd_prefix)
+                        self.log.debug('`' + script + '` returned ' + str(rc))
+
             finally:
                 if p1.poll() is None:
                     self.log.debug('Killing qemu...')
-- 
2.20.1

[PATCH v2 8/9] testsuite: Run custom commands and scripts on some qemu targets

[Uladzimir Bely]

For demonstaration purposes, build 'isar-image-ci' for some targets
and add corresponding VM run tests with custom commands and scripts.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/citest.py              | 105 ++++++++++++++++++++++++++-----
 testsuite/test_example_module.sh |   5 ++
 testsuite/test_getty_target.sh   |   7 +++
 3 files changed, 103 insertions(+), 14 deletions(-)
 create mode 100644 testsuite/test_example_module.sh
 create mode 100644 testsuite/test_getty_target.sh

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 975cb064..17a90244 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -123,8 +123,8 @@ class CrossTest(CIBaseTest):
     """
     def test_cross(self):
         targets = [
-            'mc:qemuarm-buster:isar-image-base',
-            'mc:qemuarm-bullseye:isar-image-base',
+            'mc:qemuarm-buster:isar-image-ci',
+            'mc:qemuarm-bullseye:isar-image-ci',
             'mc:de0-nano-soc-bullseye:isar-image-base',
             'mc:stm32mp15x-buster:isar-image-base'
                   ]
@@ -156,7 +156,7 @@ class CrossTest(CIBaseTest):
 
     def test_cross_bookworm(self):
         targets = [
-            'mc:qemuarm-bookworm:isar-image-base',
+            'mc:qemuarm-bookworm:isar-image-ci',
             'mc:qemuarm64-bookworm:isar-image-base'
                   ]
 
@@ -198,9 +198,9 @@ class NoCrossTest(CIBaseTest):
     """
     def test_nocross(self):
         targets = [
-            'mc:qemuarm-buster:isar-image-base',
+            'mc:qemuarm-buster:isar-image-ci',
             'mc:qemuarm-bullseye:isar-image-base',
-            'mc:qemuarm64-bullseye:isar-image-base',
+            'mc:qemuarm64-bullseye:isar-image-ci',
             'mc:qemui386-buster:isar-image-base',
             'mc:qemui386-bullseye:isar-image-base',
             'mc:qemuamd64-buster:isar-image-base',
@@ -214,7 +214,7 @@ class NoCrossTest(CIBaseTest):
             'mc:bananapi-bullseye:isar-image-base',
             'mc:nanopi-neo-bullseye:isar-image-base',
             'mc:stm32mp15x-bullseye:isar-image-base',
-            'mc:qemuamd64-focal:isar-image-base'
+            'mc:qemuamd64-focal:isar-image-ci'
                   ]
 
         self.init()
@@ -241,7 +241,7 @@ class NoCrossTest(CIBaseTest):
             'mc:qemuamd64-bookworm:isar-image-base',
             'mc:qemuarm-bookworm:isar-image-base',
             'mc:qemui386-bookworm:isar-image-base',
-            'mc:qemumipsel-bookworm:isar-image-base',
+            'mc:qemumipsel-bookworm:isar-image-ci',
             'mc:hikey-bookworm:isar-image-base'
                   ]
 
@@ -340,15 +340,48 @@ class VmBootTestFast(CIBaseTest):
     """
     def test_arm_bullseye(self):
         self.init()
-        self.vm_start('arm','bullseye')
+        self.vm_start('arm','bullseye', \
+            image='isar-image-ci')
+
+    def test_arm_bullseye_example_module(self):
+        self.init()
+        self.vm_start('arm','bullseye', \
+            image='isar-image-ci', cmd='lsmod | grep example_module')
+
+    def test_arm_bullseye_getty_target(self):
+        self.init()
+        self.vm_start('arm','bullseye', \
+            image='isar-image-ci', script='test_getty_target.sh')
 
     def test_arm_buster(self):
         self.init()
-        self.vm_start('arm','buster')
+        self.vm_start('arm','buster', \
+            image='isar-image-ci')
+
+    def test_arm_buster_getty_target(self):
+        self.init()
+        self.vm_start('arm','buster', \
+            image='isar-image-ci', cmd='systemctl is-active getty.target')
+
+    def test_arm_buster_example_module(self):
+        self.init()
+        self.vm_start('arm','buster', \
+            image='isar-image-ci', script='test_example_module.sh')
 
     def test_arm_bookworm(self):
         self.init()
-        self.vm_start('arm','bookworm')
+        self.vm_start('arm','bookworm', \
+            image='isar-image-ci')
+
+    def test_arm_bookworm_example_module(self):
+        self.init()
+        self.vm_start('arm','bookworm', \
+            image='isar-image-ci', cmd='lsmod | grep example_module')
+
+    def test_arm_bookworm_getty_target(self):
+        self.init()
+        self.vm_start('arm','bookworm', \
+            image='isar-image-ci', script='test_getty_target.sh')
 
 class VmBootTestFull(CIBaseTest):
 
@@ -363,11 +396,33 @@ class VmBootTestFull(CIBaseTest):
 
     def test_arm_buster(self):
         self.init()
-        self.vm_start('arm','buster')
+        self.vm_start('arm','buster', \
+            image='isar-image-ci')
+
+    def test_arm_buster_example_module(self):
+        self.init()
+        self.vm_start('arm','buster', \
+            image='isar-image-ci', cmd='lsmod | grep example_module')
+
+    def test_arm_buster_getty_target(self):
+        self.init()
+        self.vm_start('arm','buster', \
+            image='isar-image-ci', script='test_getty_target.sh')
 
     def test_arm64_bullseye(self):
         self.init()
-        self.vm_start('arm64','bullseye')
+        self.vm_start('arm64','bullseye', \
+            image='isar-image-ci')
+
+    def test_arm64_bullseye_getty_target(self):
+        self.init()
+        self.vm_start('arm64','bullseye', \
+            image='isar-image-ci', cmd='systemctl is-active getty.target')
+
+    def test_arm64_bullseye_example_module(self):
+        self.init()
+        self.vm_start('arm64','bullseye', \
+            image='isar-image-ci', script='test_example_module.sh')
 
     def test_i386_buster(self):
         self.init()
@@ -382,7 +437,18 @@ class VmBootTestFull(CIBaseTest):
 
     def test_amd64_focal(self):
         self.init()
-        self.vm_start('amd64','focal')
+        self.vm_start('amd64','focal', \
+            image='isar-image-ci')
+
+    def test_amd64_focal_example_module(self):
+        self.init()
+        self.vm_start('amd64','focal', \
+            image='isar-image-ci', cmd='lsmod | grep example_module')
+
+    def test_amd64_focal_getty_target(self):
+        self.init()
+        self.vm_start('amd64','focal', \
+            image='isar-image-ci', script='test_getty_target.sh')
 
     def test_amd64_bookworm(self):
         self.init()
@@ -398,4 +464,15 @@ class VmBootTestFull(CIBaseTest):
 
     def test_mipsel_bookworm(self):
         self.init()
-        self.vm_start('mipsel','bookworm')
+        self.vm_start('mipsel','bookworm', \
+            image='isar-image-ci')
+
+    def test_mipsel_bookworm_getty_target(self):
+        self.init()
+        self.vm_start('mipsel','bookworm', \
+            image='isar-image-ci', cmd='systemctl is-active getty.target')
+
+    def test_mipsel_bookworm_example_module(self):
+        self.init()
+        self.vm_start('mipsel','bookworm', \
+            image='isar-image-ci', script='test_example_module.sh')
diff --git a/testsuite/test_example_module.sh b/testsuite/test_example_module.sh
new file mode 100644
index 00000000..175030d1
--- /dev/null
+++ b/testsuite/test_example_module.sh
@@ -0,0 +1,5 @@
+#!/bin/sh
+
+set -e
+
+lsmod | grep '^example_module '
diff --git a/testsuite/test_getty_target.sh b/testsuite/test_getty_target.sh
new file mode 100644
index 00000000..38c0fc1e
--- /dev/null
+++ b/testsuite/test_getty_target.sh
@@ -0,0 +1,7 @@
+#!/bin/sh
+
+set -e
+
+sleep 10
+
+systemctl is-active getty.target
-- 
2.20.1

[PATCH v2 9/9] testsuite: Update testsuite qemu-related documentation

[Uladzimir Bely]

Added examples of avocado tests that run custom commands and scripts
under previously qemu image built.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/README.md | 41 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 39 insertions(+), 2 deletions(-)

diff --git a/testsuite/README.md b/testsuite/README.md
index 4f2a8da8..50698da5 100644
--- a/testsuite/README.md
+++ b/testsuite/README.md
@@ -48,6 +48,43 @@ $ avocado run ../testsuite/citest.py -t startvm,fast -p time_to_wait=300
 $ avocado run ../testsuite/citest.py -t startvm,full -p time_to_wait=300
 ```
 
-# Other
+# Running qemu images
 
-There is a tool start_vm.py which is the replacement for the bash script in isar/scripts directory.
+## Manual running
+
+There is a tool `start_vm.py` which is the replacement for the bash script in
+`isar/scripts` directory. It can be used to run image previously built:
+
+```
+./start_vm.py -a amd64 -b /build -d bullseye -i isar-image-base
+```
+
+# Tests for running commands under qemu images
+
+Package `isar-image-ci` configures `ci` user with non-interactive SSH access
+to the machine. Image `isar-image-ci` preinstalls this package.
+
+Example of test that runs qemu image and executes remote command under it:
+
+```
+    def test_getty_target(self):
+        self.init()
+        self.vm_start('amd64','bookworm', \
+            image='isar-image-ci',
+            cmd='systemctl is-active getty.target')
+```
+
+To run something more complex than simple command, custom test script
+can be executed instead of command:
+
+```
+    def test_getty_target(self):
+        self.init()
+        self.vm_start('amd64','bookworm', \
+            image='isar-image-ci',
+            script='test_getty_target.sh')
+```
+
+The default location of custom scripts is `isar/testsuite/`. It can be changed
+by passing `-p test_script_dir="custom_path"` to `avocado run`
+arguments.
-- 
2.20.1

Re: [PATCH v2 0/9] Support running custom commands in VM

[Uladzimir Bely]

In the email from Wednesday, 8 February 2023 18:45:02 +03 user Uladzimir Bely 
wrote:
> VM run tests implementation in Isar testsuite allows only parsing
> virtual machine boot logs, so the functionality is limited.
> 
> This patchset additionally allows to run custom commands on VM
> that runs in qemu that allows to implement any custom tests.
> 
> The concept is the following.
> 
> For running custom tests SSH is used. So:
> 
>  - SSH server on machine should be configured, so `sshd-regen-keys`
> package needs to be installed in the image.
> 
>  - Ethernet interface on machine should be automatically brought up
> to make host-to-vm port forwarding working.
> 
>  - User in VM we use to connect via SSH should have public SSH key
> in their `~/.ssh/` directory preinstalled.
> 
>  - Since base isar image don't have this included, custom image name
> for vm run tests should be supported. This also allows to make reusing
> Isar testsuite in downstream simpler.
> 
> Changes since v1:
>  - Patches "meta-isar: Fix PCI bus initialization in qemuarm machine"
> and "testsuite: Use random free port for qemu SSH forwarding" are
> excluded from patchset since they are already merged.
>  - User `ci` creation/configuration is moved to a separate package so
> now it can be simply removed from target using apt. No new user flags
> are required anymore.
>  - In addition to simple commands (cmd="<command>"), running custom
> local scripts (script="<script>.sh") on remote machine is implemented.
>  - Appropriate documentation was added.
>  - patchset rebased on top of "Add developers test" patchset which
> is supposed to be merged earlier.
> 
> Notes:
>  - Mixing positional and named arguments, e.g.
> 
> |    self.vm_start('arm','bookworm',
> |    
> |        image='isar-image-ci', script='test_getty_target.sh')
> 
> is still here, since it's easy to read.
> 
> Uladzimir Bely (9):
>   testsuite: Fix failing hostname service in qemu guest
>   testsuite: Add SSH key pair for using in CI
>   meta-isar: Add a recipe that configures ci user
>   meta-isar: Use a separate image recipe in CI
>   testsuite: Allow custom image names in start_vm.py
>   testsuite: Support running custom commands in VM
>   testsuite: Support running custom scripts in VM
>   testsuite: Run custom commands and scripts on some qemu targets
>   testsuite: Update testsuite qemu-related documentation
> 
>  meta-isar/recipes-ci/images/isar-image-ci.bb  |   9 ++
>  .../isar-ci-ssh-setup/files/postinst          |  18 +++
>  .../isar-ci-ssh-setup_0.1.bb                  |  22 ++++
>  testsuite/README.md                           |  41 ++++++-
>  testsuite/cibuilder.py                        | 105 +++++++++++++++++-
>  testsuite/citest.py                           | 105 +++++++++++++++---
>  testsuite/keys/ssh/id_rsa                     |  38 +++++++
>  testsuite/keys/ssh/id_rsa.pub                 |   1 +
>  testsuite/start_vm.py                         |  23 ++--
>  testsuite/test_example_module.sh              |   5 +
>  testsuite/test_getty_target.sh                |   7 ++
>  11 files changed, 341 insertions(+), 33 deletions(-)
>  create mode 100644 meta-isar/recipes-ci/images/isar-image-ci.bb
>  create mode 100644 meta-isar/recipes-ci/isar-ci-ssh-setup/files/postinst
>  create mode 100644
> meta-isar/recipes-ci/isar-ci-ssh-setup/isar-ci-ssh-setup_0.1.bb create mode
> 100644 testsuite/keys/ssh/id_rsa
>  create mode 100644 testsuite/keys/ssh/id_rsa.pub
>  create mode 100644 testsuite/test_example_module.sh
>  create mode 100644 testsuite/test_getty_target.sh


Applied to next.