[PATCH 0/4] Basic hardware testing support

[PATCH 0/4] Basic hardware testing support

[Uladzimir Bely]

Isar is currently able to run tests only on qemu targets over SSH.

This patchset adds basic support of running tests on real hardware
targets in similar way. There are the following limitations:

- User `ci` should be pre-configured (e.g., isar-ci-ssh-setup recipe
used when generating the image), similar to qemu testing;
- The image should be flashed on the board before running tests;
- The target should be turned on and accessible over SSH;

Possible ways to improve this:
- Support turning on/off the hardware (via USB switches or similar);
- Automation of board software update.

Uladzimir Bely (4):
  cibuilder.py: Refactor vm_start related functions
  testsuite: Add an interface to run commands over ssh
  testsuite: Separate common part of kas-based test example
  testsute: Provide an example of hardware test

 meta-isar/test/README.md                | 24 ++++++-
 meta-isar/test/common.sh                | 30 +++++++++
 meta-isar/test/run_test.sh              | 33 ++-------
 meta-isar/test/run_test_hw.sh           | 10 +++
 meta-isar/test/sample_kas_config_hw.yml | 32 +++++++++
 meta-isar/test/sample_test.py           |  0
 meta-isar/test/sample_test_hw.py        | 13 ++++
 testsuite/cibuilder.py                  | 89 +++++++++++++++++--------
 8 files changed, 174 insertions(+), 57 deletions(-)
 create mode 100755 meta-isar/test/common.sh
 create mode 100755 meta-isar/test/run_test_hw.sh
 create mode 100644 meta-isar/test/sample_kas_config_hw.yml
 mode change 100644 => 100755 meta-isar/test/sample_test.py
 create mode 100755 meta-isar/test/sample_test_hw.py

-- 
2.20.1

[PATCH 1/4] cibuilder.py: Refactor vm_start related functions

[Uladzimir Bely]

get_ssh_cmd_prefix: Add possibility to specify user and hostname
instead of default `ci@localhost`. This allows to run test on hardware
targets by their IP address or hostname.

prepare_priv_key: Copy private SSH key to build directory before use
since it's original location may be readonly.

remote_run: carve out ssh-related code from vm_start function.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/cibuilder.py | 67 ++++++++++++++++++++++++++----------------
 1 file changed, 41 insertions(+), 26 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 6aac2279..a4edb578 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -226,14 +226,12 @@ class CIBuilder(Test):
 
         return env['LAYERDIR_' + layer].strip('"')
 
-    def get_ssh_cmd_prefix(self, port, priv_key):
-        port_args = ''
-        if port:
-            port_args = ' -p ' + str(port)
+    def get_ssh_cmd_prefix(self, user, host, port, priv_key):
+        cmd_prefix = 'ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=no '\
+                     '-p %s -o IdentityFile=%s %s@%s ' \
+                     % (port, priv_key, user, host)
 
-        cmd_prefix = 'ssh' + port_args + \
-                     ' -o ConnectTimeout=5 -o IdentityFile=' + priv_key + \
-                     ' -o StrictHostKeyChecking=no ci@localhost '
+        self.log.debug('Connect command:\n' + cmd_prefix)
 
         return cmd_prefix
 
@@ -258,6 +256,9 @@ class CIBuilder(Test):
 
 
     def wait_connection(self, proc, cmd_prefix, timeout):
+        if proc is None:
+            return 0
+
         self.log.debug('Waiting for SSH server ready...')
 
         rc = None
@@ -281,6 +282,35 @@ class CIBuilder(Test):
         return rc
 
 
+    def prepare_priv_key(self):
+        # copy private key to build directory (that is writable)
+        priv_key = '%s/ci_priv_key' % self.build_dir
+        if not os.path.exists(priv_key):
+            shutil.copy(os.path.dirname(__file__) + '/keys/ssh/id_rsa', priv_key)
+        os.chmod(priv_key, 0o400)
+
+        return priv_key
+
+
+    def remote_run(self, user, host, port, cmd, script, proc=None, timeout=0):
+        priv_key = self.prepare_priv_key()
+        cmd_prefix = self.get_ssh_cmd_prefix(user, host, port, priv_key)
+
+        rc = self.wait_connection(proc, cmd_prefix, timeout)
+
+        if rc == 0:
+            if cmd is not None:
+                rc = self.exec_cmd(cmd, cmd_prefix)
+                self.log.debug('`' + cmd + '` returned ' + str(rc))
+            elif script is not None:
+                rc = self.run_script(script, cmd_prefix)
+                self.log.debug('`' + script + '` returned ' + str(rc))
+            else:
+                return None
+
+        return rc
+
+
     def vm_start(self, arch='amd64', distro='buster',
                  enforce_pcbios=False, skip_modulecheck=False,
                  image='isar-image-base', cmd=None, script=None):
@@ -342,32 +372,17 @@ class CIBuilder(Test):
                               universal_newlines=True)
 
         if cmd is not None or script is not None:
-            rc = None
             try:
-                port = None
+                user='ci'
+                host='localhost'
+                port = 22
                 for arg in cmdline:
                     match = re.match(r".*hostfwd=tcp::(\d*).*", arg)
                     if match:
                         port = match.group(1)
                         break
 
-                # copy private key to build directory
-                priv_key = '%s/ci_priv_key' % self.build_dir
-                if not os.path.exists(priv_key):
-                    shutil.copy(os.path.dirname(__file__) + '/keys/ssh/id_rsa', priv_key)
-                    os.chmod(priv_key, 0o400)
-
-                cmd_prefix = self.get_ssh_cmd_prefix(port, priv_key)
-                self.log.debug('Connect command:\n' + cmd_prefix)
-                rc = self.wait_connection(p1, cmd_prefix, timeout)
-
-                if rc == 0:
-                    if cmd is not None:
-                        rc = self.exec_cmd(cmd, cmd_prefix)
-                        self.log.debug('`' + cmd + '` returned ' + str(rc))
-                    elif script is not None:
-                        rc = self.run_script(script, cmd_prefix)
-                        self.log.debug('`' + script + '` returned ' + str(rc))
+                rc = self.remote_run(user, host, port, cmd, script, p1, timeout)
 
             finally:
                 if p1.poll() is None:
-- 
2.20.1

[PATCH 2/4] testsuite: Add an interface to run commands over ssh

[Uladzimir Bely]

Added ssh_start function for using in top-level tests for running
tests on hardware targets.

In contrast to vm_start one, it doesn't run qemu machine for testing,
but assumes that the machine is already running and can be reached
over SSH connection.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/cibuilder.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index a4edb578..a0c5711f 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -311,6 +311,28 @@ class CIBuilder(Test):
         return rc
 
 
+    def ssh_start(self, user='ci', host='localhost', port=22,
+                  cmd=None, script=None):
+        self.log.info('===================================================')
+        self.log.info('Running Isar SSH test for `%s@%s:%s`' % (user, host, port))
+        self.log.info('Remote command is `%s`' % (cmd))
+        self.log.info('Remote script is `%s`' % (script))
+        self.log.info('Isar build folder is: ' + self.build_dir)
+        self.log.info('===================================================')
+
+        self.check_init()
+
+        if cmd is not None or script is not None:
+            rc = self.remote_run(user, host, port, cmd, script)
+
+            if rc != 0:
+                self.fail('Failed with rc=%s' % rc)
+
+            return
+
+        self.fail('No command to run specified')
+
+
     def vm_start(self, arch='amd64', distro='buster',
                  enforce_pcbios=False, skip_modulecheck=False,
                  image='isar-image-base', cmd=None, script=None):
-- 
2.20.1

[PATCH 3/4] testsuite: Separate common part of kas-based test example

[Uladzimir Bely]

Avocado installation and configuration are steps required
by any kind of testing (qemu or hardware).

Separate appropriate stuff out of run_test.sh.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta-isar/test/common.sh      | 30 ++++++++++++++++++++++++++++++
 meta-isar/test/run_test.sh    | 33 +++++----------------------------
 meta-isar/test/sample_test.py |  0
 3 files changed, 35 insertions(+), 28 deletions(-)
 create mode 100755 meta-isar/test/common.sh
 mode change 100644 => 100755 meta-isar/test/sample_test.py

diff --git a/meta-isar/test/common.sh b/meta-isar/test/common.sh
new file mode 100755
index 00000000..e0e4ee6c
--- /dev/null
+++ b/meta-isar/test/common.sh
@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+
+set -e
+
+# Make Isar testsuite accessable
+export PYTHONPATH=${PYTHONPATH}:${TESTSUITEDIR}
+
+# install avocado in virtualenv in case it is not there already
+if ! command -v avocado > /dev/null; then
+    sudo apt-get -y update
+    sudo apt-get -y install virtualenv
+    rm -rf /tmp/avocado_venv
+    virtualenv --python python3 /tmp/avocado_venv
+    source /tmp/avocado_venv/bin/activate
+    pip install avocado-framework==100.1
+fi
+
+# Start tests in this dir
+BASE_DIR="/build/avocado"
+
+# Provide working path
+mkdir -p .config/avocado
+cat <<EOF > .config/avocado/avocado.conf
+[datadir.paths]
+base_dir = ${BASE_DIR}/
+data_dir = ${BASE_DIR}/data
+logs_dir = ${BASE_DIR}/logs
+test_dir = ${BASE_DIR}/test
+EOF
+export VIRTUAL_ENV="./"
diff --git a/meta-isar/test/run_test.sh b/meta-isar/test/run_test.sh
index c4a7d4ac..6083e635 100755
--- a/meta-isar/test/run_test.sh
+++ b/meta-isar/test/run_test.sh
@@ -1,17 +1,10 @@
 #!/usr/bin/env bash
 
-# Make Isar testsuite accessable
-export PYTHONPATH=${PYTHONPATH}:${TESTSUITEDIR}
+set -e
 
-# install avocado in virtualenv in case it is not there already
-if ! command -v avocado > /dev/null; then
-    sudo apt-get -y update
-    sudo apt-get -y install virtualenv
-    rm -rf /tmp/avocado_venv
-    virtualenv --python python3 /tmp/avocado_venv
-    source /tmp/avocado_venv/bin/activate
-    pip install avocado-framework==100.1
-fi
+test_dir=$(dirname $(realpath $0))
+
+. ${test_dir}/common.sh
 
 # Install qemu
 if ! command -v qemu-system-aarch64 > /dev/null; then
@@ -19,21 +12,5 @@ if ! command -v qemu-system-aarch64 > /dev/null; then
   sudo apt-get -y install --no-install-recommends qemu-system-aarch64 ipxe-qemu
 fi
 
-# Start tests in this dir
-BASE_DIR="/build/avocado"
-
-# Provide working path
-mkdir -p .config/avocado
-cat <<EOF > .config/avocado/avocado.conf
-[datadir.paths]
-base_dir = ${BASE_DIR}/
-data_dir = ${BASE_DIR}/data
-logs_dir = ${BASE_DIR}/logs
-test_dir = ${BASE_DIR}/test
-EOF
-export VIRTUAL_ENV="./"
-
-tsd=$(dirname $(realpath $0))/scripts
-
 # Run SSH tests
-avocado run --max-parallel-tasks=1 /work/sample_test.py -p test_script_dir=${tsd}
+avocado run --max-parallel-tasks=1 /work/sample_test.py -p test_script_dir=${test_dir}
diff --git a/meta-isar/test/sample_test.py b/meta-isar/test/sample_test.py
old mode 100644
new mode 100755
-- 
2.20.1

[PATCH 4/4] testsute: Provide an example of hardware test

[Uladzimir Bely]

The example is tested with Raspberry Pi 3 B+.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta-isar/test/README.md                | 24 ++++++++++++++++---
 meta-isar/test/run_test_hw.sh           | 10 ++++++++
 meta-isar/test/sample_kas_config_hw.yml | 32 +++++++++++++++++++++++++
 meta-isar/test/sample_test_hw.py        | 13 ++++++++++
 4 files changed, 76 insertions(+), 3 deletions(-)
 create mode 100755 meta-isar/test/run_test_hw.sh
 create mode 100644 meta-isar/test/sample_kas_config_hw.yml
 create mode 100755 meta-isar/test/sample_test_hw.py

diff --git a/meta-isar/test/README.md b/meta-isar/test/README.md
index f8c6338c..4619aaae 100644
--- a/meta-isar/test/README.md
+++ b/meta-isar/test/README.md
@@ -1,16 +1,34 @@
-# Example of the downstream testcase
+# Common requirements
 
 Use `meta-isar/test/` as working directory. Make sure `kas-container` is
 available in $PATH (look at https://github.com/siemens/kas for it).
 
-Build an image:
+# Example of the downstream qemu-based testcase
+
+1. Build an image:
 
 ```
 kas-container build sample_kas_config.yml
 ```
 
-Run testcase:
+2. Run testcase:
 
 ```
 kas-container shell sample_kas_config.yml -c '/work/run_test.sh'
 ```
+
+# Example of the downstream hardware testcase (example for RPi 3b+)
+
+1. Build an image:
+
+```
+kas-container build sample_kas_config.yml
+```
+
+2. Flash resulted wic image to SD-card and boot the board
+
+3. Run testcase (adjust `host` to RPi's IP address):
+
+```
+kas-container shell sample_kas_config_hw.yml -c "/work/run_test_hw.sh -p host=192.168.100.117"
+```
diff --git a/meta-isar/test/run_test_hw.sh b/meta-isar/test/run_test_hw.sh
new file mode 100755
index 00000000..a72c0204
--- /dev/null
+++ b/meta-isar/test/run_test_hw.sh
@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+
+set -e
+
+test_dir=$(dirname $(realpath $0))
+
+. ${test_dir}/common.sh
+
+# Run SSH tests
+avocado run --max-parallel-tasks=1 /work/sample_test_hw.py -p test_script_dir=${test_dir} -p host=rpi $@
diff --git a/meta-isar/test/sample_kas_config_hw.yml b/meta-isar/test/sample_kas_config_hw.yml
new file mode 100644
index 00000000..69afc9c3
--- /dev/null
+++ b/meta-isar/test/sample_kas_config_hw.yml
@@ -0,0 +1,32 @@
+header:
+  version: 11
+
+build_system: isar
+
+distro: raspios-bullseye
+machine: rpi-arm-v7
+target: mc:rpi-arm-v7-bullseye:isar-image-ci
+repos:
+  isar:
+    url: "https://github.com/ilbers/isar.git"
+    refspec: next
+    layers:
+      meta:
+      meta-isar:
+
+bblayers_conf_header:
+  standard: |
+    BBPATH = "${TOPDIR}"
+    BBFILES ?= ""
+
+local_conf_header:
+  standard: |
+    DISTRO_ARCH ??= "armhf"
+    PATCHRESOLVE = "noop"
+
+    USERS += "root"
+    USER_root[password] ??= "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"
+  crossbuild: |
+    ISAR_CROSS_COMPILE = "1"
+  ccache: |
+    USE_CCACHE = "1"
diff --git a/meta-isar/test/sample_test_hw.py b/meta-isar/test/sample_test_hw.py
new file mode 100755
index 00000000..47735823
--- /dev/null
+++ b/meta-isar/test/sample_test_hw.py
@@ -0,0 +1,13 @@
+#!/usr/bin/env python3
+
+from cibase import CIBaseTest
+
+class SampleHwTest(CIBaseTest):
+    def test_sample_script(self):
+        self.init("/build")
+
+        host = self.params.get('host', default='raspberry')
+        port = self.params.get('port', default='22')
+
+        self.ssh_start(user='ci', host=host, port=port,
+                       script='sample_script.sh')
-- 
2.20.1

Re: [PATCH 0/4] Basic hardware testing support

[MOESSBAUER, Felix]

On Wed, 2023-05-24 at 04:45 +0200, Uladzimir Bely wrote:
> Isar is currently able to run tests only on qemu targets over SSH.
> 
> This patchset adds basic support of running tests on real hardware
> targets in similar way. There are the following limitations:
> 
> - User `ci` should be pre-configured (e.g., isar-ci-ssh-setup recipe
> used when generating the image), similar to qemu testing;
> - The image should be flashed on the board before running tests;
> - The target should be turned on and accessible over SSH;
> 
> Possible ways to improve this:
> - Support turning on/off the hardware (via USB switches or similar);
> - Automation of board software update.

The flashing and powerctl features are implemented in MTDA [1]. We also
use that in a lab for CI tests of ISAR images on hardware devices.

Felix

[1] https://github.com/siemens/mtda

> 
> Uladzimir Bely (4):
>   cibuilder.py: Refactor vm_start related functions
>   testsuite: Add an interface to run commands over ssh
>   testsuite: Separate common part of kas-based test example
>   testsute: Provide an example of hardware test
> 
>  meta-isar/test/README.md                | 24 ++++++-
>  meta-isar/test/common.sh                | 30 +++++++++
>  meta-isar/test/run_test.sh              | 33 ++-------
>  meta-isar/test/run_test_hw.sh           | 10 +++
>  meta-isar/test/sample_kas_config_hw.yml | 32 +++++++++
>  meta-isar/test/sample_test.py           |  0
>  meta-isar/test/sample_test_hw.py        | 13 ++++
>  testsuite/cibuilder.py                  | 89 +++++++++++++++++------
> --
>  8 files changed, 174 insertions(+), 57 deletions(-)
>  create mode 100755 meta-isar/test/common.sh
>  create mode 100755 meta-isar/test/run_test_hw.sh
>  create mode 100644 meta-isar/test/sample_kas_config_hw.yml
>  mode change 100644 => 100755 meta-isar/test/sample_test.py
>  create mode 100755 meta-isar/test/sample_test_hw.py
> 
> -- 
> 2.20.1
> 

Re: [PATCH 0/4] Basic hardware testing support

[Florian Bezdeka]

On Wed, 2023-05-24 at 03:06 +0000, 'MOESSBAUER, Felix' via isar-users
wrote:
> On Wed, 2023-05-24 at 04:45 +0200, Uladzimir Bely wrote:
> > Isar is currently able to run tests only on qemu targets over SSH.
> > 
> > This patchset adds basic support of running tests on real hardware
> > targets in similar way. There are the following limitations:
> > 
> > - User `ci` should be pre-configured (e.g., isar-ci-ssh-setup recipe
> > used when generating the image), similar to qemu testing;
> > - The image should be flashed on the board before running tests;
> > - The target should be turned on and accessible over SSH;
> > 
> > Possible ways to improve this:
> > - Support turning on/off the hardware (via USB switches or similar);
> > - Automation of board software update.
> 
> The flashing and powerctl features are implemented in MTDA [1]. We also
> use that in a lab for CI tests of ISAR images on hardware devices.

I would say flashing is not necessary or at least no hard requirement.
We have LAVA (tool behind kernelci) setups deployed that implement NFS
based booting of ISAR generated images.

One public available instance is lava.xenomai.org. Example test runs
can be found at [1] with the CI triggers coming from the xenomai-images
project [2].

Florian

[1] https://lava.xenomai.org/scheduler/alljobs
[2] https://source.denx.de/Xenomai/xenomai-images

> 
> Felix
> 
> [1] https://github.com/siemens/mtda
> 
> > 
> > Uladzimir Bely (4):
> >   cibuilder.py: Refactor vm_start related functions
> >   testsuite: Add an interface to run commands over ssh
> >   testsuite: Separate common part of kas-based test example
> >   testsute: Provide an example of hardware test
> > 
> >  meta-isar/test/README.md                | 24 ++++++-
> >  meta-isar/test/common.sh                | 30 +++++++++
> >  meta-isar/test/run_test.sh              | 33 ++-------
> >  meta-isar/test/run_test_hw.sh           | 10 +++
> >  meta-isar/test/sample_kas_config_hw.yml | 32 +++++++++
> >  meta-isar/test/sample_test.py           |  0
> >  meta-isar/test/sample_test_hw.py        | 13 ++++
> >  testsuite/cibuilder.py                  | 89 +++++++++++++++++------
> > --
> >  8 files changed, 174 insertions(+), 57 deletions(-)
> >  create mode 100755 meta-isar/test/common.sh
> >  create mode 100755 meta-isar/test/run_test_hw.sh
> >  create mode 100644 meta-isar/test/sample_kas_config_hw.yml
> >  mode change 100644 => 100755 meta-isar/test/sample_test.py
> >  create mode 100755 meta-isar/test/sample_test_hw.py
> > 
> > -- 
> > 2.20.1
> > 
> 

Re: [PATCH 0/4] Basic hardware testing support

[Jan Kiszka]

On 24.05.23 06:30, 'Florian Bezdeka' via isar-users wrote:
> On Wed, 2023-05-24 at 03:06 +0000, 'MOESSBAUER, Felix' via isar-users
> wrote:
>> On Wed, 2023-05-24 at 04:45 +0200, Uladzimir Bely wrote:
>>> Isar is currently able to run tests only on qemu targets over SSH.
>>>
>>> This patchset adds basic support of running tests on real hardware
>>> targets in similar way. There are the following limitations:
>>>
>>> - User `ci` should be pre-configured (e.g., isar-ci-ssh-setup recipe
>>> used when generating the image), similar to qemu testing;
>>> - The image should be flashed on the board before running tests;
>>> - The target should be turned on and accessible over SSH;
>>>
>>> Possible ways to improve this:
>>> - Support turning on/off the hardware (via USB switches or similar);
>>> - Automation of board software update.
>>
>> The flashing and powerctl features are implemented in MTDA [1]. We also
>> use that in a lab for CI tests of ISAR images on hardware devices.
> 
> I would say flashing is not necessary or at least no hard requirement.
> We have LAVA (tool behind kernelci) setups deployed that implement NFS
> based booting of ISAR generated images.

If the purpose of the test is also validating the correctness of a wic
run e.g., we do need to feed a real image in. We chould use MTDA's
capability to emulate a USB stick and let the target boot from there
(doing that for many targets by now for exactly that purpose).

Jan

> 
> One public available instance is lava.xenomai.org. Example test runs
> can be found at [1] with the CI triggers coming from the xenomai-images
> project [2].
> 
> Florian
> 
> [1] https://lava.xenomai.org/scheduler/alljobs
> [2] https://source.denx.de/Xenomai/xenomai-images
> 
>>
>> Felix
>>
>> [1] https://github.com/siemens/mtda
>>
>>>
>>> Uladzimir Bely (4):
>>>   cibuilder.py: Refactor vm_start related functions
>>>   testsuite: Add an interface to run commands over ssh
>>>   testsuite: Separate common part of kas-based test example
>>>   testsute: Provide an example of hardware test
>>>
>>>  meta-isar/test/README.md                | 24 ++++++-
>>>  meta-isar/test/common.sh                | 30 +++++++++
>>>  meta-isar/test/run_test.sh              | 33 ++-------
>>>  meta-isar/test/run_test_hw.sh           | 10 +++
>>>  meta-isar/test/sample_kas_config_hw.yml | 32 +++++++++
>>>  meta-isar/test/sample_test.py           |  0
>>>  meta-isar/test/sample_test_hw.py        | 13 ++++
>>>  testsuite/cibuilder.py                  | 89 +++++++++++++++++------
>>> --
>>>  8 files changed, 174 insertions(+), 57 deletions(-)
>>>  create mode 100755 meta-isar/test/common.sh
>>>  create mode 100755 meta-isar/test/run_test_hw.sh
>>>  create mode 100644 meta-isar/test/sample_kas_config_hw.yml
>>>  mode change 100644 => 100755 meta-isar/test/sample_test.py
>>>  create mode 100755 meta-isar/test/sample_test_hw.py
>>>
>>> -- 
>>> 2.20.1
>>>
>>
> 

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH 0/4] Basic hardware testing support

[Uladzimir Bely]

On Wed, 2023-05-24 at 04:45 +0200, Uladzimir Bely wrote:
> Isar is currently able to run tests only on qemu targets over SSH.
> 
> This patchset adds basic support of running tests on real hardware
> targets in similar way. There are the following limitations:
> 
> - User `ci` should be pre-configured (e.g., isar-ci-ssh-setup recipe
> used when generating the image), similar to qemu testing;
> - The image should be flashed on the board before running tests;
> - The target should be turned on and accessible over SSH;
> 
> Possible ways to improve this:
> - Support turning on/off the hardware (via USB switches or similar);
> - Automation of board software update.
> 
> Uladzimir Bely (4):
>   cibuilder.py: Refactor vm_start related functions
>   testsuite: Add an interface to run commands over ssh
>   testsuite: Separate common part of kas-based test example
>   testsute: Provide an example of hardware test
> 
>  meta-isar/test/README.md                | 24 ++++++-
>  meta-isar/test/common.sh                | 30 +++++++++
>  meta-isar/test/run_test.sh              | 33 ++-------
>  meta-isar/test/run_test_hw.sh           | 10 +++
>  meta-isar/test/sample_kas_config_hw.yml | 32 +++++++++
>  meta-isar/test/sample_test.py           |  0
>  meta-isar/test/sample_test_hw.py        | 13 ++++
>  testsuite/cibuilder.py                  | 89 +++++++++++++++++------
> --
>  8 files changed, 174 insertions(+), 57 deletions(-)
>  create mode 100755 meta-isar/test/common.sh
>  create mode 100755 meta-isar/test/run_test_hw.sh
>  create mode 100644 meta-isar/test/sample_kas_config_hw.yml
>  mode change 100644 => 100755 meta-isar/test/sample_test.py
>  create mode 100755 meta-isar/test/sample_test_hw.py
> 
> -- 
> 2.20.1
> 

Applied to next.

Lava / MTDA / labgrid are under investigation.