* [PATCH v2 0/3] Creating custom testcases based on Isar one
@ 2023-03-24 7:01 Uladzimir Bely
2023-03-24 7:01 ` [PATCH v2 1/3] testsuite: Check availability of script to run over ssh Uladzimir Bely
` (3 more replies)
0 siblings, 4 replies; 6+ messages in thread
From: Uladzimir Bely @ 2023-03-24 7:01 UTC (permalink / raw)
To: isar-users
This documents and adds an example of creating custom avocado-based
testsute that downstreams can use. It reuses testcase classes
implemented in Isar.
Additionally, some small related improvements and fixes are done
on Isar side.
Changes since v1:
- Added working example of custom testsuite in `meta-isar` layer
Uladzimir Bely (3):
testsuite: Check availability of script to run over ssh
testsuite: Move test scripts to their own subdirectory
testsuite: Exmaple of custom testcase for downstreams
meta-isar/test/.gitignore | 2 +
meta-isar/test/README.md | 16 ++++++++
meta-isar/test/run_test.sh | 39 +++++++++++++++++++
meta-isar/test/sample_kas_config.yml | 32 +++++++++++++++
meta-isar/test/sample_test.py | 9 +++++
.../test/scripts/sample_script.sh | 0
testsuite/README.md | 19 +++++++++
testsuite/cibuilder.py | 6 ++-
.../{ => scripts}/test_example_module.sh | 0
testsuite/scripts/test_getty_target.sh | 7 ++++
10 files changed, 129 insertions(+), 1 deletion(-)
create mode 100644 meta-isar/test/.gitignore
create mode 100644 meta-isar/test/README.md
create mode 100755 meta-isar/test/run_test.sh
create mode 100644 meta-isar/test/sample_kas_config.yml
create mode 100644 meta-isar/test/sample_test.py
rename testsuite/test_getty_target.sh => meta-isar/test/scripts/sample_script.sh (100%)
mode change 100644 => 100755
rename testsuite/{ => scripts}/test_example_module.sh (100%)
mode change 100644 => 100755
create mode 100755 testsuite/scripts/test_getty_target.sh
--
2.20.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v2 1/3] testsuite: Check availability of script to run over ssh
2023-03-24 7:01 [PATCH v2 0/3] Creating custom testcases based on Isar one Uladzimir Bely
@ 2023-03-24 7:01 ` Uladzimir Bely
2023-03-24 7:01 ` [PATCH v2 2/3] testsuite: Move test scripts to their own subdirectory Uladzimir Bely
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Uladzimir Bely @ 2023-03-24 7:01 UTC (permalink / raw)
To: isar-users
Running local scripts on remote machine over ssh is implemented as
`cat <script> | ssh ...". This always returns PASSED in case script
not found at specified location since the error code `cat` produces
in this case is masked by `ssh` command that finishes OK.
This fix is intended for downstreams that set custom script location
by passing `-p test_script_dir=<path>` to avocado arguments.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
testsuite/cibuilder.py | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 9e84c3a3..484a90b3 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -248,6 +248,10 @@ class CIBuilder(Test):
script_dir = self.params.get('test_script_dir',
default=os.path.abspath(os.path.dirname(__file__))) + '/'
script_path = script_dir + script
+ if not os.path.exists(script_path):
+ self.log.error('Script not found: ' + script_path)
+ return 2
+
rc = subprocess.call('cat ' + script_path + ' | ' + str(cmd_prefix), shell=True,
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
return rc
--
2.20.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v2 2/3] testsuite: Move test scripts to their own subdirectory
2023-03-24 7:01 [PATCH v2 0/3] Creating custom testcases based on Isar one Uladzimir Bely
2023-03-24 7:01 ` [PATCH v2 1/3] testsuite: Check availability of script to run over ssh Uladzimir Bely
@ 2023-03-24 7:01 ` Uladzimir Bely
2023-03-24 7:01 ` [PATCH v2 3/3] testsuite: Exmaple of custom testcase for downstreams Uladzimir Bely
2023-03-31 6:48 ` [PATCH v2 0/3] Creating custom testcases based on Isar one Uladzimir Bely
3 siblings, 0 replies; 6+ messages in thread
From: Uladzimir Bely @ 2023-03-24 7:01 UTC (permalink / raw)
To: isar-users
Just a cosmetic change to minimize mess in `testsuite` directory by
placing test scripts to their own subdirectory.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
testsuite/cibuilder.py | 2 +-
testsuite/{ => scripts}/test_example_module.sh | 0
testsuite/{ => scripts}/test_getty_target.sh | 0
3 files changed, 1 insertion(+), 1 deletion(-)
rename testsuite/{ => scripts}/test_example_module.sh (100%)
mode change 100644 => 100755
rename testsuite/{ => scripts}/test_getty_target.sh (100%)
mode change 100644 => 100755
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 484a90b3..f37c3190 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -246,7 +246,7 @@ class CIBuilder(Test):
def run_script(self, script, cmd_prefix):
script_dir = self.params.get('test_script_dir',
- default=os.path.abspath(os.path.dirname(__file__))) + '/'
+ default=os.path.abspath(os.path.dirname(__file__))) + '/scripts/'
script_path = script_dir + script
if not os.path.exists(script_path):
self.log.error('Script not found: ' + script_path)
diff --git a/testsuite/test_example_module.sh b/testsuite/scripts/test_example_module.sh
old mode 100644
new mode 100755
similarity index 100%
rename from testsuite/test_example_module.sh
rename to testsuite/scripts/test_example_module.sh
diff --git a/testsuite/test_getty_target.sh b/testsuite/scripts/test_getty_target.sh
old mode 100644
new mode 100755
similarity index 100%
rename from testsuite/test_getty_target.sh
rename to testsuite/scripts/test_getty_target.sh
--
2.20.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v2 3/3] testsuite: Exmaple of custom testcase for downstreams
2023-03-24 7:01 [PATCH v2 0/3] Creating custom testcases based on Isar one Uladzimir Bely
2023-03-24 7:01 ` [PATCH v2 1/3] testsuite: Check availability of script to run over ssh Uladzimir Bely
2023-03-24 7:01 ` [PATCH v2 2/3] testsuite: Move test scripts to their own subdirectory Uladzimir Bely
@ 2023-03-24 7:01 ` Uladzimir Bely
2023-03-26 17:30 ` Uladzimir Bely
2023-03-31 6:48 ` [PATCH v2 0/3] Creating custom testcases based on Isar one Uladzimir Bely
3 siblings, 1 reply; 6+ messages in thread
From: Uladzimir Bely @ 2023-03-24 7:01 UTC (permalink / raw)
To: isar-users
Provide a complete example of testsuite implementation on the
downstream side in their own testcase directory.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta-isar/test/.gitignore | 2 ++
meta-isar/test/README.md | 16 ++++++++++
meta-isar/test/run_test.sh | 39 +++++++++++++++++++++++++
meta-isar/test/sample_kas_config.yml | 32 ++++++++++++++++++++
meta-isar/test/sample_test.py | 9 ++++++
meta-isar/test/scripts/sample_script.sh | 7 +++++
testsuite/README.md | 19 ++++++++++++
7 files changed, 124 insertions(+)
create mode 100644 meta-isar/test/.gitignore
create mode 100644 meta-isar/test/README.md
create mode 100755 meta-isar/test/run_test.sh
create mode 100644 meta-isar/test/sample_kas_config.yml
create mode 100644 meta-isar/test/sample_test.py
create mode 100755 meta-isar/test/scripts/sample_script.sh
diff --git a/meta-isar/test/.gitignore b/meta-isar/test/.gitignore
new file mode 100644
index 00000000..fd87a392
--- /dev/null
+++ b/meta-isar/test/.gitignore
@@ -0,0 +1,2 @@
+build/
+isar/
diff --git a/meta-isar/test/README.md b/meta-isar/test/README.md
new file mode 100644
index 00000000..f8c6338c
--- /dev/null
+++ b/meta-isar/test/README.md
@@ -0,0 +1,16 @@
+# Example of the downstream testcase
+
+Use `meta-isar/test/` as working directory. Make sure `kas-container` is
+available in $PATH (look at https://github.com/siemens/kas for it).
+
+Build an image:
+
+```
+kas-container build sample_kas_config.yml
+```
+
+Run testcase:
+
+```
+kas-container shell sample_kas_config.yml -c '/work/run_test.sh'
+```
diff --git a/meta-isar/test/run_test.sh b/meta-isar/test/run_test.sh
new file mode 100755
index 00000000..c4a7d4ac
--- /dev/null
+++ b/meta-isar/test/run_test.sh
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+# Make Isar testsuite accessable
+export PYTHONPATH=${PYTHONPATH}:${TESTSUITEDIR}
+
+# install avocado in virtualenv in case it is not there already
+if ! command -v avocado > /dev/null; then
+ sudo apt-get -y update
+ sudo apt-get -y install virtualenv
+ rm -rf /tmp/avocado_venv
+ virtualenv --python python3 /tmp/avocado_venv
+ source /tmp/avocado_venv/bin/activate
+ pip install avocado-framework==100.1
+fi
+
+# Install qemu
+if ! command -v qemu-system-aarch64 > /dev/null; then
+ sudo apt-get -y update
+ sudo apt-get -y install --no-install-recommends qemu-system-aarch64 ipxe-qemu
+fi
+
+# Start tests in this dir
+BASE_DIR="/build/avocado"
+
+# Provide working path
+mkdir -p .config/avocado
+cat <<EOF > .config/avocado/avocado.conf
+[datadir.paths]
+base_dir = ${BASE_DIR}/
+data_dir = ${BASE_DIR}/data
+logs_dir = ${BASE_DIR}/logs
+test_dir = ${BASE_DIR}/test
+EOF
+export VIRTUAL_ENV="./"
+
+tsd=$(dirname $(realpath $0))/scripts
+
+# Run SSH tests
+avocado run --max-parallel-tasks=1 /work/sample_test.py -p test_script_dir=${tsd}
diff --git a/meta-isar/test/sample_kas_config.yml b/meta-isar/test/sample_kas_config.yml
new file mode 100644
index 00000000..e7635af8
--- /dev/null
+++ b/meta-isar/test/sample_kas_config.yml
@@ -0,0 +1,32 @@
+header:
+ version: 11
+
+build_system: isar
+
+distro: debian-bullseye
+machine: qemuarm64
+target: mc:qemuarm64-bullseye:isar-image-ci
+repos:
+ isar:
+ url: "https://github.com/ilbers/isar.git"
+ refspec: next
+ layers:
+ meta:
+ meta-isar:
+
+bblayers_conf_header:
+ standard: |
+ BBPATH = "${TOPDIR}"
+ BBFILES ?= ""
+
+local_conf_header:
+ standard: |
+ DISTRO_ARCH ??= "arm64"
+ PATCHRESOLVE = "noop"
+
+ USERS += "root"
+ USER_root[password] ??= "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"
+ crossbuild: |
+ ISAR_CROSS_COMPILE = "1"
+ ccache: |
+ USE_CCACHE = "1"
diff --git a/meta-isar/test/sample_test.py b/meta-isar/test/sample_test.py
new file mode 100644
index 00000000..7020c6ff
--- /dev/null
+++ b/meta-isar/test/sample_test.py
@@ -0,0 +1,9 @@
+#!/usr/bin/env python3
+
+from cibase import CIBaseTest
+
+class SampleTest(CIBaseTest):
+ def test_sample_script(self):
+ self.init("/build")
+ self.vm_start('arm64','bullseye', image='isar-image-ci', \
+ script='sample_script.sh')
diff --git a/meta-isar/test/scripts/sample_script.sh b/meta-isar/test/scripts/sample_script.sh
new file mode 100755
index 00000000..38c0fc1e
--- /dev/null
+++ b/meta-isar/test/scripts/sample_script.sh
@@ -0,0 +1,7 @@
+#!/bin/sh
+
+set -e
+
+sleep 10
+
+systemctl is-active getty.target
diff --git a/testsuite/README.md b/testsuite/README.md
index cf2fdc10..b58a1013 100644
--- a/testsuite/README.md
+++ b/testsuite/README.md
@@ -121,3 +121,22 @@ And to execute this example:
```
$ avocado run sample.py:SampleTest.test_sample
```
+
+## Using a different directory for custom testcases
+
+Downstreams may want to keep their testcases in a different directory
+(e.g. `./test/sample.py` as top-level with test description) but reuse
+classes implemented in Isar testsuite (e.g. `./isar/testsuite/*.py`). This is
+a common case for downstream that use `kas` to handle layers they use.
+
+In this case it's important to adjust `PYTHONPATH` variable before running
+avocado so that isar testsuite files could be found:
+
+```
+# TESTSUITEDIR="/work/isar/testsuite"
+export PYTHONPATH=${PYTHONPATH}:${TESTSUITEDIR}
+```
+
+# Example of the downstream testcase
+
+See `meta-isar/test` for an example of the testcase for kas-based downstream.
--
2.20.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2 3/3] testsuite: Exmaple of custom testcase for downstreams
2023-03-24 7:01 ` [PATCH v2 3/3] testsuite: Exmaple of custom testcase for downstreams Uladzimir Bely
@ 2023-03-26 17:30 ` Uladzimir Bely
0 siblings, 0 replies; 6+ messages in thread
From: Uladzimir Bely @ 2023-03-26 17:30 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely
The typo in the commit title will be fixed in v3 or during merge.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2 0/3] Creating custom testcases based on Isar one
2023-03-24 7:01 [PATCH v2 0/3] Creating custom testcases based on Isar one Uladzimir Bely
` (2 preceding siblings ...)
2023-03-24 7:01 ` [PATCH v2 3/3] testsuite: Exmaple of custom testcase for downstreams Uladzimir Bely
@ 2023-03-31 6:48 ` Uladzimir Bely
3 siblings, 0 replies; 6+ messages in thread
From: Uladzimir Bely @ 2023-03-31 6:48 UTC (permalink / raw)
To: isar-users
In mail from Friday, 24 March 2023 10:01:39 +03 user Uladzimir Bely wrote:
> This documents and adds an example of creating custom avocado-based
> testsute that downstreams can use. It reuses testcase classes
> implemented in Isar.
>
> Additionally, some small related improvements and fixes are done
> on Isar side.
>
> Changes since v1:
> - Added working example of custom testsuite in `meta-isar` layer
>
> Uladzimir Bely (3):
> testsuite: Check availability of script to run over ssh
> testsuite: Move test scripts to their own subdirectory
> testsuite: Exmaple of custom testcase for downstreams
>
> meta-isar/test/.gitignore | 2 +
> meta-isar/test/README.md | 16 ++++++++
> meta-isar/test/run_test.sh | 39 +++++++++++++++++++
> meta-isar/test/sample_kas_config.yml | 32 +++++++++++++++
> meta-isar/test/sample_test.py | 9 +++++
> .../test/scripts/sample_script.sh | 0
> testsuite/README.md | 19 +++++++++
> testsuite/cibuilder.py | 6 ++-
> .../{ => scripts}/test_example_module.sh | 0
> testsuite/scripts/test_getty_target.sh | 7 ++++
> 10 files changed, 129 insertions(+), 1 deletion(-)
> create mode 100644 meta-isar/test/.gitignore
> create mode 100644 meta-isar/test/README.md
> create mode 100755 meta-isar/test/run_test.sh
> create mode 100644 meta-isar/test/sample_kas_config.yml
> create mode 100644 meta-isar/test/sample_test.py
> rename testsuite/test_getty_target.sh =>
> meta-isar/test/scripts/sample_script.sh (100%) mode change 100644 => 100755
> rename testsuite/{ => scripts}/test_example_module.sh (100%)
> mode change 100644 => 100755
> create mode 100755 testsuite/scripts/test_getty_target.sh
Applied to next.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-03-31 6:48 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-03-24 7:01 [PATCH v2 0/3] Creating custom testcases based on Isar one Uladzimir Bely
2023-03-24 7:01 ` [PATCH v2 1/3] testsuite: Check availability of script to run over ssh Uladzimir Bely
2023-03-24 7:01 ` [PATCH v2 2/3] testsuite: Move test scripts to their own subdirectory Uladzimir Bely
2023-03-24 7:01 ` [PATCH v2 3/3] testsuite: Exmaple of custom testcase for downstreams Uladzimir Bely
2023-03-26 17:30 ` Uladzimir Bely
2023-03-31 6:48 ` [PATCH v2 0/3] Creating custom testcases based on Isar one Uladzimir Bely
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox