[PATCH 0/3] Creating custom testcases based on Isar one

[PATCH 0/3] Creating custom testcases based on Isar one

[Uladzimir Bely]

This documents how to create custom (downstream) avocado-based
testcase that reuses testcase classes implemented in Isar.

Additionally, some small related improvements and fixes are done
on Isar side.

Uladzimir Bely (3):
  testsuite: Check availability of script to run over ssh
  testsuite: Move test scripts to their own subdirectory
  testsuite: Document custom testcase for downstreams

 testsuite/README.md                           | 101 ++++++++++++++++++
 testsuite/cibuilder.py                        |   6 +-
 .../{ => scripts}/test_example_module.sh      |   0
 testsuite/{ => scripts}/test_getty_target.sh  |   0
 4 files changed, 106 insertions(+), 1 deletion(-)
 rename testsuite/{ => scripts}/test_example_module.sh (100%)
 mode change 100644 => 100755
 rename testsuite/{ => scripts}/test_getty_target.sh (100%)
 mode change 100644 => 100755

-- 
2.20.1

[PATCH 1/3] testsuite: Check availability of script to run over ssh

[Uladzimir Bely]

Running local scripts on remote machine over ssh is implemented as
`cat <script> | ssh ...". This always returns PASSED in case script
not found at specified location since the error code `cat` produces
in this case is masked by `ssh` command that finishes OK.

This fix is intended for downstreams that set custom script location
by passing `-p test_script_dir=<path>` to avocado arguments.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/cibuilder.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 0f84ccaf..d3d90a22 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -244,6 +244,10 @@ class CIBuilder(Test):
         script_dir = self.params.get('test_script_dir',
                                      default=os.path.abspath(os.path.dirname(__file__))) + '/'
         script_path = script_dir + script
+        if not os.path.exists(script_path):
+            self.log.error('Script not found: ' + script_path)
+            return 2
+
         rc = subprocess.call('cat ' + script_path + ' | ' + str(cmd_prefix), shell=True,
                              stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
         return rc
-- 
2.20.1

[PATCH 2/3] testsuite: Move test scripts to their own subdirectory

[Uladzimir Bely]

Just a cosmetic change to minimize mess in `testsuite` directory by
placing test scripts to their own subdirectory.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/cibuilder.py                         | 2 +-
 testsuite/{ => scripts}/test_example_module.sh | 0
 testsuite/{ => scripts}/test_getty_target.sh   | 0
 3 files changed, 1 insertion(+), 1 deletion(-)
 rename testsuite/{ => scripts}/test_example_module.sh (100%)
 mode change 100644 => 100755
 rename testsuite/{ => scripts}/test_getty_target.sh (100%)
 mode change 100644 => 100755

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index d3d90a22..9728bb03 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -242,7 +242,7 @@ class CIBuilder(Test):
 
     def run_script(self, script, cmd_prefix):
         script_dir = self.params.get('test_script_dir',
-                                     default=os.path.abspath(os.path.dirname(__file__))) + '/'
+                                     default=os.path.abspath(os.path.dirname(__file__))) + '/scripts/'
         script_path = script_dir + script
         if not os.path.exists(script_path):
             self.log.error('Script not found: ' + script_path)
diff --git a/testsuite/test_example_module.sh b/testsuite/scripts/test_example_module.sh
old mode 100644
new mode 100755
similarity index 100%
rename from testsuite/test_example_module.sh
rename to testsuite/scripts/test_example_module.sh
diff --git a/testsuite/test_getty_target.sh b/testsuite/scripts/test_getty_target.sh
old mode 100644
new mode 100755
similarity index 100%
rename from testsuite/test_getty_target.sh
rename to testsuite/scripts/test_getty_target.sh
-- 
2.20.1

[PATCH 3/3] testsuite: Document custom testcase for downstreams

[Uladzimir Bely]

Provide a complete example of testsuite implementation on the
downstream side in their own testcase directory.

Mention `PYTHONPATH` that needs to be adjusted.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/README.md | 101 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 101 insertions(+)

diff --git a/testsuite/README.md b/testsuite/README.md
index 476b8607..28036bcb 100644
--- a/testsuite/README.md
+++ b/testsuite/README.md
@@ -122,3 +122,104 @@ And to execute this example:
 $ avocado run sample.py:SampleTest.test_sample
 ```
 
+## Using a different directory for custom testcases
+
+Downstreams may want to keep their testcases in a different directory
+(e.g. `./test/sample.py` as top-level with test description) but reuse
+classes implemented in Isar testsuite (e.g. `./isar/testsuite/*.py`). This is
+a common case for downstream that use `kas` to handle layers they use.
+
+In this case it's important to adjust `PYTHONPATH` variable before running
+avocado so that isar testsuite files could be found:
+
+```
+# TESTSUITEDIR="/work/isar/testsuite"
+export PYTHONPATH=${PYTHONPATH}:${TESTSUITEDIR}
+```
+
+# Example of the downstream testcase
+
+Let's assume the downstream builds `custom_image_name` image for `qemuarm64`
+target in order to running custom tests using qemu.
+
+Let's assume the [kas](https://github.com/siemens/kas) is used by
+the downstream with the following directory structure:
+
+```
+...
+├── build/
+├── isar/
+├── kas
+│   ├── qemuarm64.yml
+│   └── kas-container
+├── meta-custom-layer
+└── test
+    ├── run_test.sh
+    ├── scripts
+    │   └── sample_script.sh
+    └── sample.py
+```
+
+### test/sample.py:
+
+```
+#!/usr/bin/env python3
+
+from cibase import CIBaseTest
+
+class SampleTest(CIBaseTest):
+    def test_sample_script(self):
+        self.init("/build")
+        self.vm_start('arm64','bullseye', image='custom_image_name', \
+                    script='sample_script.sh')
+```
+
+### test/run_test.sh:
+
+```
+#!/usr/bin/env bash
+
+# Make Isar testsuite accessable
+export PYTHONPATH=${PYTHONPATH}:${TESTSUITEDIR}
+
+# install avocado in virtualenv in case it is not there already
+if ! command -v avocado > /dev/null; then
+    sudo apt-get -y update
+    sudo apt-get -y install virtualenv
+    rm -rf /tmp/avocado_venv
+    virtualenv --python python3 /tmp/avocado_venv
+    source /tmp/avocado_venv/bin/activate
+    pip install avocado-framework==100.1
+fi
+
+# Install qemu
+if ! command -v qemu-system-aarch64 > /dev/null; then
+  sudo apt-get -y update
+  sudo apt-get -y install --no-install-recommends qemu-system-aarch64 ipxe-qemu
+fi
+
+# Start tests in this dir
+BASE_DIR="/build/avocado"
+
+# Provide working path
+mkdir -p .config/avocado
+cat <<EOF > .config/avocado/avocado.conf
+[datadir.paths]
+base_dir = ${BASE_DIR}/
+data_dir = ${BASE_DIR}/data
+logs_dir = ${BASE_DIR}/logs
+test_dir = ${BASE_DIR}/test
+EOF
+export VIRTUAL_ENV="./"
+
+tsd=$(dirname $(realpath $0))/scripts
+
+# Run SSH tests
+avocado run --max-parallel-tasks=1 /work/test/test.py -p test_script_dir=${tsd}
+```
+
+Running testcase using `kas shell` may be done as:
+
+```
+./kas/kas-container shell kas/qemuarm64.yml -c '/work/test/run_test.sh'
+```
-- 
2.20.1

Re: [PATCH 3/3] testsuite: Document custom testcase for downstreams

[Henning Schild]

Would it maybe make sense to put some tests into meta-isar and
actually demo that layering, where some layer-local tests live in the
layer. Eventually we can maybe split up the whole testsuite into "isar"
and "meta-isar" so "core" and "layer".

Henning

Am Tue, 28 Feb 2023 08:47:38 +0100
schrieb Uladzimir Bely <ubely@ilbers.de>:

> Provide a complete example of testsuite implementation on the
> downstream side in their own testcase directory.
> 
> Mention `PYTHONPATH` that needs to be adjusted.
> 
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  testsuite/README.md | 101
> ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 101
> insertions(+)
> 
> diff --git a/testsuite/README.md b/testsuite/README.md
> index 476b8607..28036bcb 100644
> --- a/testsuite/README.md
> +++ b/testsuite/README.md
> @@ -122,3 +122,104 @@ And to execute this example:
>  $ avocado run sample.py:SampleTest.test_sample
>  ```
>  
> +## Using a different directory for custom testcases
> +
> +Downstreams may want to keep their testcases in a different directory
> +(e.g. `./test/sample.py` as top-level with test description) but
> reuse +classes implemented in Isar testsuite (e.g.
> `./isar/testsuite/*.py`). This is +a common case for downstream that
> use `kas` to handle layers they use. +
> +In this case it's important to adjust `PYTHONPATH` variable before
> running +avocado so that isar testsuite files could be found:
> +
> +```
> +# TESTSUITEDIR="/work/isar/testsuite"
> +export PYTHONPATH=${PYTHONPATH}:${TESTSUITEDIR}
> +```
> +
> +# Example of the downstream testcase
> +
> +Let's assume the downstream builds `custom_image_name` image for
> `qemuarm64` +target in order to running custom tests using qemu.
> +
> +Let's assume the [kas](https://github.com/siemens/kas) is used by
> +the downstream with the following directory structure:
> +
> +```
> +...
> +├── build/
> +├── isar/
> +├── kas
> +│   ├── qemuarm64.yml
> +│   └── kas-container
> +├── meta-custom-layer
> +└── test
> +    ├── run_test.sh
> +    ├── scripts
> +    │   └── sample_script.sh
> +    └── sample.py
> +```
> +
> +### test/sample.py:
> +
> +```
> +#!/usr/bin/env python3
> +
> +from cibase import CIBaseTest
> +
> +class SampleTest(CIBaseTest):
> +    def test_sample_script(self):
> +        self.init("/build")
> +        self.vm_start('arm64','bullseye', image='custom_image_name',
> \
> +                    script='sample_script.sh')
> +```
> +
> +### test/run_test.sh:
> +
> +```
> +#!/usr/bin/env bash
> +
> +# Make Isar testsuite accessable
> +export PYTHONPATH=${PYTHONPATH}:${TESTSUITEDIR}
> +
> +# install avocado in virtualenv in case it is not there already
> +if ! command -v avocado > /dev/null; then
> +    sudo apt-get -y update
> +    sudo apt-get -y install virtualenv
> +    rm -rf /tmp/avocado_venv
> +    virtualenv --python python3 /tmp/avocado_venv
> +    source /tmp/avocado_venv/bin/activate
> +    pip install avocado-framework==100.1
> +fi
> +
> +# Install qemu
> +if ! command -v qemu-system-aarch64 > /dev/null; then
> +  sudo apt-get -y update
> +  sudo apt-get -y install --no-install-recommends
> qemu-system-aarch64 ipxe-qemu +fi
> +
> +# Start tests in this dir
> +BASE_DIR="/build/avocado"
> +
> +# Provide working path
> +mkdir -p .config/avocado
> +cat <<EOF > .config/avocado/avocado.conf
> +[datadir.paths]
> +base_dir = ${BASE_DIR}/
> +data_dir = ${BASE_DIR}/data
> +logs_dir = ${BASE_DIR}/logs
> +test_dir = ${BASE_DIR}/test
> +EOF
> +export VIRTUAL_ENV="./"
> +
> +tsd=$(dirname $(realpath $0))/scripts
> +
> +# Run SSH tests
> +avocado run --max-parallel-tasks=1 /work/test/test.py -p
> test_script_dir=${tsd} +```
> +
> +Running testcase using `kas shell` may be done as:
> +
> +```
> +./kas/kas-container shell kas/qemuarm64.yml -c
> '/work/test/run_test.sh' +```