Re: [PATCH 3/3] testsuite: Document custom testcase for downstreams

[Henning Schild <henning.schild@siemens.com>]

Would it maybe make sense to put some tests into meta-isar and
actually demo that layering, where some layer-local tests live in the
layer. Eventually we can maybe split up the whole testsuite into "isar"
and "meta-isar" so "core" and "layer".

Henning

Am Tue, 28 Feb 2023 08:47:38 +0100
schrieb Uladzimir Bely <ubely@ilbers.de>:

> Provide a complete example of testsuite implementation on the
> downstream side in their own testcase directory.
> 
> Mention `PYTHONPATH` that needs to be adjusted.
> 
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  testsuite/README.md | 101
> ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 101
> insertions(+)
> 
> diff --git a/testsuite/README.md b/testsuite/README.md
> index 476b8607..28036bcb 100644
> --- a/testsuite/README.md
> +++ b/testsuite/README.md
> @@ -122,3 +122,104 @@ And to execute this example:
>  $ avocado run sample.py:SampleTest.test_sample
>  ```
>  
> +## Using a different directory for custom testcases
> +
> +Downstreams may want to keep their testcases in a different directory
> +(e.g. `./test/sample.py` as top-level with test description) but
> reuse +classes implemented in Isar testsuite (e.g.
> `./isar/testsuite/*.py`). This is +a common case for downstream that
> use `kas` to handle layers they use. +
> +In this case it's important to adjust `PYTHONPATH` variable before
> running +avocado so that isar testsuite files could be found:
> +
> +```
> +# TESTSUITEDIR="/work/isar/testsuite"
> +export PYTHONPATH=${PYTHONPATH}:${TESTSUITEDIR}
> +```
> +
> +# Example of the downstream testcase
> +
> +Let's assume the downstream builds `custom_image_name` image for
> `qemuarm64` +target in order to running custom tests using qemu.
> +
> +Let's assume the [kas](https://github.com/siemens/kas) is used by
> +the downstream with the following directory structure:
> +
> +```
> +...
> +├── build/
> +├── isar/
> +├── kas
> +│   ├── qemuarm64.yml
> +│   └── kas-container
> +├── meta-custom-layer
> +└── test
> +    ├── run_test.sh
> +    ├── scripts
> +    │   └── sample_script.sh
> +    └── sample.py
> +```
> +
> +### test/sample.py:
> +
> +```
> +#!/usr/bin/env python3
> +
> +from cibase import CIBaseTest
> +
> +class SampleTest(CIBaseTest):
> +    def test_sample_script(self):
> +        self.init("/build")
> +        self.vm_start('arm64','bullseye', image='custom_image_name',
> \
> +                    script='sample_script.sh')
> +```
> +
> +### test/run_test.sh:
> +
> +```
> +#!/usr/bin/env bash
> +
> +# Make Isar testsuite accessable
> +export PYTHONPATH=${PYTHONPATH}:${TESTSUITEDIR}
> +
> +# install avocado in virtualenv in case it is not there already
> +if ! command -v avocado > /dev/null; then
> +    sudo apt-get -y update
> +    sudo apt-get -y install virtualenv
> +    rm -rf /tmp/avocado_venv
> +    virtualenv --python python3 /tmp/avocado_venv
> +    source /tmp/avocado_venv/bin/activate
> +    pip install avocado-framework==100.1
> +fi
> +
> +# Install qemu
> +if ! command -v qemu-system-aarch64 > /dev/null; then
> +  sudo apt-get -y update
> +  sudo apt-get -y install --no-install-recommends
> qemu-system-aarch64 ipxe-qemu +fi
> +
> +# Start tests in this dir
> +BASE_DIR="/build/avocado"
> +
> +# Provide working path
> +mkdir -p .config/avocado
> +cat <<EOF > .config/avocado/avocado.conf
> +[datadir.paths]
> +base_dir = ${BASE_DIR}/
> +data_dir = ${BASE_DIR}/data
> +logs_dir = ${BASE_DIR}/logs
> +test_dir = ${BASE_DIR}/test
> +EOF
> +export VIRTUAL_ENV="./"
> +
> +tsd=$(dirname $(realpath $0))/scripts
> +
> +# Run SSH tests
> +avocado run --max-parallel-tasks=1 /work/test/test.py -p
> test_script_dir=${tsd} +```
> +
> +Running testcase using `kas shell` may be done as:
> +
> +```
> +./kas/kas-container shell kas/qemuarm64.yml -c
> '/work/test/run_test.sh' +```