Re: [PATCH v6 00/15] Update Avocado testsuite

[Jan Kiszka <jan.kiszka@siemens.com>]

On 09.05.21 08:42, Anton Mikanovich wrote:
> This patchset provides implementation of build and vm_start tests
> functionality in Python with help of Avocado framework.
> Scripts ci_build and vm_smoke_test are rewritten to call Avocado.
> ci_build and vm_smoke_test are left for compatibility and
> should be replaced by direct avocado calls later.
> Original ci_build and vm_smoke_test are renamed to *_legacy.
> Build and run logs are hidden now until an error occurs.
> 
> ---
> Changes since v5:
> - Rebase on current next.
> Changes since v4:
> - Add container tests.
> - Implement testcase skipping logic.
> - Separate and protect Bullseye tests.
> - Add checks for container dependencies.
> - Add Ilbers repo key.
> - Improve error handling.
> - Fix Ubuntu qemu image name.
> Changes since v3:
> - Add automatic bitbake init.
> - Fix line lengths.
> - Fix build without repro.
> - Fix build artifact paths.
> - Prepare for Gitlab build.
> - Change debug settings.
> - Move log splitting to start_vm.
> - Add deb package install.
> Changes since v2:
> - Make CI call Avocado tests.
> - Wait less on vm_start test.
> - Split Roman's vm_start commit.
> - Put logs to build folder.
> - Rearrange patches.
> - Split build test to classes.
> - Add test case tagging.
> - Get rid of avocado-framework-plugin-varianter-yaml-to-mux.
> - Get rid of python-subprocess32.
> - Improve logging.
> Changes since v1:
> - Merge start_vm rebuild patches.
> - Fix patch comments.
> 
> Anton Mikanovich (12):
>   start_vm.py: Fix ubuntu image name
>   vm_boot_test: Fix log file path in vm_boot_test
>   vm_boot_test: Remove external varianter
>   vm_boot_test: Improve QEMU images checking
>   build_test: Refactoring build tests cases
>   testsuite: Add Python generations for testsuite in gitignore
>   testsuite: Fix test suite prepare guide
>   gitlab-ci: Add Avocado build artifacts
>   vm_boot_test: Add automatic bitbake init
>   ci_build: Migrate to Avocado
>   vm_smoke_test: Migrate to Avocado
>   ci_build: Install Avocado if needed
> 
> Roman Pletnev (3):
>   start_vm.py: Fix target name handling
>   start_vm.py: Add output and PID file vm_start.py options
>   start_vm.py: Add MIPS support
> 
>  .gitlab-ci.yml                         |   1 +
>  scripts/ci_build.sh                    | 212 +++++---------------
>  scripts/ci_build_legacy.sh             | 267 +++++++++++++++++++++++++
>  scripts/vm_smoke_test                  | 110 ++--------
>  scripts/vm_smoke_test_legacy           | 168 ++++++++++++++++
>  testsuite/.gitignore                   |   1 +
>  testsuite/README.md                    |  21 +-
>  testsuite/build_test/build_test.py     | 208 +++++++++++++++++--
>  testsuite/build_test/cibase.py         | 145 ++++++++++++++
>  testsuite/build_test/cibuilder.py      |  90 +++++++++
>  testsuite/build_test/run.sh            |   3 -
>  testsuite/build_test/run_fast.sh       |   3 +
>  testsuite/build_test/run_full.sh       |   3 +
>  testsuite/build_test/variant.yaml      |  22 --
>  testsuite/start_vm.py                  |  27 ++-
>  testsuite/vm_boot_test/run.sh          |   3 -
>  testsuite/vm_boot_test/run_fast.sh     |   3 +
>  testsuite/vm_boot_test/run_full.sh     |   3 +
>  testsuite/vm_boot_test/variant.yaml    |  22 --
>  testsuite/vm_boot_test/vm_boot_test.py | 120 +++++++++--
>  20 files changed, 1057 insertions(+), 375 deletions(-)
>  create mode 100755 scripts/ci_build_legacy.sh
>  create mode 100755 scripts/vm_smoke_test_legacy
>  create mode 100644 testsuite/.gitignore
>  create mode 100644 testsuite/build_test/cibase.py
>  create mode 100644 testsuite/build_test/cibuilder.py
>  delete mode 100755 testsuite/build_test/run.sh
>  create mode 100755 testsuite/build_test/run_fast.sh
>  create mode 100755 testsuite/build_test/run_full.sh
>  delete mode 100644 testsuite/build_test/variant.yaml
>  delete mode 100755 testsuite/vm_boot_test/run.sh
>  create mode 100755 testsuite/vm_boot_test/run_fast.sh
>  create mode 100755 testsuite/vm_boot_test/run_full.sh
>  delete mode 100644 testsuite/vm_boot_test/variant.yaml
> 

Is that equivalent to 1c1c1dd00dcb8a9c312faa25d22da0a7ddb0238d 
(amikan/devel)? Tried that revision, but this is what our CI said when 
running fast-ci:

+ avocado run /builds/ebsy/debian/isar/testsuite/build_test/build_test.py -t fast,-repro -p build_dir=./build -p quiet=1 -p cross=1
JOB ID     : 6725f043528240b81c41c00342477799b6a592f6
JOB LOG    : /builds/ebsy/debian/isar/build/job-results/job-2021-05-09T08.28-6725f04/job.log
 (1/7) /builds/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross:  ERROR: dictionary update sequence element #48 has length 1; 2 is required (0.07 s)
 (2/7) /builds/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross_bullseye:  CANCEL: KFAIL (0.05 s)
 (3/7) /builds/ebsy/debian/isar/testsuite/build_test/build_test.py:SdkTest.test_sdk:  ERROR: dictionary update sequence element #48 has length 1; 2 is required (0.07 s)
 (4/7) /builds/ebsy/debian/isar/testsuite/build_test/build_test.py:RebuildTest.test_rebuild:  ERROR: dictionary update sequence element #48 has length 1; 2 is required (1.63 s)
 (5/7) /builds/ebsy/debian/isar/testsuite/build_test/build_test.py:WicTest.test_wic_exclude:  ERROR: dictionary update sequence element #48 has length 1; 2 is required (0.07 s)
 (6/7) /builds/ebsy/debian/isar/testsuite/build_test/build_test.py:ContainerImageTest.test_nocross:  ERROR: dictionary update sequence element #48 has length 1; 2 is required (0.07 s)
 (7/7) /builds/ebsy/debian/isar/testsuite/build_test/build_test.py:ContainerSdkTest.test_container_sdk:  ERROR: dictionary update sequence element #48 has length 1; 2 is required (0.07 s)
RESULTS    : PASS 0 | ERROR 6 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 1
JOB TIME   : 2.71 s

I can provide the full log if needed.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux