Re: [PATCH v8 00/17] Update Avocado testsuite

[Jan Kiszka <jan.kiszka@siemens.com>]

On 02.06.21 10:09, Anton Mikanovich wrote:
> This patchset provides implementation of build and vm_start tests
> functionality in Python with help of Avocado framework.
> Scripts ci_build and vm_smoke_test are rewritten to call Avocado.
> ci_build and vm_smoke_test are left for compatibility and
> should be replaced by direct avocado calls later.
> Original ci_build and vm_smoke_test are renamed to *_legacy.
> Build and run logs are hidden now until an error occurs.
> 
> ---
> Changes since v7:
> - Remove code duplication.
> - Fix build_dir parsing.
> - Add missed protect files backup logic.
> Changes since v6:
> - Improve environment parsing.
> - Protect files backup logic.
> - Rebuild full log output option.
> - Protect ubuntu target with KFAIL.
> Changes since v5:
> - Rebase on current next.
> Changes since v4:
> - Add container tests.
> - Implement testcase skipping logic.
> - Separate and protect Bullseye tests.
> - Add checks for container dependencies.
> - Add Ilbers repo key.
> - Improve error handling.
> - Fix Ubuntu qemu image name.
> Changes since v3:
> - Add automatic bitbake init.
> - Fix line lengths.
> - Fix build without repro.
> - Fix build artifact paths.
> - Prepare for Gitlab build.
> - Change debug settings.
> - Move log splitting to start_vm.
> - Add deb package install.
> Changes since v2:
> - Make CI call Avocado tests.
> - Wait less on vm_start test.
> - Split Roman's vm_start commit.
> - Put logs to build folder.
> - Rearrange patches.
> - Split build test to classes.
> - Add test case tagging.
> - Get rid of avocado-framework-plugin-varianter-yaml-to-mux.
> - Get rid of python-subprocess32.
> - Improve logging.
> Changes since v1:
> - Merge start_vm rebuild patches.
> - Fix patch comments.
> 
> Anton Mikanovich (14):
>   start_vm.py: Fix ubuntu image name
>   vm_boot_test: Fix log file path in vm_boot_test
>   vm_boot_test: Remove external varianter
>   vm_boot_test: Improve QEMU images checking
>   build_test: Refactoring build tests cases
>   testsuite: Add Python generations for testsuite in gitignore
>   testsuite: Fix test suite prepare guide
>   gitlab-ci: Add Avocado build artifacts
>   gitlab-ci: Add debug flag
>   vm_boot_test: Add automatic bitbake init
>   build_test: Protect ubuntu target with KFAIL
>   ci_build: Migrate to Avocado
>   vm_smoke_test: Migrate to Avocado
>   ci_build: Install Avocado if needed
> 
> Roman Pletnev (3):
>   start_vm.py: Fix target name handling
>   start_vm.py: Add output and PID file vm_start.py options
>   start_vm.py: Add MIPS support
> 
>  .gitlab-ci.yml                         |   5 +-
>  scripts/ci_build.sh                    | 213 ++++----------------
>  scripts/ci_build_legacy.sh             | 267 +++++++++++++++++++++++++
>  scripts/vm_smoke_test                  | 110 ++--------
>  scripts/vm_smoke_test_legacy           | 168 ++++++++++++++++
>  testsuite/.gitignore                   |   1 +
>  testsuite/README.md                    |  21 +-
>  testsuite/build_test/build_test.py     | 217 ++++++++++++++++++--
>  testsuite/build_test/cibase.py         | 110 ++++++++++
>  testsuite/build_test/cibuilder.py      | 103 ++++++++++
>  testsuite/build_test/run.sh            |   3 -
>  testsuite/build_test/run_fast.sh       |   3 +
>  testsuite/build_test/run_full.sh       |   3 +
>  testsuite/build_test/variant.yaml      |  22 --
>  testsuite/start_vm.py                  |  27 ++-
>  testsuite/vm_boot_test/run.sh          |   3 -
>  testsuite/vm_boot_test/run_fast.sh     |   3 +
>  testsuite/vm_boot_test/run_full.sh     |   3 +
>  testsuite/vm_boot_test/variant.yaml    |  22 --
>  testsuite/vm_boot_test/vm_boot_test.py | 121 +++++++++--
>  20 files changed, 1043 insertions(+), 382 deletions(-)
>  create mode 100755 scripts/ci_build_legacy.sh
>  create mode 100755 scripts/vm_smoke_test_legacy
>  create mode 100644 testsuite/.gitignore
>  create mode 100644 testsuite/build_test/cibase.py
>  create mode 100644 testsuite/build_test/cibuilder.py
>  delete mode 100755 testsuite/build_test/run.sh
>  create mode 100755 testsuite/build_test/run_fast.sh
>  create mode 100755 testsuite/build_test/run_full.sh
>  delete mode 100644 testsuite/build_test/variant.yaml
>  delete mode 100755 testsuite/vm_boot_test/run.sh
>  create mode 100755 testsuite/vm_boot_test/run_fast.sh
>  create mode 100755 testsuite/vm_boot_test/run_full.sh
>  delete mode 100644 testsuite/vm_boot_test/variant.yaml
> 

I tried that revision in our CI, but all attempts are failing,
apparently due to connectivity issues. That would be something to blame
on our infrastructure - if current next wouldn't pass... Retrying once
again.


Then the log output is not yet optimally tuned. Avocado seems to spit
out lengthy unformatted error reports of the kind

TestFail: b'ERROR: mc:container-amd64-bullseye:isar-image-base-1.0-r0
do_rootfs_postprocess: Error executing a python function in
exec_python_func() autogenerated:\n\nThe stack trace of python calls
that resulted in this exception/failure was:\nFile: \'exec_python_func()
autogenerated\', lineno: 2, function: <module>\n     0001:\n ***
0002:do_rootfs_postprocess(d)\n     0003:\nFile:
\'/builds/Vs1BETDQ/0/ebsy/debian/isar/meta/classes/rootfs.bbclass\',
lineno: 248, function: do_rootfs_postprocess\n     0244:    if cmds is
None or not cmds.strip():\n     0245:        return\n     0246:    cmds
= cmds.split()\n     0247:    for i, cmd in enumerate(cmds):\n *** 0248:
       bb.build.exec_func(cmd, d)\n     0249:
progress_reporter.update(int(i / len(cmds) * 100))\n     0250:}\n
0251:addtask rootfs_postprocess before do_rootfs\n     0252:\nFile:
\'/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/build.py\', lineno:
251, function: exec_func\n     0247:    with
bb.utils.fileslocked(lockfiles):\n     0248:        if ispython:\n
0249:            exec_func_python(func, d, runfile, cwd=adir)\n
0250:        else:\n *** 0251:            exec_func_shell(func, d,
runfile, cwd=adir)\n     0252:\n     0253:    try:\n     0254:
curcwd = os.getcwd()\n     0255:    except:\nFile:
\'/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/build.py\', lineno:
452, function: exec_func_shell\n     0448:    with open(fifopath,
\'r+b\', buffering=0) as fifo:\n     0449:        try:\n     0450:
      bb.debug(2, "Executing shell function %s" % func)\n     0451:
       with open(os.devnull, \'r+\') as stdin, logfile:\n *** 0452:
           bb.process.run(cmd, shell=False, stdin=stdin, log=logfile,
extrafiles=[(fifo,readfifo)])\n     0453:        finally:\n     0454:
         os.unlink(fifopath)\n     0455:\n     0456:    bb.debug(2,
"Shell function %s finished" % func)\nFile:
\'/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/process.py\',
lineno: 182, function: run\n     0178:        if not stderr is None:\n
   0179:            stderr = stderr.decode("utf-8")\n     0180:\n
0181:    if pipe.returncode != 0:\n *** 0182:        raise
ExecutionError(cmd, pipe.returncode, stdout, stderr)\n     0183:
return stdout, stderr\nException: bb.process.ExecutionError: Execution
of
\'/builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/debian-bullseye-amd64/isar-image-base-container-amd64-container-img/1.0-r0/temp/run.cache_deb_src.5240\'
failed with exit code 100:\nReading package lists...\nNeed to get 1839
kB of source archives.\nGet:1 http://deb.debian.org/debian bullseye/main
fuse 2.9.9-5 (dsc) [2312 B]\nGet:2 htt

Those a likely duplicates of what the verbose log reporting already
listed, and that was formatted in readable manner. These duplicate dumps
are very unhandy and bloat the log for no value.


There is also some warning-like error about journalctl during startup:

+ avocado --show=test run
/builds/Vs1BETDQ/0/ebsy/debian/isar/testsuite/build_test/build_test.py
-t fast,-repro -p build_dir=./build -p quiet=1 -p cross=1
Commands configured by file:
/usr/lib/python3.7/dist-packages/avocado/etc/avocado/sysinfo/commands
Files configured by file:
/usr/lib/python3.7/dist-packages/avocado/etc/avocado/sysinfo/files
File
/usr/lib/python3.7/dist-packages/avocado/etc/avocado/sysinfo/fail_commands
does not exist.
File
/usr/lib/python3.7/dist-packages/avocado/etc/avocado/sysinfo/fail_files
does not exist.
Profilers configured by file:
/usr/lib/python3.7/dist-packages/avocado/etc/avocado/sysinfo/profilers
Profiler disabled
Journalctl collection failed: [Errno 2] No such file or directory:
'journalctl' (journalctl --quiet --lines 1 --output json): 'journalctl'
Not logging /sys/kernel/debug/sched_features (file does not exist)
Not logging /proc/pci (file does not exist)
Not logging /proc/slabinfo (lack of permissions)
Command line: /usr/bin/avocado --show=test run
/builds/Vs1BETDQ/0/ebsy/debian/isar/testsuite/build_test/build_test.py
-t fast,-repro -p build_dir=./build -p quiet=1 -p cross=1

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux