Re: [PATCH v8 00/17] Update Avocado testsuite

[Jan Kiszka <jan.kiszka@siemens.com>]

On 21.06.21 08:49, Jan Kiszka wrote:
> On 02.06.21 10:09, Anton Mikanovich wrote:
>> This patchset provides implementation of build and vm_start tests
>> functionality in Python with help of Avocado framework.
>> Scripts ci_build and vm_smoke_test are rewritten to call Avocado.
>> ci_build and vm_smoke_test are left for compatibility and
>> should be replaced by direct avocado calls later.
>> Original ci_build and vm_smoke_test are renamed to *_legacy.
>> Build and run logs are hidden now until an error occurs.
>>
>> ---
>> Changes since v7:
>> - Remove code duplication.
>> - Fix build_dir parsing.
>> - Add missed protect files backup logic.
>> Changes since v6:
>> - Improve environment parsing.
>> - Protect files backup logic.
>> - Rebuild full log output option.
>> - Protect ubuntu target with KFAIL.
>> Changes since v5:
>> - Rebase on current next.
>> Changes since v4:
>> - Add container tests.
>> - Implement testcase skipping logic.
>> - Separate and protect Bullseye tests.
>> - Add checks for container dependencies.
>> - Add Ilbers repo key.
>> - Improve error handling.
>> - Fix Ubuntu qemu image name.
>> Changes since v3:
>> - Add automatic bitbake init.
>> - Fix line lengths.
>> - Fix build without repro.
>> - Fix build artifact paths.
>> - Prepare for Gitlab build.
>> - Change debug settings.
>> - Move log splitting to start_vm.
>> - Add deb package install.
>> Changes since v2:
>> - Make CI call Avocado tests.
>> - Wait less on vm_start test.
>> - Split Roman's vm_start commit.
>> - Put logs to build folder.
>> - Rearrange patches.
>> - Split build test to classes.
>> - Add test case tagging.
>> - Get rid of avocado-framework-plugin-varianter-yaml-to-mux.
>> - Get rid of python-subprocess32.
>> - Improve logging.
>> Changes since v1:
>> - Merge start_vm rebuild patches.
>> - Fix patch comments.
>>
>> Anton Mikanovich (14):
>>   start_vm.py: Fix ubuntu image name
>>   vm_boot_test: Fix log file path in vm_boot_test
>>   vm_boot_test: Remove external varianter
>>   vm_boot_test: Improve QEMU images checking
>>   build_test: Refactoring build tests cases
>>   testsuite: Add Python generations for testsuite in gitignore
>>   testsuite: Fix test suite prepare guide
>>   gitlab-ci: Add Avocado build artifacts
>>   gitlab-ci: Add debug flag
>>   vm_boot_test: Add automatic bitbake init
>>   build_test: Protect ubuntu target with KFAIL
>>   ci_build: Migrate to Avocado
>>   vm_smoke_test: Migrate to Avocado
>>   ci_build: Install Avocado if needed
>>
>> Roman Pletnev (3):
>>   start_vm.py: Fix target name handling
>>   start_vm.py: Add output and PID file vm_start.py options
>>   start_vm.py: Add MIPS support
>>
>>  .gitlab-ci.yml                         |   5 +-
>>  scripts/ci_build.sh                    | 213 ++++----------------
>>  scripts/ci_build_legacy.sh             | 267 +++++++++++++++++++++++++
>>  scripts/vm_smoke_test                  | 110 ++--------
>>  scripts/vm_smoke_test_legacy           | 168 ++++++++++++++++
>>  testsuite/.gitignore                   |   1 +
>>  testsuite/README.md                    |  21 +-
>>  testsuite/build_test/build_test.py     | 217 ++++++++++++++++++--
>>  testsuite/build_test/cibase.py         | 110 ++++++++++
>>  testsuite/build_test/cibuilder.py      | 103 ++++++++++
>>  testsuite/build_test/run.sh            |   3 -
>>  testsuite/build_test/run_fast.sh       |   3 +
>>  testsuite/build_test/run_full.sh       |   3 +
>>  testsuite/build_test/variant.yaml      |  22 --
>>  testsuite/start_vm.py                  |  27 ++-
>>  testsuite/vm_boot_test/run.sh          |   3 -
>>  testsuite/vm_boot_test/run_fast.sh     |   3 +
>>  testsuite/vm_boot_test/run_full.sh     |   3 +
>>  testsuite/vm_boot_test/variant.yaml    |  22 --
>>  testsuite/vm_boot_test/vm_boot_test.py | 121 +++++++++--
>>  20 files changed, 1043 insertions(+), 382 deletions(-)
>>  create mode 100755 scripts/ci_build_legacy.sh
>>  create mode 100755 scripts/vm_smoke_test_legacy
>>  create mode 100644 testsuite/.gitignore
>>  create mode 100644 testsuite/build_test/cibase.py
>>  create mode 100644 testsuite/build_test/cibuilder.py
>>  delete mode 100755 testsuite/build_test/run.sh
>>  create mode 100755 testsuite/build_test/run_fast.sh
>>  create mode 100755 testsuite/build_test/run_full.sh
>>  delete mode 100644 testsuite/build_test/variant.yaml
>>  delete mode 100755 testsuite/vm_boot_test/run.sh
>>  create mode 100755 testsuite/vm_boot_test/run_fast.sh
>>  create mode 100755 testsuite/vm_boot_test/run_full.sh
>>  delete mode 100644 testsuite/vm_boot_test/variant.yaml
>>
> 
> I tried that revision in our CI, but all attempts are failing,
> apparently due to connectivity issues. That would be something to blame
> on our infrastructure - if current next wouldn't pass... Retrying once
> again.

Seems there to be some changing fetch patterns, and that upset our local
proxies once again. Isar is not to blaim for the proxy mess. I was
shortly confused that do_rootfs_postprocess is always failing with
fetches, but those are related to cache_deb_src pulling all related
sources. Wasn't that tested so far, and therefore current next didn't
trigger that for us?

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux