* Re: [PATCH v14 00/16] Update Avocado testsuite [not found] <1633020197-26007-1-git-send-email-anton.mikanovich@promwad.com> @ 2021-10-13 14:06 ` Anton Mikanovich 0 siblings, 0 replies; 9+ messages in thread From: Anton Mikanovich @ 2021-10-13 14:06 UTC (permalink / raw) To: isar-users; +Cc: Henning Schild On 30.09.21 19:43, Anton Mikanovich wrote: > This patchset provides implementation of build and vm_start tests > functionality in Python with help of Avocado framework. > Scripts ci_build and vm_smoke_test are rewritten to call Avocado. > Build and run logs are hidden by default now until an error occurs. > > Resending v14 because v13 was just a copy of v12 by mistake. Applied to next. -- Anton Mikanovich Promwad Ltd. External service provider of ilbers GmbH Maria-Merian-Str. 8 85521 Ottobrunn, Germany +49 (89) 122 67 24-0 Commercial register Munich, HRB 214197 General Manager: Baurzhan Ismagulov ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v14 00/16] Update Avocado testsuite @ 2021-09-30 16:45 Anton Mikanovich 2021-10-14 8:54 ` Jan Kiszka 0 siblings, 1 reply; 9+ messages in thread From: Anton Mikanovich @ 2021-09-30 16:45 UTC (permalink / raw) To: isar-users; +Cc: Anton Mikanovich This patchset provides implementation of build and vm_start tests functionality in Python with help of Avocado framework. Scripts ci_build and vm_smoke_test are rewritten to call Avocado. Build and run logs are hidden by default now until an error occurs. Resending v14 because v13 was just a copy of v12 by mistake. --- Changes since v13: - Remove legacy scripts. Changes since v12: - No changes, wrong patchset. Changes since v11: - Rebase on next. - Init environment in vm_smoke_test. Changes since v10: - Rebase on next. - Add DISTRO_APT_PREMIRRORS usage. - Remove avocado deb package install. - Set default runner. Changes since v9: - Rebase on next. - Fix avocado artifacts path. Changes since v8: - Remove duplication in logs. - Disable sysinfo plugin. - Show full errors in default mode. Changes since v7: - Remove code duplication. - Fix build_dir parsing. - Add missed protect files backup logic. Changes since v6: - Improve environment parsing. - Protect files backup logic. - Rebuild full log output option. - Protect ubuntu target with KFAIL. Changes since v5: - Rebase on current next. Changes since v4: - Add container tests. - Implement testcase skipping logic. - Separate and protect Bullseye tests. - Add checks for container dependencies. - Add Ilbers repo key. - Improve error handling. - Fix Ubuntu qemu image name. Changes since v3: - Add automatic bitbake init. - Fix line lengths. - Fix build without repro. - Fix build artifact paths. - Prepare for Gitlab build. - Change debug settings. - Move log splitting to start_vm. - Add deb package install. Changes since v2: - Make CI call Avocado tests. - Wait less on vm_start test. - Split Roman's vm_start commit. - Put logs to build folder. - Rearrange patches. - Split build test to classes. - Add test case tagging. - Get rid of avocado-framework-plugin-varianter-yaml-to-mux. - Get rid of python-subprocess32. - Improve logging. Changes since v1: - Merge start_vm rebuild patches. - Fix patch comments. Anton Mikanovich (13): start_vm.py: Fix ubuntu image name vm_boot_test: Fix log file path in vm_boot_test vm_boot_test: Remove external varianter vm_boot_test: Improve QEMU images checking build_test: Refactoring build tests cases testsuite: Add Python generations for testsuite in gitignore testsuite: Fix test suite prepare guide gitlab-ci: Add Avocado build artifacts gitlab-ci: Add debug flag vm_boot_test: Add automatic bitbake init build_test: Protect ubuntu target with KFAIL ci_build: Migrate to Avocado vm_smoke_test: Migrate to Avocado Roman Pletnev (3): start_vm.py: Fix target name handling start_vm.py: Add output and PID file vm_start.py options start_vm.py: Add MIPS support .gitlab-ci.yml | 5 +- scripts/ci_build.sh | 208 ++++--------------------------- scripts/vm_smoke_test | 120 +++++------------- testsuite/.gitignore | 1 + testsuite/README.md | 21 +--- testsuite/build_test/build_test.py | 219 ++++++++++++++++++++++++++++++--- testsuite/build_test/cibase.py | 116 +++++++++++++++++ testsuite/build_test/cibuilder.py | 132 ++++++++++++++++++++ testsuite/build_test/run.sh | 3 - testsuite/build_test/run_fast.sh | 3 + testsuite/build_test/run_full.sh | 3 + testsuite/build_test/variant.yaml | 22 ---- testsuite/start_vm.py | 27 ++-- testsuite/vm_boot_test/run.sh | 3 - testsuite/vm_boot_test/run_fast.sh | 3 + testsuite/vm_boot_test/run_full.sh | 3 + testsuite/vm_boot_test/variant.yaml | 22 ---- testsuite/vm_boot_test/vm_boot_test.py | 121 +++++++++++++++--- 18 files changed, 641 insertions(+), 391 deletions(-) create mode 100644 testsuite/.gitignore create mode 100644 testsuite/build_test/cibase.py create mode 100644 testsuite/build_test/cibuilder.py delete mode 100755 testsuite/build_test/run.sh create mode 100755 testsuite/build_test/run_fast.sh create mode 100755 testsuite/build_test/run_full.sh delete mode 100644 testsuite/build_test/variant.yaml delete mode 100755 testsuite/vm_boot_test/run.sh create mode 100755 testsuite/vm_boot_test/run_fast.sh create mode 100755 testsuite/vm_boot_test/run_full.sh delete mode 100644 testsuite/vm_boot_test/variant.yaml -- 2.7.4 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v14 00/16] Update Avocado testsuite 2021-09-30 16:45 Anton Mikanovich @ 2021-10-14 8:54 ` Jan Kiszka 2021-10-14 10:25 ` Baurzhan Ismagulov 0 siblings, 1 reply; 9+ messages in thread From: Jan Kiszka @ 2021-10-14 8:54 UTC (permalink / raw) To: Anton Mikanovich, isar-users; +Cc: Schmidt, Adriaan On 30.09.21 18:45, Anton Mikanovich wrote: > This patchset provides implementation of build and vm_start tests > functionality in Python with help of Avocado framework. > Scripts ci_build and vm_smoke_test are rewritten to call Avocado. > Build and run logs are hidden by default now until an error occurs. > > Resending v14 because v13 was just a copy of v12 by mistake. > > --- > Changes since v13: > - Remove legacy scripts. > Changes since v12: > - No changes, wrong patchset. > Changes since v11: > - Rebase on next. > - Init environment in vm_smoke_test. > Changes since v10: > - Rebase on next. > - Add DISTRO_APT_PREMIRRORS usage. > - Remove avocado deb package install. > - Set default runner. > Changes since v9: > - Rebase on next. > - Fix avocado artifacts path. > Changes since v8: > - Remove duplication in logs. > - Disable sysinfo plugin. > - Show full errors in default mode. > Changes since v7: > - Remove code duplication. > - Fix build_dir parsing. > - Add missed protect files backup logic. > Changes since v6: > - Improve environment parsing. > - Protect files backup logic. > - Rebuild full log output option. > - Protect ubuntu target with KFAIL. > Changes since v5: > - Rebase on current next. > Changes since v4: > - Add container tests. > - Implement testcase skipping logic. > - Separate and protect Bullseye tests. > - Add checks for container dependencies. > - Add Ilbers repo key. > - Improve error handling. > - Fix Ubuntu qemu image name. > Changes since v3: > - Add automatic bitbake init. > - Fix line lengths. > - Fix build without repro. > - Fix build artifact paths. > - Prepare for Gitlab build. > - Change debug settings. > - Move log splitting to start_vm. > - Add deb package install. > Changes since v2: > - Make CI call Avocado tests. > - Wait less on vm_start test. > - Split Roman's vm_start commit. > - Put logs to build folder. > - Rearrange patches. > - Split build test to classes. > - Add test case tagging. > - Get rid of avocado-framework-plugin-varianter-yaml-to-mux. > - Get rid of python-subprocess32. > - Improve logging. > Changes since v1: > - Merge start_vm rebuild patches. > - Fix patch comments. > > Anton Mikanovich (13): > start_vm.py: Fix ubuntu image name > vm_boot_test: Fix log file path in vm_boot_test > vm_boot_test: Remove external varianter > vm_boot_test: Improve QEMU images checking > build_test: Refactoring build tests cases > testsuite: Add Python generations for testsuite in gitignore > testsuite: Fix test suite prepare guide > gitlab-ci: Add Avocado build artifacts > gitlab-ci: Add debug flag > vm_boot_test: Add automatic bitbake init > build_test: Protect ubuntu target with KFAIL > ci_build: Migrate to Avocado > vm_smoke_test: Migrate to Avocado > > Roman Pletnev (3): > start_vm.py: Fix target name handling > start_vm.py: Add output and PID file vm_start.py options > start_vm.py: Add MIPS support > > .gitlab-ci.yml | 5 +- > scripts/ci_build.sh | 208 ++++--------------------------- > scripts/vm_smoke_test | 120 +++++------------- > testsuite/.gitignore | 1 + > testsuite/README.md | 21 +--- > testsuite/build_test/build_test.py | 219 ++++++++++++++++++++++++++++++--- > testsuite/build_test/cibase.py | 116 +++++++++++++++++ > testsuite/build_test/cibuilder.py | 132 ++++++++++++++++++++ > testsuite/build_test/run.sh | 3 - > testsuite/build_test/run_fast.sh | 3 + > testsuite/build_test/run_full.sh | 3 + > testsuite/build_test/variant.yaml | 22 ---- > testsuite/start_vm.py | 27 ++-- > testsuite/vm_boot_test/run.sh | 3 - > testsuite/vm_boot_test/run_fast.sh | 3 + > testsuite/vm_boot_test/run_full.sh | 3 + > testsuite/vm_boot_test/variant.yaml | 22 ---- > testsuite/vm_boot_test/vm_boot_test.py | 121 +++++++++++++++--- > 18 files changed, 641 insertions(+), 391 deletions(-) > create mode 100644 testsuite/.gitignore > create mode 100644 testsuite/build_test/cibase.py > create mode 100644 testsuite/build_test/cibuilder.py > delete mode 100755 testsuite/build_test/run.sh > create mode 100755 testsuite/build_test/run_fast.sh > create mode 100755 testsuite/build_test/run_full.sh > delete mode 100644 testsuite/build_test/variant.yaml > delete mode 100755 testsuite/vm_boot_test/run.sh > create mode 100755 testsuite/vm_boot_test/run_fast.sh > create mode 100755 testsuite/vm_boot_test/run_full.sh > delete mode 100644 testsuite/vm_boot_test/variant.yaml > Didn't you once installed avocado for execution in CI? This is no longer happening - unless I missed something - and, thus, broke gitlab-ci. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v14 00/16] Update Avocado testsuite 2021-10-14 8:54 ` Jan Kiszka @ 2021-10-14 10:25 ` Baurzhan Ismagulov 2021-10-14 15:51 ` Jan Kiszka 0 siblings, 1 reply; 9+ messages in thread From: Baurzhan Ismagulov @ 2021-10-14 10:25 UTC (permalink / raw) To: isar-users On Thu, Oct 14, 2021 at 10:54:37AM +0200, Jan Kiszka wrote: > Didn't you once installed avocado for execution in CI? This is no longer > happening - unless I missed something - and, thus, broke gitlab-ci. Yes, this happened in the v10 discussion: https://groups.google.com/g/isar-users/c/eSL1-OLaErE/m/6J3EfD9YCQAJ In a nutshell, the execution environment can have the control how the necessary tools are provided (Debian packages or pip), just as it is today with the other packages required by Isar. With kind regards, Baurzhan. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v14 00/16] Update Avocado testsuite 2021-10-14 10:25 ` Baurzhan Ismagulov @ 2021-10-14 15:51 ` Jan Kiszka 2021-10-19 8:34 ` Schmidt, Adriaan ` (2 more replies) 0 siblings, 3 replies; 9+ messages in thread From: Jan Kiszka @ 2021-10-14 15:51 UTC (permalink / raw) To: isar-users On 14.10.21 12:25, Baurzhan Ismagulov wrote: > On Thu, Oct 14, 2021 at 10:54:37AM +0200, Jan Kiszka wrote: >> Didn't you once installed avocado for execution in CI? This is no longer >> happening - unless I missed something - and, thus, broke gitlab-ci. > > Yes, this happened in the v10 discussion: > > https://groups.google.com/g/isar-users/c/eSL1-OLaErE/m/6J3EfD9YCQAJ > > In a nutshell, the execution environment can have the control how the necessary > tools are provided (Debian packages or pip), just as it is today with the other > packages required by Isar. You broke gitlab-ci, please fix. This can only reasonably be done in Isar, not outside (avokado is not build dependency, this is not part of kas-isar). Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: [PATCH v14 00/16] Update Avocado testsuite 2021-10-14 15:51 ` Jan Kiszka @ 2021-10-19 8:34 ` Schmidt, Adriaan 2021-10-22 6:54 ` Anton Mikanovich 2021-10-19 9:54 ` Henning Schild 2021-10-21 9:48 ` Baurzhan Ismagulov 2 siblings, 1 reply; 9+ messages in thread From: Schmidt, Adriaan @ 2021-10-19 8:34 UTC (permalink / raw) To: jan.kiszka, isar-users > -----Original Message----- > From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf Of > Jan Kiszka > Sent: Donnerstag, 14. Oktober 2021 17:51 > To: isar-users@googlegroups.com > Subject: Re: [PATCH v14 00/16] Update Avocado testsuite > > On 14.10.21 12:25, Baurzhan Ismagulov wrote: > > On Thu, Oct 14, 2021 at 10:54:37AM +0200, Jan Kiszka wrote: > >> Didn't you once installed avocado for execution in CI? This is no longer > >> happening - unless I missed something - and, thus, broke gitlab-ci. > > > > Yes, this happened in the v10 discussion: > > > > https://groups.google.com/g/isar-users/c/eSL1-OLaErE/m/6J3EfD9YCQAJ > > > > In a nutshell, the execution environment can have the control how the > necessary > > tools are provided (Debian packages or pip), just as it is today with the > other > > packages required by Isar. > > You broke gitlab-ci, please fix. This can only reasonably be done in > Isar, not outside (avokado is not build dependency, this is not part of > kas-isar). Even when installing avocado manually (pip3 install avocado-framework), the tests (or rather the collection of results?) fail in GitLab CI using the kas-isar:latest container: Traceback (most recent call last): File "/builder/.local/lib/python3.7/site-packages/avocado/core/test.py", line 825, in _run_avocado genio.write_file(whiteboard_file, self.whiteboard) File "/builder/.local/lib/python3.7/site-packages/avocado/utils/genio.py", line 117, in write_file with open(filename, 'w') as file_obj: FileNotFoundError: [Errno 2] No such file or directory: './build/job-results/job-2021-10-15T14.02-5ab2e09/test-results/1-_builds_cEV_-yzV_0_ebsy_debian_isar_testsuite_build_test_build_test.py_CrossTest.test_cross/whiteboard' ERROR 1-/builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross -> FileNotFoundError: [Errno 2] No such file or directory: './build/job-results/job-2021-10-15T14.02-5ab2e09/test-results/1-_builds_cEV_-yzV_0_ebsy_debian_isar_testsuite_build_test_build_test.py_CrossTest.test_cross/whiteboard' ERROR: [Errno 2] No such file or directory: './build/job-results/job-2021-10-15T14.02-5ab2e09/test-results/1-_builds_cEV_-yzV_0_ebsy_debian_isar_testsuite_build_test_build_test.py_CrossTest.test_cross/whiteboard' (5528.48 s) INIT 2-/builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross_ubuntu PARAMS (key=timeout, path=*, default=None) => None Test metadata: filename: /builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py teststmpdir: /var/tmp/avocado_haez6g83 (2/8) /builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross_ubuntu: START 2-/builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross_ubuntu DATA (filename=output.expected) => NOT FOUND (data sources: variant, test, file) Adriaan ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v14 00/16] Update Avocado testsuite 2021-10-19 8:34 ` Schmidt, Adriaan @ 2021-10-22 6:54 ` Anton Mikanovich 0 siblings, 0 replies; 9+ messages in thread From: Anton Mikanovich @ 2021-10-22 6:54 UTC (permalink / raw) To: Schmidt, Adriaan, jan.kiszka, isar-users On 19.10.21 11:34, Schmidt, Adriaan wrote: > Even when installing avocado manually (pip3 install avocado-framework), the tests (or rather the collection of results?) fail in GitLab CI using the kas-isar:latest container: > > Traceback (most recent call last): > File "/builder/.local/lib/python3.7/site-packages/avocado/core/test.py", line 825, in _run_avocado > genio.write_file(whiteboard_file, self.whiteboard) > File "/builder/.local/lib/python3.7/site-packages/avocado/utils/genio.py", line 117, in write_file > with open(filename, 'w') as file_obj: > FileNotFoundError: [Errno 2] No such file or directory: './build/job-results/job-2021-10-15T14.02-5ab2e09/test-results/1-_builds_cEV_-yzV_0_ebsy_debian_isar_testsuite_build_test_build_test.py_CrossTest.test_cross/whiteboard' > ERROR 1-/builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross -> FileNotFoundError: [Errno 2] No such file or directory: './build/job-results/job-2021-10-15T14.02-5ab2e09/test-results/1-_builds_cEV_-yzV_0_ebsy_debian_isar_testsuite_build_test_build_test.py_CrossTest.test_cross/whiteboard' > ERROR: [Errno 2] No such file or directory: './build/job-results/job-2021-10-15T14.02-5ab2e09/test-results/1-_builds_cEV_-yzV_0_ebsy_debian_isar_testsuite_build_test_build_test.py_CrossTest.test_cross/whiteboard' (5528.48 s) > INIT 2-/builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross_ubuntu > PARAMS (key=timeout, path=*, default=None) => None > Test metadata: > filename: /builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py > teststmpdir: /var/tmp/avocado_haez6g83 > (2/8) /builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross_ubuntu: START 2-/builds/cEV_-yzV/0/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross_ubuntu > DATA (filename=output.expected) => NOT FOUND (data sources: variant, test, file) > > Adriaan This was caused by logdir management changes between Avocado 88.0 and 89.0, so we couldn't use absolute paths in config file. Fix provided in `[PATCH] CI: Fix paths for the latest Avocado version` -- Anton Mikanovich Promwad Ltd. External service provider of ilbers GmbH Maria-Merian-Str. 8 85521 Ottobrunn, Germany +49 (89) 122 67 24-0 Commercial register Munich, HRB 214197 General Manager: Baurzhan Ismagulov ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v14 00/16] Update Avocado testsuite 2021-10-14 15:51 ` Jan Kiszka 2021-10-19 8:34 ` Schmidt, Adriaan @ 2021-10-19 9:54 ` Henning Schild 2021-10-21 9:48 ` Baurzhan Ismagulov 2 siblings, 0 replies; 9+ messages in thread From: Henning Schild @ 2021-10-19 9:54 UTC (permalink / raw) To: Jan Kiszka; +Cc: isar-users I am on that ... have patches practically ready. Did expect ilbers to break that because my remarks have been ignored earlier. My patch will use a virtualenv and install avocado conditionally if not found. Will send them anytime soon. Henning Am Thu, 14 Oct 2021 17:51:22 +0200 schrieb Jan Kiszka <jan.kiszka@siemens.com>: > On 14.10.21 12:25, Baurzhan Ismagulov wrote: > > On Thu, Oct 14, 2021 at 10:54:37AM +0200, Jan Kiszka wrote: > >> Didn't you once installed avocado for execution in CI? This is no > >> longer happening - unless I missed something - and, thus, broke > >> gitlab-ci. > > > > Yes, this happened in the v10 discussion: > > > > https://groups.google.com/g/isar-users/c/eSL1-OLaErE/m/6J3EfD9YCQAJ > > > > In a nutshell, the execution environment can have the control how > > the necessary tools are provided (Debian packages or pip), just as > > it is today with the other packages required by Isar. > > You broke gitlab-ci, please fix. This can only reasonably be done in > Isar, not outside (avokado is not build dependency, this is not part > of kas-isar). > > Jan > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v14 00/16] Update Avocado testsuite 2021-10-14 15:51 ` Jan Kiszka 2021-10-19 8:34 ` Schmidt, Adriaan 2021-10-19 9:54 ` Henning Schild @ 2021-10-21 9:48 ` Baurzhan Ismagulov 2 siblings, 0 replies; 9+ messages in thread From: Baurzhan Ismagulov @ 2021-10-21 9:48 UTC (permalink / raw) To: isar-users On Thu, Oct 14, 2021 at 05:51:22PM +0200, Jan Kiszka wrote: > > In a nutshell, the execution environment can have the control how the necessary > > tools are provided (Debian packages or pip), just as it is today with the other > > packages required by Isar. > > You broke gitlab-ci, please fix. This can only reasonably be done in > Isar, not outside (avokado is not build dependency, this is not part of > kas-isar). We can do this to unblock the situation. However, I think that drawing the line between build dependency and test tooling is a disservice for kas users. One of the goals for migrating to a test framework is to split the previously monolithic testsuite into individual testcases and the ability to run them manually. People like kas because it hides the environment preparation issues while also providing the necessary customization options. If a user spawns a kas shell to make his changes, he could use e.g. avocado run testsuite/build_test/build_test.py -t sdk to test only the area he is interested in. The same for debugging failing CI cases without having to run the full testsuite, which meanwhile takes several hours. That is why I think providing the test tooling in the default image is actually a good thing for the developers. With kind regards, Baurzhan. ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-10-22 6:55 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <1633020197-26007-1-git-send-email-anton.mikanovich@promwad.com>
2021-10-13 14:06 ` [PATCH v14 00/16] Update Avocado testsuite Anton Mikanovich
2021-09-30 16:45 Anton Mikanovich
2021-10-14 8:54 ` Jan Kiszka
2021-10-14 10:25 ` Baurzhan Ismagulov
2021-10-14 15:51 ` Jan Kiszka
2021-10-19 8:34 ` Schmidt, Adriaan
2021-10-22 6:54 ` Anton Mikanovich
2021-10-19 9:54 ` Henning Schild
2021-10-21 9:48 ` Baurzhan Ismagulov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox