From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6953642757724307456 X-Received: by 2002:a05:6512:3888:: with SMTP id n8mr13318967lft.407.1620549931691; Sun, 09 May 2021 01:45:31 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:230f:: with SMTP id o15ls7915675lfu.1.gmail; Sun, 09 May 2021 01:45:30 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxjSCwKQVmV/QCulDD5dJ+3O8s+F9aaYixNETgudSzXXpNANO+SBNvtuh+AxEem7AhOYG1t X-Received: by 2002:ac2:4571:: with SMTP id k17mr13162552lfm.190.1620549930641; Sun, 09 May 2021 01:45:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1620549930; cv=none; d=google.com; s=arc-20160816; b=EEnuxJcTTyLKCXRlNdEyJDFQd+dFaT3G771hpkZtpxxWUda0p4FHoIcszag4tTnD6I HhM+1gut7yczY5U+3g2xVz9vmafiHaS8sjr7UVfnHUpkVqkOVwKkx3C4DdzFl+8/G5w6 wo8oZWSq17AnU+ou6m6Fb/vpY5AwN4vGOCfEjIa2DLx7jdpdQ5ewPuir/yCph0pqnnNv yWtMtUXOzjgS3I0H4AlPNWB+FhGPDnEZsxP20y2+0+hx5Vnuqy9gQjU9YGqh4ZUI4KsW dFLLSh4EIUHb69v95w3IZLFk73Z3/8TBYJTbS2KlF5ju4lZtQ45aO99Q7yd/6AJtcV14 iFZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject; bh=U1f3uJhE+dUL6aHEqGUipl0l4728NMGfNFmLGu1BBlc=; b=fLnmbsQvLMMcHd8G0s1EUiIzA+BfpfyWe9JPeqUu1iZQ+yuLSjCalUnn6aGZnwlg1n wrTAGNebbAJM6Vf5CzFFLpxgGz5CV9v4YS5B9Y3ndnJRmT3gYexFkllLu88o8pidFlA+ 0H0yvuPL57ycUucUs28noxTI42JUn6UWpFQwlcz8Hk0sgWye5tyY/MQZ4fKHFheuoBI0 pyXAbr1DOheXK/+FecjkRWTNiRp+MQCxjEvwiCFxxMWlL23iPe6jXkCXwWIUS1afcEOf 5gjvWcPDLWKzJOR0Ex+jEF53V14gnwf8dRmhzvwr5u7MMmOZzGUSx0xvS6VSTrZH4sjN V4kQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id q6si464266lfc.13.2021.05.09.01.45.30 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 09 May 2021 01:45:30 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id 1498jT27019449 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 9 May 2021 10:45:30 +0200 Received: from [167.87.1.6] ([167.87.1.6]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 1498cA6d012866; Sun, 9 May 2021 10:38:10 +0200 Subject: Re: [PATCH v6 00/15] Update Avocado testsuite To: Anton Mikanovich , isar-users@googlegroups.com References: <20210509064224.96072-1-amikan@ilbers.de> From: Jan Kiszka Message-ID: Date: Sun, 9 May 2021 10:38:09 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.0 MIME-Version: 1.0 In-Reply-To: <20210509064224.96072-1-amikan@ilbers.de> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: ZeQK2kI6QrXt On 09.05.21 08:42, Anton Mikanovich wrote: > This patchset provides implementation of build and vm_start tests > functionality in Python with help of Avocado framework. > Scripts ci_build and vm_smoke_test are rewritten to call Avocado. > ci_build and vm_smoke_test are left for compatibility and > should be replaced by direct avocado calls later. > Original ci_build and vm_smoke_test are renamed to *_legacy. > Build and run logs are hidden now until an error occurs. > > --- > Changes since v5: > - Rebase on current next. > Changes since v4: > - Add container tests. > - Implement testcase skipping logic. > - Separate and protect Bullseye tests. > - Add checks for container dependencies. > - Add Ilbers repo key. > - Improve error handling. > - Fix Ubuntu qemu image name. > Changes since v3: > - Add automatic bitbake init. > - Fix line lengths. > - Fix build without repro. > - Fix build artifact paths. > - Prepare for Gitlab build. > - Change debug settings. > - Move log splitting to start_vm. > - Add deb package install. > Changes since v2: > - Make CI call Avocado tests. > - Wait less on vm_start test. > - Split Roman's vm_start commit. > - Put logs to build folder. > - Rearrange patches. > - Split build test to classes. > - Add test case tagging. > - Get rid of avocado-framework-plugin-varianter-yaml-to-mux. > - Get rid of python-subprocess32. > - Improve logging. > Changes since v1: > - Merge start_vm rebuild patches. > - Fix patch comments. > > Anton Mikanovich (12): > start_vm.py: Fix ubuntu image name > vm_boot_test: Fix log file path in vm_boot_test > vm_boot_test: Remove external varianter > vm_boot_test: Improve QEMU images checking > build_test: Refactoring build tests cases > testsuite: Add Python generations for testsuite in gitignore > testsuite: Fix test suite prepare guide > gitlab-ci: Add Avocado build artifacts > vm_boot_test: Add automatic bitbake init > ci_build: Migrate to Avocado > vm_smoke_test: Migrate to Avocado > ci_build: Install Avocado if needed > > Roman Pletnev (3): > start_vm.py: Fix target name handling > start_vm.py: Add output and PID file vm_start.py options > start_vm.py: Add MIPS support > > .gitlab-ci.yml | 1 + > scripts/ci_build.sh | 212 +++++--------------- > scripts/ci_build_legacy.sh | 267 +++++++++++++++++++++++++ > scripts/vm_smoke_test | 110 ++-------- > scripts/vm_smoke_test_legacy | 168 ++++++++++++++++ > testsuite/.gitignore | 1 + > testsuite/README.md | 21 +- > testsuite/build_test/build_test.py | 208 +++++++++++++++++-- > testsuite/build_test/cibase.py | 145 ++++++++++++++ > testsuite/build_test/cibuilder.py | 90 +++++++++ > testsuite/build_test/run.sh | 3 - > testsuite/build_test/run_fast.sh | 3 + > testsuite/build_test/run_full.sh | 3 + > testsuite/build_test/variant.yaml | 22 -- > testsuite/start_vm.py | 27 ++- > testsuite/vm_boot_test/run.sh | 3 - > testsuite/vm_boot_test/run_fast.sh | 3 + > testsuite/vm_boot_test/run_full.sh | 3 + > testsuite/vm_boot_test/variant.yaml | 22 -- > testsuite/vm_boot_test/vm_boot_test.py | 120 +++++++++-- > 20 files changed, 1057 insertions(+), 375 deletions(-) > create mode 100755 scripts/ci_build_legacy.sh > create mode 100755 scripts/vm_smoke_test_legacy > create mode 100644 testsuite/.gitignore > create mode 100644 testsuite/build_test/cibase.py > create mode 100644 testsuite/build_test/cibuilder.py > delete mode 100755 testsuite/build_test/run.sh > create mode 100755 testsuite/build_test/run_fast.sh > create mode 100755 testsuite/build_test/run_full.sh > delete mode 100644 testsuite/build_test/variant.yaml > delete mode 100755 testsuite/vm_boot_test/run.sh > create mode 100755 testsuite/vm_boot_test/run_fast.sh > create mode 100755 testsuite/vm_boot_test/run_full.sh > delete mode 100644 testsuite/vm_boot_test/variant.yaml > Is that equivalent to 1c1c1dd00dcb8a9c312faa25d22da0a7ddb0238d (amikan/devel)? Tried that revision, but this is what our CI said when running fast-ci: + avocado run /builds/ebsy/debian/isar/testsuite/build_test/build_test.py -t fast,-repro -p build_dir=./build -p quiet=1 -p cross=1 JOB ID : 6725f043528240b81c41c00342477799b6a592f6 JOB LOG : /builds/ebsy/debian/isar/build/job-results/job-2021-05-09T08.28-6725f04/job.log (1/7) /builds/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross: ERROR: dictionary update sequence element #48 has length 1; 2 is required (0.07 s) (2/7) /builds/ebsy/debian/isar/testsuite/build_test/build_test.py:CrossTest.test_cross_bullseye: CANCEL: KFAIL (0.05 s) (3/7) /builds/ebsy/debian/isar/testsuite/build_test/build_test.py:SdkTest.test_sdk: ERROR: dictionary update sequence element #48 has length 1; 2 is required (0.07 s) (4/7) /builds/ebsy/debian/isar/testsuite/build_test/build_test.py:RebuildTest.test_rebuild: ERROR: dictionary update sequence element #48 has length 1; 2 is required (1.63 s) (5/7) /builds/ebsy/debian/isar/testsuite/build_test/build_test.py:WicTest.test_wic_exclude: ERROR: dictionary update sequence element #48 has length 1; 2 is required (0.07 s) (6/7) /builds/ebsy/debian/isar/testsuite/build_test/build_test.py:ContainerImageTest.test_nocross: ERROR: dictionary update sequence element #48 has length 1; 2 is required (0.07 s) (7/7) /builds/ebsy/debian/isar/testsuite/build_test/build_test.py:ContainerSdkTest.test_container_sdk: ERROR: dictionary update sequence element #48 has length 1; 2 is required (0.07 s) RESULTS : PASS 0 | ERROR 6 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 1 JOB TIME : 2.71 s I can provide the full log if needed. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux