From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6969105759827132416 X-Received: by 2002:a5d:528d:: with SMTP id c13mr26868786wrv.343.1624258173481; Sun, 20 Jun 2021 23:49:33 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:1d0e:: with SMTP id l14ls8436932wms.1.canary-gmail; Sun, 20 Jun 2021 23:49:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz+5V2jZHr2z5x6iCgbJRcoTl3Y+suG7+nOdXF44hwdNdP7xSkUrcwuBd/429s8d1MKZQgK X-Received: by 2002:a7b:ce16:: with SMTP id m22mr4193327wmc.156.1624258172604; Sun, 20 Jun 2021 23:49:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1624258172; cv=none; d=google.com; s=arc-20160816; b=DG7B6mCjuPJJyjm2vJ2d4nPHf7YNim6IJ3y8e5n1V1EliBDtU23cDbEQLBM0DlP2IZ JEluVoSE6K9xFwZdt5tMUI+jG2wncNzIaGkj6r+SLoguHwAyKO79g2pPqxpiGCRk/kqC g9gzeUUDkSP/HOkqkc6XwOZuF6M7JOB9kYUeiYlY35Le8v63si+ZgTxjp6YPelDumuf/ MO46oVDSXGdG/wkqxkxC7kFllBzrOQa+p20UDWy0Xyv4tnf+xw99gwBmh2Ko7oeKlflt di+88jDdLLjwfEDDnjb8EeNGgRZN2n13iKLcWaBI8ouREvjFFaaUFKBjEQf61p+DGR+D 80Dg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject; bh=txPqfzPdSD2uQV4cfW/gzZDEnS0rDCyIRF0EO+nj+So=; b=WNjhLnXMV95cIuzowv2CJJ2eb9n7a5bftW6nmvu3VCEzsNtXa49Br02RLZz6V7+hd2 m1b65/fqApPyadYi4rGcVpzoZwCrcojA6V/8a2KcrW4WTbhLIJrac1y020mJzzIslroE FH0iUBFFpTnRP4ZPyczdjbdUCawPnHZspQfotsYrElVLFsadVgQofecTNwse6nQp3e6O Y4DMnrv4SqTI9c+v6a8SNNKNirNoLUdTNVvbrf8/1745x3mDFvS6u+IdWfLbaG4YbYS8 Q5p9Ysni3KvPYZhCQGDcMnBreRTF0UzfQkGvyUtn02a/ST79M8EVeidmMIqrhKwD/JZt nJgg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id o17si719732wrp.4.2021.06.20.23.49.32 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 20 Jun 2021 23:49:32 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id 15L6nVWW007098 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 21 Jun 2021 08:49:32 +0200 Received: from [167.87.243.122] ([167.87.243.122]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 15L6nVdI009778; Mon, 21 Jun 2021 08:49:31 +0200 Subject: Re: [PATCH v8 00/17] Update Avocado testsuite To: Anton Mikanovich , isar-users@googlegroups.com References: <20210602080922.16435-1-amikan@ilbers.de> From: Jan Kiszka Message-ID: Date: Mon, 21 Jun 2021 08:49:31 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.0 MIME-Version: 1.0 In-Reply-To: <20210602080922.16435-1-amikan@ilbers.de> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: 3Ix3QimRSX0Y On 02.06.21 10:09, Anton Mikanovich wrote: > This patchset provides implementation of build and vm_start tests > functionality in Python with help of Avocado framework. > Scripts ci_build and vm_smoke_test are rewritten to call Avocado. > ci_build and vm_smoke_test are left for compatibility and > should be replaced by direct avocado calls later. > Original ci_build and vm_smoke_test are renamed to *_legacy. > Build and run logs are hidden now until an error occurs. > > --- > Changes since v7: > - Remove code duplication. > - Fix build_dir parsing. > - Add missed protect files backup logic. > Changes since v6: > - Improve environment parsing. > - Protect files backup logic. > - Rebuild full log output option. > - Protect ubuntu target with KFAIL. > Changes since v5: > - Rebase on current next. > Changes since v4: > - Add container tests. > - Implement testcase skipping logic. > - Separate and protect Bullseye tests. > - Add checks for container dependencies. > - Add Ilbers repo key. > - Improve error handling. > - Fix Ubuntu qemu image name. > Changes since v3: > - Add automatic bitbake init. > - Fix line lengths. > - Fix build without repro. > - Fix build artifact paths. > - Prepare for Gitlab build. > - Change debug settings. > - Move log splitting to start_vm. > - Add deb package install. > Changes since v2: > - Make CI call Avocado tests. > - Wait less on vm_start test. > - Split Roman's vm_start commit. > - Put logs to build folder. > - Rearrange patches. > - Split build test to classes. > - Add test case tagging. > - Get rid of avocado-framework-plugin-varianter-yaml-to-mux. > - Get rid of python-subprocess32. > - Improve logging. > Changes since v1: > - Merge start_vm rebuild patches. > - Fix patch comments. > > Anton Mikanovich (14): > start_vm.py: Fix ubuntu image name > vm_boot_test: Fix log file path in vm_boot_test > vm_boot_test: Remove external varianter > vm_boot_test: Improve QEMU images checking > build_test: Refactoring build tests cases > testsuite: Add Python generations for testsuite in gitignore > testsuite: Fix test suite prepare guide > gitlab-ci: Add Avocado build artifacts > gitlab-ci: Add debug flag > vm_boot_test: Add automatic bitbake init > build_test: Protect ubuntu target with KFAIL > ci_build: Migrate to Avocado > vm_smoke_test: Migrate to Avocado > ci_build: Install Avocado if needed > > Roman Pletnev (3): > start_vm.py: Fix target name handling > start_vm.py: Add output and PID file vm_start.py options > start_vm.py: Add MIPS support > > .gitlab-ci.yml | 5 +- > scripts/ci_build.sh | 213 ++++---------------- > scripts/ci_build_legacy.sh | 267 +++++++++++++++++++++++++ > scripts/vm_smoke_test | 110 ++-------- > scripts/vm_smoke_test_legacy | 168 ++++++++++++++++ > testsuite/.gitignore | 1 + > testsuite/README.md | 21 +- > testsuite/build_test/build_test.py | 217 ++++++++++++++++++-- > testsuite/build_test/cibase.py | 110 ++++++++++ > testsuite/build_test/cibuilder.py | 103 ++++++++++ > testsuite/build_test/run.sh | 3 - > testsuite/build_test/run_fast.sh | 3 + > testsuite/build_test/run_full.sh | 3 + > testsuite/build_test/variant.yaml | 22 -- > testsuite/start_vm.py | 27 ++- > testsuite/vm_boot_test/run.sh | 3 - > testsuite/vm_boot_test/run_fast.sh | 3 + > testsuite/vm_boot_test/run_full.sh | 3 + > testsuite/vm_boot_test/variant.yaml | 22 -- > testsuite/vm_boot_test/vm_boot_test.py | 121 +++++++++-- > 20 files changed, 1043 insertions(+), 382 deletions(-) > create mode 100755 scripts/ci_build_legacy.sh > create mode 100755 scripts/vm_smoke_test_legacy > create mode 100644 testsuite/.gitignore > create mode 100644 testsuite/build_test/cibase.py > create mode 100644 testsuite/build_test/cibuilder.py > delete mode 100755 testsuite/build_test/run.sh > create mode 100755 testsuite/build_test/run_fast.sh > create mode 100755 testsuite/build_test/run_full.sh > delete mode 100644 testsuite/build_test/variant.yaml > delete mode 100755 testsuite/vm_boot_test/run.sh > create mode 100755 testsuite/vm_boot_test/run_fast.sh > create mode 100755 testsuite/vm_boot_test/run_full.sh > delete mode 100644 testsuite/vm_boot_test/variant.yaml > I tried that revision in our CI, but all attempts are failing, apparently due to connectivity issues. That would be something to blame on our infrastructure - if current next wouldn't pass... Retrying once again. Then the log output is not yet optimally tuned. Avocado seems to spit out lengthy unformatted error reports of the kind TestFail: b'ERROR: mc:container-amd64-bullseye:isar-image-base-1.0-r0 do_rootfs_postprocess: Error executing a python function in exec_python_func() autogenerated:\n\nThe stack trace of python calls that resulted in this exception/failure was:\nFile: \'exec_python_func() autogenerated\', lineno: 2, function: \n 0001:\n *** 0002:do_rootfs_postprocess(d)\n 0003:\nFile: \'/builds/Vs1BETDQ/0/ebsy/debian/isar/meta/classes/rootfs.bbclass\', lineno: 248, function: do_rootfs_postprocess\n 0244: if cmds is None or not cmds.strip():\n 0245: return\n 0246: cmds = cmds.split()\n 0247: for i, cmd in enumerate(cmds):\n *** 0248: bb.build.exec_func(cmd, d)\n 0249: progress_reporter.update(int(i / len(cmds) * 100))\n 0250:}\n 0251:addtask rootfs_postprocess before do_rootfs\n 0252:\nFile: \'/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/build.py\', lineno: 251, function: exec_func\n 0247: with bb.utils.fileslocked(lockfiles):\n 0248: if ispython:\n 0249: exec_func_python(func, d, runfile, cwd=adir)\n 0250: else:\n *** 0251: exec_func_shell(func, d, runfile, cwd=adir)\n 0252:\n 0253: try:\n 0254: curcwd = os.getcwd()\n 0255: except:\nFile: \'/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/build.py\', lineno: 452, function: exec_func_shell\n 0448: with open(fifopath, \'r+b\', buffering=0) as fifo:\n 0449: try:\n 0450: bb.debug(2, "Executing shell function %s" % func)\n 0451: with open(os.devnull, \'r+\') as stdin, logfile:\n *** 0452: bb.process.run(cmd, shell=False, stdin=stdin, log=logfile, extrafiles=[(fifo,readfifo)])\n 0453: finally:\n 0454: os.unlink(fifopath)\n 0455:\n 0456: bb.debug(2, "Shell function %s finished" % func)\nFile: \'/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/process.py\', lineno: 182, function: run\n 0178: if not stderr is None:\n 0179: stderr = stderr.decode("utf-8")\n 0180:\n 0181: if pipe.returncode != 0:\n *** 0182: raise ExecutionError(cmd, pipe.returncode, stdout, stderr)\n 0183: return stdout, stderr\nException: bb.process.ExecutionError: Execution of \'/builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/debian-bullseye-amd64/isar-image-base-container-amd64-container-img/1.0-r0/temp/run.cache_deb_src.5240\' failed with exit code 100:\nReading package lists...\nNeed to get 1839 kB of source archives.\nGet:1 http://deb.debian.org/debian bullseye/main fuse 2.9.9-5 (dsc) [2312 B]\nGet:2 htt Those a likely duplicates of what the verbose log reporting already listed, and that was formatted in readable manner. These duplicate dumps are very unhandy and bloat the log for no value. There is also some warning-like error about journalctl during startup: + avocado --show=test run /builds/Vs1BETDQ/0/ebsy/debian/isar/testsuite/build_test/build_test.py -t fast,-repro -p build_dir=./build -p quiet=1 -p cross=1 Commands configured by file: /usr/lib/python3.7/dist-packages/avocado/etc/avocado/sysinfo/commands Files configured by file: /usr/lib/python3.7/dist-packages/avocado/etc/avocado/sysinfo/files File /usr/lib/python3.7/dist-packages/avocado/etc/avocado/sysinfo/fail_commands does not exist. File /usr/lib/python3.7/dist-packages/avocado/etc/avocado/sysinfo/fail_files does not exist. Profilers configured by file: /usr/lib/python3.7/dist-packages/avocado/etc/avocado/sysinfo/profilers Profiler disabled Journalctl collection failed: [Errno 2] No such file or directory: 'journalctl' (journalctl --quiet --lines 1 --output json): 'journalctl' Not logging /sys/kernel/debug/sched_features (file does not exist) Not logging /proc/pci (file does not exist) Not logging /proc/slabinfo (lack of permissions) Command line: /usr/bin/avocado --show=test run /builds/Vs1BETDQ/0/ebsy/debian/isar/testsuite/build_test/build_test.py -t fast,-repro -p build_dir=./build -p quiet=1 -p cross=1 Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux