From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6675671320760418304 X-Received: by 2002:a1c:e90f:: with SMTP id q15mr3012071wmc.2.1555403669636; Tue, 16 Apr 2019 01:34:29 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:1f86:: with SMTP id f128ls1885308wmf.1.canary-gmail; Tue, 16 Apr 2019 01:34:29 -0700 (PDT) X-Google-Smtp-Source: APXvYqy9WWkmk7qKqH9tm/BDUuADVHDBMQv/OqRuc455XXzaBP8FXchVdFZZq9G07UJ1VeoTatVh X-Received: by 2002:a1c:de0a:: with SMTP id v10mr2657285wmg.12.1555403669115; Tue, 16 Apr 2019 01:34:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555403669; cv=none; d=google.com; s=arc-20160816; b=BaFH5KwmzzvUyjXiWiU2WDPOnxqX+lrqC1yHZJ8wCR523vEh6POMrlvpbszdCKcgb4 2agJY/PK2RKwc/6mTXOCTpHIc/AtFGWIibkJffZQWaeQkpVuxYtUSRK6wNWEgucOSc/y I9ButqYWeCFlwALIbkj9DgraMIBaKg3Q+EVTrF9oagIHOtyCCRJyGFkU+kpzng+619eG 5L4daBEIcPHY1A9Km+lmL1LLXoeIaytabqfHoWqq11fa0gBRWMTm1HCADhHDxMsA3R9T tv74n0/oajuTAl7tHCj/RfGHUxmPONSXSmEWc7Ezc52fxTlBRfJIEEsIa9ETCABaVGyz swvg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date; bh=zIgQ/+aMpbkrNKyzEkeZNvACGqzYNwug+lDFOz/eMSY=; b=O5BD79cxCcldBQrSk/hhqGAT73lK491oXISbtY/UoQmZ+04m/7JSBqPF4hOHV6YCSX Q02yE3GlujfW/SPSdnN6dZ2UB7pIRBzBQMsUilD3TIwDPFPzuiu2C0XwFwR9NBSUs1d7 rA8lFIasUOsXIGRgs1+y+eWB2BL7gFxZAr/ZXlzFWJu4mQ/fv+3EDgOoM80LXUGMvmcn nOK6snPS6dMTAf82YhCpkCnlLb9IIFXT6bVYrvtVN0twNifP0arLt/9fcXmAOPpCHGxn 4WEimcNJL7lFHncuTuGXY4meuNFfZaMvrNpI1Wik9qufyrkLry4bCaWZ2OsMfJO85kRC 3XNQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id k15si2743788wrq.1.2019.04.16.01.34.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 01:34:29 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id x3G8YSVf012438 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 10:34:28 +0200 Received: from md1za8fc.ad001.siemens.net ([139.25.0.40]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id x3G8YSMZ027270; Tue, 16 Apr 2019 10:34:28 +0200 Date: Tue, 16 Apr 2019 10:34:27 +0200 From: Henning Schild To: "Maxim Yu. Osipov" Cc: Baurzhan Ismagulov , Subject: Re: [PATCH v2] testsuite: Initial Avocado test implementation Message-ID: <20190416103427.2ab226de@md1za8fc.ad001.siemens.net> In-Reply-To: <7079e446-fe4b-5d87-9e44-c3d4ad50ba57@ilbers.de> References: <20190403141407.17694-1-ibr@radix50.net> <7079e446-fe4b-5d87-9e44-c3d4ad50ba57@ilbers.de> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: +r/OpAp10v+S Am Fri, 12 Apr 2019 07:27:17 +0200 schrieb "Maxim Yu. Osipov" : > On 4/3/19 4:14 PM, Baurzhan Ismagulov wrote: > > From: Alexander Smirnov > > > > Currently, we have several test cases executed from shell scripts. > > As test coverage increases, adding test cases and reading the > > results becomes complex. Avocado testing framework provides the > > necessary infrastructure to organize test cases. This patch adds > > the initial Avocado test case support for future CI migration. > > Applied to the 'next'. I would suggest to remove that again and feed it back to reviewing. The content is hardly reviewed because the first round was lacking basic things like a commit message. > > > Signed-off-by: Alexander Smirnov > > Tested-by: Maxim Yu. Osipov > > --- > > testsuite/README.md | 34 +++++++++++++ > > testsuite/build_test/build_test.py | 29 +++++++++++ > > testsuite/build_test/run.sh | 3 ++ > > testsuite/build_test/variant.yaml | 22 +++++++++ > > testsuite/start_vm.py | 88 > > ++++++++++++++++++++++++++++++++++ > > testsuite/vm_boot_test/run.sh | 3 ++ > > testsuite/vm_boot_test/variant.yaml | 22 +++++++++ > > testsuite/vm_boot_test/vm_boot_test.py | 51 ++++++++++++++++++++ 8 > > files changed, 252 insertions(+) create mode 100644 > > testsuite/README.md create mode 100644 > > testsuite/build_test/build_test.py create mode 100755 > > testsuite/build_test/run.sh create mode 100644 > > testsuite/build_test/variant.yaml create mode 100755 > > testsuite/start_vm.py create mode 100755 > > testsuite/vm_boot_test/run.sh create mode 100644 > > testsuite/vm_boot_test/variant.yaml create mode 100644 > > testsuite/vm_boot_test/vm_boot_test.py > > > > diff --git a/testsuite/README.md b/testsuite/README.md > > new file mode 100644 > > index 0000000..5e64223 > > --- /dev/null > > +++ b/testsuite/README.md > > @@ -0,0 +1,34 @@ > > +# Install Avocado > > + > > +The framework could be installed by using standard HOWTO: > > + > > + > > https://github.com/avocado-framework/avocado#installing-with-standard-python-tools > > + +Then you need to install varianter yaml-to-mux plugin by > > following these instructions: + > > + > > https://github.com/avocado-framework/avocado/tree/master/optional_plugins > > + +## For Debian 9.x > > + > > + $ sudo apt-get install python-pip > > + $ pip install --user subprocess32 > > + $ pip install --user avocado-framework > > + $ pip install --user > > avocado-framework-plugin-varianter-yaml-to-mux + > > +# Pre > > + > > + $ export PATH=$PATH:~/.local/bin > > + $ cd isar > > + $ source isar-init-build-env > > + > > + > > +# Run test > > + > > +Each testsuite directory contains: > > + - run.sh - script to start tests > > + - variants.yaml - set of input data > > + - *.py - test case > > + > > +# Other > > + > > +There is a tool start_vm.py which is the replacement for the bash > > script in isar/scripts directory. diff --git > > a/testsuite/build_test/build_test.py > > b/testsuite/build_test/build_test.py new file mode 100644 index > > 0000000..4220d3a --- /dev/null > > +++ b/testsuite/build_test/build_test.py > > @@ -0,0 +1,29 @@ > > +#!/usr/bin/env python3 > > + > > +import os > > +import subprocess32 > > +import sys > > +from os.path import dirname > > + > > +from avocado import Test > > + > > +class BuildTest(Test): > > + > > + def test(self): > > + # TODO: add default values > > + build_dir = self.params.get('build_dir', default='.') > > + arch = self.params.get('arch', default='arm') > > + distro = self.params.get('distro', default='stretch') > > + > > + > > self.log.info('===================================================') > > + self.log.info('Running Isar build test for (' + distro + > > '-' + arch + ')') > > + self.log.info('Isar build folder is: ' + build_dir) > > + > > self.log.info('===================================================') > > + > > + #isar_root = dirname(__file__) + '/..' > > + os.chdir(build_dir) > > + cmdline = ['bitbake', 'multiconfig:qemu' + arch + '-' + > > distro + ':isar-image-base'] > > + p1 = subprocess32.run(cmdline) > > + > > + if p1.returncode: > > + self.fail('Test failed') > > diff --git a/testsuite/build_test/run.sh > > b/testsuite/build_test/run.sh new file mode 100755 > > index 0000000..a8ea9cc > > --- /dev/null > > +++ b/testsuite/build_test/run.sh > > @@ -0,0 +1,3 @@ > > +#!/bin/bash > > + > > +avocado run build_test.py --mux-yaml test:variant.yaml > > --mux-inject build_dir:$BUILDDIR diff --git > > a/testsuite/build_test/variant.yaml > > b/testsuite/build_test/variant.yaml new file mode 100644 index > > 0000000..9ddc634 --- /dev/null > > +++ b/testsuite/build_test/variant.yaml > > @@ -0,0 +1,22 @@ > > +variants: !mux > > + stretch-amd64: > > + distro: "stretch" > > + arch: "amd64" > > + stretch-i386: > > + distro: "stretch" > > + arch: "i386" > > + stretch-arm: > > + distro: "stretch" > > + arch: "arm" > > + stretch-arm64: > > + distro: "stretch" > > + arch: "arm64" > > + buster-amd64: > > + distro: "buster" > > + arch: "amd64" > > + buster-i386: > > + distro: "buster" > > + arch: "i386" > > + buster-arm: > > + distro: "buster" > > + arch: "arm" > > diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py > > new file mode 100755 > > index 0000000..02a4b51 > > --- /dev/null > > +++ b/testsuite/start_vm.py > > @@ -0,0 +1,88 @@ > > +#!/usr/bin/env python3 > > +# > > +# Helper script to start QEMU with Isar image > > +# Copyright (c) 2019, ilbers GmbH > > + > > +import argparse > > +import os > > +import subprocess > > +import sys > > +import time > > + > > +def get_bitbake_env(arch, distro): > > + multiconfig = 'multiconfig:qemu' + arch + '-' + distro + > > ':isar-image-base' > > + output = subprocess.check_output(['bitbake', '-e', > > str(multiconfig)]) > > + return output > > + > > +def get_bitbake_var(output, var): > > + ret = '' > > + for line in output.splitlines(): > > + if line.startswith(var): > > + ret = line.split('"')[1] > > + return ret > > + > > +def format_qemu_cmdline(arch, build, distro): > > + bb_output = get_bitbake_env(arch, distro).decode() > > + > > + rootfs_image = '' > > + extra_args = '' > > + cpu = [''] > > + > > + image_type = get_bitbake_var(bb_output, 'IMAGE_TYPE') > > + deploy_dir_image = get_bitbake_var(bb_output, > > 'DEPLOY_DIR_IMAGE') > > + if image_type == 'ext4-img': > > + rootfs_image = 'isar-image-base-debian-' + distro + > > '-qemu' + arch + '.ext4.img' > > + kernel_image = deploy_dir_image + '/' + > > get_bitbake_var(bb_output, 'KERNEL_IMAGE') > > + initrd_image = get_bitbake_var(bb_output, 'INITRD_IMAGE') > > + > > + if not initrd_image: > > + initrd_image = '/dev/null' > > + else: > > + initrd_image = deploy_dir_image + '/' + initrd_image > > + > > + serial = get_bitbake_var(bb_output, 'MACHINE_SERIAL') > > + root_dev = get_bitbake_var(bb_output, 'QEMU_ROOTFS_DEV') > > + kargs = ['-append', '"console=' + serial + ' root=/dev/' + > > root_dev + ' rw"'] + > > + extra_args = ['-kernel', kernel_image, '-initrd', > > initrd_image] > > + extra_args.extend(kargs) > > + elif image_type == 'wic-img': > > + rootfs_image = 'isar-image-base-debian-' + distro + > > '-qemu' + arch + '.wic.img' > > + extra_args = ['-snapshot'] > > + else: > > + raise ValueError('Invalid image type: ' + str(image_type)) > > + > > + qemu_arch = get_bitbake_var(bb_output, 'QEMU_ARCH') > > + qemu_machine = get_bitbake_var(bb_output, 'QEMU_MACHINE') > > + qemu_cpu = get_bitbake_var(bb_output, 'QEMU_CPU') > > + qemu_disk_args = get_bitbake_var(bb_output, 'QEMU_DISK_ARGS') > > + > > + qemu_disk_args = qemu_disk_args.replace('##ROOTFS_IMAGE##', > > deploy_dir_image + '/' + rootfs_image).split() + > > + cmd = ['qemu-system-' + qemu_arch, '-m', '1024M'] > > + > > + if qemu_machine: > > + cmd.extend(['-M', qemu_machine]) > > + > > + if qemu_cpu: > > + cmd.extend(['-cpu', qemu_cpu]) > > + > > + cmd.extend(extra_args) > > + cmd.extend(qemu_disk_args) > > + > > + return cmd > > + > > +def start_qemu(arch, build, distro): > > + cmdline = format_qemu_cmdline(arch, build, distro) > > + cmdline.insert(1, '-nographic') > > + > > + p1 = subprocess.call(cmdline) > > + > > +if __name__ == "__main__": > > + parser = argparse.ArgumentParser() > > + parser.add_argument('-a', '--arch', choices=['arm', 'arm64', > > 'amd64', 'i386'], help='set isar machine architecture.', > > default='arm') > > + parser.add_argument('-b', '--build', help='set path to build > > directory.', default=os.getcwd()) > > + parser.add_argument('-d', '--distro', choices=['jessie', > > 'stretch'], help='set isar Debian distribution.', default='stretch') When rewriting we should take the chance and call these guys "suites" not "distros". > > + args = parser.parse_args() > > + > > + start_qemu(args.arch, args.build, args.distro) > > diff --git a/testsuite/vm_boot_test/run.sh > > b/testsuite/vm_boot_test/run.sh new file mode 100755 > > index 0000000..9fdda95 > > --- /dev/null > > +++ b/testsuite/vm_boot_test/run.sh > > @@ -0,0 +1,3 @@ > > +#!/bin/bash > > + > > +avocado run vm_boot_test.py --mux-yaml test:variant.yaml > > --mux-inject build_dir:$BUILDDIR time_to_wait:300 diff --git > > a/testsuite/vm_boot_test/variant.yaml > > b/testsuite/vm_boot_test/variant.yaml new file mode 100644 index > > 0000000..9ddc634 --- /dev/null > > +++ b/testsuite/vm_boot_test/variant.yaml > > @@ -0,0 +1,22 @@ > > +variants: !mux > > + stretch-amd64: > > + distro: "stretch" > > + arch: "amd64" > > + stretch-i386: > > + distro: "stretch" > > + arch: "i386" > > + stretch-arm: > > + distro: "stretch" > > + arch: "arm" > > + stretch-arm64: > > + distro: "stretch" > > + arch: "arm64" > > + buster-amd64: > > + distro: "buster" > > + arch: "amd64" > > + buster-i386: > > + distro: "buster" > > + arch: "i386" > > + buster-arm: > > + distro: "buster" > > + arch: "arm" > > diff --git a/testsuite/vm_boot_test/vm_boot_test.py > > b/testsuite/vm_boot_test/vm_boot_test.py new file mode 100644 > > index 0000000..d4849c7 > > --- /dev/null > > +++ b/testsuite/vm_boot_test/vm_boot_test.py > > @@ -0,0 +1,51 @@ > > +#!/usr/bin/env python3 > > + > > +import os > > +import subprocess32 > > +import sys > > +import time > > + > > +from os.path import dirname > > +sys.path.append(dirname(__file__) + '/..') > > + > > +import start_vm > > + > > +from avocado import Test > > + > > +class VmBootTest(Test): > > + > > + def test(self): > > + # TODO: add default values > > + build_dir = self.params.get('build_dir', default='.') > > + arch = self.params.get('arch', default='arm') > > + distro = self.params.get('distro', default='stretch') > > + time_to_wait = self.params.get('time_to_wait', default=60) > > + > > + > > self.log.info('===================================================') > > + self.log.info('Running Isar VM boot test for (' + distro + > > '-' + arch + ')') > > + self.log.info('Isar build folder is: ' + build_dir) > > + > > self.log.info('===================================================') > > + > > + output_file = '/tmp/vm_boot_test.log' > > + if os.path.exists(output_file): > > + os.remove(output_file) Do these tests run in parrallel? If so we need a file for every run. Even if not "tempfile" might be the way to go. Henning > > + > > + cmdline = start_vm.format_qemu_cmdline(arch, build_dir, > > distro) > > + cmdline.insert(1, '-nographic') > > + cmdline.append('-serial') > > + cmdline.append('file:' + output_file) > > + > > + self.log.info('QEMU boot line: ' + str(cmdline)) > > + > > + devnull = open(os.devnull, 'w') > > + > > + p1 = subprocess32.Popen(cmdline, stdout=devnull, > > stderr=devnull) > > + time.sleep(int(time_to_wait)) > > + p1.kill() > > + p1.wait() > > + > > + if os.path.exists(output_file): > > + if 'isar login:' in open(output_file).read(): > > + return > > + > > + self.fail('Test failed') > > > >