From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6675671320760418304 X-Received: by 2002:a50:a48a:: with SMTP id w10mr5500476edb.3.1554301720336; Wed, 03 Apr 2019 07:28:40 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:906:3e4b:: with SMTP id t11ls391235eji.5.gmail; Wed, 03 Apr 2019 07:28:39 -0700 (PDT) X-Google-Smtp-Source: APXvYqwLyoAVeE9xx5vAGsIqPbZ6W1hpzCTVCsigvBNWGX+iadE26yIxtAGGyCIHVS2NSAavVFiD X-Received: by 2002:a17:906:4757:: with SMTP id j23mr420244ejs.15.1554301719821; Wed, 03 Apr 2019 07:28:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554301719; cv=none; d=google.com; s=arc-20160816; b=okVsTt/H8tpobf16ogwY9/vWSTXJeTIjSJrH3t8zyfTIOTGBShoOF/LlsGrAWs61Cl 9URzjmjZMKH1wZeWNemL/clVSIsPDd/OS0RC66ilPZqnRKecrz4QkQNTGZSWdSOZO0q9 MIk0JPY1L4KQYOPyLUhrTyMGFEKN+MVvlokgNpudNbQAJcuZQxot9TnmO4YjEmJk5XQL vFLQFrv4pQFtrKyLVJ3RJKfuaKEdTJKUqsmpAOzRv6dsS9brwNEy6OgKLoAN6icOYlbv ewBXvQyNh1pLlCh0/2pLolB25OCoXZ+YIv4vP3yfh1aHGbO5hUDy/4erT0XS5faR7/S9 pDdg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject; bh=yV8O7uNfWI2RMVjIPGuxhJ1VewSJ1cUyju0+vVzKvOk=; b=e7PXBxLElS8U6biaeqrVJmfcmm+L2UTdNApckwTBtVYBf3gySIE4v8LGjkqfWRHHnz CwYo/SmpFtXqTwdcLTyKRQpntcZGn+Hy3D5Wt7dDo+YBu3AlkOuyj1/l2YiDJ6Q40lm8 NGo1KR2gu20QVhti/XHjS10puexIfeni6I6QfHe7aSmOI646ZxTHUSu4IlvTT+cPUmkC iJfWE90BH3uvo1GVT3UHLbOxDvOYaNfuxWJDIInpyWGxj8UY8u2Er5eDPRLbMPK3lPCO CHg69YiE2rg1Z0XfE3+M4QqvoH20k7JGm+8lbunCwY03N3IYu3wUnBApH6O0ySiowDL9 7v0w== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id b12si439827ede.3.2019.04.03.07.28.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2019 07:28:39 -0700 (PDT) Received-SPF: pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id x33ESdRl028963 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 3 Apr 2019 16:28:39 +0200 Received: from [139.25.69.232] (linux-ses-ext02.ppmd.siemens.net [139.25.69.232]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id x33ESdFi031813; Wed, 3 Apr 2019 16:28:39 +0200 Subject: Re: [PATCH v2] testsuite: Initial Avocado test implementation To: Baurzhan Ismagulov , isar-users@googlegroups.com References: <20190403141407.17694-1-ibr@radix50.net> From: Claudius Heine Message-ID: Date: Wed, 3 Apr 2019 16:28:38 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.5.1 MIME-Version: 1.0 In-Reply-To: <20190403141407.17694-1-ibr@radix50.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: LwFI0b09UmPx Hi, On 03/04/2019 16.14, Baurzhan Ismagulov wrote: > From: Alexander Smirnov > > Currently, we have several test cases executed from shell scripts. As test > coverage increases, adding test cases and reading the results becomes complex. > Avocado testing framework provides the necessary infrastructure to organize > test cases. This patch adds the initial Avocado test case support for future CI > migration. > > Signed-off-by: Alexander Smirnov > Tested-by: Maxim Yu. Osipov > --- > testsuite/README.md | 34 +++++++++++++ > testsuite/build_test/build_test.py | 29 +++++++++++ > testsuite/build_test/run.sh | 3 ++ > testsuite/build_test/variant.yaml | 22 +++++++++ > testsuite/start_vm.py | 88 ++++++++++++++++++++++++++++++++++ > testsuite/vm_boot_test/run.sh | 3 ++ > testsuite/vm_boot_test/variant.yaml | 22 +++++++++ > testsuite/vm_boot_test/vm_boot_test.py | 51 ++++++++++++++++++++ > 8 files changed, 252 insertions(+) > create mode 100644 testsuite/README.md > create mode 100644 testsuite/build_test/build_test.py > create mode 100755 testsuite/build_test/run.sh > create mode 100644 testsuite/build_test/variant.yaml > create mode 100755 testsuite/start_vm.py > create mode 100755 testsuite/vm_boot_test/run.sh > create mode 100644 testsuite/vm_boot_test/variant.yaml > create mode 100644 testsuite/vm_boot_test/vm_boot_test.py > > diff --git a/testsuite/README.md b/testsuite/README.md > new file mode 100644 > index 0000000..5e64223 > --- /dev/null > +++ b/testsuite/README.md > @@ -0,0 +1,34 @@ > +# Install Avocado > + > +The framework could be installed by using standard HOWTO: > + > + https://github.com/avocado-framework/avocado#installing-with-standard-python-tools > + > +Then you need to install varianter yaml-to-mux plugin by following these instructions: > + > + https://github.com/avocado-framework/avocado/tree/master/optional_plugins > + > +## For Debian 9.x > + > + $ sudo apt-get install python-pip > + $ pip install --user subprocess32 > + $ pip install --user avocado-framework > + $ pip install --user avocado-framework-plugin-varianter-yaml-to-mux > + > +# Pre > + > + $ export PATH=$PATH:~/.local/bin > + $ cd isar > + $ source isar-init-build-env > + > + > +# Run test > + > +Each testsuite directory contains: > + - run.sh - script to start tests > + - variants.yaml - set of input data > + - *.py - test case > + > +# Other > + > +There is a tool start_vm.py which is the replacement for the bash script in isar/scripts directory. > diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py > new file mode 100644 > index 0000000..4220d3a > --- /dev/null > +++ b/testsuite/build_test/build_test.py > @@ -0,0 +1,29 @@ > +#!/usr/bin/env python3 > + > +import os > +import subprocess32 > +import sys > +from os.path import dirname > + > +from avocado import Test > + > +class BuildTest(Test): > + > + def test(self): > + # TODO: add default values > + build_dir = self.params.get('build_dir', default='.') > + arch = self.params.get('arch', default='arm') > + distro = self.params.get('distro', default='stretch') > + > + self.log.info('===================================================') > + self.log.info('Running Isar build test for (' + distro + '-' + arch + ')') > + self.log.info('Isar build folder is: ' + build_dir) > + self.log.info('===================================================') > + > + #isar_root = dirname(__file__) + '/..' > + os.chdir(build_dir) > + cmdline = ['bitbake', 'multiconfig:qemu' + arch + '-' + distro + ':isar-image-base'] > + p1 = subprocess32.run(cmdline) > + > + if p1.returncode: > + self.fail('Test failed') > diff --git a/testsuite/build_test/run.sh b/testsuite/build_test/run.sh > new file mode 100755 > index 0000000..a8ea9cc > --- /dev/null > +++ b/testsuite/build_test/run.sh > @@ -0,0 +1,3 @@ > +#!/bin/bash > + > +avocado run build_test.py --mux-yaml test:variant.yaml --mux-inject build_dir:$BUILDDIR > diff --git a/testsuite/build_test/variant.yaml b/testsuite/build_test/variant.yaml > new file mode 100644 > index 0000000..9ddc634 > --- /dev/null > +++ b/testsuite/build_test/variant.yaml > @@ -0,0 +1,22 @@ > +variants: !mux > + stretch-amd64: > + distro: "stretch" > + arch: "amd64" > + stretch-i386: > + distro: "stretch" > + arch: "i386" > + stretch-arm: > + distro: "stretch" > + arch: "arm" > + stretch-arm64: > + distro: "stretch" > + arch: "arm64" > + buster-amd64: > + distro: "buster" > + arch: "amd64" > + buster-i386: > + distro: "buster" > + arch: "i386" > + buster-arm: > + distro: "buster" > + arch: "arm" What about the non-qemu variants? At least a build test would be nice. Are you planning deploying and booting Isar with real HW in the CI as well? Claudius > diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py > new file mode 100755 > index 0000000..02a4b51 > --- /dev/null > +++ b/testsuite/start_vm.py > @@ -0,0 +1,88 @@ > +#!/usr/bin/env python3 > +# > +# Helper script to start QEMU with Isar image > +# Copyright (c) 2019, ilbers GmbH > + > +import argparse > +import os > +import subprocess > +import sys > +import time > + > +def get_bitbake_env(arch, distro): > + multiconfig = 'multiconfig:qemu' + arch + '-' + distro + ':isar-image-base' > + output = subprocess.check_output(['bitbake', '-e', str(multiconfig)]) > + return output > + > +def get_bitbake_var(output, var): > + ret = '' > + for line in output.splitlines(): > + if line.startswith(var): > + ret = line.split('"')[1] > + return ret > + > +def format_qemu_cmdline(arch, build, distro): > + bb_output = get_bitbake_env(arch, distro).decode() > + > + rootfs_image = '' > + extra_args = '' > + cpu = [''] > + > + image_type = get_bitbake_var(bb_output, 'IMAGE_TYPE') > + deploy_dir_image = get_bitbake_var(bb_output, 'DEPLOY_DIR_IMAGE') > + if image_type == 'ext4-img': > + rootfs_image = 'isar-image-base-debian-' + distro + '-qemu' + arch + '.ext4.img' > + kernel_image = deploy_dir_image + '/' + get_bitbake_var(bb_output, 'KERNEL_IMAGE') > + initrd_image = get_bitbake_var(bb_output, 'INITRD_IMAGE') > + > + if not initrd_image: > + initrd_image = '/dev/null' > + else: > + initrd_image = deploy_dir_image + '/' + initrd_image > + > + serial = get_bitbake_var(bb_output, 'MACHINE_SERIAL') > + root_dev = get_bitbake_var(bb_output, 'QEMU_ROOTFS_DEV') > + kargs = ['-append', '"console=' + serial + ' root=/dev/' + root_dev + ' rw"'] > + > + extra_args = ['-kernel', kernel_image, '-initrd', initrd_image] > + extra_args.extend(kargs) > + elif image_type == 'wic-img': > + rootfs_image = 'isar-image-base-debian-' + distro + '-qemu' + arch + '.wic.img' > + extra_args = ['-snapshot'] > + else: > + raise ValueError('Invalid image type: ' + str(image_type)) > + > + qemu_arch = get_bitbake_var(bb_output, 'QEMU_ARCH') > + qemu_machine = get_bitbake_var(bb_output, 'QEMU_MACHINE') > + qemu_cpu = get_bitbake_var(bb_output, 'QEMU_CPU') > + qemu_disk_args = get_bitbake_var(bb_output, 'QEMU_DISK_ARGS') > + > + qemu_disk_args = qemu_disk_args.replace('##ROOTFS_IMAGE##', deploy_dir_image + '/' + rootfs_image).split() > + > + cmd = ['qemu-system-' + qemu_arch, '-m', '1024M'] > + > + if qemu_machine: > + cmd.extend(['-M', qemu_machine]) > + > + if qemu_cpu: > + cmd.extend(['-cpu', qemu_cpu]) > + > + cmd.extend(extra_args) > + cmd.extend(qemu_disk_args) > + > + return cmd > + > +def start_qemu(arch, build, distro): > + cmdline = format_qemu_cmdline(arch, build, distro) > + cmdline.insert(1, '-nographic') > + > + p1 = subprocess.call(cmdline) > + > +if __name__ == "__main__": > + parser = argparse.ArgumentParser() > + parser.add_argument('-a', '--arch', choices=['arm', 'arm64', 'amd64', 'i386'], help='set isar machine architecture.', default='arm') > + parser.add_argument('-b', '--build', help='set path to build directory.', default=os.getcwd()) > + parser.add_argument('-d', '--distro', choices=['jessie', 'stretch'], help='set isar Debian distribution.', default='stretch') > + args = parser.parse_args() > + > + start_qemu(args.arch, args.build, args.distro) > diff --git a/testsuite/vm_boot_test/run.sh b/testsuite/vm_boot_test/run.sh > new file mode 100755 > index 0000000..9fdda95 > --- /dev/null > +++ b/testsuite/vm_boot_test/run.sh > @@ -0,0 +1,3 @@ > +#!/bin/bash > + > +avocado run vm_boot_test.py --mux-yaml test:variant.yaml --mux-inject build_dir:$BUILDDIR time_to_wait:300 > diff --git a/testsuite/vm_boot_test/variant.yaml b/testsuite/vm_boot_test/variant.yaml > new file mode 100644 > index 0000000..9ddc634 > --- /dev/null > +++ b/testsuite/vm_boot_test/variant.yaml > @@ -0,0 +1,22 @@ > +variants: !mux > + stretch-amd64: > + distro: "stretch" > + arch: "amd64" > + stretch-i386: > + distro: "stretch" > + arch: "i386" > + stretch-arm: > + distro: "stretch" > + arch: "arm" > + stretch-arm64: > + distro: "stretch" > + arch: "arm64" > + buster-amd64: > + distro: "buster" > + arch: "amd64" > + buster-i386: > + distro: "buster" > + arch: "i386" > + buster-arm: > + distro: "buster" > + arch: "arm" > diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py > new file mode 100644 > index 0000000..d4849c7 > --- /dev/null > +++ b/testsuite/vm_boot_test/vm_boot_test.py > @@ -0,0 +1,51 @@ > +#!/usr/bin/env python3 > + > +import os > +import subprocess32 > +import sys > +import time > + > +from os.path import dirname > +sys.path.append(dirname(__file__) + '/..') > + > +import start_vm > + > +from avocado import Test > + > +class VmBootTest(Test): > + > + def test(self): > + # TODO: add default values > + build_dir = self.params.get('build_dir', default='.') > + arch = self.params.get('arch', default='arm') > + distro = self.params.get('distro', default='stretch') > + time_to_wait = self.params.get('time_to_wait', default=60) > + > + self.log.info('===================================================') > + self.log.info('Running Isar VM boot test for (' + distro + '-' + arch + ')') > + self.log.info('Isar build folder is: ' + build_dir) > + self.log.info('===================================================') > + > + output_file = '/tmp/vm_boot_test.log' > + if os.path.exists(output_file): > + os.remove(output_file) > + > + cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro) > + cmdline.insert(1, '-nographic') > + cmdline.append('-serial') > + cmdline.append('file:' + output_file) > + > + self.log.info('QEMU boot line: ' + str(cmdline)) > + > + devnull = open(os.devnull, 'w') > + > + p1 = subprocess32.Popen(cmdline, stdout=devnull, stderr=devnull) > + time.sleep(int(time_to_wait)) > + p1.kill() > + p1.wait() > + > + if os.path.exists(output_file): > + if 'isar login:' in open(output_file).read(): > + return > + > + self.fail('Test failed') > -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de