From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6675671320760418304 X-Received: by 2002:a1c:1f10:: with SMTP id f16mr1005wmf.17.1554300850069; Wed, 03 Apr 2019 07:14:10 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:adf:ec8d:: with SMTP id z13ls566277wrn.3.gmail; Wed, 03 Apr 2019 07:14:09 -0700 (PDT) X-Google-Smtp-Source: APXvYqzO+INDEqYdNJpAU5RYj/sDY41pp4K6UA0GGOCnWzdMgdImUDvkEv81zQVl2Ctqkoevtfpx X-Received: by 2002:adf:e486:: with SMTP id i6mr18115wrm.14.1554300849385; Wed, 03 Apr 2019 07:14:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554300849; cv=none; d=google.com; s=arc-20160816; b=bsHvcA75vQgB6QTrJRrttUwJp2U/TUN9l+p+mpvWTdyqcg95eJKzlP2i/IMvRBV+G4 9Vw34N2VOh9EhBq+HZ5GkkRM4Jl0sSPqVmr235V93J6ftMhHkmOMbZvYeANnDdDpgd2/ j212S567x51djTWYflChEV5KORtgD3AdXR/wRfW5RWC7gEa/FsqcYhmZJMY3zz7NJ2Tt C+65WtHtRKAHgZwXjvM+ZEvrXjJdbhD9dENwIkSe9fv70byDDgmzETokpGQghfEy3gqe rg8u/26MIsGrr2W8F6O7RxuYKw859pEWtnH9Tdr9P+HLIrfS85H3b1XzZChc89e79GHJ ewdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:date:subject:to:from; bh=K0cZyDn4BAVoLB8OAFH1BDGULQeNiIedQ0UO8YK/LnI=; b=iU5Lsi4J7gqb1U5BosOUA2MJ/tpUGtprhVl/C8sl2K3DCQ2WdcAtIu6Lo6haYonryV 0S6ZOYjdaf1MOoeMhyoyO/W6pkXk/P6euAbkNiekjiNBj4Dz1IIReRbGwSsH77BpSDWh 7KLiaMH2TIduzIiOsKW/g4zOVK/OD3iUCrquKNM4nDHr+v7LMsq0YJf3r5NXmYZ4K34B bXzXj6/I8LTDPLEgopMFfxhtUTGHJt12OQyBHVGdXkYl8CGC1/lPPNOmpST+Taz9u+TF +ygp/9fHm++iMVdWrsvjZZjrYyNbwalMoq4tLlv9VzrK7unvIg/I30zK1Wc607g8qPuL wNyA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id d14si492311wrj.3.2019.04.03.07.14.09 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 03 Apr 2019 07:14:09 -0700 (PDT) Received-SPF: neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Received: from yssyq.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id x33EE8tb007575 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 3 Apr 2019 16:14:08 +0200 Received: from yssyq.m.ilbers.de (localhost [127.0.0.1]) by yssyq.m.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPS id x33EE7vk017735 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 3 Apr 2019 16:14:07 +0200 Received: (from ibr@localhost) by yssyq.m.ilbers.de (8.15.2/8.15.2/Submit) id x33EE7SS017734 for isar-users@googlegroups.com; Wed, 3 Apr 2019 16:14:07 +0200 From: Baurzhan Ismagulov To: isar-users@googlegroups.com Subject: [PATCH v2] testsuite: Initial Avocado test implementation Date: Wed, 3 Apr 2019 16:14:07 +0200 Message-Id: <20190403141407.17694-1-ibr@radix50.net> X-Mailer: git-send-email 2.11.0 X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: Qnt0LpLH0Z++ From: Alexander Smirnov Currently, we have several test cases executed from shell scripts. As test coverage increases, adding test cases and reading the results becomes complex. Avocado testing framework provides the necessary infrastructure to organize test cases. This patch adds the initial Avocado test case support for future CI migration. Signed-off-by: Alexander Smirnov Tested-by: Maxim Yu. Osipov --- testsuite/README.md | 34 +++++++++++++ testsuite/build_test/build_test.py | 29 +++++++++++ testsuite/build_test/run.sh | 3 ++ testsuite/build_test/variant.yaml | 22 +++++++++ testsuite/start_vm.py | 88 ++++++++++++++++++++++++++++++++++ testsuite/vm_boot_test/run.sh | 3 ++ testsuite/vm_boot_test/variant.yaml | 22 +++++++++ testsuite/vm_boot_test/vm_boot_test.py | 51 ++++++++++++++++++++ 8 files changed, 252 insertions(+) create mode 100644 testsuite/README.md create mode 100644 testsuite/build_test/build_test.py create mode 100755 testsuite/build_test/run.sh create mode 100644 testsuite/build_test/variant.yaml create mode 100755 testsuite/start_vm.py create mode 100755 testsuite/vm_boot_test/run.sh create mode 100644 testsuite/vm_boot_test/variant.yaml create mode 100644 testsuite/vm_boot_test/vm_boot_test.py diff --git a/testsuite/README.md b/testsuite/README.md new file mode 100644 index 0000000..5e64223 --- /dev/null +++ b/testsuite/README.md @@ -0,0 +1,34 @@ +# Install Avocado + +The framework could be installed by using standard HOWTO: + + https://github.com/avocado-framework/avocado#installing-with-standard-python-tools + +Then you need to install varianter yaml-to-mux plugin by following these instructions: + + https://github.com/avocado-framework/avocado/tree/master/optional_plugins + +## For Debian 9.x + + $ sudo apt-get install python-pip + $ pip install --user subprocess32 + $ pip install --user avocado-framework + $ pip install --user avocado-framework-plugin-varianter-yaml-to-mux + +# Pre + + $ export PATH=$PATH:~/.local/bin + $ cd isar + $ source isar-init-build-env + + +# Run test + +Each testsuite directory contains: + - run.sh - script to start tests + - variants.yaml - set of input data + - *.py - test case + +# Other + +There is a tool start_vm.py which is the replacement for the bash script in isar/scripts directory. diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py new file mode 100644 index 0000000..4220d3a --- /dev/null +++ b/testsuite/build_test/build_test.py @@ -0,0 +1,29 @@ +#!/usr/bin/env python3 + +import os +import subprocess32 +import sys +from os.path import dirname + +from avocado import Test + +class BuildTest(Test): + + def test(self): + # TODO: add default values + build_dir = self.params.get('build_dir', default='.') + arch = self.params.get('arch', default='arm') + distro = self.params.get('distro', default='stretch') + + self.log.info('===================================================') + self.log.info('Running Isar build test for (' + distro + '-' + arch + ')') + self.log.info('Isar build folder is: ' + build_dir) + self.log.info('===================================================') + + #isar_root = dirname(__file__) + '/..' + os.chdir(build_dir) + cmdline = ['bitbake', 'multiconfig:qemu' + arch + '-' + distro + ':isar-image-base'] + p1 = subprocess32.run(cmdline) + + if p1.returncode: + self.fail('Test failed') diff --git a/testsuite/build_test/run.sh b/testsuite/build_test/run.sh new file mode 100755 index 0000000..a8ea9cc --- /dev/null +++ b/testsuite/build_test/run.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +avocado run build_test.py --mux-yaml test:variant.yaml --mux-inject build_dir:$BUILDDIR diff --git a/testsuite/build_test/variant.yaml b/testsuite/build_test/variant.yaml new file mode 100644 index 0000000..9ddc634 --- /dev/null +++ b/testsuite/build_test/variant.yaml @@ -0,0 +1,22 @@ +variants: !mux + stretch-amd64: + distro: "stretch" + arch: "amd64" + stretch-i386: + distro: "stretch" + arch: "i386" + stretch-arm: + distro: "stretch" + arch: "arm" + stretch-arm64: + distro: "stretch" + arch: "arm64" + buster-amd64: + distro: "buster" + arch: "amd64" + buster-i386: + distro: "buster" + arch: "i386" + buster-arm: + distro: "buster" + arch: "arm" diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py new file mode 100755 index 0000000..02a4b51 --- /dev/null +++ b/testsuite/start_vm.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python3 +# +# Helper script to start QEMU with Isar image +# Copyright (c) 2019, ilbers GmbH + +import argparse +import os +import subprocess +import sys +import time + +def get_bitbake_env(arch, distro): + multiconfig = 'multiconfig:qemu' + arch + '-' + distro + ':isar-image-base' + output = subprocess.check_output(['bitbake', '-e', str(multiconfig)]) + return output + +def get_bitbake_var(output, var): + ret = '' + for line in output.splitlines(): + if line.startswith(var): + ret = line.split('"')[1] + return ret + +def format_qemu_cmdline(arch, build, distro): + bb_output = get_bitbake_env(arch, distro).decode() + + rootfs_image = '' + extra_args = '' + cpu = [''] + + image_type = get_bitbake_var(bb_output, 'IMAGE_TYPE') + deploy_dir_image = get_bitbake_var(bb_output, 'DEPLOY_DIR_IMAGE') + if image_type == 'ext4-img': + rootfs_image = 'isar-image-base-debian-' + distro + '-qemu' + arch + '.ext4.img' + kernel_image = deploy_dir_image + '/' + get_bitbake_var(bb_output, 'KERNEL_IMAGE') + initrd_image = get_bitbake_var(bb_output, 'INITRD_IMAGE') + + if not initrd_image: + initrd_image = '/dev/null' + else: + initrd_image = deploy_dir_image + '/' + initrd_image + + serial = get_bitbake_var(bb_output, 'MACHINE_SERIAL') + root_dev = get_bitbake_var(bb_output, 'QEMU_ROOTFS_DEV') + kargs = ['-append', '"console=' + serial + ' root=/dev/' + root_dev + ' rw"'] + + extra_args = ['-kernel', kernel_image, '-initrd', initrd_image] + extra_args.extend(kargs) + elif image_type == 'wic-img': + rootfs_image = 'isar-image-base-debian-' + distro + '-qemu' + arch + '.wic.img' + extra_args = ['-snapshot'] + else: + raise ValueError('Invalid image type: ' + str(image_type)) + + qemu_arch = get_bitbake_var(bb_output, 'QEMU_ARCH') + qemu_machine = get_bitbake_var(bb_output, 'QEMU_MACHINE') + qemu_cpu = get_bitbake_var(bb_output, 'QEMU_CPU') + qemu_disk_args = get_bitbake_var(bb_output, 'QEMU_DISK_ARGS') + + qemu_disk_args = qemu_disk_args.replace('##ROOTFS_IMAGE##', deploy_dir_image + '/' + rootfs_image).split() + + cmd = ['qemu-system-' + qemu_arch, '-m', '1024M'] + + if qemu_machine: + cmd.extend(['-M', qemu_machine]) + + if qemu_cpu: + cmd.extend(['-cpu', qemu_cpu]) + + cmd.extend(extra_args) + cmd.extend(qemu_disk_args) + + return cmd + +def start_qemu(arch, build, distro): + cmdline = format_qemu_cmdline(arch, build, distro) + cmdline.insert(1, '-nographic') + + p1 = subprocess.call(cmdline) + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument('-a', '--arch', choices=['arm', 'arm64', 'amd64', 'i386'], help='set isar machine architecture.', default='arm') + parser.add_argument('-b', '--build', help='set path to build directory.', default=os.getcwd()) + parser.add_argument('-d', '--distro', choices=['jessie', 'stretch'], help='set isar Debian distribution.', default='stretch') + args = parser.parse_args() + + start_qemu(args.arch, args.build, args.distro) diff --git a/testsuite/vm_boot_test/run.sh b/testsuite/vm_boot_test/run.sh new file mode 100755 index 0000000..9fdda95 --- /dev/null +++ b/testsuite/vm_boot_test/run.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +avocado run vm_boot_test.py --mux-yaml test:variant.yaml --mux-inject build_dir:$BUILDDIR time_to_wait:300 diff --git a/testsuite/vm_boot_test/variant.yaml b/testsuite/vm_boot_test/variant.yaml new file mode 100644 index 0000000..9ddc634 --- /dev/null +++ b/testsuite/vm_boot_test/variant.yaml @@ -0,0 +1,22 @@ +variants: !mux + stretch-amd64: + distro: "stretch" + arch: "amd64" + stretch-i386: + distro: "stretch" + arch: "i386" + stretch-arm: + distro: "stretch" + arch: "arm" + stretch-arm64: + distro: "stretch" + arch: "arm64" + buster-amd64: + distro: "buster" + arch: "amd64" + buster-i386: + distro: "buster" + arch: "i386" + buster-arm: + distro: "buster" + arch: "arm" diff --git a/testsuite/vm_boot_test/vm_boot_test.py b/testsuite/vm_boot_test/vm_boot_test.py new file mode 100644 index 0000000..d4849c7 --- /dev/null +++ b/testsuite/vm_boot_test/vm_boot_test.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python3 + +import os +import subprocess32 +import sys +import time + +from os.path import dirname +sys.path.append(dirname(__file__) + '/..') + +import start_vm + +from avocado import Test + +class VmBootTest(Test): + + def test(self): + # TODO: add default values + build_dir = self.params.get('build_dir', default='.') + arch = self.params.get('arch', default='arm') + distro = self.params.get('distro', default='stretch') + time_to_wait = self.params.get('time_to_wait', default=60) + + self.log.info('===================================================') + self.log.info('Running Isar VM boot test for (' + distro + '-' + arch + ')') + self.log.info('Isar build folder is: ' + build_dir) + self.log.info('===================================================') + + output_file = '/tmp/vm_boot_test.log' + if os.path.exists(output_file): + os.remove(output_file) + + cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro) + cmdline.insert(1, '-nographic') + cmdline.append('-serial') + cmdline.append('file:' + output_file) + + self.log.info('QEMU boot line: ' + str(cmdline)) + + devnull = open(os.devnull, 'w') + + p1 = subprocess32.Popen(cmdline, stdout=devnull, stderr=devnull) + time.sleep(int(time_to_wait)) + p1.kill() + p1.wait() + + if os.path.exists(output_file): + if 'isar login:' in open(output_file).read(): + return + + self.fail('Test failed') -- 2.11.0