From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6674828288478674944 X-Received: by 2002:a19:710d:: with SMTP id m13mr535161lfc.11.1555089999318; Fri, 12 Apr 2019 10:26:39 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:96ce:: with SMTP id d14ls105855ljj.5.gmail; Fri, 12 Apr 2019 10:26:38 -0700 (PDT) X-Google-Smtp-Source: APXvYqxdU5kz53OWOK8rEvFJPT7VTGoKajE8+24JiEi74GEYonwbUsqG+WV2HApgG9oJBQHyPz53 X-Received: by 2002:a2e:3015:: with SMTP id w21mr3116837ljw.20.1555089998620; Fri, 12 Apr 2019 10:26:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555089998; cv=none; d=google.com; s=arc-20160816; b=XrzvC0BpE9gAbOKSQKv5GspoZFgOU4q7CvHhbRjQEGA2XRmjTOVRC0Po2fy+oRlYU+ WA2IwihtijA5dhI1ipHy/7xhsSo4XeY6KnSsBXsNS/50XFcPodhIFEWP5zT4E8k0n4FY exyCRviXvwN6Zndg20mdozV335OUjsDuuQ3uF1yqBqulLxPFcUFFpGV9rGaMnf6havqr mlmZ/rsYWrgUMaLMt0dmOIHh/huWjgfGkTPBYSZ2Sir4aYafYRBWGap/po4L4PfrIFkE R+Nl+1aOOUvdGGwHLvuaWE8Tv2y9VBCckDJOb6P3TZrLS5XUPbpxh+0c62IPmX59rp5r 1KqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date; bh=RtLzbs5FBiNaFgWHGyigFvQfn4c13XJCjmKviEVia7E=; b=VhhQkdYAyKfJDETbm1Zoxf8rZkoER+ARkS5jmHNw7zRLOgCpSnRoCtkTD8iEOiTuHi PyNxlJvFbgnUIzyAql5uXQW2vCPt5ttyWkSrpyqnQAKLAmNQU8lBhXWlJ/iUNFjNtttG HHevd7ubn+ec3Nt+v8xjGOV2Z0YuYnHg29hGxlUh2bNKf05ruwhU42MhU8KO2stQMq84 Xqcvpi8uHoUQGJ1GpZU6nW7eoHenSLGFH3rRvlobIOkDaloVKz7Oq4sDR9YCCYet318h NMi4bQr40DMJW2Krq3ERg7xp7Sh3LXebGMbWx1KlWhtvmwJueKEpzEqUVMz+a3kBawuV 06Iw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id a22si1315974lfl.3.2019.04.12.10.26.38 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 12 Apr 2019 10:26:38 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id x3CHQb08006384 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Apr 2019 19:26:37 +0200 Received: from md1za8fc.ad001.siemens.net ([139.25.68.211]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id x3CHQbg5016137; Fri, 12 Apr 2019 19:26:37 +0200 Date: Fri, 12 Apr 2019 19:26:36 +0200 From: Henning Schild To: "[ext] Jan Kiszka" Cc: "Maxim Yu. Osipov" , Subject: Re: [PATCH] testsuite: Initial avocado tests implementation Message-ID: <20190412192636.58c098d2@md1za8fc.ad001.siemens.net> In-Reply-To: References: <20190401074224.10500-1-mosipov@ilbers.de> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: V7LKoA6s6s36 Am Wed, 3 Apr 2019 15:05:37 +0200 schrieb "[ext] Jan Kiszka" : > On 01.04.19 09:42, Maxim Yu. Osipov wrote: > > From: Alexander Smirnov > > > > Something is missing here... Why do we want this, what are the > benefits etc.? The answer to this is almost clear, but needs to be written down. Say we all agree that a testing framework is the way to go, but why do we pull in python (even pip) packages? Where is the RFC that compared testing frameworks? I bet there are many that can be installed with apt-get. https://docs.python.org/3/library/unittest.html If we go for Avocado the testing procedure should be explained using virtualenv. This path needs to be tested for sure, because we can not seriously ask anyone to mess up their system with pip. Henning > Jan > > > Signed-off-by: Alexander Smirnov > > Tested-by: Maxim Yu. Osipov > > --- > > testsuite/README.md | 34 +++++++++++++ > > testsuite/build_test/build_test.py | 29 +++++++++++ > > testsuite/build_test/run.sh | 3 ++ > > testsuite/build_test/variant.yaml | 22 +++++++++ > > testsuite/start_vm.py | 88 > > ++++++++++++++++++++++++++++++++++ > > testsuite/vm_boot_test/run.sh | 3 ++ > > testsuite/vm_boot_test/variant.yaml | 22 +++++++++ > > testsuite/vm_boot_test/vm_boot_test.py | 51 ++++++++++++++++++++ 8 > > files changed, 252 insertions(+) create mode 100644 > > testsuite/README.md create mode 100644 > > testsuite/build_test/build_test.py create mode 100755 > > testsuite/build_test/run.sh create mode 100644 > > testsuite/build_test/variant.yaml create mode 100755 > > testsuite/start_vm.py create mode 100755 > > testsuite/vm_boot_test/run.sh create mode 100644 > > testsuite/vm_boot_test/variant.yaml create mode 100644 > > testsuite/vm_boot_test/vm_boot_test.py > > > > diff --git a/testsuite/README.md b/testsuite/README.md > > new file mode 100644 > > index 0000000..5e64223 > > --- /dev/null > > +++ b/testsuite/README.md > > @@ -0,0 +1,34 @@ > > +# Install Avocado > > + > > +The framework could be installed by using standard HOWTO: > > + > > + > > https://github.com/avocado-framework/avocado#installing-with-standard-python-tools > > + +Then you need to install varianter yaml-to-mux plugin by > > following these instructions: + > > + > > https://github.com/avocado-framework/avocado/tree/master/optional_plugins > > + +## For Debian 9.x > > + > > + $ sudo apt-get install python-pip > > + $ pip install --user subprocess32 > > + $ pip install --user avocado-framework > > + $ pip install --user > > avocado-framework-plugin-varianter-yaml-to-mux + > > +# Pre > > + > > + $ export PATH=$PATH:~/.local/bin > > + $ cd isar > > + $ source isar-init-build-env > > + > > + > > +# Run test > > + > > +Each testsuite directory contains: > > + - run.sh - script to start tests > > + - variants.yaml - set of input data > > + - *.py - test case > > + > > +# Other > > + > > +There is a tool start_vm.py which is the replacement for the bash > > script in isar/scripts directory. diff --git > > a/testsuite/build_test/build_test.py > > b/testsuite/build_test/build_test.py new file mode 100644 index > > 0000000..4220d3a --- /dev/null > > +++ b/testsuite/build_test/build_test.py > > @@ -0,0 +1,29 @@ > > +#!/usr/bin/env python3 > > + > > +import os > > +import subprocess32 > > +import sys > > +from os.path import dirname > > + > > +from avocado import Test > > + > > +class BuildTest(Test): > > + > > + def test(self): > > + # TODO: add default values > > + build_dir = self.params.get('build_dir', default='.') > > + arch = self.params.get('arch', default='arm') > > + distro = self.params.get('distro', default='stretch') > > + > > + > > self.log.info('===================================================') > > + self.log.info('Running Isar build test for (' + distro + > > '-' + arch + ')') > > + self.log.info('Isar build folder is: ' + build_dir) > > + > > self.log.info('===================================================') > > + > > + #isar_root = dirname(__file__) + '/..' > > + os.chdir(build_dir) > > + cmdline = ['bitbake', 'multiconfig:qemu' + arch + '-' + > > distro + ':isar-image-base'] > > + p1 = subprocess32.run(cmdline) > > + > > + if p1.returncode: > > + self.fail('Test failed') > > diff --git a/testsuite/build_test/run.sh > > b/testsuite/build_test/run.sh new file mode 100755 > > index 0000000..a8ea9cc > > --- /dev/null > > +++ b/testsuite/build_test/run.sh > > @@ -0,0 +1,3 @@ > > +#!/bin/bash > > + > > +avocado run build_test.py --mux-yaml test:variant.yaml > > --mux-inject build_dir:$BUILDDIR diff --git > > a/testsuite/build_test/variant.yaml > > b/testsuite/build_test/variant.yaml new file mode 100644 index > > 0000000..9ddc634 --- /dev/null > > +++ b/testsuite/build_test/variant.yaml > > @@ -0,0 +1,22 @@ > > +variants: !mux > > + stretch-amd64: > > + distro: "stretch" > > + arch: "amd64" > > + stretch-i386: > > + distro: "stretch" > > + arch: "i386" > > + stretch-arm: > > + distro: "stretch" > > + arch: "arm" > > + stretch-arm64: > > + distro: "stretch" > > + arch: "arm64" > > + buster-amd64: > > + distro: "buster" > > + arch: "amd64" > > + buster-i386: > > + distro: "buster" > > + arch: "i386" > > + buster-arm: > > + distro: "buster" > > + arch: "arm" > > diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py > > new file mode 100755 > > index 0000000..02a4b51 > > --- /dev/null > > +++ b/testsuite/start_vm.py > > @@ -0,0 +1,88 @@ > > +#!/usr/bin/env python3 > > +# > > +# Helper script to start QEMU with Isar image > > +# Copyright (c) 2019, ilbers GmbH > > + > > +import argparse > > +import os > > +import subprocess > > +import sys > > +import time > > + > > +def get_bitbake_env(arch, distro): > > + multiconfig = 'multiconfig:qemu' + arch + '-' + distro + > > ':isar-image-base' > > + output = subprocess.check_output(['bitbake', '-e', > > str(multiconfig)]) > > + return output > > + > > +def get_bitbake_var(output, var): > > + ret = '' > > + for line in output.splitlines(): > > + if line.startswith(var): > > + ret = line.split('"')[1] > > + return ret > > + > > +def format_qemu_cmdline(arch, build, distro): > > + bb_output = get_bitbake_env(arch, distro).decode() > > + > > + rootfs_image = '' > > + extra_args = '' > > + cpu = [''] > > + > > + image_type = get_bitbake_var(bb_output, 'IMAGE_TYPE') > > + deploy_dir_image = get_bitbake_var(bb_output, > > 'DEPLOY_DIR_IMAGE') > > + if image_type == 'ext4-img': > > + rootfs_image = 'isar-image-base-debian-' + distro + > > '-qemu' + arch + '.ext4.img' > > + kernel_image = deploy_dir_image + '/' + > > get_bitbake_var(bb_output, 'KERNEL_IMAGE') > > + initrd_image = get_bitbake_var(bb_output, 'INITRD_IMAGE') > > + > > + if not initrd_image: > > + initrd_image = '/dev/null' > > + else: > > + initrd_image = deploy_dir_image + '/' + initrd_image > > + > > + serial = get_bitbake_var(bb_output, 'MACHINE_SERIAL') > > + root_dev = get_bitbake_var(bb_output, 'QEMU_ROOTFS_DEV') > > + kargs = ['-append', '"console=' + serial + ' root=/dev/' + > > root_dev + ' rw"'] + > > + extra_args = ['-kernel', kernel_image, '-initrd', > > initrd_image] > > + extra_args.extend(kargs) > > + elif image_type == 'wic-img': > > + rootfs_image = 'isar-image-base-debian-' + distro + > > '-qemu' + arch + '.wic.img' > > + extra_args = ['-snapshot'] > > + else: > > + raise ValueError('Invalid image type: ' + str(image_type)) > > + > > + qemu_arch = get_bitbake_var(bb_output, 'QEMU_ARCH') > > + qemu_machine = get_bitbake_var(bb_output, 'QEMU_MACHINE') > > + qemu_cpu = get_bitbake_var(bb_output, 'QEMU_CPU') > > + qemu_disk_args = get_bitbake_var(bb_output, 'QEMU_DISK_ARGS') > > + > > + qemu_disk_args = qemu_disk_args.replace('##ROOTFS_IMAGE##', > > deploy_dir_image + '/' + rootfs_image).split() + > > + cmd = ['qemu-system-' + qemu_arch, '-m', '1024M'] > > + > > + if qemu_machine: > > + cmd.extend(['-M', qemu_machine]) > > + > > + if qemu_cpu: > > + cmd.extend(['-cpu', qemu_cpu]) > > + > > + cmd.extend(extra_args) > > + cmd.extend(qemu_disk_args) > > + > > + return cmd > > + > > +def start_qemu(arch, build, distro): > > + cmdline = format_qemu_cmdline(arch, build, distro) > > + cmdline.insert(1, '-nographic') > > + > > + p1 = subprocess.call(cmdline) > > + > > +if __name__ == "__main__": > > + parser = argparse.ArgumentParser() > > + parser.add_argument('-a', '--arch', choices=['arm', 'arm64', > > 'amd64', 'i386'], help='set isar machine architecture.', > > default='arm') > > + parser.add_argument('-b', '--build', help='set path to build > > directory.', default=os.getcwd()) > > + parser.add_argument('-d', '--distro', choices=['jessie', > > 'stretch'], help='set isar Debian distribution.', default='stretch') > > + args = parser.parse_args() > > + > > + start_qemu(args.arch, args.build, args.distro) > > diff --git a/testsuite/vm_boot_test/run.sh > > b/testsuite/vm_boot_test/run.sh new file mode 100755 > > index 0000000..9fdda95 > > --- /dev/null > > +++ b/testsuite/vm_boot_test/run.sh > > @@ -0,0 +1,3 @@ > > +#!/bin/bash > > + > > +avocado run vm_boot_test.py --mux-yaml test:variant.yaml > > --mux-inject build_dir:$BUILDDIR time_to_wait:300 diff --git > > a/testsuite/vm_boot_test/variant.yaml > > b/testsuite/vm_boot_test/variant.yaml new file mode 100644 index > > 0000000..9ddc634 --- /dev/null > > +++ b/testsuite/vm_boot_test/variant.yaml > > @@ -0,0 +1,22 @@ > > +variants: !mux > > + stretch-amd64: > > + distro: "stretch" > > + arch: "amd64" > > + stretch-i386: > > + distro: "stretch" > > + arch: "i386" > > + stretch-arm: > > + distro: "stretch" > > + arch: "arm" > > + stretch-arm64: > > + distro: "stretch" > > + arch: "arm64" > > + buster-amd64: > > + distro: "buster" > > + arch: "amd64" > > + buster-i386: > > + distro: "buster" > > + arch: "i386" > > + buster-arm: > > + distro: "buster" > > + arch: "arm" > > diff --git a/testsuite/vm_boot_test/vm_boot_test.py > > b/testsuite/vm_boot_test/vm_boot_test.py new file mode 100644 > > index 0000000..d4849c7 > > --- /dev/null > > +++ b/testsuite/vm_boot_test/vm_boot_test.py > > @@ -0,0 +1,51 @@ > > +#!/usr/bin/env python3 > > + > > +import os > > +import subprocess32 > > +import sys > > +import time > > + > > +from os.path import dirname > > +sys.path.append(dirname(__file__) + '/..') > > + > > +import start_vm > > + > > +from avocado import Test > > + > > +class VmBootTest(Test): > > + > > + def test(self): > > + # TODO: add default values > > + build_dir = self.params.get('build_dir', default='.') > > + arch = self.params.get('arch', default='arm') > > + distro = self.params.get('distro', default='stretch') > > + time_to_wait = self.params.get('time_to_wait', default=60) > > + > > + > > self.log.info('===================================================') > > + self.log.info('Running Isar VM boot test for (' + distro + > > '-' + arch + ')') > > + self.log.info('Isar build folder is: ' + build_dir) > > + > > self.log.info('===================================================') > > + > > + output_file = '/tmp/vm_boot_test.log' > > + if os.path.exists(output_file): > > + os.remove(output_file) > > + > > + cmdline = start_vm.format_qemu_cmdline(arch, build_dir, > > distro) > > + cmdline.insert(1, '-nographic') > > + cmdline.append('-serial') > > + cmdline.append('file:' + output_file) > > + > > + self.log.info('QEMU boot line: ' + str(cmdline)) > > + > > + devnull = open(os.devnull, 'w') > > + > > + p1 = subprocess32.Popen(cmdline, stdout=devnull, > > stderr=devnull) > > + time.sleep(int(time_to_wait)) > > + p1.kill() > > + p1.wait() > > + > > + if os.path.exists(output_file): > > + if 'isar login:' in open(output_file).read(): > > + return > > + > > + self.fail('Test failed') > > >