From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6675671320760418304 X-Received: by 2002:adf:eb01:: with SMTP id s1mr48068wrn.5.1554306108780; Wed, 03 Apr 2019 08:41:48 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:adf:b642:: with SMTP id i2ls623008wre.13.gmail; Wed, 03 Apr 2019 08:41:48 -0700 (PDT) X-Google-Smtp-Source: APXvYqxP+wVS3gBxn+dYCu7Oqh1sSdHpi7onZFPqFa8ClkLZGDTYELjZVQrLc9RlLheZ0RAeinwU X-Received: by 2002:adf:f152:: with SMTP id y18mr51268wro.21.1554306108339; Wed, 03 Apr 2019 08:41:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554306108; cv=none; d=google.com; s=arc-20160816; b=LeUCPFCAiyaN2YuwOALL5x4BiOo76hp4ixRl1bxSlmfDZNz5xtP2L5OCrF8YhLIQFK ++xSkWGEpMaAys7rDla6cDhswPl4HLB+2TpSBub8RoAVT2pj2Ouaz7WZ6Phm3khPj3wx My4/r+Flf/F29xO2YpJ6Ot05UBtCMeXeHguCilTRJQa/A70DMGxw8zN2+8hmGCL/k3MG JaUa1ish50eo7XNbl4v6yK8xW6yEBID7aARxG3uSLjSI/XboHww+13XH4UsjbQzTHXB+ LAMu7ozN5d8FhWEvg8QVaiYHcQNvhE/1mgrH21fweYT8RZS61Mu5zaZe4n0PgWV0aFmN phJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:organization:from:references:to:subject; bh=Y+zOn6tJjp0pjVg6hAI1FyPnSOvxAcjahQUAz8mD6c4=; b=OEaO7PhLYlRoCJM5nPkHQK7h5t+ILm2Z55MErv7ydCu/MmutTbO9YDcQwjRSWNS5Ze 4sMLRcte+nkMqB6UtaaVdiKQMwdxHKMy0/4fIQCkyqukaRx/sxLjcVGisemYI7iGZ3YT vqPd2+XWVN8jMxTLUEklSsGvkl/d9p0IJP33f4vgixWTh9401yGdZJG1aG3oj5wWgOWn 3rdEk0se97WWXDR+FwAORpGExJNMa5cC9w836k+dt/HuLHZ0mUFZ8ZPjLluOxsYrtnGe YnVA1nRUHG6JCP+4SvkDV+/w5IZNozUAIy7c7kQ7YbUDsKEoEp//V4W32sWQvv5PsLuC LXSg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of mosipov@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=mosipov@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id v1si546379wrd.4.2019.04.03.08.41.48 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 03 Apr 2019 08:41:48 -0700 (PDT) Received-SPF: pass (google.com: domain of mosipov@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of mosipov@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=mosipov@ilbers.de Received: from [192.168.1.29] (4.28-242-81.adsl-dyn.isp.belgacom.be [81.242.28.4]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id x33Fffwr008099 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 3 Apr 2019 17:41:41 +0200 Subject: Re: [PATCH v2] testsuite: Initial Avocado test implementation To: Claudius Heine , Baurzhan Ismagulov , isar-users@googlegroups.com References: <20190403141407.17694-1-ibr@radix50.net> From: "Maxim Yu. Osipov" Organization: ilbers GmbH Message-ID: <4db17825-d4bc-4906-f81c-b7d055959038@ilbers.de> Date: Wed, 3 Apr 2019 17:41:35 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: PWdwsj5BpIBp On 4/3/19 4:28 PM, Claudius Heine wrote: > Hi, > > On 03/04/2019 16.14, Baurzhan Ismagulov wrote: >> From: Alexander Smirnov >> >> Currently, we have several test cases executed from shell scripts. As >> test >> coverage increases, adding test cases and reading the results becomes >> complex. >> Avocado testing framework provides the necessary infrastructure to >> organize >> test cases. This patch adds the initial Avocado test case support for >> future CI >> migration. >> >> Signed-off-by: Alexander Smirnov >> Tested-by: Maxim Yu. Osipov >> --- >>   testsuite/README.md                    | 34 +++++++++++++ >>   testsuite/build_test/build_test.py     | 29 +++++++++++ >>   testsuite/build_test/run.sh            |  3 ++ >>   testsuite/build_test/variant.yaml      | 22 +++++++++ >>   testsuite/start_vm.py                  | 88 >> ++++++++++++++++++++++++++++++++++ >>   testsuite/vm_boot_test/run.sh          |  3 ++ >>   testsuite/vm_boot_test/variant.yaml    | 22 +++++++++ >>   testsuite/vm_boot_test/vm_boot_test.py | 51 ++++++++++++++++++++ >>   8 files changed, 252 insertions(+) >>   create mode 100644 testsuite/README.md >>   create mode 100644 testsuite/build_test/build_test.py >>   create mode 100755 testsuite/build_test/run.sh >>   create mode 100644 testsuite/build_test/variant.yaml >>   create mode 100755 testsuite/start_vm.py >>   create mode 100755 testsuite/vm_boot_test/run.sh >>   create mode 100644 testsuite/vm_boot_test/variant.yaml >>   create mode 100644 testsuite/vm_boot_test/vm_boot_test.py >> >> diff --git a/testsuite/README.md b/testsuite/README.md >> new file mode 100644 >> index 0000000..5e64223 >> --- /dev/null >> +++ b/testsuite/README.md >> @@ -0,0 +1,34 @@ >> +# Install Avocado >> + >> +The framework could be installed by using standard HOWTO: >> + >> + >> https://github.com/avocado-framework/avocado#installing-with-standard-python-tools >> >> + >> +Then you need to install varianter yaml-to-mux plugin by following >> these instructions: >> + >> + >> https://github.com/avocado-framework/avocado/tree/master/optional_plugins >> + >> +## For Debian 9.x >> + >> +        $ sudo apt-get install python-pip >> +        $ pip install --user subprocess32 >> +        $ pip install --user avocado-framework >> +        $ pip install --user >> avocado-framework-plugin-varianter-yaml-to-mux >> + >> +# Pre >> + >> +        $ export PATH=$PATH:~/.local/bin >> +        $ cd isar >> +        $ source isar-init-build-env >> + >> + >> +# Run test >> + >> +Each testsuite directory contains: >> + - run.sh - script to start tests >> + - variants.yaml - set of input data >> + - *.py - test case >> + >> +# Other >> + >> +There is a tool start_vm.py which is the replacement for the bash >> script in isar/scripts directory. >> diff --git a/testsuite/build_test/build_test.py >> b/testsuite/build_test/build_test.py >> new file mode 100644 >> index 0000000..4220d3a >> --- /dev/null >> +++ b/testsuite/build_test/build_test.py >> @@ -0,0 +1,29 @@ >> +#!/usr/bin/env python3 >> + >> +import os >> +import subprocess32 >> +import sys >> +from os.path import dirname >> + >> +from avocado import Test >> + >> +class BuildTest(Test): >> + >> +    def test(self): >> +        # TODO: add default values >> +        build_dir = self.params.get('build_dir', default='.') >> +        arch = self.params.get('arch', default='arm') >> +        distro = self.params.get('distro', default='stretch') >> + >> + >> self.log.info('===================================================') >> +        self.log.info('Running Isar build test for (' + distro + '-' >> + arch + ')') >> +        self.log.info('Isar build folder is: ' + build_dir) >> + >> self.log.info('===================================================') >> + >> +        #isar_root = dirname(__file__) + '/..' >> +        os.chdir(build_dir) >> +        cmdline = ['bitbake', 'multiconfig:qemu' + arch + '-' + >> distro + ':isar-image-base'] >> +        p1 = subprocess32.run(cmdline) >> + >> +        if p1.returncode: >> +            self.fail('Test failed') >> diff --git a/testsuite/build_test/run.sh b/testsuite/build_test/run.sh >> new file mode 100755 >> index 0000000..a8ea9cc >> --- /dev/null >> +++ b/testsuite/build_test/run.sh >> @@ -0,0 +1,3 @@ >> +#!/bin/bash >> + >> +avocado run build_test.py --mux-yaml test:variant.yaml --mux-inject >> build_dir:$BUILDDIR >> diff --git a/testsuite/build_test/variant.yaml >> b/testsuite/build_test/variant.yaml >> new file mode 100644 >> index 0000000..9ddc634 >> --- /dev/null >> +++ b/testsuite/build_test/variant.yaml >> @@ -0,0 +1,22 @@ >> +variants: !mux >> +    stretch-amd64: >> +        distro: "stretch" >> +        arch:   "amd64" >> +    stretch-i386: >> +        distro: "stretch" >> +        arch:   "i386" >> +    stretch-arm: >> +        distro: "stretch" >> +        arch:   "arm" >> +    stretch-arm64: >> +        distro: "stretch" >> +        arch:   "arm64" >> +    buster-amd64: >> +        distro: "buster" >> +        arch:   "amd64" >> +    buster-i386: >> +        distro: "buster" >> +        arch:   "i386" >> +    buster-arm: >> +        distro: "buster" >> +        arch:   "arm" > > What about the non-qemu variants? At least a build test would be nice. > > Are you planning deploying and booting Isar with real HW in the CI as well? Yes, of course. This is just the first prototype. Maxim. > Claudius > >> diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py >> new file mode 100755 >> index 0000000..02a4b51 >> --- /dev/null >> +++ b/testsuite/start_vm.py >> @@ -0,0 +1,88 @@ >> +#!/usr/bin/env python3 >> +# >> +# Helper script to start QEMU with Isar image >> +# Copyright (c) 2019, ilbers GmbH >> + >> +import argparse >> +import os >> +import subprocess >> +import sys >> +import time >> + >> +def get_bitbake_env(arch, distro): >> +    multiconfig = 'multiconfig:qemu' + arch + '-' + distro + >> ':isar-image-base' >> +    output = subprocess.check_output(['bitbake', '-e', >> str(multiconfig)]) >> +    return output >> + >> +def get_bitbake_var(output, var): >> +    ret = '' >> +    for line in output.splitlines(): >> +        if line.startswith(var): >> +            ret = line.split('"')[1] >> +    return ret >> + >> +def format_qemu_cmdline(arch, build, distro): >> +    bb_output = get_bitbake_env(arch, distro).decode() >> + >> +    rootfs_image = '' >> +    extra_args = '' >> +    cpu = [''] >> + >> +    image_type = get_bitbake_var(bb_output, 'IMAGE_TYPE') >> +    deploy_dir_image = get_bitbake_var(bb_output, 'DEPLOY_DIR_IMAGE') >> +    if image_type == 'ext4-img': >> +        rootfs_image = 'isar-image-base-debian-' + distro + '-qemu' + >> arch + '.ext4.img' >> +        kernel_image = deploy_dir_image + '/' + >> get_bitbake_var(bb_output, 'KERNEL_IMAGE') >> +        initrd_image = get_bitbake_var(bb_output, 'INITRD_IMAGE') >> + >> +        if not initrd_image: >> +            initrd_image = '/dev/null' >> +        else: >> +            initrd_image = deploy_dir_image + '/' + initrd_image >> + >> +        serial = get_bitbake_var(bb_output, 'MACHINE_SERIAL') >> +        root_dev = get_bitbake_var(bb_output, 'QEMU_ROOTFS_DEV') >> +        kargs = ['-append', '"console=' + serial + ' root=/dev/' + >> root_dev + ' rw"'] >> + >> +        extra_args = ['-kernel', kernel_image, '-initrd', initrd_image] >> +        extra_args.extend(kargs) >> +    elif image_type == 'wic-img': >> +        rootfs_image = 'isar-image-base-debian-' + distro + '-qemu' + >> arch + '.wic.img' >> +        extra_args = ['-snapshot'] >> +    else: >> +        raise ValueError('Invalid image type: ' + str(image_type)) >> + >> +    qemu_arch = get_bitbake_var(bb_output, 'QEMU_ARCH') >> +    qemu_machine = get_bitbake_var(bb_output, 'QEMU_MACHINE') >> +    qemu_cpu = get_bitbake_var(bb_output, 'QEMU_CPU') >> +    qemu_disk_args = get_bitbake_var(bb_output, 'QEMU_DISK_ARGS') >> + >> +    qemu_disk_args = qemu_disk_args.replace('##ROOTFS_IMAGE##', >> deploy_dir_image + '/' + rootfs_image).split() >> + >> +    cmd = ['qemu-system-' + qemu_arch, '-m', '1024M'] >> + >> +    if qemu_machine: >> +        cmd.extend(['-M', qemu_machine]) >> + >> +    if qemu_cpu: >> +        cmd.extend(['-cpu', qemu_cpu]) >> + >> +    cmd.extend(extra_args) >> +    cmd.extend(qemu_disk_args) >> + >> +    return cmd >> + >> +def start_qemu(arch, build, distro): >> +    cmdline = format_qemu_cmdline(arch, build, distro) >> +    cmdline.insert(1, '-nographic') >> + >> +    p1 = subprocess.call(cmdline) >> + >> +if __name__ == "__main__": >> +    parser = argparse.ArgumentParser() >> +    parser.add_argument('-a', '--arch', choices=['arm', 'arm64', >> 'amd64', 'i386'], help='set isar machine architecture.', default='arm') >> +    parser.add_argument('-b', '--build', help='set path to build >> directory.', default=os.getcwd()) >> +    parser.add_argument('-d', '--distro', choices=['jessie', >> 'stretch'], help='set isar Debian distribution.', default='stretch') >> +    args = parser.parse_args() >> + >> +    start_qemu(args.arch, args.build, args.distro) >> diff --git a/testsuite/vm_boot_test/run.sh >> b/testsuite/vm_boot_test/run.sh >> new file mode 100755 >> index 0000000..9fdda95 >> --- /dev/null >> +++ b/testsuite/vm_boot_test/run.sh >> @@ -0,0 +1,3 @@ >> +#!/bin/bash >> + >> +avocado run vm_boot_test.py --mux-yaml test:variant.yaml --mux-inject >> build_dir:$BUILDDIR time_to_wait:300 >> diff --git a/testsuite/vm_boot_test/variant.yaml >> b/testsuite/vm_boot_test/variant.yaml >> new file mode 100644 >> index 0000000..9ddc634 >> --- /dev/null >> +++ b/testsuite/vm_boot_test/variant.yaml >> @@ -0,0 +1,22 @@ >> +variants: !mux >> +    stretch-amd64: >> +        distro: "stretch" >> +        arch:   "amd64" >> +    stretch-i386: >> +        distro: "stretch" >> +        arch:   "i386" >> +    stretch-arm: >> +        distro: "stretch" >> +        arch:   "arm" >> +    stretch-arm64: >> +        distro: "stretch" >> +        arch:   "arm64" >> +    buster-amd64: >> +        distro: "buster" >> +        arch:   "amd64" >> +    buster-i386: >> +        distro: "buster" >> +        arch:   "i386" >> +    buster-arm: >> +        distro: "buster" >> +        arch:   "arm" >> diff --git a/testsuite/vm_boot_test/vm_boot_test.py >> b/testsuite/vm_boot_test/vm_boot_test.py >> new file mode 100644 >> index 0000000..d4849c7 >> --- /dev/null >> +++ b/testsuite/vm_boot_test/vm_boot_test.py >> @@ -0,0 +1,51 @@ >> +#!/usr/bin/env python3 >> + >> +import os >> +import subprocess32 >> +import sys >> +import time >> + >> +from os.path import dirname >> +sys.path.append(dirname(__file__) + '/..') >> + >> +import start_vm >> + >> +from avocado import Test >> + >> +class VmBootTest(Test): >> + >> +    def test(self): >> +        # TODO: add default values >> +        build_dir = self.params.get('build_dir', default='.') >> +        arch = self.params.get('arch', default='arm') >> +        distro = self.params.get('distro', default='stretch') >> +        time_to_wait = self.params.get('time_to_wait', default=60) >> + >> + >> self.log.info('===================================================') >> +        self.log.info('Running Isar VM boot test for (' + distro + >> '-' + arch + ')') >> +        self.log.info('Isar build folder is: ' + build_dir) >> + >> self.log.info('===================================================') >> + >> +        output_file = '/tmp/vm_boot_test.log' >> +        if os.path.exists(output_file): >> +            os.remove(output_file) >> + >> +        cmdline = start_vm.format_qemu_cmdline(arch, build_dir, distro) >> +        cmdline.insert(1, '-nographic') >> +        cmdline.append('-serial') >> +        cmdline.append('file:' + output_file) >> + >> +        self.log.info('QEMU boot line: ' + str(cmdline)) >> + >> +        devnull = open(os.devnull, 'w') >> + >> +        p1 = subprocess32.Popen(cmdline, stdout=devnull, stderr=devnull) >> +        time.sleep(int(time_to_wait)) >> +        p1.kill() >> +        p1.wait() >> + >> +        if os.path.exists(output_file): >> +            if 'isar login:' in open(output_file).read(): >> +                return >> + >> +        self.fail('Test failed') >> > -- Maxim Osipov ilbers GmbH Maria-Merian-Str. 8 85521 Ottobrunn Germany +49 (151) 6517 6917 mosipov@ilbers.de http://ilbers.de/ Commercial register Munich, HRB 214197 General Manager: Baurzhan Ismagulov