* [RFC PATCH] testsuite: refactor
@ 2021-12-09 10:29 Adriaan Schmidt
2021-12-09 12:39 ` Anton Mikanovich
0 siblings, 1 reply; 3+ messages in thread
From: Adriaan Schmidt @ 2021-12-09 10:29 UTC (permalink / raw)
To: amikan, isar-users; +Cc: Adriaan Schmidt
Hi Anton,
as discussed earlier, I played around with the build tests, and tried to
simplify them (especially the bitbake configuration with the growing number
of positional arguments).
The main changes I've made:
- make test initialization more explicit.
tests now call init() to initialize build_dir and environment,
and configure() to generate the config file and bitbake_args
- build_dir is not passed via avocado parameter. each test
can set it in the call to init(). this makes dependencies
between tests explicit and permits parallelization.
- only configure() touches the config file. if config needs to change
during one test case, it is created from scratch (no appending
by the test case itself).
- remove the perform_*_test functions and put their logic directly
into the test cases. most of them were only used by one test case
anyway.
- remove cibuilder.py and put its remaining logic (which is not much
after the previous step) into cibase.py
With those changes I'm able to run the ci-fast tests. Haven't done
a thorough analysis yet, and I'm sure I've broken some details somewhere.
The work is not complete, but the general idea should be visible already.
Unfortunately the patch itself is not very readable, so best to look at the
updated files after applying.
Just let me know what you think and how you'd like to proceed.
Thanks and best regards,
Adriaan
Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
---
scripts/ci_build.sh | 19 +-
testsuite/build_test/build_test.py | 140 ++++++++++++--
testsuite/build_test/cibase.py | 286 ++++++++++++++++-------------
testsuite/build_test/cibuilder.py | 134 --------------
4 files changed, 279 insertions(+), 300 deletions(-)
delete mode 100644 testsuite/build_test/cibuilder.py
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index e9ba034..34f68df 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -27,9 +27,6 @@ fi
# Get Avocado build tests path
BUILD_TEST_DIR="$(pwd)/testsuite/build_test"
-# Start build in Isar tree by default
-BUILD_DIR=./build
-
# Check dependencies
DEPENDENCIES="umoci skopeo"
for prog in ${DEPENDENCIES} ; do
@@ -45,8 +42,6 @@ show_help() {
echo " $0 [params]"
echo
echo "Parameters:"
- echo " -b, --build BUILD_DIR set path to build directory. If not set,"
- echo " the build will be started in current path."
echo " -c, --cross enable cross-compilation."
echo " -d, --debug enable debug bitbake output."
echo " -f, --fast cross build reduced set of configurations."
@@ -73,10 +68,6 @@ do
show_help
exit 0
;;
- -b|--build)
- BUILD_DIR="$2"
- shift
- ;;
-c|--cross)
CROSS_BUILD="1"
;;
@@ -117,10 +108,10 @@ fi
mkdir -p .config/avocado
cat <<EOF > .config/avocado/avocado.conf
[datadir.paths]
-base_dir = $(realpath $BUILD_DIR)/
-test_dir = $(realpath $BUILD_DIR)/tests
-data_dir = $(realpath $BUILD_DIR)/data
-logs_dir = $(realpath $BUILD_DIR)/job-results
+base_dir = ./
+test_dir = ./tests
+data_dir = ./data
+logs_dir = ./job-results
EOF
export VIRTUAL_ENV="./"
@@ -129,4 +120,4 @@ set -x
avocado $VERBOSE run "$BUILD_TEST_DIR/build_test.py" \
-t $TAGS --test-runner=runner --disable-sysinfo \
- -p build_dir="$BUILD_DIR" -p quiet=$QUIET -p cross=$CROSS_BUILD
+ -p quiet=$QUIET -p cross=$CROSS_BUILD
diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
index de9e3fc..2643130 100644
--- a/testsuite/build_test/build_test.py
+++ b/testsuite/build_test/build_test.py
@@ -1,9 +1,12 @@
#!/usr/bin/env python3
+import glob
import os
+import re
+import tempfile
from avocado import skipUnless
-from avocado.utils import path
+from avocado.utils import path, process
from cibase import CIBaseTest
UMOCI_AVAILABLE = True
@@ -30,7 +33,27 @@ class ReproTest(CIBaseTest):
'mc:qemuarm64-stretch:isar-image-base'
]
- self.perform_repro_test(targets, 1)
+ gpg_pub_key = os.path.dirname(__file__) + '/../base-apt/test_pub.key'
+ gpg_priv_key = os.path.dirname(__file__) + '/../base-apt/test_priv.key'
+
+ self.init()
+ self.configure(gpg_pub_key=gpg_pub_key)
+
+ os.chdir(self.build_dir)
+ os.environ['GNUPGHOME'] = tempfile.mkdtemp()
+ result = process.run('gpg --import %s %s' % (gpg_pub_key, gpg_priv_key))
+
+ if result.exit_status:
+ self.fail('GPG import failed')
+
+ self.bitbake(targets)
+
+ self.delete_from_build_dir('tmp')
+ self.configure(offline=True, gpg_pub_key=gpg_pub_key)
+ self.bitbake(targets)
+
+ # Cleanup
+ self.delete_from_build_dir('tmp')
def test_repro_unsigned(self):
targets = [
@@ -38,7 +61,20 @@ class ReproTest(CIBaseTest):
'mc:qemuarm-stretch:isar-image-base'
]
- self.perform_repro_test(targets, 0)
+ self.init()
+ self.configure()
+ self.bitbake(targets)
+
+ self.delete_from_build_dir('tmp')
+ self.configure(offline=True)
+ self.bitbake(targets)
+
+ # Disable use of cached base repository
+ self.unconfigure()
+ self.bitbake(targets)
+
+ # Cleanup
+ self.delete_from_build_dir('tmp')
class CcacheTest(CIBaseTest):
@@ -49,7 +85,33 @@ class CcacheTest(CIBaseTest):
"""
def test_ccache_rebuild(self):
targets = ['mc:de0-nano-soc-buster:isar-image-base']
- self.perform_ccache_test(targets)
+
+ self.init()
+ self.configure(ccache=True)
+ self.delete_from_build_dir('tmp')
+ self.delete_from_build_dir('ccache')
+
+ self.log.info('Starting build and filling ccache dir...')
+ start = time.time()
+ self.bitbake(targets)
+ first_time = time.time() - start
+ self.log.info('Non-cached build: ' + str(round(first_time)) + 's')
+
+ self.delete_from_build_dir('tmp')
+
+ self.log.info('Starting build and using ccache dir...')
+ start = time.time()
+ self.bitbake(targets)
+ second_time = time.time() - start
+ self.log.info('Cached build: ' + str(round(second_time)) + 's')
+
+ speedup_k = 1.1
+ if first_time / second_time < speedup_k:
+ self.fail('No speedup after rebuild with ccache')
+
+ # Cleanup
+ self.delete_from_build_dir('tmp')
+ self.delete_from_build_dir('ccache')
class CrossTest(CIBaseTest):
@@ -69,15 +131,19 @@ class CrossTest(CIBaseTest):
'mc:rpi-stretch:isar-image-base'
]
- self.perform_build_test(targets, 1, None)
+ self.init()
+ self.configure(cross=True)
+ self.bitbake(targets)
def test_cross_ubuntu(self):
targets = [
'mc:qemuarm64-focal:isar-image-base'
]
+ self.init()
+ self.configure(cross=True)
try:
- self.perform_build_test(targets, 1, None)
+ self.bitbake(targets)
except:
self.cancel('KFAIL')
@@ -86,8 +152,10 @@ class CrossTest(CIBaseTest):
'mc:qemuarm-bullseye:isar-image-base'
]
+ self.init()
+ self.configure(cross=True)
try:
- self.perform_build_test(targets, 1, None)
+ self.bitbake(targets)
except:
self.cancel('KFAIL')
@@ -101,7 +169,9 @@ class SdkTest(CIBaseTest):
def test_sdk(self):
targets = ['mc:qemuarm-stretch:isar-image-base']
- self.perform_build_test(targets, 1, 'do_populate_sdk')
+ self.init()
+ self.configure(cross=True)
+ self.bitbake(targets, bitbake_cmd='do_populate_sdk')
class NoCrossTest(CIBaseTest):
@@ -130,11 +200,12 @@ class NoCrossTest(CIBaseTest):
'mc:virtualbox-ova-buster:isar-image-base'
]
+ self.init()
+ self.configure(cross=False)
# Cleanup after cross build
- self.deletetmp(self.params.get('build_dir',
- default=os.path.dirname(__file__) + '/../../build'))
+ self.delete_from_build_dir('tmp')
- self.perform_build_test(targets, 0, None)
+ self.bitbake(targets)
def test_nocross_bullseye(self):
targets = [
@@ -145,8 +216,10 @@ class NoCrossTest(CIBaseTest):
'mc:hikey-bullseye:isar-image-base'
]
+ self.init()
+ self.configure(cross=False)
try:
- self.perform_build_test(targets, 0, None)
+ self.bitbake(targets)
except:
self.cancel('KFAIL')
@@ -158,19 +231,18 @@ class RebuildTest(CIBaseTest):
:avocado: tags=rebuild,fast,full
"""
def test_rebuild(self):
- is_cross_build = int(self.params.get('cross', default=0))
+ self.init()
+ self.configure()
layerdir_core = self.getlayerdir('core')
dpkgbase_file = layerdir_core + '/classes/dpkg-base.bbclass'
-
self.backupfile(dpkgbase_file)
with open(dpkgbase_file, 'a') as file:
file.write('do_fetch_append() {\n\n}')
try:
- self.perform_build_test('mc:qemuamd64-stretch:isar-image-base',
- is_cross_build, None)
+ self.bitbake('mc:qemuamd64-stretch:isar-image-base')
finally:
self.restorefile(dpkgbase_file)
@@ -186,8 +258,32 @@ class WicTest(CIBaseTest):
wks_path = '/scripts/lib/wic/canned-wks/sdimage-efi.wks'
wic_path = '/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img'
- self.perform_wic_test('mc:qemuamd64-stretch:isar-image-base',
- wks_path, wic_path)
+ self.init()
+ self.configure()
+
+ layerdir_isar = self.getlayerdir('isar')
+ wks_file = layerdir_isar + wks_path
+ wic_img = self.build_dir + wic_path
+
+ if not os.path.isfile(wic_img):
+ self.fail('No build started before: ' + wic_img + ' not exist')
+
+ self.backupfile(wks_file)
+ self.backupmove(wic_img)
+ with open(wks_file, 'r') as file:
+ lines = file.readlines()
+ with open(wks_file, 'w') as file:
+ for line in lines:
+ file.write(re.sub(r'part \/ ', 'part \/ --exclude-path usr ',
+ line))
+
+ try:
+ self.bitbake('mc:qemuamd64-stretch:isar-image-base')
+ finally:
+ self.restorefile(wks_file)
+
+ self.restorefile(wic_img)
+
class ContainerImageTest(CIBaseTest):
@@ -204,7 +300,9 @@ class ContainerImageTest(CIBaseTest):
'mc:container-amd64-bullseye:isar-image-base'
]
- self.perform_container_test(targets, None)
+ self.init()
+ self.configure(container=True)
+ self.bitbake(targets)
class ContainerSdkTest(CIBaseTest):
@@ -217,4 +315,6 @@ class ContainerSdkTest(CIBaseTest):
def test_container_sdk(self):
targets = ['mc:container-amd64-stretch:isar-image-base']
- self.perform_container_test(targets, 'do_populate_sdk')
+ self.init()
+ self.configure(container=True)
+ self.bitbake(targets, bitbake_cmd='do_populate_sdk')
diff --git a/testsuite/build_test/cibase.py b/testsuite/build_test/cibase.py
index 0b053aa..52b01ed 100644
--- a/testsuite/build_test/cibase.py
+++ b/testsuite/build_test/cibase.py
@@ -1,149 +1,171 @@
#!/usr/bin/env python3
+import logging
import os
import re
-import tempfile
+import select
+import shutil
+import subprocess
import time
-from cibuilder import CIBuilder
+from avocado import Test
+from avocado.utils import path
from avocado.utils import process
-isar_root = os.path.dirname(__file__) + '/../..'
-
-class CIBaseTest(CIBuilder):
-
- def prep(self, testname, targets, cross, debsrc_cache):
- build_dir = self.params.get('build_dir', default=isar_root + '/build')
- build_dir = os.path.realpath(build_dir)
- quiet = int(self.params.get('quiet', default=0))
- bitbake_args = '-v'
-
- if quiet:
- bitbake_args = ''
-
+isar_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))
+backup_prefix = '.ci-backup'
+app_log = logging.getLogger("avocado.app")
+
+class CIBaseTest(Test):
+ def setUp(self):
+ super(CIBaseTest, self).setUp()
+ job_log = os.path.join(os.path.dirname(self.logdir), '..', 'job.log')
+ self._file_handler = logging.FileHandler(filename=job_log)
+ self._file_handler.setLevel(logging.ERROR)
+ fmt = ('%(asctime)s %(module)-16.16s L%(lineno)-.4d %('
+ 'levelname)-5.5s| %(message)s')
+ formatter = logging.Formatter(fmt=fmt)
+ self._file_handler.setFormatter(formatter)
+ app_log.addHandler(self._file_handler)
+
+
+ def init(self, build_dir='build'):
+ # initialize build_dir and setup environment
+ # needs to run once (per test case)
+ self.build_dir = os.path.join(isar_root, build_dir)
+ os.chdir(isar_root)
+ path.usable_rw_dir(self.build_dir)
+ output = process.getoutput('/bin/bash -c "source isar-init-build-env \
+ %s 2>&1 >/dev/null; env"' % self.build_dir)
+ env = dict(((x.split('=', 1) + [''])[:2] \
+ for x in output.splitlines() if x != ''))
+ os.environ.update(env)
+
+
+ def configure(self, compat_arch=True, cross=None, debsrc_cache=True,
+ container=False, ccache=False, offline=False,
+ gpg_pub_key=None):
+ # write configuration file and set bitbake_args
+ # can run multiple times per test case
+
+ # get parameters from avocado cmdline
+ quiet = bool(int(self.params.get('quiet', default=0)))
+ if cross is None:
+ cross = bool(int(self.params.get('cross', default=0)))
+
+ # get parameters from environment
+ distro_apt_premir = os.getenv('DISTRO_APT_PREMIRRORS')
+
+ self.log.info(f'===================================================\n'
+ f'Configuring build_dir {self.build_dir}\n'
+ f' compat_arch = {compat_arch}\n'
+ f' cross = {cross}\n'
+ f' debsrc_cache = {debsrc_cache}\n'
+ f' offline = {offline}\n'
+ f' container = {container}\n'
+ f' ccache = {ccache}\n'
+ f' gpg_pub_key = {gpg_pub_key}\n'
+ f'===================================================')
+
+ # determine bitbake_args
+ self.bitbake_args = []
+ if not quiet:
+ self.bitbake_args.append('-v')
+
+ # write ci_build.conf
+ with open(self.build_dir + '/conf/ci_build.conf', 'w') as f:
+ if compat_arch:
+ f.write('ISAR_ENABLE_COMPAT_ARCH_amd64 = "1"\n')
+ f.write('ISAR_ENABLE_COMPAT_ARCH_arm64 = "1"\n')
+ f.write('ISAR_ENABLE_COMPAT_ARCH_debian-stretch_amd64 = "0"\n')
+ f.write('IMAGE_INSTALL += "kselftest"\n')
+ if cross:
+ f.write('ISAR_CROSS_COMPILE = "1"\n')
+ if debsrc_cache:
+ f.write('BASE_REPO_FEATURES = "cache-deb-src"\n')
+ if offline:
+ f.write('ISAR_USE_CACHED_BASE_REPO = "1"\n')
+ f.write('BB_NO_NETWORK = "1"\n')
+ if container:
+ f.write('SDK_FORMATS = "docker-archive"\n')
+ f.write('IMAGE_INSTALL_remove = "example-module-${KERNEL_NAME} enable-fsck"\n')
+ if gpg_pub_key:
+ f.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
+ if distro_apt_premir:
+ f.write('DISTRO_APT_PREMIRRORS = "%s"\n' % distro_apt_premir)
+
+ # include ci_build.conf in local.conf
+ with open(self.build_dir + '/conf/local.conf', 'r+') as f:
+ for line in f:
+ if 'include ci_build.conf' in line:
+ break
+ else:
+ f.write('\ninclude ci_build.conf')
+
+ def unconfigure(self):
+ open(self.build_dir + '/conf/ci_build.conf', 'w').close()
+
+ def delete_from_build_dir(self, path):
+ process.run('rm -rf ' + self.build_dir + '/' + path, sudo=True)
+
+ def bitbake(self, target, bitbake_cmd=None):
self.log.info('===================================================')
- self.log.info('Running ' + testname + ' test for:')
- self.log.info(targets)
- self.log.info('Isar build folder is: ' + build_dir)
+ self.log.info('Building ' + str(target))
self.log.info('===================================================')
-
- self.init(build_dir)
- self.confprepare(build_dir, 1, cross, debsrc_cache)
-
- return build_dir, bitbake_args;
-
- def perform_build_test(self, targets, cross, bitbake_cmd):
- build_dir, bb_args = self.prep('Isar build', targets, cross, 1)
-
- self.log.info('Starting build...')
-
- self.bitbake(build_dir, targets, bitbake_cmd, bb_args)
-
- def perform_repro_test(self, targets, signed):
- cross = int(self.params.get('cross', default=0))
- build_dir, bb_args = self.prep('repro Isar build', targets, cross, 0)
-
- gpg_pub_key = os.path.dirname(__file__) + '/../base-apt/test_pub.key'
- gpg_priv_key = os.path.dirname(__file__) + '/../base-apt/test_priv.key'
-
- if signed:
- with open(build_dir + '/conf/ci_build.conf', 'a') as file:
- # Enable use of signed cached base repository
- file.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
-
- os.chdir(build_dir)
-
- os.environ['GNUPGHOME'] = tempfile.mkdtemp()
- result = process.run('gpg --import %s %s' % (gpg_pub_key, gpg_priv_key))
-
- if result.exit_status:
- self.fail('GPG import failed')
-
- self.bitbake(build_dir, targets, None, bb_args)
-
- self.deletetmp(build_dir)
- with open(build_dir + '/conf/ci_build.conf', 'a') as file:
- file.write('ISAR_USE_CACHED_BASE_REPO = "1"\n')
- file.write('BB_NO_NETWORK = "1"\n')
-
- self.bitbake(build_dir, targets, None, bb_args)
-
- # Disable use of cached base repository
- self.confcleanup(build_dir)
-
- if not signed:
- # Try to build with changed configuration with no cleanup
- self.bitbake(build_dir, targets, None, bb_args)
-
- # Cleanup
- self.deletetmp(build_dir)
-
- def perform_wic_test(self, targets, wks_path, wic_path):
- cross = int(self.params.get('cross', default=0))
- build_dir, bb_args = self.prep('WIC exclude build', targets, cross, 1)
-
- layerdir_isar = self.getlayerdir('isar')
-
- wks_file = layerdir_isar + wks_path
- wic_img = build_dir + wic_path
-
- if not os.path.isfile(wic_img):
- self.fail('No build started before: ' + wic_img + ' not exist')
-
- self.backupfile(wks_file)
- self.backupmove(wic_img)
-
- with open(wks_file, 'r') as file:
- lines = file.readlines()
- with open(wks_file, 'w') as file:
- for line in lines:
- file.write(re.sub(r'part \/ ', 'part \/ --exclude-path usr ',
- line))
-
+ os.chdir(self.build_dir)
+ cmdline = ['bitbake']
+ if self.bitbake_args:
+ cmdline.extend(self.bitbake_args)
+ if bitbake_cmd:
+ cmdline.append('-c')
+ cmdline.append(bitbake_cmd)
+ if isinstance(target, list):
+ cmdline.extend(target)
+ else:
+ cmdline.append(target)
+
+ with subprocess.Popen(" ".join(cmdline), stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE, universal_newlines=True,
+ shell=True) as p1:
+ poller = select.poll()
+ poller.register(p1.stdout, select.POLLIN)
+ poller.register(p1.stderr, select.POLLIN)
+ while p1.poll() is None:
+ events = poller.poll(1000)
+ for fd, event in events:
+ if event != select.POLLIN:
+ continue
+ if fd == p1.stdout.fileno():
+ self.log.info(p1.stdout.readline().rstrip())
+ if fd == p1.stderr.fileno():
+ app_log.error(p1.stderr.readline().rstrip())
+ p1.wait()
+ if p1.returncode:
+ self.fail('Bitbake failed')
+
+ def backupfile(self, path):
try:
- self.bitbake(build_dir, targets, None, bb_args)
- finally:
- self.restorefile(wks_file)
+ shutil.copy2(path, path + backup_prefix)
+ except FileNotFoundError:
+ self.log.warn(path + ' not exist')
- self.restorefile(wic_img)
-
- def perform_container_test(self, targets, bitbake_cmd):
- cross = int(self.params.get('cross', default=0))
- build_dir, bb_args = self.prep('Isar Container', targets, cross, 1)
-
- self.containerprep(build_dir)
-
- self.bitbake(build_dir, targets, bitbake_cmd, bb_args)
-
-
- def perform_ccache_test(self, targets):
- build_dir, bb_args = self.prep('Isar ccache build', targets, 0, 0)
-
- self.deletetmp(build_dir)
- process.run('rm -rf ' + build_dir + '/ccache', sudo=True)
-
- with open(build_dir + '/conf/ci_build.conf', 'a') as file:
- file.write('USE_CCACHE = "1"\n')
- file.write('CCACHE_TOP_DIR = "${TOPDIR}/ccache"')
+ def backupmove(self, path):
+ try:
+ shutil.move(path, path + backup_prefix)
+ except FileNotFoundError:
+ self.log.warn(path + ' not exist')
- self.log.info('Starting build and filling ccache dir...')
- start = time.time()
- self.bitbake(build_dir, targets, None, bb_args)
- first_time = time.time() - start
- self.log.info('Non-cached build: ' + str(round(first_time)) + 's')
+ def restorefile(self, path):
+ try:
+ shutil.move(path + backup_prefix, path)
+ except FileNotFoundError:
+ self.log.warn(path + backup_prefix + ' not exist')
- self.deletetmp(build_dir)
+ def getlayerdir(self, layer):
+ output = process.getoutput('bitbake -e | grep "^LAYERDIR_.*="')
+ env = dict(((x.split('=', 1) + [''])[:2] \
+ for x in output.splitlines() if x != ''))
- self.log.info('Starting build and using ccache dir...')
- start = time.time()
- self.bitbake(build_dir, targets, None, bb_args)
- second_time = time.time() - start
- self.log.info('Cached build: ' + str(round(second_time)) + 's')
+ return env['LAYERDIR_' + layer].strip('"')
- speedup_k = 1.1
- if first_time / second_time < speedup_k:
- self.fail('No speedup after rebuild with ccache')
- # Cleanup
- self.deletetmp(build_dir)
- process.run('rm -rf ' + build_dir + '/ccache', sudo=True)
diff --git a/testsuite/build_test/cibuilder.py b/testsuite/build_test/cibuilder.py
deleted file mode 100644
index 94786c7..0000000
--- a/testsuite/build_test/cibuilder.py
+++ /dev/null
@@ -1,134 +0,0 @@
-#!/usr/bin/env python3
-
-import logging
-import os
-import re
-import select
-import shutil
-import subprocess
-
-from avocado import Test
-from avocado.utils import path
-from avocado.utils import process
-
-isar_root = os.path.dirname(__file__) + '/../..'
-backup_prefix = '.ci-backup'
-
-app_log = logging.getLogger("avocado.app")
-
-class CIBuilder(Test):
- def setUp(self):
- super(CIBuilder, self).setUp()
- job_log = os.path.join(os.path.dirname(self.logdir), '..', 'job.log')
- self._file_handler = logging.FileHandler(filename=job_log)
- self._file_handler.setLevel(logging.ERROR)
- fmt = ('%(asctime)s %(module)-16.16s L%(lineno)-.4d %('
- 'levelname)-5.5s| %(message)s')
- formatter = logging.Formatter(fmt=fmt)
- self._file_handler.setFormatter(formatter)
- app_log.addHandler(self._file_handler)
-
- def init(self, build_dir):
- os.chdir(isar_root)
- path.usable_rw_dir(build_dir)
- output = process.getoutput('/bin/bash -c "source isar-init-build-env \
- %s 2>&1 >/dev/null; env"' % build_dir)
- env = dict(((x.split('=', 1) + [''])[:2] \
- for x in output.splitlines() if x != ''))
- os.environ.update(env)
-
- def confprepare(self, build_dir, compat_arch, cross, debsrc_cache):
- with open(build_dir + '/conf/ci_build.conf', 'w') as f:
- if compat_arch:
- f.write('ISAR_ENABLE_COMPAT_ARCH_amd64 = "1"\n')
- f.write('ISAR_ENABLE_COMPAT_ARCH_arm64 = "1"\n')
- f.write('ISAR_ENABLE_COMPAT_ARCH_debian-stretch_amd64 = "0"\n')
- f.write('IMAGE_INSTALL += "kselftest"\n')
- if cross:
- f.write('ISAR_CROSS_COMPILE = "1"\n')
- if debsrc_cache:
- f.write('BASE_REPO_FEATURES = "cache-deb-src"\n')
- distro_apt_premir = os.getenv('DISTRO_APT_PREMIRRORS')
- if distro_apt_premir:
- f.write('DISTRO_APT_PREMIRRORS = "%s"\n' % distro_apt_premir)
-
- with open(build_dir + '/conf/local.conf', 'r+') as f:
- for line in f:
- if 'include ci_build.conf' in line:
- break
- else:
- f.write('\ninclude ci_build.conf')
-
- def containerprep(self, build_dir):
- with open(build_dir + '/conf/ci_build.conf', 'a') as f:
- f.write('SDK_FORMATS = "docker-archive"\n')
- f.write('IMAGE_INSTALL_remove = "example-module-${KERNEL_NAME} enable-fsck"\n')
-
- def confcleanup(self, build_dir):
- open(build_dir + '/conf/ci_build.conf', 'w').close()
-
- def deletetmp(self, build_dir):
- process.run('rm -rf ' + build_dir + '/tmp', sudo=True)
-
- def bitbake(self, build_dir, target, cmd, args):
- os.chdir(build_dir)
- cmdline = ['bitbake']
- if args:
- cmdline.append(args)
- if cmd:
- cmdline.append('-c')
- cmdline.append(cmd)
- if isinstance(target, list):
- cmdline.extend(target)
- else:
- cmdline.append(target)
-
- with subprocess.Popen(" ".join(cmdline), stdout=subprocess.PIPE,
- stderr=subprocess.PIPE, universal_newlines=True,
- shell=True) as p1:
- poller = select.poll()
- poller.register(p1.stdout, select.POLLIN)
- poller.register(p1.stderr, select.POLLIN)
- while p1.poll() is None:
- events = poller.poll(1000)
- for fd, event in events:
- if event != select.POLLIN:
- continue
- if fd == p1.stdout.fileno():
- self.log.info(p1.stdout.readline().rstrip())
- if fd == p1.stderr.fileno():
- app_log.error(p1.stderr.readline().rstrip())
- p1.wait()
- if p1.returncode:
- self.fail('Bitbake failed')
-
- def backupfile(self, path):
- try:
- shutil.copy2(path, path + backup_prefix)
- except FileNotFoundError:
- self.log.warn(path + ' not exist')
-
- def backupmove(self, path):
- try:
- shutil.move(path, path + backup_prefix)
- except FileNotFoundError:
- self.log.warn(path + ' not exist')
-
- def restorefile(self, path):
- try:
- shutil.move(path + backup_prefix, path)
- except FileNotFoundError:
- self.log.warn(path + backup_prefix + ' not exist')
-
- def getlayerdir(self, layer):
- try:
- path.find_command('bitbake')
- except path.CmdNotFoundError:
- build_dir = self.params.get('build_dir',
- default=isar_root + '/build')
- self.init(build_dir)
- output = process.getoutput('bitbake -e | grep "^LAYERDIR_.*="')
- env = dict(((x.split('=', 1) + [''])[:2] \
- for x in output.splitlines() if x != ''))
-
- return env['LAYERDIR_' + layer].strip('"')
--
2.30.2
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [RFC PATCH] testsuite: refactor
2021-12-09 10:29 [RFC PATCH] testsuite: refactor Adriaan Schmidt
@ 2021-12-09 12:39 ` Anton Mikanovich
2021-12-10 8:14 ` Schmidt, Adriaan
0 siblings, 1 reply; 3+ messages in thread
From: Anton Mikanovich @ 2021-12-09 12:39 UTC (permalink / raw)
To: Adriaan Schmidt, isar-users
Hello Adriaan,
Test logic itself was moved to cibase to make build_test.py looks pretty
much like the list of tests and targets.
It makes target list maintaining much easier and avoid from searching
though the logic if any target changes needed.
So it's better to move existing logic from build_test.py to cibase.py,
but not vice versa.
Moreover your patch introduce a lot of code duplication. Current
implementation is also not ideal with it, but it's not a good idea to
make things even worse.
The user should be able to add any simple new test case without
copy-pasting, but with inherit some base test classes.
I also see you've reverted 94e8d23c 'CI: Fix paths for the latest
Avocado version' which probably will not work with the latest Avocado
versions.
Minimizing ci_build.sh itself is ok, because we should remove it
sometimes and move to the using of direct 'avocado run' calls instead.
09.12.2021 13:29, Adriaan Schmidt wrote:
> Hi Anton,
>
> as discussed earlier, I played around with the build tests, and tried to
> simplify them (especially the bitbake configuration with the growing number
> of positional arguments).
>
> The main changes I've made:
>
> - make test initialization more explicit.
> tests now call init() to initialize build_dir and environment,
> and configure() to generate the config file and bitbake_args
> - build_dir is not passed via avocado parameter. each test
> can set it in the call to init(). this makes dependencies
> between tests explicit and permits parallelization.
> - only configure() touches the config file. if config needs to change
> during one test case, it is created from scratch (no appending
> by the test case itself).
> - remove the perform_*_test functions and put their logic directly
> into the test cases. most of them were only used by one test case
> anyway.
> - remove cibuilder.py and put its remaining logic (which is not much
> after the previous step) into cibase.py
>
> With those changes I'm able to run the ci-fast tests. Haven't done
> a thorough analysis yet, and I'm sure I've broken some details somewhere.
> The work is not complete, but the general idea should be visible already.
> Unfortunately the patch itself is not very readable, so best to look at the
> updated files after applying.
>
> Just let me know what you think and how you'd like to proceed.
>
> Thanks and best regards,
> Adriaan
>
> Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> ---
> scripts/ci_build.sh | 19 +-
> testsuite/build_test/build_test.py | 140 ++++++++++++--
> testsuite/build_test/cibase.py | 286 ++++++++++++++++-------------
> testsuite/build_test/cibuilder.py | 134 --------------
> 4 files changed, 279 insertions(+), 300 deletions(-)
> delete mode 100644 testsuite/build_test/cibuilder.py
>
> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> index e9ba034..34f68df 100755
> --- a/scripts/ci_build.sh
> +++ b/scripts/ci_build.sh
> @@ -27,9 +27,6 @@ fi
> # Get Avocado build tests path
> BUILD_TEST_DIR="$(pwd)/testsuite/build_test"
>
> -# Start build in Isar tree by default
> -BUILD_DIR=./build
> -
> # Check dependencies
> DEPENDENCIES="umoci skopeo"
> for prog in ${DEPENDENCIES} ; do
> @@ -45,8 +42,6 @@ show_help() {
> echo " $0 [params]"
> echo
> echo "Parameters:"
> - echo " -b, --build BUILD_DIR set path to build directory. If not set,"
> - echo " the build will be started in current path."
> echo " -c, --cross enable cross-compilation."
> echo " -d, --debug enable debug bitbake output."
> echo " -f, --fast cross build reduced set of configurations."
> @@ -73,10 +68,6 @@ do
> show_help
> exit 0
> ;;
> - -b|--build)
> - BUILD_DIR="$2"
> - shift
> - ;;
> -c|--cross)
> CROSS_BUILD="1"
> ;;
> @@ -117,10 +108,10 @@ fi
> mkdir -p .config/avocado
> cat <<EOF > .config/avocado/avocado.conf
> [datadir.paths]
> -base_dir = $(realpath $BUILD_DIR)/
> -test_dir = $(realpath $BUILD_DIR)/tests
> -data_dir = $(realpath $BUILD_DIR)/data
> -logs_dir = $(realpath $BUILD_DIR)/job-results
> +base_dir = ./
> +test_dir = ./tests
> +data_dir = ./data
> +logs_dir = ./job-results
> EOF
> export VIRTUAL_ENV="./"
>
> @@ -129,4 +120,4 @@ set -x
>
> avocado $VERBOSE run "$BUILD_TEST_DIR/build_test.py" \
> -t $TAGS --test-runner=runner --disable-sysinfo \
> - -p build_dir="$BUILD_DIR" -p quiet=$QUIET -p cross=$CROSS_BUILD
> + -p quiet=$QUIET -p cross=$CROSS_BUILD
> diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
> index de9e3fc..2643130 100644
> --- a/testsuite/build_test/build_test.py
> +++ b/testsuite/build_test/build_test.py
> @@ -1,9 +1,12 @@
> #!/usr/bin/env python3
>
> +import glob
> import os
> +import re
> +import tempfile
>
> from avocado import skipUnless
> -from avocado.utils import path
> +from avocado.utils import path, process
> from cibase import CIBaseTest
>
> UMOCI_AVAILABLE = True
> @@ -30,7 +33,27 @@ class ReproTest(CIBaseTest):
> 'mc:qemuarm64-stretch:isar-image-base'
> ]
>
> - self.perform_repro_test(targets, 1)
> + gpg_pub_key = os.path.dirname(__file__) + '/../base-apt/test_pub.key'
> + gpg_priv_key = os.path.dirname(__file__) + '/../base-apt/test_priv.key'
> +
> + self.init()
> + self.configure(gpg_pub_key=gpg_pub_key)
> +
> + os.chdir(self.build_dir)
> + os.environ['GNUPGHOME'] = tempfile.mkdtemp()
> + result = process.run('gpg --import %s %s' % (gpg_pub_key, gpg_priv_key))
> +
> + if result.exit_status:
> + self.fail('GPG import failed')
> +
> + self.bitbake(targets)
> +
> + self.delete_from_build_dir('tmp')
> + self.configure(offline=True, gpg_pub_key=gpg_pub_key)
> + self.bitbake(targets)
> +
> + # Cleanup
> + self.delete_from_build_dir('tmp')
>
> def test_repro_unsigned(self):
> targets = [
> @@ -38,7 +61,20 @@ class ReproTest(CIBaseTest):
> 'mc:qemuarm-stretch:isar-image-base'
> ]
>
> - self.perform_repro_test(targets, 0)
> + self.init()
> + self.configure()
> + self.bitbake(targets)
> +
> + self.delete_from_build_dir('tmp')
> + self.configure(offline=True)
> + self.bitbake(targets)
> +
> + # Disable use of cached base repository
> + self.unconfigure()
> + self.bitbake(targets)
> +
> + # Cleanup
> + self.delete_from_build_dir('tmp')
>
> class CcacheTest(CIBaseTest):
>
> @@ -49,7 +85,33 @@ class CcacheTest(CIBaseTest):
> """
> def test_ccache_rebuild(self):
> targets = ['mc:de0-nano-soc-buster:isar-image-base']
> - self.perform_ccache_test(targets)
> +
> + self.init()
> + self.configure(ccache=True)
> + self.delete_from_build_dir('tmp')
> + self.delete_from_build_dir('ccache')
> +
> + self.log.info('Starting build and filling ccache dir...')
> + start = time.time()
> + self.bitbake(targets)
> + first_time = time.time() - start
> + self.log.info('Non-cached build: ' + str(round(first_time)) + 's')
> +
> + self.delete_from_build_dir('tmp')
> +
> + self.log.info('Starting build and using ccache dir...')
> + start = time.time()
> + self.bitbake(targets)
> + second_time = time.time() - start
> + self.log.info('Cached build: ' + str(round(second_time)) + 's')
> +
> + speedup_k = 1.1
> + if first_time / second_time < speedup_k:
> + self.fail('No speedup after rebuild with ccache')
> +
> + # Cleanup
> + self.delete_from_build_dir('tmp')
> + self.delete_from_build_dir('ccache')
>
> class CrossTest(CIBaseTest):
>
> @@ -69,15 +131,19 @@ class CrossTest(CIBaseTest):
> 'mc:rpi-stretch:isar-image-base'
> ]
>
> - self.perform_build_test(targets, 1, None)
> + self.init()
> + self.configure(cross=True)
> + self.bitbake(targets)
>
> def test_cross_ubuntu(self):
> targets = [
> 'mc:qemuarm64-focal:isar-image-base'
> ]
>
> + self.init()
> + self.configure(cross=True)
> try:
> - self.perform_build_test(targets, 1, None)
> + self.bitbake(targets)
> except:
> self.cancel('KFAIL')
>
> @@ -86,8 +152,10 @@ class CrossTest(CIBaseTest):
> 'mc:qemuarm-bullseye:isar-image-base'
> ]
>
> + self.init()
> + self.configure(cross=True)
> try:
> - self.perform_build_test(targets, 1, None)
> + self.bitbake(targets)
> except:
> self.cancel('KFAIL')
>
> @@ -101,7 +169,9 @@ class SdkTest(CIBaseTest):
> def test_sdk(self):
> targets = ['mc:qemuarm-stretch:isar-image-base']
>
> - self.perform_build_test(targets, 1, 'do_populate_sdk')
> + self.init()
> + self.configure(cross=True)
> + self.bitbake(targets, bitbake_cmd='do_populate_sdk')
>
> class NoCrossTest(CIBaseTest):
>
> @@ -130,11 +200,12 @@ class NoCrossTest(CIBaseTest):
> 'mc:virtualbox-ova-buster:isar-image-base'
> ]
>
> + self.init()
> + self.configure(cross=False)
> # Cleanup after cross build
> - self.deletetmp(self.params.get('build_dir',
> - default=os.path.dirname(__file__) + '/../../build'))
> + self.delete_from_build_dir('tmp')
>
> - self.perform_build_test(targets, 0, None)
> + self.bitbake(targets)
>
> def test_nocross_bullseye(self):
> targets = [
> @@ -145,8 +216,10 @@ class NoCrossTest(CIBaseTest):
> 'mc:hikey-bullseye:isar-image-base'
> ]
>
> + self.init()
> + self.configure(cross=False)
> try:
> - self.perform_build_test(targets, 0, None)
> + self.bitbake(targets)
> except:
> self.cancel('KFAIL')
>
> @@ -158,19 +231,18 @@ class RebuildTest(CIBaseTest):
> :avocado: tags=rebuild,fast,full
> """
> def test_rebuild(self):
> - is_cross_build = int(self.params.get('cross', default=0))
> + self.init()
> + self.configure()
>
> layerdir_core = self.getlayerdir('core')
>
> dpkgbase_file = layerdir_core + '/classes/dpkg-base.bbclass'
> -
> self.backupfile(dpkgbase_file)
> with open(dpkgbase_file, 'a') as file:
> file.write('do_fetch_append() {\n\n}')
>
> try:
> - self.perform_build_test('mc:qemuamd64-stretch:isar-image-base',
> - is_cross_build, None)
> + self.bitbake('mc:qemuamd64-stretch:isar-image-base')
> finally:
> self.restorefile(dpkgbase_file)
>
> @@ -186,8 +258,32 @@ class WicTest(CIBaseTest):
> wks_path = '/scripts/lib/wic/canned-wks/sdimage-efi.wks'
> wic_path = '/tmp/deploy/images/qemuamd64/isar-image-base-debian-stretch-qemuamd64.wic.img'
>
> - self.perform_wic_test('mc:qemuamd64-stretch:isar-image-base',
> - wks_path, wic_path)
> + self.init()
> + self.configure()
> +
> + layerdir_isar = self.getlayerdir('isar')
> + wks_file = layerdir_isar + wks_path
> + wic_img = self.build_dir + wic_path
> +
> + if not os.path.isfile(wic_img):
> + self.fail('No build started before: ' + wic_img + ' not exist')
> +
> + self.backupfile(wks_file)
> + self.backupmove(wic_img)
> + with open(wks_file, 'r') as file:
> + lines = file.readlines()
> + with open(wks_file, 'w') as file:
> + for line in lines:
> + file.write(re.sub(r'part \/ ', 'part \/ --exclude-path usr ',
> + line))
> +
> + try:
> + self.bitbake('mc:qemuamd64-stretch:isar-image-base')
> + finally:
> + self.restorefile(wks_file)
> +
> + self.restorefile(wic_img)
> +
>
> class ContainerImageTest(CIBaseTest):
>
> @@ -204,7 +300,9 @@ class ContainerImageTest(CIBaseTest):
> 'mc:container-amd64-bullseye:isar-image-base'
> ]
>
> - self.perform_container_test(targets, None)
> + self.init()
> + self.configure(container=True)
> + self.bitbake(targets)
>
> class ContainerSdkTest(CIBaseTest):
>
> @@ -217,4 +315,6 @@ class ContainerSdkTest(CIBaseTest):
> def test_container_sdk(self):
> targets = ['mc:container-amd64-stretch:isar-image-base']
>
> - self.perform_container_test(targets, 'do_populate_sdk')
> + self.init()
> + self.configure(container=True)
> + self.bitbake(targets, bitbake_cmd='do_populate_sdk')
> diff --git a/testsuite/build_test/cibase.py b/testsuite/build_test/cibase.py
> index 0b053aa..52b01ed 100644
> --- a/testsuite/build_test/cibase.py
> +++ b/testsuite/build_test/cibase.py
> @@ -1,149 +1,171 @@
> #!/usr/bin/env python3
>
> +import logging
> import os
> import re
> -import tempfile
> +import select
> +import shutil
> +import subprocess
> import time
>
> -from cibuilder import CIBuilder
> +from avocado import Test
> +from avocado.utils import path
> from avocado.utils import process
>
> -isar_root = os.path.dirname(__file__) + '/../..'
> -
> -class CIBaseTest(CIBuilder):
> -
> - def prep(self, testname, targets, cross, debsrc_cache):
> - build_dir = self.params.get('build_dir', default=isar_root + '/build')
> - build_dir = os.path.realpath(build_dir)
> - quiet = int(self.params.get('quiet', default=0))
> - bitbake_args = '-v'
> -
> - if quiet:
> - bitbake_args = ''
> -
> +isar_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))
> +backup_prefix = '.ci-backup'
> +app_log = logging.getLogger("avocado.app")
> +
> +class CIBaseTest(Test):
> + def setUp(self):
> + super(CIBaseTest, self).setUp()
> + job_log = os.path.join(os.path.dirname(self.logdir), '..', 'job.log')
> + self._file_handler = logging.FileHandler(filename=job_log)
> + self._file_handler.setLevel(logging.ERROR)
> + fmt = ('%(asctime)s %(module)-16.16s L%(lineno)-.4d %('
> + 'levelname)-5.5s| %(message)s')
> + formatter = logging.Formatter(fmt=fmt)
> + self._file_handler.setFormatter(formatter)
> + app_log.addHandler(self._file_handler)
> +
> +
> + def init(self, build_dir='build'):
> + # initialize build_dir and setup environment
> + # needs to run once (per test case)
> + self.build_dir = os.path.join(isar_root, build_dir)
> + os.chdir(isar_root)
> + path.usable_rw_dir(self.build_dir)
> + output = process.getoutput('/bin/bash -c "source isar-init-build-env \
> + %s 2>&1 >/dev/null; env"' % self.build_dir)
> + env = dict(((x.split('=', 1) + [''])[:2] \
> + for x in output.splitlines() if x != ''))
> + os.environ.update(env)
> +
> +
> + def configure(self, compat_arch=True, cross=None, debsrc_cache=True,
> + container=False, ccache=False, offline=False,
> + gpg_pub_key=None):
> + # write configuration file and set bitbake_args
> + # can run multiple times per test case
> +
> + # get parameters from avocado cmdline
> + quiet = bool(int(self.params.get('quiet', default=0)))
> + if cross is None:
> + cross = bool(int(self.params.get('cross', default=0)))
> +
> + # get parameters from environment
> + distro_apt_premir = os.getenv('DISTRO_APT_PREMIRRORS')
> +
> + self.log.info(f'===================================================\n'
> + f'Configuring build_dir {self.build_dir}\n'
> + f' compat_arch = {compat_arch}\n'
> + f' cross = {cross}\n'
> + f' debsrc_cache = {debsrc_cache}\n'
> + f' offline = {offline}\n'
> + f' container = {container}\n'
> + f' ccache = {ccache}\n'
> + f' gpg_pub_key = {gpg_pub_key}\n'
> + f'===================================================')
> +
> + # determine bitbake_args
> + self.bitbake_args = []
> + if not quiet:
> + self.bitbake_args.append('-v')
> +
> + # write ci_build.conf
> + with open(self.build_dir + '/conf/ci_build.conf', 'w') as f:
> + if compat_arch:
> + f.write('ISAR_ENABLE_COMPAT_ARCH_amd64 = "1"\n')
> + f.write('ISAR_ENABLE_COMPAT_ARCH_arm64 = "1"\n')
> + f.write('ISAR_ENABLE_COMPAT_ARCH_debian-stretch_amd64 = "0"\n')
> + f.write('IMAGE_INSTALL += "kselftest"\n')
> + if cross:
> + f.write('ISAR_CROSS_COMPILE = "1"\n')
> + if debsrc_cache:
> + f.write('BASE_REPO_FEATURES = "cache-deb-src"\n')
> + if offline:
> + f.write('ISAR_USE_CACHED_BASE_REPO = "1"\n')
> + f.write('BB_NO_NETWORK = "1"\n')
> + if container:
> + f.write('SDK_FORMATS = "docker-archive"\n')
> + f.write('IMAGE_INSTALL_remove = "example-module-${KERNEL_NAME} enable-fsck"\n')
> + if gpg_pub_key:
> + f.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
> + if distro_apt_premir:
> + f.write('DISTRO_APT_PREMIRRORS = "%s"\n' % distro_apt_premir)
> +
> + # include ci_build.conf in local.conf
> + with open(self.build_dir + '/conf/local.conf', 'r+') as f:
> + for line in f:
> + if 'include ci_build.conf' in line:
> + break
> + else:
> + f.write('\ninclude ci_build.conf')
> +
> + def unconfigure(self):
> + open(self.build_dir + '/conf/ci_build.conf', 'w').close()
> +
> + def delete_from_build_dir(self, path):
> + process.run('rm -rf ' + self.build_dir + '/' + path, sudo=True)
> +
> + def bitbake(self, target, bitbake_cmd=None):
> self.log.info('===================================================')
> - self.log.info('Running ' + testname + ' test for:')
> - self.log.info(targets)
> - self.log.info('Isar build folder is: ' + build_dir)
> + self.log.info('Building ' + str(target))
> self.log.info('===================================================')
> -
> - self.init(build_dir)
> - self.confprepare(build_dir, 1, cross, debsrc_cache)
> -
> - return build_dir, bitbake_args;
> -
> - def perform_build_test(self, targets, cross, bitbake_cmd):
> - build_dir, bb_args = self.prep('Isar build', targets, cross, 1)
> -
> - self.log.info('Starting build...')
> -
> - self.bitbake(build_dir, targets, bitbake_cmd, bb_args)
> -
> - def perform_repro_test(self, targets, signed):
> - cross = int(self.params.get('cross', default=0))
> - build_dir, bb_args = self.prep('repro Isar build', targets, cross, 0)
> -
> - gpg_pub_key = os.path.dirname(__file__) + '/../base-apt/test_pub.key'
> - gpg_priv_key = os.path.dirname(__file__) + '/../base-apt/test_priv.key'
> -
> - if signed:
> - with open(build_dir + '/conf/ci_build.conf', 'a') as file:
> - # Enable use of signed cached base repository
> - file.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
> -
> - os.chdir(build_dir)
> -
> - os.environ['GNUPGHOME'] = tempfile.mkdtemp()
> - result = process.run('gpg --import %s %s' % (gpg_pub_key, gpg_priv_key))
> -
> - if result.exit_status:
> - self.fail('GPG import failed')
> -
> - self.bitbake(build_dir, targets, None, bb_args)
> -
> - self.deletetmp(build_dir)
> - with open(build_dir + '/conf/ci_build.conf', 'a') as file:
> - file.write('ISAR_USE_CACHED_BASE_REPO = "1"\n')
> - file.write('BB_NO_NETWORK = "1"\n')
> -
> - self.bitbake(build_dir, targets, None, bb_args)
> -
> - # Disable use of cached base repository
> - self.confcleanup(build_dir)
> -
> - if not signed:
> - # Try to build with changed configuration with no cleanup
> - self.bitbake(build_dir, targets, None, bb_args)
> -
> - # Cleanup
> - self.deletetmp(build_dir)
> -
> - def perform_wic_test(self, targets, wks_path, wic_path):
> - cross = int(self.params.get('cross', default=0))
> - build_dir, bb_args = self.prep('WIC exclude build', targets, cross, 1)
> -
> - layerdir_isar = self.getlayerdir('isar')
> -
> - wks_file = layerdir_isar + wks_path
> - wic_img = build_dir + wic_path
> -
> - if not os.path.isfile(wic_img):
> - self.fail('No build started before: ' + wic_img + ' not exist')
> -
> - self.backupfile(wks_file)
> - self.backupmove(wic_img)
> -
> - with open(wks_file, 'r') as file:
> - lines = file.readlines()
> - with open(wks_file, 'w') as file:
> - for line in lines:
> - file.write(re.sub(r'part \/ ', 'part \/ --exclude-path usr ',
> - line))
> -
> + os.chdir(self.build_dir)
> + cmdline = ['bitbake']
> + if self.bitbake_args:
> + cmdline.extend(self.bitbake_args)
> + if bitbake_cmd:
> + cmdline.append('-c')
> + cmdline.append(bitbake_cmd)
> + if isinstance(target, list):
> + cmdline.extend(target)
> + else:
> + cmdline.append(target)
> +
> + with subprocess.Popen(" ".join(cmdline), stdout=subprocess.PIPE,
> + stderr=subprocess.PIPE, universal_newlines=True,
> + shell=True) as p1:
> + poller = select.poll()
> + poller.register(p1.stdout, select.POLLIN)
> + poller.register(p1.stderr, select.POLLIN)
> + while p1.poll() is None:
> + events = poller.poll(1000)
> + for fd, event in events:
> + if event != select.POLLIN:
> + continue
> + if fd == p1.stdout.fileno():
> + self.log.info(p1.stdout.readline().rstrip())
> + if fd == p1.stderr.fileno():
> + app_log.error(p1.stderr.readline().rstrip())
> + p1.wait()
> + if p1.returncode:
> + self.fail('Bitbake failed')
> +
> + def backupfile(self, path):
> try:
> - self.bitbake(build_dir, targets, None, bb_args)
> - finally:
> - self.restorefile(wks_file)
> + shutil.copy2(path, path + backup_prefix)
> + except FileNotFoundError:
> + self.log.warn(path + ' not exist')
>
> - self.restorefile(wic_img)
> -
> - def perform_container_test(self, targets, bitbake_cmd):
> - cross = int(self.params.get('cross', default=0))
> - build_dir, bb_args = self.prep('Isar Container', targets, cross, 1)
> -
> - self.containerprep(build_dir)
> -
> - self.bitbake(build_dir, targets, bitbake_cmd, bb_args)
> -
> -
> - def perform_ccache_test(self, targets):
> - build_dir, bb_args = self.prep('Isar ccache build', targets, 0, 0)
> -
> - self.deletetmp(build_dir)
> - process.run('rm -rf ' + build_dir + '/ccache', sudo=True)
> -
> - with open(build_dir + '/conf/ci_build.conf', 'a') as file:
> - file.write('USE_CCACHE = "1"\n')
> - file.write('CCACHE_TOP_DIR = "${TOPDIR}/ccache"')
> + def backupmove(self, path):
> + try:
> + shutil.move(path, path + backup_prefix)
> + except FileNotFoundError:
> + self.log.warn(path + ' not exist')
>
> - self.log.info('Starting build and filling ccache dir...')
> - start = time.time()
> - self.bitbake(build_dir, targets, None, bb_args)
> - first_time = time.time() - start
> - self.log.info('Non-cached build: ' + str(round(first_time)) + 's')
> + def restorefile(self, path):
> + try:
> + shutil.move(path + backup_prefix, path)
> + except FileNotFoundError:
> + self.log.warn(path + backup_prefix + ' not exist')
>
> - self.deletetmp(build_dir)
> + def getlayerdir(self, layer):
> + output = process.getoutput('bitbake -e | grep "^LAYERDIR_.*="')
> + env = dict(((x.split('=', 1) + [''])[:2] \
> + for x in output.splitlines() if x != ''))
>
> - self.log.info('Starting build and using ccache dir...')
> - start = time.time()
> - self.bitbake(build_dir, targets, None, bb_args)
> - second_time = time.time() - start
> - self.log.info('Cached build: ' + str(round(second_time)) + 's')
> + return env['LAYERDIR_' + layer].strip('"')
>
> - speedup_k = 1.1
> - if first_time / second_time < speedup_k:
> - self.fail('No speedup after rebuild with ccache')
>
> - # Cleanup
> - self.deletetmp(build_dir)
> - process.run('rm -rf ' + build_dir + '/ccache', sudo=True)
> diff --git a/testsuite/build_test/cibuilder.py b/testsuite/build_test/cibuilder.py
> deleted file mode 100644
> index 94786c7..0000000
> --- a/testsuite/build_test/cibuilder.py
> +++ /dev/null
> @@ -1,134 +0,0 @@
> -#!/usr/bin/env python3
> -
> -import logging
> -import os
> -import re
> -import select
> -import shutil
> -import subprocess
> -
> -from avocado import Test
> -from avocado.utils import path
> -from avocado.utils import process
> -
> -isar_root = os.path.dirname(__file__) + '/../..'
> -backup_prefix = '.ci-backup'
> -
> -app_log = logging.getLogger("avocado.app")
> -
> -class CIBuilder(Test):
> - def setUp(self):
> - super(CIBuilder, self).setUp()
> - job_log = os.path.join(os.path.dirname(self.logdir), '..', 'job.log')
> - self._file_handler = logging.FileHandler(filename=job_log)
> - self._file_handler.setLevel(logging.ERROR)
> - fmt = ('%(asctime)s %(module)-16.16s L%(lineno)-.4d %('
> - 'levelname)-5.5s| %(message)s')
> - formatter = logging.Formatter(fmt=fmt)
> - self._file_handler.setFormatter(formatter)
> - app_log.addHandler(self._file_handler)
> -
> - def init(self, build_dir):
> - os.chdir(isar_root)
> - path.usable_rw_dir(build_dir)
> - output = process.getoutput('/bin/bash -c "source isar-init-build-env \
> - %s 2>&1 >/dev/null; env"' % build_dir)
> - env = dict(((x.split('=', 1) + [''])[:2] \
> - for x in output.splitlines() if x != ''))
> - os.environ.update(env)
> -
> - def confprepare(self, build_dir, compat_arch, cross, debsrc_cache):
> - with open(build_dir + '/conf/ci_build.conf', 'w') as f:
> - if compat_arch:
> - f.write('ISAR_ENABLE_COMPAT_ARCH_amd64 = "1"\n')
> - f.write('ISAR_ENABLE_COMPAT_ARCH_arm64 = "1"\n')
> - f.write('ISAR_ENABLE_COMPAT_ARCH_debian-stretch_amd64 = "0"\n')
> - f.write('IMAGE_INSTALL += "kselftest"\n')
> - if cross:
> - f.write('ISAR_CROSS_COMPILE = "1"\n')
> - if debsrc_cache:
> - f.write('BASE_REPO_FEATURES = "cache-deb-src"\n')
> - distro_apt_premir = os.getenv('DISTRO_APT_PREMIRRORS')
> - if distro_apt_premir:
> - f.write('DISTRO_APT_PREMIRRORS = "%s"\n' % distro_apt_premir)
> -
> - with open(build_dir + '/conf/local.conf', 'r+') as f:
> - for line in f:
> - if 'include ci_build.conf' in line:
> - break
> - else:
> - f.write('\ninclude ci_build.conf')
> -
> - def containerprep(self, build_dir):
> - with open(build_dir + '/conf/ci_build.conf', 'a') as f:
> - f.write('SDK_FORMATS = "docker-archive"\n')
> - f.write('IMAGE_INSTALL_remove = "example-module-${KERNEL_NAME} enable-fsck"\n')
> -
> - def confcleanup(self, build_dir):
> - open(build_dir + '/conf/ci_build.conf', 'w').close()
> -
> - def deletetmp(self, build_dir):
> - process.run('rm -rf ' + build_dir + '/tmp', sudo=True)
> -
> - def bitbake(self, build_dir, target, cmd, args):
> - os.chdir(build_dir)
> - cmdline = ['bitbake']
> - if args:
> - cmdline.append(args)
> - if cmd:
> - cmdline.append('-c')
> - cmdline.append(cmd)
> - if isinstance(target, list):
> - cmdline.extend(target)
> - else:
> - cmdline.append(target)
> -
> - with subprocess.Popen(" ".join(cmdline), stdout=subprocess.PIPE,
> - stderr=subprocess.PIPE, universal_newlines=True,
> - shell=True) as p1:
> - poller = select.poll()
> - poller.register(p1.stdout, select.POLLIN)
> - poller.register(p1.stderr, select.POLLIN)
> - while p1.poll() is None:
> - events = poller.poll(1000)
> - for fd, event in events:
> - if event != select.POLLIN:
> - continue
> - if fd == p1.stdout.fileno():
> - self.log.info(p1.stdout.readline().rstrip())
> - if fd == p1.stderr.fileno():
> - app_log.error(p1.stderr.readline().rstrip())
> - p1.wait()
> - if p1.returncode:
> - self.fail('Bitbake failed')
> -
> - def backupfile(self, path):
> - try:
> - shutil.copy2(path, path + backup_prefix)
> - except FileNotFoundError:
> - self.log.warn(path + ' not exist')
> -
> - def backupmove(self, path):
> - try:
> - shutil.move(path, path + backup_prefix)
> - except FileNotFoundError:
> - self.log.warn(path + ' not exist')
> -
> - def restorefile(self, path):
> - try:
> - shutil.move(path + backup_prefix, path)
> - except FileNotFoundError:
> - self.log.warn(path + backup_prefix + ' not exist')
> -
> - def getlayerdir(self, layer):
> - try:
> - path.find_command('bitbake')
> - except path.CmdNotFoundError:
> - build_dir = self.params.get('build_dir',
> - default=isar_root + '/build')
> - self.init(build_dir)
> - output = process.getoutput('bitbake -e | grep "^LAYERDIR_.*="')
> - env = dict(((x.split('=', 1) + [''])[:2] \
> - for x in output.splitlines() if x != ''))
> -
> - return env['LAYERDIR_' + layer].strip('"')
--
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 3+ messages in thread
* RE: [RFC PATCH] testsuite: refactor
2021-12-09 12:39 ` Anton Mikanovich
@ 2021-12-10 8:14 ` Schmidt, Adriaan
0 siblings, 0 replies; 3+ messages in thread
From: Schmidt, Adriaan @ 2021-12-10 8:14 UTC (permalink / raw)
To: Anton Mikanovich, isar-users
Hi Anton,
Thanks for the feedback.
Anton Mikanovich, 9. Dezember 2021 13:40:
> Test logic itself was moved to cibase to make build_test.py looks pretty
> much like the list of tests and targets.
> It makes target list maintaining much easier and avoid from searching
> though the logic if any target changes needed.
> So it's better to move existing logic from build_test.py to cibase.py,
> but not vice versa.
I get your point, although I don't fully agree.
It adds a layer of abstraction that all arguments must be passed through,
And most of the perform_* functions are only called by one test case.
But I will make another proposal, keeping the original structure.
> Moreover your patch introduce a lot of code duplication. Current
> implementation is also not ideal with it, but it's not a good idea to
> make things even worse.
The only real duplication I introduced was in the repro signed/unsigned
test, which I find hard to read (one condition "if signed", the next
"if not signed", and somehow the gpg key import happens unconditionally,
but could also be that I don't fully understand what the test wants
to do...)
> The user should be able to add any simple new test case without
> copy-pasting, but with inherit some base test classes.
>
> I also see you've reverted 94e8d23c 'CI: Fix paths for the latest
> Avocado version' which probably will not work with the latest Avocado
> versions.
That was by accident (while removing that --build switch).
Next proposal will preserve that, but rename it to --base, to make the
difference "avocado output base dir" and "bitbake build dir" clear.
Best,
Adriaan
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-12-10 8:14 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-12-09 10:29 [RFC PATCH] testsuite: refactor Adriaan Schmidt
2021-12-09 12:39 ` Anton Mikanovich
2021-12-10 8:14 ` Schmidt, Adriaan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox