[PATCH v3 00/15] Add developers test

[PATCH v3 00/15] Add developers test

[Uladzimir Bely]

Add separate quick 'developers' test for fast checking new patches.
It will test only general Isar logic on basic targets without dealing with
all the functionality.
To execute dev test run avocado testsuite like:

$ avocado run ../testsuite/citest.py -t dev --nrunner-max-parallel-tasks=1

Estimated execution time for dev test is ~1.5h.

Changes since v2:
- Rebase on next (with bitbake 2.0).
- Don't fail on deprecated params, show warning if used.

Changes sinse v1:
- Rebase on next.
- Integrate some changes from other patchsets.
- Remove ci_build.sh params.
- Improve testsuite API.
- Improve testsuite readme.
- Add single target test case.

Anton Mikanovich (10):
  CI: Enable downloads dir sharing
  CI: Add IMAGE_INSTALL override
  CI: Add developers test
  CI: Remove duplicated targets
  CI: Get rid of setting cross from outside
  CI: Make cross compile to be default
  CI: Get rid of setting repro from outside
  CI: Allow example module checking disabling
  CI: Update testsuite documentation
  CI: Add single target test case

Henning Schild (4):
  CI: move to avocado to 99.0
  CI: fix shell coding style
  CI: install qemu-system when qemu testing is requested
  testsuite: remove tests from "fast" set

Uladzimir Bely (1):
  testsuite: Fix terminal broken after run test executed

 .gitlab-ci.yml         |   4 +-
 scripts/ci_build.sh    |  50 ++++++---------
 testsuite/README.md    |  26 ++++++--
 testsuite/cibuilder.py |  27 +++++---
 testsuite/citest.py    | 141 ++++++++++++++++++++++++-----------------
 5 files changed, 145 insertions(+), 103 deletions(-)

-- 
2.20.1

[PATCH v3 01/15] CI: Enable downloads dir sharing

[Uladzimir Bely]

From: Anton Mikanovich <amikan@ilbers.de>

Test cases performed in different build dir now can share DL_DIR.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/cibuilder.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 9d1b80f7..8178ea28 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -54,7 +54,8 @@ class CIBuilder(Test):
 
     def configure(self, compat_arch=True, cross=None, debsrc_cache=False,
                   container=False, ccache=False, sstate=False, offline=False,
-                  gpg_pub_key=None, wic_deploy_parts=False, source_date_epoch=None, **kwargs):
+                  gpg_pub_key=None, wic_deploy_parts=False, dl_dir=None,
+                  source_date_epoch=None, **kwargs):
         # write configuration file and set bitbake_args
         # can run multiple times per test case
         self.check_init()
@@ -64,6 +65,9 @@ class CIBuilder(Test):
         if cross is None:
             cross = bool(int(self.params.get('cross', default=0)))
 
+        if dl_dir is None:
+            dl_dir = os.path.join(isar_root, 'downloads')
+
         # get parameters from environment
         distro_apt_premir = os.getenv('DISTRO_APT_PREMIRRORS')
 
@@ -79,6 +83,7 @@ class CIBuilder(Test):
                       f'  gpg_pub_key = {gpg_pub_key}\n'
                       f'  wic_deploy_parts = {wic_deploy_parts}\n'
                       f'  source_date_epoch = {source_date_epoch} \n'
+                      f'  dl_dir = {dl_dir}\n'
                       f'===================================================')
 
         # determine bitbake_args
@@ -115,6 +120,8 @@ class CIBuilder(Test):
                 f.write('CCACHE_TOP_DIR = "${TOPDIR}/ccache"\n')
             if source_date_epoch:
                 f.write('SOURCE_DATE_EPOCH = "%s"\n' % source_date_epoch)
+            if dl_dir:
+                f.write('DL_DIR = "%s"\n' % dl_dir)
 
         # include ci_build.conf in local.conf
         with open(self.build_dir + '/conf/local.conf', 'r+') as f:
-- 
2.20.1

[PATCH v3 02/15] CI: Add IMAGE_INSTALL override

[Uladzimir Bely]

From: Anton Mikanovich <amikan@ilbers.de>

Allow user to override the list of packages to be built.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/cibuilder.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 8178ea28..27909d29 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -55,7 +55,7 @@ class CIBuilder(Test):
     def configure(self, compat_arch=True, cross=None, debsrc_cache=False,
                   container=False, ccache=False, sstate=False, offline=False,
                   gpg_pub_key=None, wic_deploy_parts=False, dl_dir=None,
-                  source_date_epoch=None, **kwargs):
+                  source_date_epoch=None, image_install=None, **kwargs):
         # write configuration file and set bitbake_args
         # can run multiple times per test case
         self.check_init()
@@ -84,6 +84,7 @@ class CIBuilder(Test):
                       f'  wic_deploy_parts = {wic_deploy_parts}\n'
                       f'  source_date_epoch = {source_date_epoch} \n'
                       f'  dl_dir = {dl_dir}\n'
+                      f'  image_install = {image_install}\n'
                       f'===================================================')
 
         # determine bitbake_args
@@ -122,6 +123,8 @@ class CIBuilder(Test):
                 f.write('SOURCE_DATE_EPOCH = "%s"\n' % source_date_epoch)
             if dl_dir:
                 f.write('DL_DIR = "%s"\n' % dl_dir)
+            if image_install is not None:
+                f.write('IMAGE_INSTALL = "%s"' % image_install)
 
         # include ci_build.conf in local.conf
         with open(self.build_dir + '/conf/local.conf', 'r+') as f:
-- 
2.20.1

[PATCH v3 03/15] CI: Add developers test

[Uladzimir Bely]

From: Anton Mikanovich <amikan@ilbers.de>

Developers test consist of minimal test cases for patches checking.
To ensure dev test is 'always green' include it into full CI.

Dev test can be executed using 'dev' avocado tag.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/citest.py | 54 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 2b6c0b29..027b965b 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -17,6 +17,60 @@ try:
 except path.CmdNotFoundError:
     SKOPEO_AVAILABLE = False
 
+class DevTest(CIBaseTest):
+
+    """
+    Developer's test
+
+    :avocado: tags=dev,fast,full
+    """
+    def test_dev(self):
+        targets = [
+            'mc:qemuamd64-bullseye:isar-image-base',
+            'mc:qemuarm-bullseye:isar-image-base',
+            'mc:qemuarm-bullseye:isar-image-base:do_populate_sdk',
+            'mc:qemuarm64-bullseye:isar-image-base'
+                  ]
+
+        self.init()
+        self.perform_build_test(targets, cross=True, image_install="example-raw")
+
+    def test_dev_apps(self):
+        targets = [
+            'mc:qemuamd64-bullseye:isar-image-base',
+            'mc:qemuarm64-bullseye:isar-image-base'
+                  ]
+
+        self.init()
+        self.perform_build_test(targets)
+
+    def test_dev_rebuild(self):
+        self.init()
+        layerdir_core = self.getlayerdir('core')
+
+        dpkgbase_file = layerdir_core + '/classes/dpkg-base.bbclass'
+
+        self.backupfile(dpkgbase_file)
+        with open(dpkgbase_file, 'a') as file:
+            file.write('do_fetch:append() {\n\n}')
+
+        try:
+            self.perform_build_test('mc:qemuamd64-bullseye:isar-image-base')
+        finally:
+            self.restorefile(dpkgbase_file)
+
+    def test_dev_run_amd64_bullseye(self):
+        self.init()
+        self.vm_start('amd64', 'bullseye')
+
+    def test_dev_run_arm64_bullseye(self):
+        self.init()
+        self.vm_start('arm64', 'bullseye')
+
+    def test_dev_run_arm_bullseye(self):
+        self.init()
+        self.vm_start('arm', 'bullseye')
+
 class ReproTest(CIBaseTest):
 
     """
-- 
2.20.1

[PATCH v3 04/15] CI: Remove duplicated targets

[Uladzimir Bely]

From: Anton Mikanovich <amikan@ilbers.de>

Some targets are already covered by dev tests, so remove retesting.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/citest.py | 49 ---------------------------------------------
 1 file changed, 49 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 027b965b..e43004d5 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -125,7 +125,6 @@ class CrossTest(CIBaseTest):
         targets = [
             'mc:qemuarm-buster:isar-image-base',
             'mc:qemuarm-bullseye:isar-image-base',
-            'mc:qemuarm64-bullseye:isar-image-base',
             'mc:de0-nano-soc-bullseye:isar-image-base',
             'mc:stm32mp15x-buster:isar-image-base'
                   ]
@@ -167,19 +166,6 @@ class CrossTest(CIBaseTest):
         except:
             self.cancel('KFAIL')
 
-class SdkTest(CIBaseTest):
-
-    """
-    In addition test SDK creation
-
-    :avocado: tags=sdk,fast,full
-    """
-    def test_sdk(self):
-        targets = ['mc:qemuarm-bullseye:isar-image-base']
-
-        self.init()
-        self.perform_build_test(targets, bitbake_cmd='do_populate_sdk')
-
 class WicTest(CIBaseTest):
 
     """
@@ -218,7 +204,6 @@ class NoCrossTest(CIBaseTest):
             'mc:qemui386-buster:isar-image-base',
             'mc:qemui386-bullseye:isar-image-base',
             'mc:qemuamd64-buster:isar-image-base',
-            'mc:qemuamd64-bullseye:isar-image-base',
             'mc:qemuamd64-bullseye:isar-initramfs',
             'mc:qemumipsel-buster:isar-image-base',
             'mc:qemumipsel-bullseye:isar-image-base',
@@ -278,28 +263,6 @@ class NoCrossTest(CIBaseTest):
         except:
             self.cancel('KFAIL')
 
-class RebuildTest(CIBaseTest):
-
-    """
-    Test image rebuild
-
-    :avocado: tags=rebuild,fast,full
-    """
-    def test_rebuild(self):
-        self.init()
-        layerdir_core = self.getlayerdir('core')
-
-        dpkgbase_file = layerdir_core + '/classes/dpkg-base.bbclass'
-
-        self.backupfile(dpkgbase_file)
-        with open(dpkgbase_file, 'a') as file:
-            file.write('do_fetch:append() {\n\n}')
-
-        try:
-            self.perform_build_test('mc:qemuamd64-bullseye:isar-image-base', debsrc_cache=True)
-        finally:
-            self.restorefile(dpkgbase_file)
-
 class ContainerImageTest(CIBaseTest):
 
     """
@@ -361,14 +324,6 @@ class VmBootTestFast(CIBaseTest):
         self.init()
         self.vm_start('arm','buster')
 
-    def test_arm64_bullseye(self):
-        self.init()
-        self.vm_start('arm64','bullseye')
-
-    def test_amd64_bullseye(self):
-        self.init()
-        self.vm_start('amd64','bullseye')
-
     def test_arm_bookworm(self):
         self.init()
         self.vm_start('arm','bookworm')
@@ -392,10 +347,6 @@ class VmBootTestFull(CIBaseTest):
         self.init()
         self.vm_start('arm64','bullseye')
 
-    def test_amd64_bullseye(self):
-        self.init()
-        self.vm_start('amd64','bullseye')
-
     def test_i386_buster(self):
         self.init()
         self.vm_start('i386','buster')
-- 
2.20.1

[PATCH v3 05/15] CI: move to avocado to 99.0

[Uladzimir Bely]

From: Henning Schild <henning.schild@siemens.com>

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 scripts/ci_build.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index fa7208c2..84a3e472 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -21,7 +21,7 @@ if ! command -v avocado > /dev/null; then
     rm -rf /tmp/avocado_venv
     virtualenv --python python3 /tmp/avocado_venv
     source /tmp/avocado_venv/bin/activate
-    pip install avocado-framework==96.0
+    pip install avocado-framework==99.0
 fi
 
 # Get Avocado build tests path
-- 
2.20.1

[PATCH v3 06/15] CI: fix shell coding style

[Uladzimir Bely]

From: Henning Schild <henning.schild@siemens.com>

Apply some suggestions from shellcheck and make sure that runs fine.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 scripts/ci_build.sh | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 84a3e472..e5f20d4f 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -20,6 +20,7 @@ if ! command -v avocado > /dev/null; then
     sudo apt-get install -y virtualenv
     rm -rf /tmp/avocado_venv
     virtualenv --python python3 /tmp/avocado_venv
+    # shellcheck source=/dev/null
     source /tmp/avocado_venv/bin/activate
     pip install avocado-framework==99.0
 fi
@@ -33,8 +34,8 @@ BASE_DIR=./build
 # Check dependencies
 DEPENDENCIES="umoci skopeo"
 for prog in ${DEPENDENCIES} ; do
-    if [ ! -x "$(which $prog)" ] ; then
-        echo "missing $prog in PATH" >&2
+    if ! command -v "${prog}" > /dev/null; then
+        echo "missing ${prog} in PATH" >&2
     fi
 done
 
@@ -132,16 +133,16 @@ fi
 mkdir -p .config/avocado
 cat <<EOF > .config/avocado/avocado.conf
 [datadir.paths]
-base_dir = $(realpath $BASE_DIR)/
-test_dir = $(realpath $BASE_DIR)/tests
-data_dir = $(realpath $BASE_DIR)/data
-logs_dir = $(realpath $BASE_DIR)/job-results
+base_dir = $(realpath "${BASE_DIR}")/
+test_dir = $(realpath "${BASE_DIR}")/tests
+data_dir = $(realpath "${BASE_DIR}")/data
+logs_dir = $(realpath "${BASE_DIR}")/job-results
 EOF
 export VIRTUAL_ENV="./"
 
 # the real stuff starts here, trace commands from now on
 set -x
 
-avocado $VERBOSE run "$TESTSUITE_DIR/citest.py" \
-    -t $TAGS --nrunner-max-parallel-tasks=1 --disable-sysinfo \
-    -p quiet=$QUIET -p cross=$CROSS_BUILD -p time_to_wait=$TIMEOUT
+avocado ${VERBOSE} run "${TESTSUITE_DIR}/citest.py" \
+    -t "${TAGS}" --nrunner-max-parallel-tasks=1 --disable-sysinfo \
+    -p quiet="${QUIET}" -p cross="${CROSS_BUILD}" -p time_to_wait="${TIMEOUT}"
-- 
2.20.1

[PATCH v3 07/15] CI: install qemu-system when qemu testing is requested

[Uladzimir Bely]

From: Henning Schild <henning.schild@siemens.com>

And also change gitlab ci config to request qemu testing.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 .gitlab-ci.yml      | 4 ++--
 scripts/ci_build.sh | 5 +++++
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index fcdefa87..6a49ce43 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -27,7 +27,7 @@ fast-ci:
   except:
     - schedules
   script:
-    - scripts/ci_build.sh -q -f -n
+    - scripts/ci_build.sh -q -f
 
 full-ci:
   <<: *common-build
@@ -37,7 +37,7 @@ full-ci:
     - PREVIOUS_SHA="$(cat .CI_COMMIT_SHA || true)"
     - if [ "$CI_COMMIT_SHA" != "$PREVIOUS_SHA" ]; then
           echo "$CI_COMMIT_SHA" > .CI_COMMIT_SHA;
-          scripts/ci_build.sh -q -c -r -n;
+          scripts/ci_build.sh -q -c -r;
       fi
   cache:
     key: "$CI_COMMIT_REF_SLUG"
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index e5f20d4f..960ee94e 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -127,6 +127,11 @@ fi
 
 if [ -n "$NORUN" ]; then
     TAGS="$TAGS,-startvm"
+else
+    if ! command -v qemu-system-x86_64 > /dev/null; then
+        sudo apt-get update -qq
+        sudo apt-get install -y qemu-system
+    fi
 fi
 
 # Provide working path
-- 
2.20.1

[PATCH v3 08/15] testsuite: remove tests from "fast" set

[Uladzimir Bely]

From: Henning Schild <henning.schild@siemens.com>

Ccache and Sstate are test infrastructure bits and should not be done
when we try to be "fast".

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/citest.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index e43004d5..0485dd7b 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -107,7 +107,7 @@ class CcacheTest(CIBaseTest):
     """
     Test rebuild speed improve with ccache
 
-    :avocado: tags=ccache,fast,full
+    :avocado: tags=ccache,full
     """
     def test_ccache_rebuild(self):
         targets = ['mc:qemuamd64-bullseye:hello-isar']
@@ -300,7 +300,7 @@ class SstateTest(CIBaseTest):
     """
     Test builds with artifacts taken from sstate cache
 
-    :avocado: tags=sstate,fast,full
+    :avocado: tags=sstate,full
     """
     def test_sstate(self):
         image_target = 'mc:qemuamd64-bullseye:isar-image-base'
-- 
2.20.1

[PATCH v3 09/15] CI: Get rid of setting cross from outside

[Uladzimir Bely]

From: Anton Mikanovich <amikan@ilbers.de>

There is no need in setting cross compilation mode from outside
because it was already done inside test cases.
This also makes test cases more unified.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 .gitlab-ci.yml         |  2 +-
 scripts/ci_build.sh    | 13 +++++--------
 testsuite/cibuilder.py |  4 +---
 3 files changed, 7 insertions(+), 12 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 6a49ce43..d1d22a6f 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -37,7 +37,7 @@ full-ci:
     - PREVIOUS_SHA="$(cat .CI_COMMIT_SHA || true)"
     - if [ "$CI_COMMIT_SHA" != "$PREVIOUS_SHA" ]; then
           echo "$CI_COMMIT_SHA" > .CI_COMMIT_SHA;
-          scripts/ci_build.sh -q -c -r;
+          scripts/ci_build.sh -q -r;
       fi
   cache:
     key: "$CI_COMMIT_REF_SLUG"
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 960ee94e..679e3f37 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -48,7 +48,6 @@ show_help() {
     echo "Parameters:"
     echo "    -b, --base BASE_DIR      set path to base directory. If not set,"
     echo "                             the tests will be started in current path."
-    echo "    -c, --cross              enable cross-compilation."
     echo "    -d, --debug              enable debug bitbake output."
     echo "    -f, --fast               cross build reduced set of configurations."
     echo "    -q, --quiet              suppress verbose bitbake output."
@@ -64,7 +63,6 @@ show_help() {
 }
 
 TAGS="full"
-CROSS_BUILD="0"
 QUIET="0"
 TIMEOUT=300
 
@@ -82,17 +80,12 @@ do
         BASE_DIR="$2"
         shift
         ;;
-    -c|--cross)
-        CROSS_BUILD="1"
-        ;;
     -d|--debug)
         VERBOSE="--show=app,test"
         ;;
     -f|--fast)
         # Start build for the reduced set of configurations
-        # Enforce cross-compilation to speed up the build
         TAGS="fast"
-        CROSS_BUILD="1"
         ;;
     -q|--quiet)
         QUIET="1"
@@ -112,6 +105,10 @@ do
         TIMEOUT=$2
         shift
         ;;
+    -c|--cross)
+        # Just not to cause CI failures on legacy configs
+        echo "warning: deprecated parameter '$key'"
+        ;;
     *)
         echo "error: invalid parameter '$key', please try '--help' to get list of supported parameters"
         exit $ES_BUG
@@ -150,4 +147,4 @@ set -x
 
 avocado ${VERBOSE} run "${TESTSUITE_DIR}/citest.py" \
     -t "${TAGS}" --nrunner-max-parallel-tasks=1 --disable-sysinfo \
-    -p quiet="${QUIET}" -p cross="${CROSS_BUILD}" -p time_to_wait="${TIMEOUT}"
+    -p quiet="${QUIET}" -p time_to_wait="${TIMEOUT}"
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 27909d29..a9d50d42 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -52,7 +52,7 @@ class CIBuilder(Test):
         if not hasattr(self, 'build_dir'):
             self.error("Broken test implementation: need to call init().")
 
-    def configure(self, compat_arch=True, cross=None, debsrc_cache=False,
+    def configure(self, compat_arch=True, cross=False, debsrc_cache=False,
                   container=False, ccache=False, sstate=False, offline=False,
                   gpg_pub_key=None, wic_deploy_parts=False, dl_dir=None,
                   source_date_epoch=None, image_install=None, **kwargs):
@@ -62,8 +62,6 @@ class CIBuilder(Test):
 
         # get parameters from avocado cmdline
         quiet = bool(int(self.params.get('quiet', default=0)))
-        if cross is None:
-            cross = bool(int(self.params.get('cross', default=0)))
 
         if dl_dir is None:
             dl_dir = os.path.join(isar_root, 'downloads')
-- 
2.20.1

[PATCH v3 10/15] CI: Make cross compile to be default

[Uladzimir Bely]

From: Anton Mikanovich <amikan@ilbers.de>

Cross compile is stable enough to be enabled by default.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/cibuilder.py |  2 +-
 testsuite/citest.py    | 14 +++++++-------
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index a9d50d42..18ca4919 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -52,7 +52,7 @@ class CIBuilder(Test):
         if not hasattr(self, 'build_dir'):
             self.error("Broken test implementation: need to call init().")
 
-    def configure(self, compat_arch=True, cross=False, debsrc_cache=False,
+    def configure(self, compat_arch=True, cross=True, debsrc_cache=False,
                   container=False, ccache=False, sstate=False, offline=False,
                   gpg_pub_key=None, wic_deploy_parts=False, dl_dir=None,
                   source_date_epoch=None, image_install=None, **kwargs):
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 0485dd7b..f5cead59 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -33,7 +33,7 @@ class DevTest(CIBaseTest):
                   ]
 
         self.init()
-        self.perform_build_test(targets, cross=True, image_install="example-raw")
+        self.perform_build_test(targets, image_install="example-raw")
 
     def test_dev_apps(self):
         targets = [
@@ -86,7 +86,7 @@ class ReproTest(CIBaseTest):
 
         self.init()
         try:
-            self.perform_repro_test(targets, signed=True, cross=True)
+            self.perform_repro_test(targets, signed=True)
         finally:
             self.move_in_build_dir('tmp', 'tmp_repro_signed')
 
@@ -98,7 +98,7 @@ class ReproTest(CIBaseTest):
 
         self.init()
         try:
-            self.perform_repro_test(targets)
+            self.perform_repro_test(targets, cross=False)
         finally:
             self.move_in_build_dir('tmp', 'tmp_repro_unsigned')
 
@@ -130,7 +130,7 @@ class CrossTest(CIBaseTest):
                   ]
 
         self.init()
-        self.perform_build_test(targets, cross=True, debsrc_cache=True)
+        self.perform_build_test(targets, debsrc_cache=True)
 
     def test_cross_rpi(self):
         targets = [
@@ -139,7 +139,7 @@ class CrossTest(CIBaseTest):
 
         self.init()
         try:
-            self.perform_build_test(targets, cross=True, debsrc_cache=True)
+            self.perform_build_test(targets, debsrc_cache=True)
         except:
             self.cancel('KFAIL')
 
@@ -150,7 +150,7 @@ class CrossTest(CIBaseTest):
 
         self.init()
         try:
-            self.perform_build_test(targets, cross=True)
+            self.perform_build_test(targets)
         except:
             self.cancel('KFAIL')
 
@@ -162,7 +162,7 @@ class CrossTest(CIBaseTest):
 
         self.init()
         try:
-            self.perform_build_test(targets, cross=True)
+            self.perform_build_test(targets)
         except:
             self.cancel('KFAIL')
 
-- 
2.20.1

[PATCH v3 11/15] CI: Get rid of setting repro from outside

[Uladzimir Bely]

From: Anton Mikanovich <amikan@ilbers.de>

To make test cases more unified do not allow to change repro test
execution. This test case will always run for "full" now.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 .gitlab-ci.yml      |  2 +-
 scripts/ci_build.sh | 15 +--------------
 2 files changed, 2 insertions(+), 15 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index d1d22a6f..d1e82492 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -37,7 +37,7 @@ full-ci:
     - PREVIOUS_SHA="$(cat .CI_COMMIT_SHA || true)"
     - if [ "$CI_COMMIT_SHA" != "$PREVIOUS_SHA" ]; then
           echo "$CI_COMMIT_SHA" > .CI_COMMIT_SHA;
-          scripts/ci_build.sh -q -r;
+          scripts/ci_build.sh -q;
       fi
   cache:
     key: "$CI_COMMIT_REF_SLUG"
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 679e3f37..3a20515b 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -51,7 +51,6 @@ show_help() {
     echo "    -d, --debug              enable debug bitbake output."
     echo "    -f, --fast               cross build reduced set of configurations."
     echo "    -q, --quiet              suppress verbose bitbake output."
-    echo "    -r, --repro              enable use of cached base repository."
     echo "    -n, --norun              do not execute QEMU run tests."
     echo "    -t, --timeout SEC        specify time in seconds to wait before stop QEMU."
     echo "                             The default is: 300"
@@ -90,14 +89,6 @@ do
     -q|--quiet)
         QUIET="1"
         ;;
-    -r|--repro)
-        REPRO_BUILD="1"
-        # This switch is deprecated, just here to not cause failing CI on
-        # legacy configs
-        case "$2" in
-        -s|--sign) shift ;;
-        esac
-        ;;
     -n|--norun)
         NORUN="1"
         ;;
@@ -105,7 +96,7 @@ do
         TIMEOUT=$2
         shift
         ;;
-    -c|--cross)
+    -c|--cross|-r|--repro|-s|--sign)
         # Just not to cause CI failures on legacy configs
         echo "warning: deprecated parameter '$key'"
         ;;
@@ -118,10 +109,6 @@ do
     shift
 done
 
-if [ -z "$REPRO_BUILD" ]; then
-    TAGS="$TAGS,-repro"
-fi
-
 if [ -n "$NORUN" ]; then
     TAGS="$TAGS,-startvm"
 else
-- 
2.20.1

[PATCH v3 12/15] CI: Allow example module checking disabling

[Uladzimir Bely]

From: Anton Mikanovich <amikan@ilbers.de>

If the target do not have example-module package compiled it can fail
on startvm test currently. Provide internal testsuite API for skipping
this test.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/cibuilder.py | 6 ++++--
 testsuite/citest.py    | 2 +-
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 18ca4919..c6a1eb67 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -209,7 +209,8 @@ class CIBuilder(Test):
 
         return env['LAYERDIR_' + layer].strip('"')
 
-    def vm_start(self, arch='amd64', distro='buster', enforce_pcbios=False):
+    def vm_start(self, arch='amd64', distro='buster', enforce_pcbios=False,
+                 skip_modulecheck=False):
         time_to_wait = self.params.get('time_to_wait', default=60)
 
         self.log.info('===================================================')
@@ -291,7 +292,8 @@ class CIBuilder(Test):
         if os.path.exists(output_file) and os.path.getsize(output_file) > 0:
             with open(output_file, "rb") as f1:
                 data = f1.read()
-                if module_output in data and login_prompt in data:
+                if (module_output in data or skip_modulecheck) \
+                   and login_prompt in data:
                     if resize_output:
                         if resize_output in data:
                             return
diff --git a/testsuite/citest.py b/testsuite/citest.py
index f5cead59..cdbede05 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -69,7 +69,7 @@ class DevTest(CIBaseTest):
 
     def test_dev_run_arm_bullseye(self):
         self.init()
-        self.vm_start('arm', 'bullseye')
+        self.vm_start('arm', 'bullseye', skip_modulecheck=True)
 
 class ReproTest(CIBaseTest):
 
-- 
2.20.1

[PATCH v3 13/15] testsuite: Fix terminal broken after run test executed

[Uladzimir Bely]

If some of run tests executed from interactive terminal by hand,
user input goes broken after VM powered off. Commands user enters
become invisible and terminal can be fixed only by executing
either 'reset' or 'stty echo' commands.

Fix this by specifying 'stdin' value for qemu subprocess.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/cibuilder.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index c6a1eb67..72522f4d 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -262,8 +262,9 @@ class CIBuilder(Test):
 
         timeout = time.time() + int(time_to_wait)
 
-        p1 = subprocess.Popen(cmdline, stdout=subprocess.PIPE,
-                              stderr=subprocess.PIPE, universal_newlines=True)
+        p1 = subprocess.Popen(cmdline,
+                              stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+                              universal_newlines=True)
         try:
             poller = select.poll()
             poller.register(p1.stdout, select.POLLIN)
-- 
2.20.1

[PATCH v3 14/15] CI: Update testsuite documentation

[Uladzimir Bely]

From: Anton Mikanovich <amikan@ilbers.de>

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/README.md | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/testsuite/README.md b/testsuite/README.md
index b83a8f98..90ba5c3d 100644
--- a/testsuite/README.md
+++ b/testsuite/README.md
@@ -4,34 +4,42 @@ The framework could be installed by using standard HOWTO:
 
   https://github.com/avocado-framework/avocado#installing-with-standard-python-tools
 
-## For Debian (tested on Debian 10.x)
+## For Debian (tested on Debian 11.x)
 
-        $ sudo dpkg -i avocado_91.0_all.deb
+```
+$ pip install avocado-framework==99.0
+```
 
 # Run test
 
+## Quick developers test
+
+```
+$ avocado run ../testsuite/citest.py -t dev --nrunner-max-parallel-tasks=1
+```
+
 ## Fast build test
 
 ```
-$ avocado run build_test.py -t fast -p quiet=1 -p cross=1
+$ avocado run ../testsuite/citest.py -t fast --nrunner-max-parallel-tasks=1 -p quiet=1
 ```
 
 ## Full build test
 
 ```
-$ avocado run build_test.py -t full -p quiet=1
+$ avocado run ../testsuite/citest.py -t full --nrunner-max-parallel-tasks=1 -p quiet=1
 ```
 
 ## Fast boot test
 
 ```
-$ avocado run vm_boot_test.py -t fast -p build_dir="$BUILDDIR" -p time_to_wait=300
+$ avocado run ../testsuite/citest.py -t startvm,fast -p time_to_wait=300
 ```
 
 ## Full boot test
 
 ```
-$ avocado run vm_boot_test.py -t full -p build_dir="$BUILDDIR" -p time_to_wait=300
+$ avocado run ../testsuite/citest.py -t startvm,full -p time_to_wait=300
 ```
 
 # Other
-- 
2.20.1

[PATCH v3 15/15] CI: Add single target test case

[Uladzimir Bely]

From: Anton Mikanovich <amikan@ilbers.de>

Add special test case for single target build and run testing.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/README.md |  6 ++++++
 testsuite/citest.py | 22 ++++++++++++++++++++++
 2 files changed, 28 insertions(+)

diff --git a/testsuite/README.md b/testsuite/README.md
index 90ba5c3d..4f2a8da8 100644
--- a/testsuite/README.md
+++ b/testsuite/README.md
@@ -18,6 +18,12 @@ $ pip install avocado-framework==99.0
 $ avocado run ../testsuite/citest.py -t dev --nrunner-max-parallel-tasks=1
 ```
 
+## Single target test
+
+```
+$ avocado run ../testsuite/citest.py -t single --nrunner-max-parallel-tasks=1 -p machine=qemuamd64 -p distro=bullseye
+```
+
 ## Fast build test
 
 ```
diff --git a/testsuite/citest.py b/testsuite/citest.py
index cdbede05..975cb064 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -309,6 +309,28 @@ class SstateTest(CIBaseTest):
         self.init('build-sstate')
         self.perform_sstate_test(image_target, package_target)
 
+class SingleTest(CIBaseTest):
+
+    """
+    Single test for selected target
+
+    :avocado: tags=single
+    """
+    def test_single_build(self):
+        self.init()
+        machine = self.params.get('machine', default='qemuamd64')
+        distro = self.params.get('distro', default='bullseye')
+        image = self.params.get('image', default='isar-image-base')
+
+        self.perform_build_test('mc:%s-%s:%s' % (machine, distro, image))
+
+    def test_single_run(self):
+        self.init()
+        machine = self.params.get('machine', default='qemuamd64')
+        distro = self.params.get('distro', default='bullseye')
+
+        self.vm_start(machine.removeprefix('qemu'), distro)
+
 class VmBootTestFast(CIBaseTest):
 
     """
-- 
2.20.1

Re: [PATCH v3 07/15] CI: install qemu-system when qemu testing is requested

[Uladzimir Bely]

In mail from Friday, 3 February 2023 07:44:18 +03 user Uladzimir Bely wrote:
> From: Henning Schild <henning.schild@siemens.com>
> 
> And also change gitlab ci config to request qemu testing.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>  .gitlab-ci.yml      | 4 ++--
>  scripts/ci_build.sh | 5 +++++
>  2 files changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index fcdefa87..6a49ce43 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -27,7 +27,7 @@ fast-ci:
>    except:
>      - schedules
>    script:
> -    - scripts/ci_build.sh -q -f -n
> +    - scripts/ci_build.sh -q -f
> 
>  full-ci:
>    <<: *common-build
> @@ -37,7 +37,7 @@ full-ci:
>      - PREVIOUS_SHA="$(cat .CI_COMMIT_SHA || true)"
>      - if [ "$CI_COMMIT_SHA" != "$PREVIOUS_SHA" ]; then
>            echo "$CI_COMMIT_SHA" > .CI_COMMIT_SHA;
> -          scripts/ci_build.sh -q -c -r -n;
> +          scripts/ci_build.sh -q -c -r;
>        fi
>    cache:
>      key: "$CI_COMMIT_REF_SLUG"
> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> index e5f20d4f..960ee94e 100755
> --- a/scripts/ci_build.sh
> +++ b/scripts/ci_build.sh
> @@ -127,6 +127,11 @@ fi
> 
>  if [ -n "$NORUN" ]; then
>      TAGS="$TAGS,-startvm"
> +else
> +    if ! command -v qemu-system-x86_64 > /dev/null; then

For now, this check is not changed to testing '/usr/share/doc/qemu-system/
copyright' or something more general.

Additionally, it might be good to use "--no-install-recommends" to install 
less dependencies. For example, under kas docker image:

```
sudo apt-get install qemu-system
...
2 upgraded, 232 newly installed, 0 to remove and 78 not upgraded.
Need to get 228 MB of archives.
After this operation, 1,084 MB of additional disk space will be used.
```
vs

```
sudo apt-get install --no-install-recommends qemu-system ipxe-qemu
...
0 upgraded, 56 newly installed, 0 to remove and 80 not upgraded.
Need to get 92.1 MB of archives.
After this operation, 423 MB of additional disk space will be used.
```

Package `ipxe-qemu` is definitely required for network working. But for now 
I'm not sure if we don't need any more packages here, this is to be checked 
under Gitlab CI.

> +        sudo apt-get update -qq
> +        sudo apt-get install -y qemu-system
> +    fi
>  fi
> 
>  # Provide working path

Re: [PATCH v3 07/15] CI: install qemu-system when qemu testing is requested

[Henning Schild]

Am Fri, 03 Feb 2023 11:33:38 +0300
schrieb Uladzimir Bely <ubely@ilbers.de>:

> In mail from Friday, 3 February 2023 07:44:18 +03 user Uladzimir Bely
> wrote:
> > From: Henning Schild <henning.schild@siemens.com>
> > 
> > And also change gitlab ci config to request qemu testing.
> > 
> > Signed-off-by: Henning Schild <henning.schild@siemens.com>
> > Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> > ---
> >  .gitlab-ci.yml      | 4 ++--
> >  scripts/ci_build.sh | 5 +++++
> >  2 files changed, 7 insertions(+), 2 deletions(-)
> > 
> > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > index fcdefa87..6a49ce43 100644
> > --- a/.gitlab-ci.yml
> > +++ b/.gitlab-ci.yml
> > @@ -27,7 +27,7 @@ fast-ci:
> >    except:
> >      - schedules
> >    script:
> > -    - scripts/ci_build.sh -q -f -n
> > +    - scripts/ci_build.sh -q -f
> > 
> >  full-ci:
> >    <<: *common-build
> > @@ -37,7 +37,7 @@ full-ci:
> >      - PREVIOUS_SHA="$(cat .CI_COMMIT_SHA || true)"
> >      - if [ "$CI_COMMIT_SHA" != "$PREVIOUS_SHA" ]; then
> >            echo "$CI_COMMIT_SHA" > .CI_COMMIT_SHA;
> > -          scripts/ci_build.sh -q -c -r -n;
> > +          scripts/ci_build.sh -q -c -r;
> >        fi
> >    cache:
> >      key: "$CI_COMMIT_REF_SLUG"
> > diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> > index e5f20d4f..960ee94e 100755
> > --- a/scripts/ci_build.sh
> > +++ b/scripts/ci_build.sh
> > @@ -127,6 +127,11 @@ fi
> > 
> >  if [ -n "$NORUN" ]; then
> >      TAGS="$TAGS,-startvm"
> > +else
> > +    if ! command -v qemu-system-x86_64 > /dev/null; then  
> 
> For now, this check is not changed to testing
> '/usr/share/doc/qemu-system/ copyright' or something more general.
> 
> Additionally, it might be good to use "--no-install-recommends" to
> install less dependencies. For example, under kas docker image:
> 
> ```
> sudo apt-get install qemu-system
> ...
> 2 upgraded, 232 newly installed, 0 to remove and 78 not upgraded.
> Need to get 228 MB of archives.
> After this operation, 1,084 MB of additional disk space will be used.
> ```
> vs
> 
> ```
> sudo apt-get install --no-install-recommends qemu-system ipxe-qemu
> ...
> 0 upgraded, 56 newly installed, 0 to remove and 80 not upgraded.
> Need to get 92.1 MB of archives.
> After this operation, 423 MB of additional disk space will be used.
> ```
> 
> Package `ipxe-qemu` is definitely required for network working. But
> for now I'm not sure if we don't need any more packages here, this is
> to be checked under Gitlab CI.

Thanks for the update. I propose you apply those changes, since you
took those patches into your series. And i will try it on gitlab and
see which packages might end up missing.

For a super quick test it would be handy to have a branch to pull,
instead of an mbox workflow. But i can do both, especially not with
patchwork it is actually as quick as branch.

Henning

> > +        sudo apt-get update -qq
> > +        sudo apt-get install -y qemu-system
> > +    fi
> >  fi
> > 
> >  # Provide working path  
> 
> 
> 
>