[RFC][PATCH v3 1/1] test: provide ci_shell as a convenience wrapper

["'Felix Moessbauer' via isar-users" <isar-users@googlegroups.com>]

From: Cedric Hombourger' via isar-users <isar-users@googlegroups.com>

Ease use of the container environment that may be used to run
tests from this project by adding a `scripts/ci_shell` script
to invoke `kas-container` for you. The script uses the existing
kas/isar.yml kas file to use your existing checkout of the Isar
code: this makes it easier to test your work-in-progress changes.

Lastly, `ci_setup.sh` will produce less output (e.g. when
installing dependencies) and will print a few hints to remind
users with useful commands.

[Felix]
- fix pyhashtables cleanup
- speedup installation by skipping recommends
- add more helper commands
- always run with sstate cache
- run everything in tmux to access logs while test is running
- provide pycache path to speedup python startup
- add builtin (memory only) squid cache
- use host network instead of container network (allows use of federated proxies)

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
@Cedric: This is RFC is based on your patchset and tries to further
simplify and speedup the testsuite invocation. Feel free to integrate
whatever you think makes sense. With your patch and my additions,
I finally found a way to efficiently execute single tests. However,
there is still much room for improvement in the testsuite itself.

Best regards,
Felix

 CONTRIBUTING.md     | 52 ++++++++++----------------------------------
 scripts/ci_setup.sh | 53 ++++++++++++++++++++++++++++++++++++++++-----
 scripts/ci_shell    | 36 ++++++++++++++++++++++++++++++
 3 files changed, 95 insertions(+), 46 deletions(-)
 create mode 100755 scripts/ci_shell

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 07621232..6970477b 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -81,54 +81,26 @@ Plan merges to `master` so that both fit the two-week window; short extensions s
      by launching CI tests scripts. The procedure is described below:
 
     ```
-    git clone https://github.com/siemens/kas
-    cat > kas.yml <<EOF
-    header:
-      version: 14
-    build_system: isar
-    defaults:
-      repos:
-        patches:
-          repo: isar
-    repos:
-      isar:
-        url: "http://github.com:/ilbers/isar"
-        branch: next
-        layers:
-          meta:
-          meta-isar:
-    EOF
-    kas/kas-container shell --command /work/isar/scripts/ci_setup.sh kas.yml
+    ./scripts/ci_shell
     ```
 
-    In kas shell:
+    This will spawn a shell where you may just run:
 
     ```
-    cd /work/isar/testsuite
-    avocado run citest.py -t dev --max-parallel-tasks=1
+    trun citest.py -t dev --max-parallel-tasks=1
     ```
 
-    Your git-formatpatches may be listed in the `kas.yml` file as illustrated below:
+    Note: (`trun` is an alias for `avocado run`).
 
-    ```
-    ...
-    repos:
-      isar:
-        url: "http://github.com:/ilbers/isar"
-        branch: next
-	patches:
-          0001:
-            path: /work/0001-my-contribution-to-isar.patch
-        layers:
-          meta:
-          meta-isar:
-    ```
-
-    Perform the above steps from a clean directory for your CI run to be as close as
-    possible to the environment that our project maintainers will be using. That
-    directory would contain: *.patch isar/ kas/ kas.yml
+    The container has `/work` bind-mounted to your local copy of the Isar code
+    base (and therefore with your changes). Make sure to track upstream changes
+    and regularly rebase your tree: your patches shall be sent against the latest
+    HEAD available when submitting. Keep an eye on changes while your series are
+    under review: you may need to send a refreshed version if contributions from
+    others were accepted and touch the same areas (make sure to re-run the test
+    suite after rebasing).
 
-    Be also mindful of community-provided resources such as deb.debian.org or
+    Be mindful of community-provided resources such as deb.debian.org or
     snapshot.debian.org and consider using a caching proxy in your setup to
     reduce traffic as much as possible.
 
diff --git a/scripts/ci_setup.sh b/scripts/ci_setup.sh
index f373d1c0..bbca4ab8 100755
--- a/scripts/ci_setup.sh
+++ b/scripts/ci_setup.sh
@@ -5,28 +5,69 @@
 # Copyright (c) Siemens AG, 2025
 # SPDX-License-Identifier: MIT
 
+# start a squid http proxy in the container
+with_squid="1"
+
 gpg_key=/etc/apt/trusted.gpg.d/debian-isar.gpg
 [ -f "${gpg_key}" ] || {
     wget -q http://deb.isar-build.org/debian-isar.key -O- \
     | gpg --dearmor \
-    | sudo dd of="${gpg_key}"
+    | sudo dd of="${gpg_key}" status=none
 }
 
 list=/etc/apt/sources.list.d/10-isar_build.list
 [ -f "${list}" ] || {
     echo "deb [signed-by=/etc/apt/trusted.gpg.d/debian-isar.gpg] \
         http://deb.isar-build.org/debian-isar bookworm-isar main" \
-    | sudo tee /etc/apt/sources.list.d/10-isar_build.list
+    | sudo tee /etc/apt/sources.list.d/10-isar_build.list >/dev/null
 }
 
-tools="avocado qemu-system-aarch64 qemu-system-arm qemu-system-i386 qemu-system-x86_64"
+echo "🔍 Checking dependencies..."
+tools="avocado qemu-system-aarch64 qemu-system-arm qemu-system-i386 qemu-system-x86_64 htop"
+if [ "${with_squid}" = "1" ]; then
+    tools="${tools} squid"
+    squid_pkg="squid"
+    start_squid="sudo /usr/sbin/squid;"
+    echo "export http_proxy=http://localhost:3128" \
+    | sudo tee -a /etc/profile.d/squid.sh >/dev/null
+fi
+
 need_install=0
 for tool in ${tools}; do
     which "${tool}" || need_install=1
 done
 [ "${need_install}" = "0" ] || {
-    sudo apt-get update
-    sudo apt-get install -y avocado qemu-system-arm qemu-system-x86
+    echo "⏳ Installing missing dependencies..."
+    (
+        sudo apt-get update
+        sudo apt-get install -y --no-install-recommends \
+            avocado qemu-system-arm qemu-system-x86 ovmf htop $squid_pkg
+    )
 }
 
-exec /container-entrypoint ${*}
+(
+    cat <<'    EOF'
+    alias tlist="avocado list"
+    alias trun="avocado run -p sstate=1 --max-parallel-tasks=1 --failfast"
+    alias trun-fg="avocado --show avocado.test run -p sstate=1 --max-parallel-tasks=1 --failfast"
+    alias tquit="tmux kill-session -t isar_ci"
+    alias thelp="cat ~/.trun-help"
+    export PYTHONPYCACHEPREFIX=/work/build/pycache
+    mkdir -p ${PYTHONPYCACHEPREFIX}
+    cd /work/testsuite
+    EOF
+) | sudo tee -a /etc/profile.d/ci.sh >/dev/null
+
+cat <<'EOF' > ~/.trun-help
+
+💡 Use "tlist citest.py" to list existing test cases
+💡 Use "trun citest.py:TestClass.test_method" to run a specific test
+💡 Use "trun-fg <test> to run a specific test and emit the logs
+💡 Use "tquit" to leave the testing environment
+💡 Use "thelp" to show this help message
+
+✅ Test environment ready for use!
+
+EOF
+
+exec /container-entrypoint tmux new-session -A -s isar_ci "$start_squid cat ~/.trun-help && bash -l"
diff --git a/scripts/ci_shell b/scripts/ci_shell
new file mode 100755
index 00000000..917d1272
--- /dev/null
+++ b/scripts/ci_shell
@@ -0,0 +1,36 @@
+#!/bin/bash
+# A simple wrapper to run a container to run CI tests
+#
+# Cedric Hombourger <cedric.hombourger@siemens.com>
+# Copyright (c) Siemens AG, 2025
+# SPDX-License-Identifier: MIT
+#
+# Usage: ci_shell [kas-container-version]
+
+self=${0}
+topdir=$(readlink -f $(dirname ${self})/..)
+
+export KAS_BUILD_DIR=$(mktemp -d)
+
+if [[ -n "$1" ]]; then
+    export KAS_IMAGE_VERSION="$1"
+fi
+
+drop_pyhashtables() {
+    # Remove possibly outdated pyhashtables.py file that is generated on-the-fly.
+    # The file is owned by the (bitbake) calling user, hence it's lifetime needs
+    # to be bound to the container's lifetime.
+    rm -f bitbake/lib/bb/pysh/pyshtables.py
+}
+cleanup() {
+    rm -rf --one-file-system ${KAS_BUILD_DIR}
+    drop_pyhashtables
+}
+trap "cleanup" EXIT
+
+cd ${topdir}
+drop_pyhashtables
+kas/kas-container \
+    --runtime-args "--network=host" \
+    shell --command "rm -rf /work/build/conf && /work/scripts/ci_setup.sh" \
+    kas/isar.yaml
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251212150124.1424230-1-felix.moessbauer%40siemens.com.