[PATCH 0/2] Enable SState cache in GitLab CI

[PATCH 0/2] Enable SState cache in GitLab CI

['Felix Moessbauer' via isar-users]

By running the GitLab CI jobs with SState cache enabled, huge speedups
can be achieved. Example:

Rebuild with 100% cache hits:

+ avocado run /build/testsuite/citest.py -t dev --max-parallel-tasks=1 --disable-sysinfo -p sstate=1
JOB ID     : 3070455808c8c6c4fa78adac9e4c4508fe001069
JOB LOG    : /build/build/job-results/job-2025-12-17T16.59-3070455/job.log
 (1/6) /build/testsuite/citest.py:DevTest.test_dev: STARTED
 (1/6) /build/testsuite/citest.py:DevTest.test_dev:  PASS (449.91 s)
 (2/6) /build/testsuite/citest.py:DevTest.test_dev_apps: STARTED
 (2/6) /build/testsuite/citest.py:DevTest.test_dev_apps:  PASS (307.69 s)
 (3/6) /build/testsuite/citest.py:DevTest.test_dev_rebuild: STARTED
 (3/6) /build/testsuite/citest.py:DevTest.test_dev_rebuild:  PASS (249.28 s)
 (4/6) /build/testsuite/citest.py:DevTest.test_dev_run_amd64_bookworm: STARTED
 (4/6) /build/testsuite/citest.py:DevTest.test_dev_run_amd64_bookworm:  PASS (96.88 s)
 (5/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm64_bookworm: STARTED
 (5/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm64_bookworm:  PASS (61.12 s)
 (6/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm_bookworm: STARTED
 (6/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm_bookworm:  PASS (64.81 s)
RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 1233.72 s

Rebuild without cache:

+ avocado run /build/testsuite/citest.py -t dev --max-parallel-tasks=1 --disable-sysinfo -p sstate=1
JOB ID     : 38c1d4fa508ab774c7774ad404a3575343c59979
JOB LOG    : /build/build/job-results/job-2025-12-17T13.29-38c1d4f/job.log
 (1/6) /build/testsuite/citest.py:DevTest.test_dev: STARTED
 (1/6) /build/testsuite/citest.py:DevTest.test_dev:  PASS (1559.37 s)
 (2/6) /build/testsuite/citest.py:DevTest.test_dev_apps: STARTED
 (2/6) /build/testsuite/citest.py:DevTest.test_dev_apps:  PASS (1429.77 s)
 (3/6) /build/testsuite/citest.py:DevTest.test_dev_rebuild: STARTED
 (3/6) /build/testsuite/citest.py:DevTest.test_dev_rebuild:  PASS (383.64 s)
 (4/6) /build/testsuite/citest.py:DevTest.test_dev_run_amd64_bookworm: STARTED
 (4/6) /build/testsuite/citest.py:DevTest.test_dev_run_amd64_bookworm:  PASS (99.56 s)
 (5/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm64_bookworm: STARTED
 (5/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm64_bookworm:  PASS (62.24 s)
 (6/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm_bookworm: STARTED
 (6/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm_bookworm:  PASS (65.56 s)
RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 3604.21 s

Note: the measurements are on top of the "Various improvements to the testsuite" series,
but the patches also work independently.

Best regards,
Felix Moessbauer

Felix Moessbauer (2):
  testsuite: forward SSTATE_MIRRORS into CI env on sstate
  ci: add support for sstate cache

 .gitlab-ci.yml         | 34 ++++++++++++++++++++++++++++------
 testsuite/cibuilder.py |  2 ++
 2 files changed, 30 insertions(+), 6 deletions(-)

-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251218073853.41121-1-felix.moessbauer%40siemens.com.

[PATCH 1/2] testsuite: forward SSTATE_MIRRORS into CI env on sstate

['Felix Moessbauer' via isar-users]

When running the testsuite with sstate caching enabled and also the
environment variable SSTATE_MIRRORS is set, add this to the local conf,
so the CI can also use remote sstate caches (important for CI).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 testsuite/cibuilder.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 26002ade..310a3836 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -232,6 +232,8 @@ class CIBuilder(Test):
                 f.write('DL_DIR = "%s"\n' % dl_dir)
             if sstate_dir:
                 f.write('SSTATE_DIR = "%s"\n' % sstate_dir)
+            if sstate and 'SSTATE_MIRRORS' in os.environ:
+                f.write('SSTATE_MIRRORS = "%s"\n' % os.environ['SSTATE_MIRRORS'])
             if image_install is not None:
                 f.write('IMAGE_INSTALL = "%s"\n' % image_install)
             else:
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251218073853.41121-2-felix.moessbauer%40siemens.com.

[PATCH 2/2] ci: add support for sstate cache

['Felix Moessbauer' via isar-users]

The gitlab CI jobs should finish in a timely manner to give quick
feedback to the developer. By running the tests with sstate cache,
simple changes require far less CI time.

We prepare the gitlab ci job description to pick up the sstate cache
configuration from the environment and enable the caching for all jobs
(we always enable it, even if running without sstate cache, as then the
cache is simply thrown away later on). We further introduce a (manual)
info task to show the state of the cache, as well as a cleanup task to
drain the cache.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 .gitlab-ci.yml | 34 ++++++++++++++++++++++++++++------
 1 file changed, 28 insertions(+), 6 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 1a7abcb8..9bd8ff07 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -12,6 +12,14 @@ variables:
     - export ftp_proxy=$FTP_PROXY
     - export no_proxy=$NO_PROXY
     - export DISTRO_APT_PREMIRRORS=$DISTRO_APT_PREMIRRORS
+  after_script:
+    - |
+      if [[ -n "${SSTATE_LOCATION}" ]] && [[ -d "sstate-cache" ]]; then
+        echo "=== Upload sstate artifacts to ${SSTATE_LOCATION} ==="
+        ./scripts/isar-sstate upload "sstate-cache" "${SSTATE_LOCATION}"
+        ./scripts/isar-sstate info -v "${SSTATE_LOCATION}"
+      fi
+
   artifacts:
     name: "logs-$CI_JOB_ID"
     paths:
@@ -48,7 +56,7 @@ dev-ci:
     - *use-default-image
     - if: $TESTSUITE == 'dev' || $CI_PIPELINE_SOURCE != 'schedule'
   script:
-    - scripts/ci_build.sh -T dev
+    - scripts/ci_build.sh --sstate 1 -T dev
 
 fast-ci:
   <<: *common-build
@@ -56,7 +64,7 @@ fast-ci:
     - *use-default-image
     - if: $TESTSUITE == 'fast'
   script:
-    - scripts/ci_build.sh -T fast
+    - scripts/ci_build.sh --sstate 1 -T fast
 
 full-ci:
   <<: *common-build
@@ -68,7 +76,7 @@ full-ci:
     - PREVIOUS_SHA="$(cat .CI_COMMIT_SHA || true)"
     - if [ "$CI_COMMIT_SHA" != "$PREVIOUS_SHA" ]; then
           echo "$CI_COMMIT_SHA" > .CI_COMMIT_SHA;
-          scripts/ci_build.sh -T full;
+          scripts/ci_build.sh --sstate 1 -T full;
       fi
   cache:
     key: "$CI_COMMIT_REF_SLUG"
@@ -83,7 +91,7 @@ dev-ci-isar:
     - *use-docker-isar-image
     - if: $TESTSUITE == 'dev'
   script:
-    - scripts/ci_build.sh -T dev
+    - scripts/ci_build.sh --sstate 1 -T dev
 
 fast-ci-isar:
   <<: *docker-isar
@@ -92,7 +100,7 @@ fast-ci-isar:
     - *use-docker-isar-image
     - if: $TESTSUITE == 'fast'
   script:
-    - scripts/ci_build.sh -T fast
+    - scripts/ci_build.sh --sstate 1 -T fast
 
 full-ci-isar:
   <<: *docker-isar
@@ -101,4 +109,18 @@ full-ci-isar:
     - *use-docker-isar-image
     - if: $TESTSUITE == 'full'
   script:
-    - scripts/ci_build.sh -T full
+    - scripts/ci_build.sh --sstate 1 -T full
+
+sstate-cache-info:
+  stage: build
+  when: manual
+  script:
+    - ./scripts/isar-sstate info -v "${SSTATE_LOCATION}"
+
+sstate-cache-clean:
+  stage: build
+  when: manual
+  variables:
+    SSTATE_MAX_AGE: "0d"
+  script:
+    - ./scripts/isar-sstate clean "${SSTATE_LOCATION}" --max-age "${SSTATE_MAX_AGE}"
-- 
2.51.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251218073853.41121-3-felix.moessbauer%40siemens.com.

Re: [PATCH 2/2] ci: add support for sstate cache

['MOESSBAUER, Felix' via isar-users]

On Thu, 2025-12-18 at 08:38 +0100, Felix Moessbauer wrote:
> The gitlab CI jobs should finish in a timely manner to give quick
> feedback to the developer. By running the tests with sstate cache,
> simple changes require far less CI time.
> 
> We prepare the gitlab ci job description to pick up the sstate cache
> configuration from the environment and enable the caching for all jobs
> (we always enable it, even if running without sstate cache, as then the
> cache is simply thrown away later on). We further introduce a (manual)
> info task to show the state of the cache, as well as a cleanup task to
> drain the cache.
> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>  .gitlab-ci.yml | 34 ++++++++++++++++++++++++++++------
>  1 file changed, 28 insertions(+), 6 deletions(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 1a7abcb8..9bd8ff07 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -12,6 +12,14 @@ variables:
>      - export ftp_proxy=$FTP_PROXY
>      - export no_proxy=$NO_PROXY
>      - export DISTRO_APT_PREMIRRORS=$DISTRO_APT_PREMIRRORS
> +  after_script:
> +    - |
> +      if [[ -n "${SSTATE_LOCATION}" ]] && [[ -d "sstate-cache" ]]; then
> +        echo "=== Upload sstate artifacts to ${SSTATE_LOCATION} ==="
> +        ./scripts/isar-sstate upload "sstate-cache" "${SSTATE_LOCATION}"
> +        ./scripts/isar-sstate info -v "${SSTATE_LOCATION}"
> +      fi

We need to increase the timeout of the after_script, as otherwise not
all artifacts might be uploaded on larger runs like "-t fast". We need
to add [1]

variables:
  RUNNER_AFTER_SCRIPT_TIMEOUT: 15m

I'll add it in case a v2 is needed. Otherwise it can also be added
later one, when more evidence around resonable timeouts are gathered.

[1]
https://docs.gitlab.com/ci/runners/configure_runners/#set-script-and-after_script-timeouts

Felix

-- 
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/4fc80c2806a1a4cc3b3019c437b76af83d59f4dd.camel%40siemens.com.