* [PATCH 0/2] Enable SState cache in GitLab CI @ 2025-12-18 7:38 'Felix Moessbauer' via isar-users 2025-12-18 7:38 ` [PATCH 1/2] testsuite: forward SSTATE_MIRRORS into CI env on sstate 'Felix Moessbauer' via isar-users 2025-12-18 7:38 ` [PATCH 2/2] ci: add support for sstate cache 'Felix Moessbauer' via isar-users 0 siblings, 2 replies; 4+ messages in thread From: 'Felix Moessbauer' via isar-users @ 2025-12-18 7:38 UTC (permalink / raw) To: isar-users; +Cc: Felix Moessbauer By running the GitLab CI jobs with SState cache enabled, huge speedups can be achieved. Example: Rebuild with 100% cache hits: + avocado run /build/testsuite/citest.py -t dev --max-parallel-tasks=1 --disable-sysinfo -p sstate=1 JOB ID : 3070455808c8c6c4fa78adac9e4c4508fe001069 JOB LOG : /build/build/job-results/job-2025-12-17T16.59-3070455/job.log (1/6) /build/testsuite/citest.py:DevTest.test_dev: STARTED (1/6) /build/testsuite/citest.py:DevTest.test_dev: PASS (449.91 s) (2/6) /build/testsuite/citest.py:DevTest.test_dev_apps: STARTED (2/6) /build/testsuite/citest.py:DevTest.test_dev_apps: PASS (307.69 s) (3/6) /build/testsuite/citest.py:DevTest.test_dev_rebuild: STARTED (3/6) /build/testsuite/citest.py:DevTest.test_dev_rebuild: PASS (249.28 s) (4/6) /build/testsuite/citest.py:DevTest.test_dev_run_amd64_bookworm: STARTED (4/6) /build/testsuite/citest.py:DevTest.test_dev_run_amd64_bookworm: PASS (96.88 s) (5/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm64_bookworm: STARTED (5/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm64_bookworm: PASS (61.12 s) (6/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm_bookworm: STARTED (6/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm_bookworm: PASS (64.81 s) RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 1233.72 s Rebuild without cache: + avocado run /build/testsuite/citest.py -t dev --max-parallel-tasks=1 --disable-sysinfo -p sstate=1 JOB ID : 38c1d4fa508ab774c7774ad404a3575343c59979 JOB LOG : /build/build/job-results/job-2025-12-17T13.29-38c1d4f/job.log (1/6) /build/testsuite/citest.py:DevTest.test_dev: STARTED (1/6) /build/testsuite/citest.py:DevTest.test_dev: PASS (1559.37 s) (2/6) /build/testsuite/citest.py:DevTest.test_dev_apps: STARTED (2/6) /build/testsuite/citest.py:DevTest.test_dev_apps: PASS (1429.77 s) (3/6) /build/testsuite/citest.py:DevTest.test_dev_rebuild: STARTED (3/6) /build/testsuite/citest.py:DevTest.test_dev_rebuild: PASS (383.64 s) (4/6) /build/testsuite/citest.py:DevTest.test_dev_run_amd64_bookworm: STARTED (4/6) /build/testsuite/citest.py:DevTest.test_dev_run_amd64_bookworm: PASS (99.56 s) (5/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm64_bookworm: STARTED (5/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm64_bookworm: PASS (62.24 s) (6/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm_bookworm: STARTED (6/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm_bookworm: PASS (65.56 s) RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 3604.21 s Note: the measurements are on top of the "Various improvements to the testsuite" series, but the patches also work independently. Best regards, Felix Moessbauer Felix Moessbauer (2): testsuite: forward SSTATE_MIRRORS into CI env on sstate ci: add support for sstate cache .gitlab-ci.yml | 34 ++++++++++++++++++++++++++++------ testsuite/cibuilder.py | 2 ++ 2 files changed, 30 insertions(+), 6 deletions(-) -- 2.51.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251218073853.41121-1-felix.moessbauer%40siemens.com. ^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH 1/2] testsuite: forward SSTATE_MIRRORS into CI env on sstate 2025-12-18 7:38 [PATCH 0/2] Enable SState cache in GitLab CI 'Felix Moessbauer' via isar-users @ 2025-12-18 7:38 ` 'Felix Moessbauer' via isar-users 2025-12-18 7:38 ` [PATCH 2/2] ci: add support for sstate cache 'Felix Moessbauer' via isar-users 1 sibling, 0 replies; 4+ messages in thread From: 'Felix Moessbauer' via isar-users @ 2025-12-18 7:38 UTC (permalink / raw) To: isar-users; +Cc: Felix Moessbauer When running the testsuite with sstate caching enabled and also the environment variable SSTATE_MIRRORS is set, add this to the local conf, so the CI can also use remote sstate caches (important for CI). Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> --- testsuite/cibuilder.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py index 26002ade..310a3836 100755 --- a/testsuite/cibuilder.py +++ b/testsuite/cibuilder.py @@ -232,6 +232,8 @@ class CIBuilder(Test): f.write('DL_DIR = "%s"\n' % dl_dir) if sstate_dir: f.write('SSTATE_DIR = "%s"\n' % sstate_dir) + if sstate and 'SSTATE_MIRRORS' in os.environ: + f.write('SSTATE_MIRRORS = "%s"\n' % os.environ['SSTATE_MIRRORS']) if image_install is not None: f.write('IMAGE_INSTALL = "%s"\n' % image_install) else: -- 2.51.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251218073853.41121-2-felix.moessbauer%40siemens.com. ^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH 2/2] ci: add support for sstate cache 2025-12-18 7:38 [PATCH 0/2] Enable SState cache in GitLab CI 'Felix Moessbauer' via isar-users 2025-12-18 7:38 ` [PATCH 1/2] testsuite: forward SSTATE_MIRRORS into CI env on sstate 'Felix Moessbauer' via isar-users @ 2025-12-18 7:38 ` 'Felix Moessbauer' via isar-users 2025-12-18 11:57 ` 'MOESSBAUER, Felix' via isar-users 1 sibling, 1 reply; 4+ messages in thread From: 'Felix Moessbauer' via isar-users @ 2025-12-18 7:38 UTC (permalink / raw) To: isar-users; +Cc: Felix Moessbauer The gitlab CI jobs should finish in a timely manner to give quick feedback to the developer. By running the tests with sstate cache, simple changes require far less CI time. We prepare the gitlab ci job description to pick up the sstate cache configuration from the environment and enable the caching for all jobs (we always enable it, even if running without sstate cache, as then the cache is simply thrown away later on). We further introduce a (manual) info task to show the state of the cache, as well as a cleanup task to drain the cache. Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> --- .gitlab-ci.yml | 34 ++++++++++++++++++++++++++++------ 1 file changed, 28 insertions(+), 6 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 1a7abcb8..9bd8ff07 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -12,6 +12,14 @@ variables: - export ftp_proxy=$FTP_PROXY - export no_proxy=$NO_PROXY - export DISTRO_APT_PREMIRRORS=$DISTRO_APT_PREMIRRORS + after_script: + - | + if [[ -n "${SSTATE_LOCATION}" ]] && [[ -d "sstate-cache" ]]; then + echo "=== Upload sstate artifacts to ${SSTATE_LOCATION} ===" + ./scripts/isar-sstate upload "sstate-cache" "${SSTATE_LOCATION}" + ./scripts/isar-sstate info -v "${SSTATE_LOCATION}" + fi + artifacts: name: "logs-$CI_JOB_ID" paths: @@ -48,7 +56,7 @@ dev-ci: - *use-default-image - if: $TESTSUITE == 'dev' || $CI_PIPELINE_SOURCE != 'schedule' script: - - scripts/ci_build.sh -T dev + - scripts/ci_build.sh --sstate 1 -T dev fast-ci: <<: *common-build @@ -56,7 +64,7 @@ fast-ci: - *use-default-image - if: $TESTSUITE == 'fast' script: - - scripts/ci_build.sh -T fast + - scripts/ci_build.sh --sstate 1 -T fast full-ci: <<: *common-build @@ -68,7 +76,7 @@ full-ci: - PREVIOUS_SHA="$(cat .CI_COMMIT_SHA || true)" - if [ "$CI_COMMIT_SHA" != "$PREVIOUS_SHA" ]; then echo "$CI_COMMIT_SHA" > .CI_COMMIT_SHA; - scripts/ci_build.sh -T full; + scripts/ci_build.sh --sstate 1 -T full; fi cache: key: "$CI_COMMIT_REF_SLUG" @@ -83,7 +91,7 @@ dev-ci-isar: - *use-docker-isar-image - if: $TESTSUITE == 'dev' script: - - scripts/ci_build.sh -T dev + - scripts/ci_build.sh --sstate 1 -T dev fast-ci-isar: <<: *docker-isar @@ -92,7 +100,7 @@ fast-ci-isar: - *use-docker-isar-image - if: $TESTSUITE == 'fast' script: - - scripts/ci_build.sh -T fast + - scripts/ci_build.sh --sstate 1 -T fast full-ci-isar: <<: *docker-isar @@ -101,4 +109,18 @@ full-ci-isar: - *use-docker-isar-image - if: $TESTSUITE == 'full' script: - - scripts/ci_build.sh -T full + - scripts/ci_build.sh --sstate 1 -T full + +sstate-cache-info: + stage: build + when: manual + script: + - ./scripts/isar-sstate info -v "${SSTATE_LOCATION}" + +sstate-cache-clean: + stage: build + when: manual + variables: + SSTATE_MAX_AGE: "0d" + script: + - ./scripts/isar-sstate clean "${SSTATE_LOCATION}" --max-age "${SSTATE_MAX_AGE}" -- 2.51.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/20251218073853.41121-3-felix.moessbauer%40siemens.com. ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH 2/2] ci: add support for sstate cache 2025-12-18 7:38 ` [PATCH 2/2] ci: add support for sstate cache 'Felix Moessbauer' via isar-users @ 2025-12-18 11:57 ` 'MOESSBAUER, Felix' via isar-users 0 siblings, 0 replies; 4+ messages in thread From: 'MOESSBAUER, Felix' via isar-users @ 2025-12-18 11:57 UTC (permalink / raw) To: isar-users On Thu, 2025-12-18 at 08:38 +0100, Felix Moessbauer wrote: > The gitlab CI jobs should finish in a timely manner to give quick > feedback to the developer. By running the tests with sstate cache, > simple changes require far less CI time. > > We prepare the gitlab ci job description to pick up the sstate cache > configuration from the environment and enable the caching for all jobs > (we always enable it, even if running without sstate cache, as then the > cache is simply thrown away later on). We further introduce a (manual) > info task to show the state of the cache, as well as a cleanup task to > drain the cache. > > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> > --- > .gitlab-ci.yml | 34 ++++++++++++++++++++++++++++------ > 1 file changed, 28 insertions(+), 6 deletions(-) > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > index 1a7abcb8..9bd8ff07 100644 > --- a/.gitlab-ci.yml > +++ b/.gitlab-ci.yml > @@ -12,6 +12,14 @@ variables: > - export ftp_proxy=$FTP_PROXY > - export no_proxy=$NO_PROXY > - export DISTRO_APT_PREMIRRORS=$DISTRO_APT_PREMIRRORS > + after_script: > + - | > + if [[ -n "${SSTATE_LOCATION}" ]] && [[ -d "sstate-cache" ]]; then > + echo "=== Upload sstate artifacts to ${SSTATE_LOCATION} ===" > + ./scripts/isar-sstate upload "sstate-cache" "${SSTATE_LOCATION}" > + ./scripts/isar-sstate info -v "${SSTATE_LOCATION}" > + fi We need to increase the timeout of the after_script, as otherwise not all artifacts might be uploaded on larger runs like "-t fast". We need to add [1] variables: RUNNER_AFTER_SCRIPT_TIMEOUT: 15m I'll add it in case a v2 is needed. Otherwise it can also be added later one, when more evidence around resonable timeouts are gathered. [1] https://docs.gitlab.com/ci/runners/configure_runners/#set-script-and-after_script-timeouts Felix -- Siemens AG Linux Expert Center Friedrich-Ludwig-Bauer-Str. 3 85748 Garching, Germany -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/4fc80c2806a1a4cc3b3019c437b76af83d59f4dd.camel%40siemens.com. ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2025-12-18 11:57 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2025-12-18 7:38 [PATCH 0/2] Enable SState cache in GitLab CI 'Felix Moessbauer' via isar-users 2025-12-18 7:38 ` [PATCH 1/2] testsuite: forward SSTATE_MIRRORS into CI env on sstate 'Felix Moessbauer' via isar-users 2025-12-18 7:38 ` [PATCH 2/2] ci: add support for sstate cache 'Felix Moessbauer' via isar-users 2025-12-18 11:57 ` 'MOESSBAUER, Felix' via isar-users
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox