RE: [PATCH v4 00/10] Add sstate-cache

["Schmidt, Adriaan" <adriaan.schmidt@siemens.com>]

Current status of the sstate patches:
- we've been using them internally for a few months without problems
- they now pass the "fast-ci" test suite (there were some issues related to multiconfig)

To see the effect of caching, SSTATE_DIR needs to be persistent between bitbake calls. When running in GitLab CI, I change the script to run the test suite twice, deleting TMPDIR between the invocations:

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 1693a8a..e2fd652 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -27,7 +27,9 @@ fast-ci:
   except:
     - schedules
   script:
     - scripts/ci_build.sh -q -f -d
+    - sudo rm -rf build/tmp/
+    - scripts/ci_build.sh -q -f -d
 
 full-ci:
   <<: *common-build
--

The first run reports:
RESULTS    : PASS 8 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 17724.65 s

and the second:
RESULTS    : PASS 8 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 1002.72 s

Adriaan

> -----Original Message-----
> From: Adriaan Schmidt <adriaan.schmidt@siemens.com>
> Sent: Dienstag, 26. Oktober 2021 14:28
> To: isar-users@googlegroups.com
> Cc: Schmidt, Adriaan (T RDA IOT SES-DE) <adriaan.schmidt@siemens.com>
> Subject: [PATCH v4 00/10] Add sstate-cache
> 
> This series adds the sstate-cache feature from OE to Isar. The cache holds
> the results of bootstrapping, rootfs generation (buildchroot, image rootfs),
> and deb package generation.
> 
> To use the cache, the only configuration neccessary is setting SSTATE_DIR.
> The contents of that directory need to be preserved across bitbake
> invocations.
> 
> One known weakness is that the package lists of cached rootfs's can run
> out of sync with upstream ("apt-get update" only happens at bootstrap time).
> But this also happens with an "old" local build dir, and is something that
> may be addressed elsewhere. For now, the recommendation is to frequently
> clear the cache (in one of our projects we run a nightly "clear&populate
> cache" CI job).
> 
> Patches 1..5 fix (unrelated) issues that would otherwise block sstate
> caching,
> patch 6 copies files from OE, and patches 7..10 add caching to Isar.
> 
> ---
> Changes since v3:
> - rebase on next
> - make sstate understand that images and initramfss are MACHINE specific
>   (fixes a false-sharing bug that occured while testing with multiconfig)
> 
> Changes since v2:
> - rebase on next
> - add the explicit isar-apt dependency to all images (not only wic as
> previously).
> - don't move existing code in isar-bootstrap.inc
> - add `do_deploy_deb[deptask] = "do_deploy_deb"` to dpkg-base. This is so
> that
>   packages depend recursively on their dependencies, even when build results
>   are taken from cache.
> - generally improve commit messages
> 
> Changes since v1:
> - fix copy/paste typo in rootfs.bbclass
> - add mounting trick to tar rootfs (because --one-file-system
>   does not stop at bind mounts)
> - have install_imager_deps also depend explicitly on isar-apt
> 
> Adriaan Schmidt (10):
>   oe imports in central location
>   images: create deploy dir
>   rootfs: recursively depend on packages
>   base: remove unneeded "before do_build" task dependencies
>   dpkg: add explicit dependency to isar-apt
>   meta: add sstate feature from oe
>   sstate: configure
>   sstate: add caching to isar-bootstrap
>   sstate: add caching to rootfs
>   sstate: add caching to debian packages
> 
>  meta/classes/base.bbclass                     |   32 +-
>  meta/classes/container-img.bbclass            |    1 +
>  meta/classes/cpiogz-img.bbclass               |    1 +
>  meta/classes/dpkg-base.bbclass                |   34 +-
>  meta/classes/dpkg.bbclass                     |    1 +
>  meta/classes/ext4-img.bbclass                 |    1 +
>  meta/classes/fit-img.bbclass                  |    1 +
>  meta/classes/image-tools-extension.bbclass    |    2 +-
>  meta/classes/image.bbclass                    |    3 +
>  meta/classes/initramfs.bbclass                |    3 +
>  meta/classes/patch.bbclass                    |    5 -
>  meta/classes/rootfs.bbclass                   |   29 +-
>  meta/classes/sstate.bbclass                   | 1311 +++++++++++++++++
>  meta/classes/targz-img.bbclass                |    1 +
>  meta/classes/ubi-img.bbclass                  |    1 +
>  meta/classes/ubifs-img.bbclass                |    1 +
>  meta/classes/wic-img.bbclass                  |    1 +
>  meta/conf/bitbake.conf                        |   10 +-
>  meta/lib/oe/gpg_sign.py                       |  130 ++
>  meta/lib/oe/sstatesig.py                      |  603 ++++++++
>  .../isar-bootstrap/isar-bootstrap.inc         |   24 +
>  21 files changed, 2178 insertions(+), 17 deletions(-)
>  create mode 100644 meta/classes/sstate.bbclass
>  create mode 100644 meta/lib/oe/gpg_sign.py
>  create mode 100644 meta/lib/oe/sstatesig.py
> 
> --
> 2.30.2