Re: [PATCH v4 00/10] Add sstate-cache

[Henning Schild <henning.schild@siemens.com>]

Can not wait to see that merged. Or maybe i can ... having to wait in
isar projects that do not yet use this! It is so cool to see it in
place and saving tons of time!

I am not sure but maybe p4 might need an entry in the API CHANGELOG,
maybe others as well. You fixed several subtile dependency bugs, which
might in fact be present in layers again (copied over from a bad
example).
It might be hard to guess which mistakes people made or which of your
changes should be considered an API change ... Maybe a generic section
in the CHANGELOG could try and say
"sstate enabled"
  things will be fast on partial rebuilds ... deps might have changed
  in a weird way ;)

Same goes for the user manual, maybe a short section on the new
feature. How to clean the cache etc. Or a pointer to OE.
Maybe also suggests on how to configure the cache on a CI runner and
how to evict it. Not with code but maybe conceptually.
"evict your sstate cache on CI runners" regularly because of "outdated
apt database from apt-get update in bootstraped rootfss"

regards,
Henning 

Am Tue, 26 Oct 2021 14:28:01 +0200
schrieb Adriaan Schmidt <adriaan.schmidt@siemens.com>:

> This series adds the sstate-cache feature from OE to Isar. The cache
> holds the results of bootstrapping, rootfs generation (buildchroot,
> image rootfs), and deb package generation.
> 
> To use the cache, the only configuration neccessary is setting
> SSTATE_DIR. The contents of that directory need to be preserved
> across bitbake invocations.
> 
> One known weakness is that the package lists of cached rootfs's can
> run out of sync with upstream ("apt-get update" only happens at
> bootstrap time). But this also happens with an "old" local build dir,
> and is something that may be addressed elsewhere. For now, the
> recommendation is to frequently clear the cache (in one of our
> projects we run a nightly "clear&populate cache" CI job).
> 
> Patches 1..5 fix (unrelated) issues that would otherwise block sstate
> caching, patch 6 copies files from OE, and patches 7..10 add caching
> to Isar.
> 
> ---
> Changes since v3:
> - rebase on next
> - make sstate understand that images and initramfss are MACHINE
> specific (fixes a false-sharing bug that occured while testing with
> multiconfig)
> 
> Changes since v2:
> - rebase on next
> - add the explicit isar-apt dependency to all images (not only wic as
> previously).
> - don't move existing code in isar-bootstrap.inc
> - add `do_deploy_deb[deptask] = "do_deploy_deb"` to dpkg-base. This
> is so that packages depend recursively on their dependencies, even
> when build results are taken from cache.
> - generally improve commit messages
> 
> Changes since v1:
> - fix copy/paste typo in rootfs.bbclass
> - add mounting trick to tar rootfs (because --one-file-system
>   does not stop at bind mounts)
> - have install_imager_deps also depend explicitly on isar-apt
> 
> Adriaan Schmidt (10):
>   oe imports in central location
>   images: create deploy dir
>   rootfs: recursively depend on packages
>   base: remove unneeded "before do_build" task dependencies
>   dpkg: add explicit dependency to isar-apt
>   meta: add sstate feature from oe
>   sstate: configure
>   sstate: add caching to isar-bootstrap
>   sstate: add caching to rootfs
>   sstate: add caching to debian packages
> 
>  meta/classes/base.bbclass                     |   32 +-
>  meta/classes/container-img.bbclass            |    1 +
>  meta/classes/cpiogz-img.bbclass               |    1 +
>  meta/classes/dpkg-base.bbclass                |   34 +-
>  meta/classes/dpkg.bbclass                     |    1 +
>  meta/classes/ext4-img.bbclass                 |    1 +
>  meta/classes/fit-img.bbclass                  |    1 +
>  meta/classes/image-tools-extension.bbclass    |    2 +-
>  meta/classes/image.bbclass                    |    3 +
>  meta/classes/initramfs.bbclass                |    3 +
>  meta/classes/patch.bbclass                    |    5 -
>  meta/classes/rootfs.bbclass                   |   29 +-
>  meta/classes/sstate.bbclass                   | 1311
> +++++++++++++++++ meta/classes/targz-img.bbclass                |
> 1 + meta/classes/ubi-img.bbclass                  |    1 +
>  meta/classes/ubifs-img.bbclass                |    1 +
>  meta/classes/wic-img.bbclass                  |    1 +
>  meta/conf/bitbake.conf                        |   10 +-
>  meta/lib/oe/gpg_sign.py                       |  130 ++
>  meta/lib/oe/sstatesig.py                      |  603 ++++++++
>  .../isar-bootstrap/isar-bootstrap.inc         |   24 +
>  21 files changed, 2178 insertions(+), 17 deletions(-)
>  create mode 100644 meta/classes/sstate.bbclass
>  create mode 100644 meta/lib/oe/gpg_sign.py
>  create mode 100644 meta/lib/oe/sstatesig.py
>