Re: [PATCH v6 00/13] Add sstate-cache

[Anton Mikanovich <amikan@ilbers.de>]

11.11.2021 09:47, Adriaan Schmidt wrote:
> This series adds the sstate-cache feature from OE to Isar. The cache holds
> the results of bootstrapping, rootfs generation (buildchroot, image rootfs),
> and deb package generation.
>
> To use the cache, the only configuration neccessary is setting SSTATE_DIR.
> The contents of that directory need to be preserved across bitbake invocations.
>
> One known weakness is that the package lists of cached rootfs's can run
> out of sync with upstream ("apt-get update" only happens at bootstrap time).
> But this also happens with an "old" local build dir, and is something that
> may be addressed elsewhere. For now, the recommendation is to frequently
> clear the cache (in one of our projects we run a nightly "clear&populate
> cache" CI job).
>
> Patches 1..5 fix (unrelated) issues that would otherwise block sstate caching,
> patch 6 copies files from OE, and patches 7..10 add caching to Isar.
> Patches 11..12 make minor adjustments to the tests, and patch 13 adds
> a test case for the new feature.
>
> ---
> Changes since v5:
> - added lib/oe/utils.py because sstate.bbclass uses oe.utils.ThreadedPool
>    when accessing a remote sstate cache (SSTATE_MIRRORS)
>
> Changes since v4:
> - rebase on next
> - add test case
>
> Changes since v3:
> - rebase on next
> - make sstate understand that images and initramfss are MACHINE specific
>    (fixes a false-sharing bug that occured while testing with multiconfig)
>
> Changes since v2:
> - rebase on next
> - add the explicit isar-apt dependency to all images (not only wic as previously).
> - don't move existing code in isar-bootstrap.inc
> - add `do_deploy_deb[deptask] = "do_deploy_deb"` to dpkg-base. This is so that
>    packages depend recursively on their dependencies, even when build results
>    are taken from cache.
> - generally improve commit messages
>
> Changes since v1:
> - fix copy/paste typo in rootfs.bbclass
> - add mounting trick to tar rootfs (because --one-file-system
>    does not stop at bind mounts)
> - have install_imager_deps also depend explicitly on isar-apt
>
> Adriaan Schmidt (13):
>    oe imports in central location
>    images: create deploy dir
>    rootfs: recursively depend on packages
>    base: remove unneeded "before do_build" task dependencies
>    dpkg: add explicit dependency to isar-apt
>    meta: add sstate feature from oe
>    sstate: configure
>    sstate: add caching to isar-bootstrap
>    sstate: add caching to rootfs
>    sstate: add caching to debian packages
>    test: pass absolute path for build_dir
>    test: make bitbake_args a list
>    sstate: add test case
>
>   meta/classes/base.bbclass                     |   32 +-
>   meta/classes/container-img.bbclass            |    1 +
>   meta/classes/cpiogz-img.bbclass               |    1 +
>   meta/classes/dpkg-base.bbclass                |   34 +-
>   meta/classes/dpkg.bbclass                     |    1 +
>   meta/classes/ext4-img.bbclass                 |    1 +
>   meta/classes/fit-img.bbclass                  |    1 +
>   meta/classes/image-tools-extension.bbclass    |    2 +-
>   meta/classes/image.bbclass                    |    3 +
>   meta/classes/initramfs.bbclass                |    3 +
>   meta/classes/patch.bbclass                    |    5 -
>   meta/classes/rootfs.bbclass                   |   29 +-
>   meta/classes/sstate.bbclass                   | 1311 +++++++++++++++++
>   meta/classes/targz-img.bbclass                |    1 +
>   meta/classes/ubi-img.bbclass                  |    1 +
>   meta/classes/ubifs-img.bbclass                |    1 +
>   meta/classes/wic-img.bbclass                  |    1 +
>   meta/conf/bitbake.conf                        |   10 +-
>   meta/lib/oe/gpg_sign.py                       |  130 ++
>   meta/lib/oe/sstatesig.py                      |  603 ++++++++
>   meta/lib/oe/utils.py                          |  569 +++++++
>   .../isar-bootstrap/isar-bootstrap.inc         |   24 +
>   scripts/ci_build.sh                           |    2 +-
>   testsuite/build_test/build_test.py            |  101 +-
>   testsuite/build_test/cibase.py                |   20 +-
>   testsuite/build_test/cibuilder.py             |    2 +-
>   26 files changed, 2853 insertions(+), 36 deletions(-)
>   create mode 100644 meta/classes/sstate.bbclass
>   create mode 100644 meta/lib/oe/gpg_sign.py
>   create mode 100644 meta/lib/oe/sstatesig.py
>   create mode 100644 meta/lib/oe/utils.py
>
Patches 01-10 applied to next, thanks.

Test API rebuild and sstate test case can be separated into the new 
patchset.

-- 
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov