[PATCH v6 00/13] Add sstate-cache

[Adriaan Schmidt <adriaan.schmidt@siemens.com>]

This series adds the sstate-cache feature from OE to Isar. The cache holds
the results of bootstrapping, rootfs generation (buildchroot, image rootfs),
and deb package generation.

To use the cache, the only configuration neccessary is setting SSTATE_DIR.
The contents of that directory need to be preserved across bitbake invocations.

One known weakness is that the package lists of cached rootfs's can run
out of sync with upstream ("apt-get update" only happens at bootstrap time).
But this also happens with an "old" local build dir, and is something that
may be addressed elsewhere. For now, the recommendation is to frequently
clear the cache (in one of our projects we run a nightly "clear&populate
cache" CI job).

Patches 1..5 fix (unrelated) issues that would otherwise block sstate caching,
patch 6 copies files from OE, and patches 7..10 add caching to Isar.
Patches 11..12 make minor adjustments to the tests, and patch 13 adds
a test case for the new feature.

---
Changes since v5:
- added lib/oe/utils.py because sstate.bbclass uses oe.utils.ThreadedPool
  when accessing a remote sstate cache (SSTATE_MIRRORS)

Changes since v4:
- rebase on next
- add test case

Changes since v3:
- rebase on next
- make sstate understand that images and initramfss are MACHINE specific
  (fixes a false-sharing bug that occured while testing with multiconfig)

Changes since v2:
- rebase on next
- add the explicit isar-apt dependency to all images (not only wic as previously).
- don't move existing code in isar-bootstrap.inc
- add `do_deploy_deb[deptask] = "do_deploy_deb"` to dpkg-base. This is so that
  packages depend recursively on their dependencies, even when build results
  are taken from cache.
- generally improve commit messages

Changes since v1:
- fix copy/paste typo in rootfs.bbclass
- add mounting trick to tar rootfs (because --one-file-system
  does not stop at bind mounts)
- have install_imager_deps also depend explicitly on isar-apt

Adriaan Schmidt (13):
  oe imports in central location
  images: create deploy dir
  rootfs: recursively depend on packages
  base: remove unneeded "before do_build" task dependencies
  dpkg: add explicit dependency to isar-apt
  meta: add sstate feature from oe
  sstate: configure
  sstate: add caching to isar-bootstrap
  sstate: add caching to rootfs
  sstate: add caching to debian packages
  test: pass absolute path for build_dir
  test: make bitbake_args a list
  sstate: add test case

 meta/classes/base.bbclass                     |   32 +-
 meta/classes/container-img.bbclass            |    1 +
 meta/classes/cpiogz-img.bbclass               |    1 +
 meta/classes/dpkg-base.bbclass                |   34 +-
 meta/classes/dpkg.bbclass                     |    1 +
 meta/classes/ext4-img.bbclass                 |    1 +
 meta/classes/fit-img.bbclass                  |    1 +
 meta/classes/image-tools-extension.bbclass    |    2 +-
 meta/classes/image.bbclass                    |    3 +
 meta/classes/initramfs.bbclass                |    3 +
 meta/classes/patch.bbclass                    |    5 -
 meta/classes/rootfs.bbclass                   |   29 +-
 meta/classes/sstate.bbclass                   | 1311 +++++++++++++++++
 meta/classes/targz-img.bbclass                |    1 +
 meta/classes/ubi-img.bbclass                  |    1 +
 meta/classes/ubifs-img.bbclass                |    1 +
 meta/classes/wic-img.bbclass                  |    1 +
 meta/conf/bitbake.conf                        |   10 +-
 meta/lib/oe/gpg_sign.py                       |  130 ++
 meta/lib/oe/sstatesig.py                      |  603 ++++++++
 meta/lib/oe/utils.py                          |  569 +++++++
 .../isar-bootstrap/isar-bootstrap.inc         |   24 +
 scripts/ci_build.sh                           |    2 +-
 testsuite/build_test/build_test.py            |  101 +-
 testsuite/build_test/cibase.py                |   20 +-
 testsuite/build_test/cibuilder.py             |    2 +-
 26 files changed, 2853 insertions(+), 36 deletions(-)
 create mode 100644 meta/classes/sstate.bbclass
 create mode 100644 meta/lib/oe/gpg_sign.py
 create mode 100644 meta/lib/oe/sstatesig.py
 create mode 100644 meta/lib/oe/utils.py

-- 
2.30.2