[PATCH v2 0/7] Sstate maintenance script

[Adriaan Schmidt <adriaan.schmidt@siemens.com>]

We have been running CI with shared sstate caches for some months now, in
several downstream projects. This is the cache maintenance script that has
evolved during that time. Detailed documentation is in the script itself.
Main features:
- upload cache artifacts to shared caches on filesystem, http, or s3
- clean old artifacts from shared caches
- analyze in detail why cache misses happen (what has changed in the signatures)
- check the sstate signatures for absolute paths pointing to the build host

The last two are especially interesting, and have already yielded some
improvements to the cacheability of Isar, some already merged, and some
more in "[PATCH 0/7] Further improve cachability of ISAR".

p1 handles another absolute path in a variable (LAYERDIR_isar).
p2..3 are minor patches to bitbake (both already upstream) that greatly
  improve accuracy and performance of the sstate analysis.
p4 refactors handling of the apt_* tasks. This was motivated by the sstate
  analysis, but I think it also makes the code cleaner.
p5..6 add the sstate maintenance script (2 authors, hence 2 patches).
p7 includes a signature check into the sstate test case. This requires the
  changes from "[PATCH 0/7] Further improve cachability of ISAR" for the
  test to pass.

One issue: testing!
This is not easy, because it involves infrastructure, and artificial tests
that provide decent coverage would be quite complex to design.

If we declare that we sufficiently trust the sstate code, we could add a
shared/persistent cache to the Isar CI infrastructure. This would further test
the sstate feature and all steps involved in maintaining such a setup.
In addition, it would significantly speed up CI builds.

changes since v1:
- generally improved script
- analysis and cachability improvements in bitbake, dpkg-base, and meta-isar
- added "sstate linting" to the testsuite

Adriaan Schmidt (6):
  meta-isar: improve cachability
  bitbake-diffsigs: make finding of changed signatures more robust
  bitbake-diffsigs: break on first dependent task difference
  dpkg-base: refactor dependencies of apt_* tasks
  scripts: add isar-sstate
  testsuite: add cachability analysis to sstate test

Felix Moessbauer (1):
  isar-sstate: add tool to check for caching issues

 bitbake/lib/bb/siggen.py       |  11 +-
 meta-isar/conf/layer.conf      |   1 +
 meta/classes/dpkg-base.bbclass |  14 +-
 meta/classes/dpkg.bbclass      |   2 +-
 scripts/isar-sstate            | 863 +++++++++++++++++++++++++++++++++
 testsuite/cibase.py            |   8 +-
 6 files changed, 884 insertions(+), 15 deletions(-)
 create mode 100755 scripts/isar-sstate

-- 
2.30.2