Re: [PATCH v2 0/7] Sstate maintenance script

[Anton Mikanovich <amikan@ilbers.de>]

09.05.2022 13:15, Adriaan Schmidt wrote:
> We have been running CI with shared sstate caches for some months now, in
> several downstream projects. This is the cache maintenance script that has
> evolved during that time. Detailed documentation is in the script itself.
> Main features:
> - upload cache artifacts to shared caches on filesystem, http, or s3
> - clean old artifacts from shared caches
> - analyze in detail why cache misses happen (what has changed in the signatures)
> - check the sstate signatures for absolute paths pointing to the build host
>
> The last two are especially interesting, and have already yielded some
> improvements to the cacheability of Isar, some already merged, and some
> more in "[PATCH 0/7] Further improve cachability of ISAR".
>
> p1 handles another absolute path in a variable (LAYERDIR_isar).
> p2..3 are minor patches to bitbake (both already upstream) that greatly
>    improve accuracy and performance of the sstate analysis.
> p4 refactors handling of the apt_* tasks. This was motivated by the sstate
>    analysis, but I think it also makes the code cleaner.
> p5..6 add the sstate maintenance script (2 authors, hence 2 patches).
> p7 includes a signature check into the sstate test case. This requires the
>    changes from "[PATCH 0/7] Further improve cachability of ISAR" for the
>    test to pass.
>
> One issue: testing!
> This is not easy, because it involves infrastructure, and artificial tests
> that provide decent coverage would be quite complex to design.
>
> If we declare that we sufficiently trust the sstate code, we could add a
> shared/persistent cache to the Isar CI infrastructure. This would further test
> the sstate feature and all steps involved in maintaining such a setup.
> In addition, it would significantly speed up CI builds.
>
> changes since v1:
> - generally improved script
> - analysis and cachability improvements in bitbake, dpkg-base, and meta-isar
> - added "sstate linting" to the testsuite
>
> Adriaan Schmidt (6):
>    meta-isar: improve cachability
>    bitbake-diffsigs: make finding of changed signatures more robust
>    bitbake-diffsigs: break on first dependent task difference
>    dpkg-base: refactor dependencies of apt_* tasks
>    scripts: add isar-sstate
>    testsuite: add cachability analysis to sstate test
>
> Felix Moessbauer (1):
>    isar-sstate: add tool to check for caching issues
>
>   bitbake/lib/bb/siggen.py       |  11 +-
>   meta-isar/conf/layer.conf      |   1 +
>   meta/classes/dpkg-base.bbclass |  14 +-
>   meta/classes/dpkg.bbclass      |   2 +-
>   scripts/isar-sstate            | 863 +++++++++++++++++++++++++++++++++
>   testsuite/cibase.py            |   8 +-
>   6 files changed, 884 insertions(+), 15 deletions(-)
>   create mode 100755 scripts/isar-sstate
>
Applied to next, thanks.