Re: [PATCHv2 0/9] Add more signature cachability tests to the testsuite

[Uladzimir Bely <ubely@ilbers.de>]

On Fri, 2024-04-05 at 16:31 +0000, chris.larson via isar-users wrote:
> From: Christopher Larson <chris.larson@siemens.com>
> 
> This series improves isar-sstate's lint command to support checking
> sigdata in tmp/stamps and to check for absolute paths in SRC_URI, and
> adds an additional test to the testsuite to exercise this capability.
> The new test checks for signature cachability issues much more
> quickly, as it does so without a full build, so it checks more target
> configurations. This uses bitbake's `-S none` option to avoid
> building anything, and then runs isar-sstate lint on the resulting
> sigdata in tmp/stamps.
> 
> This test will accept a verbose parameter on the command-line to pass
> --verbose to isar-sstate lint, which will show the "other" absolute
> paths found, not just source and build directory. This series also
> fixes two of the failures which were identified by the new test.
> 
> Here we can see the issues identified by the new test, which are
> fixed in this series:
> ```
> $ avocado run testsuite/citest.py -t signatures --max-parallel-
> tasks=1
> JOB ID     : 3a3308967946663d9b239f638b030502fa80ef0a
> JOB LOG    : /builder/avocado/job-results/job-2024-03-29T19.24-
> 3a33089/job.log
>  (1/1) testsuite/citest.py:SignatureTest.test_signature_lint: STARTED
>  (1/1) testsuite/citest.py:SignatureTest.test_signature_lint: FAIL:
> Detected cachability issues (42.93 s)
> RESULTS    : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT
> 0 | CANCEL 0
> JOB TIME   : 44.63 s
> 
> Test summary:
> 1-testsuite/citest.py:SignatureTest.test_signature_lint: FAIL
> ```
> 
> Applicable section of the avocado full.log:
> ```
> 1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29
> 19:25:10,818 avocado.test cibase           L0142 ERROR| ==== issues
> found in ubuntu-focal-amd64:isar-bootstrap-target:unpack (95af4581)
> ====
> 1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29
> 19:25:10,818 avocado.test cibase           L0142 ERROR| -> path in
> sources-dir: SRC_URI entry
> "file:///home/kergoth/Code/industrial/signatures/isar/meta-
> isar/conf/distro/ubuntu.public.key;sha256sum=36a38199a4bf4eae1e7f5748
> 91f7dfcb79b91b87a33a499383265e1224b5e989"
> 1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29
> 19:25:10,818 avocado.test cibase           L0142 ERROR| ==== issues
> found in ubuntu-focal-amd64:isar-bootstrap-target:fetch (e8249cb2)
> ====
> 1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29
> 19:25:10,818 avocado.test cibase           L0142 ERROR| -> path in
> sources-dir: SRC_URI entry
> "file:///home/kergoth/Code/industrial/signatures/isar/meta-
> isar/conf/distro/ubuntu.public.key;sha256sum=36a38199a4bf4eae1e7f5748
> 91f7dfcb79b91b87a33a499383265e1224b5e989"
> 1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29
> 19:25:10,818 avocado.test cibase           L0142 ERROR| ==== issues
> found in debian-bullseye-amd64:isar-ci-ssh-setup:install (1c0a8b21)
> ====
> 1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29
> 19:25:10,818 avocado.test cibase           L0142 ERROR| -> path in
> sources-dir: TESTSUITEDIR =
> "/home/kergoth/Code/industrial/signatures/isar/testsuite"
> 1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29
> 19:25:10,818 avocado.test cibase           L0142 ERROR| found
> cachability issues (scanned 602 signatures)
> 1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29
> 19:25:10,818 avocado.test cibase           L0142 ERROR| -> absolute
> paths: sources-dir 3, build-dir 0, other 184
> ```
> 
> If we strip off the prefix for legibility:
> ```
> ERROR| ==== issues found in ubuntu-focal-amd64:isar-bootstrap-
> target:unpack (95af4581) ====
> ERROR| -> path in sources-dir: SRC_URI entry
> "file:///home/kergoth/Code/industrial/signatures/isar/meta-
> isar/conf/distro/ubuntu.public.key;sha256sum=36a38199a4bf4eae1e7f5748
> 91f7dfcb79b91b87a33a499383265e1224b5e989"
> ERROR| ==== issues found in ubuntu-focal-amd64:isar-bootstrap-
> target:fetch (e8249cb2) ====
> ERROR| -> path in sources-dir: SRC_URI entry
> "file:///home/kergoth/Code/industrial/signatures/isar/meta-
> isar/conf/distro/ubuntu.public.key;sha256sum=36a38199a4bf4eae1e7f5748
> 91f7dfcb79b91b87a33a499383265e1224b5e989"
> ERROR| ==== issues found in debian-bullseye-amd64:isar-ci-ssh-
> setup:install (1c0a8b21) ====
> ERROR| -> path in sources-dir: TESTSUITEDIR =
> "/home/kergoth/Code/industrial/signatures/isar/testsuite"
> ERROR| found cachability issues (scanned 602 signatures)
> ERROR| -> absolute paths: sources-dir 3, build-dir 0, other 184
> ```
> 
> The sstate/signatures tests were run for each commit in this series
> using `git rebase -x` to ensure it does not break a bisect.
> 
> I welcome any and all feedback on this.
> 
> v2 updates:
> - Explain the use of --dump-signatures in the docs on the lint
> command in isar-sstate
> - Explain the stamps regex the way the sstate regex is explained, for
> consistency
> - Support wildcards for the --excluded-tasks argument and default
> exclusions
> - Drop the additional default task exclusions, as this should be
> revisited
> - Fix a lint traceback if a variable's value is only whitespacxe
> 
> Christopher Larson (9):
>   isar-bootstrap: avoid forced early expansion of key vars
>   isar-ci-ssh-setup: avoid abs path in signatures
>   isar-sstate: lint: check for absolute paths in SRC_URI
>   isar-sstate: lint: add support for checking stamps
>   isar-sstate: add --excluded-tasks argument
>   isar-sstate: fix failures if a variable is set to just whitespace
>   cibuilder.py: add -S support to the bitbake method
>   testsuite: add perform_signature_lint method
>   testsuite: add signature cachability checks
> 
>  .../isar-ci-ssh-setup_0.1.bb                  |  3 +
>  .../isar-bootstrap/isar-bootstrap.inc         |  8 +-
>  scripts/isar-sstate                           | 85 +++++++++++++++--
> --
>  testsuite/cibase.py                           | 19 +++++
>  testsuite/cibuilder.py                        |  5 +-
>  testsuite/citest.py                           | 22 ++++-
>  6 files changed, 121 insertions(+), 21 deletions(-)
> 
> -- 
> 2.39.2
> 

Applied to next, thanks.