RE: [PATCHv2 0/9] Add more signature cachability tests to the testsuite

["Larson, Chris" <chris.larson@siemens.com>]

Were there any issues with the v2 updates? Thanks.

--
Christopher Larson
Siemens AG
www.siemens.com

-----Original Message-----
From: Larson, Chris (DI CTO FDS CES LX MEL) <chris.larson@siemens.com> 
Sent: Friday, April 5, 2024 9:31 AM
To: isar-users@googlegroups.com
Cc: Larson, Chris (DI CTO FDS CES LX MEL) <chris.larson@siemens.com>
Subject: [PATCHv2 0/9] Add more signature cachability tests to the testsuite

From: Christopher Larson <chris.larson@siemens.com>

This series improves isar-sstate's lint command to support checking sigdata in tmp/stamps and to check for absolute paths in SRC_URI, and adds an additional test to the testsuite to exercise this capability. The new test checks for signature cachability issues much more quickly, as it does so without a full build, so it checks more target configurations. This uses bitbake's `-S none` option to avoid building anything, and then runs isar-sstate lint on the resulting sigdata in tmp/stamps.

This test will accept a verbose parameter on the command-line to pass --verbose to isar-sstate lint, which will show the "other" absolute paths found, not just source and build directory. This series also fixes two of the failures which were identified by the new test.

Here we can see the issues identified by the new test, which are fixed in this series:
```
$ avocado run testsuite/citest.py -t signatures --max-parallel-tasks=1
JOB ID     : 3a3308967946663d9b239f638b030502fa80ef0a
JOB LOG    : /builder/avocado/job-results/job-2024-03-29T19.24-3a33089/job.log
 (1/1) testsuite/citest.py:SignatureTest.test_signature_lint: STARTED
 (1/1) testsuite/citest.py:SignatureTest.test_signature_lint: FAIL: Detected cachability issues (42.93 s)
RESULTS    : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 44.63 s

Test summary:
1-testsuite/citest.py:SignatureTest.test_signature_lint: FAIL ```

Applicable section of the avocado full.log:
```
1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29 19:25:10,818 avocado.test cibase           L0142 ERROR| ==== issues found in ubuntu-focal-amd64:isar-bootstrap-target:unpack (95af4581) ====
1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29 19:25:10,818 avocado.test cibase           L0142 ERROR| -> path in sources-dir: SRC_URI entry "file:///home/kergoth/Code/industrial/signatures/isar/meta-isar/conf/distro/ubuntu.public.key;sha256sum=36a38199a4bf4eae1e7f574891f7dfcb79b91b87a33a499383265e1224b5e989"
1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29 19:25:10,818 avocado.test cibase           L0142 ERROR| ==== issues found in ubuntu-focal-amd64:isar-bootstrap-target:fetch (e8249cb2) ====
1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29 19:25:10,818 avocado.test cibase           L0142 ERROR| -> path in sources-dir: SRC_URI entry "file:///home/kergoth/Code/industrial/signatures/isar/meta-isar/conf/distro/ubuntu.public.key;sha256sum=36a38199a4bf4eae1e7f574891f7dfcb79b91b87a33a499383265e1224b5e989"
1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29 19:25:10,818 avocado.test cibase           L0142 ERROR| ==== issues found in debian-bullseye-amd64:isar-ci-ssh-setup:install (1c0a8b21) ====
1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29 19:25:10,818 avocado.test cibase           L0142 ERROR| -> path in sources-dir: TESTSUITEDIR = "/home/kergoth/Code/industrial/signatures/isar/testsuite"
1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29 19:25:10,818 avocado.test cibase           L0142 ERROR| found cachability issues (scanned 602 signatures)
1-testsuite/citest.py:SignatureTest.test_signature_lint: 2024-03-29 19:25:10,818 avocado.test cibase           L0142 ERROR| -> absolute paths: sources-dir 3, build-dir 0, other 184
```

If we strip off the prefix for legibility:
```
ERROR| ==== issues found in 
ERROR| ubuntu-focal-amd64:isar-bootstrap-target:unpack (95af4581) ====
ERROR| -> path in sources-dir: SRC_URI entry "file:///home/kergoth/Code/industrial/signatures/isar/meta-isar/conf/distro/ubuntu.public.key;sha256sum=36a38199a4bf4eae1e7f574891f7dfcb79b91b87a33a499383265e1224b5e989"
ERROR| ==== issues found in 
ERROR| ubuntu-focal-amd64:isar-bootstrap-target:fetch (e8249cb2) ====
ERROR| -> path in sources-dir: SRC_URI entry "file:///home/kergoth/Code/industrial/signatures/isar/meta-isar/conf/distro/ubuntu.public.key;sha256sum=36a38199a4bf4eae1e7f574891f7dfcb79b91b87a33a499383265e1224b5e989"
ERROR| ==== issues found in 
ERROR| debian-bullseye-amd64:isar-ci-ssh-setup:install (1c0a8b21) ====
ERROR| -> path in sources-dir: TESTSUITEDIR = "/home/kergoth/Code/industrial/signatures/isar/testsuite"
ERROR| found cachability issues (scanned 602 signatures)
ERROR| -> absolute paths: sources-dir 3, build-dir 0, other 184
```

The sstate/signatures tests were run for each commit in this series using `git rebase -x` to ensure it does not break a bisect.

I welcome any and all feedback on this.

v2 updates:
- Explain the use of --dump-signatures in the docs on the lint command in isar-sstate
- Explain the stamps regex the way the sstate regex is explained, for consistency
- Support wildcards for the --excluded-tasks argument and default exclusions
- Drop the additional default task exclusions, as this should be revisited
- Fix a lint traceback if a variable's value is only whitespacxe

Christopher Larson (9):
  isar-bootstrap: avoid forced early expansion of key vars
  isar-ci-ssh-setup: avoid abs path in signatures
  isar-sstate: lint: check for absolute paths in SRC_URI
  isar-sstate: lint: add support for checking stamps
  isar-sstate: add --excluded-tasks argument
  isar-sstate: fix failures if a variable is set to just whitespace
  cibuilder.py: add -S support to the bitbake method
  testsuite: add perform_signature_lint method
  testsuite: add signature cachability checks

 .../isar-ci-ssh-setup_0.1.bb                  |  3 +
 .../isar-bootstrap/isar-bootstrap.inc         |  8 +-
 scripts/isar-sstate                           | 85 +++++++++++++++----
 testsuite/cibase.py                           | 19 +++++
 testsuite/cibuilder.py                        |  5 +-
 testsuite/citest.py                           | 22 ++++-
 6 files changed, 121 insertions(+), 21 deletions(-)

--
2.39.2