RE: [PATCH] CI: Add sstate-cache testcase

["Schmidt, Adriaan" <adriaan.schmidt@siemens.com>]

Henning Schild, Donnerstag, 28. Oktober 2021 20:22:
> Am Thu, 28 Oct 2021 20:22:58 +0300
> schrieb Anton Mikanovich <amikan@ilbers.de>:
> 
> > 28.10.2021 18:34, Jan Kiszka wrote:
> > > On 28.10.21 17:20, Anton Mikanovich wrote:
> > >> Test rebuild time improve after cleanup to be sure sstate-cache
> > >> works.
> > >>
> > >> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> > >> ---
> > >>   testsuite/build_test/build_test.py | 40
> > >> ++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2
> > >> deletions(-)
> > >>
> > >> diff --git a/testsuite/build_test/build_test.py
> > >> b/testsuite/build_test/build_test.py index d39c10c0..244f6fc0
> > >> 100644 --- a/testsuite/build_test/build_test.py
> > >> +++ b/testsuite/build_test/build_test.py
> > >> @@ -1,9 +1,9 @@
> > >>   #!/usr/bin/env python3
> > >>
> > >> -import os
> > >> +import os, time
> > >>
> > >>   from avocado import skipUnless
> > >> -from avocado.utils import path
> > >> +from avocado.utils import path, process
> > >>   from cibase import CIBaseTest
> > >>
> > >>   UMOCI_AVAILABLE = True
> > >> @@ -206,3 +206,39 @@ class ContainerSdkTest(CIBaseTest):
> > >>           targets = ['mc:container-amd64-stretch:isar-image-base']
> > >>
> > >>           self.perform_container_test(targets, 'do_populate_sdk')
> > >> +
> > >> +class SstateTest(CIBaseTest):
> > >> +
> > >> +    """
> > >> +    Test rebuild speed improve with sstate-cache
> > >> +
> > >> +    :avocado: tags=sstate
> > >> +    """
> > >> +    def test_sstate_rebuild(self):
> > >> +        speedup_k = 2.0
> > >> +
> > >> +        targets = [
> > >> +            'mc:qemuamd64-bullseye:isar-image-base'
> > >> +                  ]
> > >> +
> > >> +        # Cleanup everything before build
> > >> +        build_dir = self.params.get('build_dir',
> > >> +                    default=os.path.dirname(__file__) +
> > >> '/../../build')
> > >> +        process.run('rm -rf ' + build_dir + '/sstate-cache',
> > >> sudo=True)
> > >> +        self.deletetmp(build_dir)
> > >> +
> > >> +        start = time.time()
> > >> +        self.perform_build_test(targets, 1, None)
> > >> +        first_time = time.time() - start
> > >> +        self.log.info('Non-cached build:' + str(round(first_time,
> > >> 2))) +
> > >> +        # Cleanup everything but cache files
> > >> +        self.deletetmp(build_dir)
> > >> +
> > >> +        start = time.time()
> > >> +        self.perform_build_test(targets, 1, None)
> > >> +        second_time = time.time() - start
> > >> +        self.log.info('Cached build:' + str(round(second_time,
> > >> 2))) +
> > >> +        if first_time / second_time < speedup_k:
> > >> +            self.fail('No speedup after rebuild with
> > >> sstate-cache')
> > > Is there no better way than measuring time to test if sstate was in
> > > place? Maybe some hit/miss statistics?
> > >
> > > Jan
> > >
> > I've tried to find some Avocado API to access final testcase
> > statistics from within other testcase but didn't find so far.
> > Any advice is welcome.
> 
> sstate hit rates are in the bitbake logs, i hope avocado can open
> artifacts

You want to look for the line "Sstate summary", which is shown at the beginning of the bitbake run, e.g.:

Sstate summary: Wanted 121 Local 121 Network 0 Missed 0 Current 0 (100% match, 0% complete)

But in a test it's probably better to examine log.task_order and check which tasks were executed.

> But this is infra ... we will see when it fails and do not have to test
> that.

It's worth running some tests to see that builds succeed with artifacts taken from cache. I can imagine something like:

- Build image to populate cache
- Remove TMPDIR
- Rebuild the image. Should have 100% cache hits. (This would take the target rootfs from cache, and the buildchroot required for image-building)
- Remove TMPDIR
- Build a single package (that was already contained in the image). Should have a cache hit, and only dpkg_build_setscene and deploy_deb should run (plus minimal steps to initialize isar-apt).

And optionally:
- Remove TMPDIR
- Force rebuild of one package (e.g. by deleting its sstate artifact)
- Rebuild the image (should take the buildchroot and all but the one package from cache, and rebuild the one package and the image)

> The interesting problems will happen once you build version+5 on a
> cache from version. The only known limitiation there is that your base
> rootfss must not be too old. Because your local apt-database will start
> containing package versions that might not be available to install
> anymore.

At least IMO this is limitation is unrelated to the sstate cache, and I agree that testing such cases doesn't make much sense.

I'm happy to take over and propose a test case.

Adriaan