Re: [PATCH] CI: Add sstate-cache testcase

[Henning Schild <henning.schild@siemens.com>]

Am Thu, 28 Oct 2021 20:22:58 +0300
schrieb Anton Mikanovich <amikan@ilbers.de>:

> 28.10.2021 18:34, Jan Kiszka wrote:
> > On 28.10.21 17:20, Anton Mikanovich wrote:  
> >> Test rebuild time improve after cleanup to be sure sstate-cache
> >> works.
> >>
> >> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> >> ---
> >>   testsuite/build_test/build_test.py | 40
> >> ++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2
> >> deletions(-)
> >>
> >> diff --git a/testsuite/build_test/build_test.py
> >> b/testsuite/build_test/build_test.py index d39c10c0..244f6fc0
> >> 100644 --- a/testsuite/build_test/build_test.py
> >> +++ b/testsuite/build_test/build_test.py
> >> @@ -1,9 +1,9 @@
> >>   #!/usr/bin/env python3
> >>   
> >> -import os
> >> +import os, time
> >>   
> >>   from avocado import skipUnless
> >> -from avocado.utils import path
> >> +from avocado.utils import path, process
> >>   from cibase import CIBaseTest
> >>   
> >>   UMOCI_AVAILABLE = True
> >> @@ -206,3 +206,39 @@ class ContainerSdkTest(CIBaseTest):
> >>           targets = ['mc:container-amd64-stretch:isar-image-base']
> >>   
> >>           self.perform_container_test(targets, 'do_populate_sdk')
> >> +
> >> +class SstateTest(CIBaseTest):
> >> +
> >> +    """
> >> +    Test rebuild speed improve with sstate-cache
> >> +
> >> +    :avocado: tags=sstate
> >> +    """
> >> +    def test_sstate_rebuild(self):
> >> +        speedup_k = 2.0
> >> +
> >> +        targets = [
> >> +            'mc:qemuamd64-bullseye:isar-image-base'
> >> +                  ]
> >> +
> >> +        # Cleanup everything before build
> >> +        build_dir = self.params.get('build_dir',
> >> +                    default=os.path.dirname(__file__) +
> >> '/../../build')
> >> +        process.run('rm -rf ' + build_dir + '/sstate-cache',
> >> sudo=True)
> >> +        self.deletetmp(build_dir)
> >> +
> >> +        start = time.time()
> >> +        self.perform_build_test(targets, 1, None)
> >> +        first_time = time.time() - start
> >> +        self.log.info('Non-cached build:' + str(round(first_time,
> >> 2))) +
> >> +        # Cleanup everything but cache files
> >> +        self.deletetmp(build_dir)
> >> +
> >> +        start = time.time()
> >> +        self.perform_build_test(targets, 1, None)
> >> +        second_time = time.time() - start
> >> +        self.log.info('Cached build:' + str(round(second_time,
> >> 2))) +
> >> +        if first_time / second_time < speedup_k:
> >> +            self.fail('No speedup after rebuild with
> >> sstate-cache') 
> > Is there no better way than measuring time to test if sstate was in
> > place? Maybe some hit/miss statistics?
> >
> > Jan
> >  
> I've tried to find some Avocado API to access final testcase
> statistics from within other testcase but didn't find so far.
> Any advice is welcome.

sstate hit rates are in the bitbake logs, i hope avocado can open
artifacts

But this is infra ... we will see when it fails and do not have to test
that.

The interesting problems will happen once you build version+5 on a
cache from version. The only known limitiation there is that your base
rootfss must not be too old. Because your local apt-database will start
containing package versions that might not be available to install
anymore.
So you have to find an eviction strategy. But that is very project
specific.

We currently have two strategies in place on two projects happily
working with this since weeks.
  1. nightly clean + full build
  2. "tmpreaper"-style age-greater 3 days

Third one you build against a frozen mirror and never clear, or very
infrequent.

Plus this problem is well known to isar guys, you just have to
sometimes do a full build or "bitbake -C task recipe". It just now
moves to runners ... but only if those are specially configured to
offer persistent storage.

Not sure how jenkins deals with this per default, and what sort of
configuration the ilbers jenkins might need.

Henning