Re: [PATCH] CI: Add sstate-cache testcase

[Jan Kiszka <jan.kiszka@siemens.com>]

On 02.11.21 06:57, Schmidt, Adriaan (T RDA IOT SES-DE) wrote:
> Henning Schild, Donnerstag, 28. Oktober 2021 20:22:
>> Am Thu, 28 Oct 2021 20:22:58 +0300
>> schrieb Anton Mikanovich <amikan@ilbers.de>:
>>
>>> 28.10.2021 18:34, Jan Kiszka wrote:
>>>> On 28.10.21 17:20, Anton Mikanovich wrote:
>>>>> Test rebuild time improve after cleanup to be sure sstate-cache
>>>>> works.
>>>>>
>>>>> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
>>>>> ---
>>>>>   testsuite/build_test/build_test.py | 40
>>>>> ++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2
>>>>> deletions(-)
>>>>>
>>>>> diff --git a/testsuite/build_test/build_test.py
>>>>> b/testsuite/build_test/build_test.py index d39c10c0..244f6fc0
>>>>> 100644 --- a/testsuite/build_test/build_test.py
>>>>> +++ b/testsuite/build_test/build_test.py
>>>>> @@ -1,9 +1,9 @@
>>>>>   #!/usr/bin/env python3
>>>>>
>>>>> -import os
>>>>> +import os, time
>>>>>
>>>>>   from avocado import skipUnless
>>>>> -from avocado.utils import path
>>>>> +from avocado.utils import path, process
>>>>>   from cibase import CIBaseTest
>>>>>
>>>>>   UMOCI_AVAILABLE = True
>>>>> @@ -206,3 +206,39 @@ class ContainerSdkTest(CIBaseTest):
>>>>>           targets = ['mc:container-amd64-stretch:isar-image-base']
>>>>>
>>>>>           self.perform_container_test(targets, 'do_populate_sdk')
>>>>> +
>>>>> +class SstateTest(CIBaseTest):
>>>>> +
>>>>> +    """
>>>>> +    Test rebuild speed improve with sstate-cache
>>>>> +
>>>>> +    :avocado: tags=sstate
>>>>> +    """
>>>>> +    def test_sstate_rebuild(self):
>>>>> +        speedup_k = 2.0
>>>>> +
>>>>> +        targets = [
>>>>> +            'mc:qemuamd64-bullseye:isar-image-base'
>>>>> +                  ]
>>>>> +
>>>>> +        # Cleanup everything before build
>>>>> +        build_dir = self.params.get('build_dir',
>>>>> +                    default=os.path.dirname(__file__) +
>>>>> '/../../build')
>>>>> +        process.run('rm -rf ' + build_dir + '/sstate-cache',
>>>>> sudo=True)
>>>>> +        self.deletetmp(build_dir)
>>>>> +
>>>>> +        start = time.time()
>>>>> +        self.perform_build_test(targets, 1, None)
>>>>> +        first_time = time.time() - start
>>>>> +        self.log.info('Non-cached build:' + str(round(first_time,
>>>>> 2))) +
>>>>> +        # Cleanup everything but cache files
>>>>> +        self.deletetmp(build_dir)
>>>>> +
>>>>> +        start = time.time()
>>>>> +        self.perform_build_test(targets, 1, None)
>>>>> +        second_time = time.time() - start
>>>>> +        self.log.info('Cached build:' + str(round(second_time,
>>>>> 2))) +
>>>>> +        if first_time / second_time < speedup_k:
>>>>> +            self.fail('No speedup after rebuild with
>>>>> sstate-cache')
>>>> Is there no better way than measuring time to test if sstate was in
>>>> place? Maybe some hit/miss statistics?
>>>>
>>>> Jan
>>>>
>>> I've tried to find some Avocado API to access final testcase
>>> statistics from within other testcase but didn't find so far.
>>> Any advice is welcome.
>>
>> sstate hit rates are in the bitbake logs, i hope avocado can open
>> artifacts
> 
> You want to look for the line "Sstate summary", which is shown at the beginning of the bitbake run, e.g.:
> 
> Sstate summary: Wanted 121 Local 121 Network 0 Missed 0 Current 0 (100% match, 0% complete)
> 
> But in a test it's probably better to examine log.task_order and check which tasks were executed.
> 
>> But this is infra ... we will see when it fails and do not have to test
>> that.
> 
> It's worth running some tests to see that builds succeed with artifacts taken from cache. I can imagine something like:
> 
> - Build image to populate cache
> - Remove TMPDIR
> - Rebuild the image. Should have 100% cache hits. (This would take the target rootfs from cache, and the buildchroot required for image-building)
> - Remove TMPDIR
> - Build a single package (that was already contained in the image). Should have a cache hit, and only dpkg_build_setscene and deploy_deb should run (plus minimal steps to initialize isar-apt).
> 
> And optionally:
> - Remove TMPDIR
> - Force rebuild of one package (e.g. by deleting its sstate artifact)
> - Rebuild the image (should take the buildchroot and all but the one package from cache, and rebuild the one package and the image)
> 

BTW, we likely now also need a cleansstate target, like OE has.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux