[PATCH] CI: Add sstate-cache testcase

[PATCH] CI: Add sstate-cache testcase

[Anton Mikanovich]

Test rebuild time improve after cleanup to be sure sstate-cache works.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/build_test/build_test.py | 40 ++++++++++++++++++++++++++++--
 1 file changed, 38 insertions(+), 2 deletions(-)

diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
index d39c10c0..244f6fc0 100644
--- a/testsuite/build_test/build_test.py
+++ b/testsuite/build_test/build_test.py
@@ -1,9 +1,9 @@
 #!/usr/bin/env python3
 
-import os
+import os, time
 
 from avocado import skipUnless
-from avocado.utils import path
+from avocado.utils import path, process
 from cibase import CIBaseTest
 
 UMOCI_AVAILABLE = True
@@ -206,3 +206,39 @@ class ContainerSdkTest(CIBaseTest):
         targets = ['mc:container-amd64-stretch:isar-image-base']
 
         self.perform_container_test(targets, 'do_populate_sdk')
+
+class SstateTest(CIBaseTest):
+
+    """
+    Test rebuild speed improve with sstate-cache
+
+    :avocado: tags=sstate
+    """
+    def test_sstate_rebuild(self):
+        speedup_k = 2.0
+
+        targets = [
+            'mc:qemuamd64-bullseye:isar-image-base'
+                  ]
+
+        # Cleanup everything before build
+        build_dir = self.params.get('build_dir',
+                    default=os.path.dirname(__file__) + '/../../build')
+        process.run('rm -rf ' + build_dir + '/sstate-cache', sudo=True)
+        self.deletetmp(build_dir)
+
+        start = time.time()
+        self.perform_build_test(targets, 1, None)
+        first_time = time.time() - start
+        self.log.info('Non-cached build:' + str(round(first_time, 2)))
+
+        # Cleanup everything but cache files
+        self.deletetmp(build_dir)
+
+        start = time.time()
+        self.perform_build_test(targets, 1, None)
+        second_time = time.time() - start
+        self.log.info('Cached build:' + str(round(second_time, 2)))
+
+        if first_time / second_time < speedup_k:
+            self.fail('No speedup after rebuild with sstate-cache')
-- 
2.20.1

Re: [PATCH] CI: Add sstate-cache testcase

[Jan Kiszka]

On 28.10.21 17:20, Anton Mikanovich wrote:
> Test rebuild time improve after cleanup to be sure sstate-cache works.
> 
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>  testsuite/build_test/build_test.py | 40 ++++++++++++++++++++++++++++--
>  1 file changed, 38 insertions(+), 2 deletions(-)
> 
> diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
> index d39c10c0..244f6fc0 100644
> --- a/testsuite/build_test/build_test.py
> +++ b/testsuite/build_test/build_test.py
> @@ -1,9 +1,9 @@
>  #!/usr/bin/env python3
>  
> -import os
> +import os, time
>  
>  from avocado import skipUnless
> -from avocado.utils import path
> +from avocado.utils import path, process
>  from cibase import CIBaseTest
>  
>  UMOCI_AVAILABLE = True
> @@ -206,3 +206,39 @@ class ContainerSdkTest(CIBaseTest):
>          targets = ['mc:container-amd64-stretch:isar-image-base']
>  
>          self.perform_container_test(targets, 'do_populate_sdk')
> +
> +class SstateTest(CIBaseTest):
> +
> +    """
> +    Test rebuild speed improve with sstate-cache
> +
> +    :avocado: tags=sstate
> +    """
> +    def test_sstate_rebuild(self):
> +        speedup_k = 2.0
> +
> +        targets = [
> +            'mc:qemuamd64-bullseye:isar-image-base'
> +                  ]
> +
> +        # Cleanup everything before build
> +        build_dir = self.params.get('build_dir',
> +                    default=os.path.dirname(__file__) + '/../../build')
> +        process.run('rm -rf ' + build_dir + '/sstate-cache', sudo=True)
> +        self.deletetmp(build_dir)
> +
> +        start = time.time()
> +        self.perform_build_test(targets, 1, None)
> +        first_time = time.time() - start
> +        self.log.info('Non-cached build:' + str(round(first_time, 2)))
> +
> +        # Cleanup everything but cache files
> +        self.deletetmp(build_dir)
> +
> +        start = time.time()
> +        self.perform_build_test(targets, 1, None)
> +        second_time = time.time() - start
> +        self.log.info('Cached build:' + str(round(second_time, 2)))
> +
> +        if first_time / second_time < speedup_k:
> +            self.fail('No speedup after rebuild with sstate-cache')
> 

Is there no better way than measuring time to test if sstate was in
place? Maybe some hit/miss statistics?

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [PATCH] CI: Add sstate-cache testcase

[Anton Mikanovich]

28.10.2021 18:34, Jan Kiszka wrote:
> On 28.10.21 17:20, Anton Mikanovich wrote:
>> Test rebuild time improve after cleanup to be sure sstate-cache works.
>>
>> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
>> ---
>>   testsuite/build_test/build_test.py | 40 ++++++++++++++++++++++++++++--
>>   1 file changed, 38 insertions(+), 2 deletions(-)
>>
>> diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
>> index d39c10c0..244f6fc0 100644
>> --- a/testsuite/build_test/build_test.py
>> +++ b/testsuite/build_test/build_test.py
>> @@ -1,9 +1,9 @@
>>   #!/usr/bin/env python3
>>   
>> -import os
>> +import os, time
>>   
>>   from avocado import skipUnless
>> -from avocado.utils import path
>> +from avocado.utils import path, process
>>   from cibase import CIBaseTest
>>   
>>   UMOCI_AVAILABLE = True
>> @@ -206,3 +206,39 @@ class ContainerSdkTest(CIBaseTest):
>>           targets = ['mc:container-amd64-stretch:isar-image-base']
>>   
>>           self.perform_container_test(targets, 'do_populate_sdk')
>> +
>> +class SstateTest(CIBaseTest):
>> +
>> +    """
>> +    Test rebuild speed improve with sstate-cache
>> +
>> +    :avocado: tags=sstate
>> +    """
>> +    def test_sstate_rebuild(self):
>> +        speedup_k = 2.0
>> +
>> +        targets = [
>> +            'mc:qemuamd64-bullseye:isar-image-base'
>> +                  ]
>> +
>> +        # Cleanup everything before build
>> +        build_dir = self.params.get('build_dir',
>> +                    default=os.path.dirname(__file__) + '/../../build')
>> +        process.run('rm -rf ' + build_dir + '/sstate-cache', sudo=True)
>> +        self.deletetmp(build_dir)
>> +
>> +        start = time.time()
>> +        self.perform_build_test(targets, 1, None)
>> +        first_time = time.time() - start
>> +        self.log.info('Non-cached build:' + str(round(first_time, 2)))
>> +
>> +        # Cleanup everything but cache files
>> +        self.deletetmp(build_dir)
>> +
>> +        start = time.time()
>> +        self.perform_build_test(targets, 1, None)
>> +        second_time = time.time() - start
>> +        self.log.info('Cached build:' + str(round(second_time, 2)))
>> +
>> +        if first_time / second_time < speedup_k:
>> +            self.fail('No speedup after rebuild with sstate-cache')
>>
> Is there no better way than measuring time to test if sstate was in
> place? Maybe some hit/miss statistics?
>
> Jan
>
I've tried to find some Avocado API to access final testcase statistics 
from within other testcase but didn't find so far.
Any advice is welcome.

-- 
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH] CI: Add sstate-cache testcase

[Henning Schild]

Am Thu, 28 Oct 2021 20:22:58 +0300
schrieb Anton Mikanovich <amikan@ilbers.de>:

> 28.10.2021 18:34, Jan Kiszka wrote:
> > On 28.10.21 17:20, Anton Mikanovich wrote:  
> >> Test rebuild time improve after cleanup to be sure sstate-cache
> >> works.
> >>
> >> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> >> ---
> >>   testsuite/build_test/build_test.py | 40
> >> ++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2
> >> deletions(-)
> >>
> >> diff --git a/testsuite/build_test/build_test.py
> >> b/testsuite/build_test/build_test.py index d39c10c0..244f6fc0
> >> 100644 --- a/testsuite/build_test/build_test.py
> >> +++ b/testsuite/build_test/build_test.py
> >> @@ -1,9 +1,9 @@
> >>   #!/usr/bin/env python3
> >>   
> >> -import os
> >> +import os, time
> >>   
> >>   from avocado import skipUnless
> >> -from avocado.utils import path
> >> +from avocado.utils import path, process
> >>   from cibase import CIBaseTest
> >>   
> >>   UMOCI_AVAILABLE = True
> >> @@ -206,3 +206,39 @@ class ContainerSdkTest(CIBaseTest):
> >>           targets = ['mc:container-amd64-stretch:isar-image-base']
> >>   
> >>           self.perform_container_test(targets, 'do_populate_sdk')
> >> +
> >> +class SstateTest(CIBaseTest):
> >> +
> >> +    """
> >> +    Test rebuild speed improve with sstate-cache
> >> +
> >> +    :avocado: tags=sstate
> >> +    """
> >> +    def test_sstate_rebuild(self):
> >> +        speedup_k = 2.0
> >> +
> >> +        targets = [
> >> +            'mc:qemuamd64-bullseye:isar-image-base'
> >> +                  ]
> >> +
> >> +        # Cleanup everything before build
> >> +        build_dir = self.params.get('build_dir',
> >> +                    default=os.path.dirname(__file__) +
> >> '/../../build')
> >> +        process.run('rm -rf ' + build_dir + '/sstate-cache',
> >> sudo=True)
> >> +        self.deletetmp(build_dir)
> >> +
> >> +        start = time.time()
> >> +        self.perform_build_test(targets, 1, None)
> >> +        first_time = time.time() - start
> >> +        self.log.info('Non-cached build:' + str(round(first_time,
> >> 2))) +
> >> +        # Cleanup everything but cache files
> >> +        self.deletetmp(build_dir)
> >> +
> >> +        start = time.time()
> >> +        self.perform_build_test(targets, 1, None)
> >> +        second_time = time.time() - start
> >> +        self.log.info('Cached build:' + str(round(second_time,
> >> 2))) +
> >> +        if first_time / second_time < speedup_k:
> >> +            self.fail('No speedup after rebuild with
> >> sstate-cache') 
> > Is there no better way than measuring time to test if sstate was in
> > place? Maybe some hit/miss statistics?
> >
> > Jan
> >  
> I've tried to find some Avocado API to access final testcase
> statistics from within other testcase but didn't find so far.
> Any advice is welcome.

sstate hit rates are in the bitbake logs, i hope avocado can open
artifacts

But this is infra ... we will see when it fails and do not have to test
that.

The interesting problems will happen once you build version+5 on a
cache from version. The only known limitiation there is that your base
rootfss must not be too old. Because your local apt-database will start
containing package versions that might not be available to install
anymore.
So you have to find an eviction strategy. But that is very project
specific.

We currently have two strategies in place on two projects happily
working with this since weeks.
  1. nightly clean + full build
  2. "tmpreaper"-style age-greater 3 days

Third one you build against a frozen mirror and never clear, or very
infrequent.

Plus this problem is well known to isar guys, you just have to
sometimes do a full build or "bitbake -C task recipe". It just now
moves to runners ... but only if those are specially configured to
offer persistent storage.

Not sure how jenkins deals with this per default, and what sort of
configuration the ilbers jenkins might need.

Henning

RE: [PATCH] CI: Add sstate-cache testcase

[Schmidt, Adriaan]

Henning Schild, Donnerstag, 28. Oktober 2021 20:22:
> Am Thu, 28 Oct 2021 20:22:58 +0300
> schrieb Anton Mikanovich <amikan@ilbers.de>:
> 
> > 28.10.2021 18:34, Jan Kiszka wrote:
> > > On 28.10.21 17:20, Anton Mikanovich wrote:
> > >> Test rebuild time improve after cleanup to be sure sstate-cache
> > >> works.
> > >>
> > >> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> > >> ---
> > >>   testsuite/build_test/build_test.py | 40
> > >> ++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2
> > >> deletions(-)
> > >>
> > >> diff --git a/testsuite/build_test/build_test.py
> > >> b/testsuite/build_test/build_test.py index d39c10c0..244f6fc0
> > >> 100644 --- a/testsuite/build_test/build_test.py
> > >> +++ b/testsuite/build_test/build_test.py
> > >> @@ -1,9 +1,9 @@
> > >>   #!/usr/bin/env python3
> > >>
> > >> -import os
> > >> +import os, time
> > >>
> > >>   from avocado import skipUnless
> > >> -from avocado.utils import path
> > >> +from avocado.utils import path, process
> > >>   from cibase import CIBaseTest
> > >>
> > >>   UMOCI_AVAILABLE = True
> > >> @@ -206,3 +206,39 @@ class ContainerSdkTest(CIBaseTest):
> > >>           targets = ['mc:container-amd64-stretch:isar-image-base']
> > >>
> > >>           self.perform_container_test(targets, 'do_populate_sdk')
> > >> +
> > >> +class SstateTest(CIBaseTest):
> > >> +
> > >> +    """
> > >> +    Test rebuild speed improve with sstate-cache
> > >> +
> > >> +    :avocado: tags=sstate
> > >> +    """
> > >> +    def test_sstate_rebuild(self):
> > >> +        speedup_k = 2.0
> > >> +
> > >> +        targets = [
> > >> +            'mc:qemuamd64-bullseye:isar-image-base'
> > >> +                  ]
> > >> +
> > >> +        # Cleanup everything before build
> > >> +        build_dir = self.params.get('build_dir',
> > >> +                    default=os.path.dirname(__file__) +
> > >> '/../../build')
> > >> +        process.run('rm -rf ' + build_dir + '/sstate-cache',
> > >> sudo=True)
> > >> +        self.deletetmp(build_dir)
> > >> +
> > >> +        start = time.time()
> > >> +        self.perform_build_test(targets, 1, None)
> > >> +        first_time = time.time() - start
> > >> +        self.log.info('Non-cached build:' + str(round(first_time,
> > >> 2))) +
> > >> +        # Cleanup everything but cache files
> > >> +        self.deletetmp(build_dir)
> > >> +
> > >> +        start = time.time()
> > >> +        self.perform_build_test(targets, 1, None)
> > >> +        second_time = time.time() - start
> > >> +        self.log.info('Cached build:' + str(round(second_time,
> > >> 2))) +
> > >> +        if first_time / second_time < speedup_k:
> > >> +            self.fail('No speedup after rebuild with
> > >> sstate-cache')
> > > Is there no better way than measuring time to test if sstate was in
> > > place? Maybe some hit/miss statistics?
> > >
> > > Jan
> > >
> > I've tried to find some Avocado API to access final testcase
> > statistics from within other testcase but didn't find so far.
> > Any advice is welcome.
> 
> sstate hit rates are in the bitbake logs, i hope avocado can open
> artifacts

You want to look for the line "Sstate summary", which is shown at the beginning of the bitbake run, e.g.:

Sstate summary: Wanted 121 Local 121 Network 0 Missed 0 Current 0 (100% match, 0% complete)

But in a test it's probably better to examine log.task_order and check which tasks were executed.

> But this is infra ... we will see when it fails and do not have to test
> that.

It's worth running some tests to see that builds succeed with artifacts taken from cache. I can imagine something like:

- Build image to populate cache
- Remove TMPDIR
- Rebuild the image. Should have 100% cache hits. (This would take the target rootfs from cache, and the buildchroot required for image-building)
- Remove TMPDIR
- Build a single package (that was already contained in the image). Should have a cache hit, and only dpkg_build_setscene and deploy_deb should run (plus minimal steps to initialize isar-apt).

And optionally:
- Remove TMPDIR
- Force rebuild of one package (e.g. by deleting its sstate artifact)
- Rebuild the image (should take the buildchroot and all but the one package from cache, and rebuild the one package and the image)

> The interesting problems will happen once you build version+5 on a
> cache from version. The only known limitiation there is that your base
> rootfss must not be too old. Because your local apt-database will start
> containing package versions that might not be available to install
> anymore.

At least IMO this is limitation is unrelated to the sstate cache, and I agree that testing such cases doesn't make much sense.

I'm happy to take over and propose a test case.

Adriaan

Re: [PATCH] CI: Add sstate-cache testcase

[Jan Kiszka]

On 02.11.21 06:57, Schmidt, Adriaan (T RDA IOT SES-DE) wrote:
> Henning Schild, Donnerstag, 28. Oktober 2021 20:22:
>> Am Thu, 28 Oct 2021 20:22:58 +0300
>> schrieb Anton Mikanovich <amikan@ilbers.de>:
>>
>>> 28.10.2021 18:34, Jan Kiszka wrote:
>>>> On 28.10.21 17:20, Anton Mikanovich wrote:
>>>>> Test rebuild time improve after cleanup to be sure sstate-cache
>>>>> works.
>>>>>
>>>>> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
>>>>> ---
>>>>>   testsuite/build_test/build_test.py | 40
>>>>> ++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2
>>>>> deletions(-)
>>>>>
>>>>> diff --git a/testsuite/build_test/build_test.py
>>>>> b/testsuite/build_test/build_test.py index d39c10c0..244f6fc0
>>>>> 100644 --- a/testsuite/build_test/build_test.py
>>>>> +++ b/testsuite/build_test/build_test.py
>>>>> @@ -1,9 +1,9 @@
>>>>>   #!/usr/bin/env python3
>>>>>
>>>>> -import os
>>>>> +import os, time
>>>>>
>>>>>   from avocado import skipUnless
>>>>> -from avocado.utils import path
>>>>> +from avocado.utils import path, process
>>>>>   from cibase import CIBaseTest
>>>>>
>>>>>   UMOCI_AVAILABLE = True
>>>>> @@ -206,3 +206,39 @@ class ContainerSdkTest(CIBaseTest):
>>>>>           targets = ['mc:container-amd64-stretch:isar-image-base']
>>>>>
>>>>>           self.perform_container_test(targets, 'do_populate_sdk')
>>>>> +
>>>>> +class SstateTest(CIBaseTest):
>>>>> +
>>>>> +    """
>>>>> +    Test rebuild speed improve with sstate-cache
>>>>> +
>>>>> +    :avocado: tags=sstate
>>>>> +    """
>>>>> +    def test_sstate_rebuild(self):
>>>>> +        speedup_k = 2.0
>>>>> +
>>>>> +        targets = [
>>>>> +            'mc:qemuamd64-bullseye:isar-image-base'
>>>>> +                  ]
>>>>> +
>>>>> +        # Cleanup everything before build
>>>>> +        build_dir = self.params.get('build_dir',
>>>>> +                    default=os.path.dirname(__file__) +
>>>>> '/../../build')
>>>>> +        process.run('rm -rf ' + build_dir + '/sstate-cache',
>>>>> sudo=True)
>>>>> +        self.deletetmp(build_dir)
>>>>> +
>>>>> +        start = time.time()
>>>>> +        self.perform_build_test(targets, 1, None)
>>>>> +        first_time = time.time() - start
>>>>> +        self.log.info('Non-cached build:' + str(round(first_time,
>>>>> 2))) +
>>>>> +        # Cleanup everything but cache files
>>>>> +        self.deletetmp(build_dir)
>>>>> +
>>>>> +        start = time.time()
>>>>> +        self.perform_build_test(targets, 1, None)
>>>>> +        second_time = time.time() - start
>>>>> +        self.log.info('Cached build:' + str(round(second_time,
>>>>> 2))) +
>>>>> +        if first_time / second_time < speedup_k:
>>>>> +            self.fail('No speedup after rebuild with
>>>>> sstate-cache')
>>>> Is there no better way than measuring time to test if sstate was in
>>>> place? Maybe some hit/miss statistics?
>>>>
>>>> Jan
>>>>
>>> I've tried to find some Avocado API to access final testcase
>>> statistics from within other testcase but didn't find so far.
>>> Any advice is welcome.
>>
>> sstate hit rates are in the bitbake logs, i hope avocado can open
>> artifacts
> 
> You want to look for the line "Sstate summary", which is shown at the beginning of the bitbake run, e.g.:
> 
> Sstate summary: Wanted 121 Local 121 Network 0 Missed 0 Current 0 (100% match, 0% complete)
> 
> But in a test it's probably better to examine log.task_order and check which tasks were executed.
> 
>> But this is infra ... we will see when it fails and do not have to test
>> that.
> 
> It's worth running some tests to see that builds succeed with artifacts taken from cache. I can imagine something like:
> 
> - Build image to populate cache
> - Remove TMPDIR
> - Rebuild the image. Should have 100% cache hits. (This would take the target rootfs from cache, and the buildchroot required for image-building)
> - Remove TMPDIR
> - Build a single package (that was already contained in the image). Should have a cache hit, and only dpkg_build_setscene and deploy_deb should run (plus minimal steps to initialize isar-apt).
> 
> And optionally:
> - Remove TMPDIR
> - Force rebuild of one package (e.g. by deleting its sstate artifact)
> - Rebuild the image (should take the buildchroot and all but the one package from cache, and rebuild the one package and the image)
> 

BTW, we likely now also need a cleansstate target, like OE has.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux