[PATCH 0/2] Rework and enable ccache test

[PATCH 0/2] Rework and enable ccache test

[Uladzimir Bely]

This series makes ccache test use internal ccache statistics instead
of test time measurements.

Ccache stats files are:
    CCACHE_DIR/*/stats
    CCACHE_DIR/*/*/stats

Line 22 in this files keeps corresponds to 'direct_cache_hit' value.
Summing up the values from all files, we can receive the same result
as 'ccache -s' gifes for 'Summary - Hits - Direct'.

Uladzimir Bely (2):
  ci: Rework ccache test
  ci: Add ccache test to fast and full test groups

 testsuite/cibase.py    | 33 ++++++++++++++++++++++++---------
 testsuite/cibuilder.py |  3 +++
 testsuite/citest.py    |  4 ++--
 3 files changed, 29 insertions(+), 11 deletions(-)

-- 
2.20.1

[PATCH 1/2] ci: Rework ccache test

[Uladzimir Bely]

Previous ccache test implementation was based on comparing difference
between non-cached and cached build time. It is not reliable enough
because results may depend on overall CPU/disk load.

New implementation analyzes ccache statistics provided by */*/stats
files in ccache directory. Field with index 22 is used to check hits.

To reduce test time, we compile only `hello-isar` (and `libhello` as
the dependency) amd64 target during the test.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/cibase.py    | 33 ++++++++++++++++++++++++---------
 testsuite/cibuilder.py |  3 +++
 testsuite/citest.py    |  2 +-
 3 files changed, 28 insertions(+), 10 deletions(-)

diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index 2ffb8191..722d7bd2 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -46,32 +46,47 @@ class CIBaseTest(CIBuilder):
             self.bitbake(targets, **kwargs)
 
     def perform_ccache_test(self, targets, **kwargs):
+        def ccache_stats(dir, field):
+            # Look ccache source's 'src/core/Statistic.hpp' for field meanings
+            count = 0
+            for filename in glob.iglob(dir + '/**/stats', recursive=True):
+                if os.path.isfile(filename):
+                    with open(filename,'r') as file:
+                        content = file.readlines()
+                        if (field < len(content)):
+                            count += int(content[field])
+            return count
+
         self.configure(ccache=True, **kwargs)
 
+        # Field that stores direct ccache hits
+        direct_cache_hit = 22
+
         self.delete_from_build_dir('tmp')
+        self.delete_from_build_dir('sstate-cache')
         self.delete_from_build_dir('ccache')
 
         self.log.info('Starting build and filling ccache dir...')
-        start = time.time()
         self.bitbake(targets, **kwargs)
-        first_time = time.time() - start
-        self.log.info('Non-cached build: ' + str(round(first_time)) + 's')
+        hit1 = ccache_stats(self.build_dir + '/ccache', direct_cache_hit)
+        self.log.info('Ccache hits 1: ' + str(hit1))
 
         self.delete_from_build_dir('tmp')
+        self.delete_from_build_dir('sstate-cache')
 
         self.log.info('Starting build and using ccache dir...')
-        start = time.time()
         self.bitbake(targets, **kwargs)
-        second_time = time.time() - start
-        self.log.info('Cached build: ' + str(round(second_time)) + 's')
+        hit2 = ccache_stats(self.build_dir + '/ccache', direct_cache_hit)
+        self.log.info('Ccache hits 2: ' + str(hit2))
 
-        speedup_k = 1.1
-        if first_time / second_time < speedup_k:
-            self.fail('No speedup after rebuild with ccache')
+        if hit2 <= hit1:
+            self.fail('Ccache was not used on second build')
 
         # Cleanup
         self.delete_from_build_dir('tmp')
+        self.delete_from_build_dir('sstate-cache')
         self.delete_from_build_dir('ccache')
+        self.unconfigure()
 
     def perform_sstate_test(self, image_target, package_target, **kwargs):
         def check_executed_tasks(target, expected):
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index bc48d47f..dfb0a376 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -107,6 +107,9 @@ class CIBuilder(Test):
                 f.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
             if distro_apt_premir:
                 f.write('DISTRO_APT_PREMIRRORS = "%s"\n' % distro_apt_premir)
+            if ccache:
+                f.write('USE_CCACHE = "1"\n')
+                f.write('CCACHE_TOP_DIR = "${TOPDIR}/ccache"\n')
 
         # include ci_build.conf in local.conf
         with open(self.build_dir + '/conf/local.conf', 'r+') as f:
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 16e38d07..9135475e 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -56,7 +56,7 @@ class CcacheTest(CIBaseTest):
     :avocado: tags=ccache
     """
     def test_ccache_rebuild(self):
-        targets = ['mc:de0-nano-soc-bullseye:isar-image-base']
+        targets = ['mc:qemuamd64-bullseye:hello-isar']
         self.init()
         self.perform_ccache_test(targets)
 
-- 
2.20.1

[PATCH 2/2] ci: Add ccache test to fast and full test groups

[Uladzimir Bely]

Ccache test takes less then 10 minutes to check on decent hardware so
we can use it both in fast and full groups without big overall test
time increase.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/citest.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 9135475e..6c8eb26e 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -53,7 +53,7 @@ class CcacheTest(CIBaseTest):
     """
     Test rebuild speed improve with ccache
 
-    :avocado: tags=ccache
+    :avocado: tags=ccache,fast,full
     """
     def test_ccache_rebuild(self):
         targets = ['mc:qemuamd64-bullseye:hello-isar']
-- 
2.20.1

Re: [PATCH 0/2] Rework and enable ccache test

[Anton Mikanovich]

07.06.2022 08:24, Uladzimir Bely wrote:
> This series makes ccache test use internal ccache statistics instead
> of test time measurements.
>
> Ccache stats files are:
>      CCACHE_DIR/*/stats
>      CCACHE_DIR/*/*/stats
>
> Line 22 in this files keeps corresponds to 'direct_cache_hit' value.
> Summing up the values from all files, we can receive the same result
> as 'ccache -s' gifes for 'Summary - Hits - Direct'.
>
> Uladzimir Bely (2):
>    ci: Rework ccache test
>    ci: Add ccache test to fast and full test groups
>
>   testsuite/cibase.py    | 33 ++++++++++++++++++++++++---------
>   testsuite/cibuilder.py |  3 +++
>   testsuite/citest.py    |  4 ++--
>   3 files changed, 29 insertions(+), 11 deletions(-)
>
Applied to next, thanks.