[RFC PATCH 0/2] check Isar image for reproducability

[RFC PATCH 0/2] check Isar image for reproducability

[Uladzimir Bely]

This is an reimplementation of the idea proposed in
https://groups.google.com/g/isar-users/c/4ZIuKCOQzVc/m/sPUafWDgAgAJ
that uses avocado test instead of custom script.

The test is placed under KFAIL while images are even not close to
be considered reproducable for now.

Some things to improve the patchset:
 - Check not only `tar.gz` images, but target rootfs directories,
so it should save time and allow supporting various multiconfigs.
 - Not only generate diff statistics, but also analyze "total"
number of differencies we see.
 - ...

Uladzimir Bely (2):
  Install diffoscope binary if not installed
  testsuite: check Isar image for reproducability.

 scripts/ci_build.sh |  5 +++++
 testsuite/cibase.py | 44 ++++++++++++++++++++++++++++++++++++++++++++
 testsuite/citest.py | 18 ++++++++++++++++++
 3 files changed, 67 insertions(+)

-- 
2.20.1

[PATCH 1/2] Install diffoscope binary if not installed

[Uladzimir Bely]

New `reproducable` test requires `diffoscope` utility to be
installed on the machine running CI.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 scripts/ci_build.sh | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 820a5609..f2f908d6 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -23,6 +23,11 @@ if ! command -v avocado > /dev/null; then
     source /tmp/avocado_venv/bin/activate
     pip install avocado-framework==96.0
 fi
+# install diffoscope package if not yet installed
+if ! command -v diffoscope > /dev/null; then
+    sudo apt-get update -qq
+    sudo apt-get install -y --no-install-recommends diffoscope
+fi
 
 # Get Avocado build tests path
 TESTSUITE_DIR="$(pwd)/testsuite"
-- 
2.20.1

[PATCH 2/2] testsuite: check Isar image for reproducability.

[Uladzimir Bely]

This test uses 'diffoscope'[1] tool to check the difference between
two consecutive builds and copy the differeneces in html format.

[1] https://diffoscope.org/

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/cibase.py | 44 ++++++++++++++++++++++++++++++++++++++++++++
 testsuite/citest.py | 18 ++++++++++++++++++
 2 files changed, 62 insertions(+)

diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index 6239b4de..8a6d4c6e 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -2,6 +2,7 @@
 
 import glob
 import os
+import subprocess
 import re
 import tempfile
 import time
@@ -55,6 +56,49 @@ class CIBaseTest(CIBuilder):
             # Try to build with changed configuration with no cleanup
             self.bitbake(targets, **kwargs)
 
+    def perform_reproducable_test(self, targets, **kwargs):
+        def get_bitbake_var(output, var):
+            ret = ''
+            for line in output.splitlines():
+                if line.startswith(var + '='):
+                    ret = line.split('"')[1]
+            return ret
+
+        self.configure(**kwargs)
+
+        self.log.info('Starting first build...')
+        self.bitbake(targets, **kwargs)
+
+        self.move_in_build_dir('tmp', 'tmp1')
+        self.delete_from_build_dir('sstate-cache')
+        self.delete_from_build_dir('ccache')
+
+        self.log.info('Starting second build...')
+        self.bitbake(targets, **kwargs)
+        self.bitbake(targets, **kwargs)
+
+        self.move_in_build_dir('tmp', 'tmp2')
+        self.delete_from_build_dir('sstate-cache')
+        self.delete_from_build_dir('ccache')
+
+        for target in targets:
+            output = subprocess.check_output(['bitbake', '-e', target]).decode()
+
+            machine=get_bitbake_var(output, 'MACHINE')
+            distro=get_bitbake_var(output, 'DISTRO')
+
+            cmdline = ['diffoscope']
+            cmdline.extend(['--html-dir', 'diffoscope_out'])
+            cmdline.extend(['tmp1/deploy/images/' + machine + '/isar-image-base-' + distro + '-' + machine + '.tar.gz'])
+            cmdline.extend(['tmp2/deploy/images/' + machine + '/isar-image-base-' + distro + '-' + machine + '.tar.gz'])
+
+            print(cmdline)
+
+            p1 = subprocess.call(cmdline)
+
+            if p1.returncode:
+                self.fail('Image is not reproducable')
+
     def perform_ccache_test(self, targets, **kwargs):
         def ccache_stats(dir, field):
             # Look ccache source's 'src/core/Statistic.hpp' for field meanings
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 2dc78015..aa63e848 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -48,6 +48,24 @@ class ReproTest(CIBaseTest):
         finally:
             self.move_in_build_dir('tmp', 'tmp_repro_unsigned')
 
+class ReproducableTest(CIBaseTest):
+
+    """
+    Test cached base repository
+
+    :avocado: tags=reproducable
+    """
+    def test_reproducable(self):
+        targets = [
+            'mc:qemuamd64-bullseye:isar-image-base'
+                  ]
+
+        self.init()
+        try:
+            self.perform_reproducable_test(targets)
+        except:
+            self.cancel('KFAIL')
+
 class CcacheTest(CIBaseTest):
 
     """
-- 
2.20.1