[RFC v2 0/3] PoC for isar-apt repo reusing

[RFC v2 0/3] PoC for isar-apt repo reusing

[Uladzimir Bely]

Currently, isar-apt repo is always rebuilt at second build when
temporary files except the repo directory were deleted.

This patchset implements proof-of-concept of reusing isar-apt
repo previously built. The idea is to check that package already
exists in the repo and early exit tasks related to fetching/building.

Such approach doesn't remove any tasks from the queue, but changes
their behvaior (early exit) in case of existing package found. 

Using this feature doesn't depend on sstate-cache and requires only
local repo from some of previous builds to exist.

Changes since v1:
- avocado test, not based on build time measure
- fixed isar-apt check in compat-arch mode

Uladzimir Bely (3):
  meta: Reuse existing local isar-apt repo in new builds
  doc: Add section for isar-apt reuse functionality
  ci: Added test for isar-apt reuse functionality

 doc/user_manual.md                  | 23 ++++++++++
 meta-isar/conf/local.conf.sample    |  4 ++
 meta/classes/dpkg-base.bbclass      |  1 +
 meta/classes/isar-apt-cache.bbclass | 69 +++++++++++++++++++++++++++++
 testsuite/cibase.py                 | 42 ++++++++++++++++++
 testsuite/cibuilder.py              |  7 ++-
 testsuite/citest.py                 | 12 +++++
 7 files changed, 157 insertions(+), 1 deletion(-)
 create mode 100644 meta/classes/isar-apt-cache.bbclass

-- 
2.20.1

[RFC v2 1/3] meta: Reuse existing local isar-apt repo in new builds

[Uladzimir Bely]

The idea of reusing isar-apt repo at second build is to early
exit from build-related tasks if the package with the same
version already exists in current isar-apt repo.

Also, redefining default values of REPO_ISAR_DIR and REPO_ISAR_DB_DIR
maybe useful to share isar-apt between different builds and not to
keep it under ${TMPDIR}:

REPO_ISAR_DIR = "/path/to/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt"
REPO_ISAR_DB_DIR = "/path/to/isar-apt/${DISTRO}-${DISTRO_ARCH}/db"

The variable ISAR_APT_REUSE in local.conf controls the behvaiour:

0: Don't use isar-apt cache, always rebuilt packages (default)
1: Use local isar-apt cache, rebuilt packages only if needed

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta-isar/conf/local.conf.sample    |  4 ++
 meta/classes/dpkg-base.bbclass      |  1 +
 meta/classes/isar-apt-cache.bbclass | 69 +++++++++++++++++++++++++++++
 3 files changed, 74 insertions(+)
 create mode 100644 meta/classes/isar-apt-cache.bbclass

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 58f3e1a2..523da5c3 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -221,6 +221,10 @@ ISAR_CROSS_COMPILE ?= "0"
 # does not access the network
 #BB_NO_NETWORK ?= "1"
 
+#
+# Uncomment to use previously built isar_apt repo if package version hasn't changed
+#ISAR_APT_REUSE ?= "1"
+
 # Set root password to 'root'
 # Password was encrypted using following command:
 #   mkpasswd -m sha512crypt -R 10000
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 7e032ba3..a9697fbd 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -9,6 +9,7 @@ inherit debianize
 inherit terminal
 inherit repository
 inherit deb-dl-dir
+inherit isar-apt-cache
 
 DEPENDS ?= ""
 
diff --git a/meta/classes/isar-apt-cache.bbclass b/meta/classes/isar-apt-cache.bbclass
new file mode 100644
index 00000000..4550886d
--- /dev/null
+++ b/meta/classes/isar-apt-cache.bbclass
@@ -0,0 +1,69 @@
+# This software is a part of ISAR.
+# Copyright (C) 2017-2019 Siemens AG
+# Copyright (C) 2019 ilbers GmbH
+#
+# SPDX-License-Identifier: MIT
+
+iac_check_package_in_repo() {
+    DISTS_ISAR="${REPO_ISAR_DIR}/${DISTRO}/dists/${DEBDISTRONAME}"
+    PACKAGES="${DISTS_ISAR}/main/binary-${PACKAGE_ARCH}/Packages"
+
+    if [ ! -f "${PACKAGES}" ]; then
+        return 1
+    fi
+
+    if ! grep -q "^\(Package\|Source\): ${PN}$" "${PACKAGES}"; then
+        return 2
+    fi
+
+    local repo_version=$(sed -n -e '/^\(Package\|Source\): ${PN}$/,/Version/p' ${PACKAGES} | grep '^Version' | sed -e 's/.* //' | head -n1)
+    local new_version=$(head -n1 ${S}/debian/changelog | sed -e 's/.* (\(.*\)) .*/\1/' | head -n1)
+
+    if [ "${repo_version}" != "${new_version}" ]; then
+        return 3
+    fi
+
+    bbnote "Package ${PN}-${PV} is already in isar-apt repo, reuse it"
+    return 0
+}
+
+iac_check_need_rebuild() {
+    if [ "${ISAR_APT_REUSE}" != "1" ]; then
+       return 1
+    fi
+
+    iac_check_package_in_repo
+}
+
+def iac_check_task_early_exit(d):
+    try:
+        bb.build.exec_func("iac_check_need_rebuild", d)
+        return True
+    except:
+        return False
+
+
+do_apt_fetch_prepend() {
+    iac_check_need_rebuild && return
+}
+
+do_apt_unpack_prepend() {
+    iac_check_need_rebuild && return
+}
+
+do_prepare_build_prepend() {
+    iac_check_need_rebuild && return
+}
+
+do_deploy_deb_prepend() {
+    iac_check_need_rebuild && return
+}
+
+do_install_builddeps_prepend() {
+    iac_check_need_rebuild && return
+}
+
+python do_dpkg_build_prepend() {
+    if iac_check_task_early_exit(d):
+        return 0
+}
-- 
2.20.1

[RFC v2 2/3] doc: Add section for isar-apt reuse functionality

[Uladzimir Bely]

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 doc/user_manual.md | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 02874b6d..80f79ddd 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -24,6 +24,7 @@ Copyright (C) 2016-2019, ilbers GmbH
  - [Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)
  - [Create a containerized Isar SDK root filesystem](#create-a-containerized-isar-sdk-root-filesystem)
  - [Creation of local apt repo caching upstream Debian packages](#creation-of-local-apt-repo-caching-upstream-debian-packages)
+ - [Reusing isar-apt repository](#reusing-isar-apt-repository)
 
 
 ## Introduction
@@ -1254,6 +1255,28 @@ sudo rm -rf tmp
 bitbake mc:qemuarm-buster:isar-image-base
 ```
 
+## Reusing isar-apt repository
+
+### Motivation
+
+Building custom recipes from sources may take a lot of time to download the sources and build the package. Also, reusing isar-apt makes it possible to work offline.
+
+### Solution
+
+After a custom package is built, it's placed into isar-apt repository. It's possible to reuse this repository for further builds. The behavior is controlled by `ISAR_APT_REUSE` variable in `conf/local.conf`:
+
+- `ISAR_APT_REUSE = "0"` - always rebuild custom packages from sources, even if they already exist in the repository
+- `ISAR_APT_REUSE = "1"` - reuse previously built binary if it exists in repository with the same version
+
+If `ISAR_APT_REUSE` is not set it behaves like `"0"`
+
+It also may be useful to move `isar-apt` directory out from `tmp/deploy` so that it could be easily shared between builds. To do that, modify `conf/local.conf` by adding lines similar to the following ones:
+
+```
+REPO_ISAR_DIR = "/path/to/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt"
+REPO_ISAR_DB_DIR = "/path/to/isar-apt/${DISTRO}-${DISTRO_ARCH}/db"
+```
+
 ## Add foreign packages from other repositories to the generated image
 
 ### Motivation
-- 
2.20.1

[RFC v2 3/3] ci: Added test for isar-apt reuse functionality

[Uladzimir Bely]

Test makes clean build of "libhello" package that is placed to local
isar-apt repo outside of 'tmp'.

Second build with cleaned 'tmp' should use previously built package
that is detected by missing Makefile in source directory.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/cibase.py    | 42 ++++++++++++++++++++++++++++++++++++++++++
 testsuite/cibuilder.py |  7 ++++++-
 testsuite/citest.py    | 12 ++++++++++++
 3 files changed, 60 insertions(+), 1 deletion(-)

diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index 2ffb8191..9ef9d247 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -73,6 +73,48 @@ class CIBaseTest(CIBuilder):
         self.delete_from_build_dir('tmp')
         self.delete_from_build_dir('ccache')
 
+    def perform_isarapt_test(self, package_target, **kwargs):
+        def check_compiled(package_name):
+            # Makefile should exist if there was compilation
+            makefile = glob.glob(f'{self.build_dir}/tmp/work/*/{package_name}/*/{package_name}-*/Makefile')
+            if makefile:
+                self.log.info("Makefile was generated")
+                return True
+
+            self.log.info("Makefile was not generated")
+            return False
+
+        self.configure(isar_apt_reuse=True, **kwargs)
+
+        package_name = package_target.split(':')[-1]
+
+        # Cleanup sstate and tmp before test`
+        self.delete_from_build_dir('tmp')
+        self.delete_from_build_dir('sstate-cache')
+        self.delete_from_build_dir('isar-apt')
+
+        self.log.info('Start build and populate isar-apt...')
+        self.bitbake(package_target, **kwargs)
+        compiled1 = check_compiled(package_name)
+
+        if not compiled1:
+            self.fail('isar-apt repo used at first build')
+
+        # Cleanup build stuff
+        self.delete_from_build_dir('tmp')
+        self.delete_from_build_dir('sstate-cache')
+
+        self.log.info('Starting build and reuse isar-apt repo...')
+        self.bitbake(package_target, **kwargs)
+        compiled2 = check_compiled(package_name)
+
+        if compiled2:
+            self.fail('isar-apt repo not used at second build')
+
+        # Cleanup
+        self.delete_from_build_dir('tmp')
+        self.delete_from_build_dir('sstate-cache')
+
     def perform_sstate_test(self, image_target, package_target, **kwargs):
         def check_executed_tasks(target, expected):
             taskorder_file = glob.glob(f'{self.build_dir}/tmp/work/*/{target}/*/temp/log.task_order')
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index bc48d47f..42e344a3 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -54,7 +54,7 @@ class CIBuilder(Test):
 
     def configure(self, compat_arch=True, cross=None, debsrc_cache=False,
                   container=False, ccache=False, sstate=False, offline=False,
-                  gpg_pub_key=None, **kwargs):
+                  isar_apt_reuse=False, gpg_pub_key=None, **kwargs):
         # write configuration file and set bitbake_args
         # can run multiple times per test case
         self.check_init()
@@ -76,6 +76,7 @@ class CIBuilder(Test):
                       f'  container = {container}\n'
                       f'  ccache = {ccache}\n'
                       f'  sstate = {sstate}\n'
+                      f'  isar_apt_reuse = {isar_apt_reuse}\n'
                       f'  gpg_pub_key = {gpg_pub_key}\n'
                       f'===================================================')
 
@@ -107,6 +108,10 @@ class CIBuilder(Test):
                 f.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
             if distro_apt_premir:
                 f.write('DISTRO_APT_PREMIRRORS = "%s"\n' % distro_apt_premir)
+            if isar_apt_reuse:
+                f.write('ISAR_APT_REUSE = "1"\n')
+                f.write('REPO_ISAR_DIR = "${TOPDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt"\n')
+                f.write('REPO_ISAR_DB_DIR = "${TOPDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/db"\n')
 
         # include ci_build.conf in local.conf
         with open(self.build_dir + '/conf/local.conf', 'r+') as f:
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 16e38d07..7545d9e6 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -60,6 +60,18 @@ class CcacheTest(CIBaseTest):
         self.init()
         self.perform_ccache_test(targets)
 
+class IsarAptTest(CIBaseTest):
+
+    """
+    Test rebuild speed improve with isar-apt reuse enabled
+
+    :avocado: tags=isarapt,full
+    """
+    def test_isarapt_reuse(self):
+        target = 'mc:qemuamd64-bullseye:libhello'
+        self.init()
+        self.perform_isarapt_test(target)
+
 class CrossTest(CIBaseTest):
 
     """
-- 
2.20.1