[RFC 0/4] PoC for isar-apt repo reusing

[RFC 0/4] PoC for isar-apt repo reusing

[Uladzimir Bely]

Currently, isar-apt repo is always rebuilt at second build when
temporary files except the repo directory were deleted.

This patchset implements proof-of-concept of reusing isar-apt 
repo previously built. The idea is to check that package already 
exists in the repo and early exit tasks related to fetching/building.

Such approach doesn't remove any tasks from the queue, but changes
their behvaior (early exit) in case of existing package found.

Uladzimir Bely (4):
  meta-isar: Fix do_dpkg_build override for prebuild-deb recipe.
  isar-apt: PoC of reusing isar-apt repo
  ci: Test for isar-apt reuse
  doc: Add section for isar-apt reuse functionality

 doc/user_manual.md                            | 23 ++++++
 meta-isar/conf/local.conf.sample              |  4 ++
 .../prebuilt-deb/prebuilt-deb_0.1.bb          |  2 +-
 meta/classes/dpkg-base.bbclass                |  1 +
 meta/classes/isar-apt-cache.bbclass           | 72 +++++++++++++++++++
 scripts/ci_build.sh                           | 10 +++
 6 files changed, 111 insertions(+), 1 deletion(-)
 create mode 100644 meta/classes/isar-apt-cache.bbclass

-- 
2.20.1

[RFC 1/4] meta-isar: Fix do_dpkg_build override for prebuild-deb recipe.

[Uladzimir Bely]

While original dpkg_build task in dpkg-base bbclass uses python syntax,
any overrides in recipes also should us it. Otherwise, bitbake error
is caused in case of prepends for this task.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta-isar/recipes-app/prebuilt-deb/prebuilt-deb_0.1.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta-isar/recipes-app/prebuilt-deb/prebuilt-deb_0.1.bb b/meta-isar/recipes-app/prebuilt-deb/prebuilt-deb_0.1.bb
index 2f1b699..5e2cc77 100644
--- a/meta-isar/recipes-app/prebuilt-deb/prebuilt-deb_0.1.bb
+++ b/meta-isar/recipes-app/prebuilt-deb/prebuilt-deb_0.1.bb
@@ -16,5 +16,5 @@ SRC_URI = "file://example-prebuilt_1.0.0-0_all.deb;unpack=false"
 # packages are provided by a single recipe.
 PROVIDES += "example-prebuilt"
 
-do_dpkg_build() {
+python do_dpkg_build() {
 }
-- 
2.20.1

[RFC 2/4] isar-apt: PoC of reusing isar-apt repo

[Uladzimir Bely]

The idea of reusing isar-apt repo at second build is to early
exit from build-related tasks if the package with the same
version already exists in current isar-apt.

Also, redefining default values of REPO_ISAR_DIR and REPO_ISAR_DB_DIR
maybe useful to share isar-apt between different builds

REPO_ISAR_DIR = "/path/to/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt"
REPO_ISAR_DB_DIR = "/path/to/isar-apt/${DISTRO}-${DISTRO_ARCH}/db"

The variable ISAR_APT_ALWAYS_REBUILD_ON_CHANGE in local.conf
allow to control the behvaiour:

0: Use the cache, rebuilt only if needed
1: Don't use cache, rebuilt packages always (old behaviour)

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta-isar/conf/local.conf.sample    |  4 ++
 meta/classes/dpkg-base.bbclass      |  1 +
 meta/classes/isar-apt-cache.bbclass | 72 +++++++++++++++++++++++++++++
 3 files changed, 77 insertions(+)
 create mode 100644 meta/classes/isar-apt-cache.bbclass

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 96a8beb..ae1ba45 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -201,6 +201,10 @@ ISAR_CROSS_COMPILE ?= "0"
 # does not access the network
 #BB_NO_NETWORK ?= "1"
 
+#
+# Comment or set to 0 to use previously built isar_apt repo if package version hasn't changed
+ISAR_APT_ALWAYS_REBUILD_ON_CHANGE ?= "1"
+
 # Set root password to 'root'
 # Password was encrypted using following command:
 #   mkpasswd -m sha512crypt -R 10000
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index aa529ca..59113b3 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -9,6 +9,7 @@ inherit debianize
 inherit terminal
 inherit repository
 inherit deb-dl-dir
+inherit isar-apt-cache
 
 DEPENDS ?= ""
 
diff --git a/meta/classes/isar-apt-cache.bbclass b/meta/classes/isar-apt-cache.bbclass
new file mode 100644
index 0000000..c9533f0
--- /dev/null
+++ b/meta/classes/isar-apt-cache.bbclass
@@ -0,0 +1,72 @@
+# This software is a part of ISAR.
+# Copyright (C) 2017-2019 Siemens AG
+# Copyright (C) 2019 ilbers GmbH
+#
+# SPDX-License-Identifier: MIT
+
+iac_check_package_in_repo() {
+    DISTS_ISAR="${REPO_ISAR_DIR}/${DISTRO}/dists/${DEBDISTRONAME}"
+    PACKAGES="${DISTS_ISAR}/main/binary-${DISTRO_ARCH}/Packages"
+
+    if [ ! -f "${PACKAGES}" ]; then
+        return 1
+    fi
+
+    if ! grep -q "^Package: ${PN}$" "${PACKAGES}"; then
+        return 2
+    fi
+
+    local repo_version=$(sed -n -e '/^Package: ${PN}$/,/Version/p' ${PACKAGES} | grep '^Version' | sed -e 's/.* //')
+    local new_version=$(head -n1 ${S}/debian/changelog | sed -e 's/.* (\(.*\)) .*/\1/')
+
+    if [ "${repo_version}" != "${new_version}" ]; then
+        return 3
+    fi
+
+    bbnote "Package ${PN}-${PV} is already in isar-apt repo, reuse it"
+    return 0
+}
+
+iac_check_need_rebuild() {
+    if [ "${ISAR_APT_ALWAYS_REBUILD_ON_CHANGE}" = "1" ]; then
+       return 1
+    fi
+
+    iac_check_package_in_repo
+}
+
+def iac_check_task_early_exit():
+    try:
+        bb.build.exec_func("iac_check_need_rebuild", d)
+        return True
+    except:
+        return False
+
+
+python do_apt_fetch_prepend() {
+    if iac_check_task_early_exit():
+        return 0
+}
+
+python do_apt_unpack_prepend() {
+    if iac_check_task_early_exit():
+        return 0
+}
+
+do_prepare_build_prepend() {
+    iac_check_need_rebuild && return
+}
+
+do_deploy_deb_prepend() {
+    iac_check_need_rebuild && return
+}
+
+python do_install_builddeps_prepend() {
+    if iac_check_task_early_exit():
+        return 0
+}
+
+python do_dpkg_build_prepend() {
+    if iac_check_task_early_exit():
+        return 0
+}
-- 
2.20.1

[RFC 3/4] ci: Test for isar-apt reuse

[Uladzimir Bely]

The approach: after first build of some target can remove
all temporary files, leaving tmp/deploy/isar-apt. Second
build should use packages from it without rebuilding.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 scripts/ci_build.sh | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 837cd67..e62c02a 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -203,8 +203,18 @@ if [ -n "$REPRO_BUILD" ]; then
     sed -i -e 's/^#BB_NO_NETWORK/BB_NO_NETWORK/g' conf/local.conf
     bitbake $BB_ARGS $REPRO_TARGETS_SET
     while [ -e bitbake.sock ]; do sleep 1; done
+    # Check reuse of isar-apt repository
+    sudo rm -rf tmp/cache tmp/sstate-cache tmp/work tmp/stamps
+    sed -i -e 's/^ISAR_APT_ALWAYS_REBUILD_ON_CHANGE ?= "1"/#ISAR_APT_ALWAYS_REBUILD_ON_CHANGE ?= "1"/g' conf/local.conf
+    if bitbake $BB_ARGS $REPRO_TARGETS_SET; then
+        echo "isar-apt reuse: PASSED"
+    else
+        echo "isar-apt reuse: KFAIL"
+    fi
+    while [ -e bitbake.sock ]; do sleep 1; done
     # Cleanup and disable use of unsigned cached base repository
     sudo rm -rf tmp
+    sed -i -e 's/^#ISAR_APT_ALWAYS_REBUILD_ON_CHANGE ?= "1"/ISAR_APT_ALWAYS_REBUILD_ON_CHANGE ?= "1"/g' conf/local.conf
     sed -i -e 's/ISAR_USE_CACHED_BASE_REPO ?= "1"/#ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
     sed -i -e 's/^BB_NO_NETWORK/#BB_NO_NETWORK/g' conf/local.conf
 fi
-- 
2.20.1

[RFC 4/4] doc: Add section for isar-apt reuse functionality

[Uladzimir Bely]

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 doc/user_manual.md | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 1da0e9d..94af2ea 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -21,6 +21,7 @@ Copyright (C) 2016-2019, ilbers GmbH
  - [Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)
  - [Create a containerized Isar SDK root filesystem](#create-a-containerized-isar-sdk-root-filesystem)
  - [Creation of local apt repo caching upstream Debian packages](#creation-of-local-apt-repo-caching-upstream-debian-packages)
+ - [Reusing isar-apt repository](#reusing-isar-apt-repository)
 
 
 ## Introduction
@@ -1134,6 +1135,28 @@ sudo rm -rf tmp
 bitbake mc:qemuarm-buster:isar-image-base
 ```
 
+## Reusing isar-apt repository
+
+### Motivation
+
+Building custom recipes from sources may take a lot of time to download the sources and build the package. Also, reusing isar-apt makes it possible to work offline.
+
+### Solution
+
+After a custom package is built, it's placed into isar-apt repository. It's possible to reuse this repository for further builds. The behavior is controlled by `ISAR_APT_ALWAYS_REBUILD_ON_CHANGE` variable in `conf/local.conf`:
+
+- `ISAR_APT_ALWAYS_REBUILD_ON_CHANGE = "1"` - force rebuilding custom packages from sources
+- `ISAR_APT_ALWAYS_REBUILD_ON_CHANGE = "0"` - reuse previously built binary if it exists in repository.
+
+If `ISAR_APT_ALWAYS_REBUILD_ON_CHANGE` is not set it behaves like it was set to `"0"`
+
+It also may be useful to move `isar-apt` directory out from `tmp/deploy` so that it could be easily shared between builds. To do that, modify `conf/local.conf` by adding lines similar to the following ones:
+
+```
+REPO_ISAR_DIR = "/path/to/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt"
+REPO_ISAR_DB_DIR = "/path/to/isar-apt/${DISTRO}-${DISTRO_ARCH}/db"
+```
+
 ## Add foreign packages from other repositories to the generated image
 
 ### Motivation
-- 
2.20.1

Re: [RFC 0/4] PoC for isar-apt repo reusing

[Henning Schild]

Hi,

i did not look into the details yet. Just wanted to let you know that
we do have a prototype implementation of using sstate caching. That
allows caching anything, not just isar-apt.

The speedups on rebuilds are more than just significant. But it is
still too rough to present it on the list.

I think that sstate and setscene are better and more generic than
targeting only isar-apt.

If you want i guess Adriaan can share the current state of the "sstate"
patch series for anyone to play with. We are not holding back, just are
not ready to show ... But seeing that you work on similar things it is
about time we tell people about that ongoing sstate experiment.

Henning

Am Thu,  2 Sep 2021 11:58:13 +0200
schrieb Uladzimir Bely <ubely@ilbers.de>:

> Currently, isar-apt repo is always rebuilt at second build when
> temporary files except the repo directory were deleted.
> 
> This patchset implements proof-of-concept of reusing isar-apt 
> repo previously built. The idea is to check that package already 
> exists in the repo and early exit tasks related to fetching/building.
> 
> Such approach doesn't remove any tasks from the queue, but changes
> their behvaior (early exit) in case of existing package found.
> 
> Uladzimir Bely (4):
>   meta-isar: Fix do_dpkg_build override for prebuild-deb recipe.
>   isar-apt: PoC of reusing isar-apt repo
>   ci: Test for isar-apt reuse
>   doc: Add section for isar-apt reuse functionality
> 
>  doc/user_manual.md                            | 23 ++++++
>  meta-isar/conf/local.conf.sample              |  4 ++
>  .../prebuilt-deb/prebuilt-deb_0.1.bb          |  2 +-
>  meta/classes/dpkg-base.bbclass                |  1 +
>  meta/classes/isar-apt-cache.bbclass           | 72
> +++++++++++++++++++ scripts/ci_build.sh                           |
> 10 +++ 6 files changed, 111 insertions(+), 1 deletion(-)
>  create mode 100644 meta/classes/isar-apt-cache.bbclass
> 

Re: [RFC 0/4] PoC for isar-apt repo reusing

[Jan Kiszka]

On 02.09.21 11:58, Uladzimir Bely wrote:
> Currently, isar-apt repo is always rebuilt at second build when
> temporary files except the repo directory were deleted.
> 
> This patchset implements proof-of-concept of reusing isar-apt 
> repo previously built. The idea is to check that package already 
> exists in the repo and early exit tasks related to fetching/building.
> 
> Such approach doesn't remove any tasks from the queue, but changes
> their behvaior (early exit) in case of existing package found.
> 
> Uladzimir Bely (4):
>   meta-isar: Fix do_dpkg_build override for prebuild-deb recipe.
>   isar-apt: PoC of reusing isar-apt repo
>   ci: Test for isar-apt reuse
>   doc: Add section for isar-apt reuse functionality
> 
>  doc/user_manual.md                            | 23 ++++++
>  meta-isar/conf/local.conf.sample              |  4 ++
>  .../prebuilt-deb/prebuilt-deb_0.1.bb          |  2 +-
>  meta/classes/dpkg-base.bbclass                |  1 +
>  meta/classes/isar-apt-cache.bbclass           | 72 +++++++++++++++++++
>  scripts/ci_build.sh                           | 10 +++
>  6 files changed, 111 insertions(+), 1 deletion(-)
>  create mode 100644 meta/classes/isar-apt-cache.bbclass
> 

Sounds to me like there is overlap with Adriaan's experiments around
sstate cache. Guess he can comment on that (when he's back in office).

Patch 1 should likely be pulled out and merged independently - not
really an RFC, rather a fix, no?

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [RFC 0/4] PoC for isar-apt repo reusing

[Uladzimir Bely]

In the email from пятница, 3 сентября 2021 г. 08:52:44 +03 user Jan Kiszka 
wrote:
> 
> Patch 1 should likely be pulled out and merged independently - not
> really an RFC, rather a fix, no?
> 

Yes, it may be merged independently. Currently it works only due to the empty 
contents of the overlapped task.

In the email from четверг, 2 сентября 2021 г. 14:11:45 +03 user Henning Schild 
wrote:
>
> If you want i guess Adriaan can share the current state of the "sstate"
> patch series for anyone to play with.
>

Yes, if you share these patches, we could look at them to see the idea and  
possible overlaps of this PoC functionality.

-- 
Uladzimir Bely
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [RFC 0/4] PoC for isar-apt repo reusing

[Baurzhan Ismagulov]

On Thu, Sep 02, 2021 at 01:11:45PM +0200, Henning Schild wrote:
> i did not look into the details yet. Just wanted to let you know that
> we do have a prototype implementation of using sstate caching. That
> allows caching anything, not just isar-apt.

Looking forward to the sstate-cache patches. I think both are useful. Sstate
caching is about speeding up package building, isar-apt is about Debian
workflows.

Also, Isar will eventually have to track and distinguish between base-apt and
isar-apt. This is needed for correctness if isar-apt provides Build-Depends for
other self-built packages (linux-headers, openssl-dev, etc.).

On a related note, we've also tested ccache and would hopefully share the
patches soon -- it would speed up building when packages do change.

With kind regards,
Baurzhan.