* [RFC 1/4] meta-isar: Fix do_dpkg_build override for prebuild-deb recipe.
2021-09-02 9:58 [RFC 0/4] PoC for isar-apt repo reusing Uladzimir Bely
@ 2021-09-02 9:58 ` Uladzimir Bely
2021-09-02 9:58 ` [RFC 2/4] isar-apt: PoC of reusing isar-apt repo Uladzimir Bely
` (4 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: Uladzimir Bely @ 2021-09-02 9:58 UTC (permalink / raw)
To: isar-users
While original dpkg_build task in dpkg-base bbclass uses python syntax,
any overrides in recipes also should us it. Otherwise, bitbake error
is caused in case of prepends for this task.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta-isar/recipes-app/prebuilt-deb/prebuilt-deb_0.1.bb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta-isar/recipes-app/prebuilt-deb/prebuilt-deb_0.1.bb b/meta-isar/recipes-app/prebuilt-deb/prebuilt-deb_0.1.bb
index 2f1b699..5e2cc77 100644
--- a/meta-isar/recipes-app/prebuilt-deb/prebuilt-deb_0.1.bb
+++ b/meta-isar/recipes-app/prebuilt-deb/prebuilt-deb_0.1.bb
@@ -16,5 +16,5 @@ SRC_URI = "file://example-prebuilt_1.0.0-0_all.deb;unpack=false"
# packages are provided by a single recipe.
PROVIDES += "example-prebuilt"
-do_dpkg_build() {
+python do_dpkg_build() {
}
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread* [RFC 2/4] isar-apt: PoC of reusing isar-apt repo
2021-09-02 9:58 [RFC 0/4] PoC for isar-apt repo reusing Uladzimir Bely
2021-09-02 9:58 ` [RFC 1/4] meta-isar: Fix do_dpkg_build override for prebuild-deb recipe Uladzimir Bely
@ 2021-09-02 9:58 ` Uladzimir Bely
2021-09-02 9:58 ` [RFC 3/4] ci: Test for isar-apt reuse Uladzimir Bely
` (3 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: Uladzimir Bely @ 2021-09-02 9:58 UTC (permalink / raw)
To: isar-users
The idea of reusing isar-apt repo at second build is to early
exit from build-related tasks if the package with the same
version already exists in current isar-apt.
Also, redefining default values of REPO_ISAR_DIR and REPO_ISAR_DB_DIR
maybe useful to share isar-apt between different builds
REPO_ISAR_DIR = "/path/to/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt"
REPO_ISAR_DB_DIR = "/path/to/isar-apt/${DISTRO}-${DISTRO_ARCH}/db"
The variable ISAR_APT_ALWAYS_REBUILD_ON_CHANGE in local.conf
allow to control the behvaiour:
0: Use the cache, rebuilt only if needed
1: Don't use cache, rebuilt packages always (old behaviour)
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta-isar/conf/local.conf.sample | 4 ++
meta/classes/dpkg-base.bbclass | 1 +
meta/classes/isar-apt-cache.bbclass | 72 +++++++++++++++++++++++++++++
3 files changed, 77 insertions(+)
create mode 100644 meta/classes/isar-apt-cache.bbclass
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 96a8beb..ae1ba45 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -201,6 +201,10 @@ ISAR_CROSS_COMPILE ?= "0"
# does not access the network
#BB_NO_NETWORK ?= "1"
+#
+# Comment or set to 0 to use previously built isar_apt repo if package version hasn't changed
+ISAR_APT_ALWAYS_REBUILD_ON_CHANGE ?= "1"
+
# Set root password to 'root'
# Password was encrypted using following command:
# mkpasswd -m sha512crypt -R 10000
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index aa529ca..59113b3 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -9,6 +9,7 @@ inherit debianize
inherit terminal
inherit repository
inherit deb-dl-dir
+inherit isar-apt-cache
DEPENDS ?= ""
diff --git a/meta/classes/isar-apt-cache.bbclass b/meta/classes/isar-apt-cache.bbclass
new file mode 100644
index 0000000..c9533f0
--- /dev/null
+++ b/meta/classes/isar-apt-cache.bbclass
@@ -0,0 +1,72 @@
+# This software is a part of ISAR.
+# Copyright (C) 2017-2019 Siemens AG
+# Copyright (C) 2019 ilbers GmbH
+#
+# SPDX-License-Identifier: MIT
+
+iac_check_package_in_repo() {
+ DISTS_ISAR="${REPO_ISAR_DIR}/${DISTRO}/dists/${DEBDISTRONAME}"
+ PACKAGES="${DISTS_ISAR}/main/binary-${DISTRO_ARCH}/Packages"
+
+ if [ ! -f "${PACKAGES}" ]; then
+ return 1
+ fi
+
+ if ! grep -q "^Package: ${PN}$" "${PACKAGES}"; then
+ return 2
+ fi
+
+ local repo_version=$(sed -n -e '/^Package: ${PN}$/,/Version/p' ${PACKAGES} | grep '^Version' | sed -e 's/.* //')
+ local new_version=$(head -n1 ${S}/debian/changelog | sed -e 's/.* (\(.*\)) .*/\1/')
+
+ if [ "${repo_version}" != "${new_version}" ]; then
+ return 3
+ fi
+
+ bbnote "Package ${PN}-${PV} is already in isar-apt repo, reuse it"
+ return 0
+}
+
+iac_check_need_rebuild() {
+ if [ "${ISAR_APT_ALWAYS_REBUILD_ON_CHANGE}" = "1" ]; then
+ return 1
+ fi
+
+ iac_check_package_in_repo
+}
+
+def iac_check_task_early_exit():
+ try:
+ bb.build.exec_func("iac_check_need_rebuild", d)
+ return True
+ except:
+ return False
+
+
+python do_apt_fetch_prepend() {
+ if iac_check_task_early_exit():
+ return 0
+}
+
+python do_apt_unpack_prepend() {
+ if iac_check_task_early_exit():
+ return 0
+}
+
+do_prepare_build_prepend() {
+ iac_check_need_rebuild && return
+}
+
+do_deploy_deb_prepend() {
+ iac_check_need_rebuild && return
+}
+
+python do_install_builddeps_prepend() {
+ if iac_check_task_early_exit():
+ return 0
+}
+
+python do_dpkg_build_prepend() {
+ if iac_check_task_early_exit():
+ return 0
+}
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread* [RFC 3/4] ci: Test for isar-apt reuse
2021-09-02 9:58 [RFC 0/4] PoC for isar-apt repo reusing Uladzimir Bely
2021-09-02 9:58 ` [RFC 1/4] meta-isar: Fix do_dpkg_build override for prebuild-deb recipe Uladzimir Bely
2021-09-02 9:58 ` [RFC 2/4] isar-apt: PoC of reusing isar-apt repo Uladzimir Bely
@ 2021-09-02 9:58 ` Uladzimir Bely
2021-09-02 9:58 ` [RFC 4/4] doc: Add section for isar-apt reuse functionality Uladzimir Bely
` (2 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: Uladzimir Bely @ 2021-09-02 9:58 UTC (permalink / raw)
To: isar-users
The approach: after first build of some target can remove
all temporary files, leaving tmp/deploy/isar-apt. Second
build should use packages from it without rebuilding.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
scripts/ci_build.sh | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 837cd67..e62c02a 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -203,8 +203,18 @@ if [ -n "$REPRO_BUILD" ]; then
sed -i -e 's/^#BB_NO_NETWORK/BB_NO_NETWORK/g' conf/local.conf
bitbake $BB_ARGS $REPRO_TARGETS_SET
while [ -e bitbake.sock ]; do sleep 1; done
+ # Check reuse of isar-apt repository
+ sudo rm -rf tmp/cache tmp/sstate-cache tmp/work tmp/stamps
+ sed -i -e 's/^ISAR_APT_ALWAYS_REBUILD_ON_CHANGE ?= "1"/#ISAR_APT_ALWAYS_REBUILD_ON_CHANGE ?= "1"/g' conf/local.conf
+ if bitbake $BB_ARGS $REPRO_TARGETS_SET; then
+ echo "isar-apt reuse: PASSED"
+ else
+ echo "isar-apt reuse: KFAIL"
+ fi
+ while [ -e bitbake.sock ]; do sleep 1; done
# Cleanup and disable use of unsigned cached base repository
sudo rm -rf tmp
+ sed -i -e 's/^#ISAR_APT_ALWAYS_REBUILD_ON_CHANGE ?= "1"/ISAR_APT_ALWAYS_REBUILD_ON_CHANGE ?= "1"/g' conf/local.conf
sed -i -e 's/ISAR_USE_CACHED_BASE_REPO ?= "1"/#ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
sed -i -e 's/^BB_NO_NETWORK/#BB_NO_NETWORK/g' conf/local.conf
fi
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread* [RFC 4/4] doc: Add section for isar-apt reuse functionality
2021-09-02 9:58 [RFC 0/4] PoC for isar-apt repo reusing Uladzimir Bely
` (2 preceding siblings ...)
2021-09-02 9:58 ` [RFC 3/4] ci: Test for isar-apt reuse Uladzimir Bely
@ 2021-09-02 9:58 ` Uladzimir Bely
2021-09-02 11:11 ` [RFC 0/4] PoC for isar-apt repo reusing Henning Schild
2021-09-03 5:52 ` Jan Kiszka
5 siblings, 0 replies; 9+ messages in thread
From: Uladzimir Bely @ 2021-09-02 9:58 UTC (permalink / raw)
To: isar-users
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
doc/user_manual.md | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/doc/user_manual.md b/doc/user_manual.md
index 1da0e9d..94af2ea 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -21,6 +21,7 @@ Copyright (C) 2016-2019, ilbers GmbH
- [Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)
- [Create a containerized Isar SDK root filesystem](#create-a-containerized-isar-sdk-root-filesystem)
- [Creation of local apt repo caching upstream Debian packages](#creation-of-local-apt-repo-caching-upstream-debian-packages)
+ - [Reusing isar-apt repository](#reusing-isar-apt-repository)
## Introduction
@@ -1134,6 +1135,28 @@ sudo rm -rf tmp
bitbake mc:qemuarm-buster:isar-image-base
```
+## Reusing isar-apt repository
+
+### Motivation
+
+Building custom recipes from sources may take a lot of time to download the sources and build the package. Also, reusing isar-apt makes it possible to work offline.
+
+### Solution
+
+After a custom package is built, it's placed into isar-apt repository. It's possible to reuse this repository for further builds. The behavior is controlled by `ISAR_APT_ALWAYS_REBUILD_ON_CHANGE` variable in `conf/local.conf`:
+
+- `ISAR_APT_ALWAYS_REBUILD_ON_CHANGE = "1"` - force rebuilding custom packages from sources
+- `ISAR_APT_ALWAYS_REBUILD_ON_CHANGE = "0"` - reuse previously built binary if it exists in repository.
+
+If `ISAR_APT_ALWAYS_REBUILD_ON_CHANGE` is not set it behaves like it was set to `"0"`
+
+It also may be useful to move `isar-apt` directory out from `tmp/deploy` so that it could be easily shared between builds. To do that, modify `conf/local.conf` by adding lines similar to the following ones:
+
+```
+REPO_ISAR_DIR = "/path/to/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt"
+REPO_ISAR_DB_DIR = "/path/to/isar-apt/${DISTRO}-${DISTRO_ARCH}/db"
+```
+
## Add foreign packages from other repositories to the generated image
### Motivation
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [RFC 0/4] PoC for isar-apt repo reusing
2021-09-02 9:58 [RFC 0/4] PoC for isar-apt repo reusing Uladzimir Bely
` (3 preceding siblings ...)
2021-09-02 9:58 ` [RFC 4/4] doc: Add section for isar-apt reuse functionality Uladzimir Bely
@ 2021-09-02 11:11 ` Henning Schild
2021-09-03 8:43 ` Baurzhan Ismagulov
2021-09-03 5:52 ` Jan Kiszka
5 siblings, 1 reply; 9+ messages in thread
From: Henning Schild @ 2021-09-02 11:11 UTC (permalink / raw)
To: Uladzimir Bely; +Cc: isar-users, Schmidt, Adriaan
Hi,
i did not look into the details yet. Just wanted to let you know that
we do have a prototype implementation of using sstate caching. That
allows caching anything, not just isar-apt.
The speedups on rebuilds are more than just significant. But it is
still too rough to present it on the list.
I think that sstate and setscene are better and more generic than
targeting only isar-apt.
If you want i guess Adriaan can share the current state of the "sstate"
patch series for anyone to play with. We are not holding back, just are
not ready to show ... But seeing that you work on similar things it is
about time we tell people about that ongoing sstate experiment.
Henning
Am Thu, 2 Sep 2021 11:58:13 +0200
schrieb Uladzimir Bely <ubely@ilbers.de>:
> Currently, isar-apt repo is always rebuilt at second build when
> temporary files except the repo directory were deleted.
>
> This patchset implements proof-of-concept of reusing isar-apt
> repo previously built. The idea is to check that package already
> exists in the repo and early exit tasks related to fetching/building.
>
> Such approach doesn't remove any tasks from the queue, but changes
> their behvaior (early exit) in case of existing package found.
>
> Uladzimir Bely (4):
> meta-isar: Fix do_dpkg_build override for prebuild-deb recipe.
> isar-apt: PoC of reusing isar-apt repo
> ci: Test for isar-apt reuse
> doc: Add section for isar-apt reuse functionality
>
> doc/user_manual.md | 23 ++++++
> meta-isar/conf/local.conf.sample | 4 ++
> .../prebuilt-deb/prebuilt-deb_0.1.bb | 2 +-
> meta/classes/dpkg-base.bbclass | 1 +
> meta/classes/isar-apt-cache.bbclass | 72
> +++++++++++++++++++ scripts/ci_build.sh |
> 10 +++ 6 files changed, 111 insertions(+), 1 deletion(-)
> create mode 100644 meta/classes/isar-apt-cache.bbclass
>
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [RFC 0/4] PoC for isar-apt repo reusing
2021-09-02 11:11 ` [RFC 0/4] PoC for isar-apt repo reusing Henning Schild
@ 2021-09-03 8:43 ` Baurzhan Ismagulov
0 siblings, 0 replies; 9+ messages in thread
From: Baurzhan Ismagulov @ 2021-09-03 8:43 UTC (permalink / raw)
To: isar-users
On Thu, Sep 02, 2021 at 01:11:45PM +0200, Henning Schild wrote:
> i did not look into the details yet. Just wanted to let you know that
> we do have a prototype implementation of using sstate caching. That
> allows caching anything, not just isar-apt.
Looking forward to the sstate-cache patches. I think both are useful. Sstate
caching is about speeding up package building, isar-apt is about Debian
workflows.
Also, Isar will eventually have to track and distinguish between base-apt and
isar-apt. This is needed for correctness if isar-apt provides Build-Depends for
other self-built packages (linux-headers, openssl-dev, etc.).
On a related note, we've also tested ccache and would hopefully share the
patches soon -- it would speed up building when packages do change.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RFC 0/4] PoC for isar-apt repo reusing
2021-09-02 9:58 [RFC 0/4] PoC for isar-apt repo reusing Uladzimir Bely
` (4 preceding siblings ...)
2021-09-02 11:11 ` [RFC 0/4] PoC for isar-apt repo reusing Henning Schild
@ 2021-09-03 5:52 ` Jan Kiszka
2021-09-03 8:18 ` Uladzimir Bely
5 siblings, 1 reply; 9+ messages in thread
From: Jan Kiszka @ 2021-09-03 5:52 UTC (permalink / raw)
To: Uladzimir Bely, isar-users, Schmidt, Adriaan
On 02.09.21 11:58, Uladzimir Bely wrote:
> Currently, isar-apt repo is always rebuilt at second build when
> temporary files except the repo directory were deleted.
>
> This patchset implements proof-of-concept of reusing isar-apt
> repo previously built. The idea is to check that package already
> exists in the repo and early exit tasks related to fetching/building.
>
> Such approach doesn't remove any tasks from the queue, but changes
> their behvaior (early exit) in case of existing package found.
>
> Uladzimir Bely (4):
> meta-isar: Fix do_dpkg_build override for prebuild-deb recipe.
> isar-apt: PoC of reusing isar-apt repo
> ci: Test for isar-apt reuse
> doc: Add section for isar-apt reuse functionality
>
> doc/user_manual.md | 23 ++++++
> meta-isar/conf/local.conf.sample | 4 ++
> .../prebuilt-deb/prebuilt-deb_0.1.bb | 2 +-
> meta/classes/dpkg-base.bbclass | 1 +
> meta/classes/isar-apt-cache.bbclass | 72 +++++++++++++++++++
> scripts/ci_build.sh | 10 +++
> 6 files changed, 111 insertions(+), 1 deletion(-)
> create mode 100644 meta/classes/isar-apt-cache.bbclass
>
Sounds to me like there is overlap with Adriaan's experiments around
sstate cache. Guess he can comment on that (when he's back in office).
Patch 1 should likely be pulled out and merged independently - not
really an RFC, rather a fix, no?
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [RFC 0/4] PoC for isar-apt repo reusing
2021-09-03 5:52 ` Jan Kiszka
@ 2021-09-03 8:18 ` Uladzimir Bely
0 siblings, 0 replies; 9+ messages in thread
From: Uladzimir Bely @ 2021-09-03 8:18 UTC (permalink / raw)
To: isar-users, Jan Kiszka, Henning Schild
In the email from пятница, 3 сентября 2021 г. 08:52:44 +03 user Jan Kiszka
wrote:
>
> Patch 1 should likely be pulled out and merged independently - not
> really an RFC, rather a fix, no?
>
Yes, it may be merged independently. Currently it works only due to the empty
contents of the overlapped task.
In the email from четверг, 2 сентября 2021 г. 14:11:45 +03 user Henning Schild
wrote:
>
> If you want i guess Adriaan can share the current state of the "sstate"
> patch series for anyone to play with.
>
Yes, if you share these patches, we could look at them to see the idea and
possible overlaps of this PoC functionality.
--
Uladzimir Bely
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 9+ messages in thread