* [PATCH 0/5] Rework isar-apt v2
@ 2018-02-05 9:59 Alexander Smirnov
2018-02-05 9:59 ` [PATCH 1/5] Update isar-bitbake.conf according to the changes in bitbake related to multiconfig support Alexander Smirnov
` (9 more replies)
0 siblings, 10 replies; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-05 9:59 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
Hi all,
this series intended to make buildchroot able to work with isar-apt.
I've tried to add extended comments to each patch.
Impact: with this series I'm able to build 'example-hello' <-> 'libhello'
without any hacks. So the deps are installed automatically.
Changes since v1:
- The bitbake was updated to the latest upstream version
- In build.sh the only isar-apt is updated
Alexander Smirnov (5):
Update isar-bitbake.conf according to the changes in bitbake related
to multiconfig support.
isar-apt: Introduce separate recipe
buildchroot: Enable isar-apt
build.sh: Update apt sources
build.sh: Force 'yes' for apt
meta-isar/conf/local.conf.sample | 5 +-
.../recipes-core/images/files/distributions.in | 3 -
meta/classes/dpkg-base.bbclass | 12 +++-
meta/classes/image.bbclass | 67 +---------------------
meta/classes/isar-events.bbclass | 20 +++++++
meta/conf/isar-bitbake.conf | 1 +
meta/recipes-devtools/buildchroot/buildchroot.bb | 7 +++
meta/recipes-devtools/buildchroot/files/build.sh | 13 ++++-
.../buildchroot/files/multistrap.conf.in | 8 ++-
.../isar-apt/files/distributions.in | 3 +
meta/recipes-devtools/isar-apt/isar-apt.bb | 29 ++++++++++
11 files changed, 92 insertions(+), 76 deletions(-)
delete mode 100644 meta-isar/recipes-core/images/files/distributions.in
create mode 100644 meta/classes/isar-events.bbclass
create mode 100644 meta/recipes-devtools/isar-apt/files/distributions.in
create mode 100644 meta/recipes-devtools/isar-apt/isar-apt.bb
--
2.1.4
^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH 1/5] Update isar-bitbake.conf according to the changes in bitbake related to multiconfig support.
2018-02-05 9:59 [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
@ 2018-02-05 9:59 ` Alexander Smirnov
2018-02-05 10:15 ` Jan Kiszka
2018-02-05 9:59 ` [PATCH 2/5] isar-apt: Introduce separate recipe Alexander Smirnov
` (8 subsequent siblings)
9 siblings, 1 reply; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-05 9:59 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta/conf/isar-bitbake.conf | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
index 10cd45d..b853c88 100644
--- a/meta/conf/isar-bitbake.conf
+++ b/meta/conf/isar-bitbake.conf
@@ -28,5 +28,6 @@ BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
BB_SIGNATURE_HANDLER ?= "noop"
include conf/local.conf
+include conf/multiconfig/${BB_CURRENT_MC}.conf
include conf/machine/${MACHINE}.conf
include conf/distro/${DISTRO}.conf
--
2.1.4
^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH 2/5] isar-apt: Introduce separate recipe
2018-02-05 9:59 [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
2018-02-05 9:59 ` [PATCH 1/5] Update isar-bitbake.conf according to the changes in bitbake related to multiconfig support Alexander Smirnov
@ 2018-02-05 9:59 ` Alexander Smirnov
2018-02-05 9:59 ` [PATCH 3/5] buildchroot: Enable isar-apt Alexander Smirnov
` (7 subsequent siblings)
9 siblings, 0 replies; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-05 9:59 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
Motivation: in current implementation whole isar-apt logic are done as
a part of image class. This actually not the optimal way due to the
various issues like:
- isar-apt should be singletone in current tree despite on amount of images
to be built.
- isar-apt API could be possible extended, for example by adding do_export().
- *-image is not the mandatory target to build. User may build just
example-hello, but isar-apt should handle this.
So there should be separate recipe which is responsibe for isar-apt management.
Also isar-apt should be a static part of core Isar like buildchroot.
This patch drops isar-apt from image class, and introduces isar-apt recipe.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta-isar/conf/local.conf.sample | 4 --
.../recipes-core/images/files/distributions.in | 3 -
meta/classes/dpkg-base.bbclass | 12 +++-
meta/classes/image.bbclass | 67 +---------------------
.../isar-apt/files/distributions.in | 3 +
meta/recipes-devtools/isar-apt/isar-apt.bb | 29 ++++++++++
6 files changed, 43 insertions(+), 75 deletions(-)
delete mode 100644 meta-isar/recipes-core/images/files/distributions.in
create mode 100644 meta/recipes-devtools/isar-apt/files/distributions.in
create mode 100644 meta/recipes-devtools/isar-apt/isar-apt.bb
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 33d988e..2ae43e7 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -158,7 +158,3 @@ IMAGE_INSTALL = "example-hello example-raw"
#
# Default parallel jobs for bitbake:
BB_NUMBER_THREADS = "4"
-
-#
-# Number of attempts to try to get reprepro lock for access to apt cache
-REPREPRO_LOCK_ATTEMPTS = "16"
diff --git a/meta-isar/recipes-core/images/files/distributions.in b/meta-isar/recipes-core/images/files/distributions.in
deleted file mode 100644
index cd214c6..0000000
--- a/meta-isar/recipes-core/images/files/distributions.in
+++ /dev/null
@@ -1,3 +0,0 @@
-Codename: {DISTRO_NAME}
-Architectures: i386 armhf amd64 source
-Components: main
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 4941f9b..026028f 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -31,11 +31,17 @@ do_build() {
_do_build_cleanup
}
-# Install package to dedicated deploy directory
+# Install package to Isar-apt
do_deploy_deb() {
- install -m 644 ${WORKDIR}/*.deb ${DEPLOY_DIR_DEB}/
+ reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \
+ --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \
+ -C main \
+ includedeb ${DEBDISTRONAME} \
+ ${WORKDIR}/*.deb
}
addtask deploy_deb after do_build
do_deploy_deb[dirs] = "${DEPLOY_DIR_DEB}"
-do_deploy_deb[stamp-extra-info] = "${MACHINE}"
+do_deploy_deb[stamp-extra-info] = "${DISTRO}-${MACHINE}"
+do_deploy_deb[lockfiles] = "${DEPLOY_DIR_APT}/isar.lock"
+do_deploy_deb[depends] = "isar-apt:do_cache_config"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index e2cb01b..cbd74b3 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -17,70 +17,6 @@ KERNEL_IMAGE ?= "${@get_image_name(d, 'vmlinuz')}"
INITRD_IMAGE ?= "${@get_image_name(d, 'initrd.img')}"
inherit ${IMAGE_TYPE}
-call_reprepro() {
- for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do
- # According to `sh` manual page, shell exit statuses fall between
- # 0-255. The EEXIST error code is (-17), so casting to usigned 8-bit
- # integer results value (239).
- eexist=$(python -c 'import errno; print(256-errno.EEXIST)')
- retval="0"
- reprepro $* || retval="$?"
-
- # If reprepro has failed to get database lock, it returns EEXIST code.
- # In this case we continue trying to get lock until max amount of
- # attempts is reached.
- if [ $retval -eq $eexist ]; then
- bbwarn "Failed to get reprepro lock, trying again..."
- sleep 5
- else
- break
- fi
- done
-
- if [ $retval -ne 0 ]; then
- bbfatal "reprepro failed"
- fi
-}
-
-CACHE_CONF_DIR = "${DEPLOY_DIR_APT}/${DISTRO}/conf"
-do_cache_config[dirs] = "${CACHE_CONF_DIR}"
-do_cache_config[stamp-extra-info] = "${DISTRO}"
-
-# Generate reprepro config for current distro if it doesn't exist. Once it's
-# generated, this task should do nothing.
-do_cache_config() {
- if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
- sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
- ${FILESDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
- fi
-
- path_cache="${DEPLOY_DIR_APT}/${DISTRO}"
- path_databases="${DEPLOY_DIR_DB}/${DISTRO}"
-
- if [ ! -d "${path_databases}" ]; then
- call_reprepro -b ${path_cache} \
- --dbdir ${path_databases} \
- export ${DEBDISTRONAME}
- fi
-}
-
-addtask cache_config before do_populate
-
-do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
-
-# Populate Isar apt repository by newly built packages
-do_populate() {
- if [ -n "${IMAGE_INSTALL}" ]; then
- call_reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \
- --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \
- -C main \
- includedeb ${DEBDISTRONAME} \
- ${DEPLOY_DIR_DEB}/*.deb
- fi
-}
-
-addtask populate before do_build after do_unpack
-do_populate[deptask] = "do_deploy_deb"
do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
@@ -88,7 +24,8 @@ do_rootfs() {
die "No root filesystem function defined, please implement in your recipe"
}
-addtask rootfs before do_build after do_populate
+addtask rootfs before do_build after do_unpack
+do_rootfs[deptask] = "do_deploy_deb"
do_copy_boot_files() {
KERNEL_IMAGE=${@get_image_name(d, 'vmlinuz')}
diff --git a/meta/recipes-devtools/isar-apt/files/distributions.in b/meta/recipes-devtools/isar-apt/files/distributions.in
new file mode 100644
index 0000000..cd214c6
--- /dev/null
+++ b/meta/recipes-devtools/isar-apt/files/distributions.in
@@ -0,0 +1,3 @@
+Codename: {DISTRO_NAME}
+Architectures: i386 armhf amd64 source
+Components: main
diff --git a/meta/recipes-devtools/isar-apt/isar-apt.bb b/meta/recipes-devtools/isar-apt/isar-apt.bb
new file mode 100644
index 0000000..9c31d12
--- /dev/null
+++ b/meta/recipes-devtools/isar-apt/isar-apt.bb
@@ -0,0 +1,29 @@
+# This software is a part of ISAR.
+# Copyright (C) 2015-2017 ilbers GmbH
+
+SRC_URI = "file://distributions.in"
+
+CACHE_CONF_DIR = "${DEPLOY_DIR_APT}/${DISTRO}/conf"
+do_cache_config[dirs] = "${CACHE_CONF_DIR}"
+do_cache_config[stamp-extra-info] = "${DISTRO}"
+do_cache_config[lockfiles] = "${DEPLOY_DIR_APT}/isar.lock"
+
+# Generate reprepro config for current distro if it doesn't exist. Once it's
+# generated, this task should do nothing.
+do_cache_config() {
+ if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
+ sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
+ ${WORKDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
+ fi
+
+ path_cache="${DEPLOY_DIR_APT}/${DISTRO}"
+ path_databases="${DEPLOY_DIR_DB}/${DISTRO}"
+
+ if [ ! -d "${path_databases}" ]; then
+ reprepro -b ${path_cache} \
+ --dbdir ${path_databases} \
+ export ${DEBDISTRONAME}
+ fi
+}
+
+addtask cache_config after do_unpack before do_build
--
2.1.4
^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH 3/5] buildchroot: Enable isar-apt
2018-02-05 9:59 [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
2018-02-05 9:59 ` [PATCH 1/5] Update isar-bitbake.conf according to the changes in bitbake related to multiconfig support Alexander Smirnov
2018-02-05 9:59 ` [PATCH 2/5] isar-apt: Introduce separate recipe Alexander Smirnov
@ 2018-02-05 9:59 ` Alexander Smirnov
2018-02-05 10:18 ` Jan Kiszka
2018-02-05 9:59 ` [PATCH 4/5] build.sh: Update apt sources Alexander Smirnov
` (6 subsequent siblings)
9 siblings, 1 reply; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-05 9:59 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
This patch provides access to isar-apt from buildchroot. It does the
following:
- mount isar-apt during buildchroot building.
- umount isar-apt using bitbake events.
Also it needs to keep Isar build tree clean from any mounts despite on whether
build succeed of failed. bitbake provides various events that could trigger
custom python hooks. In this patch BuildCompleted event is used, which happened
when bitbake finished its execution despite on the result.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta-isar/conf/local.conf.sample | 3 +++
meta/classes/isar-events.bbclass | 20 ++++++++++++++++++++
meta/recipes-devtools/buildchroot/buildchroot.bb | 7 +++++++
.../buildchroot/files/multistrap.conf.in | 8 +++++++-
4 files changed, 37 insertions(+), 1 deletion(-)
create mode 100644 meta/classes/isar-events.bbclass
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 2ae43e7..fc760ed 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -158,3 +158,6 @@ IMAGE_INSTALL = "example-hello example-raw"
#
# Default parallel jobs for bitbake:
BB_NUMBER_THREADS = "4"
+
+# Add event handlers for bitbake
+INHERIT += "isar-events"
diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
new file mode 100644
index 0000000..55fc106
--- /dev/null
+++ b/meta/classes/isar-events.bbclass
@@ -0,0 +1,20 @@
+# Isar event handlers.
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2017 ilbers GmbH
+
+addhandler isar_handler
+
+python isar_handler () {
+ import subprocess
+
+ devnull = open(os.devnull, 'w')
+
+ if isinstance(e, bb.event.BuildCompleted):
+ bchroot = d.getVar('BUILDCHROOT_DIR', True)
+
+ # Clean up buildchroot
+ subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot + '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True)
+
+ devnull.close()
+}
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 51f9d5d..cb90148 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -33,6 +33,7 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
do_build[dirs] = "${WORKDIR}/hooks_multistrap"
+do_build[depends] = "isar-apt:do_cache_config"
do_build() {
E="${@ bb.utils.export_proxies(d)}"
@@ -58,6 +59,9 @@ do_build() {
-e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
"${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
+ [ ! -d ${BUILDCHROOT_DIR}/isar-apt ] && install -d -m 555 ${BUILDCHROOT_DIR}/isar-apt
+ sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
+
[ ! -d ${BUILDCHROOT_DIR}/proc ] && install -d -m 555 ${BUILDCHROOT_DIR}/proc
sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
_do_build_cleanup() {
@@ -73,6 +77,9 @@ do_build() {
# Install package builder script
sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
+ # Create share point for isar-apt
+ sudo install -d ${BUILDCHROOT_DIR}/isar-apt
+
# Configure root filesystem
sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
_do_build_cleanup
diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
index a0b28e3..480a4b8 100644
--- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
+++ b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
@@ -6,7 +6,7 @@ noauth=true
unpack=true
ignorenativearch=true
bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
-aptsources=##DISTRO_MULTICONF_APTSOURCES##
+aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
configscript=##CONFIG_SCRIPT##
setupscript=##SETUP_SCRIPT##
hookdir=##DIR_HOOKS##
@@ -29,3 +29,9 @@ source=##DISTRO_APT_SOURCE_SEC##
suite=##DISTRO_SUITE##/updates
components=##DISTRO_COMPONENTS##
omitdebsrc=true
+
+[isar-apt]
+source=file:///isar-apt
+suite=isar
+components=main
+omitdebsrc=true
--
2.1.4
^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH 4/5] build.sh: Update apt sources
2018-02-05 9:59 [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
` (2 preceding siblings ...)
2018-02-05 9:59 ` [PATCH 3/5] buildchroot: Enable isar-apt Alexander Smirnov
@ 2018-02-05 9:59 ` Alexander Smirnov
2018-02-05 9:59 ` [PATCH 5/5] build.sh: Force 'yes' for apt Alexander Smirnov
` (5 subsequent siblings)
9 siblings, 0 replies; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-05 9:59 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
Trigger isar-apt packages list update before building the package to be
sure that we are in sync with the latest isar-apt content.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta/recipes-devtools/buildchroot/files/build.sh | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
index 77e0fdd..b63a27e 100644
--- a/meta/recipes-devtools/buildchroot/files/build.sh
+++ b/meta/recipes-devtools/buildchroot/files/build.sh
@@ -3,6 +3,17 @@
# This software is a part of ISAR.
# Copyright (C) 2015-2017 ilbers GmbH
+# Make sure that we have latest isar-apt content.
+# Options meaning:
+# Dir::Etc::sourcelist - specifies which source to be used
+# Dir::Etc::sourceparts - disables looking for the other sources
+# APT::Get::List-Cleanup - do not erase obsolete packages list for
+# upstream in '/var/lib/apt/lists'
+apt-get update \
+ -o Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \
+ -o Dir::Etc::sourceparts="-" \
+ -o APT::Get::List-Cleanup="0"
+
# Go to build directory
cd $1
--
2.1.4
^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH 5/5] build.sh: Force 'yes' for apt
2018-02-05 9:59 [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
` (3 preceding siblings ...)
2018-02-05 9:59 ` [PATCH 4/5] build.sh: Update apt sources Alexander Smirnov
@ 2018-02-05 9:59 ` Alexander Smirnov
2018-02-05 11:52 ` Jan Kiszka
2018-02-05 13:48 ` [PATCH 3/5 v2] buildchroot: Enable isar-apt Alexander Smirnov
` (4 subsequent siblings)
9 siblings, 1 reply; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-05 9:59 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
Sometimes the following error occurs with custom repo:
After this operation, 84.0 kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
libhello libhello-dev
E: There are problems and -y was used without --force-yes
This patch adds '--force-yes' option to be sure that no dialog will occur.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta/recipes-devtools/buildchroot/files/build.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
index b63a27e..06df1da 100644
--- a/meta/recipes-devtools/buildchroot/files/build.sh
+++ b/meta/recipes-devtools/buildchroot/files/build.sh
@@ -21,7 +21,7 @@ cd $1
# Notes:
# 1) everything before the -y switch is unchanged from the defaults
# 2) we add -y to go non-interactive
-install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y"
+install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y --force-yes"
# Install all build deps
mk-build-deps -t "${install_cmd}" -i -r debian/control
--
2.1.4
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 1/5] Update isar-bitbake.conf according to the changes in bitbake related to multiconfig support.
2018-02-05 9:59 ` [PATCH 1/5] Update isar-bitbake.conf according to the changes in bitbake related to multiconfig support Alexander Smirnov
@ 2018-02-05 10:15 ` Jan Kiszka
2018-02-05 10:26 ` Alexander Smirnov
0 siblings, 1 reply; 32+ messages in thread
From: Jan Kiszka @ 2018-02-05 10:15 UTC (permalink / raw)
To: Alexander Smirnov, isar-users
On 2018-02-05 10:59, Alexander Smirnov wrote:
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta/conf/isar-bitbake.conf | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
> index 10cd45d..b853c88 100644
> --- a/meta/conf/isar-bitbake.conf
> +++ b/meta/conf/isar-bitbake.conf
> @@ -28,5 +28,6 @@ BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
> BB_SIGNATURE_HANDLER ?= "noop"
>
> include conf/local.conf
> +include conf/multiconfig/${BB_CURRENT_MC}.conf
> include conf/machine/${MACHINE}.conf
> include conf/distro/${DISTRO}.conf
>
Is the preparatory to the bitbake update or does it have to be merged
afterwards?
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 3/5] buildchroot: Enable isar-apt
2018-02-05 9:59 ` [PATCH 3/5] buildchroot: Enable isar-apt Alexander Smirnov
@ 2018-02-05 10:18 ` Jan Kiszka
2018-02-05 10:33 ` Alexander Smirnov
0 siblings, 1 reply; 32+ messages in thread
From: Jan Kiszka @ 2018-02-05 10:18 UTC (permalink / raw)
To: Alexander Smirnov, isar-users
On 2018-02-05 10:59, Alexander Smirnov wrote:
> This patch provides access to isar-apt from buildchroot. It does the
> following:
> - mount isar-apt during buildchroot building.
> - umount isar-apt using bitbake events.
>
> Also it needs to keep Isar build tree clean from any mounts despite on whether
> build succeed of failed. bitbake provides various events that could trigger
> custom python hooks. In this patch BuildCompleted event is used, which happened
> when bitbake finished its execution despite on the result.
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta-isar/conf/local.conf.sample | 3 +++
> meta/classes/isar-events.bbclass | 20 ++++++++++++++++++++
> meta/recipes-devtools/buildchroot/buildchroot.bb | 7 +++++++
> .../buildchroot/files/multistrap.conf.in | 8 +++++++-
> 4 files changed, 37 insertions(+), 1 deletion(-)
> create mode 100644 meta/classes/isar-events.bbclass
>
> diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
> index 2ae43e7..fc760ed 100644
> --- a/meta-isar/conf/local.conf.sample
> +++ b/meta-isar/conf/local.conf.sample
> @@ -158,3 +158,6 @@ IMAGE_INSTALL = "example-hello example-raw"
> #
> # Default parallel jobs for bitbake:
> BB_NUMBER_THREADS = "4"
> +
> +# Add event handlers for bitbake
> +INHERIT += "isar-events"
> diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
> new file mode 100644
> index 0000000..55fc106
> --- /dev/null
> +++ b/meta/classes/isar-events.bbclass
> @@ -0,0 +1,20 @@
> +# Isar event handlers.
> +#
> +# This software is a part of ISAR.
> +# Copyright (C) 2015-2017 ilbers GmbH
> +
> +addhandler isar_handler
> +
> +python isar_handler () {
> + import subprocess
> +
> + devnull = open(os.devnull, 'w')
> +
> + if isinstance(e, bb.event.BuildCompleted):
> + bchroot = d.getVar('BUILDCHROOT_DIR', True)
> +
> + # Clean up buildchroot
> + subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot + '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True)
> +
> + devnull.close()
> +}
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
> index 51f9d5d..cb90148 100644
> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
> @@ -33,6 +33,7 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>
> do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> do_build[dirs] = "${WORKDIR}/hooks_multistrap"
> +do_build[depends] = "isar-apt:do_cache_config"
>
> do_build() {
> E="${@ bb.utils.export_proxies(d)}"
> @@ -58,6 +59,9 @@ do_build() {
> -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
> "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
>
> + [ ! -d ${BUILDCHROOT_DIR}/isar-apt ] && install -d -m 555 ${BUILDCHROOT_DIR}/isar-apt
> + sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
> +
> [ ! -d ${BUILDCHROOT_DIR}/proc ] && install -d -m 555 ${BUILDCHROOT_DIR}/proc
> sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
> _do_build_cleanup() {
> @@ -73,6 +77,9 @@ do_build() {
> # Install package builder script
> sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
>
> + # Create share point for isar-apt
> + sudo install -d ${BUILDCHROOT_DIR}/isar-apt
> +
> # Configure root filesystem
> sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
> _do_build_cleanup
> diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
> index a0b28e3..480a4b8 100644
> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
> +++ b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
> @@ -6,7 +6,7 @@ noauth=true
> unpack=true
> ignorenativearch=true
> bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
> +aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
> configscript=##CONFIG_SCRIPT##
> setupscript=##SETUP_SCRIPT##
> hookdir=##DIR_HOOKS##
> @@ -29,3 +29,9 @@ source=##DISTRO_APT_SOURCE_SEC##
> suite=##DISTRO_SUITE##/updates
> components=##DISTRO_COMPONENTS##
> omitdebsrc=true
> +
> +[isar-apt]
> +source=file:///isar-apt
> +suite=isar
> +components=main
> +omitdebsrc=true
>
On first glance, this looks identical to v1 which I tried over the
current bitbake version we have in Isar. There I see two issues:
- umount does not work if the build fails
- re-mount will not happen if the buildchroot was built in a previous
run and only a depending packages is built now
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 1/5] Update isar-bitbake.conf according to the changes in bitbake related to multiconfig support.
2018-02-05 10:15 ` Jan Kiszka
@ 2018-02-05 10:26 ` Alexander Smirnov
2018-02-05 10:27 ` Jan Kiszka
0 siblings, 1 reply; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-05 10:26 UTC (permalink / raw)
To: Jan Kiszka, isar-users
Hi,
On 02/05/2018 01:15 PM, Jan Kiszka wrote:
> On 2018-02-05 10:59, Alexander Smirnov wrote:
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>> meta/conf/isar-bitbake.conf | 1 +
>> 1 file changed, 1 insertion(+)
>>
>> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
>> index 10cd45d..b853c88 100644
>> --- a/meta/conf/isar-bitbake.conf
>> +++ b/meta/conf/isar-bitbake.conf
>> @@ -28,5 +28,6 @@ BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
>> BB_SIGNATURE_HANDLER ?= "noop"
>>
>> include conf/local.conf
>> +include conf/multiconfig/${BB_CURRENT_MC}.conf
>> include conf/machine/${MACHINE}.conf
>> include conf/distro/${DISTRO}.conf
>>
>
> Is the preparatory to the bitbake update or does it have to be merged
> afterwards?
>
Patch to update bitbake will applied before this series, it takes 1.3MB,
so I didn't send it here.
Here is the commit message:
Update bitbake from the upstream.
Origin: https://github.com/openembedded/bitbake.git
Commit: f8126aaf774186a6eaf0bd4067b89c074594886c
Alex
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 1/5] Update isar-bitbake.conf according to the changes in bitbake related to multiconfig support.
2018-02-05 10:26 ` Alexander Smirnov
@ 2018-02-05 10:27 ` Jan Kiszka
0 siblings, 0 replies; 32+ messages in thread
From: Jan Kiszka @ 2018-02-05 10:27 UTC (permalink / raw)
To: Alexander Smirnov, isar-users
On 2018-02-05 11:26, Alexander Smirnov wrote:
> Hi,
>
> On 02/05/2018 01:15 PM, Jan Kiszka wrote:
>> On 2018-02-05 10:59, Alexander Smirnov wrote:
>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>> ---
>>> meta/conf/isar-bitbake.conf | 1 +
>>> 1 file changed, 1 insertion(+)
>>>
>>> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
>>> index 10cd45d..b853c88 100644
>>> --- a/meta/conf/isar-bitbake.conf
>>> +++ b/meta/conf/isar-bitbake.conf
>>> @@ -28,5 +28,6 @@ BUILDCHROOT_DIR =
>>> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
>>> BB_SIGNATURE_HANDLER ?= "noop"
>>> include conf/local.conf
>>> +include conf/multiconfig/${BB_CURRENT_MC}.conf
>>> include conf/machine/${MACHINE}.conf
>>> include conf/distro/${DISTRO}.conf
>>>
>>
>> Is the preparatory to the bitbake update or does it have to be merged
>> afterwards?
>>
>
> Patch to update bitbake will applied before this series, it takes 1.3MB,
> so I didn't send it here.
>
> Here is the commit message:
>
> Update bitbake from the upstream.
>
> Origin: https://github.com/openembedded/bitbake.git
> Commit: f8126aaf774186a6eaf0bd4067b89c074594886c
>
Perfect, thanks!
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 3/5] buildchroot: Enable isar-apt
2018-02-05 10:18 ` Jan Kiszka
@ 2018-02-05 10:33 ` Alexander Smirnov
0 siblings, 0 replies; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-05 10:33 UTC (permalink / raw)
To: Jan Kiszka, isar-users
On 02/05/2018 01:18 PM, Jan Kiszka wrote:
> On 2018-02-05 10:59, Alexander Smirnov wrote:
>> This patch provides access to isar-apt from buildchroot. It does the
>> following:
>> - mount isar-apt during buildchroot building.
>> - umount isar-apt using bitbake events.
>>
>> Also it needs to keep Isar build tree clean from any mounts despite on whether
>> build succeed of failed. bitbake provides various events that could trigger
>> custom python hooks. In this patch BuildCompleted event is used, which happened
>> when bitbake finished its execution despite on the result.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>> meta-isar/conf/local.conf.sample | 3 +++
>> meta/classes/isar-events.bbclass | 20 ++++++++++++++++++++
>> meta/recipes-devtools/buildchroot/buildchroot.bb | 7 +++++++
>> .../buildchroot/files/multistrap.conf.in | 8 +++++++-
>> 4 files changed, 37 insertions(+), 1 deletion(-)
>> create mode 100644 meta/classes/isar-events.bbclass
>>
>> diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
>> index 2ae43e7..fc760ed 100644
>> --- a/meta-isar/conf/local.conf.sample
>> +++ b/meta-isar/conf/local.conf.sample
>> @@ -158,3 +158,6 @@ IMAGE_INSTALL = "example-hello example-raw"
>> #
>> # Default parallel jobs for bitbake:
>> BB_NUMBER_THREADS = "4"
>> +
>> +# Add event handlers for bitbake
>> +INHERIT += "isar-events"
>> diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
>> new file mode 100644
>> index 0000000..55fc106
>> --- /dev/null
>> +++ b/meta/classes/isar-events.bbclass
>> @@ -0,0 +1,20 @@
>> +# Isar event handlers.
>> +#
>> +# This software is a part of ISAR.
>> +# Copyright (C) 2015-2017 ilbers GmbH
>> +
>> +addhandler isar_handler
>> +
>> +python isar_handler () {
>> + import subprocess
>> +
>> + devnull = open(os.devnull, 'w')
>> +
>> + if isinstance(e, bb.event.BuildCompleted):
>> + bchroot = d.getVar('BUILDCHROOT_DIR', True)
>> +
>> + # Clean up buildchroot
>> + subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot + '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True)
>> +
>> + devnull.close()
>> +}
>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
>> index 51f9d5d..cb90148 100644
>> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
>> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
>> @@ -33,6 +33,7 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>>
>> do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>> do_build[dirs] = "${WORKDIR}/hooks_multistrap"
>> +do_build[depends] = "isar-apt:do_cache_config"
>>
>> do_build() {
>> E="${@ bb.utils.export_proxies(d)}"
>> @@ -58,6 +59,9 @@ do_build() {
>> -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
>> "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
>>
>> + [ ! -d ${BUILDCHROOT_DIR}/isar-apt ] && install -d -m 555 ${BUILDCHROOT_DIR}/isar-apt
>> + sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
>> +
>> [ ! -d ${BUILDCHROOT_DIR}/proc ] && install -d -m 555 ${BUILDCHROOT_DIR}/proc
>> sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
>> _do_build_cleanup() {
>> @@ -73,6 +77,9 @@ do_build() {
>> # Install package builder script
>> sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
>>
>> + # Create share point for isar-apt
>> + sudo install -d ${BUILDCHROOT_DIR}/isar-apt
>> +
>> # Configure root filesystem
>> sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>> _do_build_cleanup
>> diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>> index a0b28e3..480a4b8 100644
>> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>> +++ b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>> @@ -6,7 +6,7 @@ noauth=true
>> unpack=true
>> ignorenativearch=true
>> bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
>> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
>> +aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
>> configscript=##CONFIG_SCRIPT##
>> setupscript=##SETUP_SCRIPT##
>> hookdir=##DIR_HOOKS##
>> @@ -29,3 +29,9 @@ source=##DISTRO_APT_SOURCE_SEC##
>> suite=##DISTRO_SUITE##/updates
>> components=##DISTRO_COMPONENTS##
>> omitdebsrc=true
>> +
>> +[isar-apt]
>> +source=file:///isar-apt
>> +suite=isar
>> +components=main
>> +omitdebsrc=true
>>
>
> On first glance, this looks identical to v1 which I tried over the
> current bitbake version we have in Isar. There I see two issues:
>
> - umount does not work if the build fails
Started replying to your previous mail, but let's discuss it here.
Clean up hooks is mostly a hack and IMHO should be replaced by events.
I use here bitbake events, see file 'meta/classes/isar-events.bbclass'.
There is event BuildCompleted which means that bitbake has finished its
pipeline despite on the result. So I've added hook to this event which
performs unmount, this also works good for double ^C.
> - re-mount will not happen if the buildchroot was built in a previous
> run and only a depending packages is built now
Good point. Will fix it.
Alex
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 5/5] build.sh: Force 'yes' for apt
2018-02-05 9:59 ` [PATCH 5/5] build.sh: Force 'yes' for apt Alexander Smirnov
@ 2018-02-05 11:52 ` Jan Kiszka
2018-02-05 13:56 ` Alexander Smirnov
0 siblings, 1 reply; 32+ messages in thread
From: Jan Kiszka @ 2018-02-05 11:52 UTC (permalink / raw)
To: Alexander Smirnov, isar-users, Cedric Hombourger
On 2018-02-05 10:59, Alexander Smirnov wrote:
> Sometimes the following error occurs with custom repo:
>
> After this operation, 84.0 kB of additional disk space will be used.
> WARNING: The following packages cannot be authenticated!
> libhello libhello-dev
> E: There are problems and -y was used without --force-yes
>
> This patch adds '--force-yes' option to be sure that no dialog will occur.
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta/recipes-devtools/buildchroot/files/build.sh | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
> index b63a27e..06df1da 100644
> --- a/meta/recipes-devtools/buildchroot/files/build.sh
> +++ b/meta/recipes-devtools/buildchroot/files/build.sh
> @@ -21,7 +21,7 @@ cd $1
> # Notes:
> # 1) everything before the -y switch is unchanged from the defaults
> # 2) we add -y to go non-interactive
> -install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y"
> +install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y --force-yes"
>
> # Install all build deps
> mk-build-deps -t "${install_cmd}" -i -r debian/control
>
For this, I'm in favor of Cedric's patch because it has a more targeted
solution for the problem and explains it clearly this way.
However, we likely have to tune this further in the future:
- avoid that installation from all repos can be unauthenticated
- deal with key installation for 3rd-party repos, likely carrying the
keys in a meta layer
Eventually, we will also have to deal with reproducing builds from repos
that have expired keys.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH 3/5 v2] buildchroot: Enable isar-apt
2018-02-05 9:59 [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
` (4 preceding siblings ...)
2018-02-05 9:59 ` [PATCH 5/5] build.sh: Force 'yes' for apt Alexander Smirnov
@ 2018-02-05 13:48 ` Alexander Smirnov
2018-02-06 11:01 ` [PATCH 3/5 v3] " Alexander Smirnov
2018-02-05 14:41 ` [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
` (3 subsequent siblings)
9 siblings, 1 reply; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-05 13:48 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
This patch provides access to isar-apt from buildchroot. It does the
following:
- mount isar-apt during buildchroot building.
- umount isar-apt using bitbake events.
Also it needs to keep Isar build tree clean from any mounts despite on whether
build succeed of failed. bitbake provides various events that could trigger
custom python hooks. In this patch BuildCompleted event is used, which happened
when bitbake finished its execution despite on the result.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta-isar/conf/local.conf.sample | 3 +++
meta/classes/isar-events.bbclass | 20 ++++++++++++++++++++
meta/recipes-devtools/buildchroot/buildchroot.bb | 10 ++++++++++
.../buildchroot/files/multistrap.conf.in | 8 +++++++-
4 files changed, 40 insertions(+), 1 deletion(-)
create mode 100644 meta/classes/isar-events.bbclass
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 2ae43e7..fc760ed 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -158,3 +158,6 @@ IMAGE_INSTALL = "example-hello example-raw"
#
# Default parallel jobs for bitbake:
BB_NUMBER_THREADS = "4"
+
+# Add event handlers for bitbake
+INHERIT += "isar-events"
diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
new file mode 100644
index 0000000..55fc106
--- /dev/null
+++ b/meta/classes/isar-events.bbclass
@@ -0,0 +1,20 @@
+# Isar event handlers.
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2017 ilbers GmbH
+
+addhandler isar_handler
+
+python isar_handler () {
+ import subprocess
+
+ devnull = open(os.devnull, 'w')
+
+ if isinstance(e, bb.event.BuildCompleted):
+ bchroot = d.getVar('BUILDCHROOT_DIR', True)
+
+ # Clean up buildchroot
+ subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot + '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True)
+
+ devnull.close()
+}
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 51f9d5d..0044a71 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -31,8 +31,18 @@ BUILDCHROOT_PREINSTALL ?= "gcc \
WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
+do_prepare[nostamp] = "1"
+do_prepare[dirs] = "${BUILDCHROOT_DIR}/isar-apt"
+
+do_prepare() {
+ sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
+}
+
+addtask prepare before do_build after do_unpack
+
do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
do_build[dirs] = "${WORKDIR}/hooks_multistrap"
+do_build[depends] = "isar-apt:do_cache_config"
do_build() {
E="${@ bb.utils.export_proxies(d)}"
diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
index a0b28e3..480a4b8 100644
--- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
+++ b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
@@ -6,7 +6,7 @@ noauth=true
unpack=true
ignorenativearch=true
bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
-aptsources=##DISTRO_MULTICONF_APTSOURCES##
+aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
configscript=##CONFIG_SCRIPT##
setupscript=##SETUP_SCRIPT##
hookdir=##DIR_HOOKS##
@@ -29,3 +29,9 @@ source=##DISTRO_APT_SOURCE_SEC##
suite=##DISTRO_SUITE##/updates
components=##DISTRO_COMPONENTS##
omitdebsrc=true
+
+[isar-apt]
+source=file:///isar-apt
+suite=isar
+components=main
+omitdebsrc=true
--
2.1.4
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 5/5] build.sh: Force 'yes' for apt
2018-02-05 11:52 ` Jan Kiszka
@ 2018-02-05 13:56 ` Alexander Smirnov
0 siblings, 0 replies; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-05 13:56 UTC (permalink / raw)
To: Jan Kiszka, isar-users, Cedric Hombourger
On 02/05/2018 02:52 PM, Jan Kiszka wrote:
> On 2018-02-05 10:59, Alexander Smirnov wrote:
>> Sometimes the following error occurs with custom repo:
>>
>> After this operation, 84.0 kB of additional disk space will be used.
>> WARNING: The following packages cannot be authenticated!
>> libhello libhello-dev
>> E: There are problems and -y was used without --force-yes
>>
>> This patch adds '--force-yes' option to be sure that no dialog will occur.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>> meta/recipes-devtools/buildchroot/files/build.sh | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
>> index b63a27e..06df1da 100644
>> --- a/meta/recipes-devtools/buildchroot/files/build.sh
>> +++ b/meta/recipes-devtools/buildchroot/files/build.sh
>> @@ -21,7 +21,7 @@ cd $1
>> # Notes:
>> # 1) everything before the -y switch is unchanged from the defaults
>> # 2) we add -y to go non-interactive
>> -install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y"
>> +install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y --force-yes"
>>
>> # Install all build deps
>> mk-build-deps -t "${install_cmd}" -i -r debian/control
>>
>
> For this, I'm in favor of Cedric's patch because it has a more targeted
> solution for the problem and explains it clearly this way.
>
Tested this series with his patch, works good - so no objection from me.
So my one dropped and this will be applied.
> However, we likely have to tune this further in the future:
>
> - avoid that installation from all repos can be unauthenticated
> - deal with key installation for 3rd-party repos, likely carrying the
> keys in a meta layer
>
> Eventually, we will also have to deal with reproducing builds from repos
> that have expired keys.
>
The issue for me here is traceability for such problems where we disable
interactive interface. To get error message about unsigned repo I had to
run 'apt-get' manually inside buildchroot, bitbake do_build.log was
empty. :-( It would be nice to have some traceability without manual
attempts to reproduce the issue.
P.S. I just share my thoughts, no questions to this patch.
Alex
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 0/5] Rework isar-apt v2
2018-02-05 9:59 [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
` (5 preceding siblings ...)
2018-02-05 13:48 ` [PATCH 3/5 v2] buildchroot: Enable isar-apt Alexander Smirnov
@ 2018-02-05 14:41 ` Alexander Smirnov
2018-02-05 16:47 ` Jan Kiszka
2018-02-06 13:57 ` [PATCH 3/5 v4] buildchroot: Enable isar-apt Alexander Smirnov
` (2 subsequent siblings)
9 siblings, 1 reply; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-05 14:41 UTC (permalink / raw)
To: isar-users, Kiszka, Jan
Any other comments or I'll apply this to next?
Alex
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 0/5] Rework isar-apt v2
2018-02-05 14:41 ` [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
@ 2018-02-05 16:47 ` Jan Kiszka
2018-02-05 16:56 ` Alexander Smirnov
0 siblings, 1 reply; 32+ messages in thread
From: Jan Kiszka @ 2018-02-05 16:47 UTC (permalink / raw)
To: Alexander Smirnov, isar-users
On 2018-02-05 15:41, Alexander Smirnov wrote:
> Any other comments or I'll apply this to next?
>
I think it's a good step forward. I wasn't able to test the exact
version yet, already due to the bitbake update, but I will once it's in
next. We can fix on top if needed.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 0/5] Rework isar-apt v2
2018-02-05 16:47 ` Jan Kiszka
@ 2018-02-05 16:56 ` Alexander Smirnov
0 siblings, 0 replies; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-05 16:56 UTC (permalink / raw)
To: Jan Kiszka, isar-users
On 02/05/2018 07:47 PM, Jan Kiszka wrote:
> On 2018-02-05 15:41, Alexander Smirnov wrote:
>> Any other comments or I'll apply this to next?
>>
>
> I think it's a good step forward. I wasn't able to test the exact
> version yet, already due to the bitbake update, but I will once it's in
> next. We can fix on top if needed.
>
Applied to next.
Alex
^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH 3/5 v3] buildchroot: Enable isar-apt
2018-02-05 13:48 ` [PATCH 3/5 v2] buildchroot: Enable isar-apt Alexander Smirnov
@ 2018-02-06 11:01 ` Alexander Smirnov
2018-02-06 12:10 ` Jan Kiszka
0 siblings, 1 reply; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-06 11:01 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
8<--
Changes since v2:
- Move do_prepare after do_build. This will not trigger do_build for
reoccurent builds.
8<--
This patch provides access to isar-apt from buildchroot. It does the
following:
- mount isar-apt during buildchroot building.
- umount isar-apt using bitbake events.
Also it needs to keep Isar build tree clean from any mounts despite on whether
build succeed of failed. bitbake provides various events that could trigger
custom python hooks. In this patch BuildCompleted event is used, which happened
when bitbake finished its execution despite on the result.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta-isar/conf/local.conf.sample | 3 +++
meta/classes/dpkg-base.bbclass | 2 +-
meta/classes/isar-events.bbclass | 20 ++++++++++++++++++++
meta/recipes-devtools/buildchroot/buildchroot.bb | 20 +++++++++++++++++++-
.../buildchroot/files/multistrap.conf.in | 8 +++++++-
5 files changed, 50 insertions(+), 3 deletions(-)
create mode 100644 meta/classes/isar-events.bbclass
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 2ae43e7..fc760ed 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -158,3 +158,6 @@ IMAGE_INSTALL = "example-hello example-raw"
#
# Default parallel jobs for bitbake:
BB_NUMBER_THREADS = "4"
+
+# Add event handlers for bitbake
+INHERIT += "isar-events"
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 026028f..74dedc5 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -2,7 +2,7 @@
# Copyright (C) 2017 Siemens AG
# Add dependency from buildchroot creation
-do_build[depends] = "buildchroot:do_build"
+do_build[depends] = "buildchroot:do_prepare"
# Each package should have its own unique build folder, so use
# recipe name as identifier
diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
new file mode 100644
index 0000000..55fc106
--- /dev/null
+++ b/meta/classes/isar-events.bbclass
@@ -0,0 +1,20 @@
+# Isar event handlers.
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2017 ilbers GmbH
+
+addhandler isar_handler
+
+python isar_handler () {
+ import subprocess
+
+ devnull = open(os.devnull, 'w')
+
+ if isinstance(e, bb.event.BuildCompleted):
+ bchroot = d.getVar('BUILDCHROOT_DIR', True)
+
+ # Clean up buildchroot
+ subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot + '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True)
+
+ devnull.close()
+}
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 51f9d5d..60195de 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -32,7 +32,9 @@ BUILDCHROOT_PREINSTALL ?= "gcc \
WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-do_build[dirs] = "${WORKDIR}/hooks_multistrap"
+do_build[dirs] = "${WORKDIR}/hooks_multistrap \
+ ${BUILDCHROOT_DIR}/isar-apt"
+do_build[depends] = "isar-apt:do_cache_config"
do_build() {
E="${@ bb.utils.export_proxies(d)}"
@@ -67,6 +69,8 @@ do_build() {
}
trap '_do_build_cleanup' EXIT
+ do_prepare
+
# Create root filesystem
sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
@@ -76,4 +80,18 @@ do_build() {
# Configure root filesystem
sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
_do_build_cleanup
+
+ do_cleanup
+}
+
+do_prepare[nostamp] = "1"
+
+do_prepare() {
+ sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
+}
+
+addtask prepare after do_build
+
+do_cleanup() {
+ sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true
}
diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
index a0b28e3..480a4b8 100644
--- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
+++ b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
@@ -6,7 +6,7 @@ noauth=true
unpack=true
ignorenativearch=true
bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
-aptsources=##DISTRO_MULTICONF_APTSOURCES##
+aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
configscript=##CONFIG_SCRIPT##
setupscript=##SETUP_SCRIPT##
hookdir=##DIR_HOOKS##
@@ -29,3 +29,9 @@ source=##DISTRO_APT_SOURCE_SEC##
suite=##DISTRO_SUITE##/updates
components=##DISTRO_COMPONENTS##
omitdebsrc=true
+
+[isar-apt]
+source=file:///isar-apt
+suite=isar
+components=main
+omitdebsrc=true
--
2.1.4
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 3/5 v3] buildchroot: Enable isar-apt
2018-02-06 11:01 ` [PATCH 3/5 v3] " Alexander Smirnov
@ 2018-02-06 12:10 ` Jan Kiszka
2018-02-06 13:28 ` Alexander Smirnov
0 siblings, 1 reply; 32+ messages in thread
From: Jan Kiszka @ 2018-02-06 12:10 UTC (permalink / raw)
To: Alexander Smirnov, isar-users
On 2018-02-06 12:01, Alexander Smirnov wrote:
> 8<--
>
> Changes since v2:
> - Move do_prepare after do_build. This will not trigger do_build for
> reoccurent builds.
Hmm, aren't you also moving it *before every* build? That's how I read
the code.
>
> 8<--
>
> This patch provides access to isar-apt from buildchroot. It does the
> following:
> - mount isar-apt during buildchroot building.
> - umount isar-apt using bitbake events.
>
> Also it needs to keep Isar build tree clean from any mounts despite on whether
> build succeed of failed. bitbake provides various events that could trigger
> custom python hooks. In this patch BuildCompleted event is used, which happened
> when bitbake finished its execution despite on the result.
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta-isar/conf/local.conf.sample | 3 +++
> meta/classes/dpkg-base.bbclass | 2 +-
> meta/classes/isar-events.bbclass | 20 ++++++++++++++++++++
> meta/recipes-devtools/buildchroot/buildchroot.bb | 20 +++++++++++++++++++-
> .../buildchroot/files/multistrap.conf.in | 8 +++++++-
> 5 files changed, 50 insertions(+), 3 deletions(-)
> create mode 100644 meta/classes/isar-events.bbclass
>
> diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
> index 2ae43e7..fc760ed 100644
> --- a/meta-isar/conf/local.conf.sample
> +++ b/meta-isar/conf/local.conf.sample
> @@ -158,3 +158,6 @@ IMAGE_INSTALL = "example-hello example-raw"
> #
> # Default parallel jobs for bitbake:
> BB_NUMBER_THREADS = "4"
> +
> +# Add event handlers for bitbake
> +INHERIT += "isar-events"
If you are already updating, please move this to a proper place
(isar-bitbake.conf) like I suggested via my other patch.
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 026028f..74dedc5 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -2,7 +2,7 @@
> # Copyright (C) 2017 Siemens AG
>
> # Add dependency from buildchroot creation
> -do_build[depends] = "buildchroot:do_build"
> +do_build[depends] = "buildchroot:do_prepare"
>
> # Each package should have its own unique build folder, so use
> # recipe name as identifier
> diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
> new file mode 100644
> index 0000000..55fc106
> --- /dev/null
> +++ b/meta/classes/isar-events.bbclass
> @@ -0,0 +1,20 @@
> +# Isar event handlers.
> +#
> +# This software is a part of ISAR.
> +# Copyright (C) 2015-2017 ilbers GmbH
> +
> +addhandler isar_handler
> +
> +python isar_handler () {
> + import subprocess
> +
> + devnull = open(os.devnull, 'w')
> +
> + if isinstance(e, bb.event.BuildCompleted):
> + bchroot = d.getVar('BUILDCHROOT_DIR', True)
> +
> + # Clean up buildchroot
> + subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot + '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True)
> +
> + devnull.close()
> +}
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
> index 51f9d5d..60195de 100644
> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
> @@ -32,7 +32,9 @@ BUILDCHROOT_PREINSTALL ?= "gcc \
> WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>
> do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> -do_build[dirs] = "${WORKDIR}/hooks_multistrap"
> +do_build[dirs] = "${WORKDIR}/hooks_multistrap \
> + ${BUILDCHROOT_DIR}/isar-apt"
> +do_build[depends] = "isar-apt:do_cache_config"
>
> do_build() {
> E="${@ bb.utils.export_proxies(d)}"
> @@ -67,6 +69,8 @@ do_build() {
> }
> trap '_do_build_cleanup' EXIT
>
> + do_prepare
> +
> # Create root filesystem
> sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
>
> @@ -76,4 +80,18 @@ do_build() {
> # Configure root filesystem
> sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
> _do_build_cleanup
> +
> + do_cleanup
> +}
> +
> +do_prepare[nostamp] = "1"
> +
> +do_prepare() {
> + sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
What prevents that this is run multiple times? Or will it simply
silently fail if the mount already exists?
> +}
Maybe better call this "do_setup_mounts" or so
> +
> +addtask prepare after do_build
That is weird. Don't we call this prior to every build anyway?
> +
> +do_cleanup() {
> + sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true
> }
> diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
> index a0b28e3..480a4b8 100644
> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
> +++ b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
> @@ -6,7 +6,7 @@ noauth=true
> unpack=true
> ignorenativearch=true
> bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
> +aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
> configscript=##CONFIG_SCRIPT##
> setupscript=##SETUP_SCRIPT##
> hookdir=##DIR_HOOKS##
> @@ -29,3 +29,9 @@ source=##DISTRO_APT_SOURCE_SEC##
> suite=##DISTRO_SUITE##/updates
> components=##DISTRO_COMPONENTS##
> omitdebsrc=true
> +
> +[isar-apt]
> +source=file:///isar-apt
> +suite=isar
> +components=main
> +omitdebsrc=true
>
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 3/5 v3] buildchroot: Enable isar-apt
2018-02-06 12:10 ` Jan Kiszka
@ 2018-02-06 13:28 ` Alexander Smirnov
0 siblings, 0 replies; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-06 13:28 UTC (permalink / raw)
To: Jan Kiszka, isar-users
On 02/06/2018 03:10 PM, Jan Kiszka wrote:
> On 2018-02-06 12:01, Alexander Smirnov wrote:
>> 8<--
>>
>> Changes since v2:
>> - Move do_prepare after do_build. This will not trigger do_build for
>> reoccurent builds.
>
> Hmm, aren't you also moving it *before every* build? That's how I read
> the code.
>
Yes, there is dependency dpkg-base:
do_build[depends] = "buildchroot:do_prepare"
so it's a mandatory dependency for each recipe to be built.
>>
>> 8<--
>>
>> This patch provides access to isar-apt from buildchroot. It does the
>> following:
>> - mount isar-apt during buildchroot building.
>> - umount isar-apt using bitbake events.
>>
>> Also it needs to keep Isar build tree clean from any mounts despite on whether
>> build succeed of failed. bitbake provides various events that could trigger
>> custom python hooks. In this patch BuildCompleted event is used, which happened
>> when bitbake finished its execution despite on the result.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>> meta-isar/conf/local.conf.sample | 3 +++
>> meta/classes/dpkg-base.bbclass | 2 +-
>> meta/classes/isar-events.bbclass | 20 ++++++++++++++++++++
>> meta/recipes-devtools/buildchroot/buildchroot.bb | 20 +++++++++++++++++++-
>> .../buildchroot/files/multistrap.conf.in | 8 +++++++-
>> 5 files changed, 50 insertions(+), 3 deletions(-)
>> create mode 100644 meta/classes/isar-events.bbclass
>>
>> diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
>> index 2ae43e7..fc760ed 100644
>> --- a/meta-isar/conf/local.conf.sample
>> +++ b/meta-isar/conf/local.conf.sample
>> @@ -158,3 +158,6 @@ IMAGE_INSTALL = "example-hello example-raw"
>> #
>> # Default parallel jobs for bitbake:
>> BB_NUMBER_THREADS = "4"
>> +
>> +# Add event handlers for bitbake
>> +INHERIT += "isar-events"
>
> If you are already updating, please move this to a proper place
> (isar-bitbake.conf) like I suggested via my other patch.
Will be done in v4.
>
>> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
>> index 026028f..74dedc5 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -2,7 +2,7 @@
>> # Copyright (C) 2017 Siemens AG
>>
>> # Add dependency from buildchroot creation
>> -do_build[depends] = "buildchroot:do_build"
>> +do_build[depends] = "buildchroot:do_prepare"
>>
>> # Each package should have its own unique build folder, so use
>> # recipe name as identifier
>> diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
>> new file mode 100644
>> index 0000000..55fc106
>> --- /dev/null
>> +++ b/meta/classes/isar-events.bbclass
>> @@ -0,0 +1,20 @@
>> +# Isar event handlers.
>> +#
>> +# This software is a part of ISAR.
>> +# Copyright (C) 2015-2017 ilbers GmbH
>> +
>> +addhandler isar_handler
>> +
>> +python isar_handler () {
>> + import subprocess
>> +
>> + devnull = open(os.devnull, 'w')
>> +
>> + if isinstance(e, bb.event.BuildCompleted):
>> + bchroot = d.getVar('BUILDCHROOT_DIR', True)
>> +
>> + # Clean up buildchroot
>> + subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot + '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True)
>> +
>> + devnull.close()
>> +}
>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
>> index 51f9d5d..60195de 100644
>> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
>> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
>> @@ -32,7 +32,9 @@ BUILDCHROOT_PREINSTALL ?= "gcc \
>> WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>>
>> do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>> -do_build[dirs] = "${WORKDIR}/hooks_multistrap"
>> +do_build[dirs] = "${WORKDIR}/hooks_multistrap \
>> + ${BUILDCHROOT_DIR}/isar-apt"
>> +do_build[depends] = "isar-apt:do_cache_config"
>>
>> do_build() {
>> E="${@ bb.utils.export_proxies(d)}"
>> @@ -67,6 +69,8 @@ do_build() {
>> }
>> trap '_do_build_cleanup' EXIT
>>
>> + do_prepare
>> +
>> # Create root filesystem
>> sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
>>
>> @@ -76,4 +80,18 @@ do_build() {
>> # Configure root filesystem
>> sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>> _do_build_cleanup
>> +
>> + do_cleanup
>> +}
>> +
>> +do_prepare[nostamp] = "1"
>> +
>> +do_prepare() {
>> + sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
>
> What prevents that this is run multiple times? Or will it simply
> silently fail if the mount already exists?
>
Here nothing, will fix in v4.
>> +}
>
> Maybe better call this "do_setup_mounts" or so
>
Agreed.
>> +
>> +addtask prepare after do_build
>
> That is weird. Don't we call this prior to every build anyway?
>
We do. See the first comment above.
Alex
>> +
>> +do_cleanup() {
>> + sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true
>> }
>> diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>> index a0b28e3..480a4b8 100644
>> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>> +++ b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>> @@ -6,7 +6,7 @@ noauth=true
>> unpack=true
>> ignorenativearch=true
>> bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
>> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
>> +aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
>> configscript=##CONFIG_SCRIPT##
>> setupscript=##SETUP_SCRIPT##
>> hookdir=##DIR_HOOKS##
>> @@ -29,3 +29,9 @@ source=##DISTRO_APT_SOURCE_SEC##
>> suite=##DISTRO_SUITE##/updates
>> components=##DISTRO_COMPONENTS##
>> omitdebsrc=true
>> +
>> +[isar-apt]
>> +source=file:///isar-apt
>> +suite=isar
>> +components=main
>> +omitdebsrc=true
>>
>
> Jan
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH 3/5 v4] buildchroot: Enable isar-apt
2018-02-05 9:59 [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
` (6 preceding siblings ...)
2018-02-05 14:41 ` [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
@ 2018-02-06 13:57 ` Alexander Smirnov
2018-02-06 17:31 ` Jan Kiszka
2018-02-08 14:37 ` Henning Schild
2018-02-06 20:24 ` [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
2018-02-08 14:59 ` Henning Schild
9 siblings, 2 replies; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-06 13:57 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
8<--
Branch to test: asmirnov/devel
8<--
This patch provides access to isar-apt from buildchroot. It does the
following:
- mount isar-apt during buildchroot building.
- umount isar-apt using bitbake events.
Also it needs to keep Isar build tree clean from any mounts despite on whether
build succeed of failed. bitbake provides various events that could trigger
custom python hooks. In this patch BuildCompleted event is used, which happened
when bitbake finished its execution despite on the result.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta/classes/dpkg-base.bbclass | 2 +-
meta/classes/isar-events.bbclass | 20 ++++++++++++++++
meta/conf/isar-bitbake.conf | 3 +++
meta/recipes-devtools/buildchroot/buildchroot.bb | 28 +++++++++++++++++++++-
.../buildchroot/files/multistrap.conf.in | 8 ++++++-
5 files changed, 58 insertions(+), 3 deletions(-)
create mode 100644 meta/classes/isar-events.bbclass
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 026028f..5d5a924 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -2,7 +2,7 @@
# Copyright (C) 2017 Siemens AG
# Add dependency from buildchroot creation
-do_build[depends] = "buildchroot:do_build"
+do_build[depends] = "buildchroot:do_setup_mounts"
# Each package should have its own unique build folder, so use
# recipe name as identifier
diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
new file mode 100644
index 0000000..55fc106
--- /dev/null
+++ b/meta/classes/isar-events.bbclass
@@ -0,0 +1,20 @@
+# Isar event handlers.
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2017 ilbers GmbH
+
+addhandler isar_handler
+
+python isar_handler () {
+ import subprocess
+
+ devnull = open(os.devnull, 'w')
+
+ if isinstance(e, bb.event.BuildCompleted):
+ bchroot = d.getVar('BUILDCHROOT_DIR', True)
+
+ # Clean up buildchroot
+ subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot + '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True)
+
+ devnull.close()
+}
diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
index b853c88..36cca29 100644
--- a/meta/conf/isar-bitbake.conf
+++ b/meta/conf/isar-bitbake.conf
@@ -27,6 +27,9 @@ BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
# Setup our default hash policy
BB_SIGNATURE_HANDLER ?= "noop"
+# Add event handlers for bitbake
+INHERIT += "isar-events"
+
include conf/local.conf
include conf/multiconfig/${BB_CURRENT_MC}.conf
include conf/machine/${MACHINE}.conf
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 51f9d5d..304c67e 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -32,7 +32,9 @@ BUILDCHROOT_PREINSTALL ?= "gcc \
WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-do_build[dirs] = "${WORKDIR}/hooks_multistrap"
+do_build[dirs] = "${WORKDIR}/hooks_multistrap \
+ ${BUILDCHROOT_DIR}/isar-apt"
+do_build[depends] = "isar-apt:do_cache_config"
do_build() {
E="${@ bb.utils.export_proxies(d)}"
@@ -67,6 +69,8 @@ do_build() {
}
trap '_do_build_cleanup' EXIT
+ do_setup_mounts
+
# Create root filesystem
sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
@@ -76,4 +80,26 @@ do_build() {
# Configure root filesystem
sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
_do_build_cleanup
+
+ do_cleanup_mounts
+}
+
+# Invalidate stamp for do_setup_mounts before each build start.
+# This will guarantee that this function will be executed once
+# per build.
+python __anonymous() {
+ stamp = d.getVar("STAMP") + ".do_setup_mounts." + d.getVarFlag("do_setup_mounts", 'stamp-extra-info')
+ os.remove(stamp) if os.path.exists(stamp) else None
+}
+
+do_setup_mounts[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+
+do_setup_mounts() {
+ sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
+}
+
+addtask setup_mounts after do_build
+
+do_cleanup_mounts() {
+ sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true
}
diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
index a0b28e3..480a4b8 100644
--- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
+++ b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
@@ -6,7 +6,7 @@ noauth=true
unpack=true
ignorenativearch=true
bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
-aptsources=##DISTRO_MULTICONF_APTSOURCES##
+aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
configscript=##CONFIG_SCRIPT##
setupscript=##SETUP_SCRIPT##
hookdir=##DIR_HOOKS##
@@ -29,3 +29,9 @@ source=##DISTRO_APT_SOURCE_SEC##
suite=##DISTRO_SUITE##/updates
components=##DISTRO_COMPONENTS##
omitdebsrc=true
+
+[isar-apt]
+source=file:///isar-apt
+suite=isar
+components=main
+omitdebsrc=true
--
2.1.4
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 3/5 v4] buildchroot: Enable isar-apt
2018-02-06 13:57 ` [PATCH 3/5 v4] buildchroot: Enable isar-apt Alexander Smirnov
@ 2018-02-06 17:31 ` Jan Kiszka
2018-02-06 18:29 ` Alexander Smirnov
2018-02-08 14:37 ` Henning Schild
1 sibling, 1 reply; 32+ messages in thread
From: Jan Kiszka @ 2018-02-06 17:31 UTC (permalink / raw)
To: Alexander Smirnov, isar-users
On 2018-02-06 14:57, Alexander Smirnov wrote:
> 8<--
>
> Branch to test: asmirnov/devel
>
> 8<--
>
> This patch provides access to isar-apt from buildchroot. It does the
> following:
> - mount isar-apt during buildchroot building.
> - umount isar-apt using bitbake events.
>
> Also it needs to keep Isar build tree clean from any mounts despite on whether
> build succeed of failed. bitbake provides various events that could trigger
> custom python hooks. In this patch BuildCompleted event is used, which happened
> when bitbake finished its execution despite on the result.
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta/classes/dpkg-base.bbclass | 2 +-
> meta/classes/isar-events.bbclass | 20 ++++++++++++++++
> meta/conf/isar-bitbake.conf | 3 +++
> meta/recipes-devtools/buildchroot/buildchroot.bb | 28 +++++++++++++++++++++-
> .../buildchroot/files/multistrap.conf.in | 8 ++++++-
> 5 files changed, 58 insertions(+), 3 deletions(-)
> create mode 100644 meta/classes/isar-events.bbclass
>
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 026028f..5d5a924 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -2,7 +2,7 @@
> # Copyright (C) 2017 Siemens AG
>
> # Add dependency from buildchroot creation
> -do_build[depends] = "buildchroot:do_build"
> +do_build[depends] = "buildchroot:do_setup_mounts"
>
> # Each package should have its own unique build folder, so use
> # recipe name as identifier
> diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
> new file mode 100644
> index 0000000..55fc106
> --- /dev/null
> +++ b/meta/classes/isar-events.bbclass
> @@ -0,0 +1,20 @@
> +# Isar event handlers.
> +#
> +# This software is a part of ISAR.
> +# Copyright (C) 2015-2017 ilbers GmbH
> +
> +addhandler isar_handler
> +
> +python isar_handler () {
> + import subprocess
> +
> + devnull = open(os.devnull, 'w')
> +
> + if isinstance(e, bb.event.BuildCompleted):
> + bchroot = d.getVar('BUILDCHROOT_DIR', True)
> +
> + # Clean up buildchroot
> + subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot + '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True)
> +
> + devnull.close()
> +}
> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
> index b853c88..36cca29 100644
> --- a/meta/conf/isar-bitbake.conf
> +++ b/meta/conf/isar-bitbake.conf
> @@ -27,6 +27,9 @@ BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
> # Setup our default hash policy
> BB_SIGNATURE_HANDLER ?= "noop"
>
> +# Add event handlers for bitbake
> +INHERIT += "isar-events"
> +
> include conf/local.conf
> include conf/multiconfig/${BB_CURRENT_MC}.conf
> include conf/machine/${MACHINE}.conf
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
> index 51f9d5d..304c67e 100644
> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
> @@ -32,7 +32,9 @@ BUILDCHROOT_PREINSTALL ?= "gcc \
> WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>
> do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> -do_build[dirs] = "${WORKDIR}/hooks_multistrap"
> +do_build[dirs] = "${WORKDIR}/hooks_multistrap \
> + ${BUILDCHROOT_DIR}/isar-apt"
> +do_build[depends] = "isar-apt:do_cache_config"
>
> do_build() {
> E="${@ bb.utils.export_proxies(d)}"
> @@ -67,6 +69,8 @@ do_build() {
> }
> trap '_do_build_cleanup' EXIT
>
> + do_setup_mounts
> +
> # Create root filesystem
> sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
>
> @@ -76,4 +80,26 @@ do_build() {
> # Configure root filesystem
> sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
> _do_build_cleanup
> +
> + do_cleanup_mounts
> +}
> +
> +# Invalidate stamp for do_setup_mounts before each build start.
> +# This will guarantee that this function will be executed once
> +# per build.
> +python __anonymous() {
> + stamp = d.getVar("STAMP") + ".do_setup_mounts." + d.getVarFlag("do_setup_mounts", 'stamp-extra-info')
> + os.remove(stamp) if os.path.exists(stamp) else None
> +}
> +
> +do_setup_mounts[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +
> +do_setup_mounts() {
> + sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
> +}
> +
> +addtask setup_mounts after do_build
> +
> +do_cleanup_mounts() {
> + sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true
> }
> diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
> index a0b28e3..480a4b8 100644
> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
> +++ b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
> @@ -6,7 +6,7 @@ noauth=true
> unpack=true
> ignorenativearch=true
> bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
> +aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
> configscript=##CONFIG_SCRIPT##
> setupscript=##SETUP_SCRIPT##
> hookdir=##DIR_HOOKS##
> @@ -29,3 +29,9 @@ source=##DISTRO_APT_SOURCE_SEC##
> suite=##DISTRO_SUITE##/updates
> components=##DISTRO_COMPONENTS##
> omitdebsrc=true
> +
> +[isar-apt]
> +source=file:///isar-apt
> +suite=isar
> +components=main
> +omitdebsrc=true
>
Looks good! I'm testing it via your devel branch, and I also
successfully rebased "Permanently mount '/dev', '/sys' and '/proc'" on top.
Regarding the remaining cleanup of WORKDIR mounting: Can't each
dpkg-base class register their own cleanup handler in the same fashion?
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 3/5 v4] buildchroot: Enable isar-apt
2018-02-06 17:31 ` Jan Kiszka
@ 2018-02-06 18:29 ` Alexander Smirnov
2018-02-06 18:31 ` Jan Kiszka
0 siblings, 1 reply; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-06 18:29 UTC (permalink / raw)
To: Jan Kiszka, isar-users
On 02/06/2018 08:31 PM, Jan Kiszka wrote:
> On 2018-02-06 14:57, Alexander Smirnov wrote:
>> 8<--
>>
>> Branch to test: asmirnov/devel
>>
>> 8<--
>>
>> This patch provides access to isar-apt from buildchroot. It does the
>> following:
>> - mount isar-apt during buildchroot building.
>> - umount isar-apt using bitbake events.
>>
>> Also it needs to keep Isar build tree clean from any mounts despite on whether
>> build succeed of failed. bitbake provides various events that could trigger
>> custom python hooks. In this patch BuildCompleted event is used, which happened
>> when bitbake finished its execution despite on the result.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>> meta/classes/dpkg-base.bbclass | 2 +-
>> meta/classes/isar-events.bbclass | 20 ++++++++++++++++
>> meta/conf/isar-bitbake.conf | 3 +++
>> meta/recipes-devtools/buildchroot/buildchroot.bb | 28 +++++++++++++++++++++-
>> .../buildchroot/files/multistrap.conf.in | 8 ++++++-
>> 5 files changed, 58 insertions(+), 3 deletions(-)
>> create mode 100644 meta/classes/isar-events.bbclass
>>
>> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
>> index 026028f..5d5a924 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -2,7 +2,7 @@
>> # Copyright (C) 2017 Siemens AG
>>
>> # Add dependency from buildchroot creation
>> -do_build[depends] = "buildchroot:do_build"
>> +do_build[depends] = "buildchroot:do_setup_mounts"
>>
>> # Each package should have its own unique build folder, so use
>> # recipe name as identifier
>> diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
>> new file mode 100644
>> index 0000000..55fc106
>> --- /dev/null
>> +++ b/meta/classes/isar-events.bbclass
>> @@ -0,0 +1,20 @@
>> +# Isar event handlers.
>> +#
>> +# This software is a part of ISAR.
>> +# Copyright (C) 2015-2017 ilbers GmbH
>> +
>> +addhandler isar_handler
>> +
>> +python isar_handler () {
>> + import subprocess
>> +
>> + devnull = open(os.devnull, 'w')
>> +
>> + if isinstance(e, bb.event.BuildCompleted):
>> + bchroot = d.getVar('BUILDCHROOT_DIR', True)
>> +
>> + # Clean up buildchroot
>> + subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot + '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True)
>> +
>> + devnull.close()
>> +}
>> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
>> index b853c88..36cca29 100644
>> --- a/meta/conf/isar-bitbake.conf
>> +++ b/meta/conf/isar-bitbake.conf
>> @@ -27,6 +27,9 @@ BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
>> # Setup our default hash policy
>> BB_SIGNATURE_HANDLER ?= "noop"
>>
>> +# Add event handlers for bitbake
>> +INHERIT += "isar-events"
>> +
>> include conf/local.conf
>> include conf/multiconfig/${BB_CURRENT_MC}.conf
>> include conf/machine/${MACHINE}.conf
>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
>> index 51f9d5d..304c67e 100644
>> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
>> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
>> @@ -32,7 +32,9 @@ BUILDCHROOT_PREINSTALL ?= "gcc \
>> WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>>
>> do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>> -do_build[dirs] = "${WORKDIR}/hooks_multistrap"
>> +do_build[dirs] = "${WORKDIR}/hooks_multistrap \
>> + ${BUILDCHROOT_DIR}/isar-apt"
>> +do_build[depends] = "isar-apt:do_cache_config"
>>
>> do_build() {
>> E="${@ bb.utils.export_proxies(d)}"
>> @@ -67,6 +69,8 @@ do_build() {
>> }
>> trap '_do_build_cleanup' EXIT
>>
>> + do_setup_mounts
>> +
>> # Create root filesystem
>> sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
>>
>> @@ -76,4 +80,26 @@ do_build() {
>> # Configure root filesystem
>> sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>> _do_build_cleanup
>> +
>> + do_cleanup_mounts
>> +}
>> +
>> +# Invalidate stamp for do_setup_mounts before each build start.
>> +# This will guarantee that this function will be executed once
>> +# per build.
>> +python __anonymous() {
>> + stamp = d.getVar("STAMP") + ".do_setup_mounts." + d.getVarFlag("do_setup_mounts", 'stamp-extra-info')
>> + os.remove(stamp) if os.path.exists(stamp) else None
>> +}
>> +
>> +do_setup_mounts[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>> +
>> +do_setup_mounts() {
>> + sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
>> +}
>> +
>> +addtask setup_mounts after do_build
>> +
>> +do_cleanup_mounts() {
>> + sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true
>> }
>> diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>> index a0b28e3..480a4b8 100644
>> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>> +++ b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>> @@ -6,7 +6,7 @@ noauth=true
>> unpack=true
>> ignorenativearch=true
>> bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
>> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
>> +aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
>> configscript=##CONFIG_SCRIPT##
>> setupscript=##SETUP_SCRIPT##
>> hookdir=##DIR_HOOKS##
>> @@ -29,3 +29,9 @@ source=##DISTRO_APT_SOURCE_SEC##
>> suite=##DISTRO_SUITE##/updates
>> components=##DISTRO_COMPONENTS##
>> omitdebsrc=true
>> +
>> +[isar-apt]
>> +source=file:///isar-apt
>> +suite=isar
>> +components=main
>> +omitdebsrc=true
>>
>
> Looks good! I'm testing it via your devel branch, and I also
> successfully rebased "Permanently mount '/dev', '/sys' and '/proc'" on top.
>
Thanks!
> Regarding the remaining cleanup of WORKDIR mounting: Can't each
> dpkg-base class register their own cleanup handler in the same fashion?
>
I've remembered, that for Jenkins I've used something following to clean
the build automatically:
8<--
#!/bin/bash
W=${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}
for m in $(cat /proc/mounts | grep $W); do
m=$(echo $m | cut -d ' ' -f 2)
umount $m
done
8<--
There are some facts:
1. Proc folder must be available for correct Isar operating.
2. /proc/mounts contains all the active mounts.
3. Folder $W contains only mounts created by current build.
Based on this, I think if we insert the code above to the event handler,
it will be robust enough solution to clean up everything. Also we should
not anymore care if somebody adds some extra mount.
What do you think?
Alex
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 3/5 v4] buildchroot: Enable isar-apt
2018-02-06 18:29 ` Alexander Smirnov
@ 2018-02-06 18:31 ` Jan Kiszka
0 siblings, 0 replies; 32+ messages in thread
From: Jan Kiszka @ 2018-02-06 18:31 UTC (permalink / raw)
To: Alexander Smirnov, isar-users
On 2018-02-06 19:29, Alexander Smirnov wrote:
> On 02/06/2018 08:31 PM, Jan Kiszka wrote:
>> On 2018-02-06 14:57, Alexander Smirnov wrote:
>>> 8<--
>>>
>>> Branch to test: asmirnov/devel
>>>
>>> 8<--
>>>
>>> This patch provides access to isar-apt from buildchroot. It does the
>>> following:
>>> - mount isar-apt during buildchroot building.
>>> - umount isar-apt using bitbake events.
>>>
>>> Also it needs to keep Isar build tree clean from any mounts despite
>>> on whether
>>> build succeed of failed. bitbake provides various events that could
>>> trigger
>>> custom python hooks. In this patch BuildCompleted event is used,
>>> which happened
>>> when bitbake finished its execution despite on the result.
>>>
>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>> ---
>>> meta/classes/dpkg-base.bbclass | 2 +-
>>> meta/classes/isar-events.bbclass | 20
>>> ++++++++++++++++
>>> meta/conf/isar-bitbake.conf | 3 +++
>>> meta/recipes-devtools/buildchroot/buildchroot.bb | 28
>>> +++++++++++++++++++++-
>>> .../buildchroot/files/multistrap.conf.in | 8 ++++++-
>>> 5 files changed, 58 insertions(+), 3 deletions(-)
>>> create mode 100644 meta/classes/isar-events.bbclass
>>>
>>> diff --git a/meta/classes/dpkg-base.bbclass
>>> b/meta/classes/dpkg-base.bbclass
>>> index 026028f..5d5a924 100644
>>> --- a/meta/classes/dpkg-base.bbclass
>>> +++ b/meta/classes/dpkg-base.bbclass
>>> @@ -2,7 +2,7 @@
>>> # Copyright (C) 2017 Siemens AG
>>> # Add dependency from buildchroot creation
>>> -do_build[depends] = "buildchroot:do_build"
>>> +do_build[depends] = "buildchroot:do_setup_mounts"
>>> # Each package should have its own unique build folder, so use
>>> # recipe name as identifier
>>> diff --git a/meta/classes/isar-events.bbclass
>>> b/meta/classes/isar-events.bbclass
>>> new file mode 100644
>>> index 0000000..55fc106
>>> --- /dev/null
>>> +++ b/meta/classes/isar-events.bbclass
>>> @@ -0,0 +1,20 @@
>>> +# Isar event handlers.
>>> +#
>>> +# This software is a part of ISAR.
>>> +# Copyright (C) 2015-2017 ilbers GmbH
>>> +
>>> +addhandler isar_handler
>>> +
>>> +python isar_handler () {
>>> + import subprocess
>>> +
>>> + devnull = open(os.devnull, 'w')
>>> +
>>> + if isinstance(e, bb.event.BuildCompleted):
>>> + bchroot = d.getVar('BUILDCHROOT_DIR', True)
>>> +
>>> + # Clean up buildchroot
>>> + subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot +
>>> '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True)
>>> +
>>> + devnull.close()
>>> +}
>>> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
>>> index b853c88..36cca29 100644
>>> --- a/meta/conf/isar-bitbake.conf
>>> +++ b/meta/conf/isar-bitbake.conf
>>> @@ -27,6 +27,9 @@ BUILDCHROOT_DIR =
>>> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
>>> # Setup our default hash policy
>>> BB_SIGNATURE_HANDLER ?= "noop"
>>> +# Add event handlers for bitbake
>>> +INHERIT += "isar-events"
>>> +
>>> include conf/local.conf
>>> include conf/multiconfig/${BB_CURRENT_MC}.conf
>>> include conf/machine/${MACHINE}.conf
>>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
>>> b/meta/recipes-devtools/buildchroot/buildchroot.bb
>>> index 51f9d5d..304c67e 100644
>>> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
>>> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
>>> @@ -32,7 +32,9 @@ BUILDCHROOT_PREINSTALL ?= "gcc \
>>> WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>>> do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>> -do_build[dirs] = "${WORKDIR}/hooks_multistrap"
>>> +do_build[dirs] = "${WORKDIR}/hooks_multistrap \
>>> + ${BUILDCHROOT_DIR}/isar-apt"
>>> +do_build[depends] = "isar-apt:do_cache_config"
>>> do_build() {
>>> E="${@ bb.utils.export_proxies(d)}"
>>> @@ -67,6 +69,8 @@ do_build() {
>>> }
>>> trap '_do_build_cleanup' EXIT
>>> + do_setup_mounts
>>> +
>>> # Create root filesystem
>>> sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f
>>> "${WORKDIR}/multistrap.conf"
>>> @@ -76,4 +80,26 @@ do_build() {
>>> # Configure root filesystem
>>> sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>>> _do_build_cleanup
>>> +
>>> + do_cleanup_mounts
>>> +}
>>> +
>>> +# Invalidate stamp for do_setup_mounts before each build start.
>>> +# This will guarantee that this function will be executed once
>>> +# per build.
>>> +python __anonymous() {
>>> + stamp = d.getVar("STAMP") + ".do_setup_mounts." +
>>> d.getVarFlag("do_setup_mounts", 'stamp-extra-info')
>>> + os.remove(stamp) if os.path.exists(stamp) else None
>>> +}
>>> +
>>> +do_setup_mounts[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>> +
>>> +do_setup_mounts() {
>>> + sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO}
>>> ${BUILDCHROOT_DIR}/isar-apt
>>> +}
>>> +
>>> +addtask setup_mounts after do_build
>>> +
>>> +do_cleanup_mounts() {
>>> + sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true
>>> }
>>> diff --git
>>> a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>>> b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>>> index a0b28e3..480a4b8 100644
>>> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>>> +++ b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>>> @@ -6,7 +6,7 @@ noauth=true
>>> unpack=true
>>> ignorenativearch=true
>>> bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
>>> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
>>> +aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
>>> configscript=##CONFIG_SCRIPT##
>>> setupscript=##SETUP_SCRIPT##
>>> hookdir=##DIR_HOOKS##
>>> @@ -29,3 +29,9 @@ source=##DISTRO_APT_SOURCE_SEC##
>>> suite=##DISTRO_SUITE##/updates
>>> components=##DISTRO_COMPONENTS##
>>> omitdebsrc=true
>>> +
>>> +[isar-apt]
>>> +source=file:///isar-apt
>>> +suite=isar
>>> +components=main
>>> +omitdebsrc=true
>>>
>>
>> Looks good! I'm testing it via your devel branch, and I also
>> successfully rebased "Permanently mount '/dev', '/sys' and '/proc'" on
>> top.
>>
>
> Thanks!
>
>> Regarding the remaining cleanup of WORKDIR mounting: Can't each
>> dpkg-base class register their own cleanup handler in the same fashion?
>>
>
> I've remembered, that for Jenkins I've used something following to clean
> the build automatically:
>
> 8<--
>
> #!/bin/bash
>
> W=${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}
>
> for m in $(cat /proc/mounts | grep $W); do
> m=$(echo $m | cut -d ' ' -f 2)
> umount $m
> done
>
> 8<--
>
> There are some facts:
>
> 1. Proc folder must be available for correct Isar operating.
> 2. /proc/mounts contains all the active mounts.
> 3. Folder $W contains only mounts created by current build.
>
> Based on this, I think if we insert the code above to the event handler,
> it will be robust enough solution to clean up everything. Also we should
> not anymore care if somebody adds some extra mount.
>
> What do you think?
Could work...
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 0/5] Rework isar-apt v2
2018-02-05 9:59 [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
` (7 preceding siblings ...)
2018-02-06 13:57 ` [PATCH 3/5 v4] buildchroot: Enable isar-apt Alexander Smirnov
@ 2018-02-06 20:24 ` Alexander Smirnov
2018-02-08 14:59 ` Henning Schild
9 siblings, 0 replies; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-06 20:24 UTC (permalink / raw)
To: isar-users
On 02/05/2018 12:59 PM, Alexander Smirnov wrote:
> Hi all,
>
> this series intended to make buildchroot able to work with isar-apt.
> I've tried to add extended comments to each patch.
>
> Impact: with this series I'm able to build 'example-hello' <-> 'libhello'
> without any hacks. So the deps are installed automatically.
>
> Changes since v1:
> - The bitbake was updated to the latest upstream version
> - In build.sh the only isar-apt is updated
>
>
> Alexander Smirnov (5):
> Update isar-bitbake.conf according to the changes in bitbake related
> to multiconfig support.
> isar-apt: Introduce separate recipe
> buildchroot: Enable isar-apt
> build.sh: Update apt sources
> build.sh: Force 'yes' for apt
>
> meta-isar/conf/local.conf.sample | 5 +-
> .../recipes-core/images/files/distributions.in | 3 -
> meta/classes/dpkg-base.bbclass | 12 +++-
> meta/classes/image.bbclass | 67 +---------------------
> meta/classes/isar-events.bbclass | 20 +++++++
> meta/conf/isar-bitbake.conf | 1 +
> meta/recipes-devtools/buildchroot/buildchroot.bb | 7 +++
> meta/recipes-devtools/buildchroot/files/build.sh | 13 ++++-
> .../buildchroot/files/multistrap.conf.in | 8 ++-
> .../isar-apt/files/distributions.in | 3 +
> meta/recipes-devtools/isar-apt/isar-apt.bb | 29 ++++++++++
> 11 files changed, 92 insertions(+), 76 deletions(-)
> delete mode 100644 meta-isar/recipes-core/images/files/distributions.in
> create mode 100644 meta/classes/isar-events.bbclass
> create mode 100644 meta/recipes-devtools/isar-apt/files/distributions.in
> create mode 100644 meta/recipes-devtools/isar-apt/isar-apt.bb
Applied to next.
Alex
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 3/5 v4] buildchroot: Enable isar-apt
2018-02-06 13:57 ` [PATCH 3/5 v4] buildchroot: Enable isar-apt Alexander Smirnov
2018-02-06 17:31 ` Jan Kiszka
@ 2018-02-08 14:37 ` Henning Schild
2018-02-08 14:57 ` Alexander Smirnov
1 sibling, 1 reply; 32+ messages in thread
From: Henning Schild @ 2018-02-08 14:37 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users
Am Tue, 6 Feb 2018 16:57:25 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 8<--
>
> Branch to test: asmirnov/devel
>
> 8<--
>
> This patch provides access to isar-apt from buildchroot. It does the
> following:
> - mount isar-apt during buildchroot building.
> - umount isar-apt using bitbake events.
>
> Also it needs to keep Isar build tree clean from any mounts despite
> on whether build succeed of failed. bitbake provides various events
> that could trigger custom python hooks. In this patch BuildCompleted
> event is used, which happened when bitbake finished its execution
> despite on the result.
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta/classes/dpkg-base.bbclass | 2 +-
> meta/classes/isar-events.bbclass | 20
> ++++++++++++++++ meta/conf/isar-bitbake.conf
> | 3 +++ meta/recipes-devtools/buildchroot/buildchroot.bb | 28
> +++++++++++++++++++++- .../buildchroot/files/multistrap.conf.in
> | 8 ++++++- 5 files changed, 58 insertions(+), 3 deletions(-)
> create mode 100644 meta/classes/isar-events.bbclass
>
> diff --git a/meta/classes/dpkg-base.bbclass
> b/meta/classes/dpkg-base.bbclass index 026028f..5d5a924 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -2,7 +2,7 @@
> # Copyright (C) 2017 Siemens AG
>
> # Add dependency from buildchroot creation
> -do_build[depends] = "buildchroot:do_build"
> +do_build[depends] = "buildchroot:do_setup_mounts"
>
> # Each package should have its own unique build folder, so use
> # recipe name as identifier
> diff --git a/meta/classes/isar-events.bbclass
> b/meta/classes/isar-events.bbclass new file mode 100644
> index 0000000..55fc106
> --- /dev/null
> +++ b/meta/classes/isar-events.bbclass
> @@ -0,0 +1,20 @@
> +# Isar event handlers.
> +#
> +# This software is a part of ISAR.
> +# Copyright (C) 2015-2017 ilbers GmbH
> +
> +addhandler isar_handler
> +
> +python isar_handler () {
> + import subprocess
> +
> + devnull = open(os.devnull, 'w')
> +
> + if isinstance(e, bb.event.BuildCompleted):
> + bchroot = d.getVar('BUILDCHROOT_DIR', True)
> +
> + # Clean up buildchroot
> + subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot +
> '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True)
I would suggest to drop the hardcoded paths where possible. Imagine i
had my sudo sitting in /usr/local/bin/ or similar. And sudo internally
will take care of its own PATH so the /bin/ for umount is also not
required.
Another use-case could be a wrapper around the actual sudo where people
try to handle the sudo-problem of Isar by having a wrapper that uses a
whitelist of allowed commands.
Henning
> +
> + devnull.close()
> +}
> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
> index b853c88..36cca29 100644
> --- a/meta/conf/isar-bitbake.conf
> +++ b/meta/conf/isar-bitbake.conf
> @@ -27,6 +27,9 @@ BUILDCHROOT_DIR =
> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs" # Setup
> our default hash policy BB_SIGNATURE_HANDLER ?= "noop"
>
> +# Add event handlers for bitbake
> +INHERIT += "isar-events"
> +
> include conf/local.conf
> include conf/multiconfig/${BB_CURRENT_MC}.conf
> include conf/machine/${MACHINE}.conf
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
> 51f9d5d..304c67e 100644 ---
> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -32,7 +32,9 @@
> BUILDCHROOT_PREINSTALL ?= "gcc \ WORKDIR =
> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
> do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> -do_build[dirs] = "${WORKDIR}/hooks_multistrap"
> +do_build[dirs] = "${WORKDIR}/hooks_multistrap \
> + ${BUILDCHROOT_DIR}/isar-apt"
> +do_build[depends] = "isar-apt:do_cache_config"
>
> do_build() {
> E="${@ bb.utils.export_proxies(d)}"
> @@ -67,6 +69,8 @@ do_build() {
> }
> trap '_do_build_cleanup' EXIT
>
> + do_setup_mounts
> +
> # Create root filesystem
> sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f
> "${WORKDIR}/multistrap.conf"
> @@ -76,4 +80,26 @@ do_build() {
> # Configure root filesystem
> sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
> _do_build_cleanup
> +
> + do_cleanup_mounts
> +}
> +
> +# Invalidate stamp for do_setup_mounts before each build start.
> +# This will guarantee that this function will be executed once
> +# per build.
> +python __anonymous() {
> + stamp = d.getVar("STAMP") + ".do_setup_mounts." +
> d.getVarFlag("do_setup_mounts", 'stamp-extra-info')
> + os.remove(stamp) if os.path.exists(stamp) else None
> +}
> +
> +do_setup_mounts[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +
> +do_setup_mounts() {
> + sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO}
> ${BUILDCHROOT_DIR}/isar-apt +}
> +
> +addtask setup_mounts after do_build
> +
> +do_cleanup_mounts() {
> + sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true
> }
> diff --git
> a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
> b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in index
> a0b28e3..480a4b8 100644 ---
> a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in +++
> b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in @@ -6,7
> +6,7 @@ noauth=true unpack=true ignorenativearch=true
> bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
> +aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
> configscript=##CONFIG_SCRIPT##
> setupscript=##SETUP_SCRIPT##
> hookdir=##DIR_HOOKS##
> @@ -29,3 +29,9 @@ source=##DISTRO_APT_SOURCE_SEC##
> suite=##DISTRO_SUITE##/updates
> components=##DISTRO_COMPONENTS##
> omitdebsrc=true
> +
> +[isar-apt]
> +source=file:///isar-apt
> +suite=isar
> +components=main
> +omitdebsrc=true
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 3/5 v4] buildchroot: Enable isar-apt
2018-02-08 14:37 ` Henning Schild
@ 2018-02-08 14:57 ` Alexander Smirnov
0 siblings, 0 replies; 32+ messages in thread
From: Alexander Smirnov @ 2018-02-08 14:57 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On 02/08/2018 05:37 PM, Henning Schild wrote:
> Am Tue, 6 Feb 2018 16:57:25 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>
>> 8<--
>>
>> Branch to test: asmirnov/devel
>>
>> 8<--
>>
>> This patch provides access to isar-apt from buildchroot. It does the
>> following:
>> - mount isar-apt during buildchroot building.
>> - umount isar-apt using bitbake events.
>>
>> Also it needs to keep Isar build tree clean from any mounts despite
>> on whether build succeed of failed. bitbake provides various events
>> that could trigger custom python hooks. In this patch BuildCompleted
>> event is used, which happened when bitbake finished its execution
>> despite on the result.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>> meta/classes/dpkg-base.bbclass | 2 +-
>> meta/classes/isar-events.bbclass | 20
>> ++++++++++++++++ meta/conf/isar-bitbake.conf
>> | 3 +++ meta/recipes-devtools/buildchroot/buildchroot.bb | 28
>> +++++++++++++++++++++- .../buildchroot/files/multistrap.conf.in
>> | 8 ++++++- 5 files changed, 58 insertions(+), 3 deletions(-)
>> create mode 100644 meta/classes/isar-events.bbclass
>>
>> diff --git a/meta/classes/dpkg-base.bbclass
>> b/meta/classes/dpkg-base.bbclass index 026028f..5d5a924 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -2,7 +2,7 @@
>> # Copyright (C) 2017 Siemens AG
>>
>> # Add dependency from buildchroot creation
>> -do_build[depends] = "buildchroot:do_build"
>> +do_build[depends] = "buildchroot:do_setup_mounts"
>>
>> # Each package should have its own unique build folder, so use
>> # recipe name as identifier
>> diff --git a/meta/classes/isar-events.bbclass
>> b/meta/classes/isar-events.bbclass new file mode 100644
>> index 0000000..55fc106
>> --- /dev/null
>> +++ b/meta/classes/isar-events.bbclass
>> @@ -0,0 +1,20 @@
>> +# Isar event handlers.
>> +#
>> +# This software is a part of ISAR.
>> +# Copyright (C) 2015-2017 ilbers GmbH
>> +
>> +addhandler isar_handler
>> +
>> +python isar_handler () {
>> + import subprocess
>> +
>> + devnull = open(os.devnull, 'w')
>> +
>> + if isinstance(e, bb.event.BuildCompleted):
>> + bchroot = d.getVar('BUILDCHROOT_DIR', True)
>> +
>> + # Clean up buildchroot
>> + subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot +
>> '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True)
>
> I would suggest to drop the hardcoded paths where possible. Imagine i
> had my sudo sitting in /usr/local/bin/ or similar. And sudo internally
> will take care of its own PATH so the /bin/ for umount is also not
> required.
> Another use-case could be a wrapper around the actual sudo where people
> try to handle the sudo-problem of Isar by having a wrapper that uses a
> whitelist of allowed commands.
>
This issue resolved in patch:
[PATCH] isar: Clean mount point on bitbake exit
Alex
> Henning
>
>> +
>> + devnull.close()
>> +}
>> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
>> index b853c88..36cca29 100644
>> --- a/meta/conf/isar-bitbake.conf
>> +++ b/meta/conf/isar-bitbake.conf
>> @@ -27,6 +27,9 @@ BUILDCHROOT_DIR =
>> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs" # Setup
>> our default hash policy BB_SIGNATURE_HANDLER ?= "noop"
>>
>> +# Add event handlers for bitbake
>> +INHERIT += "isar-events"
>> +
>> include conf/local.conf
>> include conf/multiconfig/${BB_CURRENT_MC}.conf
>> include conf/machine/${MACHINE}.conf
>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
>> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
>> 51f9d5d..304c67e 100644 ---
>> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
>> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -32,7 +32,9 @@
>> BUILDCHROOT_PREINSTALL ?= "gcc \ WORKDIR =
>> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>> do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>> -do_build[dirs] = "${WORKDIR}/hooks_multistrap"
>> +do_build[dirs] = "${WORKDIR}/hooks_multistrap \
>> + ${BUILDCHROOT_DIR}/isar-apt"
>> +do_build[depends] = "isar-apt:do_cache_config"
>>
>> do_build() {
>> E="${@ bb.utils.export_proxies(d)}"
>> @@ -67,6 +69,8 @@ do_build() {
>> }
>> trap '_do_build_cleanup' EXIT
>>
>> + do_setup_mounts
>> +
>> # Create root filesystem
>> sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f
>> "${WORKDIR}/multistrap.conf"
>> @@ -76,4 +80,26 @@ do_build() {
>> # Configure root filesystem
>> sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>> _do_build_cleanup
>> +
>> + do_cleanup_mounts
>> +}
>> +
>> +# Invalidate stamp for do_setup_mounts before each build start.
>> +# This will guarantee that this function will be executed once
>> +# per build.
>> +python __anonymous() {
>> + stamp = d.getVar("STAMP") + ".do_setup_mounts." +
>> d.getVarFlag("do_setup_mounts", 'stamp-extra-info')
>> + os.remove(stamp) if os.path.exists(stamp) else None
>> +}
>> +
>> +do_setup_mounts[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>> +
>> +do_setup_mounts() {
>> + sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO}
>> ${BUILDCHROOT_DIR}/isar-apt +}
>> +
>> +addtask setup_mounts after do_build
>> +
>> +do_cleanup_mounts() {
>> + sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true
>> }
>> diff --git
>> a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>> b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in index
>> a0b28e3..480a4b8 100644 ---
>> a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in +++
>> b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in @@ -6,7
>> +6,7 @@ noauth=true unpack=true ignorenativearch=true
>> bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
>> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
>> +aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
>> configscript=##CONFIG_SCRIPT##
>> setupscript=##SETUP_SCRIPT##
>> hookdir=##DIR_HOOKS##
>> @@ -29,3 +29,9 @@ source=##DISTRO_APT_SOURCE_SEC##
>> suite=##DISTRO_SUITE##/updates
>> components=##DISTRO_COMPONENTS##
>> omitdebsrc=true
>> +
>> +[isar-apt]
>> +source=file:///isar-apt
>> +suite=isar
>> +components=main
>> +omitdebsrc=true
>
--
With best regards,
Alexander Smirnov
ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 0/5] Rework isar-apt v2
2018-02-05 9:59 [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
` (8 preceding siblings ...)
2018-02-06 20:24 ` [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
@ 2018-02-08 14:59 ` Henning Schild
2018-02-08 15:09 ` Jan Kiszka
2018-02-08 17:03 ` Benedikt Niedermayr
9 siblings, 2 replies; 32+ messages in thread
From: Henning Schild @ 2018-02-08 14:59 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users
Hi,
if that is the series that went into next already, it can not deal with
IMAGE_INSTALL not containing any recipes.
In that case we do not built a buildchroot and at install time we do
not have a repo for the image-rootfs.
> W: Failed to fetch
> copy://builder/build/tmp/deploy/apt/debian-stretch/dists/isar/main/binary-amd64/Packages
> Failed to stat - stat (2: No such file or directory)
The easiest solution is probably to call do_cache_config independant of
whether we will actually fill it with packages later.
Henning
Am Mon, 5 Feb 2018 12:59:26 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> Hi all,
>
> this series intended to make buildchroot able to work with isar-apt.
> I've tried to add extended comments to each patch.
>
> Impact: with this series I'm able to build 'example-hello' <->
> 'libhello' without any hacks. So the deps are installed automatically.
>
> Changes since v1:
> - The bitbake was updated to the latest upstream version
> - In build.sh the only isar-apt is updated
>
>
> Alexander Smirnov (5):
> Update isar-bitbake.conf according to the changes in bitbake related
> to multiconfig support.
> isar-apt: Introduce separate recipe
> buildchroot: Enable isar-apt
> build.sh: Update apt sources
> build.sh: Force 'yes' for apt
>
> meta-isar/conf/local.conf.sample | 5 +-
> .../recipes-core/images/files/distributions.in | 3 -
> meta/classes/dpkg-base.bbclass | 12 +++-
> meta/classes/image.bbclass | 67
> +---------------------
> meta/classes/isar-events.bbclass | 20 +++++++
> meta/conf/isar-bitbake.conf | 1 +
> meta/recipes-devtools/buildchroot/buildchroot.bb | 7 +++
> meta/recipes-devtools/buildchroot/files/build.sh | 13
> ++++- .../buildchroot/files/multistrap.conf.in | 8
> ++- .../isar-apt/files/distributions.in | 3 +
> meta/recipes-devtools/isar-apt/isar-apt.bb | 29 ++++++++++ 11
> files changed, 92 insertions(+), 76 deletions(-) delete mode 100644
> meta-isar/recipes-core/images/files/distributions.in create mode
> 100644 meta/classes/isar-events.bbclass create mode 100644
> meta/recipes-devtools/isar-apt/files/distributions.in create mode
> 100644 meta/recipes-devtools/isar-apt/isar-apt.bb
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 0/5] Rework isar-apt v2
2018-02-08 14:59 ` Henning Schild
@ 2018-02-08 15:09 ` Jan Kiszka
2018-02-09 10:30 ` Henning Schild
2018-02-08 17:03 ` Benedikt Niedermayr
1 sibling, 1 reply; 32+ messages in thread
From: Jan Kiszka @ 2018-02-08 15:09 UTC (permalink / raw)
To: [ext] Henning Schild, Alexander Smirnov; +Cc: isar-users
On 2018-02-08 15:59, [ext] Henning Schild wrote:
> Hi,
>
> if that is the series that went into next already, it can not deal with
> IMAGE_INSTALL not containing any recipes.
> In that case we do not built a buildchroot and at install time we do
> not have a repo for the image-rootfs.
>
>> W: Failed to fetch
>> copy://builder/build/tmp/deploy/apt/debian-stretch/dists/isar/main/binary-amd64/Packages
>> Failed to stat - stat (2: No such file or directory)
>
> The easiest solution is probably to call do_cache_config independant of
> whether we will actually fill it with packages later.
Good catch. It's papered over by my "Install kernel via replaceable
recipe" because that will now always need at least one recipe. Probably
worth fixing nevertheless to once support kernel-free builds.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 0/5] Rework isar-apt v2
2018-02-08 14:59 ` Henning Schild
2018-02-08 15:09 ` Jan Kiszka
@ 2018-02-08 17:03 ` Benedikt Niedermayr
2018-02-08 18:27 ` Benedikt Niedermayr
1 sibling, 1 reply; 32+ messages in thread
From: Benedikt Niedermayr @ 2018-02-08 17:03 UTC (permalink / raw)
To: Henning Schild, Alexander Smirnov; +Cc: isar-users
Am 08.02.2018 um 15:59 schrieb Henning Schild:
> Hi,
>
> if that is the series that went into next already, it can not deal with
> IMAGE_INSTALL not containing any recipes.
> In that case we do not built a buildchroot and at install time we do
> not have a repo for the image-rootfs.
>
>> W: Failed to fetch
>> copy://builder/build/tmp/deploy/apt/debian-stretch/dists/isar/main/binary-amd64/Packages
>> Failed to stat - stat (2: No such file or directory)
> The easiest solution is probably to call do_cache_config independant of
> whether we will actually fill it with packages later.
>
> Henning
>
> Am Mon, 5 Feb 2018 12:59:26 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>
>> Hi all,
>>
>> this series intended to make buildchroot able to work with isar-apt.
>> I've tried to add extended comments to each patch.
>>
>> Impact: with this series I'm able to build 'example-hello' <->
>> 'libhello' without any hacks. So the deps are installed automatically.
>>
>> Changes since v1:
>> - The bitbake was updated to the latest upstream version
>> - In build.sh the only isar-apt is updated
>>
>>
>> Alexander Smirnov (5):
>> Update isar-bitbake.conf according to the changes in bitbake related
>> to multiconfig support.
>> isar-apt: Introduce separate recipe
>> buildchroot: Enable isar-apt
>> build.sh: Update apt sources
>> build.sh: Force 'yes' for apt
>>
>> meta-isar/conf/local.conf.sample | 5 +-
>> .../recipes-core/images/files/distributions.in | 3 -
>> meta/classes/dpkg-base.bbclass | 12 +++-
>> meta/classes/image.bbclass | 67
>> +---------------------
>> meta/classes/isar-events.bbclass | 20 +++++++
>> meta/conf/isar-bitbake.conf | 1 +
>> meta/recipes-devtools/buildchroot/buildchroot.bb | 7 +++
>> meta/recipes-devtools/buildchroot/files/build.sh | 13
>> ++++- .../buildchroot/files/multistrap.conf.in | 8
>> ++- .../isar-apt/files/distributions.in | 3 +
>> meta/recipes-devtools/isar-apt/isar-apt.bb | 29 ++++++++++ 11
>> files changed, 92 insertions(+), 76 deletions(-) delete mode 100644
>> meta-isar/recipes-core/images/files/distributions.in create mode
>> 100644 meta/classes/isar-events.bbclass create mode 100644
>> meta/recipes-devtools/isar-apt/files/distributions.in create mode
>> 100644 meta/recipes-devtools/isar-apt/isar-apt.bb
>>
No, this approach is independent from Alex's series....
Yes.. good catch. Should be fixed soon.
Benni
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 0/5] Rework isar-apt v2
2018-02-08 17:03 ` Benedikt Niedermayr
@ 2018-02-08 18:27 ` Benedikt Niedermayr
0 siblings, 0 replies; 32+ messages in thread
From: Benedikt Niedermayr @ 2018-02-08 18:27 UTC (permalink / raw)
To: Henning Schild, Alexander Smirnov; +Cc: isar-users
Am 08.02.2018 um 18:03 schrieb Benedikt Niedermayr:
> Am 08.02.2018 um 15:59 schrieb Henning Schild:
>> Hi,
>>
>> if that is the series that went into next already, it can not deal with
>> IMAGE_INSTALL not containing any recipes.
>> In that case we do not built a buildchroot and at install time we do
>> not have a repo for the image-rootfs.
>>
>>> W: Failed to fetch
>>> copy://builder/build/tmp/deploy/apt/debian-stretch/dists/isar/main/binary-amd64/Packages
>>>
>>> Failed to stat - stat (2: No such file or directory)
>> The easiest solution is probably to call do_cache_config independant of
>> whether we will actually fill it with packages later.
>>
>> Henning
>>
>> Am Mon, 5 Feb 2018 12:59:26 +0300
>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>
>>> Hi all,
>>>
>>> this series intended to make buildchroot able to work with isar-apt.
>>> I've tried to add extended comments to each patch.
>>>
>>> Impact: with this series I'm able to build 'example-hello' <->
>>> 'libhello' without any hacks. So the deps are installed automatically.
>>>
>>> Changes since v1:
>>> - The bitbake was updated to the latest upstream version
>>> - In build.sh the only isar-apt is updated
>>>
>>>
>>> Alexander Smirnov (5):
>>> Update isar-bitbake.conf according to the changes in bitbake related
>>> to multiconfig support.
>>> isar-apt: Introduce separate recipe
>>> buildchroot: Enable isar-apt
>>> build.sh: Update apt sources
>>> build.sh: Force 'yes' for apt
>>>
>>> meta-isar/conf/local.conf.sample | 5 +-
>>> .../recipes-core/images/files/distributions.in | 3 -
>>> meta/classes/dpkg-base.bbclass | 12 +++-
>>> meta/classes/image.bbclass | 67
>>> +---------------------
>>> meta/classes/isar-events.bbclass | 20 +++++++
>>> meta/conf/isar-bitbake.conf | 1 +
>>> meta/recipes-devtools/buildchroot/buildchroot.bb | 7 +++
>>> meta/recipes-devtools/buildchroot/files/build.sh | 13
>>> ++++- .../buildchroot/files/multistrap.conf.in | 8
>>> ++- .../isar-apt/files/distributions.in | 3 +
>>> meta/recipes-devtools/isar-apt/isar-apt.bb | 29 ++++++++++ 11
>>> files changed, 92 insertions(+), 76 deletions(-) delete mode 100644
>>> meta-isar/recipes-core/images/files/distributions.in create mode
>>> 100644 meta/classes/isar-events.bbclass create mode 100644
>>> meta/recipes-devtools/isar-apt/files/distributions.in create mode
>>> 100644 meta/recipes-devtools/isar-apt/isar-apt.bb
>>>
> No, this approach is independent from Alex's series....
>
> Yes.. good catch. Should be fixed soon.
>
>
> Benni
>
>
>
>
Sorry don't pay attention to my last message. I've mistaken the Subject.....
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH 0/5] Rework isar-apt v2
2018-02-08 15:09 ` Jan Kiszka
@ 2018-02-09 10:30 ` Henning Schild
0 siblings, 0 replies; 32+ messages in thread
From: Henning Schild @ 2018-02-09 10:30 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Alexander Smirnov, isar-users
Am Thu, 8 Feb 2018 16:09:14 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> On 2018-02-08 15:59, [ext] Henning Schild wrote:
> > Hi,
> >
> > if that is the series that went into next already, it can not deal
> > with IMAGE_INSTALL not containing any recipes.
> > In that case we do not built a buildchroot and at install time we do
> > not have a repo for the image-rootfs.
> >
> >> W: Failed to fetch
> >> copy://builder/build/tmp/deploy/apt/debian-stretch/dists/isar/main/binary-amd64/Packages
> >> Failed to stat - stat (2: No such file or directory)
> >
> > The easiest solution is probably to call do_cache_config
> > independant of whether we will actually fill it with packages
> > later.
>
> Good catch. It's papered over by my "Install kernel via replaceable
> recipe" because that will now always need at least one recipe.
> Probably worth fixing nevertheless to once support kernel-free builds.
Yes, it should be fixed and actually got fixed in current next
"image: Add dependency from isar-apt"
Thanks Alex,
Henning
> Jan
>
^ permalink raw reply [flat|nested] 32+ messages in thread
end of thread, other threads:[~2018-02-09 10:30 UTC | newest]
Thread overview: 32+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-05 9:59 [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
2018-02-05 9:59 ` [PATCH 1/5] Update isar-bitbake.conf according to the changes in bitbake related to multiconfig support Alexander Smirnov
2018-02-05 10:15 ` Jan Kiszka
2018-02-05 10:26 ` Alexander Smirnov
2018-02-05 10:27 ` Jan Kiszka
2018-02-05 9:59 ` [PATCH 2/5] isar-apt: Introduce separate recipe Alexander Smirnov
2018-02-05 9:59 ` [PATCH 3/5] buildchroot: Enable isar-apt Alexander Smirnov
2018-02-05 10:18 ` Jan Kiszka
2018-02-05 10:33 ` Alexander Smirnov
2018-02-05 9:59 ` [PATCH 4/5] build.sh: Update apt sources Alexander Smirnov
2018-02-05 9:59 ` [PATCH 5/5] build.sh: Force 'yes' for apt Alexander Smirnov
2018-02-05 11:52 ` Jan Kiszka
2018-02-05 13:56 ` Alexander Smirnov
2018-02-05 13:48 ` [PATCH 3/5 v2] buildchroot: Enable isar-apt Alexander Smirnov
2018-02-06 11:01 ` [PATCH 3/5 v3] " Alexander Smirnov
2018-02-06 12:10 ` Jan Kiszka
2018-02-06 13:28 ` Alexander Smirnov
2018-02-05 14:41 ` [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
2018-02-05 16:47 ` Jan Kiszka
2018-02-05 16:56 ` Alexander Smirnov
2018-02-06 13:57 ` [PATCH 3/5 v4] buildchroot: Enable isar-apt Alexander Smirnov
2018-02-06 17:31 ` Jan Kiszka
2018-02-06 18:29 ` Alexander Smirnov
2018-02-06 18:31 ` Jan Kiszka
2018-02-08 14:37 ` Henning Schild
2018-02-08 14:57 ` Alexander Smirnov
2018-02-06 20:24 ` [PATCH 0/5] Rework isar-apt v2 Alexander Smirnov
2018-02-08 14:59 ` Henning Schild
2018-02-08 15:09 ` Jan Kiszka
2018-02-09 10:30 ` Henning Schild
2018-02-08 17:03 ` Benedikt Niedermayr
2018-02-08 18:27 ` Benedikt Niedermayr
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox