public inbox for isar-users@googlegroups.com
 help / color / mirror / Atom feed
* [PATCH 0/4] Rework isar-apt
@ 2018-02-01 11:29 Alexander Smirnov
  2018-02-01 11:29 ` [PATCH 1/4] isar-apt: Introduce separate recipe Alexander Smirnov
                   ` (4 more replies)
  0 siblings, 5 replies; 23+ messages in thread
From: Alexander Smirnov @ 2018-02-01 11:29 UTC (permalink / raw)
  To: isar-users; +Cc: Alexander Smirnov

Hi all,

this series intended to make buildchroot able to work with isar-apt.
I've tried to add extended comments to each patch.

Impact: with this series I'm able to build 'example-hello' <-> 'libhello'
without any hacks. So the deps are installed automatically.

NOTE: I've migrated to bitbake [lockfiles] mechanism, don't know how robust it
is, but build in the loop started in the evening didn't fail during the night.

Alexander Smirnov (3):
  isar-apt: Introduce separate recipe
  buildchroot: Enable isar-apt
  build.sh: Update apt sources

 meta-isar/conf/local.conf.sample                   |  5 +-
 .../recipes-core/images/files/distributions.in     |  3 -
 meta/classes/dpkg-base.bbclass                     | 12 +++-
 meta/classes/image.bbclass                         | 67 +---------------------
 meta/classes/isar-events.bbclass                   | 20 +++++++
 meta/recipes-devtools/buildchroot/buildchroot.bb   |  7 +++
 meta/recipes-devtools/buildchroot/files/build.sh   |  3 +
 .../buildchroot/files/multistrap.conf.in           |  8 ++-
 .../isar-apt/files/distributions.in                |  3 +
 meta/recipes-devtools/isar-apt/isar-apt.bb         | 29 ++++++++++
 10 files changed, 82 insertions(+), 75 deletions(-)
 delete mode 100644 meta-isar/recipes-core/images/files/distributions.in
 create mode 100644 meta/classes/isar-events.bbclass
 create mode 100644 meta/recipes-devtools/isar-apt/files/distributions.in
 create mode 100644 meta/recipes-devtools/isar-apt/isar-apt.bb

-- 
2.1.4


^ permalink raw reply	[flat|nested] 23+ messages in thread
* [PATCH 1/4] isar-apt: Introduce separate recipe
  2018-02-01 11:29 [PATCH 0/4] Rework isar-apt Alexander Smirnov
@ 2018-02-01 11:29 ` Alexander Smirnov
  2018-02-01 11:29 ` [PATCH 2/4] buildchroot: Enable isar-apt Alexander Smirnov
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 23+ messages in thread
From: Alexander Smirnov @ 2018-02-01 11:29 UTC (permalink / raw)
  To: isar-users; +Cc: Alexander Smirnov

Motivation: in current implementation whole isar-apt logic are done as
a part of image class. This actually not the optimal way due to the
various issues like:
 - isar-apt should be singletone in current tree despite on amount of images
   to be built.
 - isar-apt API could be possible extended, for example by adding do_export().
 - *-image is not the mandatory target to build. User may build just
   example-hello, but isar-apt should handle this.
So there should be separate recipe which is responsibe for isar-apt management.
Also isar-apt should be a static part of core Isar like buildchroot.

This patch drops isar-apt from image class, and introduces isar-apt recipe.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar/conf/local.conf.sample                   |  4 --
 .../recipes-core/images/files/distributions.in     |  3 -
 meta/classes/dpkg-base.bbclass                     | 12 +++-
 meta/classes/image.bbclass                         | 67 +---------------------
 .../isar-apt/files/distributions.in                |  3 +
 meta/recipes-devtools/isar-apt/isar-apt.bb         | 29 ++++++++++
 6 files changed, 43 insertions(+), 75 deletions(-)
 delete mode 100644 meta-isar/recipes-core/images/files/distributions.in
 create mode 100644 meta/recipes-devtools/isar-apt/files/distributions.in
 create mode 100644 meta/recipes-devtools/isar-apt/isar-apt.bb

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 33d988e..2ae43e7 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -158,7 +158,3 @@ IMAGE_INSTALL = "example-hello example-raw"
 #
 # Default parallel jobs for bitbake:
 BB_NUMBER_THREADS = "4"
-
-#
-# Number of attempts to try to get reprepro lock for access to apt cache
-REPREPRO_LOCK_ATTEMPTS = "16"
diff --git a/meta-isar/recipes-core/images/files/distributions.in b/meta-isar/recipes-core/images/files/distributions.in
deleted file mode 100644
index cd214c6..0000000
--- a/meta-isar/recipes-core/images/files/distributions.in
+++ /dev/null
@@ -1,3 +0,0 @@
-Codename: {DISTRO_NAME}
-Architectures: i386 armhf amd64 source
-Components: main
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 4941f9b..026028f 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -31,11 +31,17 @@ do_build() {
     _do_build_cleanup
 }
 
-# Install package to dedicated deploy directory
+# Install package to Isar-apt
 do_deploy_deb() {
-    install -m 644 ${WORKDIR}/*.deb ${DEPLOY_DIR_DEB}/
+    reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \
+             --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \
+             -C main \
+             includedeb ${DEBDISTRONAME} \
+             ${WORKDIR}/*.deb
 }
 
 addtask deploy_deb after do_build
 do_deploy_deb[dirs] = "${DEPLOY_DIR_DEB}"
-do_deploy_deb[stamp-extra-info] = "${MACHINE}"
+do_deploy_deb[stamp-extra-info] = "${DISTRO}-${MACHINE}"
+do_deploy_deb[lockfiles] = "${DEPLOY_DIR_APT}/isar.lock"
+do_deploy_deb[depends] = "isar-apt:do_cache_config"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index e2cb01b..cbd74b3 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -17,70 +17,6 @@ KERNEL_IMAGE ?= "${@get_image_name(d, 'vmlinuz')}"
 INITRD_IMAGE ?= "${@get_image_name(d, 'initrd.img')}"
 
 inherit ${IMAGE_TYPE}
-call_reprepro() {
-    for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do
-        #  According to `sh` manual page, shell exit statuses fall between
-        # 0-255. The EEXIST error code is (-17), so casting to usigned 8-bit
-        # integer results value (239).
-        eexist=$(python -c 'import errno; print(256-errno.EEXIST)')
-        retval="0"
-        reprepro $* || retval="$?"
-
-        # If reprepro has failed to get database lock, it returns EEXIST code.
-        # In this case we continue trying to get lock until max amount of
-        # attempts is reached.
-        if [ $retval -eq $eexist ]; then
-            bbwarn "Failed to get reprepro lock, trying again..."
-            sleep 5
-        else
-            break
-        fi
-    done
-
-    if [ $retval -ne 0 ]; then
-        bbfatal "reprepro failed"
-    fi
-}
-
-CACHE_CONF_DIR = "${DEPLOY_DIR_APT}/${DISTRO}/conf"
-do_cache_config[dirs] = "${CACHE_CONF_DIR}"
-do_cache_config[stamp-extra-info] = "${DISTRO}"
-
-# Generate reprepro config for current distro if it doesn't exist. Once it's
-# generated, this task should do nothing.
-do_cache_config() {
-    if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
-        sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
-            ${FILESDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
-    fi
-
-    path_cache="${DEPLOY_DIR_APT}/${DISTRO}"
-    path_databases="${DEPLOY_DIR_DB}/${DISTRO}"
-
-    if [ ! -d "${path_databases}" ]; then
-        call_reprepro -b ${path_cache} \
-                      --dbdir ${path_databases} \
-                      export ${DEBDISTRONAME}
-    fi
-}
-
-addtask cache_config before do_populate
-
-do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
-
-# Populate Isar apt repository by newly built packages
-do_populate() {
-    if [ -n "${IMAGE_INSTALL}" ]; then
-        call_reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \
-                      --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \
-                      -C main \
-                      includedeb ${DEBDISTRONAME} \
-                      ${DEPLOY_DIR_DEB}/*.deb
-    fi
-}
-
-addtask populate before do_build after do_unpack
-do_populate[deptask] = "do_deploy_deb"
 
 do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
 
@@ -88,7 +24,8 @@ do_rootfs() {
     die "No root filesystem function defined, please implement in your recipe"
 }
 
-addtask rootfs before do_build after do_populate
+addtask rootfs before do_build after do_unpack
+do_rootfs[deptask] = "do_deploy_deb"
 
 do_copy_boot_files() {
     KERNEL_IMAGE=${@get_image_name(d, 'vmlinuz')}
diff --git a/meta/recipes-devtools/isar-apt/files/distributions.in b/meta/recipes-devtools/isar-apt/files/distributions.in
new file mode 100644
index 0000000..cd214c6
--- /dev/null
+++ b/meta/recipes-devtools/isar-apt/files/distributions.in
@@ -0,0 +1,3 @@
+Codename: {DISTRO_NAME}
+Architectures: i386 armhf amd64 source
+Components: main
diff --git a/meta/recipes-devtools/isar-apt/isar-apt.bb b/meta/recipes-devtools/isar-apt/isar-apt.bb
new file mode 100644
index 0000000..9c31d12
--- /dev/null
+++ b/meta/recipes-devtools/isar-apt/isar-apt.bb
@@ -0,0 +1,29 @@
+# This software is a part of ISAR.
+# Copyright (C) 2015-2017 ilbers GmbH
+
+SRC_URI = "file://distributions.in"
+
+CACHE_CONF_DIR = "${DEPLOY_DIR_APT}/${DISTRO}/conf"
+do_cache_config[dirs] = "${CACHE_CONF_DIR}"
+do_cache_config[stamp-extra-info] = "${DISTRO}"
+do_cache_config[lockfiles] = "${DEPLOY_DIR_APT}/isar.lock"
+
+# Generate reprepro config for current distro if it doesn't exist. Once it's
+# generated, this task should do nothing.
+do_cache_config() {
+    if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
+        sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
+            ${WORKDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
+    fi
+
+    path_cache="${DEPLOY_DIR_APT}/${DISTRO}"
+    path_databases="${DEPLOY_DIR_DB}/${DISTRO}"
+
+    if [ ! -d "${path_databases}" ]; then
+        reprepro -b ${path_cache} \
+                 --dbdir ${path_databases} \
+                 export ${DEBDISTRONAME}
+    fi
+}
+
+addtask cache_config after do_unpack before do_build
-- 
2.1.4


^ permalink raw reply	[flat|nested] 23+ messages in thread
* [PATCH 2/4] buildchroot: Enable isar-apt
  2018-02-01 11:29 [PATCH 0/4] Rework isar-apt Alexander Smirnov
  2018-02-01 11:29 ` [PATCH 1/4] isar-apt: Introduce separate recipe Alexander Smirnov
@ 2018-02-01 11:29 ` Alexander Smirnov
  2018-02-04 12:06   ` Jan Kiszka
  2018-02-01 11:29 ` [PATCH 3/4] build.sh: Update apt sources Alexander Smirnov
                   ` (2 subsequent siblings)
  4 siblings, 1 reply; 23+ messages in thread
From: Alexander Smirnov @ 2018-02-01 11:29 UTC (permalink / raw)
  To: isar-users; +Cc: Alexander Smirnov

This patch provides access to isar-apt from buildchroot. It does the
following:
 - mount isar-apt during buildchroot building.
 - umount isar-apt using bitbake events.

Also it needs to keep Isar build tree clean from any mounts despite on whether
build succeed of failed. bitbake provides various events that could trigger
custom python hooks. In this patch BuildCompleted event is used, which happened
when bitbake finished its execution despite on the result.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar/conf/local.conf.sample                     |  3 +++
 meta/classes/isar-events.bbclass                     | 20 ++++++++++++++++++++
 meta/recipes-devtools/buildchroot/buildchroot.bb     |  7 +++++++
 .../buildchroot/files/multistrap.conf.in             |  8 +++++++-
 4 files changed, 37 insertions(+), 1 deletion(-)
 create mode 100644 meta/classes/isar-events.bbclass

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 2ae43e7..fc760ed 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -158,3 +158,6 @@ IMAGE_INSTALL = "example-hello example-raw"
 #
 # Default parallel jobs for bitbake:
 BB_NUMBER_THREADS = "4"
+
+# Add event handlers for bitbake
+INHERIT += "isar-events"
diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
new file mode 100644
index 0000000..55fc106
--- /dev/null
+++ b/meta/classes/isar-events.bbclass
@@ -0,0 +1,20 @@
+# Isar event handlers.
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2017 ilbers GmbH
+
+addhandler isar_handler
+
+python isar_handler () {
+    import subprocess
+
+    devnull = open(os.devnull, 'w')
+
+    if isinstance(e, bb.event.BuildCompleted):
+        bchroot = d.getVar('BUILDCHROOT_DIR', True)
+
+        # Clean up buildchroot
+        subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot + '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True)
+
+    devnull.close()
+}
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 51f9d5d..cb90148 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -33,6 +33,7 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
 
 do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 do_build[dirs] = "${WORKDIR}/hooks_multistrap"
+do_build[depends] = "isar-apt:do_cache_config"
 
 do_build() {
     E="${@ bb.utils.export_proxies(d)}"
@@ -58,6 +59,9 @@ do_build() {
         -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
            "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
 
+    [ ! -d ${BUILDCHROOT_DIR}/isar-apt ] && install -d -m 555 ${BUILDCHROOT_DIR}/isar-apt
+    sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
+
     [ ! -d ${BUILDCHROOT_DIR}/proc ] && install -d -m 555 ${BUILDCHROOT_DIR}/proc
     sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
     _do_build_cleanup() {
@@ -73,6 +77,9 @@ do_build() {
     # Install package builder script
     sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
 
+    # Create share point for isar-apt
+    sudo install -d ${BUILDCHROOT_DIR}/isar-apt
+
     # Configure root filesystem
     sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
     _do_build_cleanup
diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
index a0b28e3..480a4b8 100644
--- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
+++ b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
@@ -6,7 +6,7 @@ noauth=true
 unpack=true
 ignorenativearch=true
 bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
-aptsources=##DISTRO_MULTICONF_APTSOURCES##
+aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
 configscript=##CONFIG_SCRIPT##
 setupscript=##SETUP_SCRIPT##
 hookdir=##DIR_HOOKS##
@@ -29,3 +29,9 @@ source=##DISTRO_APT_SOURCE_SEC##
 suite=##DISTRO_SUITE##/updates
 components=##DISTRO_COMPONENTS##
 omitdebsrc=true
+
+[isar-apt]
+source=file:///isar-apt
+suite=isar
+components=main
+omitdebsrc=true
-- 
2.1.4


^ permalink raw reply	[flat|nested] 23+ messages in thread
* [PATCH 3/4] build.sh: Update apt sources
  2018-02-01 11:29 [PATCH 0/4] Rework isar-apt Alexander Smirnov
  2018-02-01 11:29 ` [PATCH 1/4] isar-apt: Introduce separate recipe Alexander Smirnov
  2018-02-01 11:29 ` [PATCH 2/4] buildchroot: Enable isar-apt Alexander Smirnov
@ 2018-02-01 11:29 ` Alexander Smirnov
  2018-02-01 11:40   ` Jan Kiszka
  2018-02-01 11:29 ` [PATCH 4/4] build.sh: Force 'yes' for apt Alexander Smirnov
  2018-02-01 16:14 ` [PATCH 0/4] Rework isar-apt Jan Kiszka
  4 siblings, 1 reply; 23+ messages in thread
From: Alexander Smirnov @ 2018-02-01 11:29 UTC (permalink / raw)
  To: isar-users; +Cc: Alexander Smirnov

Trigger apt sources update before building the package to be sure that
we are in sync with isar-apt content.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/recipes-devtools/buildchroot/files/build.sh | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
index 77e0fdd..5c59193 100644
--- a/meta/recipes-devtools/buildchroot/files/build.sh
+++ b/meta/recipes-devtools/buildchroot/files/build.sh
@@ -3,6 +3,9 @@
 # This software is a part of ISAR.
 # Copyright (C) 2015-2017 ilbers GmbH
 
+# Make sure that we have latest isar-apt content
+apt-get update
+
 # Go to build directory
 cd $1
 
-- 
2.1.4


^ permalink raw reply	[flat|nested] 23+ messages in thread
* [PATCH 4/4] build.sh: Force 'yes' for apt
  2018-02-01 11:29 [PATCH 0/4] Rework isar-apt Alexander Smirnov
                   ` (2 preceding siblings ...)
  2018-02-01 11:29 ` [PATCH 3/4] build.sh: Update apt sources Alexander Smirnov
@ 2018-02-01 11:29 ` Alexander Smirnov
  2018-02-01 11:41   ` Jan Kiszka
  2018-02-01 16:14 ` [PATCH 0/4] Rework isar-apt Jan Kiszka
  4 siblings, 1 reply; 23+ messages in thread
From: Alexander Smirnov @ 2018-02-01 11:29 UTC (permalink / raw)
  To: isar-users; +Cc: Alexander Smirnov

Sometimes the following error occurs with custom repo:

After this operation, 84.0 kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  libhello libhello-dev
E: There are problems and -y was used without --force-yes

This patch adds '--force-yes' option to be sure that no dialog will occur.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/recipes-devtools/buildchroot/files/build.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
index 5c59193..cdf5888 100644
--- a/meta/recipes-devtools/buildchroot/files/build.sh
+++ b/meta/recipes-devtools/buildchroot/files/build.sh
@@ -13,7 +13,7 @@ cd $1
 # Notes:
 #   1) everything before the -y switch is unchanged from the defaults
 #   2) we add -y to go non-interactive
-install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y"
+install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y --force-yes"
 
 # Install all build deps
 mk-build-deps -t "${install_cmd}" -i -r debian/control
-- 
2.1.4


^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 3/4] build.sh: Update apt sources
  2018-02-01 11:29 ` [PATCH 3/4] build.sh: Update apt sources Alexander Smirnov
@ 2018-02-01 11:40   ` Jan Kiszka
  2018-02-01 11:48     ` Alexander Smirnov
  0 siblings, 1 reply; 23+ messages in thread
From: Jan Kiszka @ 2018-02-01 11:40 UTC (permalink / raw)
  To: Alexander Smirnov, isar-users

On 2018-02-01 12:29, Alexander Smirnov wrote:
> Trigger apt sources update before building the package to be sure that
> we are in sync with isar-apt content.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/recipes-devtools/buildchroot/files/build.sh | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
> index 77e0fdd..5c59193 100644
> --- a/meta/recipes-devtools/buildchroot/files/build.sh
> +++ b/meta/recipes-devtools/buildchroot/files/build.sh
> @@ -3,6 +3,9 @@
>  # This software is a part of ISAR.
>  # Copyright (C) 2015-2017 ilbers GmbH
>  
> +# Make sure that we have latest isar-apt content
> +apt-get update
> +

Can we confine this to the isar-apt repo? Or is there no risk that an
upstream repo update will result in a buildchroot change that is
inconsistent with an earlier build stage?

Jan

>  # Go to build directory
>  cd $1
>  
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 4/4] build.sh: Force 'yes' for apt
  2018-02-01 11:29 ` [PATCH 4/4] build.sh: Force 'yes' for apt Alexander Smirnov
@ 2018-02-01 11:41   ` Jan Kiszka
  2018-02-01 12:27     ` Alexander Smirnov
  0 siblings, 1 reply; 23+ messages in thread
From: Jan Kiszka @ 2018-02-01 11:41 UTC (permalink / raw)
  To: Alexander Smirnov, isar-users

On 2018-02-01 12:29, Alexander Smirnov wrote:
> Sometimes the following error occurs with custom repo:
> 
> After this operation, 84.0 kB of additional disk space will be used.
> WARNING: The following packages cannot be authenticated!
>   libhello libhello-dev
> E: There are problems and -y was used without --force-yes

Means they or the repo lack signatures?

Jan

> 
> This patch adds '--force-yes' option to be sure that no dialog will occur.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/recipes-devtools/buildchroot/files/build.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
> index 5c59193..cdf5888 100644
> --- a/meta/recipes-devtools/buildchroot/files/build.sh
> +++ b/meta/recipes-devtools/buildchroot/files/build.sh
> @@ -13,7 +13,7 @@ cd $1
>  # Notes:
>  #   1) everything before the -y switch is unchanged from the defaults
>  #   2) we add -y to go non-interactive
> -install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y"
> +install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y --force-yes"
>  
>  # Install all build deps
>  mk-build-deps -t "${install_cmd}" -i -r debian/control
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 3/4] build.sh: Update apt sources
  2018-02-01 11:40   ` Jan Kiszka
@ 2018-02-01 11:48     ` Alexander Smirnov
  2018-02-01 13:13       ` Jan Kiszka
  0 siblings, 1 reply; 23+ messages in thread
From: Alexander Smirnov @ 2018-02-01 11:48 UTC (permalink / raw)
  To: Jan Kiszka, isar-users

On 02/01/2018 02:40 PM, Jan Kiszka wrote:
> On 2018-02-01 12:29, Alexander Smirnov wrote:
>> Trigger apt sources update before building the package to be sure that
>> we are in sync with isar-apt content.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta/recipes-devtools/buildchroot/files/build.sh | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
>> index 77e0fdd..5c59193 100644
>> --- a/meta/recipes-devtools/buildchroot/files/build.sh
>> +++ b/meta/recipes-devtools/buildchroot/files/build.sh
>> @@ -3,6 +3,9 @@
>>   # This software is a part of ISAR.
>>   # Copyright (C) 2015-2017 ilbers GmbH
>>   
>> +# Make sure that we have latest isar-apt content
>> +apt-get update
>> +
> 
> Can we confine this to the isar-apt repo? Or is there no risk that an
> upstream repo update will result in a buildchroot change that is
> inconsistent with an earlier build stage?

That's a good point, I've already checked it, it seems to be possible like:

sudo apt-get update -o Dir::Etc::sourcelist="sources.list.d/${source}" \
-o Dir::Etc::sourceparts="-" -o APT::Get::List-Cleanup="0"

[https://askubuntu.com/questions/65245/apt-get-update-only-for-a-specific-repository]

But I decided that it makes no sense, because target image will be 
generated later and use upstream apt. So if something has been changed 
during buildchroot operating, this will affect target image anyway.

Again I think this should be handled outside buildchroot by caching 
upstream apts.

Alex

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 4/4] build.sh: Force 'yes' for apt
  2018-02-01 11:41   ` Jan Kiszka
@ 2018-02-01 12:27     ` Alexander Smirnov
  0 siblings, 0 replies; 23+ messages in thread
From: Alexander Smirnov @ 2018-02-01 12:27 UTC (permalink / raw)
  To: Jan Kiszka, isar-users

On 02/01/2018 02:41 PM, Jan Kiszka wrote:
> On 2018-02-01 12:29, Alexander Smirnov wrote:
>> Sometimes the following error occurs with custom repo:
>>
>> After this operation, 84.0 kB of additional disk space will be used.
>> WARNING: The following packages cannot be authenticated!
>>    libhello libhello-dev
>> E: There are problems and -y was used without --force-yes
> 
> Means they or the repo lack signatures?

Probably it's an issue of isar-apt, AFAIK isar-apt has no signature. But 
I suppose any other possible WARNING will cause this problem.

BTW: isar-apt is our local repo that created from the scratch and 
contains only Isar packages that have been built from recipes, so I 
don't think we need to sign it, at least for now especially due to 
possible migration to aptly. But other opinions are welcome! :-)

Alex

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 3/4] build.sh: Update apt sources
  2018-02-01 11:48     ` Alexander Smirnov
@ 2018-02-01 13:13       ` Jan Kiszka
  2018-02-01 13:43         ` Alexander Smirnov
  0 siblings, 1 reply; 23+ messages in thread
From: Jan Kiszka @ 2018-02-01 13:13 UTC (permalink / raw)
  To: Alexander Smirnov, isar-users

On 2018-02-01 12:48, Alexander Smirnov wrote:
> On 02/01/2018 02:40 PM, Jan Kiszka wrote:
>> On 2018-02-01 12:29, Alexander Smirnov wrote:
>>> Trigger apt sources update before building the package to be sure that
>>> we are in sync with isar-apt content.
>>>
>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>> ---
>>>   meta/recipes-devtools/buildchroot/files/build.sh | 3 +++
>>>   1 file changed, 3 insertions(+)
>>>
>>> diff --git a/meta/recipes-devtools/buildchroot/files/build.sh
>>> b/meta/recipes-devtools/buildchroot/files/build.sh
>>> index 77e0fdd..5c59193 100644
>>> --- a/meta/recipes-devtools/buildchroot/files/build.sh
>>> +++ b/meta/recipes-devtools/buildchroot/files/build.sh
>>> @@ -3,6 +3,9 @@
>>>   # This software is a part of ISAR.
>>>   # Copyright (C) 2015-2017 ilbers GmbH
>>>   +# Make sure that we have latest isar-apt content
>>> +apt-get update
>>> +
>>
>> Can we confine this to the isar-apt repo? Or is there no risk that an
>> upstream repo update will result in a buildchroot change that is
>> inconsistent with an earlier build stage?
> 
> That's a good point, I've already checked it, it seems to be possible like:
> 
> sudo apt-get update -o Dir::Etc::sourcelist="sources.list.d/${source}" \
> -o Dir::Etc::sourceparts="-" -o APT::Get::List-Cleanup="0"
> 
> [https://askubuntu.com/questions/65245/apt-get-update-only-for-a-specific-repository]
> 
> 
> But I decided that it makes no sense, because target image will be
> generated later and use upstream apt. So if something has been changed
> during buildchroot operating, this will affect target image anyway.

buildchroot shall not deviate from target rootfs. We will eventually
(rather sooner than later) have to provide it for SDK use, and than you
definitely don't want surprises, but actually already earlier.

> 
> Again I think this should be handled outside buildchroot by caching
> upstream apts.

Well, do you plan to /always/ cache them? Last time I asked you said we
need both cases, cached and "live" ;). But even when live, we must keep
things atomic over a single build at least.

So, if you can avoid pulling from upstream here, please add that. We can
still drop it again when the external infrastructure can always
guarantee atomicity in the repo state.

Thanks,
Jan

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 3/4] build.sh: Update apt sources
  2018-02-01 13:13       ` Jan Kiszka
@ 2018-02-01 13:43         ` Alexander Smirnov
  2018-02-01 14:28           ` Jan Kiszka
  0 siblings, 1 reply; 23+ messages in thread
From: Alexander Smirnov @ 2018-02-01 13:43 UTC (permalink / raw)
  To: Jan Kiszka, isar-users

On 02/01/2018 04:13 PM, Jan Kiszka wrote:
> On 2018-02-01 12:48, Alexander Smirnov wrote:
>> On 02/01/2018 02:40 PM, Jan Kiszka wrote:
>>> On 2018-02-01 12:29, Alexander Smirnov wrote:
>>>> Trigger apt sources update before building the package to be sure that
>>>> we are in sync with isar-apt content.
>>>>
>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>> ---
>>>>    meta/recipes-devtools/buildchroot/files/build.sh | 3 +++
>>>>    1 file changed, 3 insertions(+)
>>>>
>>>> diff --git a/meta/recipes-devtools/buildchroot/files/build.sh
>>>> b/meta/recipes-devtools/buildchroot/files/build.sh
>>>> index 77e0fdd..5c59193 100644
>>>> --- a/meta/recipes-devtools/buildchroot/files/build.sh
>>>> +++ b/meta/recipes-devtools/buildchroot/files/build.sh
>>>> @@ -3,6 +3,9 @@
>>>>    # This software is a part of ISAR.
>>>>    # Copyright (C) 2015-2017 ilbers GmbH
>>>>    +# Make sure that we have latest isar-apt content
>>>> +apt-get update
>>>> +
>>>
>>> Can we confine this to the isar-apt repo? Or is there no risk that an
>>> upstream repo update will result in a buildchroot change that is
>>> inconsistent with an earlier build stage?
>>
>> That's a good point, I've already checked it, it seems to be possible like:
>>
>> sudo apt-get update -o Dir::Etc::sourcelist="sources.list.d/${source}" \
>> -o Dir::Etc::sourceparts="-" -o APT::Get::List-Cleanup="0"
>>
>> [https://askubuntu.com/questions/65245/apt-get-update-only-for-a-specific-repository]
>>
>>
>> But I decided that it makes no sense, because target image will be
>> generated later and use upstream apt. So if something has been changed
>> during buildchroot operating, this will affect target image anyway.
> 
> buildchroot shall not deviate from target rootfs. We will eventually
> (rather sooner than later) have to provide it for SDK use, and than you
> definitely don't want surprises, but actually already earlier.
> 

I mean that the issue is not in buildchroot, I'll fix this line to 
update isar-apt only, but this doesn't solve the issue of atomic 
upstream fetch. You can't generate target image using only buildchroot. 
So for example:

  - You've generated buildchroot SDK in Feb.
  - You want to use this SDK and build target image in March.

No warranty that this will work until you prefetch packages for *both* 
target image and buildchroot.

>>
>> Again I think this should be handled outside buildchroot by caching
>> upstream apts.
> 
> Well, do you plan to /always/ cache them? Last time I asked you said we
> need both cases, cached and "live" ;). But even when live, we must keep
> things atomic over a single build at least.

No, I meant a bit different thing: if build_rep assumes to cache 
*everything* exists in Isar tree, I'd like to have an option to disable 
it, I don't want to waste the time by downloading tons of unneeded packages.

In my opinion reproducibility should be always enabled and should cache 
everything needed for *the current* bitbake target, i.e.:

1. $ bitbake example-hello
Isar prefetches everything needed to build example-hello (buildchroot + 
build deps) and use local cache only.

2. $ bitabke isar-image-base
Isar prefetches everything needed to build target image (buildchroot + 
build deps + target image packages) and use local cache only.


Without this Isar reproducibility and build correctness will be based on 
user's hopes and prays :-)

> 
> So, if you can avoid pulling from upstream here, please add that. We can
> still drop it again when the external infrastructure can always
> guarantee atomicity in the repo state.
> 

Ok.

Alex

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 3/4] build.sh: Update apt sources
  2018-02-01 13:43         ` Alexander Smirnov
@ 2018-02-01 14:28           ` Jan Kiszka
  0 siblings, 0 replies; 23+ messages in thread
From: Jan Kiszka @ 2018-02-01 14:28 UTC (permalink / raw)
  To: Alexander Smirnov, isar-users

On 2018-02-01 14:43, Alexander Smirnov wrote:
> On 02/01/2018 04:13 PM, Jan Kiszka wrote:
>> On 2018-02-01 12:48, Alexander Smirnov wrote:
>>> On 02/01/2018 02:40 PM, Jan Kiszka wrote:
>>>> On 2018-02-01 12:29, Alexander Smirnov wrote:
>>>>> Trigger apt sources update before building the package to be sure that
>>>>> we are in sync with isar-apt content.
>>>>>
>>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>>> ---
>>>>>    meta/recipes-devtools/buildchroot/files/build.sh | 3 +++
>>>>>    1 file changed, 3 insertions(+)
>>>>>
>>>>> diff --git a/meta/recipes-devtools/buildchroot/files/build.sh
>>>>> b/meta/recipes-devtools/buildchroot/files/build.sh
>>>>> index 77e0fdd..5c59193 100644
>>>>> --- a/meta/recipes-devtools/buildchroot/files/build.sh
>>>>> +++ b/meta/recipes-devtools/buildchroot/files/build.sh
>>>>> @@ -3,6 +3,9 @@
>>>>>    # This software is a part of ISAR.
>>>>>    # Copyright (C) 2015-2017 ilbers GmbH
>>>>>    +# Make sure that we have latest isar-apt content
>>>>> +apt-get update
>>>>> +
>>>>
>>>> Can we confine this to the isar-apt repo? Or is there no risk that an
>>>> upstream repo update will result in a buildchroot change that is
>>>> inconsistent with an earlier build stage?
>>>
>>> That's a good point, I've already checked it, it seems to be possible
>>> like:
>>>
>>> sudo apt-get update -o Dir::Etc::sourcelist="sources.list.d/${source}" \
>>> -o Dir::Etc::sourceparts="-" -o APT::Get::List-Cleanup="0"
>>>
>>> [https://askubuntu.com/questions/65245/apt-get-update-only-for-a-specific-repository]
>>>
>>>
>>>
>>> But I decided that it makes no sense, because target image will be
>>> generated later and use upstream apt. So if something has been changed
>>> during buildchroot operating, this will affect target image anyway.
>>
>> buildchroot shall not deviate from target rootfs. We will eventually
>> (rather sooner than later) have to provide it for SDK use, and than you
>> definitely don't want surprises, but actually already earlier.
>>
> 
> I mean that the issue is not in buildchroot, I'll fix this line to
> update isar-apt only, but this doesn't solve the issue of atomic
> upstream fetch. You can't generate target image using only buildchroot.
> So for example:
> 
>  - You've generated buildchroot SDK in Feb.
>  - You want to use this SDK and build target image in March.
> 
> No warranty that this will work until you prefetch packages for *both*
> target image and buildchroot.

That... is true. :)

> 
>>>
>>> Again I think this should be handled outside buildchroot by caching
>>> upstream apts.
>>
>> Well, do you plan to /always/ cache them? Last time I asked you said we
>> need both cases, cached and "live" ;). But even when live, we must keep
>> things atomic over a single build at least.
> 
> No, I meant a bit different thing: if build_rep assumes to cache
> *everything* exists in Isar tree, I'd like to have an option to disable
> it, I don't want to waste the time by downloading tons of unneeded
> packages.
> 
> In my opinion reproducibility should be always enabled and should cache
> everything needed for *the current* bitbake target, i.e.:
> 
> 1. $ bitbake example-hello
> Isar prefetches everything needed to build example-hello (buildchroot +
> build deps) and use local cache only.
> 
> 2. $ bitabke isar-image-base
> Isar prefetches everything needed to build target image (buildchroot +
> build deps + target image packages) and use local cache only.
> 
> 
> Without this Isar reproducibility and build correctness will be based on
> user's hopes and prays :-)

We don't disagree.

> 
>>
>> So, if you can avoid pulling from upstream here, please add that. We can
>> still drop it again when the external infrastructure can always
>> guarantee atomicity in the repo state.
>>
> 
> Ok.

Let's consider it a "mitigation" measure for "improving" consistency,
not able to guarantee it. As you correctly said, as long as we do not
cache and freeze our fetches, we are lost with the two rootfs'es we
create asynchronously.

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 0/4] Rework isar-apt
  2018-02-01 11:29 [PATCH 0/4] Rework isar-apt Alexander Smirnov
                   ` (3 preceding siblings ...)
  2018-02-01 11:29 ` [PATCH 4/4] build.sh: Force 'yes' for apt Alexander Smirnov
@ 2018-02-01 16:14 ` Jan Kiszka
  2018-02-01 16:22   ` Alexander Smirnov
  4 siblings, 1 reply; 23+ messages in thread
From: Jan Kiszka @ 2018-02-01 16:14 UTC (permalink / raw)
  To: Alexander Smirnov, isar-users

On 2018-02-01 12:29, Alexander Smirnov wrote:
> Hi all,
> 
> this series intended to make buildchroot able to work with isar-apt.
> I've tried to add extended comments to each patch.
> 
> Impact: with this series I'm able to build 'example-hello' <-> 'libhello'
> without any hacks. So the deps are installed automatically.
> 
> NOTE: I've migrated to bitbake [lockfiles] mechanism, don't know how robust it
> is, but build in the loop started in the evening didn't fail during the night.
> 

I can happily report: It works as promised also for my use case
(jailhouse.bb -> linux-jailhouse.bb, both Isar-built).

Great!
Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 0/4] Rework isar-apt
  2018-02-01 16:14 ` [PATCH 0/4] Rework isar-apt Jan Kiszka
@ 2018-02-01 16:22   ` Alexander Smirnov
  2018-02-01 16:25     ` Jan Kiszka
  0 siblings, 1 reply; 23+ messages in thread
From: Alexander Smirnov @ 2018-02-01 16:22 UTC (permalink / raw)
  To: Jan Kiszka, isar-users

On 02/01/2018 07:14 PM, Jan Kiszka wrote:
> On 2018-02-01 12:29, Alexander Smirnov wrote:
>> Hi all,
>>
>> this series intended to make buildchroot able to work with isar-apt.
>> I've tried to add extended comments to each patch.
>>
>> Impact: with this series I'm able to build 'example-hello' <-> 'libhello'
>> without any hacks. So the deps are installed automatically.
>>
>> NOTE: I've migrated to bitbake [lockfiles] mechanism, don't know how robust it
>> is, but build in the loop started in the evening didn't fail during the night.
>>
> 
> I can happily report: It works as promised also for my use case
> (jailhouse.bb -> linux-jailhouse.bb, both Isar-built).
> 

Thanks! But I've found an issue with events. :-( Our bitbake didn't 
handle them for multiconfig, the following patch seems to add this:

https://patchwork.openembedded.org/patch/141626/

Events is very good mechanism to clean up Isar build from pending mounts 
without headache with build fails, so I think it would be valuable to 
try latest bitbake. I'll report the results as soon as build finishes.

Alex

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 0/4] Rework isar-apt
  2018-02-01 16:22   ` Alexander Smirnov
@ 2018-02-01 16:25     ` Jan Kiszka
  2018-02-01 16:54       ` Alexander Smirnov
  2018-02-01 18:37       ` Claudius Heine
  0 siblings, 2 replies; 23+ messages in thread
From: Jan Kiszka @ 2018-02-01 16:25 UTC (permalink / raw)
  To: Alexander Smirnov, isar-users, Claudius Heine

On 2018-02-01 17:22, Alexander Smirnov wrote:
> On 02/01/2018 07:14 PM, Jan Kiszka wrote:
>> On 2018-02-01 12:29, Alexander Smirnov wrote:
>>> Hi all,
>>>
>>> this series intended to make buildchroot able to work with isar-apt.
>>> I've tried to add extended comments to each patch.
>>>
>>> Impact: with this series I'm able to build 'example-hello' <->
>>> 'libhello'
>>> without any hacks. So the deps are installed automatically.
>>>
>>> NOTE: I've migrated to bitbake [lockfiles] mechanism, don't know how
>>> robust it
>>> is, but build in the loop started in the evening didn't fail during
>>> the night.
>>>
>>
>> I can happily report: It works as promised also for my use case
>> (jailhouse.bb -> linux-jailhouse.bb, both Isar-built).
>>
> 
> Thanks! But I've found an issue with events. :-( Our bitbake didn't
> handle them for multiconfig, the following patch seems to add this:
> 
> https://patchwork.openembedded.org/patch/141626/
> 
> Events is very good mechanism to clean up Isar build from pending mounts
> without headache with build fails, so I think it would be valuable to
> try latest bitbake. I'll report the results as soon as build finishes.

Perfect: Claudius just told me we need to update bitbake anyway to have
proper multiconfig support (i.e. no more hacky copying of files during
isar-init-build-env). Claudius, which version at least?

BTW, why are we copying bitbake in? Wouldn't a submodule make more sense?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 0/4] Rework isar-apt
  2018-02-01 16:25     ` Jan Kiszka
@ 2018-02-01 16:54       ` Alexander Smirnov
  2018-02-01 17:03         ` Jan Kiszka
  2018-02-01 18:37       ` Claudius Heine
  1 sibling, 1 reply; 23+ messages in thread
From: Alexander Smirnov @ 2018-02-01 16:54 UTC (permalink / raw)
  To: Jan Kiszka, isar-users, Claudius Heine



On 02/01/2018 07:25 PM, Jan Kiszka wrote:
> On 2018-02-01 17:22, Alexander Smirnov wrote:
>> On 02/01/2018 07:14 PM, Jan Kiszka wrote:
>>> On 2018-02-01 12:29, Alexander Smirnov wrote:
>>>> Hi all,
>>>>
>>>> this series intended to make buildchroot able to work with isar-apt.
>>>> I've tried to add extended comments to each patch.
>>>>
>>>> Impact: with this series I'm able to build 'example-hello' <->
>>>> 'libhello'
>>>> without any hacks. So the deps are installed automatically.
>>>>
>>>> NOTE: I've migrated to bitbake [lockfiles] mechanism, don't know how
>>>> robust it
>>>> is, but build in the loop started in the evening didn't fail during
>>>> the night.
>>>>
>>>
>>> I can happily report: It works as promised also for my use case
>>> (jailhouse.bb -> linux-jailhouse.bb, both Isar-built).
>>>
>>
>> Thanks! But I've found an issue with events. :-( Our bitbake didn't
>> handle them for multiconfig, the following patch seems to add this:
>>
>> https://patchwork.openembedded.org/patch/141626/
>>
>> Events is very good mechanism to clean up Isar build from pending mounts
>> without headache with build fails, so I think it would be valuable to
>> try latest bitbake. I'll report the results as soon as build finishes.
> 

 From the first like I like it:

  - build output looks more user-friendly:
0: mc:qemui386-jessie:example-hello-0.2+7bf716d2-r0 do_build - 222s (pid 
15637)

  - Yes! Events are now handled by all instances, so no more pending 
mounts. :-)

> Perfect: Claudius just told me we need to update bitbake anyway to have
> proper multiconfig support (i.e. no more hacky copying of files during
> isar-init-build-env). Claudius, which version at least?
> 

Hmm, didn't get what is "hacky copying"...

> BTW, why are we copying bitbake in? Wouldn't a submodule make more sense?
> 

In my opinion not, the main idea is the same as for Yocto - have no 
external dependencies. Isar should be self-contained.

Alex

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 0/4] Rework isar-apt
  2018-02-01 16:54       ` Alexander Smirnov
@ 2018-02-01 17:03         ` Jan Kiszka
  2018-02-01 20:32           ` Alexander Smirnov
  0 siblings, 1 reply; 23+ messages in thread
From: Jan Kiszka @ 2018-02-01 17:03 UTC (permalink / raw)
  To: Alexander Smirnov, isar-users, Claudius Heine

On 2018-02-01 17:54, Alexander Smirnov wrote:
> 
> 
> On 02/01/2018 07:25 PM, Jan Kiszka wrote:
>> On 2018-02-01 17:22, Alexander Smirnov wrote:
>>> On 02/01/2018 07:14 PM, Jan Kiszka wrote:
>>>> On 2018-02-01 12:29, Alexander Smirnov wrote:
>>>>> Hi all,
>>>>>
>>>>> this series intended to make buildchroot able to work with isar-apt.
>>>>> I've tried to add extended comments to each patch.
>>>>>
>>>>> Impact: with this series I'm able to build 'example-hello' <->
>>>>> 'libhello'
>>>>> without any hacks. So the deps are installed automatically.
>>>>>
>>>>> NOTE: I've migrated to bitbake [lockfiles] mechanism, don't know how
>>>>> robust it
>>>>> is, but build in the loop started in the evening didn't fail during
>>>>> the night.
>>>>>
>>>>
>>>> I can happily report: It works as promised also for my use case
>>>> (jailhouse.bb -> linux-jailhouse.bb, both Isar-built).
>>>>
>>>
>>> Thanks! But I've found an issue with events. :-( Our bitbake didn't
>>> handle them for multiconfig, the following patch seems to add this:
>>>
>>> https://patchwork.openembedded.org/patch/141626/
>>>
>>> Events is very good mechanism to clean up Isar build from pending mounts
>>> without headache with build fails, so I think it would be valuable to
>>> try latest bitbake. I'll report the results as soon as build finishes.
>>
> 
> From the first like I like it:
> 
>  - build output looks more user-friendly:
> 0: mc:qemui386-jessie:example-hello-0.2+7bf716d2-r0 do_build - 222s (pid
> 15637)
> 
>  - Yes! Events are now handled by all instances, so no more pending
> mounts. :-)

Even on ^C^C (forced termination)? That would be valuable!

> 
>> Perfect: Claudius just told me we need to update bitbake anyway to have
>> proper multiconfig support (i.e. no more hacky copying of files during
>> isar-init-build-env). Claudius, which version at least?
>>
> 
> Hmm, didn't get what is "hacky copying"...

That we only pick up an use multiconfig/ files from meta-isar, not any
other layer. See
https://groups.google.com/forum/#!msg/isar-users/IjQTuBFLPLo/IqRREtAHBgAJ

> 
>> BTW, why are we copying bitbake in? Wouldn't a submodule make more sense?
>>
> 
> In my opinion not, the main idea is the same as for Yocto - have no
> external dependencies. Isar should be self-contained.

That wouldn't change. You would still have both artifacts in the tree,
after checkout. And you could tar them both together for release
packaging. Or what are the concrete downsides?

I'm not using submodules heavily, but here would be a good use case IMHO.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 0/4] Rework isar-apt
  2018-02-01 16:25     ` Jan Kiszka
  2018-02-01 16:54       ` Alexander Smirnov
@ 2018-02-01 18:37       ` Claudius Heine
  2018-02-01 19:37         ` Alexander Smirnov
  1 sibling, 1 reply; 23+ messages in thread
From: Claudius Heine @ 2018-02-01 18:37 UTC (permalink / raw)
  To: Jan Kiszka, Alexander Smirnov, isar-users

[-- Attachment #1: Type: text/plain, Size: 2143 bytes --]

Hi,

On Thu, 2018-02-01 at 17:25 +0100, Jan Kiszka wrote:
> On 2018-02-01 17:22, Alexander Smirnov wrote:
> > On 02/01/2018 07:14 PM, Jan Kiszka wrote:
> > > On 2018-02-01 12:29, Alexander Smirnov wrote:
> > > > Hi all,
> > > > 
> > > > this series intended to make buildchroot able to work with
> > > > isar-apt.
> > > > I've tried to add extended comments to each patch.
> > > > 
> > > > Impact: with this series I'm able to build 'example-hello' <->
> > > > 'libhello'
> > > > without any hacks. So the deps are installed automatically.
> > > > 
> > > > NOTE: I've migrated to bitbake [lockfiles] mechanism, don't
> > > > know how
> > > > robust it
> > > > is, but build in the loop started in the evening didn't fail
> > > > during
> > > > the night.
> > > > 
> > > 
> > > I can happily report: It works as promised also for my use case
> > > (jailhouse.bb -> linux-jailhouse.bb, both Isar-built).
> > > 
> > 
> > Thanks! But I've found an issue with events. :-( Our bitbake didn't
> > handle them for multiconfig, the following patch seems to add this:
> > 
> > https://patchwork.openembedded.org/patch/141626/
> > 
> > Events is very good mechanism to clean up Isar build from pending
> > mounts
> > without headache with build fails, so I think it would be valuable
> > to
> > try latest bitbake. I'll report the results as soon as build
> > finishes.
> 
> Perfect: Claudius just told me we need to update bitbake anyway to
> have
> proper multiconfig support (i.e. no more hacky copying of files
> during
> isar-init-build-env). Claudius, which version at least?

I haven't tested it but I think its this one:

http://git.openembedded.org/bitbake/commit/?id=518b9015c2be8d3894277a8e
54890d6f04d656c0

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 0/4] Rework isar-apt
  2018-02-01 18:37       ` Claudius Heine
@ 2018-02-01 19:37         ` Alexander Smirnov
  2018-02-01 20:09           ` Claudius Heine
  0 siblings, 1 reply; 23+ messages in thread
From: Alexander Smirnov @ 2018-02-01 19:37 UTC (permalink / raw)
  To: Claudius Heine, Jan Kiszka, isar-users



On 02/01/2018 09:37 PM, Claudius Heine wrote:
> Hi,
> 
> On Thu, 2018-02-01 at 17:25 +0100, Jan Kiszka wrote:
>> On 2018-02-01 17:22, Alexander Smirnov wrote:
>>> On 02/01/2018 07:14 PM, Jan Kiszka wrote:
>>>> On 2018-02-01 12:29, Alexander Smirnov wrote:
>>>>> Hi all,
>>>>>
>>>>> this series intended to make buildchroot able to work with
>>>>> isar-apt.
>>>>> I've tried to add extended comments to each patch.
>>>>>
>>>>> Impact: with this series I'm able to build 'example-hello' <->
>>>>> 'libhello'
>>>>> without any hacks. So the deps are installed automatically.
>>>>>
>>>>> NOTE: I've migrated to bitbake [lockfiles] mechanism, don't
>>>>> know how
>>>>> robust it
>>>>> is, but build in the loop started in the evening didn't fail
>>>>> during
>>>>> the night.
>>>>>
>>>>
>>>> I can happily report: It works as promised also for my use case
>>>> (jailhouse.bb -> linux-jailhouse.bb, both Isar-built).
>>>>
>>>
>>> Thanks! But I've found an issue with events. :-( Our bitbake didn't
>>> handle them for multiconfig, the following patch seems to add this:
>>>
>>> https://patchwork.openembedded.org/patch/141626/
>>>
>>> Events is very good mechanism to clean up Isar build from pending
>>> mounts
>>> without headache with build fails, so I think it would be valuable
>>> to
>>> try latest bitbake. I'll report the results as soon as build
>>> finishes.
>>
>> Perfect: Claudius just told me we need to update bitbake anyway to
>> have
>> proper multiconfig support (i.e. no more hacky copying of files
>> during
>> isar-init-build-env). Claudius, which version at least?
> 
> I haven't tested it but I think its this one:
> 
> http://git.openembedded.org/bitbake/commit/?id=518b9015c2be8d3894277a8e
> 54890d6f04d656c0
> 

The following patch is needed for latest bitbake (derived from Yocto):

diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
index 10cd45d..b853c88 100644
--- a/meta/conf/isar-bitbake.conf
+++ b/meta/conf/isar-bitbake.conf
@@ -28,5 +28,6 @@ BUILDCHROOT_DIR = 
"${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
  BB_SIGNATURE_HANDLER ?= "noop"

  include conf/local.conf
+include conf/multiconfig/${BB_CURRENT_MC}.conf
  include conf/machine/${MACHINE}.conf
  include conf/distro/${DISTRO}.conf

Alex

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 0/4] Rework isar-apt
  2018-02-01 19:37         ` Alexander Smirnov
@ 2018-02-01 20:09           ` Claudius Heine
  2018-02-01 20:13             ` Claudius Heine
  0 siblings, 1 reply; 23+ messages in thread
From: Claudius Heine @ 2018-02-01 20:09 UTC (permalink / raw)
  To: Alexander Smirnov, Jan Kiszka, isar-users

[-- Attachment #1: Type: text/plain, Size: 3421 bytes --]

On Thu, 2018-02-01 at 22:37 +0300, Alexander Smirnov wrote:
> 
> On 02/01/2018 09:37 PM, Claudius Heine wrote:
> > Hi,
> > 
> > On Thu, 2018-02-01 at 17:25 +0100, Jan Kiszka wrote:
> > > On 2018-02-01 17:22, Alexander Smirnov wrote:
> > > > On 02/01/2018 07:14 PM, Jan Kiszka wrote:
> > > > > On 2018-02-01 12:29, Alexander Smirnov wrote:
> > > > > > Hi all,
> > > > > > 
> > > > > > this series intended to make buildchroot able to work with
> > > > > > isar-apt.
> > > > > > I've tried to add extended comments to each patch.
> > > > > > 
> > > > > > Impact: with this series I'm able to build 'example-hello'
> > > > > > <->
> > > > > > 'libhello'
> > > > > > without any hacks. So the deps are installed automatically.
> > > > > > 
> > > > > > NOTE: I've migrated to bitbake [lockfiles] mechanism, don't
> > > > > > know how
> > > > > > robust it
> > > > > > is, but build in the loop started in the evening didn't
> > > > > > fail
> > > > > > during
> > > > > > the night.
> > > > > > 
> > > > > 
> > > > > I can happily report: It works as promised also for my use
> > > > > case
> > > > > (jailhouse.bb -> linux-jailhouse.bb, both Isar-built).
> > > > > 
> > > > 
> > > > Thanks! But I've found an issue with events. :-( Our bitbake
> > > > didn't
> > > > handle them for multiconfig, the following patch seems to add
> > > > this:
> > > > 
> > > > https://patchwork.openembedded.org/patch/141626/
> > > > 
> > > > Events is very good mechanism to clean up Isar build from
> > > > pending
> > > > mounts
> > > > without headache with build fails, so I think it would be
> > > > valuable
> > > > to
> > > > try latest bitbake. I'll report the results as soon as build
> > > > finishes.
> > > 
> > > Perfect: Claudius just told me we need to update bitbake anyway
> > > to
> > > have
> > > proper multiconfig support (i.e. no more hacky copying of files
> > > during
> > > isar-init-build-env). Claudius, which version at least?
> > 
> > I haven't tested it but I think its this one:
> > 
> > http://git.openembedded.org/bitbake/commit/?id=518b9015c2be8d389427
> > 7a8e
> > 54890d6f04d656c0
> > 
> 
> The following patch is needed for latest bitbake (derived from
> Yocto):
> 
> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-
> bitbake.conf
> index 10cd45d..b853c88 100644
> --- a/meta/conf/isar-bitbake.conf
> +++ b/meta/conf/isar-bitbake.conf
> @@ -28,5 +28,6 @@ BUILDCHROOT_DIR = 
> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
>   BB_SIGNATURE_HANDLER ?= "noop"
> 
>   include conf/local.conf
> +include conf/multiconfig/${BB_CURRENT_MC}.conf
>   include conf/machine/${MACHINE}.conf
>   include conf/distro/${DISTRO}.conf

Right and that is exactly what is needed to be able to have own
multiconfigs in separate layers.

From the dates I suppose that your new bitbake version will contain
this commit, so then we could remove ISARMULTICONF stuff from isar-
init-build-env since copying is no longer needed.

Cheers,
Claudius



-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 0/4] Rework isar-apt
  2018-02-01 20:09           ` Claudius Heine
@ 2018-02-01 20:13             ` Claudius Heine
  0 siblings, 0 replies; 23+ messages in thread
From: Claudius Heine @ 2018-02-01 20:13 UTC (permalink / raw)
  To: Alexander Smirnov, Jan Kiszka, isar-users

[-- Attachment #1: Type: text/plain, Size: 4166 bytes --]

On Thu, 2018-02-01 at 21:09 +0100, Claudius Heine wrote:
> On Thu, 2018-02-01 at 22:37 +0300, Alexander Smirnov wrote:
> > 
> > On 02/01/2018 09:37 PM, Claudius Heine wrote:
> > > Hi,
> > > 
> > > On Thu, 2018-02-01 at 17:25 +0100, Jan Kiszka wrote:
> > > > On 2018-02-01 17:22, Alexander Smirnov wrote:
> > > > > On 02/01/2018 07:14 PM, Jan Kiszka wrote:
> > > > > > On 2018-02-01 12:29, Alexander Smirnov wrote:
> > > > > > > Hi all,
> > > > > > > 
> > > > > > > this series intended to make buildchroot able to work
> > > > > > > with
> > > > > > > isar-apt.
> > > > > > > I've tried to add extended comments to each patch.
> > > > > > > 
> > > > > > > Impact: with this series I'm able to build 'example-
> > > > > > > hello'
> > > > > > > <->
> > > > > > > 'libhello'
> > > > > > > without any hacks. So the deps are installed
> > > > > > > automatically.
> > > > > > > 
> > > > > > > NOTE: I've migrated to bitbake [lockfiles] mechanism,
> > > > > > > don't
> > > > > > > know how
> > > > > > > robust it
> > > > > > > is, but build in the loop started in the evening didn't
> > > > > > > fail
> > > > > > > during
> > > > > > > the night.
> > > > > > > 
> > > > > > 
> > > > > > I can happily report: It works as promised also for my use
> > > > > > case
> > > > > > (jailhouse.bb -> linux-jailhouse.bb, both Isar-built).
> > > > > > 
> > > > > 
> > > > > Thanks! But I've found an issue with events. :-( Our bitbake
> > > > > didn't
> > > > > handle them for multiconfig, the following patch seems to add
> > > > > this:
> > > > > 
> > > > > https://patchwork.openembedded.org/patch/141626/
> > > > > 
> > > > > Events is very good mechanism to clean up Isar build from
> > > > > pending
> > > > > mounts
> > > > > without headache with build fails, so I think it would be
> > > > > valuable
> > > > > to
> > > > > try latest bitbake. I'll report the results as soon as build
> > > > > finishes.
> > > > 
> > > > Perfect: Claudius just told me we need to update bitbake anyway
> > > > to
> > > > have
> > > > proper multiconfig support (i.e. no more hacky copying of files
> > > > during
> > > > isar-init-build-env). Claudius, which version at least?
> > > 
> > > I haven't tested it but I think its this one:
> > > 
> > > http://git.openembedded.org/bitbake/commit/?id=518b9015c2be8d3894
> > > 27
> > > 7a8e
> > > 54890d6f04d656c0
> > > 
> > 
> > The following patch is needed for latest bitbake (derived from
> > Yocto):
> > 
> > diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-
> > bitbake.conf
> > index 10cd45d..b853c88 100644
> > --- a/meta/conf/isar-bitbake.conf
> > +++ b/meta/conf/isar-bitbake.conf
> > @@ -28,5 +28,6 @@ BUILDCHROOT_DIR = 
> > "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
> >   BB_SIGNATURE_HANDLER ?= "noop"
> > 
> >   include conf/local.conf
> > +include conf/multiconfig/${BB_CURRENT_MC}.conf
> >   include conf/machine/${MACHINE}.conf
> >   include conf/distro/${DISTRO}.conf
> 
> Right and that is exactly what is needed to be able to have own
> multiconfigs in separate layers.
> 
> From the dates I suppose that your new bitbake version will contain
> this commit, so then we could remove ISARMULTICONF stuff from isar-
> init-build-env since copying is no longer needed.

Sorry, I meant 'isar-setup-builddir'.

> 
> Cheers,
> Claudius
> 
> 
> 
> -- 
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
> 
>             PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808
> B153
>                               Keyserver: hkp://pool.sks-
> keyservers.net
> 
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 0/4] Rework isar-apt
  2018-02-01 17:03         ` Jan Kiszka
@ 2018-02-01 20:32           ` Alexander Smirnov
  0 siblings, 0 replies; 23+ messages in thread
From: Alexander Smirnov @ 2018-02-01 20:32 UTC (permalink / raw)
  To: Jan Kiszka, isar-users, Claudius Heine

On 02/01/2018 08:03 PM, Jan Kiszka wrote:
> On 2018-02-01 17:54, Alexander Smirnov wrote:
>>
>>
>> On 02/01/2018 07:25 PM, Jan Kiszka wrote:
>>> On 2018-02-01 17:22, Alexander Smirnov wrote:
>>>> On 02/01/2018 07:14 PM, Jan Kiszka wrote:
>>>>> On 2018-02-01 12:29, Alexander Smirnov wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> this series intended to make buildchroot able to work with isar-apt.
>>>>>> I've tried to add extended comments to each patch.
>>>>>>
>>>>>> Impact: with this series I'm able to build 'example-hello' <->
>>>>>> 'libhello'
>>>>>> without any hacks. So the deps are installed automatically.
>>>>>>
>>>>>> NOTE: I've migrated to bitbake [lockfiles] mechanism, don't know how
>>>>>> robust it
>>>>>> is, but build in the loop started in the evening didn't fail during
>>>>>> the night.
>>>>>>
>>>>>
>>>>> I can happily report: It works as promised also for my use case
>>>>> (jailhouse.bb -> linux-jailhouse.bb, both Isar-built).
>>>>>
>>>>
>>>> Thanks! But I've found an issue with events. :-( Our bitbake didn't
>>>> handle them for multiconfig, the following patch seems to add this:
>>>>
>>>> https://patchwork.openembedded.org/patch/141626/
>>>>
>>>> Events is very good mechanism to clean up Isar build from pending mounts
>>>> without headache with build fails, so I think it would be valuable to
>>>> try latest bitbake. I'll report the results as soon as build finishes.
>>>
>>
>>  From the first like I like it:
>>
>>   - build output looks more user-friendly:
>> 0: mc:qemui386-jessie:example-hello-0.2+7bf716d2-r0 do_build - 222s (pid
>> 15637)
>>
>>   - Yes! Events are now handled by all instances, so no more pending
>> mounts. :-)
> 
> Even on ^C^C (forced termination)? That would be valuable!

It seems so, I've tested several times by interrupting the build with 
double CTRL+C, isar-apt is being unmounted in 1-2 seconds for *all* 
multiconfigs. :-)

So I definitely want this bitbake and add /proc, /dev to clean up hook.

> 
>>
>>> Perfect: Claudius just told me we need to update bitbake anyway to have
>>> proper multiconfig support (i.e. no more hacky copying of files during
>>> isar-init-build-env). Claudius, which version at least?
>>>
>>
>> Hmm, didn't get what is "hacky copying"...
> 
> That we only pick up an use multiconfig/ files from meta-isar, not any
> other layer. See
> https://groups.google.com/forum/#!msg/isar-users/IjQTuBFLPLo/IqRREtAHBgAJ
> 

Ok.

>>
>>> BTW, why are we copying bitbake in? Wouldn't a submodule make more sense?
>>>
>>
>> In my opinion not, the main idea is the same as for Yocto - have no
>> external dependencies. Isar should be self-contained.
> 
> That wouldn't change. You would still have both artifacts in the tree,
> after checkout. And you could tar them both together for release
> packaging. Or what are the concrete downsides?
> 
> I'm not using submodules heavily, but here would be a good use case IMHO.
> 

Have no experience in creation of submodules but sounds good, let me 
check this first in details, thanks for the idea!

Alex

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 2/4] buildchroot: Enable isar-apt
  2018-02-01 11:29 ` [PATCH 2/4] buildchroot: Enable isar-apt Alexander Smirnov
@ 2018-02-04 12:06   ` Jan Kiszka
  0 siblings, 0 replies; 23+ messages in thread
From: Jan Kiszka @ 2018-02-04 12:06 UTC (permalink / raw)
  To: Alexander Smirnov, isar-users

On 2018-02-01 12:29, Alexander Smirnov wrote:
> This patch provides access to isar-apt from buildchroot. It does the
> following:
>  - mount isar-apt during buildchroot building.
>  - umount isar-apt using bitbake events.
> 
> Also it needs to keep Isar build tree clean from any mounts despite on whether
> build succeed of failed. bitbake provides various events that could trigger
> custom python hooks. In this patch BuildCompleted event is used, which happened
> when bitbake finished its execution despite on the result.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta-isar/conf/local.conf.sample                     |  3 +++
>  meta/classes/isar-events.bbclass                     | 20 ++++++++++++++++++++
>  meta/recipes-devtools/buildchroot/buildchroot.bb     |  7 +++++++
>  .../buildchroot/files/multistrap.conf.in             |  8 +++++++-
>  4 files changed, 37 insertions(+), 1 deletion(-)
>  create mode 100644 meta/classes/isar-events.bbclass
> 
> diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
> index 2ae43e7..fc760ed 100644
> --- a/meta-isar/conf/local.conf.sample
> +++ b/meta-isar/conf/local.conf.sample
> @@ -158,3 +158,6 @@ IMAGE_INSTALL = "example-hello example-raw"
>  #
>  # Default parallel jobs for bitbake:
>  BB_NUMBER_THREADS = "4"
> +
> +# Add event handlers for bitbake
> +INHERIT += "isar-events"
> diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
> new file mode 100644
> index 0000000..55fc106
> --- /dev/null
> +++ b/meta/classes/isar-events.bbclass
> @@ -0,0 +1,20 @@
> +# Isar event handlers.
> +#
> +# This software is a part of ISAR.
> +# Copyright (C) 2015-2017 ilbers GmbH
> +
> +addhandler isar_handler
> +
> +python isar_handler () {
> +    import subprocess
> +
> +    devnull = open(os.devnull, 'w')
> +
> +    if isinstance(e, bb.event.BuildCompleted):
> +        bchroot = d.getVar('BUILDCHROOT_DIR', True)
> +
> +        # Clean up buildchroot
> +        subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot + '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True)
> +
> +    devnull.close()
> +}
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
> index 51f9d5d..cb90148 100644
> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
> @@ -33,6 +33,7 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>  
>  do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>  do_build[dirs] = "${WORKDIR}/hooks_multistrap"
> +do_build[depends] = "isar-apt:do_cache_config"
>  
>  do_build() {
>      E="${@ bb.utils.export_proxies(d)}"
> @@ -58,6 +59,9 @@ do_build() {
>          -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
>             "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
>  
> +    [ ! -d ${BUILDCHROOT_DIR}/isar-apt ] && install -d -m 555 ${BUILDCHROOT_DIR}/isar-apt
> +    sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
> +

You likely also want to update the cleanup hook to unmount that again...

Jan

>      [ ! -d ${BUILDCHROOT_DIR}/proc ] && install -d -m 555 ${BUILDCHROOT_DIR}/proc
>      sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
>      _do_build_cleanup() {
> @@ -73,6 +77,9 @@ do_build() {
>      # Install package builder script
>      sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
>  
> +    # Create share point for isar-apt
> +    sudo install -d ${BUILDCHROOT_DIR}/isar-apt
> +
>      # Configure root filesystem
>      sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>      _do_build_cleanup
> diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
> index a0b28e3..480a4b8 100644
> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
> +++ b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
> @@ -6,7 +6,7 @@ noauth=true
>  unpack=true
>  ignorenativearch=true
>  bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
> +aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
>  configscript=##CONFIG_SCRIPT##
>  setupscript=##SETUP_SCRIPT##
>  hookdir=##DIR_HOOKS##
> @@ -29,3 +29,9 @@ source=##DISTRO_APT_SOURCE_SEC##
>  suite=##DISTRO_SUITE##/updates
>  components=##DISTRO_COMPONENTS##
>  omitdebsrc=true
> +
> +[isar-apt]
> +source=file:///isar-apt
> +suite=isar
> +components=main
> +omitdebsrc=true
> 


^ permalink raw reply	[flat|nested] 23+ messages in thread
end of thread, other threads:[~2018-02-04 12:08 UTC | newest]

Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-01 11:29 [PATCH 0/4] Rework isar-apt Alexander Smirnov
2018-02-01 11:29 ` [PATCH 1/4] isar-apt: Introduce separate recipe Alexander Smirnov
2018-02-01 11:29 ` [PATCH 2/4] buildchroot: Enable isar-apt Alexander Smirnov
2018-02-04 12:06   ` Jan Kiszka
2018-02-01 11:29 ` [PATCH 3/4] build.sh: Update apt sources Alexander Smirnov
2018-02-01 11:40   ` Jan Kiszka
2018-02-01 11:48     ` Alexander Smirnov
2018-02-01 13:13       ` Jan Kiszka
2018-02-01 13:43         ` Alexander Smirnov
2018-02-01 14:28           ` Jan Kiszka
2018-02-01 11:29 ` [PATCH 4/4] build.sh: Force 'yes' for apt Alexander Smirnov
2018-02-01 11:41   ` Jan Kiszka
2018-02-01 12:27     ` Alexander Smirnov
2018-02-01 16:14 ` [PATCH 0/4] Rework isar-apt Jan Kiszka
2018-02-01 16:22   ` Alexander Smirnov
2018-02-01 16:25     ` Jan Kiszka
2018-02-01 16:54       ` Alexander Smirnov
2018-02-01 17:03         ` Jan Kiszka
2018-02-01 20:32           ` Alexander Smirnov
2018-02-01 18:37       ` Claudius Heine
2018-02-01 19:37         ` Alexander Smirnov
2018-02-01 20:09           ` Claudius Heine
2018-02-01 20:13             ` Claudius Heine

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox