[PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[Jan Kiszka]

Yeah, finally Yocto/OE-like usability: This ensures for many cases that
changes to recipes lead to rebuilds of dependent recipes, including the
final image. Some extra measures are needed so that those rebuilds work
with clean dirs.

And if the change detection should not work, e.g. changes to file://
resources are not detected, then a clean or cleanall task is now
available and ensures a proper manual rebuild.

This massively increases the fun factor when developing Isar projects.

Changes in v2:
- include patch 1 (dpkg-raw: Install hooks automatically) to clarify
  that this is a dependency
- add patch 5 (Move buildchroot mounts into dpkg-base class) to fix
  broken mounts when doing rebuilds
- add patch 9 (buildchroot: Clean up rootfs prior to populating it),
  analogously to patch 7 and 8

Jan

Jan Kiszka (9):
  dpkg-raw: Install hooks automatically
  Fix indention of base_do_build
  Add clean and cleanall tasks
  Enable recipe caching
  Move buildchroot mounts into dpkg-base class
  Enable proper rebuilds on dependency changes
  dpkg-raw: Clean DEBIAN dir prior to filling it
  isar-image-base: Clean rootfs folder prior to building
  buildchroot: Clean up rootfs prior to populating it

 .../recipes-app/example-raw/example-raw_0.2.bb     |  8 -----
 meta-isar/recipes-app/example-raw/files/postinst   |  4 +++
 meta-isar/recipes-core/images/isar-image-base.bb   |  2 ++
 meta/classes/base.bbclass                          | 39 +++++++++++++++++++--
 meta/classes/dpkg-base.bbclass                     | 27 ++++++++++++++-
 meta/classes/dpkg-raw.bbclass                      | 10 +++---
 meta/conf/isar-bitbake.conf                        | 18 ++++++++--
 meta/recipes-devtools/buildchroot/buildchroot.bb   | 40 +++++-----------------
 8 files changed, 99 insertions(+), 49 deletions(-)

-- 
2.13.6

[PATCH v2 1/9] dpkg-raw: Install hooks automatically

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Simplify the process of adding a pre/postinst or -rm script to a
dpkg-raw package by auto-installing all files of that name from the
WORKDIR.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta-isar/recipes-app/example-raw/example-raw_0.2.bb | 8 --------
 meta-isar/recipes-app/example-raw/files/postinst     | 4 ++++
 meta/classes/dpkg-raw.bbclass                        | 5 ++++-
 3 files changed, 8 insertions(+), 9 deletions(-)

diff --git a/meta-isar/recipes-app/example-raw/example-raw_0.2.bb b/meta-isar/recipes-app/example-raw/example-raw_0.2.bb
index 657430e..9200e0d 100644
--- a/meta-isar/recipes-app/example-raw/example-raw_0.2.bb
+++ b/meta-isar/recipes-app/example-raw/example-raw_0.2.bb
@@ -25,14 +25,6 @@ do_install() {
 	install -v -d ${D}/usr/local/doc/
 	install -v -m 644 ${WORKDIR}/README ${D}/usr/local/doc/README-${P}
 
-	bbnote "Now for a debian hook, see dpkg-deb"
-	install -v -m 755 ${WORKDIR}/postinst ${D}/DEBIAN/postinst
-
-	# this wins over meta-isar/recipes-core/images/files/*configscript.sh
-	# but we take the same password for this example
-	bbnote "Set the root password"
-	echo 'echo "root:root" | chpasswd' >> ${D}/DEBIAN/postinst
-
 	bbnote "Now for a fake config file"
 	echo "# empty config file" > ${WORKDIR}/${PN}.conf
 	install -v -d ${D}/usr/local/etc/
diff --git a/meta-isar/recipes-app/example-raw/files/postinst b/meta-isar/recipes-app/example-raw/files/postinst
index a28afa3..b82fccd 100644
--- a/meta-isar/recipes-app/example-raw/files/postinst
+++ b/meta-isar/recipes-app/example-raw/files/postinst
@@ -14,3 +14,7 @@ if ! getent passwd isar >/dev/null; then
 fi
 
 chown -R isar:isar /var/lib/isar
+
+# this wins over meta-isar/recipes-core/images/files/*configscript.sh
+# but we take the same password for this example
+echo "root:root" | chpasswd
diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index d132066..44aa078 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -35,7 +35,10 @@ do_deb_package_prepare() {
 	do
 		for a in inst rm
 		do
-			chmod -f +x ${D}/DEBIAN/${t}${a} || true
+			if [ -f ${WORKDIR}/${t}${a} ]; then
+				install -v -m 755 ${WORKDIR}/${t}${a} \
+					${D}/DEBIAN/${t}${a}
+			fi
 		done
 	done
 }
-- 
2.13.6

[PATCH v2 2/9] Fix indention of base_do_build

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Python code must not contain tabs.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/base.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index ae84677..0fa27db 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -106,8 +106,8 @@ addtask unpack after do_fetch before do_build
 addtask build
 do_build[dirs] = "${TOPDIR}"
 python base_do_build () {
-	bb.note("The included, default BB base.bbclass does not define a useful default task.")
-	bb.note("Try running the 'listtasks' task against a .bb to see what tasks are defined.")
+    bb.note("The included, default BB base.bbclass does not define a useful default task.")
+    bb.note("Try running the 'listtasks' task against a .bb to see what tasks are defined.")
 }
 
 EXPORT_FUNCTIONS do_clean do_mrproper do_build
-- 
2.13.6

[PATCH v2 3/9] Add clean and cleanall tasks

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Derived from OE, this adds clean and cleanall tasks for Isar. The clean
task comes with CLEANFUNCS hook, we just run them prior to cleaning the
WORKDIR. This is used by dpkg-base to remove all packages from isar-apt
that the build may have deployed.

As we are at it, also do a repo clean prior to deploying a package -
just in case something triggers a partial rebuild without prior
cleaning...

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/base.bbclass      | 35 ++++++++++++++++++++++++++++++++++-
 meta/classes/dpkg-base.bbclass | 14 ++++++++++++++
 2 files changed, 48 insertions(+), 1 deletion(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 0fa27db..b0e174e 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -110,4 +110,37 @@ python base_do_build () {
     bb.note("Try running the 'listtasks' task against a .bb to see what tasks are defined.")
 }
 
-EXPORT_FUNCTIONS do_clean do_mrproper do_build
+EXPORT_FUNCTIONS do_build
+
+CLEANFUNCS ?= ""
+
+# Derived from OpenEmbedded Core: meta/classes/utility-tasks.bbclass
+addtask clean
+do_clean[nostamp] = "1"
+python do_clean() {
+    import subprocess
+
+    for f in (d.getVar('CLEANFUNCS', True) or '').split():
+        bb.build.exec_func(f, d)
+
+    dir = d.expand("${WORKDIR}")
+    subprocess.call('sudo rm -rf ' + dir, shell=True)
+
+    dir = "%s.*" % bb.data.expand(d.getVar('STAMP', False), d)
+    subprocess.call('sudo rm -rf ' + dir, shell=True)
+}
+
+# Derived from OpenEmbedded Core: meta/classes/base.bbclass
+addtask cleanall after do_clean
+do_cleanall[nostamp] = "1"
+python do_cleanall() {
+    src_uri = (d.getVar('SRC_URI', True) or "").split()
+    if len(src_uri) == 0:
+        return
+
+    try:
+        fetcher = bb.fetch2.Fetch(src_uri, d)
+        fetcher.clean()
+    except bb.fetch2.BBFetchException as e:
+        bb.fatal(str(e))
+}
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 69c8ad5..a45fbce 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -31,8 +31,22 @@ do_build() {
     sudo rmdir ${BUILDROOT} 2>/dev/null || true
 }
 
+CLEANFUNCS += "repo_clean"
+
+repo_clean() {
+    PACKAGES=$(cd ${WORKDIR}; ls *.deb | sed 's/\([^_]*\).*/\1/')
+    if [ -n "${PACKAGES}" ]; then
+        reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \
+                 --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \
+                 -C main \
+                 remove ${DEBDISTRONAME} \
+                 ${PACKAGES}
+    fi
+}
+
 # Install package to Isar-apt
 do_deploy_deb() {
+    repo_clean
     reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \
              --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \
              -C main \
-- 
2.13.6

[PATCH v3 3/9] Add clean and cleanall tasks

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Derived from OE, this adds clean and cleanall tasks for Isar. The clean
task comes with CLEANFUNCS hook, we just run them prior to cleaning the
WORKDIR. This is used by dpkg-base to remove all packages from isar-apt
that the build may have deployed.

As we are at it, also do a repo clean prior to deploying a package -
just in case something triggers a partial rebuild without prior
cleaning...

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---

Changes in v3:
 - restrict repo cleanup to DISTRO_ARCH

Alex, could you check in CI if that fixes the issue you saw? I can look 
into making the parallel build work again by synchronizing dpkg 
accesses.

 meta/classes/base.bbclass      | 35 ++++++++++++++++++++++++++++++++++-
 meta/classes/dpkg-base.bbclass | 14 ++++++++++++++
 2 files changed, 48 insertions(+), 1 deletion(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 0fa27db..b0e174e 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -110,4 +110,37 @@ python base_do_build () {
     bb.note("Try running the 'listtasks' task against a .bb to see what tasks are defined.")
 }
 
-EXPORT_FUNCTIONS do_clean do_mrproper do_build
+EXPORT_FUNCTIONS do_build
+
+CLEANFUNCS ?= ""
+
+# Derived from OpenEmbedded Core: meta/classes/utility-tasks.bbclass
+addtask clean
+do_clean[nostamp] = "1"
+python do_clean() {
+    import subprocess
+
+    for f in (d.getVar('CLEANFUNCS', True) or '').split():
+        bb.build.exec_func(f, d)
+
+    dir = d.expand("${WORKDIR}")
+    subprocess.call('sudo rm -rf ' + dir, shell=True)
+
+    dir = "%s.*" % bb.data.expand(d.getVar('STAMP', False), d)
+    subprocess.call('sudo rm -rf ' + dir, shell=True)
+}
+
+# Derived from OpenEmbedded Core: meta/classes/base.bbclass
+addtask cleanall after do_clean
+do_cleanall[nostamp] = "1"
+python do_cleanall() {
+    src_uri = (d.getVar('SRC_URI', True) or "").split()
+    if len(src_uri) == 0:
+        return
+
+    try:
+        fetcher = bb.fetch2.Fetch(src_uri, d)
+        fetcher.clean()
+    except bb.fetch2.BBFetchException as e:
+        bb.fatal(str(e))
+}
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 69c8ad5..ce4b1e3 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -31,8 +31,22 @@ do_build() {
     sudo rmdir ${BUILDROOT} 2>/dev/null || true
 }
 
+CLEANFUNCS += "repo_clean"
+
+repo_clean() {
+    PACKAGES=$(cd ${WORKDIR}; ls *.deb | sed 's/\([^_]*\).*/\1/')
+    if [ -n "${PACKAGES}" ]; then
+        reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \
+                 --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \
+                 -C main -A ${DISTRO_ARCH} \
+                 remove ${DEBDISTRONAME} \
+                 ${PACKAGES}
+    fi
+}
+
 # Install package to Isar-apt
 do_deploy_deb() {
+    repo_clean
     reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \
              --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \
              -C main \
-- 
2.13.6

[PATCH v2 4/9] Enable recipe caching

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Import BB_HASHBASE_WHITELIST and BB_HASHCONFIG_WHITELIST from OE,
removing some items that are surely OE-specific, and set CACHE. That's
all what's needed to enable bitbake's recipe parsing cache.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/conf/isar-bitbake.conf | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
index f7f751c..660e0c0 100644
--- a/meta/conf/isar-bitbake.conf
+++ b/meta/conf/isar-bitbake.conf
@@ -22,6 +22,21 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}"
 DL_DIR = "${TOPDIR}/downloads"
 SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
 BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
+CACHE = "${TMPDIR}/cache"
+
+BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DIR \
+    SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
+    USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
+    PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
+    CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_NOHASHDIR LICENSE_PATH SDKPKGSUFFIX \
+    WORKDIR STAMPCLEAN PKGDATA_DIR BUILD_ARCH SSTATE_PKGARCH \
+    BB_WORKERCONTEXT BB_LIMITEDDEPS DEPLOY_DIR"
+BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SSH_AGENT_PID \
+    SSH_AUTH_SOCK PSEUDO_BUILD BB_ENV_EXTRAWHITE DISABLE_SANITY_CHECKS \
+    BB_NUMBER_THREADS BB_ORIGENV BB_INVALIDCONF BBINCLUDED \
+    GIT_PROXY_COMMAND ALL_PROXY all_proxy NO_PROXY no_proxy FTP_PROXY ftp_proxy \
+    HTTP_PROXY http_proxy HTTPS_PROXY https_proxy SOCKS5_USER SOCKS5_PASSWD \
+    BB_SETSCENE_ENFORCE BB_CMDLINE BB_SERVER_TIMEOUT"
 
 # Setup our default hash policy
 BB_SIGNATURE_HANDLER ?= "noop"
-- 
2.13.6

[PATCH v2 5/9] Move buildchroot mounts into dpkg-base class

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This avoids the unfortunate setup_mounts task which is either not
executed on rebuilds or causes excessive package rebuilds. We just need
ensure prior to switching into the buildchroot that all mount points are
populated.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass                   | 13 +++++++++-
 meta/recipes-devtools/buildchroot/buildchroot.bb | 30 +++---------------------
 2 files changed, 15 insertions(+), 28 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index a45fbce..66132da 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -2,7 +2,7 @@
 # Copyright (C) 2017 Siemens AG
 
 # Add dependency from buildchroot creation
-do_build[depends] = "buildchroot:do_setup_mounts"
+do_build[depends] = "buildchroot:do_build"
 
 # Add dependency between Isar recipes
 DEPENDS ?= ""
@@ -20,11 +20,22 @@ dpkg_runbuild() {
     die "This should never be called, overwrite it in your derived class"
 }
 
+mount_if_needed() {
+
+    # The test may race with parallel builds. So probe, mount if needed, and
+    # then only fail if both the mount and yet another probe failed as well.
+    grep -q $2 /proc/mounts || sudo mount $1 $2 || grep -q $2 /proc/mounts
+}
+
 # Wrap the function dpkg_runbuild with the bind mount for buildroot
 do_build() {
     mkdir -p ${BUILDROOT}
     sudo mount --bind ${WORKDIR} ${BUILDROOT}
 
+    mount_if_needed "--bind ${DEPLOY_DIR_APT}/${DISTRO}" ${BUILDCHROOT_DIR}/isar-apt
+    mount_if_needed "-t devtmpfs -o mode=0755,nosuid devtmpfs" ${BUILDCHROOT_DIR}/dev
+    mount_if_needed "-t proc none" ${BUILDCHROOT_DIR}/proc
+
     dpkg_runbuild
 
     sudo umount ${BUILDROOT} 2>/dev/null || true
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 570c0ad..cd91228 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -60,7 +60,9 @@ do_build() {
         -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
            "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
 
-    do_setup_mounts
+    sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
+    sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
+    sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
 
     # Create root filesystem
     sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
@@ -70,30 +72,4 @@ do_build() {
 
     # Configure root filesystem
     sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
-
-    do_cleanup_mounts
-}
-
-# Invalidate stamp for do_setup_mounts before each build start.
-# This will guarantee that this function will be executed once
-# per build.
-python __anonymous() {
-    stamp = d.getVar("STAMP") + ".do_setup_mounts." + d.getVarFlag("do_setup_mounts", 'stamp-extra-info')
-    os.remove(stamp) if os.path.exists(stamp) else None
-}
-
-do_setup_mounts[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-
-do_setup_mounts() {
-    sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
-    sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
-    sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
-}
-
-addtask setup_mounts after do_build
-
-do_cleanup_mounts() {
-    sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true
-    sudo umount ${BUILDCHROOT_DIR}/dev 2>/dev/null || true
-    sudo umount ${BUILDCHROOT_DIR}/proc 2>/dev/null || true
 }
-- 
2.13.6

Re: [PATCH v2 5/9] Move buildchroot mounts into dpkg-base class

[Alexander Smirnov]

On 02/13/2018 11:05 PM, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This avoids the unfortunate setup_mounts task which is either not
> executed on rebuilds or causes excessive package rebuilds. We just need
> ensure prior to switching into the buildchroot that all mount points are
> populated.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>   meta/classes/dpkg-base.bbclass                   | 13 +++++++++-
>   meta/recipes-devtools/buildchroot/buildchroot.bb | 30 +++---------------------
>   2 files changed, 15 insertions(+), 28 deletions(-)
> 
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index a45fbce..66132da 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -2,7 +2,7 @@
>   # Copyright (C) 2017 Siemens AG
>   
>   # Add dependency from buildchroot creation
> -do_build[depends] = "buildchroot:do_setup_mounts"
> +do_build[depends] = "buildchroot:do_build"
>   
>   # Add dependency between Isar recipes
>   DEPENDS ?= ""
> @@ -20,11 +20,22 @@ dpkg_runbuild() {
>       die "This should never be called, overwrite it in your derived class"
>   }
>   
> +mount_if_needed() {
> +
> +    # The test may race with parallel builds. So probe, mount if needed, and
> +    # then only fail if both the mount and yet another probe failed as well.
> +    grep -q $2 /proc/mounts || sudo mount $1 $2 || grep -q $2 /proc/mounts

This could lead to have lots of remounts on the same folder during 
parallel execution. This should be done in a separate task and protected 
by [lockfiles].

Alex

> +}
> +
>   # Wrap the function dpkg_runbuild with the bind mount for buildroot
>   do_build() {
>       mkdir -p ${BUILDROOT}
>       sudo mount --bind ${WORKDIR} ${BUILDROOT}
>   
> +    mount_if_needed "--bind ${DEPLOY_DIR_APT}/${DISTRO}" ${BUILDCHROOT_DIR}/isar-apt
> +    mount_if_needed "-t devtmpfs -o mode=0755,nosuid devtmpfs" ${BUILDCHROOT_DIR}/dev
> +    mount_if_needed "-t proc none" ${BUILDCHROOT_DIR}/proc
> +
>       dpkg_runbuild
>   
>       sudo umount ${BUILDROOT} 2>/dev/null || true
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
> index 570c0ad..cd91228 100644
> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
> @@ -60,7 +60,9 @@ do_build() {
>           -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
>              "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
>   
> -    do_setup_mounts
> +    sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
> +    sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
> +    sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
>   
>       # Create root filesystem
>       sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
> @@ -70,30 +72,4 @@ do_build() {
>   
>       # Configure root filesystem
>       sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
> -
> -    do_cleanup_mounts
> -}
> -
> -# Invalidate stamp for do_setup_mounts before each build start.
> -# This will guarantee that this function will be executed once
> -# per build.
> -python __anonymous() {
> -    stamp = d.getVar("STAMP") + ".do_setup_mounts." + d.getVarFlag("do_setup_mounts", 'stamp-extra-info')
> -    os.remove(stamp) if os.path.exists(stamp) else None
> -}
> -
> -do_setup_mounts[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> -
> -do_setup_mounts() {
> -    sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
> -    sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
> -    sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
> -}
> -
> -addtask setup_mounts after do_build
> -
> -do_cleanup_mounts() {
> -    sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true
> -    sudo umount ${BUILDCHROOT_DIR}/dev 2>/dev/null || true
> -    sudo umount ${BUILDCHROOT_DIR}/proc 2>/dev/null || true
>   }
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH v2 5/9] Move buildchroot mounts into dpkg-base class

[Jan Kiszka]

On 2018-02-13 21:17, Alexander Smirnov wrote:
> 
> 
> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This avoids the unfortunate setup_mounts task which is either not
>> executed on rebuilds or causes excessive package rebuilds. We just need
>> ensure prior to switching into the buildchroot that all mount points are
>> populated.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>   meta/classes/dpkg-base.bbclass                   | 13 +++++++++-
>>   meta/recipes-devtools/buildchroot/buildchroot.bb | 30
>> +++---------------------
>>   2 files changed, 15 insertions(+), 28 deletions(-)
>>
>> diff --git a/meta/classes/dpkg-base.bbclass
>> b/meta/classes/dpkg-base.bbclass
>> index a45fbce..66132da 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -2,7 +2,7 @@
>>   # Copyright (C) 2017 Siemens AG
>>     # Add dependency from buildchroot creation
>> -do_build[depends] = "buildchroot:do_setup_mounts"
>> +do_build[depends] = "buildchroot:do_build"
>>     # Add dependency between Isar recipes
>>   DEPENDS ?= ""
>> @@ -20,11 +20,22 @@ dpkg_runbuild() {
>>       die "This should never be called, overwrite it in your derived
>> class"
>>   }
>>   +mount_if_needed() {
>> +
>> +    # The test may race with parallel builds. So probe, mount if
>> needed, and
>> +    # then only fail if both the mount and yet another probe failed
>> as well.
>> +    grep -q $2 /proc/mounts || sudo mount $1 $2 || grep -q $2
>> /proc/mounts
> 
> This could lead to have lots of remounts on the same folder during
> parallel execution. This should be done in a separate task and protected
> by [lockfiles].

No, it won't - please re-read the code. It's a classic retry pattern,
and it's even detecting errors.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v2 5/9] Move buildchroot mounts into dpkg-base class

[Alexander Smirnov]

Jan Kiszka <jan.kiszka@siemens.com> 13 февраля 2018 г. 23:48:02 написал:

> On 2018-02-13 21:17, Alexander Smirnov wrote:
>>
>>
>> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>
>>> This avoids the unfortunate setup_mounts task which is either not
>>> executed on rebuilds or causes excessive package rebuilds. We just need
>>> ensure prior to switching into the buildchroot that all mount points are
>>> populated.
>>>
>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>> ---
>>>   meta/classes/dpkg-base.bbclass                   | 13 +++++++++-
>>>   meta/recipes-devtools/buildchroot/buildchroot.bb | 30
>>> +++---------------------
>>>   2 files changed, 15 insertions(+), 28 deletions(-)
>>>
>>> diff --git a/meta/classes/dpkg-base.bbclass
>>> b/meta/classes/dpkg-base.bbclass
>>> index a45fbce..66132da 100644
>>> --- a/meta/classes/dpkg-base.bbclass
>>> +++ b/meta/classes/dpkg-base.bbclass
>>> @@ -2,7 +2,7 @@
>>>   # Copyright (C) 2017 Siemens AG
>>>     # Add dependency from buildchroot creation
>>> -do_build[depends] = "buildchroot:do_setup_mounts"
>>> +do_build[depends] = "buildchroot:do_build"
>>>     # Add dependency between Isar recipes
>>>   DEPENDS ?= ""
>>> @@ -20,11 +20,22 @@ dpkg_runbuild() {
>>>       die "This should never be called, overwrite it in your derived
>>> class"
>>>   }
>>>   +mount_if_needed() {
>>> +
>>> +    # The test may race with parallel builds. So probe, mount if
>>> needed, and
>>> +    # then only fail if both the mount and yet another probe failed
>>> as well.
>>> +    grep -q $2 /proc/mounts || sudo mount $1 $2 || grep -q $2
>>> /proc/mounts
>>
>> This could lead to have lots of remounts on the same folder during
>> parallel execution. This should be done in a separate task and protected
>> by [lockfiles].
>
> No, it won't - please re-read the code. It's a classic retry pattern,
> and it's even detecting errors.
>

'mount --bind' never fails. So if several do_build() have passed first grep 
in parallel, then each one will mount 'isar-apt'. Or I don't understand this.

Alex

> Jan
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux
--
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH v2 5/9] Move buildchroot mounts into dpkg-base class

[Jan Kiszka]

On 2018-02-14 06:54, Alexander Smirnov wrote:
> 
> Jan Kiszka <jan.kiszka@siemens.com> 13 февраля 2018 г. 23:48:02 написал:
> 
>> On 2018-02-13 21:17, Alexander Smirnov wrote:
>>>
>>>
>>> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>
>>>> This avoids the unfortunate setup_mounts task which is either not
>>>> executed on rebuilds or causes excessive package rebuilds. We just need
>>>> ensure prior to switching into the buildchroot that all mount points
>>>> are
>>>> populated.
>>>>
>>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>>> ---
>>>>   meta/classes/dpkg-base.bbclass                   | 13 +++++++++-
>>>>   meta/recipes-devtools/buildchroot/buildchroot.bb | 30
>>>> +++---------------------
>>>>   2 files changed, 15 insertions(+), 28 deletions(-)
>>>>
>>>> diff --git a/meta/classes/dpkg-base.bbclass
>>>> b/meta/classes/dpkg-base.bbclass
>>>> index a45fbce..66132da 100644
>>>> --- a/meta/classes/dpkg-base.bbclass
>>>> +++ b/meta/classes/dpkg-base.bbclass
>>>> @@ -2,7 +2,7 @@
>>>>   # Copyright (C) 2017 Siemens AG
>>>>     # Add dependency from buildchroot creation
>>>> -do_build[depends] = "buildchroot:do_setup_mounts"
>>>> +do_build[depends] = "buildchroot:do_build"
>>>>     # Add dependency between Isar recipes
>>>>   DEPENDS ?= ""
>>>> @@ -20,11 +20,22 @@ dpkg_runbuild() {
>>>>       die "This should never be called, overwrite it in your derived
>>>> class"
>>>>   }
>>>>   +mount_if_needed() {
>>>> +
>>>> +    # The test may race with parallel builds. So probe, mount if
>>>> needed, and
>>>> +    # then only fail if both the mount and yet another probe failed
>>>> as well.
>>>> +    grep -q $2 /proc/mounts || sudo mount $1 $2 || grep -q $2
>>>> /proc/mounts
>>>
>>> This could lead to have lots of remounts on the same folder during
>>> parallel execution. This should be done in a separate task and protected
>>> by [lockfiles].
>>
>> No, it won't - please re-read the code. It's a classic retry pattern,
>> and it's even detecting errors.
>>
> 
> 'mount --bind' never fails. So if several do_build() have passed first
> grep in parallel, then each one will mount 'isar-apt'. Or I don't
> understand this.

That is a valid remark - I wasn't aware of that behaviour (and
explaining this earlier would have helped a lot).

I will come up with a safe variant using flock soon.

And before you pull out your task idea again, I'd like to cite from the
bitbake manual:

"Any task that depends (possibly indirectly) on a [nostamp] task will
always be executed as well. This can cause unnecessary rebuilding if you
are not careful."

That's why its broken by design.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[PATCH v3 5/9] Move buildchroot mounts into dpkg-base class

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This avoids the unfortunate setup_mounts task which is either not
executed on rebuilds or causes excessive package rebuilds. We just need
ensure prior to switching into the buildchroot that all mount points are
populated. The mounts may race with parallel build, so lock-protect the
check if they are needed and their execution.

Note that the more beautiful syntax using ( flock -n 9; ... ) 9>lockfile
as described in the flock man page does not work in bitbake shell
functions.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---

Changes in v3:
 - use flock instead of unreliable check pattern

 meta/classes/dpkg-base.bbclass                   | 11 ++++++++-
 meta/recipes-devtools/buildchroot/buildchroot.bb | 30 +++---------------------
 2 files changed, 13 insertions(+), 28 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index a45fbce..895ff26 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -2,7 +2,7 @@
 # Copyright (C) 2017 Siemens AG
 
 # Add dependency from buildchroot creation
-do_build[depends] = "buildchroot:do_setup_mounts"
+do_build[depends] = "buildchroot:do_build"
 
 # Add dependency between Isar recipes
 DEPENDS ?= ""
@@ -20,11 +20,20 @@ dpkg_runbuild() {
     die "This should never be called, overwrite it in your derived class"
 }
 
+MOUNT_LOCKFILE = "${TMPDIR}/buildchroot-mount.lock"
+
 # Wrap the function dpkg_runbuild with the bind mount for buildroot
 do_build() {
     mkdir -p ${BUILDROOT}
     sudo mount --bind ${WORKDIR} ${BUILDROOT}
 
+    sudo flock ${MOUNT_LOCKFILE} -c ' \
+        if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then \
+            mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt; \
+            mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev; \
+            mount -t proc none ${BUILDCHROOT_DIR}/proc; \
+        fi'
+
     dpkg_runbuild
 
     sudo umount ${BUILDROOT} 2>/dev/null || true
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 570c0ad..cd91228 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -60,7 +60,9 @@ do_build() {
         -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
            "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
 
-    do_setup_mounts
+    sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
+    sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
+    sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
 
     # Create root filesystem
     sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
@@ -70,30 +72,4 @@ do_build() {
 
     # Configure root filesystem
     sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
-
-    do_cleanup_mounts
-}
-
-# Invalidate stamp for do_setup_mounts before each build start.
-# This will guarantee that this function will be executed once
-# per build.
-python __anonymous() {
-    stamp = d.getVar("STAMP") + ".do_setup_mounts." + d.getVarFlag("do_setup_mounts", 'stamp-extra-info')
-    os.remove(stamp) if os.path.exists(stamp) else None
-}
-
-do_setup_mounts[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-
-do_setup_mounts() {
-    sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
-    sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
-    sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
-}
-
-addtask setup_mounts after do_build
-
-do_cleanup_mounts() {
-    sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true
-    sudo umount ${BUILDCHROOT_DIR}/dev 2>/dev/null || true
-    sudo umount ${BUILDCHROOT_DIR}/proc 2>/dev/null || true
 }
-- 
2.13.6

Re: [PATCH v3 5/9] Move buildchroot mounts into dpkg-base class

[Alexander Smirnov]

On 02/14/2018 11:52 AM, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This avoids the unfortunate setup_mounts task which is either not
> executed on rebuilds or causes excessive package rebuilds. We just need
> ensure prior to switching into the buildchroot that all mount points are
> populated. The mounts may race with parallel build, so lock-protect the
> check if they are needed and their execution.
> 
> Note that the more beautiful syntax using ( flock -n 9; ... ) 9>lockfile
> as described in the flock man page does not work in bitbake shell
> functions.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
> 
> Changes in v3:
>   - use flock instead of unreliable check pattern
> 
>   meta/classes/dpkg-base.bbclass                   | 11 ++++++++-
>   meta/recipes-devtools/buildchroot/buildchroot.bb | 30 +++---------------------
>   2 files changed, 13 insertions(+), 28 deletions(-)
> 
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index a45fbce..895ff26 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -2,7 +2,7 @@
>   # Copyright (C) 2017 Siemens AG
>   
>   # Add dependency from buildchroot creation
> -do_build[depends] = "buildchroot:do_setup_mounts"
> +do_build[depends] = "buildchroot:do_build"
>   
>   # Add dependency between Isar recipes
>   DEPENDS ?= ""
> @@ -20,11 +20,20 @@ dpkg_runbuild() {
>       die "This should never be called, overwrite it in your derived class"
>   }
>   
> +MOUNT_LOCKFILE = "${TMPDIR}/buildchroot-mount.lock"

With this name lockfile will be shared for all multiconfigs. I realize 
that it's small, but unnecessary serialization of parallel parts.

Would it be ok?

MOUNT_LOCKFILE = "${TMPDIR}/buildchroot-mount-${DISTRO}-${DISTRO_ARCH}.lock"

or

MOUNT_LOCKFILE = 
"${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/mount.lock"

or

MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock"

Alex

> +
>   # Wrap the function dpkg_runbuild with the bind mount for buildroot
>   do_build() {
>       mkdir -p ${BUILDROOT}
>       sudo mount --bind ${WORKDIR} ${BUILDROOT}
>   
> +    sudo flock ${MOUNT_LOCKFILE} -c ' \
> +        if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then \
> +            mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt; \
> +            mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev; \
> +            mount -t proc none ${BUILDCHROOT_DIR}/proc; \
> +        fi'
> +
>       dpkg_runbuild
>   
>       sudo umount ${BUILDROOT} 2>/dev/null || true
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
> index 570c0ad..cd91228 100644
> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
> @@ -60,7 +60,9 @@ do_build() {
>           -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
>              "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
>   
> -    do_setup_mounts
> +    sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
> +    sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
> +    sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
>   
>       # Create root filesystem
>       sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
> @@ -70,30 +72,4 @@ do_build() {
>   
>       # Configure root filesystem
>       sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
> -
> -    do_cleanup_mounts
> -}
> -
> -# Invalidate stamp for do_setup_mounts before each build start.
> -# This will guarantee that this function will be executed once
> -# per build.
> -python __anonymous() {
> -    stamp = d.getVar("STAMP") + ".do_setup_mounts." + d.getVarFlag("do_setup_mounts", 'stamp-extra-info')
> -    os.remove(stamp) if os.path.exists(stamp) else None
> -}
> -
> -do_setup_mounts[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> -
> -do_setup_mounts() {
> -    sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
> -    sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
> -    sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
> -}
> -
> -addtask setup_mounts after do_build
> -
> -do_cleanup_mounts() {
> -    sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true
> -    sudo umount ${BUILDCHROOT_DIR}/dev 2>/dev/null || true
> -    sudo umount ${BUILDCHROOT_DIR}/proc 2>/dev/null || true
>   }
> 

Re: [PATCH v3 5/9] Move buildchroot mounts into dpkg-base class

[Jan Kiszka]

On 2018-02-14 10:09, Alexander Smirnov wrote:
> On 02/14/2018 11:52 AM, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This avoids the unfortunate setup_mounts task which is either not
>> executed on rebuilds or causes excessive package rebuilds. We just need
>> ensure prior to switching into the buildchroot that all mount points are
>> populated. The mounts may race with parallel build, so lock-protect the
>> check if they are needed and their execution.
>>
>> Note that the more beautiful syntax using ( flock -n 9; ... ) 9>lockfile
>> as described in the flock man page does not work in bitbake shell
>> functions.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>
>> Changes in v3:
>>   - use flock instead of unreliable check pattern
>>
>>   meta/classes/dpkg-base.bbclass                   | 11 ++++++++-
>>   meta/recipes-devtools/buildchroot/buildchroot.bb | 30
>> +++---------------------
>>   2 files changed, 13 insertions(+), 28 deletions(-)
>>
>> diff --git a/meta/classes/dpkg-base.bbclass
>> b/meta/classes/dpkg-base.bbclass
>> index a45fbce..895ff26 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -2,7 +2,7 @@
>>   # Copyright (C) 2017 Siemens AG
>>     # Add dependency from buildchroot creation
>> -do_build[depends] = "buildchroot:do_setup_mounts"
>> +do_build[depends] = "buildchroot:do_build"
>>     # Add dependency between Isar recipes
>>   DEPENDS ?= ""
>> @@ -20,11 +20,20 @@ dpkg_runbuild() {
>>       die "This should never be called, overwrite it in your derived
>> class"
>>   }
>>   +MOUNT_LOCKFILE = "${TMPDIR}/buildchroot-mount.lock"
> 
> With this name lockfile will be shared for all multiconfigs. I realize
> that it's small, but unnecessary serialization of parallel parts.
> 
> Would it be ok?
> 
> MOUNT_LOCKFILE =
> "${TMPDIR}/buildchroot-mount-${DISTRO}-${DISTRO_ARCH}.lock"
> 
> or
> 
> MOUNT_LOCKFILE =
> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/mount.lock"
> 
> or
> 
> MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock"
> 

Yes, good point! Feel free to pick what ever is preferred when merging,
unless there is a reason I need to resend the whole thing anyway.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[PATCH v2 6/9] Enable proper rebuilds on dependency changes

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Install a basichash as signature handler and set the stamp policy to
full - and suddenly we get proper rebuilds of the image and all affected
packages when some recipe down the dependency chain changed!

We are using the legacy bitbake mechanism here as we do not have
setscene machinery like OE. Still good enough for us.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/conf/isar-bitbake.conf | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
index 660e0c0..c01e486 100644
--- a/meta/conf/isar-bitbake.conf
+++ b/meta/conf/isar-bitbake.conf
@@ -37,9 +37,8 @@ BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SSH_AGENT_PID \
     GIT_PROXY_COMMAND ALL_PROXY all_proxy NO_PROXY no_proxy FTP_PROXY ftp_proxy \
     HTTP_PROXY http_proxy HTTPS_PROXY https_proxy SOCKS5_USER SOCKS5_PASSWD \
     BB_SETSCENE_ENFORCE BB_CMDLINE BB_SERVER_TIMEOUT"
-
-# Setup our default hash policy
-BB_SIGNATURE_HANDLER ?= "noop"
+BB_SIGNATURE_HANDLER ?= "basichash"
+BB_STAMP_POLICY ?= "full"
 
 # Add event handlers for bitbake
 INHERIT += "isar-events"
-- 
2.13.6

[PATCH v2 7/9] dpkg-raw: Clean DEBIAN dir prior to filling it

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

When we run unprivileged, writing files in that directory will not work
after the first run because everything is root-owned at the end.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-raw.bbclass | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index 44aa078..e3ae9e8 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -16,10 +16,9 @@ do_install() {
 do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 addtask install after do_unpack before do_deb_package_prepare
 
-# so we can put hooks etc. in there already
-do_install[dirs] = "${D}/DEBIAN"
-
 do_deb_package_prepare() {
+	sudo rm -rf ${D}/DEBIAN
+	mkdir ${D}/DEBIAN
 	cat<<-__EOF__ > ${D}/DEBIAN/control
 		Package: ${PN}
 		Architecture: ${DISTRO_ARCH}
-- 
2.13.6

Re: [PATCH v2 7/9] dpkg-raw: Clean DEBIAN dir prior to filling it

[Claudius Heine]

Hi Jan,

On 02/13/2018 09:05 PM, [ext] Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> When we run unprivileged, writing files in that directory will not work
> after the first run because everything is root-owned at the end.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>   meta/classes/dpkg-raw.bbclass | 5 ++---
>   1 file changed, 2 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
> index 44aa078..e3ae9e8 100644
> --- a/meta/classes/dpkg-raw.bbclass
> +++ b/meta/classes/dpkg-raw.bbclass
> @@ -16,10 +16,9 @@ do_install() {
>   do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>   addtask install after do_unpack before do_deb_package_prepare
>   
> -# so we can put hooks etc. in there already
> -do_install[dirs] = "${D}/DEBIAN"
> -
>   do_deb_package_prepare() {
> +	sudo rm -rf ${D}/DEBIAN
> +	mkdir ${D}/DEBIAN
>   	cat<<-__EOF__ > ${D}/DEBIAN/control
>   		Package: ${PN}
>   		Architecture: ${DISTRO_ARCH}
> 

This:

   do_install() {
     sudo install -v -m 755 ${WORKDIR}/postinst ${D}/DEBIAN/postinst
   }

causes now:

   install: cannot create regular file ‘/[...]/DEBIAN/postinst’: No such 
file or directory

How should that be fixed?

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH v2 7/9] dpkg-raw: Clean DEBIAN dir prior to filling it

[Jan Kiszka]

On 2018-02-15 10:14, Claudius Heine wrote:
> Hi Jan,
> 
> On 02/13/2018 09:05 PM, [ext] Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> When we run unprivileged, writing files in that directory will not work
>> after the first run because everything is root-owned at the end.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>   meta/classes/dpkg-raw.bbclass | 5 ++---
>>   1 file changed, 2 insertions(+), 3 deletions(-)
>>
>> diff --git a/meta/classes/dpkg-raw.bbclass
>> b/meta/classes/dpkg-raw.bbclass
>> index 44aa078..e3ae9e8 100644
>> --- a/meta/classes/dpkg-raw.bbclass
>> +++ b/meta/classes/dpkg-raw.bbclass
>> @@ -16,10 +16,9 @@ do_install() {
>>   do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   addtask install after do_unpack before do_deb_package_prepare
>>   -# so we can put hooks etc. in there already
>> -do_install[dirs] = "${D}/DEBIAN"
>> -
>>   do_deb_package_prepare() {
>> +    sudo rm -rf ${D}/DEBIAN
>> +    mkdir ${D}/DEBIAN
>>       cat<<-__EOF__ > ${D}/DEBIAN/control
>>           Package: ${PN}
>>           Architecture: ${DISTRO_ARCH}
>>
> 
> This:
> 
>   do_install() {
>     sudo install -v -m 755 ${WORKDIR}/postinst ${D}/DEBIAN/postinst
>   }
> 
> causes now:
> 
>   install: cannot create regular file ‘/[...]/DEBIAN/postinst’: No such
> file or directory
> 
> How should that be fixed?

Simply by removing the install above. "dpkg-raw: Install hooks
automatically" makes it obsolete.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v2 7/9] dpkg-raw: Clean DEBIAN dir prior to filling it

[Claudius Heine]

On 02/15/2018 10:18 AM, Jan Kiszka wrote:
> On 2018-02-15 10:14, Claudius Heine wrote:
>> Hi Jan,
>>
>> On 02/13/2018 09:05 PM, [ext] Jan Kiszka wrote:
>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>
>>> When we run unprivileged, writing files in that directory will not work
>>> after the first run because everything is root-owned at the end.
>>>
>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>> ---
>>>    meta/classes/dpkg-raw.bbclass | 5 ++---
>>>    1 file changed, 2 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/meta/classes/dpkg-raw.bbclass
>>> b/meta/classes/dpkg-raw.bbclass
>>> index 44aa078..e3ae9e8 100644
>>> --- a/meta/classes/dpkg-raw.bbclass
>>> +++ b/meta/classes/dpkg-raw.bbclass
>>> @@ -16,10 +16,9 @@ do_install() {
>>>    do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>>    addtask install after do_unpack before do_deb_package_prepare
>>>    -# so we can put hooks etc. in there already
>>> -do_install[dirs] = "${D}/DEBIAN"
>>> -
>>>    do_deb_package_prepare() {
>>> +    sudo rm -rf ${D}/DEBIAN
>>> +    mkdir ${D}/DEBIAN
>>>        cat<<-__EOF__ > ${D}/DEBIAN/control
>>>            Package: ${PN}
>>>            Architecture: ${DISTRO_ARCH}
>>>
>>
>> This:
>>
>>    do_install() {
>>      sudo install -v -m 755 ${WORKDIR}/postinst ${D}/DEBIAN/postinst
>>    }
>>
>> causes now:
>>
>>    install: cannot create regular file ‘/[...]/DEBIAN/postinst’: No such
>> file or directory
>>
>> How should that be fixed?
> 
> Simply by removing the install above. "dpkg-raw: Install hooks
> automatically" makes it obsolete.

Thx! But now it shows:

   DEBUG: Executing shell function do_deb_package_prepare
   mkdir: cannot create directory ‘/[...]/DEBIAN’: No such file or directory

I suppose it needs "mkdir -p ${D}/DEBIAN".

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH v2 7/9] dpkg-raw: Clean DEBIAN dir prior to filling it

[Alexander Smirnov]

On 02/15/2018 01:26 PM, Claudius Heine wrote:
> On 02/15/2018 10:18 AM, Jan Kiszka wrote:
>> On 2018-02-15 10:14, Claudius Heine wrote:
>>> Hi Jan,
>>>
>>> On 02/13/2018 09:05 PM, [ext] Jan Kiszka wrote:
>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>
>>>> When we run unprivileged, writing files in that directory will not work
>>>> after the first run because everything is root-owned at the end.
>>>>
>>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>>> ---
>>>>    meta/classes/dpkg-raw.bbclass | 5 ++---
>>>>    1 file changed, 2 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/meta/classes/dpkg-raw.bbclass
>>>> b/meta/classes/dpkg-raw.bbclass
>>>> index 44aa078..e3ae9e8 100644
>>>> --- a/meta/classes/dpkg-raw.bbclass
>>>> +++ b/meta/classes/dpkg-raw.bbclass
>>>> @@ -16,10 +16,9 @@ do_install() {
>>>>    do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>>>    addtask install after do_unpack before do_deb_package_prepare
>>>>    -# so we can put hooks etc. in there already
>>>> -do_install[dirs] = "${D}/DEBIAN"
>>>> -
>>>>    do_deb_package_prepare() {
>>>> +    sudo rm -rf ${D}/DEBIAN
>>>> +    mkdir ${D}/DEBIAN
>>>>        cat<<-__EOF__ > ${D}/DEBIAN/control
>>>>            Package: ${PN}
>>>>            Architecture: ${DISTRO_ARCH}
>>>>
>>>
>>> This:
>>>
>>>    do_install() {
>>>      sudo install -v -m 755 ${WORKDIR}/postinst ${D}/DEBIAN/postinst
>>>    }
>>>
>>> causes now:
>>>
>>>    install: cannot create regular file ‘/[...]/DEBIAN/postinst’: No such
>>> file or directory
>>>
>>> How should that be fixed?
>>
>> Simply by removing the install above. "dpkg-raw: Install hooks
>> automatically" makes it obsolete.
> 
> Thx! But now it shows:
> 
>    DEBUG: Executing shell function do_deb_package_prepare
>    mkdir: cannot create directory ‘/[...]/DEBIAN’: No such file or 
> directory
> 
> I suppose it needs "mkdir -p ${D}/DEBIAN".

BTW: this series is already in next and it has passed all CI tests. 
Which git head you use?

Alex

[PATCH] dpkg-raw: Fix creation of DEBIAN folder

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

If do_install did not do this for us, we failed.

Fixes: b04530d19c79 (dpkg-raw: Clean DEBIAN dir prior to filling it)
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-raw.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index e0ea811..28233ac 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -18,7 +18,7 @@ addtask install after do_unpack before do_deb_package_prepare
 
 do_deb_package_prepare() {
 	sudo rm -rf ${D}/DEBIAN
-	mkdir ${D}/DEBIAN
+	mkdir -p ${D}/DEBIAN
 	cat<<-__EOF__ > ${D}/DEBIAN/control
 		Package: ${PN}
 		Architecture: ${DISTRO_ARCH}
-- 
2.13.6

Re: [PATCH] dpkg-raw: Fix creation of DEBIAN folder

[Alexander Smirnov]

On 02/15/2018 01:53 PM, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> If do_install did not do this for us, we failed.
> 
> Fixes: b04530d19c79 (dpkg-raw: Clean DEBIAN dir prior to filling it)
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>

Applied to next, thanks!

Alex

Re: [PATCH] dpkg-raw: Fix creation of DEBIAN folder

[Andreas Reichel]

On Thu, Feb 15, 2018 at 11:53:39AM +0100, [ext] Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>

I just noticed, for me it tries to create a DEBIAN folder inside
an "image" folder, that has root-only write permissions and thus
fails since the function is executed as user "builder".

I reproduced this with a clean build and a dpkg-raw recipe.

No time to debug this atm.

> 
> If do_install did not do this for us, we failed.
> 
> Fixes: b04530d19c79 (dpkg-raw: Clean DEBIAN dir prior to filling it)
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/classes/dpkg-raw.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
> index e0ea811..28233ac 100644
> --- a/meta/classes/dpkg-raw.bbclass
> +++ b/meta/classes/dpkg-raw.bbclass
> @@ -18,7 +18,7 @@ addtask install after do_unpack before do_deb_package_prepare
>  
>  do_deb_package_prepare() {
>  	sudo rm -rf ${D}/DEBIAN
> -	mkdir ${D}/DEBIAN
> +	mkdir -p ${D}/DEBIAN
>  	cat<<-__EOF__ > ${D}/DEBIAN/control
>  		Package: ${PN}
>  		Architecture: ${DISTRO_ARCH}
> -- 
> 2.13.6
> 
> -- 
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
> To post to this group, send email to isar-users@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/3ae97982-962e-be4d-22a6-477db2fad432%40siemens.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: [PATCH] dpkg-raw: Fix creation of DEBIAN folder

[Jan Kiszka]

On 2018-02-26 12:39, Andreas Reichel wrote:
> On Thu, Feb 15, 2018 at 11:53:39AM +0100, [ext] Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> I just noticed, for me it tries to create a DEBIAN folder inside
> an "image" folder, that has root-only write permissions and thus
> fails since the function is executed as user "builder".
> 
> I reproduced this with a clean build and a dpkg-raw recipe.
> 
> No time to debug this atm.

I suppose that was a rebuild, not a clean build (otherwise the ownership
would not be on root already). Anyway, yet another example for our mess
with ownerships vs. cleanups.

Maybe there is some way to overload the functions that bitbake uses when
it tries to do mytask[cleandirs] and switch back to a consistent bitbake
model. But I didn't figure this out yet.

Jan

> 
>>
>> If do_install did not do this for us, we failed.
>>
>> Fixes: b04530d19c79 (dpkg-raw: Clean DEBIAN dir prior to filling it)
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>  meta/classes/dpkg-raw.bbclass | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
>> index e0ea811..28233ac 100644
>> --- a/meta/classes/dpkg-raw.bbclass
>> +++ b/meta/classes/dpkg-raw.bbclass
>> @@ -18,7 +18,7 @@ addtask install after do_unpack before do_deb_package_prepare
>>  
>>  do_deb_package_prepare() {
>>  	sudo rm -rf ${D}/DEBIAN
>> -	mkdir ${D}/DEBIAN
>> +	mkdir -p ${D}/DEBIAN
>>  	cat<<-__EOF__ > ${D}/DEBIAN/control
>>  		Package: ${PN}
>>  		Architecture: ${DISTRO_ARCH}
>> -- 
>> 2.13.6
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups "isar-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
>> To post to this group, send email to isar-users@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/3ae97982-962e-be4d-22a6-477db2fad432%40siemens.com.
>> For more options, visit https://groups.google.com/d/optout.
> 


-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH] dpkg-raw: Fix creation of DEBIAN folder

[Andreas Reichel]

On Mon, Feb 26, 2018 at 12:39:34PM +0100, Andreas Reichel wrote:
> On Thu, Feb 15, 2018 at 11:53:39AM +0100, [ext] Jan Kiszka wrote:
> > From: Jan Kiszka <jan.kiszka@siemens.com>
> 
Please ignore this report :) my do_install was wrong

> I just noticed, for me it tries to create a DEBIAN folder inside
> an "image" folder, that has root-only write permissions and thus
> fails since the function is executed as user "builder".
> 
> I reproduced this with a clean build and a dpkg-raw recipe.
> 
> No time to debug this atm.
> 
> > 
> > If do_install did not do this for us, we failed.
> > 
> > Fixes: b04530d19c79 (dpkg-raw: Clean DEBIAN dir prior to filling it)
> > Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> > ---
> >  meta/classes/dpkg-raw.bbclass | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
> > index e0ea811..28233ac 100644
> > --- a/meta/classes/dpkg-raw.bbclass
> > +++ b/meta/classes/dpkg-raw.bbclass
> > @@ -18,7 +18,7 @@ addtask install after do_unpack before do_deb_package_prepare
> >  
> >  do_deb_package_prepare() {
> >  	sudo rm -rf ${D}/DEBIAN
> > -	mkdir ${D}/DEBIAN
> > +	mkdir -p ${D}/DEBIAN
> >  	cat<<-__EOF__ > ${D}/DEBIAN/control
> >  		Package: ${PN}
> >  		Architecture: ${DISTRO_ARCH}
> > -- 
> > 2.13.6
> > 
> > -- 
> > You received this message because you are subscribed to the Google Groups "isar-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
> > To post to this group, send email to isar-users@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/3ae97982-962e-be4d-22a6-477db2fad432%40siemens.com.
> > For more options, visit https://groups.google.com/d/optout.
> 
> -- 
> Andreas Reichel
> Dipl.-Phys. (Univ.)
> Software Consultant
> 
> Andreas.Reichel@tngtech.com, +49-174-3180074
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
> Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
> Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082
> 

-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: [PATCH v2 7/9] dpkg-raw: Clean DEBIAN dir prior to filling it

[Claudius Heine]

Hi,

On 02/15/2018 11:30 AM, Alexander Smirnov wrote:
> 
> 
> On 02/15/2018 01:26 PM, Claudius Heine wrote:
>> On 02/15/2018 10:18 AM, Jan Kiszka wrote:
>>> On 2018-02-15 10:14, Claudius Heine wrote:
>>>> Hi Jan,
>>>>
>>>> On 02/13/2018 09:05 PM, [ext] Jan Kiszka wrote:
>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>
>>>>> When we run unprivileged, writing files in that directory will not 
>>>>> work
>>>>> after the first run because everything is root-owned at the end.
>>>>>
>>>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>>>> ---
>>>>>    meta/classes/dpkg-raw.bbclass | 5 ++---
>>>>>    1 file changed, 2 insertions(+), 3 deletions(-)
>>>>>
>>>>> diff --git a/meta/classes/dpkg-raw.bbclass
>>>>> b/meta/classes/dpkg-raw.bbclass
>>>>> index 44aa078..e3ae9e8 100644
>>>>> --- a/meta/classes/dpkg-raw.bbclass
>>>>> +++ b/meta/classes/dpkg-raw.bbclass
>>>>> @@ -16,10 +16,9 @@ do_install() {
>>>>>    do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>>>>    addtask install after do_unpack before do_deb_package_prepare
>>>>>    -# so we can put hooks etc. in there already
>>>>> -do_install[dirs] = "${D}/DEBIAN"
>>>>> -
>>>>>    do_deb_package_prepare() {
>>>>> +    sudo rm -rf ${D}/DEBIAN
>>>>> +    mkdir ${D}/DEBIAN
>>>>>        cat<<-__EOF__ > ${D}/DEBIAN/control
>>>>>            Package: ${PN}
>>>>>            Architecture: ${DISTRO_ARCH}
>>>>>
>>>>
>>>> This:
>>>>
>>>>    do_install() {
>>>>      sudo install -v -m 755 ${WORKDIR}/postinst ${D}/DEBIAN/postinst
>>>>    }
>>>>
>>>> causes now:
>>>>
>>>>    install: cannot create regular file ‘/[...]/DEBIAN/postinst’: No 
>>>> such
>>>> file or directory
>>>>
>>>> How should that be fixed?
>>>
>>> Simply by removing the install above. "dpkg-raw: Install hooks
>>> automatically" makes it obsolete.
>>
>> Thx! But now it shows:
>>
>>    DEBUG: Executing shell function do_deb_package_prepare
>>    mkdir: cannot create directory ‘/[...]/DEBIAN’: No such file or 
>> directory
>>
>> I suppose it needs "mkdir -p ${D}/DEBIAN".
> 
> BTW: this series is already in next and it has passed all CI tests. 
> Which git head you use?

I rebased my own branch to next (e7e40a68adbfa60), and my stuff broke. 
AFAIK next is there to test stuff with our own setup and provide 
feedback, so that commits can be fixed there with sometimes even some 
history rewriting.

I wouldn't be against squashing Jans patch into this one on next.

regards,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

[PATCH v2 8/9] isar-image-base: Clean rootfs folder prior to building

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This ensures we have a clean base when rerunning the task.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta-isar/recipes-core/images/isar-image-base.bb | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index 8ddbabb..8678b51 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -31,6 +31,8 @@ do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap"
 do_rootfs() {
     E="${@ bb.utils.export_proxies(d)}"
 
+    sudo rm -rf ${IMAGE_ROOTFS}
+
     chmod +x "${WORKDIR}/${DISTRO_CONFIG_SCRIPT}"
     chmod +x "${WORKDIR}/setup.sh"
     install -m 755 "${WORKDIR}/download_dev-random" "${WORKDIR}/hooks_multistrap/"
-- 
2.13.6

[PATCH v2 9/9] buildchroot: Clean up rootfs prior to populating it

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This ensures we are always re-building a clean image.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-devtools/buildchroot/buildchroot.bb | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index cd91228..62979f5 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -31,15 +31,17 @@ BUILDCHROOT_PREINSTALL ?= "gcc \
 WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
 
 do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \
-                  ${BUILDCHROOT_DIR}/dev \
-                  ${BUILDCHROOT_DIR}/proc \
-                  ${BUILDCHROOT_DIR}/sys"
 do_build[depends] = "isar-apt:do_cache_config"
 
 do_build() {
     E="${@ bb.utils.export_proxies(d)}"
 
+    sudo rm -rf ${BUILDCHROOT_DIR}
+    mkdir -p ${BUILDCHROOT_DIR}/isar-apt
+    mkdir -p ${BUILDCHROOT_DIR}/dev
+    mkdir -p ${BUILDCHROOT_DIR}/proc
+    mkdir -p ${BUILDCHROOT_DIR}/sys
+
     chmod +x "${WORKDIR}/setup.sh"
     chmod +x "${WORKDIR}/configscript.sh"
 
-- 
2.13.6

Re: [PATCH v2 9/9] buildchroot: Clean up rootfs prior to populating it

[Alexander Smirnov]

On 02/13/2018 11:05 PM, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This ensures we are always re-building a clean image.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>   meta/recipes-devtools/buildchroot/buildchroot.bb | 10 ++++++----
>   1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
> index cd91228..62979f5 100644
> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
> @@ -31,15 +31,17 @@ BUILDCHROOT_PREINSTALL ?= "gcc \
>   WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>   
>   do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> -do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \
> -                  ${BUILDCHROOT_DIR}/dev \
> -                  ${BUILDCHROOT_DIR}/proc \
> -                  ${BUILDCHROOT_DIR}/sys"
>   do_build[depends] = "isar-apt:do_cache_config"
>   
>   do_build() {
>       E="${@ bb.utils.export_proxies(d)}"
>   

Do you have any concerns if I duplicate commit comment here? Otherwise I 
expect in near future patch which moves this to [dirs]. Already had 
several cycles of doing this in `git log` :-)

> +    sudo rm -rf ${BUILDCHROOT_DIR}
> +    mkdir -p ${BUILDCHROOT_DIR}/isar-apt
> +    mkdir -p ${BUILDCHROOT_DIR}/dev
> +    mkdir -p ${BUILDCHROOT_DIR}/proc
> +    mkdir -p ${BUILDCHROOT_DIR}/sys
> +
>       chmod +x "${WORKDIR}/setup.sh"
>       chmod +x "${WORKDIR}/configscript.sh"
>   
> 

Alex

Re: [PATCH v2 9/9] buildchroot: Clean up rootfs prior to populating it

[Jan Kiszka]

On 2018-02-14 18:43, Alexander Smirnov wrote:
> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This ensures we are always re-building a clean image.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>   meta/recipes-devtools/buildchroot/buildchroot.bb | 10 ++++++----
>>   1 file changed, 6 insertions(+), 4 deletions(-)
>>
>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
>> b/meta/recipes-devtools/buildchroot/buildchroot.bb
>> index cd91228..62979f5 100644
>> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
>> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
>> @@ -31,15 +31,17 @@ BUILDCHROOT_PREINSTALL ?= "gcc \
>>   WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>>     do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>> -do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \
>> -                  ${BUILDCHROOT_DIR}/dev \
>> -                  ${BUILDCHROOT_DIR}/proc \
>> -                  ${BUILDCHROOT_DIR}/sys"
>>   do_build[depends] = "isar-apt:do_cache_config"
>>     do_build() {
>>       E="${@ bb.utils.export_proxies(d)}"
>>   
> 
> Do you have any concerns if I duplicate commit comment here? Otherwise I
> expect in near future patch which moves this to [dirs]. Already had
> several cycles of doing this in `git log` :-)

Yes, absolutely!

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[Alexander Smirnov]

On 02/13/2018 11:05 PM, Jan Kiszka wrote:
> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
> changes to recipes lead to rebuilds of dependent recipes, including the
> final image. Some extra measures are needed so that those rebuilds work
> with clean dirs.
> 
> And if the change detection should not work, e.g. changes to file://
> resources are not detected, then a clean or cleanall task is now
> available and ensures a proper manual rebuild.
> 
> This massively increases the fun factor when developing Isar projects.


Wow, just noticed that with this series bitbake has started to run tasks 
in parallel.. How it's working? :-)


Currently  4 running tasks (54 of 217)  24% 
|######################################## 
 
                      |
0: mc:rpi-jessie:buildchroot-1.0-r0 do_build - 101s (pid 3271)
1: mc:qemui386-stretch:buildchroot-1.0-r0 do_build - 101s (pid 3336)
2: mc:qemui386-jessie:buildchroot-1.0-r0 do_build - 100s (pid 3457)
3: mc:qemuarm-wheezy:buildchroot-1.0-r0 do_build - 100s (pid 3524)

Alex

Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[Jan Kiszka]

On 2018-02-14 12:33, Alexander Smirnov wrote:
> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
>> changes to recipes lead to rebuilds of dependent recipes, including the
>> final image. Some extra measures are needed so that those rebuilds work
>> with clean dirs.
>>
>> And if the change detection should not work, e.g. changes to file://
>> resources are not detected, then a clean or cleanall task is now
>> available and ensures a proper manual rebuild.
>>
>> This massively increases the fun factor when developing Isar projects.
> 
> 
> Wow, just noticed that with this series bitbake has started to run tasks
> in parallel.. How it's working? :-)
> 
> 
> Currently  4 running tasks (54 of 217)  24%
> |########################################
>                      |
> 0: mc:rpi-jessie:buildchroot-1.0-r0 do_build - 101s (pid 3271)
> 1: mc:qemui386-stretch:buildchroot-1.0-r0 do_build - 101s (pid 3336)
> 2: mc:qemui386-jessie:buildchroot-1.0-r0 do_build - 100s (pid 3457)
> 3: mc:qemuarm-wheezy:buildchroot-1.0-r0 do_build - 100s (pid 3524)

Didn't notice that this wasn't the case before (most of my workload had
linear ordering), but I could imagine that the lacking signatures forced
bitbake to serialize. Just guessing.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[Alexander Smirnov]

[-- Attachment #1: Type: text/plain, Size: 2071 bytes --]

On 02/14/2018 03:41 PM, Jan Kiszka wrote:
> On 2018-02-14 12:33, Alexander Smirnov wrote:
>> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>>> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
>>> changes to recipes lead to rebuilds of dependent recipes, including the
>>> final image. Some extra measures are needed so that those rebuilds work
>>> with clean dirs.
>>>
>>> And if the change detection should not work, e.g. changes to file://
>>> resources are not detected, then a clean or cleanall task is now
>>> available and ensures a proper manual rebuild.
>>>
>>> This massively increases the fun factor when developing Isar projects.
>>
>>
>> Wow, just noticed that with this series bitbake has started to run tasks
>> in parallel.. How it's working? :-)
>>
>>
>> Currently  4 running tasks (54 of 217)  24%
>> |########################################
>>                       |
>> 0: mc:rpi-jessie:buildchroot-1.0-r0 do_build - 101s (pid 3271)
>> 1: mc:qemui386-stretch:buildchroot-1.0-r0 do_build - 101s (pid 3336)
>> 2: mc:qemui386-jessie:buildchroot-1.0-r0 do_build - 100s (pid 3457)
>> 3: mc:qemuarm-wheezy:buildchroot-1.0-r0 do_build - 100s (pid 3524)
> 
> Didn't notice that this wasn't the case before (most of my workload had
> linear ordering), but I could imagine that the lacking signatures forced
> bitbake to serialize. Just guessing.
> 

There is still the same build problem as reported yesterday, log is 
attached. Dropping patch #3 fixes it.

Line I used to build:

$ time bitbake multiconfig:qemuarm-wheezy:isar-image-base 
multiconfig:qemuarm-jessie:isar-image-base 
multiconfig:qemuarm-stretch:isar-image-base 
multiconfig:qemui386-jessie:isar-image-base 
multiconfig:qemui386-stretch:isar-image-base 
multiconfig:qemuamd64-jessie:isar-image-base 
multiconfig:qemuamd64-stretch:isar-image-base 
multiconfig:rpi-jessie:isar-image-base

Just for sure stated build in CI server:
http://isar-build.org:8080/job/isar_asmirnov_devel/8/console

Also failed. Could I dropped this patch from the series?

Alex

[-- Attachment #2: log.txt --]
[-- Type: text/plain, Size: 3183 bytes --]

builder@zbook:~/isar$ cat /home/builder/isar/build/tmp/work/debian-stretch-armhf/example-hello-0.2-86cc719-r0/temp/log.do_build.21911
DEBUG: Executing shell function do_build
Get:1 file:/isar-apt isar InRelease
Ign:1 file:/isar-apt isar InRelease
Get:2 file:/isar-apt isar Release [2,861 B]
Get:2 file:/isar-apt isar Release [2,861 B]
Get:3 file:/isar-apt isar Release.gpg
Ign:3 file:/isar-apt isar Release.gpg
Reading package lists...
W: The repository 'file:/isar-apt isar Release' is not signed.
hostname: Name or service not known
dh_testdir
dh_testroot
dh_prep
dh_testdir
dh_testroot
dh_install
dh_install: Compatibility levels before 9 are deprecated (level 7 in use)
dh_installdocs
dh_installdocs: Compatibility levels before 9 are deprecated (level 7 in use)
dh_installchangelogs
dh_compress
dh_fixperms
dh_installdeb
dh_installdeb: Compatibility levels before 9 are deprecated (level 7 in use)
dh_gencontrol
dh_md5sums
dh_builddeb
dpkg-deb: building package 'hello-build-deps' in '../hello-build-deps_0.2_all.deb'.

The package has been created.
Attention, the package has been created in the current directory,
not in ".." as indicated by the message above!
Selecting previously unselected package hello-build-deps.
(Reading database ... 15754 files and directories currently installed.)
Preparing to unpack hello-build-deps_0.2_all.deb ...
Unpacking hello-build-deps (0.2) ...
Reading package lists...
Building dependency tree...
Reading state information...
Correcting dependencies...Starting pkgProblemResolver with broken count: 1
Starting 2 pkgProblemResolver with broken count: 1
Investigating (0) hello-build-deps:armhf < 0.2 @iU mK Nb Ib >
Broken hello-build-deps:armhf Depends on libhello-dev:armhf < none @un H >
  Removing hello-build-deps:armhf because I can't find libhello-dev:armhf
Done
 Done
Starting pkgProblemResolver with broken count: 0
Starting 2 pkgProblemResolver with broken count: 0
Done
The following packages will be REMOVED:
  hello-build-deps
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 9,216 B disk space will be freed.
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (19: No such device)
(Reading database ... 15758 files and directories currently installed.)
Removing hello-build-deps (0.2) ...
mk-build-deps: Unable to install hello-build-deps at /usr/bin/mk-build-deps line 402.
mk-build-deps: Unable to install all build-dep packages
dpkg-buildpackage: info: source package hello
dpkg-buildpackage: info: source version 0.2
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Alexander Smirnov <asmirnov@ilbers.de>
 dpkg-source --before-build git
dpkg-buildpackage: info: host architecture armhf
dpkg-checkbuilddeps: error: Unmet build dependencies: libhello-dev
dpkg-buildpackage: warning: build dependencies/conflicts unsatisfied; aborting
dpkg-buildpackage: warning: (Use -d flag to override.)
WARNING: exit code 3 from a shell command.
ERROR: Function failed: do_build (log file is located at /home/builder/isar/build/tmp/work/debian-stretch-armhf/example-hello-0.2-86cc719-r0/temp/log.do_build.21911)

Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[Jan Kiszka]

On 2018-02-14 13:57, Alexander Smirnov wrote:
> On 02/14/2018 03:41 PM, Jan Kiszka wrote:
>> On 2018-02-14 12:33, Alexander Smirnov wrote:
>>> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>>>> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
>>>> changes to recipes lead to rebuilds of dependent recipes, including the
>>>> final image. Some extra measures are needed so that those rebuilds work
>>>> with clean dirs.
>>>>
>>>> And if the change detection should not work, e.g. changes to file://
>>>> resources are not detected, then a clean or cleanall task is now
>>>> available and ensures a proper manual rebuild.
>>>>
>>>> This massively increases the fun factor when developing Isar projects.
>>>
>>>
>>> Wow, just noticed that with this series bitbake has started to run tasks
>>> in parallel.. How it's working? :-)
>>>
>>>
>>> Currently  4 running tasks (54 of 217)  24%
>>> |########################################
>>>                       |
>>> 0: mc:rpi-jessie:buildchroot-1.0-r0 do_build - 101s (pid 3271)
>>> 1: mc:qemui386-stretch:buildchroot-1.0-r0 do_build - 101s (pid 3336)
>>> 2: mc:qemui386-jessie:buildchroot-1.0-r0 do_build - 100s (pid 3457)
>>> 3: mc:qemuarm-wheezy:buildchroot-1.0-r0 do_build - 100s (pid 3524)
>>
>> Didn't notice that this wasn't the case before (most of my workload had
>> linear ordering), but I could imagine that the lacking signatures forced
>> bitbake to serialize. Just guessing.
>>
> 
> There is still the same build problem as reported yesterday, log is
> attached. Dropping patch #3 fixes it.
> 
> Line I used to build:
> 
> $ time bitbake multiconfig:qemuarm-wheezy:isar-image-base
> multiconfig:qemuarm-jessie:isar-image-base
> multiconfig:qemuarm-stretch:isar-image-base
> multiconfig:qemui386-jessie:isar-image-base
> multiconfig:qemui386-stretch:isar-image-base
> multiconfig:qemuamd64-jessie:isar-image-base
> multiconfig:qemuamd64-stretch:isar-image-base
> multiconfig:rpi-jessie:isar-image-base
> 
> Just for sure stated build in CI server:
> http://isar-build.org:8080/job/isar_asmirnov_devel/8/console
> 
> Also failed. Could I dropped this patch from the series?

Not before you can explain why things fail and why that patch causes
this. We need to understand the problem. Let me reproduce...

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[Jan Kiszka]

On 2018-02-14 14:10, Jan Kiszka wrote:
> On 2018-02-14 13:57, Alexander Smirnov wrote:
>> On 02/14/2018 03:41 PM, Jan Kiszka wrote:
>>> On 2018-02-14 12:33, Alexander Smirnov wrote:
>>>> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>>>>> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
>>>>> changes to recipes lead to rebuilds of dependent recipes, including the
>>>>> final image. Some extra measures are needed so that those rebuilds work
>>>>> with clean dirs.
>>>>>
>>>>> And if the change detection should not work, e.g. changes to file://
>>>>> resources are not detected, then a clean or cleanall task is now
>>>>> available and ensures a proper manual rebuild.
>>>>>
>>>>> This massively increases the fun factor when developing Isar projects.
>>>>
>>>>
>>>> Wow, just noticed that with this series bitbake has started to run tasks
>>>> in parallel.. How it's working? :-)
>>>>
>>>>
>>>> Currently  4 running tasks (54 of 217)  24%
>>>> |########################################
>>>>                       |
>>>> 0: mc:rpi-jessie:buildchroot-1.0-r0 do_build - 101s (pid 3271)
>>>> 1: mc:qemui386-stretch:buildchroot-1.0-r0 do_build - 101s (pid 3336)
>>>> 2: mc:qemui386-jessie:buildchroot-1.0-r0 do_build - 100s (pid 3457)
>>>> 3: mc:qemuarm-wheezy:buildchroot-1.0-r0 do_build - 100s (pid 3524)
>>>
>>> Didn't notice that this wasn't the case before (most of my workload had
>>> linear ordering), but I could imagine that the lacking signatures forced
>>> bitbake to serialize. Just guessing.
>>>
>>
>> There is still the same build problem as reported yesterday, log is
>> attached. Dropping patch #3 fixes it.
>>
>> Line I used to build:
>>
>> $ time bitbake multiconfig:qemuarm-wheezy:isar-image-base
>> multiconfig:qemuarm-jessie:isar-image-base
>> multiconfig:qemuarm-stretch:isar-image-base
>> multiconfig:qemui386-jessie:isar-image-base
>> multiconfig:qemui386-stretch:isar-image-base
>> multiconfig:qemuamd64-jessie:isar-image-base
>> multiconfig:qemuamd64-stretch:isar-image-base
>> multiconfig:rpi-jessie:isar-image-base
>>
>> Just for sure stated build in CI server:
>> http://isar-build.org:8080/job/isar_asmirnov_devel/8/console
>>
>> Also failed. Could I dropped this patch from the series?
> 
> Not before you can explain why things fail and why that patch causes
> this. We need to understand the problem. Let me reproduce...

Yeah, I get some bug as well, but a different & well-known one:

dpkg: error: dpkg status database is locked by another process

So, I'm afraid we will find more of these sporadic breakage, now that
parallelization works better....

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[Jan Kiszka]

On 2018-02-14 14:29, Jan Kiszka wrote:
> On 2018-02-14 14:10, Jan Kiszka wrote:
>> On 2018-02-14 13:57, Alexander Smirnov wrote:
>>> On 02/14/2018 03:41 PM, Jan Kiszka wrote:
>>>> On 2018-02-14 12:33, Alexander Smirnov wrote:
>>>>> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>>>>>> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
>>>>>> changes to recipes lead to rebuilds of dependent recipes, including the
>>>>>> final image. Some extra measures are needed so that those rebuilds work
>>>>>> with clean dirs.
>>>>>>
>>>>>> And if the change detection should not work, e.g. changes to file://
>>>>>> resources are not detected, then a clean or cleanall task is now
>>>>>> available and ensures a proper manual rebuild.
>>>>>>
>>>>>> This massively increases the fun factor when developing Isar projects.
>>>>>
>>>>>
>>>>> Wow, just noticed that with this series bitbake has started to run tasks
>>>>> in parallel.. How it's working? :-)
>>>>>
>>>>>
>>>>> Currently  4 running tasks (54 of 217)  24%
>>>>> |########################################
>>>>>                       |
>>>>> 0: mc:rpi-jessie:buildchroot-1.0-r0 do_build - 101s (pid 3271)
>>>>> 1: mc:qemui386-stretch:buildchroot-1.0-r0 do_build - 101s (pid 3336)
>>>>> 2: mc:qemui386-jessie:buildchroot-1.0-r0 do_build - 100s (pid 3457)
>>>>> 3: mc:qemuarm-wheezy:buildchroot-1.0-r0 do_build - 100s (pid 3524)
>>>>
>>>> Didn't notice that this wasn't the case before (most of my workload had
>>>> linear ordering), but I could imagine that the lacking signatures forced
>>>> bitbake to serialize. Just guessing.
>>>>
>>>
>>> There is still the same build problem as reported yesterday, log is
>>> attached. Dropping patch #3 fixes it.
>>>
>>> Line I used to build:
>>>
>>> $ time bitbake multiconfig:qemuarm-wheezy:isar-image-base
>>> multiconfig:qemuarm-jessie:isar-image-base
>>> multiconfig:qemuarm-stretch:isar-image-base
>>> multiconfig:qemui386-jessie:isar-image-base
>>> multiconfig:qemui386-stretch:isar-image-base
>>> multiconfig:qemuamd64-jessie:isar-image-base
>>> multiconfig:qemuamd64-stretch:isar-image-base
>>> multiconfig:rpi-jessie:isar-image-base
>>>
>>> Just for sure stated build in CI server:
>>> http://isar-build.org:8080/job/isar_asmirnov_devel/8/console
>>>
>>> Also failed. Could I dropped this patch from the series?
>>
>> Not before you can explain why things fail and why that patch causes
>> this. We need to understand the problem. Let me reproduce...
> 
> Yeah, I get some bug as well, but a different & well-known one:
> 
> dpkg: error: dpkg status database is locked by another process
> 
> So, I'm afraid we will find more of these sporadic breakage, now that
> parallelization works better....

BTW, I'm testing with
https://github.com/siemens/isar/commits/jan/staging, i.e. the sum of all
my misdoing.

Good news is that I just reproduced your issue, now digging into what
corrupts the isar-apt repo /wrt libhello-dev. Bad news: I had to restart
5 times due to the contention bug above.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[Jan Kiszka]

On 2018-02-14 14:38, Jan Kiszka wrote:
> On 2018-02-14 14:29, Jan Kiszka wrote:
>> On 2018-02-14 14:10, Jan Kiszka wrote:
>>> On 2018-02-14 13:57, Alexander Smirnov wrote:
>>>> On 02/14/2018 03:41 PM, Jan Kiszka wrote:
>>>>> On 2018-02-14 12:33, Alexander Smirnov wrote:
>>>>>> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>>>>>>> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
>>>>>>> changes to recipes lead to rebuilds of dependent recipes, including the
>>>>>>> final image. Some extra measures are needed so that those rebuilds work
>>>>>>> with clean dirs.
>>>>>>>
>>>>>>> And if the change detection should not work, e.g. changes to file://
>>>>>>> resources are not detected, then a clean or cleanall task is now
>>>>>>> available and ensures a proper manual rebuild.
>>>>>>>
>>>>>>> This massively increases the fun factor when developing Isar projects.
>>>>>>
>>>>>>
>>>>>> Wow, just noticed that with this series bitbake has started to run tasks
>>>>>> in parallel.. How it's working? :-)
>>>>>>
>>>>>>
>>>>>> Currently  4 running tasks (54 of 217)  24%
>>>>>> |########################################
>>>>>>                       |
>>>>>> 0: mc:rpi-jessie:buildchroot-1.0-r0 do_build - 101s (pid 3271)
>>>>>> 1: mc:qemui386-stretch:buildchroot-1.0-r0 do_build - 101s (pid 3336)
>>>>>> 2: mc:qemui386-jessie:buildchroot-1.0-r0 do_build - 100s (pid 3457)
>>>>>> 3: mc:qemuarm-wheezy:buildchroot-1.0-r0 do_build - 100s (pid 3524)
>>>>>
>>>>> Didn't notice that this wasn't the case before (most of my workload had
>>>>> linear ordering), but I could imagine that the lacking signatures forced
>>>>> bitbake to serialize. Just guessing.
>>>>>
>>>>
>>>> There is still the same build problem as reported yesterday, log is
>>>> attached. Dropping patch #3 fixes it.
>>>>
>>>> Line I used to build:
>>>>
>>>> $ time bitbake multiconfig:qemuarm-wheezy:isar-image-base
>>>> multiconfig:qemuarm-jessie:isar-image-base
>>>> multiconfig:qemuarm-stretch:isar-image-base
>>>> multiconfig:qemui386-jessie:isar-image-base
>>>> multiconfig:qemui386-stretch:isar-image-base
>>>> multiconfig:qemuamd64-jessie:isar-image-base
>>>> multiconfig:qemuamd64-stretch:isar-image-base
>>>> multiconfig:rpi-jessie:isar-image-base
>>>>
>>>> Just for sure stated build in CI server:
>>>> http://isar-build.org:8080/job/isar_asmirnov_devel/8/console
>>>>
>>>> Also failed. Could I dropped this patch from the series?
>>>
>>> Not before you can explain why things fail and why that patch causes
>>> this. We need to understand the problem. Let me reproduce...
>>
>> Yeah, I get some bug as well, but a different & well-known one:
>>
>> dpkg: error: dpkg status database is locked by another process
>>
>> So, I'm afraid we will find more of these sporadic breakage, now that
>> parallelization works better....
> 
> BTW, I'm testing with
> https://github.com/siemens/isar/commits/jan/staging, i.e. the sum of all
> my misdoing.
> 
> Good news is that I just reproduced your issue, now digging into what
> corrupts the isar-apt repo /wrt libhello-dev. Bad news: I had to restart
> 5 times due to the contention bug above.

OK, understood this one: repo_clean. The arm task kills the results of
the i386 or vice versa. Need to work on the filtering rules here. Or we
need to split the isar-apt repos per multiconf target...

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[Alexander Smirnov]

On 02/14/2018 04:49 PM, Jan Kiszka wrote:
> On 2018-02-14 14:38, Jan Kiszka wrote:
>> On 2018-02-14 14:29, Jan Kiszka wrote:
>>> On 2018-02-14 14:10, Jan Kiszka wrote:
>>>> On 2018-02-14 13:57, Alexander Smirnov wrote:
>>>>> On 02/14/2018 03:41 PM, Jan Kiszka wrote:
>>>>>> On 2018-02-14 12:33, Alexander Smirnov wrote:
>>>>>>> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>>>>>>>> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
>>>>>>>> changes to recipes lead to rebuilds of dependent recipes, including the
>>>>>>>> final image. Some extra measures are needed so that those rebuilds work
>>>>>>>> with clean dirs.
>>>>>>>>
>>>>>>>> And if the change detection should not work, e.g. changes to file://
>>>>>>>> resources are not detected, then a clean or cleanall task is now
>>>>>>>> available and ensures a proper manual rebuild.
>>>>>>>>
>>>>>>>> This massively increases the fun factor when developing Isar projects.
>>>>>>>
>>>>>>>
>>>>>>> Wow, just noticed that with this series bitbake has started to run tasks
>>>>>>> in parallel.. How it's working? :-)
>>>>>>>
>>>>>>>
>>>>>>> Currently  4 running tasks (54 of 217)  24%
>>>>>>> |########################################
>>>>>>>                        |
>>>>>>> 0: mc:rpi-jessie:buildchroot-1.0-r0 do_build - 101s (pid 3271)
>>>>>>> 1: mc:qemui386-stretch:buildchroot-1.0-r0 do_build - 101s (pid 3336)
>>>>>>> 2: mc:qemui386-jessie:buildchroot-1.0-r0 do_build - 100s (pid 3457)
>>>>>>> 3: mc:qemuarm-wheezy:buildchroot-1.0-r0 do_build - 100s (pid 3524)
>>>>>>
>>>>>> Didn't notice that this wasn't the case before (most of my workload had
>>>>>> linear ordering), but I could imagine that the lacking signatures forced
>>>>>> bitbake to serialize. Just guessing.
>>>>>>
>>>>>
>>>>> There is still the same build problem as reported yesterday, log is
>>>>> attached. Dropping patch #3 fixes it.
>>>>>
>>>>> Line I used to build:
>>>>>
>>>>> $ time bitbake multiconfig:qemuarm-wheezy:isar-image-base
>>>>> multiconfig:qemuarm-jessie:isar-image-base
>>>>> multiconfig:qemuarm-stretch:isar-image-base
>>>>> multiconfig:qemui386-jessie:isar-image-base
>>>>> multiconfig:qemui386-stretch:isar-image-base
>>>>> multiconfig:qemuamd64-jessie:isar-image-base
>>>>> multiconfig:qemuamd64-stretch:isar-image-base
>>>>> multiconfig:rpi-jessie:isar-image-base
>>>>>
>>>>> Just for sure stated build in CI server:
>>>>> http://isar-build.org:8080/job/isar_asmirnov_devel/8/console
>>>>>
>>>>> Also failed. Could I dropped this patch from the series?
>>>>
>>>> Not before you can explain why things fail and why that patch causes
>>>> this. We need to understand the problem. Let me reproduce...
>>>
>>> Yeah, I get some bug as well, but a different & well-known one:
>>>
>>> dpkg: error: dpkg status database is locked by another process
>>>
>>> So, I'm afraid we will find more of these sporadic breakage, now that
>>> parallelization works better....
>>
>> BTW, I'm testing with
>> https://github.com/siemens/isar/commits/jan/staging, i.e. the sum of all
>> my misdoing.
>>
>> Good news is that I just reproduced your issue, now digging into what
>> corrupts the isar-apt repo /wrt libhello-dev. Bad news: I had to restart
>> 5 times due to the contention bug above.
> 
> OK, understood this one: repo_clean. The arm task kills the results of
> the i386 or vice versa. Need to work on the filtering rules here. Or we
> need to split the isar-apt repos per multiconf target...

This how ftp.debian.org is organized, the pool contains all the 
supported architectures for specific package:

http://ftp.de.debian.org/debian/pool/main/liba/libapache-mod-auth-radius/

Alex

Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[Alexander Smirnov]

On 02/14/2018 04:38 PM, Jan Kiszka wrote:
> On 2018-02-14 14:29, Jan Kiszka wrote:
>> On 2018-02-14 14:10, Jan Kiszka wrote:
>>> On 2018-02-14 13:57, Alexander Smirnov wrote:
>>>> On 02/14/2018 03:41 PM, Jan Kiszka wrote:
>>>>> On 2018-02-14 12:33, Alexander Smirnov wrote:
>>>>>> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>>>>>>> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
>>>>>>> changes to recipes lead to rebuilds of dependent recipes, including the
>>>>>>> final image. Some extra measures are needed so that those rebuilds work
>>>>>>> with clean dirs.
>>>>>>>
>>>>>>> And if the change detection should not work, e.g. changes to file://
>>>>>>> resources are not detected, then a clean or cleanall task is now
>>>>>>> available and ensures a proper manual rebuild.
>>>>>>>
>>>>>>> This massively increases the fun factor when developing Isar projects.
>>>>>>
>>>>>>
>>>>>> Wow, just noticed that with this series bitbake has started to run tasks
>>>>>> in parallel.. How it's working? :-)
>>>>>>
>>>>>>
>>>>>> Currently  4 running tasks (54 of 217)  24%
>>>>>> |########################################
>>>>>>                        |
>>>>>> 0: mc:rpi-jessie:buildchroot-1.0-r0 do_build - 101s (pid 3271)
>>>>>> 1: mc:qemui386-stretch:buildchroot-1.0-r0 do_build - 101s (pid 3336)
>>>>>> 2: mc:qemui386-jessie:buildchroot-1.0-r0 do_build - 100s (pid 3457)
>>>>>> 3: mc:qemuarm-wheezy:buildchroot-1.0-r0 do_build - 100s (pid 3524)
>>>>>
>>>>> Didn't notice that this wasn't the case before (most of my workload had
>>>>> linear ordering), but I could imagine that the lacking signatures forced
>>>>> bitbake to serialize. Just guessing.
>>>>>
>>>>
>>>> There is still the same build problem as reported yesterday, log is
>>>> attached. Dropping patch #3 fixes it.
>>>>
>>>> Line I used to build:
>>>>
>>>> $ time bitbake multiconfig:qemuarm-wheezy:isar-image-base
>>>> multiconfig:qemuarm-jessie:isar-image-base
>>>> multiconfig:qemuarm-stretch:isar-image-base
>>>> multiconfig:qemui386-jessie:isar-image-base
>>>> multiconfig:qemui386-stretch:isar-image-base
>>>> multiconfig:qemuamd64-jessie:isar-image-base
>>>> multiconfig:qemuamd64-stretch:isar-image-base
>>>> multiconfig:rpi-jessie:isar-image-base
>>>>
>>>> Just for sure stated build in CI server:
>>>> http://isar-build.org:8080/job/isar_asmirnov_devel/8/console
>>>>
>>>> Also failed. Could I dropped this patch from the series?
>>>
>>> Not before you can explain why things fail and why that patch causes
>>> this. We need to understand the problem. Let me reproduce...
>>
>> Yeah, I get some bug as well, but a different & well-known one:
>>
>> dpkg: error: dpkg status database is locked by another process
>>
>> So, I'm afraid we will find more of these sporadic breakage, now that
>> parallelization works better....
> 
> BTW, I'm testing with
> https://github.com/siemens/isar/commits/jan/staging, i.e. the sum of all
> my misdoing.
> 
> Good news is that I just reproduced your issue, now digging into what
> corrupts the isar-apt repo /wrt libhello-dev. Bad news: I had to restart
> 5 times due to the contention bug above.
> 

I think that the issue is in repo_clean function. "isar_apt" splits 
databases by ${DISTRO}:

builder@zbook:~/isar/build$ ls tmp/deploy/apt/ -1
debian-jessie
debian-stretch
debian-wheezy
raspbian-jessie

And each distro could contains packages of whatever arch.

I suppose that clean_repo() as it's called, remove all the architectures 
for specified package name. So building for ARM erases content for AMD 
and i386. Looking into the manual, -A option should be passed to reprepro.

Alex

Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[Jan Kiszka]

On 2018-02-14 15:03, Alexander Smirnov wrote:
> 
> 
> On 02/14/2018 04:38 PM, Jan Kiszka wrote:
>> On 2018-02-14 14:29, Jan Kiszka wrote:
>>> On 2018-02-14 14:10, Jan Kiszka wrote:
>>>> On 2018-02-14 13:57, Alexander Smirnov wrote:
>>>>> On 02/14/2018 03:41 PM, Jan Kiszka wrote:
>>>>>> On 2018-02-14 12:33, Alexander Smirnov wrote:
>>>>>>> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>>>>>>>> Yeah, finally Yocto/OE-like usability: This ensures for many
>>>>>>>> cases that
>>>>>>>> changes to recipes lead to rebuilds of dependent recipes,
>>>>>>>> including the
>>>>>>>> final image. Some extra measures are needed so that those
>>>>>>>> rebuilds work
>>>>>>>> with clean dirs.
>>>>>>>>
>>>>>>>> And if the change detection should not work, e.g. changes to
>>>>>>>> file://
>>>>>>>> resources are not detected, then a clean or cleanall task is now
>>>>>>>> available and ensures a proper manual rebuild.
>>>>>>>>
>>>>>>>> This massively increases the fun factor when developing Isar
>>>>>>>> projects.
>>>>>>>
>>>>>>>
>>>>>>> Wow, just noticed that with this series bitbake has started to
>>>>>>> run tasks
>>>>>>> in parallel.. How it's working? :-)
>>>>>>>
>>>>>>>
>>>>>>> Currently  4 running tasks (54 of 217)  24%
>>>>>>> |########################################
>>>>>>>                        |
>>>>>>> 0: mc:rpi-jessie:buildchroot-1.0-r0 do_build - 101s (pid 3271)
>>>>>>> 1: mc:qemui386-stretch:buildchroot-1.0-r0 do_build - 101s (pid 3336)
>>>>>>> 2: mc:qemui386-jessie:buildchroot-1.0-r0 do_build - 100s (pid 3457)
>>>>>>> 3: mc:qemuarm-wheezy:buildchroot-1.0-r0 do_build - 100s (pid 3524)
>>>>>>
>>>>>> Didn't notice that this wasn't the case before (most of my
>>>>>> workload had
>>>>>> linear ordering), but I could imagine that the lacking signatures
>>>>>> forced
>>>>>> bitbake to serialize. Just guessing.
>>>>>>
>>>>>
>>>>> There is still the same build problem as reported yesterday, log is
>>>>> attached. Dropping patch #3 fixes it.
>>>>>
>>>>> Line I used to build:
>>>>>
>>>>> $ time bitbake multiconfig:qemuarm-wheezy:isar-image-base
>>>>> multiconfig:qemuarm-jessie:isar-image-base
>>>>> multiconfig:qemuarm-stretch:isar-image-base
>>>>> multiconfig:qemui386-jessie:isar-image-base
>>>>> multiconfig:qemui386-stretch:isar-image-base
>>>>> multiconfig:qemuamd64-jessie:isar-image-base
>>>>> multiconfig:qemuamd64-stretch:isar-image-base
>>>>> multiconfig:rpi-jessie:isar-image-base
>>>>>
>>>>> Just for sure stated build in CI server:
>>>>> http://isar-build.org:8080/job/isar_asmirnov_devel/8/console
>>>>>
>>>>> Also failed. Could I dropped this patch from the series?
>>>>
>>>> Not before you can explain why things fail and why that patch causes
>>>> this. We need to understand the problem. Let me reproduce...
>>>
>>> Yeah, I get some bug as well, but a different & well-known one:
>>>
>>> dpkg: error: dpkg status database is locked by another process
>>>
>>> So, I'm afraid we will find more of these sporadic breakage, now that
>>> parallelization works better....
>>
>> BTW, I'm testing with
>> https://github.com/siemens/isar/commits/jan/staging, i.e. the sum of all
>> my misdoing.
>>
>> Good news is that I just reproduced your issue, now digging into what
>> corrupts the isar-apt repo /wrt libhello-dev. Bad news: I had to restart
>> 5 times due to the contention bug above.
>>
> 
> I think that the issue is in repo_clean function. "isar_apt" splits
> databases by ${DISTRO}:
> 
> builder@zbook:~/isar/build$ ls tmp/deploy/apt/ -1
> debian-jessie
> debian-stretch
> debian-wheezy
> raspbian-jessie
> 
> And each distro could contains packages of whatever arch.
> 
> I suppose that clean_repo() as it's called, remove all the architectures
> for specified package name. So building for ARM erases content for AMD
> and i386. Looking into the manual, -A option should be passed to reprepro.

Yes, exactly. I will try the -A next.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[Alexander Smirnov]

On 02/13/2018 11:05 PM, Jan Kiszka wrote:
> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
> changes to recipes lead to rebuilds of dependent recipes, including the
> final image. Some extra measures are needed so that those rebuilds work
> with clean dirs.
> 
> And if the change detection should not work, e.g. changes to file://
> resources are not detected, then a clean or cleanall task is now
> available and ensures a proper manual rebuild.
> 
> This massively increases the fun factor when developing Isar projects.
> 
> Changes in v2:
> - include patch 1 (dpkg-raw: Install hooks automatically) to clarify
>    that this is a dependency
> - add patch 5 (Move buildchroot mounts into dpkg-base class) to fix
>    broken mounts when doing rebuilds
> - add patch 9 (buildchroot: Clean up rootfs prior to populating it),
>    analogously to patch 7 and 8

Applied to next, thanks!

Alex

Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[Jan Kiszka]

On 2018-02-14 18:51, Alexander Smirnov wrote:
> 
> 
> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
>> changes to recipes lead to rebuilds of dependent recipes, including the
>> final image. Some extra measures are needed so that those rebuilds work
>> with clean dirs.
>>
>> And if the change detection should not work, e.g. changes to file://
>> resources are not detected, then a clean or cleanall task is now
>> available and ensures a proper manual rebuild.
>>
>> This massively increases the fun factor when developing Isar projects.
>>
>> Changes in v2:
>> - include patch 1 (dpkg-raw: Install hooks automatically) to clarify
>>    that this is a dependency
>> - add patch 5 (Move buildchroot mounts into dpkg-base class) to fix
>>    broken mounts when doing rebuilds
>> - add patch 9 (buildchroot: Clean up rootfs prior to populating it),
>>    analogously to patch 7 and 8
> 
> Applied to next, thanks!

Oh... Does it pass your CI?

I just started a build that tests flock around the dependency
installations so that it can complete all targets in one run here.
Should be simple (if it works...), will send the patch on top ASAP.

Jan

Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[Alexander Smirnov]

On 02/14/2018 08:54 PM, Jan Kiszka wrote:
> On 2018-02-14 18:51, Alexander Smirnov wrote:
>>
>>
>> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>>> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
>>> changes to recipes lead to rebuilds of dependent recipes, including the
>>> final image. Some extra measures are needed so that those rebuilds work
>>> with clean dirs.
>>>
>>> And if the change detection should not work, e.g. changes to file://
>>> resources are not detected, then a clean or cleanall task is now
>>> available and ensures a proper manual rebuild.
>>>
>>> This massively increases the fun factor when developing Isar projects.
>>>
>>> Changes in v2:
>>> - include patch 1 (dpkg-raw: Install hooks automatically) to clarify
>>>     that this is a dependency
>>> - add patch 5 (Move buildchroot mounts into dpkg-base class) to fix
>>>     broken mounts when doing rebuilds
>>> - add patch 9 (buildchroot: Clean up rootfs prior to populating it),
>>>     analogously to patch 7 and 8
>>
>> Applied to next, thanks!
> 
> Oh... Does it pass your CI?
> 
> I just started a build that tests flock around the dependency
> installations so that it can complete all targets in one run here.
> Should be simple (if it works...), will send the patch on top ASAP.
> 

Local build passed :-)

Started CI one: http://isar-build.org:8080/job/isar_next/103/console

Will try local one again, possibly I missed something.

Alex

Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[Jan Kiszka]

On 2018-02-14 18:57, Alexander Smirnov wrote:
> 
> 
> On 02/14/2018 08:54 PM, Jan Kiszka wrote:
>> On 2018-02-14 18:51, Alexander Smirnov wrote:
>>>
>>>
>>> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>>>> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
>>>> changes to recipes lead to rebuilds of dependent recipes, including the
>>>> final image. Some extra measures are needed so that those rebuilds work
>>>> with clean dirs.
>>>>
>>>> And if the change detection should not work, e.g. changes to file://
>>>> resources are not detected, then a clean or cleanall task is now
>>>> available and ensures a proper manual rebuild.
>>>>
>>>> This massively increases the fun factor when developing Isar projects.
>>>>
>>>> Changes in v2:
>>>> - include patch 1 (dpkg-raw: Install hooks automatically) to clarify
>>>>     that this is a dependency
>>>> - add patch 5 (Move buildchroot mounts into dpkg-base class) to fix
>>>>     broken mounts when doing rebuilds
>>>> - add patch 9 (buildchroot: Clean up rootfs prior to populating it),
>>>>     analogously to patch 7 and 8
>>>
>>> Applied to next, thanks!
>>
>> Oh... Does it pass your CI?
>>
>> I just started a build that tests flock around the dependency
>> installations so that it can complete all targets in one run here.
>> Should be simple (if it works...), will send the patch on top ASAP.
>>
> 
> Local build passed :-)
> 
> Started CI one: http://isar-build.org:8080/job/isar_next/103/console

BTW, is there a way for me to extract also the bitbake locks from those
builds? Generally, you are lost without them.

> 
> Will try local one again, possibly I missed something.

How many cores do you have? I've also lifted BB_NUMBER_THREADS to all
available ones, but already 4 gave enough trouble.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[Alexander Smirnov]

On 02/14/2018 09:02 PM, Jan Kiszka wrote:
> On 2018-02-14 18:57, Alexander Smirnov wrote:
>>
>>
>> On 02/14/2018 08:54 PM, Jan Kiszka wrote:
>>> On 2018-02-14 18:51, Alexander Smirnov wrote:
>>>>
>>>>
>>>> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>>>>> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
>>>>> changes to recipes lead to rebuilds of dependent recipes, including the
>>>>> final image. Some extra measures are needed so that those rebuilds work
>>>>> with clean dirs.
>>>>>
>>>>> And if the change detection should not work, e.g. changes to file://
>>>>> resources are not detected, then a clean or cleanall task is now
>>>>> available and ensures a proper manual rebuild.
>>>>>
>>>>> This massively increases the fun factor when developing Isar projects.
>>>>>
>>>>> Changes in v2:
>>>>> - include patch 1 (dpkg-raw: Install hooks automatically) to clarify
>>>>>      that this is a dependency
>>>>> - add patch 5 (Move buildchroot mounts into dpkg-base class) to fix
>>>>>      broken mounts when doing rebuilds
>>>>> - add patch 9 (buildchroot: Clean up rootfs prior to populating it),
>>>>>      analogously to patch 7 and 8
>>>>
>>>> Applied to next, thanks!
>>>
>>> Oh... Does it pass your CI?
>>>
>>> I just started a build that tests flock around the dependency
>>> installations so that it can complete all targets in one run here.
>>> Should be simple (if it works...), will send the patch on top ASAP.
>>>
>>
>> Local build passed :-)
>>
>> Started CI one: http://isar-build.org:8080/job/isar_next/103/console
> 
> BTW, is there a way for me to extract also the bitbake locks from those
> builds? Generally, you are lost without them.

Jenkins provides possibility to browse build folders, but for now they 
are not enabled in UI. This task is in my todo-list but with low 
priority. So if anybody else except me is going to use this server, I 
could handle this soon.

> 
>>
>> Will try local one again, possibly I missed something.
> 
> How many cores do you have? I've also lifted BB_NUMBER_THREADS to all
> available ones, but already 4 gave enough trouble.

I use only the default settings, so 4 threads. The second build is also 
passed.

Alex