Re: [PATCH 4/6] Enable proper rebuilds on dependency changes

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2018-02-13 19:08, Alexander Smirnov wrote:
> On 02/13/2018 08:57 PM, Jan Kiszka wrote:
>> On 2018-02-13 17:21, Jan Kiszka wrote:
>>> On 2018-02-13 14:03, Alexander Smirnov wrote:
>>>> On 02/12/2018 10:59 AM, Jan Kiszka wrote:
>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>
>>>>> Install a basichash as signature handler and set the stamp policy to
>>>>> full - and suddenly we get proper rebuilds of the image and all
>>>>> affected
>>>>> packages when some recipe down the dependency chain changed!
>>>>>
>>>>> We are using the legacy bitbake mechanism here as we do not have
>>>>> setscene machinery like OE. Still good enough for us.
>>>>>
>>>>
>>>> BTW: have you tried this? Two years ago I tried it in Yocto and it
>>>> didn't work properly, not all the dependents were rebuilt. If it works,
>>>> it's a very good feature!
>>>
>>> Yes, I'm using it since this weekend for both Isar as well as
>>> jailhouse-images development, and it helped a lot already. The only
>>> limitation I found, but I do not remember right now if that isn't
>>> inherent to bitbake, is that it does not detect changes in files that
>>> the recipes carries in its SRC_URI (file://...). Then you need "-c
>>> clean" for the affected target - which now works as well.
>>>
>>
>> Found another issue, which worked surprisingly well so far despite being
>> fairly broken: do_setup_mounts is not re-run on rebuilds.
> 
> Investigating exactly this issue. It occures for clean builds also. My
> guess is patch #3, currently I'm building with only 2 first applied.
> With 3rd one applied build failed.
> 

That's not the right approach, the concept of the setup_mounts task is
broken.

I'm still testing, but it looks like as if this massive simplification
can also fix the rebuild issue:

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index a45fbce..aa46f45 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -2,7 +2,7 @@
 # Copyright (C) 2017 Siemens AG
 
 # Add dependency from buildchroot creation
-do_build[depends] = "buildchroot:do_setup_mounts"
+do_build[depends] = "buildchroot:do_build"
 
 # Add dependency between Isar recipes
 DEPENDS ?= ""
@@ -25,6 +25,14 @@ do_build() {
     mkdir -p ${BUILDROOT}
     sudo mount --bind ${WORKDIR} ${BUILDROOT}
 
+    # These tests may race with parallel builds, so ignore errors.
+    grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts &&
+        sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt || true
+    grep -q ${BUILDCHROOT_DIR}/dev /proc/mounts &&
+        sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev || true
+    grep -q ${BUILDCHROOT_DIR}/proc /proc/mounts &&
+        sudo mount -t proc none ${BUILDCHROOT_DIR}/proc || true
+
     dpkg_runbuild
 
     sudo umount ${BUILDROOT} 2>/dev/null || true
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 570c0ad..cd91228 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -60,7 +60,9 @@ do_build() {
         -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
            "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
 
-    do_setup_mounts
+    sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
+    sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
+    sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
 
     # Create root filesystem
     sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
@@ -70,30 +72,4 @@ do_build() {
 
     # Configure root filesystem
     sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
-
-    do_cleanup_mounts
-}
-
-# Invalidate stamp for do_setup_mounts before each build start.
-# This will guarantee that this function will be executed once
-# per build.
-python __anonymous() {
-    stamp = d.getVar("STAMP") + ".do_setup_mounts." + d.getVarFlag("do_setup_mounts", 'stamp-extra-info')
-    os.remove(stamp) if os.path.exists(stamp) else None
-}
-
-do_setup_mounts[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-
-do_setup_mounts() {
-    sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
-    sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
-    sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
-}
-
-addtask setup_mounts after do_build
-
-do_cleanup_mounts() {
-    sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true
-    sudo umount ${BUILDCHROOT_DIR}/dev 2>/dev/null || true
-    sudo umount ${BUILDCHROOT_DIR}/proc 2>/dev/null || true
 }


Jan