Re: [PATCH 4/6] Enable proper rebuilds on dependency changes

[Alexander Smirnov <asmirnov@ilbers.de>]

On 02/13/2018 09:44 PM, Jan Kiszka wrote:
> On 2018-02-13 19:08, Alexander Smirnov wrote:
>> On 02/13/2018 08:57 PM, Jan Kiszka wrote:
>>> On 2018-02-13 17:21, Jan Kiszka wrote:
>>>> On 2018-02-13 14:03, Alexander Smirnov wrote:
>>>>> On 02/12/2018 10:59 AM, Jan Kiszka wrote:
>>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>>
>>>>>> Install a basichash as signature handler and set the stamp policy to
>>>>>> full - and suddenly we get proper rebuilds of the image and all
>>>>>> affected
>>>>>> packages when some recipe down the dependency chain changed!
>>>>>>
>>>>>> We are using the legacy bitbake mechanism here as we do not have
>>>>>> setscene machinery like OE. Still good enough for us.
>>>>>>
>>>>>
>>>>> BTW: have you tried this? Two years ago I tried it in Yocto and it
>>>>> didn't work properly, not all the dependents were rebuilt. If it works,
>>>>> it's a very good feature!
>>>>
>>>> Yes, I'm using it since this weekend for both Isar as well as
>>>> jailhouse-images development, and it helped a lot already. The only
>>>> limitation I found, but I do not remember right now if that isn't
>>>> inherent to bitbake, is that it does not detect changes in files that
>>>> the recipes carries in its SRC_URI (file://...). Then you need "-c
>>>> clean" for the affected target - which now works as well.
>>>>
>>>
>>> Found another issue, which worked surprisingly well so far despite being
>>> fairly broken: do_setup_mounts is not re-run on rebuilds.
>>
>> Investigating exactly this issue. It occures for clean builds also. My
>> guess is patch #3, currently I'm building with only 2 first applied.
>> With 3rd one applied build failed.
>>
> 
> That's not the right approach, the concept of the setup_mounts task is
> broken.

Why you think it's broken? Just tested, it works good:

--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -88,6 +88,8 @@ do_setup_mounts() {
      sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} 
${BUILDCHROOT_DIR}/isar-apt
      sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs 
${BUILDCHROOT_DIR}/dev
      sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
+
+    echo XXX > /tmp/test
  }

Applying this patch I have /tmp/test on re-run.

> 
> I'm still testing, but it looks like as if this massive simplification
> can also fix the rebuild issue:

The issue is with the patch#2, dropping it fixes the build. Probably the 
issue is that do_clean has no before/after specifier in the pipeline. 
Still testing.

Alex

> 
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index a45fbce..aa46f45 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -2,7 +2,7 @@
>   # Copyright (C) 2017 Siemens AG
>   
>   # Add dependency from buildchroot creation
> -do_build[depends] = "buildchroot:do_setup_mounts"
> +do_build[depends] = "buildchroot:do_build"
>   
>   # Add dependency between Isar recipes
>   DEPENDS ?= ""
> @@ -25,6 +25,14 @@ do_build() {
>       mkdir -p ${BUILDROOT}
>       sudo mount --bind ${WORKDIR} ${BUILDROOT}
>   
> +    # These tests may race with parallel builds, so ignore errors.
> +    grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts &&
> +        sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt || true
> +    grep -q ${BUILDCHROOT_DIR}/dev /proc/mounts &&
> +        sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev || true
> +    grep -q ${BUILDCHROOT_DIR}/proc /proc/mounts &&
> +        sudo mount -t proc none ${BUILDCHROOT_DIR}/proc || true
> +
>       dpkg_runbuild
>   
>       sudo umount ${BUILDROOT} 2>/dev/null || true
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
> index 570c0ad..cd91228 100644
> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
> @@ -60,7 +60,9 @@ do_build() {
>           -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
>              "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
>   
> -    do_setup_mounts
> +    sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
> +    sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
> +    sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
>   
>       # Create root filesystem
>       sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
> @@ -70,30 +72,4 @@ do_build() {
>   
>       # Configure root filesystem
>       sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
> -
> -    do_cleanup_mounts
> -}
> -
> -# Invalidate stamp for do_setup_mounts before each build start.
> -# This will guarantee that this function will be executed once
> -# per build.
> -python __anonymous() {
> -    stamp = d.getVar("STAMP") + ".do_setup_mounts." + d.getVarFlag("do_setup_mounts", 'stamp-extra-info')
> -    os.remove(stamp) if os.path.exists(stamp) else None
> -}
> -
> -do_setup_mounts[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> -
> -do_setup_mounts() {
> -    sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
> -    sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
> -    sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
> -}
> -
> -addtask setup_mounts after do_build
> -
> -do_cleanup_mounts() {
> -    sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true
> -    sudo umount ${BUILDCHROOT_DIR}/dev 2>/dev/null || true
> -    sudo umount ${BUILDCHROOT_DIR}/proc 2>/dev/null || true
>   }
> 
> 
> Jan
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov