Re: [PATCH] isar-bootstrap: Fix and cleanup bind mounting

["Maxim Yu. Osipov" <mosipov@ilbers.de>]

Hi Jan, Cedric,

Another question:

Which debian/kernel do you use inside your VM/docker?

Is it also stretch?

The problem is reproducible at the the same point on stretch systems 
(with kernel SMP Debian 4.9.110-3+deb9u6 (2018-10-08) x86_64 GNU/Linux)

when commands are launched by hand:

Command 1)
bitbake  -c cache_base_repo multiconfig:qemuarm-stretch:isar-image-base 
multiconfig:qemuarm64-stretch:isar-image-base multiconfig:qemuamd64-
stretch:isar-image-base

Command 2)
sudo rm -rf tmp

Command 3)
sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO 
?= "1"/g' conf/local.conf

No problems detected at this point - the same mounts etc.

The next command hungs on the last task (according strace bitbake tries 
to unmount /sys /dev)

Command 4)
bitbake multiconfig:qemuarm-stretch:isar-image-base 
multiconfig:qemuarm64-stretch:isar-image-base 
multiconfig:qemuamd64-stretch:isar-image-base

Maxim.

On 12/4/18 6:45 PM, Hombourger, Cedric wrote:
> Good catch & analysis Jan!
> In our CI, our build script is checking for any mounts relative to the current directory before purging them
> 
> -----Original Message-----
> From: Jan Kiszka [mailto:jan.kiszka@siemens.com]
> Sent: Tuesday, December 4, 2018 6:43 PM
> To: Maxim Yu. Osipov <mosipov@ilbers.de>
> Cc: Henning Schild <henning.schild@siemens.com>; isar-users <isar-users@googlegroups.com>; Hombourger, Cedric <Cedric_Hombourger@mentor.com>
> Subject: Re: [PATCH] isar-bootstrap: Fix and cleanup bind mounting
> 
> On 04.12.18 15:24, [ext] Jan Kiszka wrote:
>> On 04.12.18 11:49, Maxim Yu. Osipov wrote:
>>> On 12/3/18 3:59 PM, Jan Kiszka wrote:
>>>> On 30.11.18 10:20, Maxim Yu. Osipov wrote:
>>>>> Hi Jan,
>>>>>
>>>>> I've just tried this patch (on the 'next' with reverted patch
>>>>> d40a9ac0) and ran "fast" CI
>>>>>
>>>>> isar$mount | wc -l
>>>>> 34
>>>>>
>>>>> isar$./scripts/ci_build.sh -q -f
>>>>>
>>>>> CI script hung on CI stage when dpkg-base is modified causing
>>>>> rebuilding recipes based on dpkg-base.
>>>>>
>>>>> The mount reports less (!) mount points than before launching the script.
>>>>>
>>>>> mount | wc -l
>>>>> 31
>>>>
>>>> Any news on what's different on your side? Where exactly does your
>>>> build hang? Was your CI environment in a clean state when running
>>>> this test? Before the comment lots of things leaked.
>>>
>>>
>>> On my stretch laptop (i7-6820HQ CPU @ 2.70GHz (8 cores) with SSD)
>>> the reported problem is reproducible (I rerun 'ci_build.sh -q -f'
>>> several times in clean state) it hung and with the less mount points
>>> (the mount points before and after running are attached).
>>>
>>> The strange thing that I observe two bitbake processes:
>>>
>>> myo      26373  0.0  0.3 153116 29732 pts/0    Sl+  12:31   0:01
>>> python3 /home/myo/work/isar/src/trunk/isar/bitbake/bin/bitbake
>>> multiconfig:qemuarm-stretch:isar-image-base
>>> multiconfig:qemuarm64-stretch:isar-image-base
>>> multiconfig:qemuamd64-stretch:isar-image-base
>>>
>>> myo      26379  2.5  0.6 328476 50028 ?        Sl   12:31   0:40
>>> python3 /home/myo/work/isar/src/trunk/isar/bitbake/bin/bitbake
>>> multiconfig:qemuarm-stretch:isar-image-base
>>> multiconfig:qemuarm64-stretch:isar-image-base
>>> multiconfig:qemuamd64-stretch:isar-image-base
>>>
>>
>> We run multiple bitbake sessions after each other. Maybe the first one never
>> terminates (get stuck), and that is also why the rm after the first session
>> fails. You need to stop the build there and analyse what is keeping the mount
>> points busy.
> 
> Wait... If I terminate a build from inside the container (i.e. "natively") and
> then quickly try to delete the build artifacts, I can trigger that infamous
> empty /dev bug - on the host. That has always been the problem, and that is one
> reason why we encapsulate things into containers.
> 
> The reason for this is that bitbake's cooker waits for the last sub-process to
> finish before it calls the cleanup hook that does all the unmounting. If you
> delete something before that, you step into the mount point and purge its
> content. That /may/ be the issue here as well as we run rm directly after bitbake.
> 
> IOW: Possibly just a known limitation of current Isar design /wrt to unmounting
> in isar_handler() that now surfaces in CI. I would not be surprised you can
> resolve that by waiting for the last cooker instance to terminate before
> deleting tmp.
> 
> Jan
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov