Re: [PATCH] isar-bootstrap: Fix and cleanup bind mounting

[Jan Kiszka <jan.kiszka@siemens.com>]

On 04.12.18 16:42, Jan Kiszka wrote:
> On 04.12.18 15:24, [ext] Jan Kiszka wrote:
>> On 04.12.18 11:49, Maxim Yu. Osipov wrote:
>>> On 12/3/18 3:59 PM, Jan Kiszka wrote:
>>>> On 30.11.18 10:20, Maxim Yu. Osipov wrote:
>>>>> Hi Jan,
>>>>>
>>>>> I've just tried this patch (on the 'next' with reverted patch d40a9ac0) and 
>>>>> ran "fast" CI
>>>>>
>>>>> isar$mount | wc -l
>>>>> 34
>>>>>
>>>>> isar$./scripts/ci_build.sh -q -f
>>>>>
>>>>> CI script hung on CI stage when dpkg-base is modified
>>>>> causing rebuilding recipes based on dpkg-base.
>>>>>
>>>>> The mount reports less (!) mount points than before launching the script.
>>>>>
>>>>> mount | wc -l
>>>>> 31
>>>>
>>>> Any news on what's different on your side? Where exactly does your build 
>>>> hang? Was your CI environment in a clean state when running this test? 
>>>> Before the comment lots of things leaked.
>>>
>>>
>>> On my stretch laptop (i7-6820HQ CPU @ 2.70GHz (8 cores) with SSD)  the 
>>> reported problem is reproducible (I rerun 'ci_build.sh -q -f' several times 
>>> in clean state) it hung and with the less mount points (the mount points 
>>> before and after running are attached).
>>>
>>> The strange thing that I observe two bitbake processes:
>>>
>>> myo      26373  0.0  0.3 153116 29732 pts/0    Sl+  12:31   0:01 python3 
>>> /home/myo/work/isar/src/trunk/isar/bitbake/bin/bitbake 
>>> multiconfig:qemuarm-stretch:isar-image-base 
>>> multiconfig:qemuarm64-stretch:isar-image-base 
>>> multiconfig:qemuamd64-stretch:isar-image-base
>>>
>>> myo      26379  2.5  0.6 328476 50028 ?        Sl   12:31   0:40 python3 
>>> /home/myo/work/isar/src/trunk/isar/bitbake/bin/bitbake 
>>> multiconfig:qemuarm-stretch:isar-image-base 
>>> multiconfig:qemuarm64-stretch:isar-image-base 
>>> multiconfig:qemuamd64-stretch:isar-image-base
>>>
>>
>> We run multiple bitbake sessions after each other. Maybe the first one never 
>> terminates (get stuck), and that is also why the rm after the first session 
>> fails. You need to stop the build there and analyse what is keeping the mount 
>> points busy.
> 
> Wait... If I terminate a build from inside the container (i.e. "natively") and 
> then quickly try to delete the build artifacts, I can trigger that infamous 
> empty /dev bug - on the host. That has always been the problem, and that is one 
> reason why we encapsulate things into containers.
> 
> The reason for this is that bitbake's cooker waits for the last sub-process to 
> finish before it calls the cleanup hook that does all the unmounting. If you 
> delete something before that, you step into the mount point and purge its 
> content. That /may/ be the issue here as well as we run rm directly after bitbake.
> 
> IOW: Possibly just a known limitation of current Isar design /wrt to unmounting 
> in isar_handler() that now surfaces in CI. I would not be surprised you can 
> resolve that by waiting for the last cooker instance to terminate before 
> deleting tmp.
>

...or we are just missing "-l" as umount parameter isar_handler(), like Claudius 
added elsewhere. Though that may not conceptually resolve the race window 
between bitbake terminating and cooker still running the handler.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux