Re: [PATCH] isar-bootstrap: Fix and cleanup bind mounting

[Jan Kiszka <jan.kiszka@siemens.com>]

On 04.12.18 15:24, [ext] Jan Kiszka wrote:
> On 04.12.18 11:49, Maxim Yu. Osipov wrote:
>> On 12/3/18 3:59 PM, Jan Kiszka wrote:
>>> On 30.11.18 10:20, Maxim Yu. Osipov wrote:
>>>> Hi Jan,
>>>>
>>>> I've just tried this patch (on the 'next' with reverted patch d40a9ac0) and 
>>>> ran "fast" CI
>>>>
>>>> isar$mount | wc -l
>>>> 34
>>>>
>>>> isar$./scripts/ci_build.sh -q -f
>>>>
>>>> CI script hung on CI stage when dpkg-base is modified
>>>> causing rebuilding recipes based on dpkg-base.
>>>>
>>>> The mount reports less (!) mount points than before launching the script.
>>>>
>>>> mount | wc -l
>>>> 31
>>>
>>> Any news on what's different on your side? Where exactly does your build 
>>> hang? Was your CI environment in a clean state when running this test? Before 
>>> the comment lots of things leaked.
>>
>>
>> On my stretch laptop (i7-6820HQ CPU @ 2.70GHz (8 cores) with SSD)  the 
>> reported problem is reproducible (I rerun 'ci_build.sh -q -f' several times in 
>> clean state) it hung and with the less mount points (the mount points before 
>> and after running are attached).
>>
>> The strange thing that I observe two bitbake processes:
>>
>> myo      26373  0.0  0.3 153116 29732 pts/0    Sl+  12:31   0:01 python3 
>> /home/myo/work/isar/src/trunk/isar/bitbake/bin/bitbake 
>> multiconfig:qemuarm-stretch:isar-image-base 
>> multiconfig:qemuarm64-stretch:isar-image-base 
>> multiconfig:qemuamd64-stretch:isar-image-base
>>
>> myo      26379  2.5  0.6 328476 50028 ?        Sl   12:31   0:40 python3 
>> /home/myo/work/isar/src/trunk/isar/bitbake/bin/bitbake 
>> multiconfig:qemuarm-stretch:isar-image-base 
>> multiconfig:qemuarm64-stretch:isar-image-base 
>> multiconfig:qemuamd64-stretch:isar-image-base
>>
> 
> We run multiple bitbake sessions after each other. Maybe the first one never 
> terminates (get stuck), and that is also why the rm after the first session 
> fails. You need to stop the build there and analyse what is keeping the mount 
> points busy.

Wait... If I terminate a build from inside the container (i.e. "natively") and 
then quickly try to delete the build artifacts, I can trigger that infamous 
empty /dev bug - on the host. That has always been the problem, and that is one 
reason why we encapsulate things into containers.

The reason for this is that bitbake's cooker waits for the last sub-process to 
finish before it calls the cleanup hook that does all the unmounting. If you 
delete something before that, you step into the mount point and purge its 
content. That /may/ be the issue here as well as we run rm directly after bitbake.

IOW: Possibly just a known limitation of current Isar design /wrt to unmounting 
in isar_handler() that now surfaces in CI. I would not be surprised you can 
resolve that by waiting for the last cooker instance to terminate before 
deleting tmp.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux