Re: [PATCH] isar-bootstrap: Fix and cleanup bind mounting

[Jan Kiszka <jan.kiszka@siemens.com>]

On 04.12.18 17:59, Maxim Yu. Osipov wrote:
> Hi Jan, Cedric,
> 
> Another question:
> 
> Which debian/kernel do you use inside your VM/docker?

Container is kasproject/kas-isar:latest here, kernel is $random-host.

> 
> Is it also stretch?

Yes.

> 
> The problem is reproducible at the the same point on stretch systems (with 
> kernel SMP Debian 4.9.110-3+deb9u6 (2018-10-08) x86_64 GNU/Linux)
> 
> when commands are launched by hand:

What was your Isar baseline for that? I tested with this branch:

https://github.com/siemens/isar/commit/03e394b2aa58b5b1404fde76881774baf4a541bc

> 
> Command 1)
> bitbake  -c cache_base_repo multiconfig:qemuarm-stretch:isar-image-base 
> multiconfig:qemuarm64-stretch:isar-image-base multiconfig:qemuamd64-
> stretch:isar-image-base

Did you check if everything was unmounted at this point already?

> 
> Command 2)
> sudo rm -rf tmp
> 
> Command 3)
> sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= 
> "1"/g' conf/local.conf
> 
> No problems detected at this point - the same mounts etc.
> 
> The next command hungs on the last task (according strace bitbake tries to 
> unmount /sys /dev)

And who is holding back that mounts (lsof)? How does pstree -apl look like?

BTW, can you recover the build system for this stage? As I said, one of the 
symptoms or side-effect is removal of device nodes on those side when /dev is 
mounted at the wrong time.

Jan

> 
> Command 4)
> bitbake multiconfig:qemuarm-stretch:isar-image-base 
> multiconfig:qemuarm64-stretch:isar-image-base 
> multiconfig:qemuamd64-stretch:isar-image-base
> 
> Maxim.
> 
> On 12/4/18 6:45 PM, Hombourger, Cedric wrote:
>> Good catch & analysis Jan!
>> In our CI, our build script is checking for any mounts relative to the current 
>> directory before purging them
>>
>> -----Original Message-----
>> From: Jan Kiszka [mailto:jan.kiszka@siemens.com]
>> Sent: Tuesday, December 4, 2018 6:43 PM
>> To: Maxim Yu. Osipov <mosipov@ilbers.de>
>> Cc: Henning Schild <henning.schild@siemens.com>; isar-users 
>> <isar-users@googlegroups.com>; Hombourger, Cedric <Cedric_Hombourger@mentor.com>
>> Subject: Re: [PATCH] isar-bootstrap: Fix and cleanup bind mounting
>>
>> On 04.12.18 15:24, [ext] Jan Kiszka wrote:
>>> On 04.12.18 11:49, Maxim Yu. Osipov wrote:
>>>> On 12/3/18 3:59 PM, Jan Kiszka wrote:
>>>>> On 30.11.18 10:20, Maxim Yu. Osipov wrote:
>>>>>> Hi Jan,
>>>>>>
>>>>>> I've just tried this patch (on the 'next' with reverted patch
>>>>>> d40a9ac0) and ran "fast" CI
>>>>>>
>>>>>> isar$mount | wc -l
>>>>>> 34
>>>>>>
>>>>>> isar$./scripts/ci_build.sh -q -f
>>>>>>
>>>>>> CI script hung on CI stage when dpkg-base is modified causing
>>>>>> rebuilding recipes based on dpkg-base.
>>>>>>
>>>>>> The mount reports less (!) mount points than before launching the script.
>>>>>>
>>>>>> mount | wc -l
>>>>>> 31
>>>>>
>>>>> Any news on what's different on your side? Where exactly does your
>>>>> build hang? Was your CI environment in a clean state when running
>>>>> this test? Before the comment lots of things leaked.
>>>>
>>>>
>>>> On my stretch laptop (i7-6820HQ CPU @ 2.70GHz (8 cores) with SSD)
>>>> the reported problem is reproducible (I rerun 'ci_build.sh -q -f'
>>>> several times in clean state) it hung and with the less mount points
>>>> (the mount points before and after running are attached).
>>>>
>>>> The strange thing that I observe two bitbake processes:
>>>>
>>>> myo      26373  0.0  0.3 153116 29732 pts/0    Sl+  12:31   0:01
>>>> python3 /home/myo/work/isar/src/trunk/isar/bitbake/bin/bitbake
>>>> multiconfig:qemuarm-stretch:isar-image-base
>>>> multiconfig:qemuarm64-stretch:isar-image-base
>>>> multiconfig:qemuamd64-stretch:isar-image-base
>>>>
>>>> myo      26379  2.5  0.6 328476 50028 ?        Sl   12:31   0:40
>>>> python3 /home/myo/work/isar/src/trunk/isar/bitbake/bin/bitbake
>>>> multiconfig:qemuarm-stretch:isar-image-base
>>>> multiconfig:qemuarm64-stretch:isar-image-base
>>>> multiconfig:qemuamd64-stretch:isar-image-base
>>>>
>>>
>>> We run multiple bitbake sessions after each other. Maybe the first one never
>>> terminates (get stuck), and that is also why the rm after the first session
>>> fails. You need to stop the build there and analyse what is keeping the mount
>>> points busy.
>>
>> Wait... If I terminate a build from inside the container (i.e. "natively") and
>> then quickly try to delete the build artifacts, I can trigger that infamous
>> empty /dev bug - on the host. That has always been the problem, and that is one
>> reason why we encapsulate things into containers.
>>
>> The reason for this is that bitbake's cooker waits for the last sub-process to
>> finish before it calls the cleanup hook that does all the unmounting. If you
>> delete something before that, you step into the mount point and purge its
>> content. That /may/ be the issue here as well as we run rm directly after 
>> bitbake.
>>
>> IOW: Possibly just a known limitation of current Isar design /wrt to unmounting
>> in isar_handler() that now surfaces in CI. I would not be surprised you can
>> resolve that by waiting for the last cooker instance to terminate before
>> deleting tmp.
>>
>> Jan
>>
> 
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux