From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6629320495720300544 X-Received: by 2002:a2e:8112:: with SMTP id d18-v6mr1985653ljg.10.1543942753385; Tue, 04 Dec 2018 08:59:13 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:5d9d:: with SMTP id v29-v6ls2409108lje.12.gmail; Tue, 04 Dec 2018 08:59:12 -0800 (PST) X-Google-Smtp-Source: AFSGD/UDJfNpE5iQcxEnhLO3de9edqvOKY8RSHasq9AryIcucxipojwONQ3Fv4OjylG50036urzp X-Received: by 2002:a2e:844d:: with SMTP id u13-v6mr1940601ljh.17.1543942752815; Tue, 04 Dec 2018 08:59:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1543942752; cv=none; d=google.com; s=arc-20160816; b=jGPY0MYedvCfPwY5KqIqi7DWwbDAp5q2mmBxHUFKXGlkUNI5SDhiCkjk7Y64NbtzO4 f7V0mJ68pNu7uj48X6wgcC+4u33zPx1GkRooXeXgvKuyWyH+d2Y/dxGFbz21E48zES4v lmUlfg21sXHWZUGuYEMp6m1Bm+KOsNtvHJlo7rnWREB91KdTiDUHjCrmLjKNoDu2ZYx0 PQKMfxX/JLpwN56/4/Q8pSs32vTNHoQQGkFwJGHhUBS6QXxqUVfM7zJSGKid/SAZQUdk dgEVyMebGQ0RpuYkNtS811G4ym8uav5loLbAOOiqA+xGUUO/6GCBRRZxsntBInRaef8J r6JQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:organization:from:references:cc:to :subject; bh=NE5tCg6IibBkjGE2TZtWWRvICfLyhiFCoDo55ZCJWSQ=; b=rvlqMOrdkOXUgfsjr8c3ikV2ufof9BvJ6bufea7C9l44RIa94GeM1smE0+xde1aTI3 Tn7wSlenqnQbgN4hW8jFDsBuao/pIK2cIFCabwZ34CTOWzYWlVbjL+u7mwuU9k5HR+4y IhsyZMLHU4GYDzhK1eJR8u9sSIITP8rPGCsPyigBHwxMwKJwlaB8fPtlOQ5V7IhKNYcD xkmzdsPVM0LgQtusLR5TLmRP2uIgm3sWaJTHDY63QBeK2mAOah16YsOvqrOajYYnPzvx fU8NDDdB8Q/8Gvy+7G9eE+lLqNw3YrH2aYMIVXwhhmCu9Fzd0YTPLDR388MeG3KzaAmw a3Cw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of mosipov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=mosipov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id 134si553886lfa.1.2018.12.04.08.59.12 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Dec 2018 08:59:12 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of mosipov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of mosipov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=mosipov@ilbers.de Received: from [192.168.50.180] (nat-ppp-217.71.235.199-satnet-spb.ru [217.71.235.199] (may be forged)) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id wB4Gx9Tf025065 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Tue, 4 Dec 2018 17:59:10 +0100 Subject: Re: [PATCH] isar-bootstrap: Fix and cleanup bind mounting To: "Hombourger, Cedric" , Jan Kiszka Cc: Henning Schild , isar-users References: <6f5714bc-d5f5-c08f-c408-b32bab9169fc@siemens.com> <20181129193929.61a35056@md1za8fc.ad001.siemens.net> <2229f975-0752-ebe3-c165-979e1d5864b2@siemens.com> <61a6a17c-06e0-a13b-591e-3ea8bc09632e@ilbers.de> <405c22d0-48cd-4ea4-4c1b-c78e6c5570ed@siemens.com> <3daa2bd836424990a478b3981f9ca222@svr-ies-mbx-02.mgc.mentorg.com> From: "Maxim Yu. Osipov" Organization: ilbers GmbH Message-ID: Date: Tue, 4 Dec 2018 19:59:03 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.3.0 MIME-Version: 1.0 In-Reply-To: <3daa2bd836424990a478b3981f9ca222@svr-ies-mbx-02.mgc.mentorg.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: EDEi6HS3wSYg Hi Jan, Cedric, Another question: Which debian/kernel do you use inside your VM/docker? Is it also stretch? The problem is reproducible at the the same point on stretch systems (with kernel SMP Debian 4.9.110-3+deb9u6 (2018-10-08) x86_64 GNU/Linux) when commands are launched by hand: Command 1) bitbake -c cache_base_repo multiconfig:qemuarm-stretch:isar-image-base multiconfig:qemuarm64-stretch:isar-image-base multiconfig:qemuamd64- stretch:isar-image-base Command 2) sudo rm -rf tmp Command 3) sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf No problems detected at this point - the same mounts etc. The next command hungs on the last task (according strace bitbake tries to unmount /sys /dev) Command 4) bitbake multiconfig:qemuarm-stretch:isar-image-base multiconfig:qemuarm64-stretch:isar-image-base multiconfig:qemuamd64-stretch:isar-image-base Maxim. On 12/4/18 6:45 PM, Hombourger, Cedric wrote: > Good catch & analysis Jan! > In our CI, our build script is checking for any mounts relative to the current directory before purging them > > -----Original Message----- > From: Jan Kiszka [mailto:jan.kiszka@siemens.com] > Sent: Tuesday, December 4, 2018 6:43 PM > To: Maxim Yu. Osipov > Cc: Henning Schild ; isar-users ; Hombourger, Cedric > Subject: Re: [PATCH] isar-bootstrap: Fix and cleanup bind mounting > > On 04.12.18 15:24, [ext] Jan Kiszka wrote: >> On 04.12.18 11:49, Maxim Yu. Osipov wrote: >>> On 12/3/18 3:59 PM, Jan Kiszka wrote: >>>> On 30.11.18 10:20, Maxim Yu. Osipov wrote: >>>>> Hi Jan, >>>>> >>>>> I've just tried this patch (on the 'next' with reverted patch >>>>> d40a9ac0) and ran "fast" CI >>>>> >>>>> isar$mount | wc -l >>>>> 34 >>>>> >>>>> isar$./scripts/ci_build.sh -q -f >>>>> >>>>> CI script hung on CI stage when dpkg-base is modified causing >>>>> rebuilding recipes based on dpkg-base. >>>>> >>>>> The mount reports less (!) mount points than before launching the script. >>>>> >>>>> mount | wc -l >>>>> 31 >>>> >>>> Any news on what's different on your side? Where exactly does your >>>> build hang? Was your CI environment in a clean state when running >>>> this test? Before the comment lots of things leaked. >>> >>> >>> On my stretch laptop (i7-6820HQ CPU @ 2.70GHz (8 cores) with SSD) >>> the reported problem is reproducible (I rerun 'ci_build.sh -q -f' >>> several times in clean state) it hung and with the less mount points >>> (the mount points before and after running are attached). >>> >>> The strange thing that I observe two bitbake processes: >>> >>> myo      26373  0.0  0.3 153116 29732 pts/0    Sl+  12:31   0:01 >>> python3 /home/myo/work/isar/src/trunk/isar/bitbake/bin/bitbake >>> multiconfig:qemuarm-stretch:isar-image-base >>> multiconfig:qemuarm64-stretch:isar-image-base >>> multiconfig:qemuamd64-stretch:isar-image-base >>> >>> myo      26379  2.5  0.6 328476 50028 ?        Sl   12:31   0:40 >>> python3 /home/myo/work/isar/src/trunk/isar/bitbake/bin/bitbake >>> multiconfig:qemuarm-stretch:isar-image-base >>> multiconfig:qemuarm64-stretch:isar-image-base >>> multiconfig:qemuamd64-stretch:isar-image-base >>> >> >> We run multiple bitbake sessions after each other. Maybe the first one never >> terminates (get stuck), and that is also why the rm after the first session >> fails. You need to stop the build there and analyse what is keeping the mount >> points busy. > > Wait... If I terminate a build from inside the container (i.e. "natively") and > then quickly try to delete the build artifacts, I can trigger that infamous > empty /dev bug - on the host. That has always been the problem, and that is one > reason why we encapsulate things into containers. > > The reason for this is that bitbake's cooker waits for the last sub-process to > finish before it calls the cleanup hook that does all the unmounting. If you > delete something before that, you step into the mount point and purge its > content. That /may/ be the issue here as well as we run rm directly after bitbake. > > IOW: Possibly just a known limitation of current Isar design /wrt to unmounting > in isar_handler() that now surfaces in CI. I would not be surprised you can > resolve that by waiting for the last cooker instance to terminate before > deleting tmp. > > Jan > -- Maxim Osipov ilbers GmbH Maria-Merian-Str. 8 85521 Ottobrunn Germany +49 (151) 6517 6917 mosipov@ilbers.de http://ilbers.de/ Commercial register Munich, HRB 214197 General Manager: Baurzhan Ismagulov