From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6629320495720300544 X-Received: by 2002:a2e:574f:: with SMTP id r15-v6mr1910556ljd.19.1543938362584; Tue, 04 Dec 2018 07:46:02 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a19:520f:: with SMTP id m15ls1660819lfb.5.gmail; Tue, 04 Dec 2018 07:46:01 -0800 (PST) X-Google-Smtp-Source: AFSGD/XLcO6qjjhqQ7SXLDrugzCtHpljJ8wyBUfT//Cd3CpKM8ErutYlIMFxGGt9LzIZTp41tC65 X-Received: by 2002:a19:f202:: with SMTP id q2mr1649731lfh.8.1543938361924; Tue, 04 Dec 2018 07:46:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1543938361; cv=none; d=google.com; s=arc-20160816; b=Mi7KwaIFvU4u1+UUR1iUvifIO/7QMs69vPjriR05EXgmjcuAFeMxd/QTwAQkvZ4y69 DCsWzuTXby/FwduXpSsw5KbxZs0CKu2HiwSpYzyGhBBVQatkNszJ1pGxs88ARr1e1wsk qj4DTlIwQCN9JMRTa5droQUjLDezg7Rc5Q2ihe0r30OeV88a+IWjag4XtsMDlcdTRWcR TNPgjCF3CrwtJON79ojHQNyrybyIVOsvzaUUq8Ff7kJ/YMtK8N93XH5l71xu59pRRMsV eMZ7SN7bOhvdqcq5Ve5rmescDqq7HdUj6Gz4LhOFn8WagZpGgiLknPx0VyCDxugpqeP+ 8wyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:references:cc:to:from:subject; bh=Z7PQoqRCyPfTud7jEkgZOCGxq4Mx/bS5dqdXzafT1bQ=; b=JMhi/cWKT8AiB/gSJsfnXsiPlpJzycdab4/VD6+6ADTkPV0eIIi4fuIey7NS9kaGd/ GHEeuXWWYWlmVDyXOPweXm24KhOfVOnp/31c2Pb2iYld9vLairUCMSPfFZM8u2drihiL KLVgPBrsxZiYz7WZvUeRgMXdXMqqi6nErubot/Kr7lOC/MwC+G6q583y8NVxNSo0C514 Dk43Mefn+lOHS3bAPJr07XJcedJohiy7bypqdPTw3jA6eGwNh9gsNEz1j9uoKMx3d6X1 sMEDFoohypNrRVSigTpMfHSxZFXD7XHdjilKHx92+j6SeQb2InkehWayIx0hYM0cLBjV Tn9Q== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id w10si521155lfc.5.2018.12.04.07.46.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Dec 2018 07:46:01 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id wB4Fjxnj012217 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 4 Dec 2018 16:45:59 +0100 Received: from [167.87.41.97] ([167.87.41.97]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id wB4FjwIl013559; Tue, 4 Dec 2018 16:45:58 +0100 Subject: Re: [PATCH] isar-bootstrap: Fix and cleanup bind mounting From: Jan Kiszka To: "Maxim Yu. Osipov" , Claudius Heine Cc: Henning Schild , isar-users , Cedric Hombourger References: <6f5714bc-d5f5-c08f-c408-b32bab9169fc@siemens.com> <20181129193929.61a35056@md1za8fc.ad001.siemens.net> <2229f975-0752-ebe3-c165-979e1d5864b2@siemens.com> <61a6a17c-06e0-a13b-591e-3ea8bc09632e@ilbers.de> <405c22d0-48cd-4ea4-4c1b-c78e6c5570ed@siemens.com> Message-ID: <01b4aa88-f62b-cff3-c6fa-04dac500062e@siemens.com> Date: Tue, 4 Dec 2018 16:45:58 +0100 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <405c22d0-48cd-4ea4-4c1b-c78e6c5570ed@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: IVJtPAL0xoJ6 On 04.12.18 16:42, Jan Kiszka wrote: > On 04.12.18 15:24, [ext] Jan Kiszka wrote: >> On 04.12.18 11:49, Maxim Yu. Osipov wrote: >>> On 12/3/18 3:59 PM, Jan Kiszka wrote: >>>> On 30.11.18 10:20, Maxim Yu. Osipov wrote: >>>>> Hi Jan, >>>>> >>>>> I've just tried this patch (on the 'next' with reverted patch d40a9ac0) and >>>>> ran "fast" CI >>>>> >>>>> isar$mount | wc -l >>>>> 34 >>>>> >>>>> isar$./scripts/ci_build.sh -q -f >>>>> >>>>> CI script hung on CI stage when dpkg-base is modified >>>>> causing rebuilding recipes based on dpkg-base. >>>>> >>>>> The mount reports less (!) mount points than before launching the script. >>>>> >>>>> mount | wc -l >>>>> 31 >>>> >>>> Any news on what's different on your side? Where exactly does your build >>>> hang? Was your CI environment in a clean state when running this test? >>>> Before the comment lots of things leaked. >>> >>> >>> On my stretch laptop (i7-6820HQ CPU @ 2.70GHz (8 cores) with SSD)  the >>> reported problem is reproducible (I rerun 'ci_build.sh -q -f' several times >>> in clean state) it hung and with the less mount points (the mount points >>> before and after running are attached). >>> >>> The strange thing that I observe two bitbake processes: >>> >>> myo      26373  0.0  0.3 153116 29732 pts/0    Sl+  12:31   0:01 python3 >>> /home/myo/work/isar/src/trunk/isar/bitbake/bin/bitbake >>> multiconfig:qemuarm-stretch:isar-image-base >>> multiconfig:qemuarm64-stretch:isar-image-base >>> multiconfig:qemuamd64-stretch:isar-image-base >>> >>> myo      26379  2.5  0.6 328476 50028 ?        Sl   12:31   0:40 python3 >>> /home/myo/work/isar/src/trunk/isar/bitbake/bin/bitbake >>> multiconfig:qemuarm-stretch:isar-image-base >>> multiconfig:qemuarm64-stretch:isar-image-base >>> multiconfig:qemuamd64-stretch:isar-image-base >>> >> >> We run multiple bitbake sessions after each other. Maybe the first one never >> terminates (get stuck), and that is also why the rm after the first session >> fails. You need to stop the build there and analyse what is keeping the mount >> points busy. > > Wait... If I terminate a build from inside the container (i.e. "natively") and > then quickly try to delete the build artifacts, I can trigger that infamous > empty /dev bug - on the host. That has always been the problem, and that is one > reason why we encapsulate things into containers. > > The reason for this is that bitbake's cooker waits for the last sub-process to > finish before it calls the cleanup hook that does all the unmounting. If you > delete something before that, you step into the mount point and purge its > content. That /may/ be the issue here as well as we run rm directly after bitbake. > > IOW: Possibly just a known limitation of current Isar design /wrt to unmounting > in isar_handler() that now surfaces in CI. I would not be surprised you can > resolve that by waiting for the last cooker instance to terminate before > deleting tmp. > ...or we are just missing "-l" as umount parameter isar_handler(), like Claudius added elsewhere. Though that may not conceptually resolve the race window between bitbake terminating and cooker still running the handler. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux