From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6631438923266523136 X-Received: by 2002:a50:8eca:: with SMTP id x10mr5080734edx.8.1544051335368; Wed, 05 Dec 2018 15:08:55 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a50:a4aa:: with SMTP id w39ls6279937edb.8.gmail; Wed, 05 Dec 2018 15:08:55 -0800 (PST) X-Google-Smtp-Source: AFSGD/V4IhZtNv80Lk0sf/px/M2E6JwGa0y/C2qMFwkx8Zo9vds7wrSQ3su6uVe6cqoDhj/pmv8U X-Received: by 2002:a50:b4a1:: with SMTP id w30mr5138791edd.2.1544051334888; Wed, 05 Dec 2018 15:08:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544051334; cv=none; d=google.com; s=arc-20160816; b=hrDHngxN7WQGqWT3yrnf+49FhXxo4dVo9ZHceMrD5kdpaqDj7RhoZL9OXiz8NcicBQ meC9Q4hs5LCB5C29CNzScOl35ZWh5BX/OkPabG9Flxv5ETcptfwkUK2cE/T4D++OxERV YC9EESMBtCZ3B6cXffaMM8pvlGifdIdZTwUoI6gQ7609FwhU1Pl4W1kAM94ZbntlFRYM G4p/78JibDrvZ6QsNiB+L/EHOwgkol+iByJ+oTLP4bhHhFjS9zEsdUfmqnl6H4fjkOug UQ/oTzNNO4gydM7r7dGgbxCtWnmkkHUDqtr4m93bN13Wms7zZc4vu6vn9csfxpTdoNSB PI9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:references:cc:to:from:subject; bh=1CkU4Vv9jRRyPOmFHRLVMtfcPCi796PgWKl7qREFzXc=; b=NKQfg/rWZSgPY3TopAOrOIlyqZiRHEDbUHWnGHehisbWccRWdMcqu2XF7fpZBO3o7h cCaev/eNDhsbtJao36OOIKA/iMOofvkWbWnyJfEhKJNd3vvKnmtBYA6zVaGAUapACD3v hCxDBZ+gAriL7f+u5e1UebFc4+nMkvr1Fys5KSIsCD8il/cUFTcWaH5zIUXkr77rpTMb mf7SCVSQ7JnChB7vLmVRbI+WQFZS+X/FkpOPlZHIl6ef4bjh7Ag6OXmfrjQdKNEWddqk /mmkBrRNvLDcC07GPhhlKpXzMMLZUD14ouivJyqJByopdHKrhm+aoF8R0IxN0TTPXCbh G9uw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@web.de designates 212.227.17.11 as permitted sender) smtp.mailfrom=jan.kiszka@web.de Return-Path: Received: from mout.web.de (mout.web.de. [212.227.17.11]) by gmr-mx.google.com with ESMTPS id k25si855514edd.1.2018.12.05.15.08.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Dec 2018 15:08:54 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@web.de designates 212.227.17.11 as permitted sender) client-ip=212.227.17.11; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@web.de designates 212.227.17.11 as permitted sender) smtp.mailfrom=jan.kiszka@web.de Received: from [192.168.1.10] ([95.157.57.47]) by smtp.web.de (mrweb103 [213.165.67.124]) with ESMTPSA (Nemesis) id 0MLgQp-1gVVMl3ay3-000sOY; Thu, 06 Dec 2018 00:08:52 +0100 Subject: Re: [PATCH 0/3] umount fixes and cleanups From: Jan Kiszka To: "Maxim Yu. Osipov" , isar-users , Henning Schild Cc: Cedric Hombourger References: <02edaac4-fc2d-44ea-6471-52ad8fc3d421@siemens.com> <7267182f-b841-0763-9870-833c0ccdfee8@ilbers.de> <936cda18-e1af-6182-cf10-2f41b0a166e7@siemens.com> Message-ID: <0ddc2307-4095-0656-140a-7f4b10ae6722@web.de> Date: Thu, 6 Dec 2018 00:08:51 +0100 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <936cda18-e1af-6182-cf10-2f41b0a166e7@siemens.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Provags-ID: V03:K1:eqyuJnrO/mBZ6H5i3uY/4CmgOhg3KmgiJwRsBc6NEvWjQ39oEUt G6mZsbBlfa1tb02Wm+snz0jwtL++NHYers9rtQyDHu8MeoGRGvUb59TWDwlBfVD9YjcN1nt +e9M8O27uP9Jdgq/HE/L9fk7PAfcBwW0KoACXwNEuRqpjogvSPFKxdwfzanGV62Zy/EAw72 1j5O+TqcNK6r0dg263WVQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:G0HFnLDOBXY=:Iydpg4lYLIpMqpIOR9gw8y ckNH5aUcR9JBTuvXZwdx3fSioyOLzbcyrLJO2uStda6gPiWCjn9IRzhVlUd7WmafDWiyTQAmT ToOa9YbVkKgtkPhtQ2dj4K4guUg9iJpPQgmACMANJFk/E0x4Bx9c9kwTFmiJBb2wYbQyPqSzO 1uiCI1IWAbuP0B4npwy7RlLJjEPcLa1wF5XBSJHTrnAhuoRpz3clruRq9mqZ3Yb/+J8M1fp8c aOTd5FlWawC9klZ+8HwxXYdYkKlP1qgJxFXJj4pfyNjzU5sgEVED10Czq/9UcWm2Y6iUVL9r/ JU6JhmkLBFdhGXCGg+u5L5NMDzxi8+gZ4Wg9B2e24ykIEKdjl5+ENfQDmNsvexMP/abjuFSSa hObHj2a0nkLb4eepehgb23v2gViecCtipuBYHQvDgv3a71gGztVImGaR0hGyjlU1n7u0COed0 Wi9JC6XcQmeeVj2rG/WN70pIkbo5gvr9tu02zT7wQnk2XITuX2PsIPHZ02bbEV4sDM0UXqDsR mzYDkS+1SKblg7C7FbDQ7DcsLv1gOZntofF0KrQbZTwKU7sCipfLQwcW+iiU1gGjL8r0je6Ql x5NxiELJm8TJj2LfuH4LgKPs2lqv3ZZDxykMJsG6QDd/myWvGjJtVFsXDbFc3x0c0vx8WBxZB gc3tZtXkblvJBaLzecjr4qY3J5VgrwJfAb8UeQl6GbN2kcPro/Dd0SNGFQYRpAH0U0gQprvdq YQX+Ao7SFmdWA9KqcbeTdXJNJzIza5YCB8ioiVOW6VfSkgm58Gmd9tiODMASqQqhxKmI4GnL3 /Eo5WOgDMqTFLhWJkKRxCuCtRXlXg9u+zmiVU32i8WPDlTRODRk4Y+PqSuUaMAoBp8jbly/ch 89O5dagRn8piF9LpJScTj4MHNxgN459EcU7dQLW9F4NtN2/eoFRPvauGtsLEUo X-TUID: peF42pajD3Su On 05.12.18 15:52, [ext] Jan Kiszka wrote: > On 05.12.18 15:26, Maxim Yu. Osipov wrote: >> On 12/5/18 3:14 PM, Jan Kiszka wrote: >>> On 05.12.18 12:12, Maxim Yu. Osipov wrote: >>>> On 12/5/18 12:29 PM, Jan Kiszka wrote: >>>>> Patches related to resolving the pending CI issues as well as >>>>> simplifying the umounts used during cleanup. >>>> >>>> Tried to run in patch queue: >>>> >>>> 027b7cf Remove redundant recursive umounts >>>> c1bdc33 isar-events: Improve umount handler >>>> b354273 ci: Wait for bitbake worker to finish before deleting artifacts >>>> 9cf29e6 isar-bootstrap: Fix and cleanup bind mounting >>>> b354026 isar-image: umount base-apt when doing offline build >>>> e965c0d gitlab-ci: Switch to ci_build.sh >>>> ... >>>> >>>> After execution of problematic test case (I rebooted PC and executed steps >>>> in clean tree): >>>> >>>> my stretch Debian system was entered into unusable state >>>> as many important mounts were disappeared (see log of mount points >>>> before and after execution  of last command attached). >>> >>> OK, so I took the time and extracted the root cause. We may see a Debian 4.9 >>> kernel bug here: >>> >>> # mkdir my-mnt >>> # mount --rbind /sys my-mnt >>> # mount --make-rslave my-mnt >>> # umount {-R,-l,-R -l} my-mnt >>> # rmdir my-mnt >>> rmdir: failed to remove 'my-mnt': Device or resource busy >>> >>> This works fine on my Leap 15.0 kernel (4.12) as well as the 4.4 kernel >>> (Ubuntu) used on our CI server. >> >> This also works fine on my Debian 4.9 kernel (I've ran this sequence for every >> combination of {-R, -l , -R -l}): >> >>> root@tiberius:~# uname -a >>> Linux tiberius 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u6 (2018-10-08) >>> x86_64 GNU/Linux > > Hmpf, rechecking... I was still on 3.16 from the jessie that I upgraded from. > Works fine with 4.9, indeed. > >>> root@tiberius:~# mkdir my-mnt >>> root@tiberius:~# mount --rbind /sys my-mnt >>> root@tiberius:~# mount --make-rslave my-mnt >>> root@tiberius:~# umount -R my-mnt/ >>> root@tiberius:~# rmdir my-mnt/ >>> root@tiberius:~# mkdir my-mnt >>> root@tiberius:~# mount --rbind /sys my-mnt >>> root@tiberius:~# mount --make-rslave my-mnt >>> root@tiberius:~# umount -l my-mnt >>> root@tiberius:~# rmdir my-mnt/ >>> root@tiberius:~# mkdir my-mnt >>> root@tiberius:~# mount --rbind /sys my-mnt >>> root@tiberius:~# mount --make-rslave my-mnt >>> root@tiberius:~# umount -R -l my-mnt >>> root@tiberius:~# rmdir my-mnt/ >>> root@tiberius:~# >> > > OK, then bind-mounting itself likely not the root cause, and we need to dig > further. > Here is the real solution, or at least a much better workaround, without any reverts: diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass index 225463e..c66d208 100644 --- a/meta/classes/wic-img.bbclass +++ b/meta/classes/wic-img.bbclass @@ -90,7 +90,7 @@ do_wic_image() { for dir in ${BBLAYERS} ${STAGING_DIR} ${ISARROOT}/scripts; do sudo mkdir -p ${BUILDCHROOT_DIR}/$dir mountpoint ${BUILDCHROOT_DIR}/$dir >/dev/null 2>&1 \ - || sudo mount --bind $dir ${BUILDCHROOT_DIR}/$dir + || sudo mount --bind --make-private $dir ${BUILDCHROOT_DIR}/$dir done export FAKEROOTCMD=${FAKEROOTCMD} export BUILDDIR=${BUILDDIR} I'll explain tomorrow, also what else is broken in do_wic_image. Jan