From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6959801776553852928 X-Received: by 2002:a1c:ddd6:: with SMTP id u205mr28045053wmg.63.1624290852554; Mon, 21 Jun 2021 08:54:12 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:5122:: with SMTP id o34ls1458328wms.3.gmail; Mon, 21 Jun 2021 08:54:11 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzAk9DjlxnVXGxghpGe8Q2vzEgoUbTozAJoStVvMcQZMjiL8LgWNknnrKCdu7w9BsPkRPmn X-Received: by 2002:a1c:f215:: with SMTP id s21mr4897281wmc.179.1624290851691; Mon, 21 Jun 2021 08:54:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1624290851; cv=none; d=google.com; s=arc-20160816; b=gfVjNebd22omIAaTGLeKo42tn3mmZKaWwI7L2ZX/ykBxK6RNq/86r/cezWoOKCPPPq n/joGjHGpgrMfa+X+G6vVKBJvfwwSYOp2ARaSqoGOzjQLI5MIBT82vUytfS2wmPKxVAe qGbn5b9gPAAr0n0QgHLzZxn4hfTIqiHbuZCpY0RXNLF2fRkvaODrCmrr4lccgj1t2SaS rsWHSksOD7bZeavOiq2ozCQSLcwVCo714XRoLUBcTJoV4BXFICo8tprkFE3431Y8TKWj WGjYRRpeua/lf+pq8EWeH42fbYbx6P42LvuNp4Q8slI36DJudUqfXwEJa/cN0ikNS/nX DSIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject; bh=5H6LaHCC+SE7swJydme5qIviCFRsWB5I9dmGvu/l14I=; b=lDJ7JeLsD/dEk3+1vmdWKZN7+GRwmiq3G8pCvrGamUg3XZQ5BT5GjwsDhExkFn+yW+ acN9rtPOAIZ0b8b+BWwxKyKl/nSI1Wi+Wto/ngn9AOTyPMYdpOnJQvA0Xnb4PEQu6JOC 3yie3flbc3QUN73EnTNl24Eu2DNWP7vJtCBD3oGai+VdHURH0WEy+7IbG8pnXCQJwbJE h/YNdOHEEyRG3FcX2Ish1XxxpEZ3tbkr7T4uA7rdqt4brBOQ01mfVzN4z8eNFIx7qNM5 44WpWyqi5qBv2IYrv6fcOoDQsYRfHB+qGHood6Dh9kMr083GPPNA+Ra4W0XPO4mRLylU 8KSA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id z70si843457wmc.0.2021.06.21.08.54.11 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Jun 2021 08:54:11 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id 15LFsBqu024907 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 21 Jun 2021 17:54:11 +0200 Received: from [167.87.243.122] ([167.87.243.122]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 15LFsAhL024890; Mon, 21 Jun 2021 17:54:10 +0200 Subject: Re: [PATCH v3 0/5] Rebuild mount logic To: Anton Mikanovich , isar-users@googlegroups.com References: <20210508062518.83852-1-amikan@ilbers.de> From: Jan Kiszka Message-ID: <246fd202-0ec2-563b-90fe-3c4177fda44e@siemens.com> Date: Mon, 21 Jun 2021 17:54:10 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.0 MIME-Version: 1.0 In-Reply-To: <20210508062518.83852-1-amikan@ilbers.de> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: S8P64ikOlFt2 On 08.05.21 08:25, Anton Mikanovich wrote: > Mounting is perfomed very inconsistent currently in Isar. A lot of mount > points are left after task execution and stays mounted until cleanup in > build_completed hook. > > We already experienced issues with stuck forever on unmounting at > "Couldn't unmount, retrying..." error. > It is also quite easy to face with a lot of ramaining mounts after the > failed or interrupted manual builds. > > This patchset implements unmounting paths after usage. In the cases of > mount sharing corresponding request counting is performed, so real mount > will be done only at first request, real unmount will be done only after > the last task will try to unmount the path. > > Meantime build_completed will use emergency only, so following warning > added to show something goes wrong. > > --- > Changes since v2: > - Prevent race conditions. > - Fix building qemuarm64-stretch. > - Minimize locks. > - Cleanup counter files. > - Unmount /proc/sys/fs/binfmt_misc also. > Changes since v1: > - Make all mounts be unmounted after use. > - Warn about lost mounts after finish. > > Anton Mikanovich (5): > dpkg: Make mount buildroot reliable > buildchroot: Unmount buildchroot mounts if not needed > rootfs: Unmount rootfs mounts if not needed > wic: Unmount dirs after usage > events: Warn if mounted paths left > > meta/classes/buildchroot.bbclass | 47 +++++++ > meta/classes/dpkg-base.bbclass | 127 +++++++++++------- > meta/classes/dpkg-gbp.bbclass | 8 +- > meta/classes/dpkg.bbclass | 14 +- > meta/classes/ext4-img.bbclass | 2 + > meta/classes/fit-img.bbclass | 2 + > meta/classes/image-tools-extension.bbclass | 2 + > meta/classes/image.bbclass | 24 ++-- > meta/classes/initramfs.bbclass | 2 + > meta/classes/isar-events.bbclass | 1 + > meta/classes/rootfs.bbclass | 62 ++++++++- > meta/classes/ubi-img.bbclass | 2 + > meta/classes/ubifs-img.bbclass | 2 + > meta/classes/wic-img.bbclass | 10 ++ > .../buildchroot/buildchroot.inc | 8 ++ > 15 files changed, 232 insertions(+), 81 deletions(-) > This one is still not stable (reverting my claim "current next builds fine"): NOTE: recipe buildchroot-target-1.0-r0: task do_rootfs_postprocess: Started ERROR: mc:rpi-stretch:buildchroot-target-1.0-r0 do_rootfs_postprocess: Error executing a python function in exec_python_func() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: 0001: *** 0002:do_rootfs_postprocess(d) 0003: File: '/builds/Vs1BETDQ/0/ebsy/debian/isar/meta/classes/rootfs.bbclass', lineno: 298, function: do_rootfs_postprocess 0294: for i, cmd in enumerate(cmds): 0295: bb.build.exec_func(cmd, d) 0296: progress_reporter.update(int(i / len(cmds) * 100)) 0297: *** 0298: bb.build.exec_func('rootfs_undo_mounts', d) 0299:} 0300:addtask rootfs_postprocess before do_rootfs 0301: 0302:python do_rootfs() { File: '/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/build.py', lineno: 251, function: exec_func 0247: with bb.utils.fileslocked(lockfiles): 0248: if ispython: 0249: exec_func_python(func, d, runfile, cwd=adir) 0250: else: *** 0251: exec_func_shell(func, d, runfile, cwd=adir) 0252: 0253: try: 0254: curcwd = os.getcwd() 0255: except: File: '/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/build.py', lineno: 452, function: exec_func_shell 0448: with open(fifopath, 'r+b', buffering=0) as fifo: 0449: try: 0450: bb.debug(2, "Executing shell function %s" % func) 0451: with open(os.devnull, 'r+') as stdin, logfile: *** 0452: bb.process.run(cmd, shell=False, stdin=stdin, log=logfile, extrafiles=[(fifo,readfifo)]) 0453: finally: 0454: os.unlink(fifopath) 0455: 0456: bb.debug(2, "Shell function %s finished" % func) File: '/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/process.py', lineno: 182, function: run 0178: if not stderr is None: 0179: stderr = stderr.decode("utf-8") 0180: 0181: if pipe.returncode != 0: *** 0182: raise ExecutionError(cmd, pipe.returncode, stdout, stderr) 0183: return stdout, stderr Exception: bb.process.ExecutionError: Execution of '/builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/temp/run.rootfs_undo_mounts.19080' failed with exit code 32: umount: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/sys/fs/cgroup: target is busy. WARNING: exit code 32 from a shell command. ERROR: Logfile of failure stored in: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/temp/log.do_rootfs_postprocess.19080 NOTE: recipe buildchroot-target-1.0-r0: task do_rootfs_postprocess: Failed ERROR: Task (mc:rpi-stretch:/builds/Vs1BETDQ/0/ebsy/debian/isar/meta/recipes-dev And then also NOTE: Tasks Summary: Attempted 1181 tasks of which 214 didn't need to be rerun and 1 failed. WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev left mounted, unmounting... WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev/pts left mounted, unmounting... WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev/mqueue left mounted, unmounting... WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev/termination-log left mounted, unmounting... WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev/shm left mounted, unmounting... WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/proc left mounted, unmounting... WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/sys left mounted, unmounting... WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/sys/fs/cgroup left mounted, unmounting... WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/sys/fs/cgroup/cpuset left mounted, unmounting... WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/downloads left mounted, unmounting... Summary: 1 task failed: mc:rpi-stretch:/builds/Vs1BETDQ/0/ebsy/debian/isar/meta/recipes-devtools/buildchroot/buildchroot-target.bb:do_rootfs_postprocess Though I suspect and hope the latter is related to the former. Retrying, to see if we have a race (likely) or something stable. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux