From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6959801776553852928 X-Received: by 2002:a50:fd1a:: with SMTP id i26mr5385847eds.181.1624372831502; Tue, 22 Jun 2021 07:40:31 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6402:90b:: with SMTP id g11ls5672612edz.2.gmail; Tue, 22 Jun 2021 07:40:30 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwyFMbXmuSHWZd1/qUTsm5g3w5a3RxoxNmoqat0u0sKU18nI4fqh7YMO5X4RKvaXHYRdVkQ X-Received: by 2002:a05:6402:524d:: with SMTP id t13mr5485183edd.303.1624372830606; Tue, 22 Jun 2021 07:40:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1624372830; cv=none; d=google.com; s=arc-20160816; b=eL95xwGyMDnKQTn0mUaGGBzRuMTb7j2iHHdnwE3Q/7eJ0DG5SWTEK033UPWODcFb2m YdVSP+gGxaymrWHWkhhy2pdEXrq2iUtTKDd5W7JZhKvXC7GLdudyH3IQQFDfRZiuhuac m2D/K85HeI14OLnO4mUwJW+YruDLlCdjCzEKF+v6//Jbo/NRwzx7isDMmmLtaB2Wo+Ao zLOU1s8mdsvbMIvLn5d4iX1vzbN+0qV7CsajjN67O8J67f002UabeRehs1clxmrVbcAF 4sL5/UkF2mQyezPYPNdA66qYCcfwx/qEZ+Y89tT8luuxfzzN4fnc2m8SUy1tKl6Fqnya cauA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-language:content-transfer-encoding:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject; bh=B1Nqye59niPW8irmH2/RMNtf35cxOXbTbEGKOjcjeXI=; b=KMnAeqL8I2RQ0vJQPCr1BiwbhOBLZB60yZ5Qz16YWVH4uM8ZD64roapHef2x3gp8jQ EEVS32oFLgS+uEMi8Uo7RgKrO3ZfCd78EVeA6vRHWZ7Rbc5x8wZkVxReoUMcrrU4qUWY VUbypEGyzhWziF/Anq/vYsZijwYnXUDoYMF2XkNjnp5/qr7PZBfd7tP+xopqiisLgKry OSbkmlZwlhWKnrWcT0nQD7WAGN0RVZA5MOwUcmLZxTG7LudB9tCtvOdnQzTUrllh4zbR UCco/PR4KfgvW0lNVSzmChm4ZrlpctYgXA/UjT4oKOjOyQ+gDCfD04XOxJhiH31a0gqk VLZg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id o12si81104edc.0.2021.06.22.07.40.30 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 22 Jun 2021 07:40:30 -0700 (PDT) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from [192.168.67.164] (mm-26-60-214-37.mgts.dynamic.pppoe.byfly.by [37.214.60.26] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 15MEeT0u003763 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 22 Jun 2021 16:40:29 +0200 Subject: Re: [PATCH v3 0/5] Rebuild mount logic To: Jan Kiszka , isar-users@googlegroups.com References: <20210508062518.83852-1-amikan@ilbers.de> <246fd202-0ec2-563b-90fe-3c4177fda44e@siemens.com> From: Anton Mikanovich Message-ID: <94ad1d2f-ddfd-ebc3-8687-86dcef0a0b9a@ilbers.de> Date: Tue, 22 Jun 2021 17:40:23 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: <246fd202-0ec2-563b-90fe-3c4177fda44e@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: uzPZxVRYox2V 21.06.2021 18:54, Jan Kiszka wrote: > This one is still not stable (reverting my claim "current next builds > fine"): > > NOTE: recipe buildchroot-target-1.0-r0: task do_rootfs_postprocess: Started > ERROR: mc:rpi-stretch:buildchroot-target-1.0-r0 do_rootfs_postprocess: Error executing a python function in exec_python_func() autogenerated: > > The stack trace of python calls that resulted in this exception/failure was: > File: 'exec_python_func() autogenerated', lineno: 2, function: > 0001: > *** 0002:do_rootfs_postprocess(d) > 0003: > File: '/builds/Vs1BETDQ/0/ebsy/debian/isar/meta/classes/rootfs.bbclass', lineno: 298, function: do_rootfs_postprocess > 0294: for i, cmd in enumerate(cmds): > 0295: bb.build.exec_func(cmd, d) > 0296: progress_reporter.update(int(i / len(cmds) * 100)) > 0297: > *** 0298: bb.build.exec_func('rootfs_undo_mounts', d) > 0299:} > 0300:addtask rootfs_postprocess before do_rootfs > 0301: > 0302:python do_rootfs() { > File: '/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/build.py', lineno: 251, function: exec_func > 0247: with bb.utils.fileslocked(lockfiles): > 0248: if ispython: > 0249: exec_func_python(func, d, runfile, cwd=adir) > 0250: else: > *** 0251: exec_func_shell(func, d, runfile, cwd=adir) > 0252: > 0253: try: > 0254: curcwd = os.getcwd() > 0255: except: > File: '/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/build.py', lineno: 452, function: exec_func_shell > 0448: with open(fifopath, 'r+b', buffering=0) as fifo: > 0449: try: > 0450: bb.debug(2, "Executing shell function %s" % func) > 0451: with open(os.devnull, 'r+') as stdin, logfile: > *** 0452: bb.process.run(cmd, shell=False, stdin=stdin, log=logfile, extrafiles=[(fifo,readfifo)]) > 0453: finally: > 0454: os.unlink(fifopath) > 0455: > 0456: bb.debug(2, "Shell function %s finished" % func) > File: '/builds/Vs1BETDQ/0/ebsy/debian/isar/bitbake/lib/bb/process.py', lineno: 182, function: run > 0178: if not stderr is None: > 0179: stderr = stderr.decode("utf-8") > 0180: > 0181: if pipe.returncode != 0: > *** 0182: raise ExecutionError(cmd, pipe.returncode, stdout, stderr) > 0183: return stdout, stderr > Exception: bb.process.ExecutionError: Execution of '/builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/temp/run.rootfs_undo_mounts.19080' failed with exit code 32: > umount: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/sys/fs/cgroup: target is busy. > WARNING: exit code 32 from a shell command. > > > ERROR: Logfile of failure stored in: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/temp/log.do_rootfs_postprocess.19080 > NOTE: recipe buildchroot-target-1.0-r0: task do_rootfs_postprocess: Failed > ERROR: Task (mc:rpi-stretch:/builds/Vs1BETDQ/0/ebsy/debian/isar/meta/recipes-dev Looks like a race. It was not a problem when we left mounts till the end of the build. I will try to look for any possible reasons. > And then also > > NOTE: Tasks Summary: Attempted 1181 tasks of which 214 didn't need to be rerun and 1 failed. > WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev left mounted, unmounting... > WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev/pts left mounted, unmounting... > WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev/mqueue left mounted, unmounting... > WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev/termination-log left mounted, unmounting... > WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/dev/shm left mounted, unmounting... > WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/proc left mounted, unmounting... > WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/sys left mounted, unmounting... > WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/sys/fs/cgroup left mounted, unmounting... > WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/sys/fs/cgroup/cpuset left mounted, unmounting... > WARNING: /builds/Vs1BETDQ/0/ebsy/debian/isar/build/tmp/work/raspbian-stretch-armhf/buildchroot-target/1.0-r0/rootfs/downloads left mounted, unmounting... > > Summary: 1 task failed: > mc:rpi-stretch:/builds/Vs1BETDQ/0/ebsy/debian/isar/meta/recipes-devtools/buildchroot/buildchroot-target.bb:do_rootfs_postprocess > > > Though I suspect and hope the latter is related to the former. Yes, it's just a 'visualization' of mount points cleanup at finalize task, called just after the build was failed. It supposed to be empty for successful builds, but not for failed ones. > Retrying, to see if we have a race (likely) or something stable. > > Jan -- Anton Mikanovich Promwad Ltd. External service provider of ilbers GmbH Maria-Merian-Str. 8 85521 Ottobrunn, Germany +49 (89) 122 67 24-0 Commercial register Munich, HRB 214197 General Manager: Baurzhan Ismagulov