From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6519532471426482176 X-Received: by 10.28.106.25 with SMTP id f25mr381343wmc.16.1517949925107; Tue, 06 Feb 2018 12:45:25 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.199.10 with SMTP id x10ls88380wmf.9.gmail; Tue, 06 Feb 2018 12:45:24 -0800 (PST) X-Google-Smtp-Source: AH8x225RNmoR/KktBJxVjomOtXGa47wdNjuTb8MqnSH3KLUfnA+FJKTtt5K64bbesCqY63yv3jn9 X-Received: by 10.28.172.4 with SMTP id v4mr424892wme.2.1517949924609; Tue, 06 Feb 2018 12:45:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1517949924; cv=none; d=google.com; s=arc-20160816; b=m2aM/2CS/BH5IaUFn6RHclauhzLfrSqMmpEBP8VaVBlfoS3bMEKUoE3Z36NwNvVwE8 XGsFLk/cMc+ZyABR1ZTco/YqIU4y7tHZpwTPBCHTjuF5CymeW2j05gy65f5NJQbUFZ9+ P3P/ZUB3X9ggsKUXGfA09PnLPrpvLDFZ8UJK83KqGzBucc6pKeQ/mYWLk7GbgkTdv1p8 TMNqw3t9nIH8OkmE+zgnAiRmyLO2t9Qk6KeNSKSow4W16inS73nOyaCmwqVbx9czfu+K sb9qExE4agtFKZKxR6tgp/W86xfRJpsh9JtpVb49q8YMHvtUE0WWbb7V658+3qPTMCNO PzLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=Mc/9Gy2iZ0KaHbivWcWLVJaLRy1k/b83sy+NzaVKslQ=; b=tbecAdOotEpgk+nDHMB0cjVUtgVFNWDTAPAL3LOKr6KNB7W9TYW3WZVIraeWRq9WeE 8F8JD4RB6ovVIDgtOS3GLhTlIYL3ng8tnSG5WFIHnatZxlJZgJ7R3KBi+4JgHpQ8xhHk iOP4iNusNtrSMSjnQ+f71Er3toaEY1KjEUcs0m2cuvkCP7eUm48sID45WNyW+C22p5Vu OSkNbrPThbDZYn8R7mLkjUaV+vM2ln/c4vzbDEJO4BfNyvq1O89/+Vm5xXRcPE17LCqn W4lqk8NId6LBvMugaA9kFuySBPRYFAbes9TcSXUgCU7hydDVFS2S+fZSgJaw9Gyamffo HQ2g== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id y42si850406wry.5.2018.02.06.12.45.24 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 06 Feb 2018 12:45:24 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w16KjKfW030706 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 6 Feb 2018 21:45:22 +0100 Subject: Re: [PATCH] isar: Clean mount point on bitbake exit To: Jan Kiszka , isar-users@googlegroups.com References: <20180206195516.32153-1-asmirnov@ilbers.de> <1e69c65a-3c5d-abdd-ccb5-103e09371d92@siemens.com> From: Alexander Smirnov Message-ID: Date: Tue, 6 Feb 2018 23:45:15 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <1e69c65a-3c5d-abdd-ccb5-103e09371d92@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: 4ScUQOek1MCY On 02/06/2018 11:31 PM, Jan Kiszka wrote: > On 2018-02-06 20:55, Alexander Smirnov wrote: >> 8<-- >> >> That's it! Branch 'asmirnov/devel', please test and enjoy :-) >> >> 8<-- >> >> Now each multiconfig has registered handler for BuildCompleted event (see >> class 'isar-event.bbclass'). Moreover, the '/proc/mounts' file contains >> all the active mounts. In addition, from event handler we could derive >> all the variables like ${TMPDIR}, ${DISTRO} etc. So it's possible to >> find all the active mounts for current multiconfig and clean them. >> >> NOTE: if build is interrupted by double ^C, some mount points could stay >> uncleaned. This is caused by remaining processes started by bitbake, for >> example: >> - 'chroot build.sh ...' >> - 'multistrap ...' > > Can you explain what the race condition is exactly? It seems to work for > me so far, no forgotten mounts. Sometimes after double ^C I see several mounted artifacts in buildchroot. But I've also noticed, that there are running processes like 'multistrap' and 'build.sh' running *exactly* in this buildchroot. So, simple test, if you press double ^C and run 'ps ax' after, you will likely see running processes. IIRC the same bitbake behavior I saw with Yocto, interrupting bitbake doesn't mean to kill the process with current task. Alex > > Jan > >> >> So please be careful when interrupting build. >> >> Signed-off-by: Alexander Smirnov >> --- >> meta-isar/recipes-core/images/isar-image-base.bb | 11 ++++------ >> meta/classes/dpkg-base.bbclass | 12 ++++------- >> meta/classes/isar-events.bbclass | 15 +++++++++++--- >> meta/recipes-devtools/buildchroot/buildchroot.bb | 24 +++++++++------------- >> .../buildchroot/files/configscript.sh | 4 ---- >> .../buildchroot/files/download_dev-random | 13 ------------ >> 6 files changed, 30 insertions(+), 49 deletions(-) >> delete mode 100644 meta/recipes-devtools/buildchroot/files/download_dev-random >> >> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb >> index e359ac3..8ddbabb 100644 >> --- a/meta-isar/recipes-core/images/isar-image-base.bb >> +++ b/meta-isar/recipes-core/images/isar-image-base.bb >> @@ -55,14 +55,10 @@ do_rootfs() { >> -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \ >> "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf" >> >> + # Do not use bitbake flag [dirs] here because this folder should have >> + # specific ownership. >> [ ! -d ${IMAGE_ROOTFS}/proc ] && sudo install -d -o 0 -g 0 -m 555 ${IMAGE_ROOTFS}/proc >> sudo mount -t proc none ${IMAGE_ROOTFS}/proc >> - _do_rootfs_cleanup() { >> - ret=$? >> - sudo umount ${IMAGE_ROOTFS}/proc 2>/dev/null || true >> - (exit $ret) || bb_exit_handler >> - } >> - trap '_do_rootfs_cleanup' EXIT >> >> # Create root filesystem. We must use sudo -E here to preserve the environment >> # because of proxy settings >> @@ -72,5 +68,6 @@ do_rootfs() { >> sudo chroot ${IMAGE_ROOTFS} /${DISTRO_CONFIG_SCRIPT} ${MACHINE_SERIAL} ${BAUDRATE_TTY} \ >> ${ROOTFS_DEV} >> sudo rm "${IMAGE_ROOTFS}/${DISTRO_CONFIG_SCRIPT}" >> - _do_rootfs_cleanup >> + >> + sudo umount ${IMAGE_ROOTFS}/proc 2>/dev/null || true >> } >> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass >> index 5d5a924..a34c21f 100644 >> --- a/meta/classes/dpkg-base.bbclass >> +++ b/meta/classes/dpkg-base.bbclass >> @@ -20,15 +20,11 @@ dpkg_runbuild() { >> do_build() { >> mkdir -p ${BUILDROOT} >> sudo mount --bind ${WORKDIR} ${BUILDROOT} >> - _do_build_cleanup() { >> - ret=$? >> - sudo umount ${BUILDROOT} 2>/dev/null || true >> - sudo rmdir ${BUILDROOT} 2>/dev/null || true >> - (exit $ret) || bb_exit_handler >> - } >> - trap '_do_build_cleanup' EXIT >> + >> dpkg_runbuild >> - _do_build_cleanup >> + >> + sudo umount ${BUILDROOT} 2>/dev/null || true >> + sudo rmdir ${BUILDROOT} 2>/dev/null || true >> } >> >> # Install package to Isar-apt >> diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass >> index 55fc106..ae0f791 100644 >> --- a/meta/classes/isar-events.bbclass >> +++ b/meta/classes/isar-events.bbclass >> @@ -11,10 +11,19 @@ python isar_handler () { >> devnull = open(os.devnull, 'w') >> >> if isinstance(e, bb.event.BuildCompleted): >> - bchroot = d.getVar('BUILDCHROOT_DIR', True) >> + tmpdir = d.getVar('TMPDIR', True) >> + distro = d.getVar('DISTRO', True) >> + arch = d.getVar('DISTRO_ARCH', True) >> >> - # Clean up buildchroot >> - subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot + '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True) >> + w = tmpdir + '/work/' + distro + '-' + arch >> + >> + # '/proc/mounts' contains all the active mounts, so knowing 'w' we >> + # could get the list of mounts for the specific multiconfig and >> + # clean them. >> + with open('/proc/mounts', 'rU') as f: >> + for line in f: >> + if w in line: >> + subprocess.call('sudo umount -f ' + line.split()[1], stdout=devnull, stderr=devnull, shell=True) >> >> devnull.close() >> } >> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb >> index 304c67e..df9df19 100644 >> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb >> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb >> @@ -12,7 +12,6 @@ FILESPATH =. "${LAYERDIR_core}/recipes-devtools/buildchroot/files:" >> SRC_URI = "file://multistrap.conf.in \ >> file://configscript.sh \ >> file://setup.sh \ >> - file://download_dev-random \ >> file://build.sh" >> PV = "1.0" >> >> @@ -32,8 +31,10 @@ BUILDCHROOT_PREINSTALL ?= "gcc \ >> WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" >> >> do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >> -do_build[dirs] = "${WORKDIR}/hooks_multistrap \ >> - ${BUILDCHROOT_DIR}/isar-apt" >> +do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \ >> + ${BUILDCHROOT_DIR}/dev \ >> + ${BUILDCHROOT_DIR}/proc \ >> + ${BUILDCHROOT_DIR}/sys" >> do_build[depends] = "isar-apt:do_cache_config" >> >> do_build() { >> @@ -41,7 +42,6 @@ do_build() { >> >> chmod +x "${WORKDIR}/setup.sh" >> chmod +x "${WORKDIR}/configscript.sh" >> - install -m 755 "${WORKDIR}/download_dev-random" "${WORKDIR}/hooks_multistrap/" >> >> # Multistrap accepts only relative path in configuration files, so get it: >> cd ${TOPDIR} >> @@ -60,15 +60,6 @@ do_build() { >> -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ >> "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf" >> >> - [ ! -d ${BUILDCHROOT_DIR}/proc ] && install -d -m 555 ${BUILDCHROOT_DIR}/proc >> - sudo mount -t proc none ${BUILDCHROOT_DIR}/proc >> - _do_build_cleanup() { >> - ret=$? >> - sudo umount ${BUILDCHROOT_DIR}/proc 2>/dev/null || true >> - (exit $ret) || bb_exit_handler >> - } >> - trap '_do_build_cleanup' EXIT >> - >> do_setup_mounts >> >> # Create root filesystem >> @@ -79,7 +70,6 @@ do_build() { >> >> # Configure root filesystem >> sudo chroot ${BUILDCHROOT_DIR} /configscript.sh >> - _do_build_cleanup >> >> do_cleanup_mounts >> } >> @@ -96,10 +86,16 @@ do_setup_mounts[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >> >> do_setup_mounts() { >> sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt >> + sudo mount --bind /dev ${BUILDCHROOT_DIR}/dev >> + sudo mount -t proc none ${BUILDCHROOT_DIR}/proc >> + sudo mount -t sysfs none ${BUILDCHROOT_DIR}/sys >> } >> >> addtask setup_mounts after do_build >> >> do_cleanup_mounts() { >> sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true >> + sudo umount ${BUILDCHROOT_DIR}/dev 2>/dev/null || true >> + sudo umount ${BUILDCHROOT_DIR}/proc 2>/dev/null || true >> + sudo umount ${BUILDCHROOT_DIR}/sys 2>/dev/null || true >> } >> diff --git a/meta/recipes-devtools/buildchroot/files/configscript.sh b/meta/recipes-devtools/buildchroot/files/configscript.sh >> index 9813c9a..524e50c 100644 >> --- a/meta/recipes-devtools/buildchroot/files/configscript.sh >> +++ b/meta/recipes-devtools/buildchroot/files/configscript.sh >> @@ -39,10 +39,6 @@ export LC_ALL=C LANGUAGE=C LANG=C >> #run pre installation script >> /var/lib/dpkg/info/dash.preinst install >> >> -# apt-get http method, gpg require /dev/null >> -mount -t devtmpfs -o mode=0755,nosuid devtmpfs /dev >> - >> #configuring packages >> dpkg --configure -a >> apt-get update >> -umount /dev >> diff --git a/meta/recipes-devtools/buildchroot/files/download_dev-random b/meta/recipes-devtools/buildchroot/files/download_dev-random >> deleted file mode 100644 >> index 5b5b96b..0000000 >> --- a/meta/recipes-devtools/buildchroot/files/download_dev-random >> +++ /dev/null >> @@ -1,13 +0,0 @@ >> -#!/bin/sh >> - >> -set -e >> - >> -readonly ROOTFS="$1" >> - >> -mknod "${ROOTFS}/dev/random" c 1 8 >> -chmod 640 "${ROOTFS}/dev/random" >> -chown 0:0 "${ROOTFS}/dev/random" >> - >> -mknod "${ROOTFS}/dev/urandom" c 1 9 >> -chmod 640 "${ROOTFS}/dev/urandom" >> -chown 0:0 "${ROOTFS}/dev/urandom" >> > -- With best regards, Alexander Smirnov ilbers GmbH Baierbrunner Str. 28c D-81379 Munich +49 (89) 122 67 24-0 http://ilbers.de/ Commercial register Munich, HRB 214197 General manager: Baurzhan Ismagulov