From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6959801776553852928 X-Received: by 2002:a05:6402:214:: with SMTP id t20mr54600665edv.20.1625140519509; Thu, 01 Jul 2021 04:55:19 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6402:3514:: with SMTP id b20ls5809546edd.1.gmail; Thu, 01 Jul 2021 04:55:18 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzTopiOllfX2d6dRiDN2ImZs/+3yIWQ+o3Lh00IcNZ+s/pNZ8tpZSUAdFJxGhV3vhIu5t3B X-Received: by 2002:a05:6402:90b:: with SMTP id g11mr53483262edz.336.1625140518561; Thu, 01 Jul 2021 04:55:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1625140518; cv=none; d=google.com; s=arc-20160816; b=kswyGT21jkR/peTpiXuMr778GYqnWaK4hjQTfYByd8+bHeTrZFyZqNyrJcsGMltlTW CQM2Uda/ftR5UA+GJMjxZ7FcM+xl5p7dNebnVNWCAA82n/hKMV71zA3G9XGtfenp4wE5 Ff2R7pox2WhPeLlfl8wXtmGI0BZtJvr4AMGVllLKxcuuYPiwgG2/1tTGB/jUFwbznM66 rUwRdSkoQl1OqX5E4vjG83GlKtwBL/gVrfQHXfXfCOpqcWhiByZ/BIdx1HuCre37SMeT Ym3JKYRELlaKPl5Ym/3bBTKqMvIlM3d+O+ic8VmfAhvYAIcXev5pVrsHrHpZviyhONPt DdoQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject; bh=bB2vV5rffYI5phkcug2Ata895xC1bdZ3Pu4jz7jVxVk=; b=orO/RGAo1+Cm0jNfm6aMV07ud2ReK0aeRhfM94NFG7dfLCOvmc4+EOpW3E+0Ix+ClW Pk3q8fEiAUEFobPDp8Xm58zqo3/MBQeyGPwF07Gvvli2/hvuH+PxxdIivPjOKYeFzmbv VnrvpSkeuMJnOmsVnbPvh8ZaMfHpd7tCzLd5qVtUBfMtRk/4yWW09KmZTP7djYkCH0Gf 4Qipu7gM/XGZr9feDIYCc0KRFQcXVvzsvr9/LMg8mj8ELjs1Bui0BPaidYgP/t1l2E4L AvkaCX6CtERNdF/mBZHrponno7uP98jak6CDjqJAhqaOr66KtBIaqUHe8Gmfd+M+cVMo xhaA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from gecko.sbs.de (gecko.sbs.de. [194.138.37.40]) by gmr-mx.google.com with ESMTPS id c16si905638edj.3.2021.07.01.04.55.18 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 01 Jul 2021 04:55:18 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) client-ip=194.138.37.40; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by gecko.sbs.de (8.15.2/8.15.2) with ESMTPS id 161BtIJY005243 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 1 Jul 2021 13:55:18 +0200 Received: from [167.87.0.185] ([167.87.0.185]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 161BtHqm021566; Thu, 1 Jul 2021 13:55:17 +0200 Subject: Re: [PATCH v3 1/5] dpkg: Make mount buildroot reliable To: Anton Mikanovich , isar-users@googlegroups.com References: <20210508062518.83852-1-amikan@ilbers.de> <20210508062518.83852-2-amikan@ilbers.de> From: Jan Kiszka Message-ID: <8fa0548c-fe58-dc9f-4935-1f2d3f988b3d@siemens.com> Date: Thu, 1 Jul 2021 13:55:17 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <20210508062518.83852-2-amikan@ilbers.de> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: ukn7h897NhQH On 08.05.21 08:25, Anton Mikanovich wrote: > We use mounting for several tasks. Mounting and unmounting is performed > in shell scripts. If a command fails before unmounting, the directories > remain mounted. > > Implement exception handling around the scripts to ensure that > unmounting is performed also in case of script failure. The number of > unmounting attempts has been limited to avoid infinite loops. > > Signed-off-by: Anton Mikanovich > --- > meta/classes/dpkg-base.bbclass | 126 ++++++++++++++++++++------------- > meta/classes/dpkg-gbp.bbclass | 8 +-- > meta/classes/dpkg.bbclass | 14 ++-- > 3 files changed, 89 insertions(+), 59 deletions(-) > > diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass > index 97661a6..d5e70f1 100644 > --- a/meta/classes/dpkg-base.bbclass > +++ b/meta/classes/dpkg-base.bbclass > @@ -61,12 +61,7 @@ addtask patch before do_adjust_git > > SRC_APT ?= "" > > -do_apt_fetch() { > - if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then > - return 0 > - fi > - dpkg_do_mounts > - E="${@ isar_export_proxies(d)}" > +fetch_apt() { > sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \ > -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \ > -o Dir::Etc::SourceParts="-" \ > @@ -76,21 +71,25 @@ do_apt_fetch() { > sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ > sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}" > done > +} > + > +python do_apt_fetch() { > + src_apt = d.getVar("SRC_APT", True) > + if not src_apt: > + return 0 > > - dpkg_undo_mounts > + dpkg_do_mounts(d) > + try: > + isar_export_proxies(d) > + bb.build.exec_func("fetch_apt", d) > + finally: > + dpkg_undo_mounts(d) > } > > addtask apt_fetch after do_unpack before do_apt_unpack > do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" > > -do_apt_unpack() { > - if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then > - return 0 > - fi > - rm -rf ${S} > - dpkg_do_mounts > - E="${@ isar_export_proxies(d)}" > - > +unpack_apt() { > for uri in "${SRC_APT}"; do > sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ > sh -c ' \ > @@ -101,8 +100,25 @@ do_apt_unpack() { > dpkg-source -x "${dscfile}" "${PPS}"' \ > my_script "${DISTRO}" "${uri}" > done > +} > > - dpkg_undo_mounts > +python do_apt_unpack() { > + import shutil > + > + src_apt = d.getVar("SRC_APT", True) > + if not src_apt: > + return 0 > + > + srcsubdir = d.getVar('S', True) > + if os.path.exists(srcsubdir): > + shutil.rmtree(srcsubdir) > + > + dpkg_do_mounts(d) > + try: > + isar_export_proxies(d) > + bb.build.exec_func("unpack_apt", d) > + finally: > + dpkg_undo_mounts(d) > } > > addtask apt_unpack after do_apt_fetch before do_patch > @@ -146,25 +162,37 @@ do_prepare_build[deptask] = "do_deploy_deb" > > BUILDROOT = "${BUILDCHROOT_DIR}/${PP}" > > -dpkg_do_mounts() { > - mkdir -p ${BUILDROOT} > - sudo mount --bind ${WORKDIR} ${BUILDROOT} > - > - buildchroot_do_mounts > -} > - > -dpkg_undo_mounts() { > - i=1 > - while ! sudo umount ${BUILDROOT}; do > - sleep 0.1 > - i=`expr $i + 1` > - if [ $i -gt 100 ]; then > - bbwarn "${BUILDROOT}: Couldn't unmount, retrying..." > - i=1 > - fi > - done > - sudo rmdir ${BUILDROOT} > -} > +def ismount(path): > + real = os.path.realpath(path) > + with open('/proc/mounts') as f: > + for line in f.readlines(): > + if len(line.split()) > 2 and real == line.split()[1]: > + return True > + return False > + > +def dpkg_do_mounts(d): > + buildroot = d.getVar('BUILDROOT', True) > + if ismount(buildroot): > + bb.warn('Path %s already mounted!' % buildroot) > + return > + workdir = d.getVar('WORKDIR', True) > + os.makedirs(buildroot, exist_ok=True) > + os.system('sudo mount --bind %s %s' % (workdir, buildroot)) > + bb.build.exec_func("buildchroot_do_mounts", d) > + > +def dpkg_undo_mounts(d): > + buildroot = d.getVar('BUILDROOT', True) > + if not ismount(buildroot): > + bb.warn('Path %s not mounted!' % buildroot) > + return > + for i in range(200): > + if not os.system('sudo umount %s' % buildroot): > + os.rmdir(buildroot) > + return > + if i % 100 == 0: > + bb.warn("%s: Couldn't unmount, retrying..." % buildroot) > + time.sleep(0.1) > + bb.fatal("Couldn't unmount, exiting...") > > # Placeholder for actual dpkg_runbuild() implementation > dpkg_runbuild() { > @@ -174,10 +202,12 @@ dpkg_runbuild() { > python do_dpkg_build() { > lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock", > shared=True) > - bb.build.exec_func("dpkg_do_mounts", d) > - bb.build.exec_func("dpkg_runbuild", d) > - bb.build.exec_func("dpkg_undo_mounts", d) > - bb.utils.unlockfile(lock) > + dpkg_do_mounts(d) > + try: > + bb.build.exec_func("dpkg_runbuild", d) > + finally: > + dpkg_undo_mounts(d) > + bb.utils.unlockfile(lock) > } > > addtask dpkg_build before do_build > @@ -221,16 +251,16 @@ python do_devshell() { > oe_lib_path = os.path.join(d.getVar('LAYERDIR_core'), 'lib') > sys.path.insert(0, oe_lib_path) > > - bb.build.exec_func('dpkg_do_mounts', d) > - > - isar_export_proxies(d) > - > - buildchroot = d.getVar('BUILDCHROOT_DIR') > - pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS')) > - termcmd = "sudo -E chroot {0} sh -c 'cd {1}; $SHELL -i'" > - oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d) > + dpkg_do_mounts(d) > + try: > + isar_export_proxies(d) > > - bb.build.exec_func('dpkg_undo_mounts', d) > + buildchroot = d.getVar('BUILDCHROOT_DIR') > + pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS')) > + termcmd = "sudo -E chroot {0} sh -c 'cd {1}; $SHELL -i'" > + oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d) > + finally: > + dpkg_undo_mounts(d) > } > > addtask devshell after do_prepare_build > diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass > index d956e8c..20d2d4c 100644 > --- a/meta/classes/dpkg-gbp.bbclass > +++ b/meta/classes/dpkg-gbp.bbclass > @@ -12,12 +12,7 @@ PATCHTOOL ?= "git" > GBP_DEPENDS ?= "git-buildpackage pristine-tar" > GBP_EXTRA_OPTIONS ?= "--git-pristine-tar" > > -do_install_builddeps_append() { > - dpkg_do_mounts > - distro="${DISTRO}" > - if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then > - distro="${HOST_DISTRO}" > - fi > +builddeps_install_append() { > deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}" > sudo -E chroot ${BUILDCHROOT_DIR} \ > apt-get install -y -o Debug::pkgProblemResolver=yes \ > @@ -26,7 +21,6 @@ do_install_builddeps_append() { > sudo -E chroot ${BUILDCHROOT_DIR} \ > apt-get install -y -o Debug::pkgProblemResolver=yes \ > --no-install-recommends ${GBP_DEPENDS} > - dpkg_undo_mounts > } > > dpkg_runbuild_prepend() { > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass > index 4e7c2f7..29e2b89 100644 > --- a/meta/classes/dpkg.bbclass > +++ b/meta/classes/dpkg.bbclass > @@ -6,9 +6,7 @@ inherit dpkg-base > PACKAGE_ARCH ?= "${DISTRO_ARCH}" > > # Install build dependencies for package > -do_install_builddeps() { > - dpkg_do_mounts > - E="${@ isar_export_proxies(d)}" > +builddeps_install() { > distro="${DISTRO}" > if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then > distro="${HOST_DISTRO}" > @@ -19,7 +17,15 @@ do_install_builddeps() { > deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}" > sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \ > ${PP}/${PPS} ${PACKAGE_ARCH} > - dpkg_undo_mounts > +} > + > +python do_install_builddeps() { > + dpkg_do_mounts(d) > + isar_export_proxies(d) > + try: > + bb.build.exec_func("builddeps_install", d) > + finally: > + dpkg_undo_mounts(d) > } > > addtask install_builddeps after do_prepare_build before do_dpkg_build > As written before, this is the only class applying try-finally for cleaning up mounts on failing task. One price is API breakage because dpkg_do_mounts and dpkg_undo_mounts are converted to python functions. Given that all other mount/unmout services do not do this extra effort, I would consider that price too high. Just let the build_completed handler clean up, that's what it's for. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux