From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6982225001537601536 X-Received: by 2002:adf:fd86:: with SMTP id d6mr34241755wrr.84.1625746581475; Thu, 08 Jul 2021 05:16:21 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:1c9d:: with SMTP id k29ls1022322wms.0.gmail; Thu, 08 Jul 2021 05:16:20 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwAIF+kNQMfs7z+ndFuxH4V0fIoVmHvxSU0MVvU91fyLxB7f0BQZsk0XmL+O/zakX9Q4SKk X-Received: by 2002:a1c:1dc6:: with SMTP id d189mr33392481wmd.154.1625746580520; Thu, 08 Jul 2021 05:16:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1625746580; cv=none; d=google.com; s=arc-20160816; b=knvwNkqR/gWyR4b1mTMOt/LmtUSMf+vUQvIf6c/W0qfLd4oQVqyDbBMkg38DPDFSL0 K2QT9YEJ4w98HyG7TJ1SM9akTM2Eg7vl4aR/9mt2VNHkJ3YUV0O7ONEjLQjMhdfASer5 bVJ6FNSzv5hBlLCEjb7UZuCdKXJHOylFyZV+o5nm6WWHmgd5Xyv3f4U8ftw0XTLehaAg sbODF1Uwpg06NExLaO23tCGOtraiXvwljlGXLZo5Vove4Ihnaw/bDjhacd1fwI1zoKm4 rhsAhXFy5Tl79oh+aKVS8b6MFopOewh3T08qSQO721YTvHoaHIuEFGaYKTGtea1R7cv4 z4Lg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject; bh=D/n9AFN95QXTXVSbtn4CKqnzHwSWp036eHfyNvJ3Iyg=; b=JPBNPpd9WNuoxZuFiriSZ2S2nRx7jCKlo+qcyoDKyBgMMLpo7aZ0T1E/EzlIQTv+F3 3aFb303gYHX7vgVoAeph4kQH3p9Ywm/XahNpQaGlCFffoN9VV+OO66dWmPR3NAaf25nK 2KEHuCIoK0QiGTpfNY4HAuZ4tR/xJedMF9z4h5WmCBTK7oh1oS1pWI9LWIHZEiwvzUAI /oKA8SBIljnPtGundG6lUzkP18TNQ4NQx2ELfDFv+Ks2im/ssYzDZL52Ozm+VxnoZ2jJ EMu5qKjEKp8TmoDdRg9tjYGBByg5UAB0/EBvOIcDB1he27l23rcKw5iY3x7lDsa/FWfZ D3eA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id b6si90049wro.3.2021.07.08.05.16.20 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jul 2021 05:16:20 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id 168CGJuC007706 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 8 Jul 2021 14:16:20 +0200 Received: from [167.87.42.31] ([167.87.42.31]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 168CGJWo010433; Thu, 8 Jul 2021 14:16:19 +0200 Subject: Re: [PATCH v1 1/5] Revert "dpkg: Make mount buildroot reliable" To: Anton Mikanovich , isar-users@googlegroups.com References: <20210707163851.204296-1-amikan@ilbers.de> <20210707163851.204296-2-amikan@ilbers.de> From: Jan Kiszka Message-ID: <24c1da80-550c-8e40-aea8-c635a75380e8@siemens.com> Date: Thu, 8 Jul 2021 14:16:19 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <20210707163851.204296-2-amikan@ilbers.de> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: yO1mWjl+jwoq On 07.07.21 18:38, Anton Mikanovich wrote: > This reverts commit d21d49578e5a3b0019075d1946bd93a95914fcca. Signed-off and a short reasoning is missing. Jan > --- > meta/classes/dpkg-base.bbclass | 127 +++++++++++++-------------------- > meta/classes/dpkg-gbp.bbclass | 8 ++- > meta/classes/dpkg.bbclass | 14 ++-- > 3 files changed, 59 insertions(+), 90 deletions(-) > > diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass > index ed162b3..97661a6 100644 > --- a/meta/classes/dpkg-base.bbclass > +++ b/meta/classes/dpkg-base.bbclass > @@ -61,7 +61,12 @@ addtask patch before do_adjust_git > > SRC_APT ?= "" > > -fetch_apt() { > +do_apt_fetch() { > + if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then > + return 0 > + fi > + dpkg_do_mounts > + E="${@ isar_export_proxies(d)}" > sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \ > -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \ > -o Dir::Etc::SourceParts="-" \ > @@ -71,25 +76,21 @@ fetch_apt() { > sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ > sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}" > done > -} > - > -python do_apt_fetch() { > - src_apt = d.getVar("SRC_APT", True) > - if not src_apt: > - return 0 > > - dpkg_do_mounts(d) > - try: > - isar_export_proxies(d) > - bb.build.exec_func("fetch_apt", d) > - finally: > - dpkg_undo_mounts(d) > + dpkg_undo_mounts > } > > addtask apt_fetch after do_unpack before do_apt_unpack > do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" > > -unpack_apt() { > +do_apt_unpack() { > + if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then > + return 0 > + fi > + rm -rf ${S} > + dpkg_do_mounts > + E="${@ isar_export_proxies(d)}" > + > for uri in "${SRC_APT}"; do > sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ > sh -c ' \ > @@ -100,25 +101,8 @@ unpack_apt() { > dpkg-source -x "${dscfile}" "${PPS}"' \ > my_script "${DISTRO}" "${uri}" > done > -} > > -python do_apt_unpack() { > - import shutil > - > - src_apt = d.getVar("SRC_APT", True) > - if not src_apt: > - return 0 > - > - srcsubdir = d.getVar('S', True) > - if os.path.exists(srcsubdir): > - shutil.rmtree(srcsubdir) > - > - dpkg_do_mounts(d) > - try: > - isar_export_proxies(d) > - bb.build.exec_func("unpack_apt", d) > - finally: > - dpkg_undo_mounts(d) > + dpkg_undo_mounts > } > > addtask apt_unpack after do_apt_fetch before do_patch > @@ -162,38 +146,25 @@ do_prepare_build[deptask] = "do_deploy_deb" > > BUILDROOT = "${BUILDCHROOT_DIR}/${PP}" > > -def ismount(path): > - real = os.path.realpath(path) > - with open('/proc/mounts') as f: > - for line in f.readlines(): > - if len(line.split()) > 2 and real == line.split()[1]: > - return True > - return False > - > -def dpkg_do_mounts(d): > - buildroot = d.getVar('BUILDROOT', True) > - if ismount(buildroot): > - bb.warn('Path %s already mounted!' % buildroot) > - return > - workdir = d.getVar('WORKDIR', True) > - os.makedirs(buildroot, exist_ok=True) > - os.system('sudo mount --bind %s %s' % (workdir, buildroot)) > - bb.build.exec_func("buildchroot_do_mounts", d) > - > -def dpkg_undo_mounts(d): > - bb.build.exec_func("buildchroot_undo_mounts", d) > - buildroot = d.getVar('BUILDROOT', True) > - if not ismount(buildroot): > - bb.warn('Path %s not mounted!' % buildroot) > - return > - for i in range(200): > - if not os.system('sudo umount %s' % buildroot): > - os.rmdir(buildroot) > - return > - if i % 100 == 0: > - bb.warn("%s: Couldn't unmount, retrying..." % buildroot) > - time.sleep(0.1) > - bb.fatal("Couldn't unmount, exiting...") > +dpkg_do_mounts() { > + mkdir -p ${BUILDROOT} > + sudo mount --bind ${WORKDIR} ${BUILDROOT} > + > + buildchroot_do_mounts > +} > + > +dpkg_undo_mounts() { > + i=1 > + while ! sudo umount ${BUILDROOT}; do > + sleep 0.1 > + i=`expr $i + 1` > + if [ $i -gt 100 ]; then > + bbwarn "${BUILDROOT}: Couldn't unmount, retrying..." > + i=1 > + fi > + done > + sudo rmdir ${BUILDROOT} > +} > > # Placeholder for actual dpkg_runbuild() implementation > dpkg_runbuild() { > @@ -203,12 +174,10 @@ dpkg_runbuild() { > python do_dpkg_build() { > lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock", > shared=True) > - dpkg_do_mounts(d) > - try: > - bb.build.exec_func("dpkg_runbuild", d) > - finally: > - dpkg_undo_mounts(d) > - bb.utils.unlockfile(lock) > + bb.build.exec_func("dpkg_do_mounts", d) > + bb.build.exec_func("dpkg_runbuild", d) > + bb.build.exec_func("dpkg_undo_mounts", d) > + bb.utils.unlockfile(lock) > } > > addtask dpkg_build before do_build > @@ -252,16 +221,16 @@ python do_devshell() { > oe_lib_path = os.path.join(d.getVar('LAYERDIR_core'), 'lib') > sys.path.insert(0, oe_lib_path) > > - dpkg_do_mounts(d) > - try: > - isar_export_proxies(d) > + bb.build.exec_func('dpkg_do_mounts', d) > + > + isar_export_proxies(d) > + > + buildchroot = d.getVar('BUILDCHROOT_DIR') > + pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS')) > + termcmd = "sudo -E chroot {0} sh -c 'cd {1}; $SHELL -i'" > + oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d) > > - buildchroot = d.getVar('BUILDCHROOT_DIR') > - pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS')) > - termcmd = "sudo -E chroot {0} sh -c 'cd {1}; $SHELL -i'" > - oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d) > - finally: > - dpkg_undo_mounts(d) > + bb.build.exec_func('dpkg_undo_mounts', d) > } > > addtask devshell after do_prepare_build > diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass > index 20d2d4c..d956e8c 100644 > --- a/meta/classes/dpkg-gbp.bbclass > +++ b/meta/classes/dpkg-gbp.bbclass > @@ -12,7 +12,12 @@ PATCHTOOL ?= "git" > GBP_DEPENDS ?= "git-buildpackage pristine-tar" > GBP_EXTRA_OPTIONS ?= "--git-pristine-tar" > > -builddeps_install_append() { > +do_install_builddeps_append() { > + dpkg_do_mounts > + distro="${DISTRO}" > + if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then > + distro="${HOST_DISTRO}" > + fi > deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}" > sudo -E chroot ${BUILDCHROOT_DIR} \ > apt-get install -y -o Debug::pkgProblemResolver=yes \ > @@ -21,6 +26,7 @@ builddeps_install_append() { > sudo -E chroot ${BUILDCHROOT_DIR} \ > apt-get install -y -o Debug::pkgProblemResolver=yes \ > --no-install-recommends ${GBP_DEPENDS} > + dpkg_undo_mounts > } > > dpkg_runbuild_prepend() { > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass > index 29e2b89..4e7c2f7 100644 > --- a/meta/classes/dpkg.bbclass > +++ b/meta/classes/dpkg.bbclass > @@ -6,7 +6,9 @@ inherit dpkg-base > PACKAGE_ARCH ?= "${DISTRO_ARCH}" > > # Install build dependencies for package > -builddeps_install() { > +do_install_builddeps() { > + dpkg_do_mounts > + E="${@ isar_export_proxies(d)}" > distro="${DISTRO}" > if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then > distro="${HOST_DISTRO}" > @@ -17,15 +19,7 @@ builddeps_install() { > deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}" > sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \ > ${PP}/${PPS} ${PACKAGE_ARCH} > -} > - > -python do_install_builddeps() { > - dpkg_do_mounts(d) > - isar_export_proxies(d) > - try: > - bb.build.exec_func("builddeps_install", d) > - finally: > - dpkg_undo_mounts(d) > + dpkg_undo_mounts > } > > addtask install_builddeps after do_prepare_build before do_dpkg_build >