From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6729358150130991104 X-Received: by 2002:aa7:851a:: with SMTP id v26mr18179465pfn.238.1566801550286; Sun, 25 Aug 2019 23:39:10 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a65:68c8:: with SMTP id k8ls3444539pgt.8.gmail; Sun, 25 Aug 2019 23:39:09 -0700 (PDT) X-Google-Smtp-Source: APXvYqyBrMjoOeXVDRIiP5RinFz12gzmIvUIo4HL6XU5Zu48+U/3emNFm02cqUyhJUqSmTG8XMBx X-Received: by 2002:a62:60c7:: with SMTP id u190mr18699345pfb.54.1566801549898; Sun, 25 Aug 2019 23:39:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566801549; cv=none; d=google.com; s=arc-20160816; b=hD+Kk/MtiYVxo2IuauHhK6QS4XBtMkcyqwZuqeYhn3C2x8zDsnOiTVx/wELrdCfPsY 6zrHq7Wc+na0/PpzSM7PRv/nPcyJS0YzZ2QnMlqeriGNZI8szwWLmZvY7g4GI19eJWUf 8bbXAs4EpKJoBy4K0ISyOKyeejSGLZTS17jyX2mu6/t7W2BrcVT6PTsdsizNAnp1QtWV yiY+TPSuxsw3TJwxItsR4vpZqpcB5Oj6lp2flyNUKrPURN93HyUXpxCSPYimmA/pNKVN nl3xEzZ+eMtCL/gq5W+0CHFYsl1V3D2mUAe/cWUeXgyZrQgieo12D4miuGeYte5nLwNc +wVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-language:content-transfer-encoding:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject:ironport-sdr :ironport-sdr; bh=2Tg7VvYdo1Zz5Ze/34fum+wS0jwP1BqfcFgnSfniFtY=; b=VIJN05cqQ9/WSf9tFGxUSwJX3eulYJa+hcYRD2SsXy1SVW4dail0XLNllgly6VDxWr BMmW7lQ5HzFpwNWv3NKT1d09+3BHASkYDKo7+me1sx8kxkMtWvgLetH3hmf9X0IE5ein +2BlQBsjzyF8xE7Fp3uvrsypGgDBRaWZedZKViBZDlStZhWp3ShfYWmX/Hu5r32qx9mg h1izbbVm8CL9ou4aGAe+6O0AkgF/BHNy8sXkUrvT1EGavKjMfECvA8Ol4mG1lKe8KWvO 4NomkSRT/WboLQq2LxRxELifB8cOsU9YJXq8dvmullpoN0l4DDaDowu7kLoYP7a1/GbF vOVQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of cedric_hombourger@mentor.com designates 68.232.137.252 as permitted sender) smtp.mailfrom=Cedric_Hombourger@mentor.com Return-Path: Received: from esa4.mentor.iphmx.com (esa4.mentor.iphmx.com. [68.232.137.252]) by gmr-mx.google.com with ESMTPS id f125si545327pgc.4.2019.08.25.23.39.09 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 25 Aug 2019 23:39:09 -0700 (PDT) Received-SPF: pass (google.com: domain of cedric_hombourger@mentor.com designates 68.232.137.252 as permitted sender) client-ip=68.232.137.252; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of cedric_hombourger@mentor.com designates 68.232.137.252 as permitted sender) smtp.mailfrom=Cedric_Hombourger@mentor.com IronPort-SDR: PyLY2lglPL15+waYroQvnqtnRZz7QSNpILAAa0sYomkK2VYalK2TVetQ9Fi+wnyxtuGDeQ5hpK 2BeN7fJFIgsDTGUOuiv0fkWIwOKpc2t8XzU8fB39hTDzWE2L/9WlK65RHwJQnL1pmr620u4egA LdqMNLj4GTAda2+RByEhaIQ6Q2FqNh6UjmSdH6BpArzXOCKEkpQGM8Gt+DUBB8R3aTjHYgV6IJ Bf2FAxN0e8sTikjPu4HjXlGOOcJXS2WkLqk9nrk15zkTXamYX9CQnTS7bYqql311ES0f6qOmV0 KxM= X-IronPort-AV: E=Sophos;i="5.64,431,1559548800"; d="scan'208";a="40758757" Received: from orw-gwy-01-in.mentorg.com ([192.94.38.165]) by esa4.mentor.iphmx.com with ESMTP; 25 Aug 2019 22:39:09 -0800 IronPort-SDR: kYXWJsJK69CaJ1KTLcXu4koYJ3N3VZrrea7bezMqg4FleHrh3KKW3dIO0OmxlhMKuvZT7iL/Gz 0dt5pu49BMIka3Qng+w/lpIRBknsjLW45eqUtjyFqrKQzU752DSr2QxPaLUUQDnGCEUQRnQFyP uZM3pnSeqkUu3ELzsrCpXPqktbXoJJQ+YVGYnkAHuXWpUhPt23B6yl2+DNov9u56iMfh9k19Yb urBQtzxT/F14eaF9Dc33MrKkV+yiRN7lr1IQ9MqJSHlIXID3qf6sht0mc5KtQcdp+AvvWnfoAU cIo= Subject: Re: [PATCH 05/26] dpkg: Fix shared-locking of build task To: Jan Kiszka , isar-users References: <0342bed12dfefb93ee9c2cc93282641a97a12042.1566800787.git.jan.kiszka@siemens.com> From: Cedric Hombourger Message-ID: <5668b1d7-2b16-bebd-a605-dac9f0a7823e@mentor.com> Date: Mon, 26 Aug 2019 08:38:42 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <0342bed12dfefb93ee9c2cc93282641a97a12042.1566800787.git.jan.kiszka@siemens.com> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US Return-Path: cedric_hombourger@mentor.com X-Originating-IP: [137.202.0.90] X-ClientProxiedBy: SVR-IES-MBX-09.mgc.mentorg.com (139.181.222.9) To svr-ies-mbx-02.mgc.mentorg.com (139.181.222.2) X-TUID: EHBWTjfc3DKZ On 8/26/2019 8:26 AM, Jan Kiszka wrote: > From: Jan Kiszka > > Bitbake file locks are not just exclusive flock calls. For whatever > reason, bitbake prefers to delete the lock file before unlocking them. > That obviously creates a nasty race with other locker requests: Those > already waiting will get access to the removed lock, those coming later > will create a new one. To resolve that race, bitbakes lockfile() tests > if the inode of the acquired lock file is identical to the one that is > currently visible via the lock name. If not, it retries the lock. What a > pointless mess... > > Our open-coded "flock -s" does not do that, thus will often acquire only > a stale lock. Fix that by using bitbake's services for locking. That > requires to move the lock acquisition at the top level of the task, ie. > do_build. Thus, also dpkg-raw and kernel builds are now under the shared > (read) lock. The better fix for this issue would actually be private > buildchroots for each build, but that will come later. Very good catch! ack > CC: Cedric Hombourger > Fixes: 568072765b08 ("dpkg: acquire a read (shared) locks while building packages") > Signed-off-by: Jan Kiszka > --- > meta/classes/dpkg-base.bbclass | 11 +++++++---- > meta/classes/dpkg.bbclass | 5 ++--- > 2 files changed, 9 insertions(+), 7 deletions(-) > > diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass > index 2d26418..8146328 100644 > --- a/meta/classes/dpkg-base.bbclass > +++ b/meta/classes/dpkg-base.bbclass > @@ -87,10 +87,13 @@ dpkg_runbuild() { > die "This should never be called, overwrite it in your derived class" > } > > -do_build() { > - dpkg_do_mounts > - dpkg_runbuild > - dpkg_undo_mounts > +python do_build() { > + lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock", > + shared=True) > + bb.build.exec_func("dpkg_do_mounts", d) > + bb.build.exec_func("dpkg_runbuild", d) > + bb.build.exec_func("dpkg_undo_mounts", d) > + bb.utils.unlockfile(lock) > } > > CLEANFUNCS += "repo_clean" > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass > index dd123c3..df47ced 100644 > --- a/meta/classes/dpkg.bbclass > +++ b/meta/classes/dpkg.bbclass > @@ -18,7 +18,6 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" > # Build package from sources using build script > dpkg_runbuild() { > E="${@ bb.utils.export_proxies(d)}" > - flock -s "${REPO_ISAR_DIR}/isar.lock" \ > - sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ > - /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH} > + sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ > + /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH} > }