From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6729358150130991104 X-Received: by 2002:a7b:c246:: with SMTP id b6mr20174755wmj.13.1566800790529; Sun, 25 Aug 2019 23:26:30 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:adf:eb44:: with SMTP id u4ls4142226wrn.2.gmail; Sun, 25 Aug 2019 23:26:30 -0700 (PDT) X-Google-Smtp-Source: APXvYqySS4sr+vghXJwPBj4bMJBUI+sxTXJJo71iA3cIm1n+Y26GbWgjI5EqmGMkMSAmBbONIAp7 X-Received: by 2002:adf:eec5:: with SMTP id a5mr20085699wrp.352.1566800790045; Sun, 25 Aug 2019 23:26:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566800790; cv=none; d=google.com; s=arc-20160816; b=u0jrje20qDkqMkfRp4GKXTMgvOfydjleM7OSPyYY4ZIseXByS8w9Ew2ufDKIgfHF6W 56UWZomZRvffTJkvZG6j0JEvbrnKZ2KH9qdXoYnsRjjKFW9EdqrewvkdIRdh+s/J1LWc vQWt9oXIw4E9CFhMmNe0vU9Myf40QU6KI2QDPDNj5P8tF7BgNu8FiMJR/wF3rmhtIUNM xTIV1GWvlqnlQDkEtlwl0Jh1G/6cknqpzIWbuiOCspDN4X7oMeer93mAp5vzXEgLuMmg drrz+mKB1vffCoi6MsbPg6xjFEK/EOVeIjTv75odqi25nj7EWVuTOKZgB0Wqp+1ePBZ+ /pVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from; bh=BtlPCv2uB/J/WV/tpZXB+/3vAOAeR2Pf4SzfN8BNrHw=; b=sWw1HGZ9faFWaJOO/KMRdJyBDhefYH4Gc40B9pGRXAn4euQO2O0gPAQydAvki/XOzr /Ql69r2k+pnXVhB3b935sQ5O2pW0wP/BmfaHMeMtB+D9PXaCONdSs1v/Nlt81s1up2A1 ZRhor7UPcRcKu7OW/zdB4M9AXZOCR+oCReAGT1KZ4CJsFEMH82ALPwJCc5SmFj3VNTdB ljdtbVm1TyotJqZPg92r5GtaG2Wrc3f/fSq34rdelD5/6SaGEtTyu5yP5ZXlVvwy0won EUtaeabzsSBhQeK9Moy44+1IpcqNtg2I1TAbDAE5RWYCu2Rd8rAhZiqINow+0zaZuoGp PAKg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id e7si420415wru.1.2019.08.25.23.26.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 25 Aug 2019 23:26:30 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id x7Q6QTqO013586 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 26 Aug 2019 08:26:29 +0200 Received: from md1f2u6c.ad001.siemens.net ([139.22.35.99]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id x7Q6QRGx015165; Mon, 26 Aug 2019 08:26:28 +0200 From: Jan Kiszka To: isar-users Cc: Cedric Hombourger Subject: [PATCH 05/26] dpkg: Fix shared-locking of build task Date: Mon, 26 Aug 2019 08:26:06 +0200 Message-Id: <0342bed12dfefb93ee9c2cc93282641a97a12042.1566800787.git.jan.kiszka@siemens.com> X-Mailer: git-send-email 2.16.4 In-Reply-To: References: In-Reply-To: References: X-TUID: YREQInFymCei From: Jan Kiszka Bitbake file locks are not just exclusive flock calls. For whatever reason, bitbake prefers to delete the lock file before unlocking them. That obviously creates a nasty race with other locker requests: Those already waiting will get access to the removed lock, those coming later will create a new one. To resolve that race, bitbakes lockfile() tests if the inode of the acquired lock file is identical to the one that is currently visible via the lock name. If not, it retries the lock. What a pointless mess... Our open-coded "flock -s" does not do that, thus will often acquire only a stale lock. Fix that by using bitbake's services for locking. That requires to move the lock acquisition at the top level of the task, ie. do_build. Thus, also dpkg-raw and kernel builds are now under the shared (read) lock. The better fix for this issue would actually be private buildchroots for each build, but that will come later. CC: Cedric Hombourger Fixes: 568072765b08 ("dpkg: acquire a read (shared) locks while building packages") Signed-off-by: Jan Kiszka --- meta/classes/dpkg-base.bbclass | 11 +++++++---- meta/classes/dpkg.bbclass | 5 ++--- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 2d26418..8146328 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -87,10 +87,13 @@ dpkg_runbuild() { die "This should never be called, overwrite it in your derived class" } -do_build() { - dpkg_do_mounts - dpkg_runbuild - dpkg_undo_mounts +python do_build() { + lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock", + shared=True) + bb.build.exec_func("dpkg_do_mounts", d) + bb.build.exec_func("dpkg_runbuild", d) + bb.build.exec_func("dpkg_undo_mounts", d) + bb.utils.unlockfile(lock) } CLEANFUNCS += "repo_clean" diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index dd123c3..df47ced 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -18,7 +18,6 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" # Build package from sources using build script dpkg_runbuild() { E="${@ bb.utils.export_proxies(d)}" - flock -s "${REPO_ISAR_DIR}/isar.lock" \ - sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ - /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH} + sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ + /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH} } -- 2.16.4