From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6729358150130991104 X-Received: by 2002:a05:651c:20d:: with SMTP id y13mr9942132ljn.112.1566808257043; Mon, 26 Aug 2019 01:30:57 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:4201:: with SMTP id y1ls1432627lfh.9.gmail; Mon, 26 Aug 2019 01:30:56 -0700 (PDT) X-Google-Smtp-Source: APXvYqxIcccMqfNjwXTH6lInLT1AJYMqdbs/g60QzASY1SpNW5LvP5HH/YjCq/4LKTQbFHoRXyBh X-Received: by 2002:ac2:50c9:: with SMTP id h9mr10176904lfm.51.1566808256576; Mon, 26 Aug 2019 01:30:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566808256; cv=none; d=google.com; s=arc-20160816; b=gi59WOl4xgTXLvfAEmik5TaoFL2vwuNCj8FWyEF/4f032R8D3Qad1DSeMFAYUM9Zb/ qhVy1acBDC7TVivr2fmanCEjL+v+twmkAM0wRnqIqASZVuw677fa+AhTVr9oE2zPqYxt 1AX/ctJX8wMC0RhFH/ZDnt3uPVIebZj8IdDIpBlkemCFo/ihQ6E9Gu4UkfBbYMDxoPyg mx1UWy6Ro+VWEfYY2GoprP2Vg2KSlvXjVmrB4yRR9S57bkO83qpbv31uHcoc8GmbydIL dQuYEa4erBeKjwkavYt14Oucs1NVEi04qzdVXB2q66jRgYGQHYq1I1sjzc16v7oWcWNA sj6A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject; bh=bsTyrIPneqadNinvVl+lm9YCXjRCSsWnkHvyxU25yCI=; b=cW7arwYVAARPnDeTcg9c5LpMlsozcRD1bYbYD/pn0XTqi/G8iA04MCE1CfBUvvqnY9 azRINyQ3us52fwc7q72WHsPUqkfsjnputLDuaFqt7DyHqgyY7fG1wWrVWiI//HU/19qz Nd82X8OOdMdaGQWW0G1AGpI6rTOr1cmkIe/C3ZA35/3VeX69wOMhsVKNkBH9fh55hiUL PfP6ADUOjb8fv8DH9LibbygVMHwi+Gyg/RTLBpjyxP3XdIQksE5tuM8M05adryIx38q8 EhOD+/BIp8o9a758D7skZwvXotSepFkHn4kJRKqkAVk5D93v1p5Bq4EHQPsodfUx8rph XTFg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id h11si500646lja.2.2019.08.26.01.30.56 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Aug 2019 01:30:56 -0700 (PDT) Received-SPF: pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id x7Q8UtM6001182 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 26 Aug 2019 10:30:55 +0200 Received: from [139.25.69.208] (linux-ses-ext02.ppmd.siemens.net [139.25.69.208]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id x7Q8Ut1j009383; Mon, 26 Aug 2019 10:30:55 +0200 Subject: Re: [PATCH 05/26] dpkg: Fix shared-locking of build task To: "[ext] Jan Kiszka" , isar-users Cc: Cedric Hombourger References: <0342bed12dfefb93ee9c2cc93282641a97a12042.1566800787.git.jan.kiszka@siemens.com> From: Claudius Heine Message-ID: <68263e5e-36ba-16d0-be9b-e8dfc8697ccf@siemens.com> Date: Mon, 26 Aug 2019 10:30:55 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <0342bed12dfefb93ee9c2cc93282641a97a12042.1566800787.git.jan.kiszka@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: fxwIwuFdj+dY Hi Jan, On 26/08/2019 08.26, [ext] Jan Kiszka wrote: > From: Jan Kiszka > > Bitbake file locks are not just exclusive flock calls. For whatever > reason, bitbake prefers to delete the lock file before unlocking them. > That obviously creates a nasty race with other locker requests: Those > already waiting will get access to the removed lock, those coming later > will create a new one. To resolve that race, bitbakes lockfile() tests > if the inode of the acquired lock file is identical to the one that is > currently visible via the lock name. If not, it retries the lock. What a > pointless mess... > > Our open-coded "flock -s" does not do that, thus will often acquire only > a stale lock. Fix that by using bitbake's services for locking. That > requires to move the lock acquisition at the top level of the task, ie. > do_build. Thus, also dpkg-raw and kernel builds are now under the shared > (read) lock. The better fix for this issue would actually be private > buildchroots for each build, but that will come later. > > CC: Cedric Hombourger > Fixes: 568072765b08 ("dpkg: acquire a read (shared) locks while building packages") > Signed-off-by: Jan Kiszka > --- > meta/classes/dpkg-base.bbclass | 11 +++++++---- > meta/classes/dpkg.bbclass | 5 ++--- > 2 files changed, 9 insertions(+), 7 deletions(-) > > diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass > index 2d26418..8146328 100644 > --- a/meta/classes/dpkg-base.bbclass > +++ b/meta/classes/dpkg-base.bbclass > @@ -87,10 +87,13 @@ dpkg_runbuild() { > die "This should never be called, overwrite it in your derived class" > } > > -do_build() { > - dpkg_do_mounts > - dpkg_runbuild > - dpkg_undo_mounts > +python do_build() { > + lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock", > + shared=True) > + bb.build.exec_func("dpkg_do_mounts", d) > + bb.build.exec_func("dpkg_runbuild", d) > + bb.build.exec_func("dpkg_undo_mounts", d) > + bb.utils.unlockfile(lock) What is the difference between this and do_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" ? regards, Claudius > } > > CLEANFUNCS += "repo_clean" > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass > index dd123c3..df47ced 100644 > --- a/meta/classes/dpkg.bbclass > +++ b/meta/classes/dpkg.bbclass > @@ -18,7 +18,6 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" > # Build package from sources using build script > dpkg_runbuild() { > E="${@ bb.utils.export_proxies(d)}" > - flock -s "${REPO_ISAR_DIR}/isar.lock" \ > - sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ > - /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH} > + sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ > + /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH} > } > -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de