From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6729358150130991104 X-Received: by 2002:a19:2d15:: with SMTP id k21mr9908049lfj.188.1566808733502; Mon, 26 Aug 2019 01:38:53 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a19:8c0c:: with SMTP id o12ls1434901lfd.12.gmail; Mon, 26 Aug 2019 01:38:52 -0700 (PDT) X-Google-Smtp-Source: APXvYqxM+aW7DLiEwrpdREdYKLjCeUP09avtYIsoxnxvsXSGY8AzHxOuRQZbrSKleVNfCL8HP7r+ X-Received: by 2002:a05:6512:481:: with SMTP id v1mr1418111lfq.120.1566808732882; Mon, 26 Aug 2019 01:38:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566808732; cv=none; d=google.com; s=arc-20160816; b=nCT1BU7mTjehHP6e6jgfqoSV5NtitmwSWDx/VWoxTBp+mKaDVXSZhkocGjFzamcppn fHFfHcSCJrekVXJaWMsNh4raIaihqJjyLkGYttEEnZw5ngyy+ASwrGpfHzkXPBOOnvnh WFMc3cqIXCpjYGKhvGAMlRf1Y8nb7ZzdILBbFBt5oNnCR+AloSAl96SnEn9MPo0AnM1l EEH8Z5KEU+9ZG5ShxvZ01ZcuSj89MTSNDQAY6J6ZbBhVrrZe3F/3YBe8PQxlJliPD3A9 kvcZYKYiaEN513xRZ15B6RjtOwB5q75717ePFp+D3k2s2wRx43C1FhVIufVNr66BBX8h 7gOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject; bh=SN+cWn5xg3dUCahv5Fip/JgxgNTnVXl3hX+nXtdhLqM=; b=pmZBDiZqKZFmwcxAEfz4zycYS3jAGUqZ3Q2DLPwQydY8vayLAIqBOgMRsnbo/N+3xK xukb6rSUYdG1RNDzFRkLoU7BkH1r3izkYcVAH2UMPM4qFiNyGiGP/ADI072gdenqkl3z cjTLoM0M1KlUo/w5IeTS2KIBpQ/ENWrPdmRKYBLUDGvEO1qwAXthFnm93jxPWjOf93Rv hyN/ukLA0shcKyaaaKhFcraTsp9vPaVjGAXonetnjRHro44f5fVv/XLURaPVvi48veny pbiiLhrwFA0XwZz5ZmZwhrayV8dzria8JfVCDnU/URSx0T2rThjA4fBsgbix2Rf6JNZ1 K+Tw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from gecko.sbs.de (gecko.sbs.de. [194.138.37.40]) by gmr-mx.google.com with ESMTPS id a9si64954lfk.5.2019.08.26.01.38.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Aug 2019 01:38:52 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) client-ip=194.138.37.40; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by gecko.sbs.de (8.15.2/8.15.2) with ESMTPS id x7Q8cqK6017166 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 26 Aug 2019 10:38:52 +0200 Received: from [139.22.35.99] ([139.22.35.99]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id x7Q8cpOv022876; Mon, 26 Aug 2019 10:38:51 +0200 Subject: Re: [PATCH 05/26] dpkg: Fix shared-locking of build task To: Claudius Heine , isar-users Cc: Cedric Hombourger References: <0342bed12dfefb93ee9c2cc93282641a97a12042.1566800787.git.jan.kiszka@siemens.com> <68263e5e-36ba-16d0-be9b-e8dfc8697ccf@siemens.com> From: Jan Kiszka Message-ID: Date: Mon, 26 Aug 2019 10:38:50 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <68263e5e-36ba-16d0-be9b-e8dfc8697ccf@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: 7d/tdDlsXYxG On 26.08.19 10:30, Claudius Heine wrote: > Hi Jan, > > On 26/08/2019 08.26, [ext] Jan Kiszka wrote: >> From: Jan Kiszka >> >> Bitbake file locks are not just exclusive flock calls. For whatever >> reason, bitbake prefers to delete the lock file before unlocking them. >> That obviously creates a nasty race with other locker requests: Those >> already waiting will get access to the removed lock, those coming later >> will create a new one. To resolve that race, bitbakes lockfile() tests >> if the inode of the acquired lock file is identical to the one that is >> currently visible via the lock name. If not, it retries the lock. What a >> pointless mess... >> >> Our open-coded "flock -s" does not do that, thus will often acquire only >> a stale lock. Fix that by using bitbake's services for locking. That >> requires to move the lock acquisition at the top level of the task, ie. >> do_build. Thus, also dpkg-raw and kernel builds are now under the shared >> (read) lock. The better fix for this issue would actually be private >> buildchroots for each build, but that will come later. >> >> CC: Cedric Hombourger >> Fixes: 568072765b08 ("dpkg: acquire a read (shared) locks while building >> packages") >> Signed-off-by: Jan Kiszka >> --- >>   meta/classes/dpkg-base.bbclass | 11 +++++++---- >>   meta/classes/dpkg.bbclass      |  5 ++--- >>   2 files changed, 9 insertions(+), 7 deletions(-) >> >> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass >> index 2d26418..8146328 100644 >> --- a/meta/classes/dpkg-base.bbclass >> +++ b/meta/classes/dpkg-base.bbclass >> @@ -87,10 +87,13 @@ dpkg_runbuild() { >>       die "This should never be called, overwrite it in your derived class" >>   } >> -do_build() { >> -    dpkg_do_mounts >> -    dpkg_runbuild >> -    dpkg_undo_mounts >> +python do_build() { >> +    lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock", >> +                             shared=True) >> +    bb.build.exec_func("dpkg_do_mounts", d) >> +    bb.build.exec_func("dpkg_runbuild", d) >> +    bb.build.exec_func("dpkg_undo_mounts", d) >> +    bb.utils.unlockfile(lock) > > What is the difference between this and > > do_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" > shared=True. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux