From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6592507174472122368 X-Received: by 2002:a65:580c:: with SMTP id g12-v6mr13666046pgr.119.1534942039613; Wed, 22 Aug 2018 05:47:19 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:902:6a88:: with SMTP id n8-v6ls382456plk.8.gmail; Wed, 22 Aug 2018 05:47:19 -0700 (PDT) X-Google-Smtp-Source: AA+uWPz0tpX+mVIwbhQrAcj8l7DQe8IXPhdr5d5dP1yrulfzAJIMI7saBi6viVObmqYhFP0ZlvUM X-Received: by 2002:a17:902:b206:: with SMTP id t6-v6mr13024212plr.30.1534942039325; Wed, 22 Aug 2018 05:47:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1534942039; cv=none; d=google.com; s=arc-20160816; b=kGV3taDZroHDsQVPGPjjoIZOEGiJLXyiphNCfEY6syTscdKwruWx8cgLRGtrsxLYUt aIqTIEz8+6y9tLX0U6VfFTPrxjydYlgXgsAsP/u2Y8y5pE3k3xbZuCxhunYXwxzs0LUh GXiJszbIEtE/csZaIZ5ITYWGdQbBvPyfIrToFzobNiBqZREc15+2h/7qf0b2Ni26hN74 r1g2GV+ztNCvRvDDUlNFrLkxPCzszXW+NZA8zx2sRr84vHM14WOjg/QmArvE0HacBlBq Jot/wKgngnLMw/HbPVzfJr5ML57aH4wSE/nP+0vxB2jAh249zP1iFh4IliV3PM/DTfc0 rjIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:arc-authentication-results; bh=PZg090k1BNkVdHSUs1NwbJtwtBribwRZcgjGE8+DG4g=; b=xw4OTIRNldLV+2Ir0QJLfXhO9FbuVuWBI5HeLfkGAu7kv5VdeQCruJcSrt1BC1lMSe YjyBZuOrwCKWuzSxjjUHT3tniyWkp8i2uoRjD6jh9Hp1KlxmhrzDkc6uCFS0i9bc94hK wJRIBPXXJPemWaF4EVfOnLd3yXroiZNpUsVadoHzd1KmDUmGDZNB4kz4Puz2reGLklVU 5E9ndsV0iL4m6H1CJr3VSqGB0hNLn+6p9hibndY4XZNHFSiH1Xn2KSac0UQae7MS01Sc 96rRmPomzJsSM65P9PLkkFSylMX8R0UHWq6v2HKa1BmmL8ccZA7ggyapYUUZbUmWmFmz 211g== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of cedric_hombourger@mentor.com designates 192.94.38.131 as permitted sender) smtp.mailfrom=Cedric_Hombourger@mentor.com Return-Path: Received: from relay1.mentorg.com (relay1.mentorg.com. [192.94.38.131]) by gmr-mx.google.com with ESMTPS id h1-v6si42634pfi.4.2018.08.22.05.47.19 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Aug 2018 05:47:19 -0700 (PDT) Received-SPF: pass (google.com: domain of cedric_hombourger@mentor.com designates 192.94.38.131 as permitted sender) client-ip=192.94.38.131; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of cedric_hombourger@mentor.com designates 192.94.38.131 as permitted sender) smtp.mailfrom=Cedric_Hombourger@mentor.com Received: from nat-ies.mentorg.com ([192.94.31.2] helo=svr-ies-mbx-02.mgc.mentorg.com) by relay1.mentorg.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-SHA384:256) id 1fsSXe-0002ke-QC from Cedric_Hombourger@mentor.com for isar-users@googlegroups.com; Wed, 22 Aug 2018 05:47:19 -0700 Received: from FRG-W10-HOMBOUR.world.mentorg.com (137.202.0.87) by svr-ies-mbx-02.mgc.mentorg.com (139.181.222.2) with Microsoft SMTP Server (TLS) id 15.0.1320.4; Wed, 22 Aug 2018 13:47:14 +0100 From: Cedric Hombourger To: CC: Cedric Hombourger Subject: [PATCH v2] dpkg: acquire lock when calling reprepro Date: Wed, 22 Aug 2018 14:47:07 +0200 Message-ID: <1534942027-7806-1-git-send-email-Cedric_Hombourger@mentor.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1534937676-7628-1-git-send-email-Cedric_Hombourger@mentor.com> References: <1534937676-7628-1-git-send-email-Cedric_Hombourger@mentor.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [137.202.0.87] X-ClientProxiedBy: svr-ies-mbx-01.mgc.mentorg.com (139.181.222.1) To svr-ies-mbx-02.mgc.mentorg.com (139.181.222.2) X-TUID: YqLMwbcqqDey As bitbake performs multi-threaded builds, packages may be built while others get deployed. A lock was already acquired within dpkg_runbuild but accesses to the isar-apt database were not protected from do_deploy_deb. Move the lock from the buildchroot to the isar-apt repository and acquire the (same) lock from dpkg_runbuild, repo_clean and do_deploy_deb. Signed-off-by: Cedric Hombourger --- meta/classes/dpkg-base.bbclass | 9 ++++++--- meta/recipes-devtools/buildchroot/files/build.sh | 2 +- meta/recipes-devtools/isar-apt/isar-apt-host.bb | 4 ++-- meta/recipes-devtools/isar-apt/isar-apt.bb | 4 ++-- meta/recipes-kernel/linux/files/build-kernel.sh | 2 +- 5 files changed, 12 insertions(+), 9 deletions(-) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index a41df2c..108d942 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -71,6 +71,8 @@ do_build() { mount -t proc none ${BUILDCHROOT_DIR}/proc fi' + # Create database lock file here so we own it (not root) and perform actual build + touch ${DEPLOY_DIR_APT}/${DISTRO}/db.lock dpkg_runbuild sudo umount ${BUILDROOT} 2>/dev/null || true @@ -82,7 +84,8 @@ CLEANFUNCS += "repo_clean" repo_clean() { PACKAGES=$(cd ${S}/..; ls *.deb | sed 's/\([^_]*\).*/\1/') if [ -n "${PACKAGES}" ]; then - reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \ + flock ${DEPLOY_DIR_APT}/${DISTRO}/db.lock \ + reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \ --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \ -C main -A ${DISTRO_ARCH} \ remove ${DEBDISTRONAME} \ @@ -93,7 +96,8 @@ repo_clean() { # Install package to Isar-apt do_deploy_deb() { repo_clean - reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \ + flock ${DEPLOY_DIR_APT}/${DISTRO}/db.lock \ + reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \ --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \ -C main \ includedeb ${DEBDISTRONAME} \ @@ -102,6 +106,5 @@ do_deploy_deb() { addtask deploy_deb after do_build do_deploy_deb[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" -do_deploy_deb[lockfiles] = "${DEPLOY_DIR_APT}/isar.lock" do_deploy_deb[depends] = "isar-apt:do_cache_config" do_deploy_deb[dirs] = "${S}" diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh index e2dabab..e8a9c17 100644 --- a/meta/recipes-devtools/buildchroot/files/build.sh +++ b/meta/recipes-devtools/buildchroot/files/build.sh @@ -50,7 +50,7 @@ install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y # Install all build deps mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control -) 42>/dpkg.lock +) 42>/isar-apt/db.lock # If autotools files have been created, update their timestamp to # prevent them from being regenerated diff --git a/meta/recipes-devtools/isar-apt/isar-apt-host.bb b/meta/recipes-devtools/isar-apt/isar-apt-host.bb index 0cf5ce1..cab84a7 100644 --- a/meta/recipes-devtools/isar-apt/isar-apt-host.bb +++ b/meta/recipes-devtools/isar-apt/isar-apt-host.bb @@ -8,7 +8,6 @@ SRC_URI = "file://distributions.in" CACHE_CONF_DIR = "${DEPLOY_DIR_APT}/${HOST_DISTRO}/conf" do_cache_config[dirs] = "${CACHE_CONF_DIR}" do_cache_config[stamp-extra-info] = "${HOST_DISTRO}" -do_cache_config[lockfiles] = "${DEPLOY_DIR_APT}/isar.lock" # Generate reprepro config for current host distro if it doesn't exist. Once it's # generated, this task should do nothing. @@ -22,7 +21,8 @@ do_cache_config() { path_databases="${DEPLOY_DIR_DB}/${HOST_DISTRO}" if [ ! -d "${path_databases}" ]; then - reprepro -b ${path_cache} \ + flock ${DEPLOY_DIR_APT}/db.lock \ + reprepro -b ${path_cache} \ --dbdir ${path_databases} \ export ${DEBDISTRONAME} fi diff --git a/meta/recipes-devtools/isar-apt/isar-apt.bb b/meta/recipes-devtools/isar-apt/isar-apt.bb index 9c31d12..e9dfec1 100644 --- a/meta/recipes-devtools/isar-apt/isar-apt.bb +++ b/meta/recipes-devtools/isar-apt/isar-apt.bb @@ -6,7 +6,6 @@ SRC_URI = "file://distributions.in" CACHE_CONF_DIR = "${DEPLOY_DIR_APT}/${DISTRO}/conf" do_cache_config[dirs] = "${CACHE_CONF_DIR}" do_cache_config[stamp-extra-info] = "${DISTRO}" -do_cache_config[lockfiles] = "${DEPLOY_DIR_APT}/isar.lock" # Generate reprepro config for current distro if it doesn't exist. Once it's # generated, this task should do nothing. @@ -20,7 +19,8 @@ do_cache_config() { path_databases="${DEPLOY_DIR_DB}/${DISTRO}" if [ ! -d "${path_databases}" ]; then - reprepro -b ${path_cache} \ + flock ${DEPLOY_DIR_APT}/db.lock \ + reprepro -b ${path_cache} \ --dbdir ${path_databases} \ export ${DEBDISTRONAME} fi diff --git a/meta/recipes-kernel/linux/files/build-kernel.sh b/meta/recipes-kernel/linux/files/build-kernel.sh index 8b58e93..2ff5a1e 100644 --- a/meta/recipes-kernel/linux/files/build-kernel.sh +++ b/meta/recipes-kernel/linux/files/build-kernel.sh @@ -34,7 +34,7 @@ REPACK_LINUX_IMAGE_DIR="${REPACK_DIR}/linux-image" REPACK_LINUX_HEADERS_DIR="${REPACK_DIR}/linux-headers" # Lock-protected because apt and dpkg do not wait in case of contention -flock /dpkg.lock \ +flock /isar-apt/db.lock \ apt-get install -y -o Debug::pkgProblemResolver=yes \ --no-install-recommends ${KBUILD_DEPENDS} -- 2.11.0