From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6592507174472122368 X-Received: by 2002:aa7:c510:: with SMTP id o16-v6mr3261972edq.7.1534945689191; Wed, 22 Aug 2018 06:48:09 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a50:8825:: with SMTP id b34-v6ls811225edb.8.gmail; Wed, 22 Aug 2018 06:48:08 -0700 (PDT) X-Google-Smtp-Source: AA+uWPy41RhcCiNia303gHEhyGEjLaVNWmPn1neP7Y5gN9eJ5RMRA7F3955H0JDNU5xBsoVJcLwA X-Received: by 2002:a50:8879:: with SMTP id c54-v6mr3408630edc.1.1534945688569; Wed, 22 Aug 2018 06:48:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1534945688; cv=none; d=google.com; s=arc-20160816; b=ye5PFvw8j60ZqjTiaaQh55nVN71M3u0N0ywaFDXtmnXPupRVezgyNDJBpEfHZuxz2E UXugKIoLnyo7bsrR0Nwr/PT/rsAacL2oSi4Fs4i3sZBAistbG5VTOQmb88gZhg3HiLoz +QFSuxfQe8ErVdzAJRG1J3r94Ux1ea6fd7Jbp0woyimvUZoGsrGGM3Im/J6sAsQhIuVV ALjFwNJLfIumVruP7zc1wrAvavXEPj4UdRzA/UuAgP/208qHbotYkXNddoel5mJsSxhb zKC5+mpJ+St4OBAuTnF1EYRop2QXOBRxPkg4lyZ3v7pipJoCtYDkaMUavgBkWAJ4D2Qj 7LWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=hX7J74MOjf5ofxSsyMAtfRSbixBH6V5DhL7mhX4qC90=; b=PTP4yHAsaYZ/j+QE1IeZ9cq2QMUmrTLM8ZfQZ/ipqTgqKKSTh6scQIezZjjglE1JVL HGl3RkJQECjlD64ZBEtkrWpSYuiG/+bSz1jLccZfbwNoqWG7DjhnDnDWhjdIgpSoL7tF Z2HDRDZdF1Rx1T1rKLMjsWarO9j1X0g/jlYkRX0Y0xatfwYUNjeBXXkETBXW7LDooD4C aU9nKYmDeuEpzoQDMnjmUMiMvjGZaylZ6WXgAGkwX7dLtBPBcNnx84svNyWhy2smtVZw As5pQJ9FCbfX+nUz7/zisbSfGDx89ipbE1CvDpborWkoyqqn/nZpfmAXMPhAWcoaYYeN hdHA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id y4-v6si71867edh.4.2018.08.22.06.48.08 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Aug 2018 06:48:08 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id w7MDm78m030842 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 22 Aug 2018 15:48:07 +0200 Received: from [139.25.68.37] (md1q0hnc.ad001.siemens.net [139.25.68.37] (may be forged)) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id w7MDm7gj018161; Wed, 22 Aug 2018 15:48:07 +0200 Subject: Re: [PATCH 1/1] dpkg: acquire lock when calling reprepro To: Henning Schild , "Hombourger, Cedric" Cc: "isar-users@googlegroups.com" References: <1534937676-7628-1-git-send-email-Cedric_Hombourger@mentor.com> <1534937676-7628-2-git-send-email-Cedric_Hombourger@mentor.com> <004d3802fb104470a68eada0c12fae1f@svr-ies-mbx-02.mgc.mentorg.com> <20180822154145.07d4a2ca@md1pvb1c.ad001.siemens.net> From: Jan Kiszka Message-ID: Date: Wed, 22 Aug 2018 15:48:07 +0200 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <20180822154145.07d4a2ca@md1pvb1c.ad001.siemens.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: XhvY1VuTt/93 On 2018-08-22 15:41, Henning Schild wrote: > My first reaction was the same, but in the other direction. I would > prefer a bitbake lock over the flock, if possible. That would at least involve factoring out the critical section into dedicated tasks - otherwise we to over-locking and serialize harmless (and potentially lengthy) steps needlessly. For now, I would be infavor of fixing in-place and trying to cleanup on top. Jan > > Henning > > Am Wed, 22 Aug 2018 12:48:14 +0000 > schrieb "Hombourger, Cedric" : > >> Thanks Jan for your quick feedback. I have submitted a v2 with >> isar.lock removed >> >> Cedric >> >> -----Original Message----- >> From: Jan Kiszka [mailto:jan.kiszka@siemens.com] >> Sent: Wednesday, August 22, 2018 2:20 PM >> To: Hombourger, Cedric ; >> isar-users@googlegroups.com Subject: Re: [PATCH 1/1] dpkg: acquire >> lock when calling reprepro >> >> On 2018-08-22 13:34, Cedric Hombourger wrote: >>> As bitbake performs multi-threaded builds, packages may be built >>> while others get deployed. A lock was already acquired within >>> dpkg_runbuild but accesses to the isar-apt database were not >>> protected from do_deploy_deb. Move the lock from the buildchroot to >>> the isar-apt repository and acquire the (same) lock from >>> dpkg_runbuild, repo_clean and do_deploy_deb. >> >> I think I've seen this issue once but forgot to dig into the details. >> It looks consistent. >> >>> >>> Signed-off-by: Cedric Hombourger >>> --- >>> meta/classes/dpkg-base.bbclass | 8 ++++++-- >>> meta/recipes-devtools/buildchroot/files/build.sh | 2 +- >>> meta/recipes-kernel/linux/files/build-kernel.sh | 2 +- >>> 3 files changed, 8 insertions(+), 4 deletions(-) >>> >>> diff --git a/meta/classes/dpkg-base.bbclass >>> b/meta/classes/dpkg-base.bbclass index a41df2c..7137968 100644 >>> --- a/meta/classes/dpkg-base.bbclass >>> +++ b/meta/classes/dpkg-base.bbclass >>> @@ -71,6 +71,8 @@ do_build() { >>> mount -t proc none ${BUILDCHROOT_DIR}/proc >>> fi' >>> >>> + # Create database lock file here so we own it (not root) and >>> perform actual build >>> + touch ${DEPLOY_DIR_APT}/${DISTRO}/db.lock >>> dpkg_runbuild >>> >>> sudo umount ${BUILDROOT} 2>/dev/null || true @@ -82,7 +84,8 >>> @@ CLEANFUNCS += "repo_clean" >>> repo_clean() { >>> PACKAGES=$(cd ${S}/..; ls *.deb | sed 's/\([^_]*\).*/\1/') >>> if [ -n "${PACKAGES}" ]; then >>> - reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \ >>> + flock ${DEPLOY_DIR_APT}/${DISTRO}/db.lock \ >>> + reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \ >>> --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \ >>> -C main -A ${DISTRO_ARCH} \ >>> remove ${DEBDISTRONAME} \ @@ -93,7 +96,8 @@ >>> repo_clean() { >>> # Install package to Isar-apt >>> do_deploy_deb() { >>> repo_clean >>> - reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \ >>> + flock ${DEPLOY_DIR_APT}/${DISTRO}/db.lock \ >>> + reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \ >>> --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \ >>> -C main \ >>> includedeb ${DEBDISTRONAME} \ >> >> There is also >> >> do_deploy_deb[lockfiles] = "${DEPLOY_DIR_APT}/isar.lock" >> >> in that file. I suppose that can now be dropped, can't it? >> >>> diff --git a/meta/recipes-devtools/buildchroot/files/build.sh >>> b/meta/recipes-devtools/buildchroot/files/build.sh >>> index e2dabab..e8a9c17 100644 >>> --- a/meta/recipes-devtools/buildchroot/files/build.sh >>> +++ b/meta/recipes-devtools/buildchroot/files/build.sh >>> @@ -50,7 +50,7 @@ install_cmd="apt-get -o >>> Debug::pkgProblemResolver=yes --no-install-recommends -y >>> >>> # Install all build deps >>> mk-build-deps $set_arch -t "${install_cmd}" -i -r >>> debian/control -) 42>/dpkg.lock >>> +) 42>/isar-apt/db.lock >>> >>> # If autotools files have been created, update their timestamp to >>> # prevent them from being regenerated diff --git >>> a/meta/recipes-kernel/linux/files/build-kernel.sh >>> b/meta/recipes-kernel/linux/files/build-kernel.sh >>> index 8b58e93..2ff5a1e 100644 >>> --- a/meta/recipes-kernel/linux/files/build-kernel.sh >>> +++ b/meta/recipes-kernel/linux/files/build-kernel.sh >>> @@ -34,7 +34,7 @@ REPACK_LINUX_IMAGE_DIR="${REPACK_DIR}/linux-image" >>> REPACK_LINUX_HEADERS_DIR="${REPACK_DIR}/linux-headers" >>> >>> # Lock-protected because apt and dpkg do not wait in case of >>> contention -flock /dpkg.lock \ >>> +flock /isar-apt/db.lock \ >>> apt-get install -y -o Debug::pkgProblemResolver=yes \ >>> --no-install-recommends ${KBUILD_DEPENDS} >>> >>> >> >> Jan >> >> -- >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate >> Competence Center Embedded Linux >> > -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux