From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6522497667596025856 X-Received: by 10.223.190.141 with SMTP id i13mr793521wrh.15.1518684394314; Thu, 15 Feb 2018 00:46:34 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.223.163.205 with SMTP id m13ls4702345wrb.0.gmail; Thu, 15 Feb 2018 00:46:33 -0800 (PST) X-Google-Smtp-Source: AH8x224qIfMrYmc5tySvebg2a5nmRgNv8w8eljWelPb3/cfOzUti8v2L0mEZ+VO1rSA0w7mSCDFv X-Received: by 10.223.197.204 with SMTP id v12mr43710wrg.29.1518684393739; Thu, 15 Feb 2018 00:46:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1518684393; cv=none; d=google.com; s=arc-20160816; b=0SLgRZAMrXtAl7QztRVLNBpBCzCutHDCNClNAk1/3ZNpABKYrLZCwLbEbu7bQk0VCo i4Z+1JPN5aX6GbcARCV/CGNVoj40SiyESWw8BJN/HtVQ/Be/10ptT7PnREwN+OVw0yIN YQYrMx0QxDmrD4Vf0JR9V96hUzGCpeuQvusQICOC4lhlEJ+JaGc6erM6KbzJ9WbXqI5L /i3KCF2KxSqsgUySh3DGBKZ10rOZGtjbL8pajjXZTzfATGoDmEX6wudW/9exGyA1Tsa9 lVX79ovTn2FQxccPwi8kH76/cJvMe9VIiDABr7KO0fuT9RABoCxM4gwHZZWqcQKhm1EN ZI4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=dQjmHl9PU3rCGDJWyMlKRXAIaek2WC+lbmRF2+hZ8Dg=; b=UV2RkEjUBkoK776vaIBEV/SzLh7ui4TIvLz3YvbSsxbFDvSTAPMXws3W/nQHeBcbwP jNqQDEqAQDsMvshRVqIJu+nTPy1xEXxDwAD0fPlx6ClXOpYWUp0vfvsugwqIbPRAcJSF X2nkZzyrpAJsbjVfwa2iqp1hI3VtWk4zzeUIk857NKpN/gUnHZEvbdaDlQvWsefwMLsx traR0PtDsMGSo9BjhbBD4lt3HYKuk2ruYm1qzA27N5e4AXQrcCb0c5+Cz9kCslcwesLg a/jXi3JuPf8BcfxT9fz/9ejRk8CLb5NLO70DD9Rj4L26h0lem0Gn/D6p3wQOtP3X0jVn voAw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id a138si76243wmd.0.2018.02.15.00.46.33 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Feb 2018 00:46:33 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail3.siemens.de (mail3.siemens.de [139.25.208.14]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id w1F8kXIZ028181 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 15 Feb 2018 09:46:33 +0100 Received: from [167.87.42.35] ([167.87.42.35]) by mail3.siemens.de (8.15.2/8.15.2) with ESMTP id w1F8kWYF003301; Thu, 15 Feb 2018 09:46:33 +0100 Subject: Re: [PATCH] buildchroot: Lock-protect apt/dpkg operations To: Alexander Smirnov , isar-users References: <92b096db-6db3-401c-0bba-09a31cff2376@siemens.com> <7ed33476-a553-5a2d-8eab-b39a4fe5b055@ilbers.de> From: Jan Kiszka Message-ID: <609f10a7-6670-337c-9ae1-b8f291e7125f@siemens.com> Date: Thu, 15 Feb 2018 09:46:32 +0100 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <7ed33476-a553-5a2d-8eab-b39a4fe5b055@ilbers.de> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: 88+xGiWraXpP On 2018-02-15 09:16, Alexander Smirnov wrote: > On 02/14/2018 10:42 PM, Jan Kiszka wrote: >> From: Jan Kiszka >> >> When running multiple build targets in parallel, they will compete for >> access on the package database while the related commands do not wait >> but fail immediately. >> >> Signed-off-by: Jan Kiszka >> --- >> >> Fixes parallel builds (CI setup) for me. >> >> I just recalled that Chris was reporting the same issue a long time ago, >> but his suggestion to lock down whole do_build was not that well >> received... >> >>   meta/recipes-devtools/buildchroot/files/build.sh | 32 >> +++++++++++++++--------- >>   1 file changed, 20 insertions(+), 12 deletions(-) >> >> diff --git a/meta/recipes-devtools/buildchroot/files/build.sh >> b/meta/recipes-devtools/buildchroot/files/build.sh >> index 023a6d6..048f76f 100644 >> --- a/meta/recipes-devtools/buildchroot/files/build.sh >> +++ b/meta/recipes-devtools/buildchroot/files/build.sh >> @@ -2,17 +2,9 @@ >>   # >>   # This software is a part of ISAR. >>   # Copyright (C) 2015-2017 ilbers GmbH >> +# Copyright (c) 2018 Siemens AG >>   -# Make sure that we have latest isar-apt content. >> -# Options meaning: >> -#   Dir::Etc::sourcelist - specifies which source to be used >> -#   Dir::Etc::sourceparts - disables looking for the other sources >> -#   APT::Get::List-Cleanup - do not erase obsolete packages list for >> -#                            upstream in '/var/lib/apt/lists' >> -apt-get update \ >> -    -o Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \ >> -    -o Dir::Etc::sourceparts="-" \ >> -    -o APT::Get::List-Cleanup="0" >> +set -e >>     # Go to build directory >>   cd $1 >> @@ -26,8 +18,24 @@ install_cmd="apt-get -o >> Debug::pkgProblemResolver=yes --no-install-recommends -y >>   # Allow unauthenticated feeds >>   install_cmd="${install_cmd} --allow-unauthenticated" >>   -# Install all build deps >> -mk-build-deps -t "${install_cmd}" -i -r debian/control >> +( >> +    # Lock-protected because apt and dpkg do not wait in case of >> contention >> +    flock 42 || exit 1 >> + >> +    # Make sure that we have latest isar-apt content. >> +    # Options meaning: >> +    #   Dir::Etc::sourcelist - specifies which source to be used >> +    #   Dir::Etc::sourceparts - disables looking for the other sources >> +    #   APT::Get::List-Cleanup - do not erase obsolete packages list for >> +    #                            upstream in '/var/lib/apt/lists' >> +    apt-get update \ >> +        -o >> Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \ >> +        -o Dir::Etc::sourceparts="-" \ >> +        -o APT::Get::List-Cleanup="0" >> + >> +    # Install all build deps >> +    mk-build-deps -t "${install_cmd}" -i -r debian/control >> +) 42>/dpkg.lock > > BTW, what is the difference between the construction above and the > construction you used in dpkg-base.bbclass: > > 8<-- > sudo flock ${MOUNT_LOCKFILE} -c ' \ >     if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then \ >         mount --bind ${DEPLOY_DIR_APT}/${DISTRO} > ${BUILDCHROOT_DIR}/isar-apt; \ >         mount -t devtmpfs -o mode=0755,nosuid devtmpfs > ${BUILDCHROOT_DIR}/dev; \ >         mount -t proc none ${BUILDCHROOT_DIR}/proc; \ >     fi' > 8<-- > > If there is no difference, could we somehow unify usage of flock in Isar? > See the comment in the other commit: The pattern used here is apparently not supported in bitbake shell functions. This one here seems nicer, though, as it does not require those line continuations. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux