From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6522497667596025856 X-Received: by 10.28.239.20 with SMTP id n20mr791686wmh.23.1518682576355; Thu, 15 Feb 2018 00:16:16 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.223.208.133 with SMTP id y5ls5870676wrh.5.gmail; Thu, 15 Feb 2018 00:16:16 -0800 (PST) X-Google-Smtp-Source: AH8x2252+gDzbMHJ7weR/eEehJ2FRvcasi1Bblrrhga8bJkaN5JpsM1TtRIC7Y0NJ+YlrvJNbsz6 X-Received: by 10.223.157.147 with SMTP id p19mr798902wre.30.1518682576032; Thu, 15 Feb 2018 00:16:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1518682575; cv=none; d=google.com; s=arc-20160816; b=Hnc7g4hv+u+hNW2feKqcQYvF5bbsaLj3N+Qbdooewym69gAP0m+ACv8gwQ8sdnotTw CXZDh37TdPxGDhsq5+G+HviP5aKXOjbDgO/G8sGZ03fSDGduCHDHmzu7eCQ4JpYI3vSR ex/dq0DCz955aMt/lX/jLfpGsyksOLZjrgrg0Uz18EPyPKS9NMKNgUhVeZ8Ry4VqFlLq jphxqlBtH/Ef9WQHLXxy42vpsEnpWuxLCkU54/DlHMy3u801rx1gwDVrbMn/mh/KUXsx QOlpZsf3NmSO6jWFa4/Z17Y7D89Az6ggu1vnteDqGQCMDAKsStB03fwAef5xvGR0NZOp ErDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=t4kGtQ8n2Frtif0D94IfpL1TZHijv+CvZsoJCox0byA=; b=BiNXaTYe5LqQ69tkyAJpjjgVvfTSj+E6C2Z6DnlUGHB+DA8BIVOwcPv0hw8pLpeDWz PeoaHVMDB9DD/PqmOM1GVXIm9XXZPW+ryQ4P4XMmGqUCaSSDE8HU8YUvVH2EUq1EhZVS T7we5/YYqO4/rfIRB5qHuZmjXovusV4Y8pORoagCoZbMm8Na1jvgd1Iqw7/M6eAEGbwW /LGWgU4o9cMtNYRIWXSi0zMc8nSUTtzsNgukmEA9Vl0GaZVf2UjtCsiU/N9MW6B0XK8I 7cBFvCUHQN3yIjg7q603FG882BpfW+lkB2dfoekrZ44sLh/bEU701DcF4AS5+ela0B7Y B8fQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id k21si863482wrd.1.2018.02.15.00.16.15 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Feb 2018 00:16:15 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w1F8GCbR013319 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 15 Feb 2018 09:16:14 +0100 Subject: Re: [PATCH] buildchroot: Lock-protect apt/dpkg operations To: Jan Kiszka , isar-users References: <92b096db-6db3-401c-0bba-09a31cff2376@siemens.com> From: Alexander Smirnov Message-ID: <7ed33476-a553-5a2d-8eab-b39a4fe5b055@ilbers.de> Date: Thu, 15 Feb 2018 11:16:07 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <92b096db-6db3-401c-0bba-09a31cff2376@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: eOZijcEZXN/6 On 02/14/2018 10:42 PM, Jan Kiszka wrote: > From: Jan Kiszka > > When running multiple build targets in parallel, they will compete for > access on the package database while the related commands do not wait > but fail immediately. > > Signed-off-by: Jan Kiszka > --- > > Fixes parallel builds (CI setup) for me. > > I just recalled that Chris was reporting the same issue a long time ago, > but his suggestion to lock down whole do_build was not that well > received... > > meta/recipes-devtools/buildchroot/files/build.sh | 32 +++++++++++++++--------- > 1 file changed, 20 insertions(+), 12 deletions(-) > > diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh > index 023a6d6..048f76f 100644 > --- a/meta/recipes-devtools/buildchroot/files/build.sh > +++ b/meta/recipes-devtools/buildchroot/files/build.sh > @@ -2,17 +2,9 @@ > # > # This software is a part of ISAR. > # Copyright (C) 2015-2017 ilbers GmbH > +# Copyright (c) 2018 Siemens AG > > -# Make sure that we have latest isar-apt content. > -# Options meaning: > -# Dir::Etc::sourcelist - specifies which source to be used > -# Dir::Etc::sourceparts - disables looking for the other sources > -# APT::Get::List-Cleanup - do not erase obsolete packages list for > -# upstream in '/var/lib/apt/lists' > -apt-get update \ > - -o Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \ > - -o Dir::Etc::sourceparts="-" \ > - -o APT::Get::List-Cleanup="0" > +set -e > > # Go to build directory > cd $1 > @@ -26,8 +18,24 @@ install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y > # Allow unauthenticated feeds > install_cmd="${install_cmd} --allow-unauthenticated" > > -# Install all build deps > -mk-build-deps -t "${install_cmd}" -i -r debian/control > +( > + # Lock-protected because apt and dpkg do not wait in case of contention > + flock 42 || exit 1 > + > + # Make sure that we have latest isar-apt content. > + # Options meaning: > + # Dir::Etc::sourcelist - specifies which source to be used > + # Dir::Etc::sourceparts - disables looking for the other sources > + # APT::Get::List-Cleanup - do not erase obsolete packages list for > + # upstream in '/var/lib/apt/lists' > + apt-get update \ > + -o Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \ > + -o Dir::Etc::sourceparts="-" \ > + -o APT::Get::List-Cleanup="0" > + > + # Install all build deps > + mk-build-deps -t "${install_cmd}" -i -r debian/control > +) 42>/dpkg.lock BTW, what is the difference between the construction above and the construction you used in dpkg-base.bbclass: 8<-- sudo flock ${MOUNT_LOCKFILE} -c ' \ if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then \ mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt; \ mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev; \ mount -t proc none ${BUILDCHROOT_DIR}/proc; \ fi' 8<-- If there is no difference, could we somehow unify usage of flock in Isar? Alex > > # If autotools files have been created, update their timestamp to > # prevent them from being regenerated >