From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6592507174472122368 X-Received: by 2002:a02:4d:: with SMTP id 74-v6mr26147594jaa.15.1534945661860; Wed, 22 Aug 2018 06:47:41 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5e:834b:: with SMTP id y11-v6ls180673iom.7.gmail; Wed, 22 Aug 2018 06:47:41 -0700 (PDT) X-Google-Smtp-Source: AA+uWPyyFi2YoPRXWhiaSEkUwN1ERk9QLeGD3QLqK8mDP/iPJMXrHb6sQk0tWMTcmVlpWOv9RTC3 X-Received: by 2002:a6b:9710:: with SMTP id z16-v6mr22201742iod.85.1534945661576; Wed, 22 Aug 2018 06:47:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1534945661; cv=none; d=google.com; s=arc-20160816; b=0CkgN8l14ZN2BuQQw4aui/evbfVaTIvu9BbrtXW8duHkf296iGWzb0nkzZs8tzgxyR LDFzPcKqeHNtjo4C3iwe6bqE7wZJzOlrALYc98cN03chH2NWLZQshag4CtTHMEBsvj6H /a+a0yDvGqTd2cYnozv8LFUiUojvO1bsjQoMxjR11Vv3/g4SAg5Uu8WestClNKS27zQ8 3mCoEMIUjqpygAJ+FOTmRhYAPJbkT2ji/V1wEbbAVFtwHx+SGEPX8kZ2tcCRaSCzC59I TyAa212VI27e1F5ERYj2iNvsmf1NcR4soNsccgVq7Wl6+YoKoggo8kDZTdx/TdqmrAkm 2zyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:content-language :accept-language:in-reply-to:references:message-id:date:thread-index :thread-topic:subject:cc:to:from:arc-authentication-results; bh=awniNTN7v1mvUwnk5mIwu4YINUxXya9Ss953PEYI694=; b=yChfhfCIA0jFIguc0gu0Bk9LZxnrN0dkcnZ8BCBdKO4EcqEg36NalL2f5f9fz7hJIi 0+B6qTf2kreBbmYQPvjhJWaw9BEogXyerKLXvu8hvWqiZlVa+ofINqSa6CqUakyTh+zp ym53x2IFE08uCX156IWAxoYmnWTcyP4Z78lS/7OYq+DyLVAHXgWGYfpJKscEwMGx0zHy pr4U8TQuiAiuR6/68DuHbVBr6TL5VXrbjhHYJRdR0Bd6/lSR1LqTq2nmuvaLQxa3FTCm ktC0RMOcNIZKKkj4C2YYewCQPiVMahw3vjPRugGTkiMPhgb0kPPxtcjWt3BcCgiY2yx0 MPRQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of cedric_hombourger@mentor.com designates 192.94.38.131 as permitted sender) smtp.mailfrom=Cedric_Hombourger@mentor.com Return-Path: Received: from relay1.mentorg.com (relay1.mentorg.com. [192.94.38.131]) by gmr-mx.google.com with ESMTPS id p16-v6si984423itf.0.2018.08.22.06.47.41 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Aug 2018 06:47:41 -0700 (PDT) Received-SPF: pass (google.com: domain of cedric_hombourger@mentor.com designates 192.94.38.131 as permitted sender) client-ip=192.94.38.131; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of cedric_hombourger@mentor.com designates 192.94.38.131 as permitted sender) smtp.mailfrom=Cedric_Hombourger@mentor.com Received: from nat-ies.mentorg.com ([192.94.31.2] helo=svr-ies-mbx-01.mgc.mentorg.com) by relay1.mentorg.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-SHA384:256) id 1fsTU4-000732-DL from Cedric_Hombourger@mentor.com ; Wed, 22 Aug 2018 06:47:40 -0700 Received: from svr-ies-mbx-02.mgc.mentorg.com (139.181.222.2) by svr-ies-mbx-01.mgc.mentorg.com (139.181.222.1) with Microsoft SMTP Server (TLS) id 15.0.1320.4; Wed, 22 Aug 2018 14:47:36 +0100 Received: from svr-ies-mbx-02.mgc.mentorg.com ([fe80::a01f:51c9:5b6c:e0c]) by svr-ies-mbx-02.mgc.mentorg.com ([fe80::a01f:51c9:5b6c:e0c%22]) with mapi id 15.00.1320.000; Wed, 22 Aug 2018 14:47:36 +0100 From: "Hombourger, Cedric" To: Henning Schild CC: Jan Kiszka , "isar-users@googlegroups.com" Subject: RE: [PATCH 1/1] dpkg: acquire lock when calling reprepro Thread-Topic: [PATCH 1/1] dpkg: acquire lock when calling reprepro Thread-Index: AQHUOgwjyodHCWwHxUe2bsZOCAaC9qTLoA8AgAAYbiD///5igIAAEQAg Date: Wed, 22 Aug 2018 13:47:36 +0000 Message-ID: <75371d2352c041d8bb22986084bc479c@svr-ies-mbx-02.mgc.mentorg.com> References: <1534937676-7628-1-git-send-email-Cedric_Hombourger@mentor.com> <1534937676-7628-2-git-send-email-Cedric_Hombourger@mentor.com> <004d3802fb104470a68eada0c12fae1f@svr-ies-mbx-02.mgc.mentorg.com> <20180822154145.07d4a2ca@md1pvb1c.ad001.siemens.net> In-Reply-To: <20180822154145.07d4a2ca@md1pvb1c.ad001.siemens.net> Accept-Language: en-US, en-IE Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [137.202.0.87] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-TUID: LfqLxtGq3hRp Hi Henning, I am not sure this can be easily done without additional some additional re= factoring More specifically, I believe this would require splitting dpkg_runbuild & b= uild.sh into 2 distinct tasks: 1) run apt-get update build the package dependencies package (aka the 1st s= ection of build.sh) 2) perform the actual package build Without this, we would keep the lock acquired for as long as the full packa= ge build goes (several minutes for a kernel build) I will investigate This may make it easier to implement the other thing I had in mind: do not = perform builds as root Cedric -----Original Message----- From: Henning Schild [mailto:henning.schild@siemens.com]=20 Sent: Wednesday, August 22, 2018 3:42 PM To: Hombourger, Cedric Cc: Jan Kiszka ; isar-users@googlegroups.com Subject: Re: [PATCH 1/1] dpkg: acquire lock when calling reprepro My first reaction was the same, but in the other direction. I would prefer = a bitbake lock over the flock, if possible. Henning Am Wed, 22 Aug 2018 12:48:14 +0000 schrieb "Hombourger, Cedric" : > Thanks Jan for your quick feedback. I have submitted a v2 with=20 > isar.lock removed >=20 > Cedric >=20 > -----Original Message----- > From: Jan Kiszka [mailto:jan.kiszka@siemens.com] > Sent: Wednesday, August 22, 2018 2:20 PM > To: Hombourger, Cedric ;=20 > isar-users@googlegroups.com Subject: Re: [PATCH 1/1] dpkg: acquire=20 > lock when calling reprepro >=20 > On 2018-08-22 13:34, Cedric Hombourger wrote: > > As bitbake performs multi-threaded builds, packages may be built=20 > > while others get deployed. A lock was already acquired within=20 > > dpkg_runbuild but accesses to the isar-apt database were not=20 > > protected from do_deploy_deb. Move the lock from the buildchroot to=20 > > the isar-apt repository and acquire the (same) lock from=20 > > dpkg_runbuild, repo_clean and do_deploy_deb. >=20 > I think I've seen this issue once but forgot to dig into the details. > It looks consistent. >=20 > >=20 > > Signed-off-by: Cedric Hombourger > > --- > > meta/classes/dpkg-base.bbclass | 8 ++++++-- > > meta/recipes-devtools/buildchroot/files/build.sh | 2 +- > > meta/recipes-kernel/linux/files/build-kernel.sh | 2 +- > > 3 files changed, 8 insertions(+), 4 deletions(-) > >=20 > > diff --git a/meta/classes/dpkg-base.bbclass=20 > > b/meta/classes/dpkg-base.bbclass index a41df2c..7137968 100644 > > --- a/meta/classes/dpkg-base.bbclass > > +++ b/meta/classes/dpkg-base.bbclass > > @@ -71,6 +71,8 @@ do_build() { > > mount -t proc none ${BUILDCHROOT_DIR}/proc > > fi' > > =20 > > + # Create database lock file here so we own it (not root) and > > perform actual build > > + touch ${DEPLOY_DIR_APT}/${DISTRO}/db.lock > > dpkg_runbuild > > =20 > > sudo umount ${BUILDROOT} 2>/dev/null || true @@ -82,7 +84,8 @@=20 > > CLEANFUNCS +=3D "repo_clean" > > repo_clean() { > > PACKAGES=3D$(cd ${S}/..; ls *.deb | sed 's/\([^_]*\).*/\1/') > > if [ -n "${PACKAGES}" ]; then > > - reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \ > > + flock ${DEPLOY_DIR_APT}/${DISTRO}/db.lock \ > > + reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \ > > --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \ > > -C main -A ${DISTRO_ARCH} \ > > remove ${DEBDISTRONAME} \ @@ -93,7 +96,8 @@ > > repo_clean() { > > # Install package to Isar-apt > > do_deploy_deb() { > > repo_clean > > - reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \ > > + flock ${DEPLOY_DIR_APT}/${DISTRO}/db.lock \ > > + reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \ > > --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \ > > -C main \ > > includedeb ${DEBDISTRONAME} \ >=20 > There is also >=20 > do_deploy_deb[lockfiles] =3D "${DEPLOY_DIR_APT}/isar.lock" >=20 > in that file. I suppose that can now be dropped, can't it? >=20 > > diff --git a/meta/recipes-devtools/buildchroot/files/build.sh > > b/meta/recipes-devtools/buildchroot/files/build.sh > > index e2dabab..e8a9c17 100644 > > --- a/meta/recipes-devtools/buildchroot/files/build.sh > > +++ b/meta/recipes-devtools/buildchroot/files/build.sh > > @@ -50,7 +50,7 @@ install_cmd=3D"apt-get -o=20 > > Debug::pkgProblemResolver=3Dyes --no-install-recommends -y > > =20 > > # Install all build deps > > mk-build-deps $set_arch -t "${install_cmd}" -i -r=20 > > debian/control -) 42>/dpkg.lock > > +) 42>/isar-apt/db.lock > > =20 > > # If autotools files have been created, update their timestamp to > > # prevent them from being regenerated diff --git=20 > > a/meta/recipes-kernel/linux/files/build-kernel.sh > > b/meta/recipes-kernel/linux/files/build-kernel.sh > > index 8b58e93..2ff5a1e 100644 > > --- a/meta/recipes-kernel/linux/files/build-kernel.sh > > +++ b/meta/recipes-kernel/linux/files/build-kernel.sh > > @@ -34,7 +34,7 @@ REPACK_LINUX_IMAGE_DIR=3D"${REPACK_DIR}/linux-image" > > REPACK_LINUX_HEADERS_DIR=3D"${REPACK_DIR}/linux-headers" > > =20 > > # Lock-protected because apt and dpkg do not wait in case of=20 > > contention -flock /dpkg.lock \ > > +flock /isar-apt/db.lock \ > > apt-get install -y -o Debug::pkgProblemResolver=3Dyes \ > > --no-install-recommends ${KBUILD_DEPENDS} > > =20 > > =20 >=20 > Jan >=20 > -- > Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate=20 > Competence Center Embedded Linux >=20