From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6521316140522668032 X-Received: by 10.28.129.195 with SMTP id c186mr239323wmd.10.1518373092611; Sun, 11 Feb 2018 10:18:12 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.165.130 with SMTP id o124ls684761wme.9.canary-gmail; Sun, 11 Feb 2018 10:18:12 -0800 (PST) X-Google-Smtp-Source: AH8x227nNQs1XEiTw/0VtBTFQBTAat+x4ptzgYzxM03dnam+ZtB9/HGwoG2e14Yn3pLMo+6bMsJ9 X-Received: by 10.28.63.6 with SMTP id m6mr222280wma.3.1518373092009; Sun, 11 Feb 2018 10:18:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1518373091; cv=none; d=google.com; s=arc-20160816; b=DB/JTXuefMbxYCAJIN+Oimbn79+WgwKQBPao1o4rRuy3rI5RdxR0vW7SGJWqBLLo3y W29y2tc763k/hfnhLCcycuwatUTA3vxIZqHxuxEqRMIo8/JexeEJ416NNHsqua5p9OIF Jz2e3kzoHoeMWNBDGlT5L5ElE3wOpK4ffMJeByyKWeXsRvdsMJv8moY7/YGytgchRX6K d7j5OBf7OXrsTggRaGl4pjiOQN8hxSC4bITbjP2O5uqOf8uScmGNCv3Bj+iKEsx1ziA/ dxPOjWkcOkQ95FLqoluRXvbMAspbsui4+T7oDiqVcl4M6E+Ty8bLYuUXWw4g54OuQMhM y7GA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=DAxcLqOHZ3c1iRRhD+JndTHIfDcm0GZC+rC8tsuN3C4=; b=dlp+n885B9kex4u9qJRis3EmCLIBXRI1dy0N5im2UwkKFCsvWHpxIPIqCptym4DlAd g2kAdBTACP0ZvEmrJKDPoFNn/TTRx0Sm0LRl92nGswuTYxGoZFSwPsLyZY3j7jG0Cw69 YmxGIGITqgwoKqpVAvRrJEZuAnIh/EPR/zrEAU+3ZlTlBSXAAHZx2qDQg3c5Jn/2A1n0 Cjselj69RP+KsyssUFAuv6aD7zUklmuXMXQdkttjHUxysASh1nOLel9iqlGBuE+veTN1 lBVR4dw6+T+5/seUBlLXg6vgnK5MYCTLfhFOPmRqH+ibHHWuBZrK7INyEvmIcJS/SUn9 3LUA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id i75si151226wmg.1.2018.02.11.10.18.11 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 11 Feb 2018 10:18:11 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail2.siemens.de (mail2.siemens.de [139.25.208.11]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id w1BIIBrB030479 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 11 Feb 2018 19:18:11 +0100 Received: from [167.87.43.174] ([167.87.43.174]) by mail2.siemens.de (8.15.2/8.15.2) with ESMTP id w1BIIATA026563; Sun, 11 Feb 2018 19:18:11 +0100 Subject: Re: Build can sporadically fail due to dpkg contention in buildchroot To: Alexander Smirnov , isar-users References: <79bd216e-53ca-5485-4f6c-66050d08ed5f@siemens.com> <4d2b9322-54ed-ddff-739e-d0a3d5c6cc7b@ilbers.de> From: Jan Kiszka Message-ID: Date: Sun, 11 Feb 2018 19:18:10 +0100 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <4d2b9322-54ed-ddff-739e-d0a3d5c6cc7b@ilbers.de> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: VllkojgbX2j8 On 2018-02-11 17:55, Alexander Smirnov wrote: > Hi, > > On 02/11/2018 06:17 PM, Jan Kiszka wrote: >> Hi all, >> >> I got this failure of example-hello:do_build already twice while doing >> rebuild tests with my kernel series (ie. with more independent >> buildchroot users): >> >> DEBUG: Executing shell function do_build >> Get:1 file:/isar-apt isar InRelease >> Ign:1 file:/isar-apt isar InRelease >> Get:2 file:/isar-apt isar Release [2,864 B] >> Get:2 file:/isar-apt isar Release [2,864 B] >> Get:3 file:/isar-apt isar Release.gpg >> Ign:3 file:/isar-apt isar Release.gpg >> Get:4 file:/isar-apt isar/main amd64 Packages [1,135 B] >> Reading package lists... >> W: The repository 'file:/isar-apt isar Release' is not signed. >> hostname: No address associated with hostname >> dh_testdir >> dh_testroot >> dh_prep >> dh_testdir >> dh_testroot >> dh_install >> dh_install: Compatibility levels before 9 are deprecated (level 7 in use) >> dh_installdocs >> dh_installdocs: Compatibility levels before 9 are deprecated (level 7 >> in use) >> dh_installchangelogs >> dh_compress >> dh_fixperms >> dh_installdeb >> dh_installdeb: Compatibility levels before 9 are deprecated (level 7 >> in use) >> dh_gencontrol >> dh_md5sums >> dh_builddeb >> dpkg-deb: building package 'hello-build-deps' in >> '../hello-build-deps_0.2_all.deb'. >> > > Good catch! > >> The package has been created. >> Attention, the package has been created in the current directory, >> not in ".." as indicated by the message above! >> dpkg: error: dpkg status database is locked by another process >> mk-build-deps: dpkg --unpack failed >> [...] >> >> So we have a concurrency problem when building over the same dpkg >> database. Looks like we need to synchronize (lock-protect) the access to >> it, which also means pulling out the dependency installation from the >> regular build step. Is that feasible at all? Any alternatives (besides >> retrying such builds...)? > > In general we could do this easily: > > 1. Split the content of build.sh into two functions, for example: >  - install_build_deps >  - build_package > > 2. Spit the bitbake do_build() into two tasks: > > do_install_build_deps() { >     ... build.sh install_build_deps ... > } > > addtask install_build_deps before do_build after do_unpack > > do_build_package() { >     ... build.sh build_package ... > } > > 3. Using bitbake synchronization primitives, protect the fist task from > parallel execution. > > If you are OK with this, I could do this tomorrow. I'm still concerned how well this will scale: a) We have additional users we already know of (linux-kernel.bbclass). We will need to provide them the same means. b) There might be more users hidden in today's or future recipes... Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux