From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6521316140522668032 X-Received: by 10.223.198.82 with SMTP id u18mr50868wrg.17.1518631088451; Wed, 14 Feb 2018 09:58:08 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.223.161.148 with SMTP id u20ls1362581wru.7.gmail; Wed, 14 Feb 2018 09:58:08 -0800 (PST) X-Google-Smtp-Source: AH8x225TsgEdk+6ZS3HNsGYcs4NBdUPPzAu/eT0zpn2YI4nYx1ZUPRgxjQ/6Mdm6JPwK/3Ba6x2v X-Received: by 10.223.196.13 with SMTP id v13mr639721wrf.22.1518631088104; Wed, 14 Feb 2018 09:58:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1518631088; cv=none; d=google.com; s=arc-20160816; b=HmkzRmGQ5y/Me0+w3pvEnmJuDJ9BqpQBvuXVJXoSIPs5ERMWpJkfvTwcItPSD0UeA+ eGREXlrSBSLpfvMqtA8jV1ZmoIBHzHGWYeKTBDjsriH+lAYjObCh7kA5gaMu27CI5+DW sLBnaTWcHPlEB30E1OaOHktZiC0Ru7/IBbi5nGxhAcCzYQqWtTqNODu+2DN7iSYaAq5i +8tuDQigSGCDoUeXO/MkJeAA9gTMbA0vthqH5uQS6ELDmYa8CdAKf/OBG1EdShEr/dCJ TDeJBXrajhORZd2j7KZkN6/OMFoaU8KPBDW+kfmP8AKXSOFsSdb5dUWXfLgYgJ50kng5 LO3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=5zSCHDpNOM1QUyoJhSA9IR8eY12CwubrObrLSV/8syU=; b=P2Cm10luHyuyc0MQ7S0KE+8bLJHlCPIWYMKkEgyqpU2UmPZT5k7xhsE0rdt9lz6j0Y RFrBat0l5d8L1WQ/XdbbhyeAgf1pKcWfv6hsqqYC+73A4aek52hvpb8QE4yvzcj2j1E1 N6x6IU2WG8Y6X69lRgCOcvHo/zKjyAWpdZJFtkenwELNQLBxxdilmDWLu5e9or0QxVe8 clkhQvzVYQSFTQho9AyVQ8T/jm/NH/wAiT5sxxFl0lhtld9VThTDezIcH3v0h0nAIEiO fFojgwqv5rJJd4CTybU7mcTSeMJM6fVNluxfnQZa5iYXo7r4lXw6mqi4vpNLLIWIQ0S/ DN/g== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id g70si555772wmc.3.2018.02.14.09.58.07 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 14 Feb 2018 09:58:08 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w1EHw41p010963 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 14 Feb 2018 18:58:06 +0100 Subject: Re: Build can sporadically fail due to dpkg contention in buildchroot To: Jan Kiszka , isar-users References: <79bd216e-53ca-5485-4f6c-66050d08ed5f@siemens.com> <4d2b9322-54ed-ddff-739e-d0a3d5c6cc7b@ilbers.de> <040312d8-2496-dc62-5cbc-744dbf6e953c@ilbers.de> <9a978df9-2c52-7bd5-ad75-d6a1ff0269bc@siemens.com> <3668e211-0a0a-5837-c52f-4e0dbb1b94bf@siemens.com> From: Alexander Smirnov Message-ID: <30016591-1d6b-7a5d-20ad-a8ac30413585@ilbers.de> Date: Wed, 14 Feb 2018 20:57:59 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <3668e211-0a0a-5837-c52f-4e0dbb1b94bf@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: 8tzl1XBvGzs+ On 02/14/2018 06:02 PM, Jan Kiszka wrote: > On 2018-02-12 08:42, [ext] Jan Kiszka wrote: >> On 2018-02-11 19:44, Alexander Smirnov wrote: >>> >>> >>> On 02/11/2018 09:18 PM, Jan Kiszka wrote: >>>> On 2018-02-11 17:55, Alexander Smirnov wrote: >>>>> Hi, >>>>> >>>>> On 02/11/2018 06:17 PM, Jan Kiszka wrote: >>>>>> Hi all, >>>>>> >>>>>> I got this failure of example-hello:do_build already twice while doing >>>>>> rebuild tests with my kernel series (ie. with more independent >>>>>> buildchroot users): >>>>>> >>>>>> DEBUG: Executing shell function do_build >>>>>> Get:1 file:/isar-apt isar InRelease >>>>>> Ign:1 file:/isar-apt isar InRelease >>>>>> Get:2 file:/isar-apt isar Release [2,864 B] >>>>>> Get:2 file:/isar-apt isar Release [2,864 B] >>>>>> Get:3 file:/isar-apt isar Release.gpg >>>>>> Ign:3 file:/isar-apt isar Release.gpg >>>>>> Get:4 file:/isar-apt isar/main amd64 Packages [1,135 B] >>>>>> Reading package lists... >>>>>> W: The repository 'file:/isar-apt isar Release' is not signed. >>>>>> hostname: No address associated with hostname >>>>>> dh_testdir >>>>>> dh_testroot >>>>>> dh_prep >>>>>> dh_testdir >>>>>> dh_testroot >>>>>> dh_install >>>>>> dh_install: Compatibility levels before 9 are deprecated (level 7 in >>>>>> use) >>>>>> dh_installdocs >>>>>> dh_installdocs: Compatibility levels before 9 are deprecated (level 7 >>>>>> in use) >>>>>> dh_installchangelogs >>>>>> dh_compress >>>>>> dh_fixperms >>>>>> dh_installdeb >>>>>> dh_installdeb: Compatibility levels before 9 are deprecated (level 7 >>>>>> in use) >>>>>> dh_gencontrol >>>>>> dh_md5sums >>>>>> dh_builddeb >>>>>> dpkg-deb: building package 'hello-build-deps' in >>>>>> '../hello-build-deps_0.2_all.deb'. >>>>>> >>>>> >>>>> Good catch! >>>>> >>>>>> The package has been created. >>>>>> Attention, the package has been created in the current directory, >>>>>> not in ".." as indicated by the message above! >>>>>> dpkg: error: dpkg status database is locked by another process >>>>>> mk-build-deps: dpkg --unpack failed >>>>>> [...] >>>>>> >>>>>> So we have a concurrency problem when building over the same dpkg >>>>>> database. Looks like we need to synchronize (lock-protect) the >>>>>> access to >>>>>> it, which also means pulling out the dependency installation from the >>>>>> regular build step. Is that feasible at all? Any alternatives (besides >>>>>> retrying such builds...)? >>>>> >>>>> In general we could do this easily: >>>>> >>>>> 1. Split the content of build.sh into two functions, for example: >>>>>   - install_build_deps >>>>>   - build_package >>>>> >>>>> 2. Spit the bitbake do_build() into two tasks: >>>>> >>>>> do_install_build_deps() { >>>>>      ... build.sh install_build_deps ... >>>>> } >>>>> >>>>> addtask install_build_deps before do_build after do_unpack >>>>> >>>>> do_build_package() { >>>>>      ... build.sh build_package ... >>>>> } >>>>> >>>>> 3. Using bitbake synchronization primitives, protect the fist task from >>>>> parallel execution. >>>>> >>>>> If you are OK with this, I could do this tomorrow. >>>> >>>> I'm still concerned how well this will scale: >>>> >>>> a) We have additional users we already know of (linux-kernel.bbclass). >>>>     We will need to provide them the same means. >>>> >>>> b) There might be more users hidden in today's or future recipes... >>>> >>> >>> Now we have pipeline: >>>  - do_fetch, do_unpack, do_build. >>> >>> I propose to extend this pipeline by one extra task: >>>  - do_fetch, do_unpack, do_install_build_deps, do_build. >>> >>> These are the core tasks that have default payload defined in calsses, >>> so you should not touch them in custom recipes. >>> >>> For better scale-ability, a separate class could be created: >>> >>> 8<-- >>> >>> dpkg-build-deps.bbclass: >>> >>> do_install_build_deps() { >>>    ... call mk-build-deps >>> } >>> >>> addtask install_build_deps before do_build after do_unpack >>> >>> 8<-- >>> >>> So if you want this functionality in your class, for example in >>> dpkg.bbclass, so just include it. I can't imagine if we will have so >>> many different classes to build something... >> >> I'm concern about all dpkg calls. Are we sure there are no others to >> just query the database eg. which cannot fail? >> >> Otherwise, this sounds good to me. > > Already coded this into a path, or should I look into it? We desperately > need this to restore CI with all its concurrent builds - once they run > truly concurrent. If you mean whether I've done this or not, then no, I haven't. Could handle this in the next one-two-ays, at the moment there are several series pending for review/applying. Alex