From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6521574339082452992 X-Received: by 10.28.145.13 with SMTP id t13mr545681wmd.14.1518616175221; Wed, 14 Feb 2018 05:49:35 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.47.194 with SMTP id v185ls2157916wmv.4.canary-gmail; Wed, 14 Feb 2018 05:49:34 -0800 (PST) X-Google-Smtp-Source: AH8x2263guSsbO+yN67Ru+NH9pbM+UCXRd5FHR3hjqwBgJozAvhaRjouTx7qUxv05vyna8MYXrIe X-Received: by 10.28.87.5 with SMTP id l5mr561028wmb.23.1518616174498; Wed, 14 Feb 2018 05:49:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1518616174; cv=none; d=google.com; s=arc-20160816; b=IH15kPTCv1LDbVJGnf4b2AjOx/8c6iN/UFX/HKYXV8fblzKGL/xOq4ZLrfdOzaQRMh A5ZSdi6dhRZlgEE5X0QyC1NOOUdKiv/6J17BTRbnDkxFfuLrrMSdNlmOOTujLdmWsd1M Aks0sZUc/PLe8KoX+WKL9rnydhmfhGmfudOdS9qVkgiPzvHScZpcCmGgwjz4V/yNFypd 2z6+EWesGWKwJ0m5PkToG1JKYnK0F4wtX2bhFVhKRKcoQcgweDS1C02aa9GzBatTrQhV YfHnIDvAtBEaQtnr0CrfD4HGxntNr+5rMcXAO3xDRmKNTDrFDlFAG2FDjroZX7Qj5y8u JwNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:references:to:from:subject :arc-authentication-results; bh=zEoaQP3qfaEWKvYlC9bEsKChXkjIft4iqEk+WEjMW5Q=; b=HhzxoWmZfiDd/eMCf92orf/x7o8ScsmuC8RwLJeZPwDSyKVgz66ymNaLSyzuEIKdKN ddomNqr5D6ObPz5HBRKb2JmdXBpsUQKUITT+rYiU7+c+4TyKLe6Gh6bec6lp/hvBqyOn mJiwREoM7ZmgJGbIlhMbxmZOO51dCL8QrkK4FPFo63potDNVG2HMr8UWb5ZGUaz2pAZ2 YyL2mfZWNa7P07D2wa44w9gdqSdhfibza5cocpipXHYSm/VTVIjvIaWv8AGR52esi3ev 3ISfKd+7RTUFoYZ90cmxJYhDVyee9AclMlJPJQwtHenQkHLeGtNmzy7qXnCqS03tUYNy qa1A== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id s81si652911wmd.2.2018.02.14.05.49.34 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 14 Feb 2018 05:49:34 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail3.siemens.de (mail3.siemens.de [139.25.208.14]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id w1EDnXod028553 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Feb 2018 14:49:34 +0100 Received: from [139.25.68.37] (md1q0hnc.ad001.siemens.net [139.25.68.37] (may be forged)) by mail3.siemens.de (8.15.2/8.15.2) with ESMTP id w1EDnXAh022673; Wed, 14 Feb 2018 14:49:33 +0100 Subject: Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes From: Jan Kiszka To: Alexander Smirnov , isar-users References: <96a5d7df-b7a3-08ec-3c28-85ecf332d32b@siemens.com> <01a34333-06a2-4df0-a51e-17aced9f9ee3@ilbers.de> <7bd0cc56-0dba-6a04-767a-de4edcc3be1e@siemens.com> <96dd5116-fc00-d5df-d48b-c91b6d332cae@siemens.com> Message-ID: <43c8ff54-68d3-e55c-6e0c-a2e60f53db30@siemens.com> Date: Wed, 14 Feb 2018 14:49:33 +0100 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <96dd5116-fc00-d5df-d48b-c91b6d332cae@siemens.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: CU/qKtLmEVmd On 2018-02-14 14:38, Jan Kiszka wrote: > On 2018-02-14 14:29, Jan Kiszka wrote: >> On 2018-02-14 14:10, Jan Kiszka wrote: >>> On 2018-02-14 13:57, Alexander Smirnov wrote: >>>> On 02/14/2018 03:41 PM, Jan Kiszka wrote: >>>>> On 2018-02-14 12:33, Alexander Smirnov wrote: >>>>>> On 02/13/2018 11:05 PM, Jan Kiszka wrote: >>>>>>> Yeah, finally Yocto/OE-like usability: This ensures for many cases that >>>>>>> changes to recipes lead to rebuilds of dependent recipes, including the >>>>>>> final image. Some extra measures are needed so that those rebuilds work >>>>>>> with clean dirs. >>>>>>> >>>>>>> And if the change detection should not work, e.g. changes to file:// >>>>>>> resources are not detected, then a clean or cleanall task is now >>>>>>> available and ensures a proper manual rebuild. >>>>>>> >>>>>>> This massively increases the fun factor when developing Isar projects. >>>>>> >>>>>> >>>>>> Wow, just noticed that with this series bitbake has started to run tasks >>>>>> in parallel.. How it's working? :-) >>>>>> >>>>>> >>>>>> Currently  4 running tasks (54 of 217)  24% >>>>>> |######################################## >>>>>>                       | >>>>>> 0: mc:rpi-jessie:buildchroot-1.0-r0 do_build - 101s (pid 3271) >>>>>> 1: mc:qemui386-stretch:buildchroot-1.0-r0 do_build - 101s (pid 3336) >>>>>> 2: mc:qemui386-jessie:buildchroot-1.0-r0 do_build - 100s (pid 3457) >>>>>> 3: mc:qemuarm-wheezy:buildchroot-1.0-r0 do_build - 100s (pid 3524) >>>>> >>>>> Didn't notice that this wasn't the case before (most of my workload had >>>>> linear ordering), but I could imagine that the lacking signatures forced >>>>> bitbake to serialize. Just guessing. >>>>> >>>> >>>> There is still the same build problem as reported yesterday, log is >>>> attached. Dropping patch #3 fixes it. >>>> >>>> Line I used to build: >>>> >>>> $ time bitbake multiconfig:qemuarm-wheezy:isar-image-base >>>> multiconfig:qemuarm-jessie:isar-image-base >>>> multiconfig:qemuarm-stretch:isar-image-base >>>> multiconfig:qemui386-jessie:isar-image-base >>>> multiconfig:qemui386-stretch:isar-image-base >>>> multiconfig:qemuamd64-jessie:isar-image-base >>>> multiconfig:qemuamd64-stretch:isar-image-base >>>> multiconfig:rpi-jessie:isar-image-base >>>> >>>> Just for sure stated build in CI server: >>>> http://isar-build.org:8080/job/isar_asmirnov_devel/8/console >>>> >>>> Also failed. Could I dropped this patch from the series? >>> >>> Not before you can explain why things fail and why that patch causes >>> this. We need to understand the problem. Let me reproduce... >> >> Yeah, I get some bug as well, but a different & well-known one: >> >> dpkg: error: dpkg status database is locked by another process >> >> So, I'm afraid we will find more of these sporadic breakage, now that >> parallelization works better.... > > BTW, I'm testing with > https://github.com/siemens/isar/commits/jan/staging, i.e. the sum of all > my misdoing. > > Good news is that I just reproduced your issue, now digging into what > corrupts the isar-apt repo /wrt libhello-dev. Bad news: I had to restart > 5 times due to the contention bug above. OK, understood this one: repo_clean. The arm task kills the results of the i386 or vice versa. Need to work on the filtering rules here. Or we need to split the isar-apt repos per multiconf target... Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux