From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6521574339082452992 X-Received: by 10.28.15.145 with SMTP id 139mr540234wmp.27.1518617406401; Wed, 14 Feb 2018 06:10:06 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.138.19 with SMTP id m19ls2158471wmd.5.gmail; Wed, 14 Feb 2018 06:10:05 -0800 (PST) X-Google-Smtp-Source: AH8x226LdEDHH33u8w9OamUFW/zZW4WL6QFt+eL1TbhrjWmJe96dVeJxDYAlhplpqHXDaKKsS0PG X-Received: by 10.28.239.20 with SMTP id n20mr539535wmh.23.1518617405723; Wed, 14 Feb 2018 06:10:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1518617405; cv=none; d=google.com; s=arc-20160816; b=rnUMS2oPwfAhH5yK4mehTlDEoR0puEUwahiX0a3fT9L3G9li9PruS0M2hOuGFo1ajQ 5P6A9SnZqpz3VCFJnEpfEvrXn75Tnlf4CNw45C10vuBx8ovqImBpFCfixGEXUS3t+RVD lXGyUX4pRjZSy512d0Lard8t34nKOkOa3L4Xb+VaxouRmqeiPp8L+gjsHpi2ds0eUwvx 8cEdTqtS/GJw/+YikY5z5wwFDBBuUSlIyeyOJMql7qAE4V1clFVXiMmLiptyz8iMtYU6 CFRHuKn93B/2ONPiJIcZ5z30Hg41Ev2V4kQ6aZlZZKP0RYAtH8o+EGRxb9xLTiR1Jp62 0MfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=4IvWUmAfZhslOSEGw1zwyWiFovdI9PqFoFl1rIUqpZk=; b=S8CvCNwgWJ9R6+7ZdmxlLxqczRkQerkC0s3zXvO7sv/vjl5dluAY1hSNCOVr3jf/qm RBLzPZrngtarWyc914hJMwAauUODQiuUmrhCuyd00gsX/zFf/8uAbdnywjKxR3Ei7Nln MYNiVU2wUQzG3XyW7Un46sG+big876w4UlmExxeMb04mY3Tm6Uq5VgrKtDMyIRtUE6n+ Ej152v5gRxdLI6BkLHmxzOoP1KMTsWNKI7MUUAzdCmmCm6N/djz6hYxyv1J52zUuDkvf s0YWMqVtLQTWWjLKiptXoAn2E5WOZGfR2xNM+RzoJa4unP7+NUlkX25emJnKhpZlGSyH A5UQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id r20si138074wmg.0.2018.02.14.06.10.05 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 14 Feb 2018 06:10:05 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail3.siemens.de (mail3.siemens.de [139.25.208.14]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id w1EEA5lv001808 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Feb 2018 15:10:05 +0100 Received: from [139.25.68.37] (md1q0hnc.ad001.siemens.net [139.25.68.37] (may be forged)) by mail3.siemens.de (8.15.2/8.15.2) with ESMTP id w1EEA5dZ009564; Wed, 14 Feb 2018 15:10:05 +0100 Subject: Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes To: Alexander Smirnov , isar-users References: <96a5d7df-b7a3-08ec-3c28-85ecf332d32b@siemens.com> <01a34333-06a2-4df0-a51e-17aced9f9ee3@ilbers.de> <7bd0cc56-0dba-6a04-767a-de4edcc3be1e@siemens.com> <96dd5116-fc00-d5df-d48b-c91b6d332cae@siemens.com> From: Jan Kiszka Message-ID: <3a736b21-e9a8-199c-7329-39cbde57139e@siemens.com> Date: Wed, 14 Feb 2018 15:10:04 +0100 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: H4plNYP30cfq On 2018-02-14 15:03, Alexander Smirnov wrote: > > > On 02/14/2018 04:38 PM, Jan Kiszka wrote: >> On 2018-02-14 14:29, Jan Kiszka wrote: >>> On 2018-02-14 14:10, Jan Kiszka wrote: >>>> On 2018-02-14 13:57, Alexander Smirnov wrote: >>>>> On 02/14/2018 03:41 PM, Jan Kiszka wrote: >>>>>> On 2018-02-14 12:33, Alexander Smirnov wrote: >>>>>>> On 02/13/2018 11:05 PM, Jan Kiszka wrote: >>>>>>>> Yeah, finally Yocto/OE-like usability: This ensures for many >>>>>>>> cases that >>>>>>>> changes to recipes lead to rebuilds of dependent recipes, >>>>>>>> including the >>>>>>>> final image. Some extra measures are needed so that those >>>>>>>> rebuilds work >>>>>>>> with clean dirs. >>>>>>>> >>>>>>>> And if the change detection should not work, e.g. changes to >>>>>>>> file:// >>>>>>>> resources are not detected, then a clean or cleanall task is now >>>>>>>> available and ensures a proper manual rebuild. >>>>>>>> >>>>>>>> This massively increases the fun factor when developing Isar >>>>>>>> projects. >>>>>>> >>>>>>> >>>>>>> Wow, just noticed that with this series bitbake has started to >>>>>>> run tasks >>>>>>> in parallel.. How it's working? :-) >>>>>>> >>>>>>> >>>>>>> Currently  4 running tasks (54 of 217)  24% >>>>>>> |######################################## >>>>>>>                        | >>>>>>> 0: mc:rpi-jessie:buildchroot-1.0-r0 do_build - 101s (pid 3271) >>>>>>> 1: mc:qemui386-stretch:buildchroot-1.0-r0 do_build - 101s (pid 3336) >>>>>>> 2: mc:qemui386-jessie:buildchroot-1.0-r0 do_build - 100s (pid 3457) >>>>>>> 3: mc:qemuarm-wheezy:buildchroot-1.0-r0 do_build - 100s (pid 3524) >>>>>> >>>>>> Didn't notice that this wasn't the case before (most of my >>>>>> workload had >>>>>> linear ordering), but I could imagine that the lacking signatures >>>>>> forced >>>>>> bitbake to serialize. Just guessing. >>>>>> >>>>> >>>>> There is still the same build problem as reported yesterday, log is >>>>> attached. Dropping patch #3 fixes it. >>>>> >>>>> Line I used to build: >>>>> >>>>> $ time bitbake multiconfig:qemuarm-wheezy:isar-image-base >>>>> multiconfig:qemuarm-jessie:isar-image-base >>>>> multiconfig:qemuarm-stretch:isar-image-base >>>>> multiconfig:qemui386-jessie:isar-image-base >>>>> multiconfig:qemui386-stretch:isar-image-base >>>>> multiconfig:qemuamd64-jessie:isar-image-base >>>>> multiconfig:qemuamd64-stretch:isar-image-base >>>>> multiconfig:rpi-jessie:isar-image-base >>>>> >>>>> Just for sure stated build in CI server: >>>>> http://isar-build.org:8080/job/isar_asmirnov_devel/8/console >>>>> >>>>> Also failed. Could I dropped this patch from the series? >>>> >>>> Not before you can explain why things fail and why that patch causes >>>> this. We need to understand the problem. Let me reproduce... >>> >>> Yeah, I get some bug as well, but a different & well-known one: >>> >>> dpkg: error: dpkg status database is locked by another process >>> >>> So, I'm afraid we will find more of these sporadic breakage, now that >>> parallelization works better.... >> >> BTW, I'm testing with >> https://github.com/siemens/isar/commits/jan/staging, i.e. the sum of all >> my misdoing. >> >> Good news is that I just reproduced your issue, now digging into what >> corrupts the isar-apt repo /wrt libhello-dev. Bad news: I had to restart >> 5 times due to the contention bug above. >> > > I think that the issue is in repo_clean function. "isar_apt" splits > databases by ${DISTRO}: > > builder@zbook:~/isar/build$ ls tmp/deploy/apt/ -1 > debian-jessie > debian-stretch > debian-wheezy > raspbian-jessie > > And each distro could contains packages of whatever arch. > > I suppose that clean_repo() as it's called, remove all the architectures > for specified package name. So building for ARM erases content for AMD > and i386. Looking into the manual, -A option should be passed to reprepro. Yes, exactly. I will try the -A next. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux