Re: [PATCH v2 0/9] Add support for automatic partial rebuilds on recipe changes

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2018-02-14 15:03, Alexander Smirnov wrote:
> 
> 
> On 02/14/2018 04:38 PM, Jan Kiszka wrote:
>> On 2018-02-14 14:29, Jan Kiszka wrote:
>>> On 2018-02-14 14:10, Jan Kiszka wrote:
>>>> On 2018-02-14 13:57, Alexander Smirnov wrote:
>>>>> On 02/14/2018 03:41 PM, Jan Kiszka wrote:
>>>>>> On 2018-02-14 12:33, Alexander Smirnov wrote:
>>>>>>> On 02/13/2018 11:05 PM, Jan Kiszka wrote:
>>>>>>>> Yeah, finally Yocto/OE-like usability: This ensures for many
>>>>>>>> cases that
>>>>>>>> changes to recipes lead to rebuilds of dependent recipes,
>>>>>>>> including the
>>>>>>>> final image. Some extra measures are needed so that those
>>>>>>>> rebuilds work
>>>>>>>> with clean dirs.
>>>>>>>>
>>>>>>>> And if the change detection should not work, e.g. changes to
>>>>>>>> file://
>>>>>>>> resources are not detected, then a clean or cleanall task is now
>>>>>>>> available and ensures a proper manual rebuild.
>>>>>>>>
>>>>>>>> This massively increases the fun factor when developing Isar
>>>>>>>> projects.
>>>>>>>
>>>>>>>
>>>>>>> Wow, just noticed that with this series bitbake has started to
>>>>>>> run tasks
>>>>>>> in parallel.. How it's working? :-)
>>>>>>>
>>>>>>>
>>>>>>> Currently  4 running tasks (54 of 217)  24%
>>>>>>> |########################################
>>>>>>>                        |
>>>>>>> 0: mc:rpi-jessie:buildchroot-1.0-r0 do_build - 101s (pid 3271)
>>>>>>> 1: mc:qemui386-stretch:buildchroot-1.0-r0 do_build - 101s (pid 3336)
>>>>>>> 2: mc:qemui386-jessie:buildchroot-1.0-r0 do_build - 100s (pid 3457)
>>>>>>> 3: mc:qemuarm-wheezy:buildchroot-1.0-r0 do_build - 100s (pid 3524)
>>>>>>
>>>>>> Didn't notice that this wasn't the case before (most of my
>>>>>> workload had
>>>>>> linear ordering), but I could imagine that the lacking signatures
>>>>>> forced
>>>>>> bitbake to serialize. Just guessing.
>>>>>>
>>>>>
>>>>> There is still the same build problem as reported yesterday, log is
>>>>> attached. Dropping patch #3 fixes it.
>>>>>
>>>>> Line I used to build:
>>>>>
>>>>> $ time bitbake multiconfig:qemuarm-wheezy:isar-image-base
>>>>> multiconfig:qemuarm-jessie:isar-image-base
>>>>> multiconfig:qemuarm-stretch:isar-image-base
>>>>> multiconfig:qemui386-jessie:isar-image-base
>>>>> multiconfig:qemui386-stretch:isar-image-base
>>>>> multiconfig:qemuamd64-jessie:isar-image-base
>>>>> multiconfig:qemuamd64-stretch:isar-image-base
>>>>> multiconfig:rpi-jessie:isar-image-base
>>>>>
>>>>> Just for sure stated build in CI server:
>>>>> http://isar-build.org:8080/job/isar_asmirnov_devel/8/console
>>>>>
>>>>> Also failed. Could I dropped this patch from the series?
>>>>
>>>> Not before you can explain why things fail and why that patch causes
>>>> this. We need to understand the problem. Let me reproduce...
>>>
>>> Yeah, I get some bug as well, but a different & well-known one:
>>>
>>> dpkg: error: dpkg status database is locked by another process
>>>
>>> So, I'm afraid we will find more of these sporadic breakage, now that
>>> parallelization works better....
>>
>> BTW, I'm testing with
>> https://github.com/siemens/isar/commits/jan/staging, i.e. the sum of all
>> my misdoing.
>>
>> Good news is that I just reproduced your issue, now digging into what
>> corrupts the isar-apt repo /wrt libhello-dev. Bad news: I had to restart
>> 5 times due to the contention bug above.
>>
> 
> I think that the issue is in repo_clean function. "isar_apt" splits
> databases by ${DISTRO}:
> 
> builder@zbook:~/isar/build$ ls tmp/deploy/apt/ -1
> debian-jessie
> debian-stretch
> debian-wheezy
> raspbian-jessie
> 
> And each distro could contains packages of whatever arch.
> 
> I suppose that clean_repo() as it's called, remove all the architectures
> for specified package name. So building for ARM erases content for AMD
> and i386. Looking into the manual, -A option should be passed to reprepro.

Yes, exactly. I will try the -A next.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux