Re: isar-bootstrap fails to rebuild after config changes

[Jan Kiszka <jan.kiszka@siemens.com>]

On 13.08.19 10:14, [ext] Jan Kiszka wrote:
> On 13.08.19 10:01, Claudius Heine wrote:
>> Hi Jan,
>>
>> On 12/08/2019 19.18, [ext] Jan Kiszka wrote:
>>> Hi,
>>>
>>> I noticed today that I'm not getting a new bootstrap rootfs after adding a 
>>> repo to DISTRO_APT_SOURCES. Digging deeper, this revealed missing STAMPCLEAN 
>>> setting, but also more:
>>>
>>> Why are we probing for the existence of the DEPLOY_ISAR_BOOTSTRAP link in 
>>> do_apt_config_prepare(), Claudius? This obviously exists after a first run, 
>>> and that blocks refreshing all the apt settings if they have changed.
>>
>> I do not know. That was added by a patch of Maxim 
>> (ab0a1c8c357a18ddd67fe4c0efc66a296d93dae2), maybe he knows.
>>
>>      isar-bootstrap: Eliminate multiple debootstraps for the same distro/host
>>
>>      After applying this patch only single debootstrap for
>>      particular platform is created and pointed by corresponding
>>      link under DEPLOY_DIR_IMAGE.
>>
>>      For proper locking of parallel builds shell tasks
>>      apt_config_install, set_locale, setup_chroot, apt_update
>>      and deploy were collapsed to common debootstrap helper.
>>      Due to problems in bitbake with shell functions expansion
>>      under quotes these functions were substituted  by their
>>      bodies.
>>
>>      Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
>>
>> That is by the way a pretty big patch. Also parts of it could probably be 
>> reverted, since the 'problems in bitbake with shell functions expansion under 
>> quotes' are resolved in the current version by removing these quotes IIUC.
>>
>>> There is CLEANFUNCS set with clean_deploy(), but that is only run on true 
>>> do_clean(), not on rebuilds.
>>
>> Right.
>>
>>>
>>> And finally, when I look at isar_bootstrap(), run by do_bootstrap(), it 
>>> protects checking and creating DEPLOY_ISAR_BOOTSTRAP with a lock. But 
>>> do_apt_config_prepare() runs lockless. This cannot work reliably.
>>
>> When I initially wrote isar-bootstrap, it didn't contain any looks. IIRC the 
>> locks where added when I wasn't active with isar and I haven't yet looked into 
>> why exactly they are needed and what they protect against etc. So you know 
>> probably more about them than me.
> 
> Ah, sorry, wrongly accounted you for this.
> 
> Maxim, can you comment on the design idea and the issues I found?
> 

Actually, I do not even see the point of locking here: If a bootstrap target is 
shared, only one multiconfig target will request its build and the others will 
wait automatically for the result. That's what bitbake is supposed to do for us. 
If it's not shared, there is no conflict to use locks on. We only need locks if 
different tasks of different recipes compete for common resources.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux