Re: Build can sporadically fail due to dpkg contention in buildchroot

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2018-02-12 08:42, [ext] Jan Kiszka wrote:
> On 2018-02-11 19:44, Alexander Smirnov wrote:
>>
>>
>> On 02/11/2018 09:18 PM, Jan Kiszka wrote:
>>> On 2018-02-11 17:55, Alexander Smirnov wrote:
>>>> Hi,
>>>>
>>>> On 02/11/2018 06:17 PM, Jan Kiszka wrote:
>>>>> Hi all,
>>>>>
>>>>> I got this failure of example-hello:do_build already twice while doing
>>>>> rebuild tests with my kernel series (ie. with more independent
>>>>> buildchroot users):
>>>>>
>>>>> DEBUG: Executing shell function do_build
>>>>> Get:1 file:/isar-apt isar InRelease
>>>>> Ign:1 file:/isar-apt isar InRelease
>>>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>>>> Get:3 file:/isar-apt isar Release.gpg
>>>>> Ign:3 file:/isar-apt isar Release.gpg
>>>>> Get:4 file:/isar-apt isar/main amd64 Packages [1,135 B]
>>>>> Reading package lists...
>>>>> W: The repository 'file:/isar-apt isar Release' is not signed.
>>>>> hostname: No address associated with hostname
>>>>> dh_testdir
>>>>> dh_testroot
>>>>> dh_prep
>>>>> dh_testdir
>>>>> dh_testroot
>>>>> dh_install
>>>>> dh_install: Compatibility levels before 9 are deprecated (level 7 in
>>>>> use)
>>>>> dh_installdocs
>>>>> dh_installdocs: Compatibility levels before 9 are deprecated (level 7
>>>>> in use)
>>>>> dh_installchangelogs
>>>>> dh_compress
>>>>> dh_fixperms
>>>>> dh_installdeb
>>>>> dh_installdeb: Compatibility levels before 9 are deprecated (level 7
>>>>> in use)
>>>>> dh_gencontrol
>>>>> dh_md5sums
>>>>> dh_builddeb
>>>>> dpkg-deb: building package 'hello-build-deps' in
>>>>> '../hello-build-deps_0.2_all.deb'.
>>>>>
>>>>
>>>> Good catch!
>>>>
>>>>> The package has been created.
>>>>> Attention, the package has been created in the current directory,
>>>>> not in ".." as indicated by the message above!
>>>>> dpkg: error: dpkg status database is locked by another process
>>>>> mk-build-deps: dpkg --unpack failed
>>>>> [...]
>>>>>
>>>>> So we have a concurrency problem when building over the same dpkg
>>>>> database. Looks like we need to synchronize (lock-protect) the
>>>>> access to
>>>>> it, which also means pulling out the dependency installation from the
>>>>> regular build step. Is that feasible at all? Any alternatives (besides
>>>>> retrying such builds...)?
>>>>
>>>> In general we could do this easily:
>>>>
>>>> 1. Split the content of build.sh into two functions, for example:
>>>>   - install_build_deps
>>>>   - build_package
>>>>
>>>> 2. Spit the bitbake do_build() into two tasks:
>>>>
>>>> do_install_build_deps() {
>>>>      ... build.sh install_build_deps ...
>>>> }
>>>>
>>>> addtask install_build_deps before do_build after do_unpack
>>>>
>>>> do_build_package() {
>>>>      ... build.sh build_package ...
>>>> }
>>>>
>>>> 3. Using bitbake synchronization primitives, protect the fist task from
>>>> parallel execution.
>>>>
>>>> If you are OK with this, I could do this tomorrow.
>>>
>>> I'm still concerned how well this will scale:
>>>
>>> a) We have additional users we already know of (linux-kernel.bbclass).
>>>     We will need to provide them the same means.
>>>
>>> b) There might be more users hidden in today's or future recipes...
>>>
>>
>> Now we have pipeline:
>>  - do_fetch, do_unpack, do_build.
>>
>> I propose to extend this pipeline by one extra task:
>>  - do_fetch, do_unpack, do_install_build_deps, do_build.
>>
>> These are the core tasks that have default payload defined in calsses,
>> so you should not touch them in custom recipes.
>>
>> For better scale-ability, a separate class could be created:
>>
>> 8<--
>>
>> dpkg-build-deps.bbclass:
>>
>> do_install_build_deps() {
>>    ... call mk-build-deps
>> }
>>
>> addtask install_build_deps before do_build after do_unpack
>>
>> 8<--
>>
>> So if you want this functionality in your class, for example in
>> dpkg.bbclass, so just include it. I can't imagine if we will have so
>> many different classes to build something...
> 
> I'm concern about all dpkg calls. Are we sure there are no others to
> just query the database eg. which cannot fail?
> 
> Otherwise, this sounds good to me.

Already coded this into a path, or should I look into it? We desperately
need this to restore CI with all its concurrent builds - once they run
truly concurrent.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux