Re: Build can sporadically fail due to dpkg contention in buildchroot

[Alexander Smirnov <asmirnov@ilbers.de>]

On 02/14/2018 06:02 PM, Jan Kiszka wrote:
> On 2018-02-12 08:42, [ext] Jan Kiszka wrote:
>> On 2018-02-11 19:44, Alexander Smirnov wrote:
>>>
>>>
>>> On 02/11/2018 09:18 PM, Jan Kiszka wrote:
>>>> On 2018-02-11 17:55, Alexander Smirnov wrote:
>>>>> Hi,
>>>>>
>>>>> On 02/11/2018 06:17 PM, Jan Kiszka wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I got this failure of example-hello:do_build already twice while doing
>>>>>> rebuild tests with my kernel series (ie. with more independent
>>>>>> buildchroot users):
>>>>>>
>>>>>> DEBUG: Executing shell function do_build
>>>>>> Get:1 file:/isar-apt isar InRelease
>>>>>> Ign:1 file:/isar-apt isar InRelease
>>>>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>>>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>>>>> Get:3 file:/isar-apt isar Release.gpg
>>>>>> Ign:3 file:/isar-apt isar Release.gpg
>>>>>> Get:4 file:/isar-apt isar/main amd64 Packages [1,135 B]
>>>>>> Reading package lists...
>>>>>> W: The repository 'file:/isar-apt isar Release' is not signed.
>>>>>> hostname: No address associated with hostname
>>>>>> dh_testdir
>>>>>> dh_testroot
>>>>>> dh_prep
>>>>>> dh_testdir
>>>>>> dh_testroot
>>>>>> dh_install
>>>>>> dh_install: Compatibility levels before 9 are deprecated (level 7 in
>>>>>> use)
>>>>>> dh_installdocs
>>>>>> dh_installdocs: Compatibility levels before 9 are deprecated (level 7
>>>>>> in use)
>>>>>> dh_installchangelogs
>>>>>> dh_compress
>>>>>> dh_fixperms
>>>>>> dh_installdeb
>>>>>> dh_installdeb: Compatibility levels before 9 are deprecated (level 7
>>>>>> in use)
>>>>>> dh_gencontrol
>>>>>> dh_md5sums
>>>>>> dh_builddeb
>>>>>> dpkg-deb: building package 'hello-build-deps' in
>>>>>> '../hello-build-deps_0.2_all.deb'.
>>>>>>
>>>>>
>>>>> Good catch!
>>>>>
>>>>>> The package has been created.
>>>>>> Attention, the package has been created in the current directory,
>>>>>> not in ".." as indicated by the message above!
>>>>>> dpkg: error: dpkg status database is locked by another process
>>>>>> mk-build-deps: dpkg --unpack failed
>>>>>> [...]
>>>>>>
>>>>>> So we have a concurrency problem when building over the same dpkg
>>>>>> database. Looks like we need to synchronize (lock-protect) the
>>>>>> access to
>>>>>> it, which also means pulling out the dependency installation from the
>>>>>> regular build step. Is that feasible at all? Any alternatives (besides
>>>>>> retrying such builds...)?
>>>>>
>>>>> In general we could do this easily:
>>>>>
>>>>> 1. Split the content of build.sh into two functions, for example:
>>>>>    - install_build_deps
>>>>>    - build_package
>>>>>
>>>>> 2. Spit the bitbake do_build() into two tasks:
>>>>>
>>>>> do_install_build_deps() {
>>>>>       ... build.sh install_build_deps ...
>>>>> }
>>>>>
>>>>> addtask install_build_deps before do_build after do_unpack
>>>>>
>>>>> do_build_package() {
>>>>>       ... build.sh build_package ...
>>>>> }
>>>>>
>>>>> 3. Using bitbake synchronization primitives, protect the fist task from
>>>>> parallel execution.
>>>>>
>>>>> If you are OK with this, I could do this tomorrow.
>>>>
>>>> I'm still concerned how well this will scale:
>>>>
>>>> a) We have additional users we already know of (linux-kernel.bbclass).
>>>>      We will need to provide them the same means.
>>>>
>>>> b) There might be more users hidden in today's or future recipes...
>>>>
>>>
>>> Now we have pipeline:
>>>   - do_fetch, do_unpack, do_build.
>>>
>>> I propose to extend this pipeline by one extra task:
>>>   - do_fetch, do_unpack, do_install_build_deps, do_build.
>>>
>>> These are the core tasks that have default payload defined in calsses,
>>> so you should not touch them in custom recipes.
>>>
>>> For better scale-ability, a separate class could be created:
>>>
>>> 8<--
>>>
>>> dpkg-build-deps.bbclass:
>>>
>>> do_install_build_deps() {
>>>     ... call mk-build-deps
>>> }
>>>
>>> addtask install_build_deps before do_build after do_unpack
>>>
>>> 8<--
>>>
>>> So if you want this functionality in your class, for example in
>>> dpkg.bbclass, so just include it. I can't imagine if we will have so
>>> many different classes to build something...
>>
>> I'm concern about all dpkg calls. Are we sure there are no others to
>> just query the database eg. which cannot fail?
>>
>> Otherwise, this sounds good to me.
> 
> Already coded this into a path, or should I look into it? We desperately
> need this to restore CI with all its concurrent builds - once they run
> truly concurrent.

If you mean whether I've done this or not, then no, I haven't. Could 
handle this in the next one-two-ays, at the moment there are several 
series pending for review/applying.

Alex