Re: Build can sporadically fail due to dpkg contention in buildchroot

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2018-02-11 19:44, Alexander Smirnov wrote:
> 
> 
> On 02/11/2018 09:18 PM, Jan Kiszka wrote:
>> On 2018-02-11 17:55, Alexander Smirnov wrote:
>>> Hi,
>>>
>>> On 02/11/2018 06:17 PM, Jan Kiszka wrote:
>>>> Hi all,
>>>>
>>>> I got this failure of example-hello:do_build already twice while doing
>>>> rebuild tests with my kernel series (ie. with more independent
>>>> buildchroot users):
>>>>
>>>> DEBUG: Executing shell function do_build
>>>> Get:1 file:/isar-apt isar InRelease
>>>> Ign:1 file:/isar-apt isar InRelease
>>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>>> Get:3 file:/isar-apt isar Release.gpg
>>>> Ign:3 file:/isar-apt isar Release.gpg
>>>> Get:4 file:/isar-apt isar/main amd64 Packages [1,135 B]
>>>> Reading package lists...
>>>> W: The repository 'file:/isar-apt isar Release' is not signed.
>>>> hostname: No address associated with hostname
>>>> dh_testdir
>>>> dh_testroot
>>>> dh_prep
>>>> dh_testdir
>>>> dh_testroot
>>>> dh_install
>>>> dh_install: Compatibility levels before 9 are deprecated (level 7 in
>>>> use)
>>>> dh_installdocs
>>>> dh_installdocs: Compatibility levels before 9 are deprecated (level 7
>>>> in use)
>>>> dh_installchangelogs
>>>> dh_compress
>>>> dh_fixperms
>>>> dh_installdeb
>>>> dh_installdeb: Compatibility levels before 9 are deprecated (level 7
>>>> in use)
>>>> dh_gencontrol
>>>> dh_md5sums
>>>> dh_builddeb
>>>> dpkg-deb: building package 'hello-build-deps' in
>>>> '../hello-build-deps_0.2_all.deb'.
>>>>
>>>
>>> Good catch!
>>>
>>>> The package has been created.
>>>> Attention, the package has been created in the current directory,
>>>> not in ".." as indicated by the message above!
>>>> dpkg: error: dpkg status database is locked by another process
>>>> mk-build-deps: dpkg --unpack failed
>>>> [...]
>>>>
>>>> So we have a concurrency problem when building over the same dpkg
>>>> database. Looks like we need to synchronize (lock-protect) the
>>>> access to
>>>> it, which also means pulling out the dependency installation from the
>>>> regular build step. Is that feasible at all? Any alternatives (besides
>>>> retrying such builds...)?
>>>
>>> In general we could do this easily:
>>>
>>> 1. Split the content of build.sh into two functions, for example:
>>>   - install_build_deps
>>>   - build_package
>>>
>>> 2. Spit the bitbake do_build() into two tasks:
>>>
>>> do_install_build_deps() {
>>>      ... build.sh install_build_deps ...
>>> }
>>>
>>> addtask install_build_deps before do_build after do_unpack
>>>
>>> do_build_package() {
>>>      ... build.sh build_package ...
>>> }
>>>
>>> 3. Using bitbake synchronization primitives, protect the fist task from
>>> parallel execution.
>>>
>>> If you are OK with this, I could do this tomorrow.
>>
>> I'm still concerned how well this will scale:
>>
>> a) We have additional users we already know of (linux-kernel.bbclass).
>>     We will need to provide them the same means.
>>
>> b) There might be more users hidden in today's or future recipes...
>>
> 
> Now we have pipeline:
>  - do_fetch, do_unpack, do_build.
> 
> I propose to extend this pipeline by one extra task:
>  - do_fetch, do_unpack, do_install_build_deps, do_build.
> 
> These are the core tasks that have default payload defined in calsses,
> so you should not touch them in custom recipes.
> 
> For better scale-ability, a separate class could be created:
> 
> 8<--
> 
> dpkg-build-deps.bbclass:
> 
> do_install_build_deps() {
>    ... call mk-build-deps
> }
> 
> addtask install_build_deps before do_build after do_unpack
> 
> 8<--
> 
> So if you want this functionality in your class, for example in
> dpkg.bbclass, so just include it. I can't imagine if we will have so
> many different classes to build something...

I'm concern about all dpkg calls. Are we sure there are no others to
just query the database eg. which cannot fail?

Otherwise, this sounds good to me.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux