Re: Build can sporadically fail due to dpkg contention in buildchroot

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2018-02-14 18:57, Alexander Smirnov wrote:
> On 02/14/2018 06:02 PM, Jan Kiszka wrote:
>> On 2018-02-12 08:42, [ext] Jan Kiszka wrote:
>>> On 2018-02-11 19:44, Alexander Smirnov wrote:
>>>>
>>>>
>>>> On 02/11/2018 09:18 PM, Jan Kiszka wrote:
>>>>> On 2018-02-11 17:55, Alexander Smirnov wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On 02/11/2018 06:17 PM, Jan Kiszka wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I got this failure of example-hello:do_build already twice while
>>>>>>> doing
>>>>>>> rebuild tests with my kernel series (ie. with more independent
>>>>>>> buildchroot users):
>>>>>>>
>>>>>>> DEBUG: Executing shell function do_build
>>>>>>> Get:1 file:/isar-apt isar InRelease
>>>>>>> Ign:1 file:/isar-apt isar InRelease
>>>>>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>>>>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>>>>>> Get:3 file:/isar-apt isar Release.gpg
>>>>>>> Ign:3 file:/isar-apt isar Release.gpg
>>>>>>> Get:4 file:/isar-apt isar/main amd64 Packages [1,135 B]
>>>>>>> Reading package lists...
>>>>>>> W: The repository 'file:/isar-apt isar Release' is not signed.
>>>>>>> hostname: No address associated with hostname
>>>>>>> dh_testdir
>>>>>>> dh_testroot
>>>>>>> dh_prep
>>>>>>> dh_testdir
>>>>>>> dh_testroot
>>>>>>> dh_install
>>>>>>> dh_install: Compatibility levels before 9 are deprecated (level 7 in
>>>>>>> use)
>>>>>>> dh_installdocs
>>>>>>> dh_installdocs: Compatibility levels before 9 are deprecated
>>>>>>> (level 7
>>>>>>> in use)
>>>>>>> dh_installchangelogs
>>>>>>> dh_compress
>>>>>>> dh_fixperms
>>>>>>> dh_installdeb
>>>>>>> dh_installdeb: Compatibility levels before 9 are deprecated (level 7
>>>>>>> in use)
>>>>>>> dh_gencontrol
>>>>>>> dh_md5sums
>>>>>>> dh_builddeb
>>>>>>> dpkg-deb: building package 'hello-build-deps' in
>>>>>>> '../hello-build-deps_0.2_all.deb'.
>>>>>>>
>>>>>>
>>>>>> Good catch!
>>>>>>
>>>>>>> The package has been created.
>>>>>>> Attention, the package has been created in the current directory,
>>>>>>> not in ".." as indicated by the message above!
>>>>>>> dpkg: error: dpkg status database is locked by another process
>>>>>>> mk-build-deps: dpkg --unpack failed
>>>>>>> [...]
>>>>>>>
>>>>>>> So we have a concurrency problem when building over the same dpkg
>>>>>>> database. Looks like we need to synchronize (lock-protect) the
>>>>>>> access to
>>>>>>> it, which also means pulling out the dependency installation from
>>>>>>> the
>>>>>>> regular build step. Is that feasible at all? Any alternatives
>>>>>>> (besides
>>>>>>> retrying such builds...)?
>>>>>>
>>>>>> In general we could do this easily:
>>>>>>
>>>>>> 1. Split the content of build.sh into two functions, for example:
>>>>>>    - install_build_deps
>>>>>>    - build_package
>>>>>>
>>>>>> 2. Spit the bitbake do_build() into two tasks:
>>>>>>
>>>>>> do_install_build_deps() {
>>>>>>       ... build.sh install_build_deps ...
>>>>>> }
>>>>>>
>>>>>> addtask install_build_deps before do_build after do_unpack
>>>>>>
>>>>>> do_build_package() {
>>>>>>       ... build.sh build_package ...
>>>>>> }
>>>>>>
>>>>>> 3. Using bitbake synchronization primitives, protect the fist task
>>>>>> from
>>>>>> parallel execution.
>>>>>>
>>>>>> If you are OK with this, I could do this tomorrow.
>>>>>
>>>>> I'm still concerned how well this will scale:
>>>>>
>>>>> a) We have additional users we already know of (linux-kernel.bbclass).
>>>>>      We will need to provide them the same means.
>>>>>
>>>>> b) There might be more users hidden in today's or future recipes...
>>>>>
>>>>
>>>> Now we have pipeline:
>>>>   - do_fetch, do_unpack, do_build.
>>>>
>>>> I propose to extend this pipeline by one extra task:
>>>>   - do_fetch, do_unpack, do_install_build_deps, do_build.
>>>>
>>>> These are the core tasks that have default payload defined in calsses,
>>>> so you should not touch them in custom recipes.
>>>>
>>>> For better scale-ability, a separate class could be created:
>>>>
>>>> 8<--
>>>>
>>>> dpkg-build-deps.bbclass:
>>>>
>>>> do_install_build_deps() {
>>>>     ... call mk-build-deps
>>>> }
>>>>
>>>> addtask install_build_deps before do_build after do_unpack
>>>>
>>>> 8<--
>>>>
>>>> So if you want this functionality in your class, for example in
>>>> dpkg.bbclass, so just include it. I can't imagine if we will have so
>>>> many different classes to build something...
>>>
>>> I'm concern about all dpkg calls. Are we sure there are no others to
>>> just query the database eg. which cannot fail?
>>>
>>> Otherwise, this sounds good to me.
>>
>> Already coded this into a path, or should I look into it? We desperately
>> need this to restore CI with all its concurrent builds - once they run
>> truly concurrent.
> 
> If you mean whether I've done this or not, then no, I haven't. Could
> handle this in the next one-two-ays, at the moment there are several
> series pending for review/applying.

Yeah, as I said, I'm already looking into this in order to be able to
reproduce your CI setup. I will not model it via locked tasks, to
complex, but likely just via flock in build.sh.

Jan