Build can sporadically fail due to dpkg contention in buildchroot

Build can sporadically fail due to dpkg contention in buildchroot

[Jan Kiszka]

Hi all,

I got this failure of example-hello:do_build already twice while doing
rebuild tests with my kernel series (ie. with more independent
buildchroot users):

DEBUG: Executing shell function do_build
Get:1 file:/isar-apt isar InRelease
Ign:1 file:/isar-apt isar InRelease
Get:2 file:/isar-apt isar Release [2,864 B]
Get:2 file:/isar-apt isar Release [2,864 B]
Get:3 file:/isar-apt isar Release.gpg
Ign:3 file:/isar-apt isar Release.gpg
Get:4 file:/isar-apt isar/main amd64 Packages [1,135 B]
Reading package lists...
W: The repository 'file:/isar-apt isar Release' is not signed.
hostname: No address associated with hostname
dh_testdir
dh_testroot
dh_prep
dh_testdir
dh_testroot
dh_install
dh_install: Compatibility levels before 9 are deprecated (level 7 in use)
dh_installdocs
dh_installdocs: Compatibility levels before 9 are deprecated (level 7 in use)
dh_installchangelogs
dh_compress
dh_fixperms
dh_installdeb
dh_installdeb: Compatibility levels before 9 are deprecated (level 7 in use)
dh_gencontrol
dh_md5sums
dh_builddeb
dpkg-deb: building package 'hello-build-deps' in '../hello-build-deps_0.2_all.deb'.

The package has been created.
Attention, the package has been created in the current directory,
not in ".." as indicated by the message above!
dpkg: error: dpkg status database is locked by another process
mk-build-deps: dpkg --unpack failed
[...]

So we have a concurrency problem when building over the same dpkg
database. Looks like we need to synchronize (lock-protect) the access to
it, which also means pulling out the dependency installation from the
regular build step. Is that feasible at all? Any alternatives (besides
retrying such builds...)?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Build can sporadically fail due to dpkg contention in buildchroot

[Alexander Smirnov]

Hi,

On 02/11/2018 06:17 PM, Jan Kiszka wrote:
> Hi all,
> 
> I got this failure of example-hello:do_build already twice while doing
> rebuild tests with my kernel series (ie. with more independent
> buildchroot users):
> 
> DEBUG: Executing shell function do_build
> Get:1 file:/isar-apt isar InRelease
> Ign:1 file:/isar-apt isar InRelease
> Get:2 file:/isar-apt isar Release [2,864 B]
> Get:2 file:/isar-apt isar Release [2,864 B]
> Get:3 file:/isar-apt isar Release.gpg
> Ign:3 file:/isar-apt isar Release.gpg
> Get:4 file:/isar-apt isar/main amd64 Packages [1,135 B]
> Reading package lists...
> W: The repository 'file:/isar-apt isar Release' is not signed.
> hostname: No address associated with hostname
> dh_testdir
> dh_testroot
> dh_prep
> dh_testdir
> dh_testroot
> dh_install
> dh_install: Compatibility levels before 9 are deprecated (level 7 in use)
> dh_installdocs
> dh_installdocs: Compatibility levels before 9 are deprecated (level 7 in use)
> dh_installchangelogs
> dh_compress
> dh_fixperms
> dh_installdeb
> dh_installdeb: Compatibility levels before 9 are deprecated (level 7 in use)
> dh_gencontrol
> dh_md5sums
> dh_builddeb
> dpkg-deb: building package 'hello-build-deps' in '../hello-build-deps_0.2_all.deb'.
> 

Good catch!

> The package has been created.
> Attention, the package has been created in the current directory,
> not in ".." as indicated by the message above!
> dpkg: error: dpkg status database is locked by another process
> mk-build-deps: dpkg --unpack failed
> [...]
> 
> So we have a concurrency problem when building over the same dpkg
> database. Looks like we need to synchronize (lock-protect) the access to
> it, which also means pulling out the dependency installation from the
> regular build step. Is that feasible at all? Any alternatives (besides
> retrying such builds...)?

In general we could do this easily:

1. Split the content of build.sh into two functions, for example:
  - install_build_deps
  - build_package

2. Spit the bitbake do_build() into two tasks:

do_install_build_deps() {
     ... build.sh install_build_deps ...
}

addtask install_build_deps before do_build after do_unpack

do_build_package() {
     ... build.sh build_package ...
}

3. Using bitbake synchronization primitives, protect the fist task from 
parallel execution.

If you are OK with this, I could do this tomorrow.

Alex

Re: Build can sporadically fail due to dpkg contention in buildchroot

[Jan Kiszka]

On 2018-02-11 17:55, Alexander Smirnov wrote:
> Hi,
> 
> On 02/11/2018 06:17 PM, Jan Kiszka wrote:
>> Hi all,
>>
>> I got this failure of example-hello:do_build already twice while doing
>> rebuild tests with my kernel series (ie. with more independent
>> buildchroot users):
>>
>> DEBUG: Executing shell function do_build
>> Get:1 file:/isar-apt isar InRelease
>> Ign:1 file:/isar-apt isar InRelease
>> Get:2 file:/isar-apt isar Release [2,864 B]
>> Get:2 file:/isar-apt isar Release [2,864 B]
>> Get:3 file:/isar-apt isar Release.gpg
>> Ign:3 file:/isar-apt isar Release.gpg
>> Get:4 file:/isar-apt isar/main amd64 Packages [1,135 B]
>> Reading package lists...
>> W: The repository 'file:/isar-apt isar Release' is not signed.
>> hostname: No address associated with hostname
>> dh_testdir
>> dh_testroot
>> dh_prep
>> dh_testdir
>> dh_testroot
>> dh_install
>> dh_install: Compatibility levels before 9 are deprecated (level 7 in use)
>> dh_installdocs
>> dh_installdocs: Compatibility levels before 9 are deprecated (level 7
>> in use)
>> dh_installchangelogs
>> dh_compress
>> dh_fixperms
>> dh_installdeb
>> dh_installdeb: Compatibility levels before 9 are deprecated (level 7
>> in use)
>> dh_gencontrol
>> dh_md5sums
>> dh_builddeb
>> dpkg-deb: building package 'hello-build-deps' in
>> '../hello-build-deps_0.2_all.deb'.
>>
> 
> Good catch!
> 
>> The package has been created.
>> Attention, the package has been created in the current directory,
>> not in ".." as indicated by the message above!
>> dpkg: error: dpkg status database is locked by another process
>> mk-build-deps: dpkg --unpack failed
>> [...]
>>
>> So we have a concurrency problem when building over the same dpkg
>> database. Looks like we need to synchronize (lock-protect) the access to
>> it, which also means pulling out the dependency installation from the
>> regular build step. Is that feasible at all? Any alternatives (besides
>> retrying such builds...)?
> 
> In general we could do this easily:
> 
> 1. Split the content of build.sh into two functions, for example:
>  - install_build_deps
>  - build_package
> 
> 2. Spit the bitbake do_build() into two tasks:
> 
> do_install_build_deps() {
>     ... build.sh install_build_deps ...
> }
> 
> addtask install_build_deps before do_build after do_unpack
> 
> do_build_package() {
>     ... build.sh build_package ...
> }
> 
> 3. Using bitbake synchronization primitives, protect the fist task from
> parallel execution.
> 
> If you are OK with this, I could do this tomorrow.

I'm still concerned how well this will scale:

a) We have additional users we already know of (linux-kernel.bbclass).
   We will need to provide them the same means.

b) There might be more users hidden in today's or future recipes...

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Build can sporadically fail due to dpkg contention in buildchroot

[Alexander Smirnov]

On 02/11/2018 09:18 PM, Jan Kiszka wrote:
> On 2018-02-11 17:55, Alexander Smirnov wrote:
>> Hi,
>>
>> On 02/11/2018 06:17 PM, Jan Kiszka wrote:
>>> Hi all,
>>>
>>> I got this failure of example-hello:do_build already twice while doing
>>> rebuild tests with my kernel series (ie. with more independent
>>> buildchroot users):
>>>
>>> DEBUG: Executing shell function do_build
>>> Get:1 file:/isar-apt isar InRelease
>>> Ign:1 file:/isar-apt isar InRelease
>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>> Get:3 file:/isar-apt isar Release.gpg
>>> Ign:3 file:/isar-apt isar Release.gpg
>>> Get:4 file:/isar-apt isar/main amd64 Packages [1,135 B]
>>> Reading package lists...
>>> W: The repository 'file:/isar-apt isar Release' is not signed.
>>> hostname: No address associated with hostname
>>> dh_testdir
>>> dh_testroot
>>> dh_prep
>>> dh_testdir
>>> dh_testroot
>>> dh_install
>>> dh_install: Compatibility levels before 9 are deprecated (level 7 in use)
>>> dh_installdocs
>>> dh_installdocs: Compatibility levels before 9 are deprecated (level 7
>>> in use)
>>> dh_installchangelogs
>>> dh_compress
>>> dh_fixperms
>>> dh_installdeb
>>> dh_installdeb: Compatibility levels before 9 are deprecated (level 7
>>> in use)
>>> dh_gencontrol
>>> dh_md5sums
>>> dh_builddeb
>>> dpkg-deb: building package 'hello-build-deps' in
>>> '../hello-build-deps_0.2_all.deb'.
>>>
>>
>> Good catch!
>>
>>> The package has been created.
>>> Attention, the package has been created in the current directory,
>>> not in ".." as indicated by the message above!
>>> dpkg: error: dpkg status database is locked by another process
>>> mk-build-deps: dpkg --unpack failed
>>> [...]
>>>
>>> So we have a concurrency problem when building over the same dpkg
>>> database. Looks like we need to synchronize (lock-protect) the access to
>>> it, which also means pulling out the dependency installation from the
>>> regular build step. Is that feasible at all? Any alternatives (besides
>>> retrying such builds...)?
>>
>> In general we could do this easily:
>>
>> 1. Split the content of build.sh into two functions, for example:
>>   - install_build_deps
>>   - build_package
>>
>> 2. Spit the bitbake do_build() into two tasks:
>>
>> do_install_build_deps() {
>>      ... build.sh install_build_deps ...
>> }
>>
>> addtask install_build_deps before do_build after do_unpack
>>
>> do_build_package() {
>>      ... build.sh build_package ...
>> }
>>
>> 3. Using bitbake synchronization primitives, protect the fist task from
>> parallel execution.
>>
>> If you are OK with this, I could do this tomorrow.
> 
> I'm still concerned how well this will scale:
> 
> a) We have additional users we already know of (linux-kernel.bbclass).
>     We will need to provide them the same means.
> 
> b) There might be more users hidden in today's or future recipes...
> 

Now we have pipeline:
  - do_fetch, do_unpack, do_build.

I propose to extend this pipeline by one extra task:
  - do_fetch, do_unpack, do_install_build_deps, do_build.

These are the core tasks that have default payload defined in calsses, 
so you should not touch them in custom recipes.

For better scale-ability, a separate class could be created:

8<--

dpkg-build-deps.bbclass:

do_install_build_deps() {
    ... call mk-build-deps
}

addtask install_build_deps before do_build after do_unpack

8<--

So if you want this functionality in your class, for example in 
dpkg.bbclass, so just include it. I can't imagine if we will have so 
many different classes to build something...

Alex

Re: Build can sporadically fail due to dpkg contention in buildchroot

[Jan Kiszka]

On 2018-02-11 19:44, Alexander Smirnov wrote:
> 
> 
> On 02/11/2018 09:18 PM, Jan Kiszka wrote:
>> On 2018-02-11 17:55, Alexander Smirnov wrote:
>>> Hi,
>>>
>>> On 02/11/2018 06:17 PM, Jan Kiszka wrote:
>>>> Hi all,
>>>>
>>>> I got this failure of example-hello:do_build already twice while doing
>>>> rebuild tests with my kernel series (ie. with more independent
>>>> buildchroot users):
>>>>
>>>> DEBUG: Executing shell function do_build
>>>> Get:1 file:/isar-apt isar InRelease
>>>> Ign:1 file:/isar-apt isar InRelease
>>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>>> Get:3 file:/isar-apt isar Release.gpg
>>>> Ign:3 file:/isar-apt isar Release.gpg
>>>> Get:4 file:/isar-apt isar/main amd64 Packages [1,135 B]
>>>> Reading package lists...
>>>> W: The repository 'file:/isar-apt isar Release' is not signed.
>>>> hostname: No address associated with hostname
>>>> dh_testdir
>>>> dh_testroot
>>>> dh_prep
>>>> dh_testdir
>>>> dh_testroot
>>>> dh_install
>>>> dh_install: Compatibility levels before 9 are deprecated (level 7 in
>>>> use)
>>>> dh_installdocs
>>>> dh_installdocs: Compatibility levels before 9 are deprecated (level 7
>>>> in use)
>>>> dh_installchangelogs
>>>> dh_compress
>>>> dh_fixperms
>>>> dh_installdeb
>>>> dh_installdeb: Compatibility levels before 9 are deprecated (level 7
>>>> in use)
>>>> dh_gencontrol
>>>> dh_md5sums
>>>> dh_builddeb
>>>> dpkg-deb: building package 'hello-build-deps' in
>>>> '../hello-build-deps_0.2_all.deb'.
>>>>
>>>
>>> Good catch!
>>>
>>>> The package has been created.
>>>> Attention, the package has been created in the current directory,
>>>> not in ".." as indicated by the message above!
>>>> dpkg: error: dpkg status database is locked by another process
>>>> mk-build-deps: dpkg --unpack failed
>>>> [...]
>>>>
>>>> So we have a concurrency problem when building over the same dpkg
>>>> database. Looks like we need to synchronize (lock-protect) the
>>>> access to
>>>> it, which also means pulling out the dependency installation from the
>>>> regular build step. Is that feasible at all? Any alternatives (besides
>>>> retrying such builds...)?
>>>
>>> In general we could do this easily:
>>>
>>> 1. Split the content of build.sh into two functions, for example:
>>>   - install_build_deps
>>>   - build_package
>>>
>>> 2. Spit the bitbake do_build() into two tasks:
>>>
>>> do_install_build_deps() {
>>>      ... build.sh install_build_deps ...
>>> }
>>>
>>> addtask install_build_deps before do_build after do_unpack
>>>
>>> do_build_package() {
>>>      ... build.sh build_package ...
>>> }
>>>
>>> 3. Using bitbake synchronization primitives, protect the fist task from
>>> parallel execution.
>>>
>>> If you are OK with this, I could do this tomorrow.
>>
>> I'm still concerned how well this will scale:
>>
>> a) We have additional users we already know of (linux-kernel.bbclass).
>>     We will need to provide them the same means.
>>
>> b) There might be more users hidden in today's or future recipes...
>>
> 
> Now we have pipeline:
>  - do_fetch, do_unpack, do_build.
> 
> I propose to extend this pipeline by one extra task:
>  - do_fetch, do_unpack, do_install_build_deps, do_build.
> 
> These are the core tasks that have default payload defined in calsses,
> so you should not touch them in custom recipes.
> 
> For better scale-ability, a separate class could be created:
> 
> 8<--
> 
> dpkg-build-deps.bbclass:
> 
> do_install_build_deps() {
>    ... call mk-build-deps
> }
> 
> addtask install_build_deps before do_build after do_unpack
> 
> 8<--
> 
> So if you want this functionality in your class, for example in
> dpkg.bbclass, so just include it. I can't imagine if we will have so
> many different classes to build something...

I'm concern about all dpkg calls. Are we sure there are no others to
just query the database eg. which cannot fail?

Otherwise, this sounds good to me.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Build can sporadically fail due to dpkg contention in buildchroot

[Jan Kiszka]

On 2018-02-12 08:42, [ext] Jan Kiszka wrote:
> On 2018-02-11 19:44, Alexander Smirnov wrote:
>>
>>
>> On 02/11/2018 09:18 PM, Jan Kiszka wrote:
>>> On 2018-02-11 17:55, Alexander Smirnov wrote:
>>>> Hi,
>>>>
>>>> On 02/11/2018 06:17 PM, Jan Kiszka wrote:
>>>>> Hi all,
>>>>>
>>>>> I got this failure of example-hello:do_build already twice while doing
>>>>> rebuild tests with my kernel series (ie. with more independent
>>>>> buildchroot users):
>>>>>
>>>>> DEBUG: Executing shell function do_build
>>>>> Get:1 file:/isar-apt isar InRelease
>>>>> Ign:1 file:/isar-apt isar InRelease
>>>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>>>> Get:3 file:/isar-apt isar Release.gpg
>>>>> Ign:3 file:/isar-apt isar Release.gpg
>>>>> Get:4 file:/isar-apt isar/main amd64 Packages [1,135 B]
>>>>> Reading package lists...
>>>>> W: The repository 'file:/isar-apt isar Release' is not signed.
>>>>> hostname: No address associated with hostname
>>>>> dh_testdir
>>>>> dh_testroot
>>>>> dh_prep
>>>>> dh_testdir
>>>>> dh_testroot
>>>>> dh_install
>>>>> dh_install: Compatibility levels before 9 are deprecated (level 7 in
>>>>> use)
>>>>> dh_installdocs
>>>>> dh_installdocs: Compatibility levels before 9 are deprecated (level 7
>>>>> in use)
>>>>> dh_installchangelogs
>>>>> dh_compress
>>>>> dh_fixperms
>>>>> dh_installdeb
>>>>> dh_installdeb: Compatibility levels before 9 are deprecated (level 7
>>>>> in use)
>>>>> dh_gencontrol
>>>>> dh_md5sums
>>>>> dh_builddeb
>>>>> dpkg-deb: building package 'hello-build-deps' in
>>>>> '../hello-build-deps_0.2_all.deb'.
>>>>>
>>>>
>>>> Good catch!
>>>>
>>>>> The package has been created.
>>>>> Attention, the package has been created in the current directory,
>>>>> not in ".." as indicated by the message above!
>>>>> dpkg: error: dpkg status database is locked by another process
>>>>> mk-build-deps: dpkg --unpack failed
>>>>> [...]
>>>>>
>>>>> So we have a concurrency problem when building over the same dpkg
>>>>> database. Looks like we need to synchronize (lock-protect) the
>>>>> access to
>>>>> it, which also means pulling out the dependency installation from the
>>>>> regular build step. Is that feasible at all? Any alternatives (besides
>>>>> retrying such builds...)?
>>>>
>>>> In general we could do this easily:
>>>>
>>>> 1. Split the content of build.sh into two functions, for example:
>>>>   - install_build_deps
>>>>   - build_package
>>>>
>>>> 2. Spit the bitbake do_build() into two tasks:
>>>>
>>>> do_install_build_deps() {
>>>>      ... build.sh install_build_deps ...
>>>> }
>>>>
>>>> addtask install_build_deps before do_build after do_unpack
>>>>
>>>> do_build_package() {
>>>>      ... build.sh build_package ...
>>>> }
>>>>
>>>> 3. Using bitbake synchronization primitives, protect the fist task from
>>>> parallel execution.
>>>>
>>>> If you are OK with this, I could do this tomorrow.
>>>
>>> I'm still concerned how well this will scale:
>>>
>>> a) We have additional users we already know of (linux-kernel.bbclass).
>>>     We will need to provide them the same means.
>>>
>>> b) There might be more users hidden in today's or future recipes...
>>>
>>
>> Now we have pipeline:
>>  - do_fetch, do_unpack, do_build.
>>
>> I propose to extend this pipeline by one extra task:
>>  - do_fetch, do_unpack, do_install_build_deps, do_build.
>>
>> These are the core tasks that have default payload defined in calsses,
>> so you should not touch them in custom recipes.
>>
>> For better scale-ability, a separate class could be created:
>>
>> 8<--
>>
>> dpkg-build-deps.bbclass:
>>
>> do_install_build_deps() {
>>    ... call mk-build-deps
>> }
>>
>> addtask install_build_deps before do_build after do_unpack
>>
>> 8<--
>>
>> So if you want this functionality in your class, for example in
>> dpkg.bbclass, so just include it. I can't imagine if we will have so
>> many different classes to build something...
> 
> I'm concern about all dpkg calls. Are we sure there are no others to
> just query the database eg. which cannot fail?
> 
> Otherwise, this sounds good to me.

Already coded this into a path, or should I look into it? We desperately
need this to restore CI with all its concurrent builds - once they run
truly concurrent.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Build can sporadically fail due to dpkg contention in buildchroot

[Alexander Smirnov]

On 02/14/2018 06:02 PM, Jan Kiszka wrote:
> On 2018-02-12 08:42, [ext] Jan Kiszka wrote:
>> On 2018-02-11 19:44, Alexander Smirnov wrote:
>>>
>>>
>>> On 02/11/2018 09:18 PM, Jan Kiszka wrote:
>>>> On 2018-02-11 17:55, Alexander Smirnov wrote:
>>>>> Hi,
>>>>>
>>>>> On 02/11/2018 06:17 PM, Jan Kiszka wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I got this failure of example-hello:do_build already twice while doing
>>>>>> rebuild tests with my kernel series (ie. with more independent
>>>>>> buildchroot users):
>>>>>>
>>>>>> DEBUG: Executing shell function do_build
>>>>>> Get:1 file:/isar-apt isar InRelease
>>>>>> Ign:1 file:/isar-apt isar InRelease
>>>>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>>>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>>>>> Get:3 file:/isar-apt isar Release.gpg
>>>>>> Ign:3 file:/isar-apt isar Release.gpg
>>>>>> Get:4 file:/isar-apt isar/main amd64 Packages [1,135 B]
>>>>>> Reading package lists...
>>>>>> W: The repository 'file:/isar-apt isar Release' is not signed.
>>>>>> hostname: No address associated with hostname
>>>>>> dh_testdir
>>>>>> dh_testroot
>>>>>> dh_prep
>>>>>> dh_testdir
>>>>>> dh_testroot
>>>>>> dh_install
>>>>>> dh_install: Compatibility levels before 9 are deprecated (level 7 in
>>>>>> use)
>>>>>> dh_installdocs
>>>>>> dh_installdocs: Compatibility levels before 9 are deprecated (level 7
>>>>>> in use)
>>>>>> dh_installchangelogs
>>>>>> dh_compress
>>>>>> dh_fixperms
>>>>>> dh_installdeb
>>>>>> dh_installdeb: Compatibility levels before 9 are deprecated (level 7
>>>>>> in use)
>>>>>> dh_gencontrol
>>>>>> dh_md5sums
>>>>>> dh_builddeb
>>>>>> dpkg-deb: building package 'hello-build-deps' in
>>>>>> '../hello-build-deps_0.2_all.deb'.
>>>>>>
>>>>>
>>>>> Good catch!
>>>>>
>>>>>> The package has been created.
>>>>>> Attention, the package has been created in the current directory,
>>>>>> not in ".." as indicated by the message above!
>>>>>> dpkg: error: dpkg status database is locked by another process
>>>>>> mk-build-deps: dpkg --unpack failed
>>>>>> [...]
>>>>>>
>>>>>> So we have a concurrency problem when building over the same dpkg
>>>>>> database. Looks like we need to synchronize (lock-protect) the
>>>>>> access to
>>>>>> it, which also means pulling out the dependency installation from the
>>>>>> regular build step. Is that feasible at all? Any alternatives (besides
>>>>>> retrying such builds...)?
>>>>>
>>>>> In general we could do this easily:
>>>>>
>>>>> 1. Split the content of build.sh into two functions, for example:
>>>>>    - install_build_deps
>>>>>    - build_package
>>>>>
>>>>> 2. Spit the bitbake do_build() into two tasks:
>>>>>
>>>>> do_install_build_deps() {
>>>>>       ... build.sh install_build_deps ...
>>>>> }
>>>>>
>>>>> addtask install_build_deps before do_build after do_unpack
>>>>>
>>>>> do_build_package() {
>>>>>       ... build.sh build_package ...
>>>>> }
>>>>>
>>>>> 3. Using bitbake synchronization primitives, protect the fist task from
>>>>> parallel execution.
>>>>>
>>>>> If you are OK with this, I could do this tomorrow.
>>>>
>>>> I'm still concerned how well this will scale:
>>>>
>>>> a) We have additional users we already know of (linux-kernel.bbclass).
>>>>      We will need to provide them the same means.
>>>>
>>>> b) There might be more users hidden in today's or future recipes...
>>>>
>>>
>>> Now we have pipeline:
>>>   - do_fetch, do_unpack, do_build.
>>>
>>> I propose to extend this pipeline by one extra task:
>>>   - do_fetch, do_unpack, do_install_build_deps, do_build.
>>>
>>> These are the core tasks that have default payload defined in calsses,
>>> so you should not touch them in custom recipes.
>>>
>>> For better scale-ability, a separate class could be created:
>>>
>>> 8<--
>>>
>>> dpkg-build-deps.bbclass:
>>>
>>> do_install_build_deps() {
>>>     ... call mk-build-deps
>>> }
>>>
>>> addtask install_build_deps before do_build after do_unpack
>>>
>>> 8<--
>>>
>>> So if you want this functionality in your class, for example in
>>> dpkg.bbclass, so just include it. I can't imagine if we will have so
>>> many different classes to build something...
>>
>> I'm concern about all dpkg calls. Are we sure there are no others to
>> just query the database eg. which cannot fail?
>>
>> Otherwise, this sounds good to me.
> 
> Already coded this into a path, or should I look into it? We desperately
> need this to restore CI with all its concurrent builds - once they run
> truly concurrent.

If you mean whether I've done this or not, then no, I haven't. Could 
handle this in the next one-two-ays, at the moment there are several 
series pending for review/applying.

Alex

Re: Build can sporadically fail due to dpkg contention in buildchroot

[Jan Kiszka]

On 2018-02-14 18:57, Alexander Smirnov wrote:
> On 02/14/2018 06:02 PM, Jan Kiszka wrote:
>> On 2018-02-12 08:42, [ext] Jan Kiszka wrote:
>>> On 2018-02-11 19:44, Alexander Smirnov wrote:
>>>>
>>>>
>>>> On 02/11/2018 09:18 PM, Jan Kiszka wrote:
>>>>> On 2018-02-11 17:55, Alexander Smirnov wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On 02/11/2018 06:17 PM, Jan Kiszka wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I got this failure of example-hello:do_build already twice while
>>>>>>> doing
>>>>>>> rebuild tests with my kernel series (ie. with more independent
>>>>>>> buildchroot users):
>>>>>>>
>>>>>>> DEBUG: Executing shell function do_build
>>>>>>> Get:1 file:/isar-apt isar InRelease
>>>>>>> Ign:1 file:/isar-apt isar InRelease
>>>>>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>>>>>> Get:2 file:/isar-apt isar Release [2,864 B]
>>>>>>> Get:3 file:/isar-apt isar Release.gpg
>>>>>>> Ign:3 file:/isar-apt isar Release.gpg
>>>>>>> Get:4 file:/isar-apt isar/main amd64 Packages [1,135 B]
>>>>>>> Reading package lists...
>>>>>>> W: The repository 'file:/isar-apt isar Release' is not signed.
>>>>>>> hostname: No address associated with hostname
>>>>>>> dh_testdir
>>>>>>> dh_testroot
>>>>>>> dh_prep
>>>>>>> dh_testdir
>>>>>>> dh_testroot
>>>>>>> dh_install
>>>>>>> dh_install: Compatibility levels before 9 are deprecated (level 7 in
>>>>>>> use)
>>>>>>> dh_installdocs
>>>>>>> dh_installdocs: Compatibility levels before 9 are deprecated
>>>>>>> (level 7
>>>>>>> in use)
>>>>>>> dh_installchangelogs
>>>>>>> dh_compress
>>>>>>> dh_fixperms
>>>>>>> dh_installdeb
>>>>>>> dh_installdeb: Compatibility levels before 9 are deprecated (level 7
>>>>>>> in use)
>>>>>>> dh_gencontrol
>>>>>>> dh_md5sums
>>>>>>> dh_builddeb
>>>>>>> dpkg-deb: building package 'hello-build-deps' in
>>>>>>> '../hello-build-deps_0.2_all.deb'.
>>>>>>>
>>>>>>
>>>>>> Good catch!
>>>>>>
>>>>>>> The package has been created.
>>>>>>> Attention, the package has been created in the current directory,
>>>>>>> not in ".." as indicated by the message above!
>>>>>>> dpkg: error: dpkg status database is locked by another process
>>>>>>> mk-build-deps: dpkg --unpack failed
>>>>>>> [...]
>>>>>>>
>>>>>>> So we have a concurrency problem when building over the same dpkg
>>>>>>> database. Looks like we need to synchronize (lock-protect) the
>>>>>>> access to
>>>>>>> it, which also means pulling out the dependency installation from
>>>>>>> the
>>>>>>> regular build step. Is that feasible at all? Any alternatives
>>>>>>> (besides
>>>>>>> retrying such builds...)?
>>>>>>
>>>>>> In general we could do this easily:
>>>>>>
>>>>>> 1. Split the content of build.sh into two functions, for example:
>>>>>>    - install_build_deps
>>>>>>    - build_package
>>>>>>
>>>>>> 2. Spit the bitbake do_build() into two tasks:
>>>>>>
>>>>>> do_install_build_deps() {
>>>>>>       ... build.sh install_build_deps ...
>>>>>> }
>>>>>>
>>>>>> addtask install_build_deps before do_build after do_unpack
>>>>>>
>>>>>> do_build_package() {
>>>>>>       ... build.sh build_package ...
>>>>>> }
>>>>>>
>>>>>> 3. Using bitbake synchronization primitives, protect the fist task
>>>>>> from
>>>>>> parallel execution.
>>>>>>
>>>>>> If you are OK with this, I could do this tomorrow.
>>>>>
>>>>> I'm still concerned how well this will scale:
>>>>>
>>>>> a) We have additional users we already know of (linux-kernel.bbclass).
>>>>>      We will need to provide them the same means.
>>>>>
>>>>> b) There might be more users hidden in today's or future recipes...
>>>>>
>>>>
>>>> Now we have pipeline:
>>>>   - do_fetch, do_unpack, do_build.
>>>>
>>>> I propose to extend this pipeline by one extra task:
>>>>   - do_fetch, do_unpack, do_install_build_deps, do_build.
>>>>
>>>> These are the core tasks that have default payload defined in calsses,
>>>> so you should not touch them in custom recipes.
>>>>
>>>> For better scale-ability, a separate class could be created:
>>>>
>>>> 8<--
>>>>
>>>> dpkg-build-deps.bbclass:
>>>>
>>>> do_install_build_deps() {
>>>>     ... call mk-build-deps
>>>> }
>>>>
>>>> addtask install_build_deps before do_build after do_unpack
>>>>
>>>> 8<--
>>>>
>>>> So if you want this functionality in your class, for example in
>>>> dpkg.bbclass, so just include it. I can't imagine if we will have so
>>>> many different classes to build something...
>>>
>>> I'm concern about all dpkg calls. Are we sure there are no others to
>>> just query the database eg. which cannot fail?
>>>
>>> Otherwise, this sounds good to me.
>>
>> Already coded this into a path, or should I look into it? We desperately
>> need this to restore CI with all its concurrent builds - once they run
>> truly concurrent.
> 
> If you mean whether I've done this or not, then no, I haven't. Could
> handle this in the next one-two-ays, at the moment there are several
> series pending for review/applying.

Yeah, as I said, I'm already looking into this in order to be able to
reproduce your CI setup. I will not model it via locked tasks, to
complex, but likely just via flock in build.sh.

Jan