Re: Build can sporadically fail due to dpkg contention in buildchroot

[Alexander Smirnov <asmirnov@ilbers.de>]

Hi,

On 02/11/2018 06:17 PM, Jan Kiszka wrote:
> Hi all,
> 
> I got this failure of example-hello:do_build already twice while doing
> rebuild tests with my kernel series (ie. with more independent
> buildchroot users):
> 
> DEBUG: Executing shell function do_build
> Get:1 file:/isar-apt isar InRelease
> Ign:1 file:/isar-apt isar InRelease
> Get:2 file:/isar-apt isar Release [2,864 B]
> Get:2 file:/isar-apt isar Release [2,864 B]
> Get:3 file:/isar-apt isar Release.gpg
> Ign:3 file:/isar-apt isar Release.gpg
> Get:4 file:/isar-apt isar/main amd64 Packages [1,135 B]
> Reading package lists...
> W: The repository 'file:/isar-apt isar Release' is not signed.
> hostname: No address associated with hostname
> dh_testdir
> dh_testroot
> dh_prep
> dh_testdir
> dh_testroot
> dh_install
> dh_install: Compatibility levels before 9 are deprecated (level 7 in use)
> dh_installdocs
> dh_installdocs: Compatibility levels before 9 are deprecated (level 7 in use)
> dh_installchangelogs
> dh_compress
> dh_fixperms
> dh_installdeb
> dh_installdeb: Compatibility levels before 9 are deprecated (level 7 in use)
> dh_gencontrol
> dh_md5sums
> dh_builddeb
> dpkg-deb: building package 'hello-build-deps' in '../hello-build-deps_0.2_all.deb'.
> 

Good catch!

> The package has been created.
> Attention, the package has been created in the current directory,
> not in ".." as indicated by the message above!
> dpkg: error: dpkg status database is locked by another process
> mk-build-deps: dpkg --unpack failed
> [...]
> 
> So we have a concurrency problem when building over the same dpkg
> database. Looks like we need to synchronize (lock-protect) the access to
> it, which also means pulling out the dependency installation from the
> regular build step. Is that feasible at all? Any alternatives (besides
> retrying such builds...)?

In general we could do this easily:

1. Split the content of build.sh into two functions, for example:
  - install_build_deps
  - build_package

2. Spit the bitbake do_build() into two tasks:

do_install_build_deps() {
     ... build.sh install_build_deps ...
}

addtask install_build_deps before do_build after do_unpack

do_build_package() {
     ... build.sh build_package ...
}

3. Using bitbake synchronization primitives, protect the fist task from 
parallel execution.

If you are OK with this, I could do this tomorrow.

Alex