[PATCH] dpkg.bbclass: add lockfile for do_build

[PATCH] dpkg.bbclass: add lockfile for do_build

[Christopher Larson]

All recipes using this class will run build.sh in the same chroot, and
build.sh runs apt-get to install deps, so the build can fail due to dpkg
being unable to acquire its lock. Add a bitbake lockfile to ensure that
only one do_build task for recipes inheriting this class runs at a time.

Signed-off-by: Christopher Larson <kergoth@gmail.com>
---
 meta/classes/dpkg.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index c10f5ba..3966a71 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -7,3 +7,4 @@ inherit dpkg-base
 dpkg_runbuild() {
     sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
 }
+do_build[lockfiles] += "${TMPDIR}/dpkg.lock"
-- 
2.11.0

Re: [PATCH] dpkg.bbclass: add lockfile for do_build

[Alexander Smirnov]

Hello Christopher,

thank you for this catch!

On 01/10/2018 07:04 PM, Christopher Larson wrote:
> All recipes using this class will run build.sh in the same chroot, and
> build.sh runs apt-get to install deps, so the build can fail due to dpkg
> being unable to acquire its lock. Add a bitbake lockfile to ensure that
> only one do_build task for recipes inheriting this class runs at a time.
> 

Serialization of do_build tasks leads to whole Isar serialization, what 
is significant issue if you have lots of packages to build. So I think 
the "build" and "deps installation" should be split in separate tasks 
and only the second one should be protected by locks.

Alex

> Signed-off-by: Christopher Larson <kergoth@gmail.com>
> ---
>   meta/classes/dpkg.bbclass | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index c10f5ba..3966a71 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -7,3 +7,4 @@ inherit dpkg-base
>   dpkg_runbuild() {
>       sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
>   }
> +do_build[lockfiles] += "${TMPDIR}/dpkg.lock"
>

Re: [PATCH] dpkg.bbclass: add lockfile for do_build

[Henning Schild]

Am Thu, 11 Jan 2018 13:31:45 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Hello Christopher,
> 
> thank you for this catch!
> 
> On 01/10/2018 07:04 PM, Christopher Larson wrote:
> > All recipes using this class will run build.sh in the same chroot,
> > and build.sh runs apt-get to install deps, so the build can fail
> > due to dpkg being unable to acquire its lock. Add a bitbake
> > lockfile to ensure that only one do_build task for recipes
> > inheriting this class runs at a time. 
> 
> Serialization of do_build tasks leads to whole Isar serialization,
> what is significant issue if you have lots of packages to build. So I
> think the "build" and "deps installation" should be split in separate
> tasks and only the second one should be protected by locks.

The fact that Isar uses multiconfig to build many targets from the same
tree is not intuitive, we discussed this before. So i am not surprised
new contributors get that "wrong". Alex i suggest you split the task
and adopt Christophers changes.

Christopher, have a look at scripts/ci_build.sh and you will see that
Isar is capable of building multiple images at once in the same tree.
If you are building 10 images you will have 10 buildchroots that have
to be serialized seperately. A feature that probably nobody but the
CI-bot will ever use ;).

Henning

> Alex
> 
> > Signed-off-by: Christopher Larson <kergoth@gmail.com>
> > ---
> >   meta/classes/dpkg.bbclass | 1 +
> >   1 file changed, 1 insertion(+)
> > 
> > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> > index c10f5ba..3966a71 100644
> > --- a/meta/classes/dpkg.bbclass
> > +++ b/meta/classes/dpkg.bbclass
> > @@ -7,3 +7,4 @@ inherit dpkg-base
> >   dpkg_runbuild() {
> >       sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
> >   }
> > +do_build[lockfiles] += "${TMPDIR}/dpkg.lock"
> >  
> 

Re: [PATCH] dpkg.bbclass: add lockfile for do_build

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1773 bytes --]

On Thu, Jan 11, 2018 at 5:14 AM, Henning Schild <henning.schild@siemens.com>
wrote:

> Am Thu, 11 Jan 2018 13:31:45 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>
> > Hello Christopher,
> >
> > thank you for this catch!
> >
> > On 01/10/2018 07:04 PM, Christopher Larson wrote:
> > > All recipes using this class will run build.sh in the same chroot,
> > > and build.sh runs apt-get to install deps, so the build can fail
> > > due to dpkg being unable to acquire its lock. Add a bitbake
> > > lockfile to ensure that only one do_build task for recipes
> > > inheriting this class runs at a time.
> >
> > Serialization of do_build tasks leads to whole Isar serialization,
> > what is significant issue if you have lots of packages to build. So I
> > think the "build" and "deps installation" should be split in separate
> > tasks and only the second one should be protected by locks.
>
> The fact that Isar uses multiconfig to build many targets from the same
> tree is not intuitive, we discussed this before. So i am not surprised
> new contributors get that "wrong". Alex i suggest you split the task
> and adopt Christophers changes.
>
> Christopher, have a look at scripts/ci_build.sh and you will see that
> Isar is capable of building multiple images at once in the same tree.
> If you are building 10 images you will have 10 buildchroots that have
> to be serialized seperately. A feature that probably nobody but the
> CI-bot will ever use ;).


Fair enough, thanks, I’ll re-test with 1) a lockfile in the buildchroot or
2) having build.sh wait on a lock itself, if possible.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2434 bytes --]

Re: [PATCH] dpkg.bbclass: add lockfile for do_build

[Henning Schild]

Am Thu, 11 Jan 2018 08:20:54 -0700
schrieb Christopher Larson <kergoth@gmail.com>:

> On Thu, Jan 11, 2018 at 5:14 AM, Henning Schild
> <henning.schild@siemens.com> wrote:
> 
> > Am Thu, 11 Jan 2018 13:31:45 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >  
> > > Hello Christopher,
> > >
> > > thank you for this catch!
> > >
> > > On 01/10/2018 07:04 PM, Christopher Larson wrote:  
> > > > All recipes using this class will run build.sh in the same
> > > > chroot, and build.sh runs apt-get to install deps, so the build
> > > > can fail due to dpkg being unable to acquire its lock. Add a
> > > > bitbake lockfile to ensure that only one do_build task for
> > > > recipes inheriting this class runs at a time.  
> > >
> > > Serialization of do_build tasks leads to whole Isar serialization,
> > > what is significant issue if you have lots of packages to build.
> > > So I think the "build" and "deps installation" should be split in
> > > separate tasks and only the second one should be protected by
> > > locks.  
> >
> > The fact that Isar uses multiconfig to build many targets from the
> > same tree is not intuitive, we discussed this before. So i am not
> > surprised new contributors get that "wrong". Alex i suggest you
> > split the task and adopt Christophers changes.
> >
> > Christopher, have a look at scripts/ci_build.sh and you will see
> > that Isar is capable of building multiple images at once in the
> > same tree. If you are building 10 images you will have 10
> > buildchroots that have to be serialized seperately. A feature that
> > probably nobody but the CI-bot will ever use ;).  
> 
> 
> Fair enough, thanks, I’ll re-test with 1) a lockfile in the
> buildchroot or 2) having build.sh wait on a lock itself, if possible.

I just read your signature, and now i guess you can handle the
complexity of multiconf without Alex helping you out. Watch out for
changes to that code in current next.

Henning