[PATCH] buildchroot: Lock-protect apt/dpkg operations

[PATCH] buildchroot: Lock-protect apt/dpkg operations

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

When running multiple build targets in parallel, they will compete for
access on the package database while the related commands do not wait
but fail immediately.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---

Fixes parallel builds (CI setup) for me.

I just recalled that Chris was reporting the same issue a long time ago,
but his suggestion to lock down whole do_build was not that well
received...

 meta/recipes-devtools/buildchroot/files/build.sh | 32 +++++++++++++++---------
 1 file changed, 20 insertions(+), 12 deletions(-)

diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
index 023a6d6..048f76f 100644
--- a/meta/recipes-devtools/buildchroot/files/build.sh
+++ b/meta/recipes-devtools/buildchroot/files/build.sh
@@ -2,17 +2,9 @@
 #
 # This software is a part of ISAR.
 # Copyright (C) 2015-2017 ilbers GmbH
+# Copyright (c) 2018 Siemens AG
 
-# Make sure that we have latest isar-apt content.
-# Options meaning:
-#   Dir::Etc::sourcelist - specifies which source to be used
-#   Dir::Etc::sourceparts - disables looking for the other sources
-#   APT::Get::List-Cleanup - do not erase obsolete packages list for
-#                            upstream in '/var/lib/apt/lists'
-apt-get update \
-    -o Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \
-    -o Dir::Etc::sourceparts="-" \
-    -o APT::Get::List-Cleanup="0"
+set -e
 
 # Go to build directory
 cd $1
@@ -26,8 +18,24 @@ install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y
 # Allow unauthenticated feeds
 install_cmd="${install_cmd} --allow-unauthenticated"
 
-# Install all build deps
-mk-build-deps -t "${install_cmd}" -i -r debian/control
+(
+    # Lock-protected because apt and dpkg do not wait in case of contention
+    flock 42 || exit 1
+
+    # Make sure that we have latest isar-apt content.
+    # Options meaning:
+    #   Dir::Etc::sourcelist - specifies which source to be used
+    #   Dir::Etc::sourceparts - disables looking for the other sources
+    #   APT::Get::List-Cleanup - do not erase obsolete packages list for
+    #                            upstream in '/var/lib/apt/lists'
+    apt-get update \
+        -o Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \
+        -o Dir::Etc::sourceparts="-" \
+        -o APT::Get::List-Cleanup="0"
+
+    # Install all build deps
+    mk-build-deps -t "${install_cmd}" -i -r debian/control
+) 42>/dpkg.lock
 
 # If autotools files have been created, update their timestamp to
 # prevent them from being regenerated
-- 
2.13.6

Re: [PATCH] buildchroot: Lock-protect apt/dpkg operations

[Alexander Smirnov]

On 02/14/2018 10:42 PM, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> When running multiple build targets in parallel, they will compete for
> access on the package database while the related commands do not wait
> but fail immediately.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
> 
> Fixes parallel builds (CI setup) for me.
> 
> I just recalled that Chris was reporting the same issue a long time ago,
> but his suggestion to lock down whole do_build was not that well
> received...
> 
>   meta/recipes-devtools/buildchroot/files/build.sh | 32 +++++++++++++++---------
>   1 file changed, 20 insertions(+), 12 deletions(-)
> 
> diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
> index 023a6d6..048f76f 100644
> --- a/meta/recipes-devtools/buildchroot/files/build.sh
> +++ b/meta/recipes-devtools/buildchroot/files/build.sh
> @@ -2,17 +2,9 @@
>   #
>   # This software is a part of ISAR.
>   # Copyright (C) 2015-2017 ilbers GmbH
> +# Copyright (c) 2018 Siemens AG
>   
> -# Make sure that we have latest isar-apt content.
> -# Options meaning:
> -#   Dir::Etc::sourcelist - specifies which source to be used
> -#   Dir::Etc::sourceparts - disables looking for the other sources
> -#   APT::Get::List-Cleanup - do not erase obsolete packages list for
> -#                            upstream in '/var/lib/apt/lists'
> -apt-get update \
> -    -o Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \
> -    -o Dir::Etc::sourceparts="-" \
> -    -o APT::Get::List-Cleanup="0"
> +set -e
>   
>   # Go to build directory
>   cd $1
> @@ -26,8 +18,24 @@ install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y
>   # Allow unauthenticated feeds
>   install_cmd="${install_cmd} --allow-unauthenticated"
>   
> -# Install all build deps
> -mk-build-deps -t "${install_cmd}" -i -r debian/control
> +(
> +    # Lock-protected because apt and dpkg do not wait in case of contention
> +    flock 42 || exit 1
> +
> +    # Make sure that we have latest isar-apt content.
> +    # Options meaning:
> +    #   Dir::Etc::sourcelist - specifies which source to be used
> +    #   Dir::Etc::sourceparts - disables looking for the other sources
> +    #   APT::Get::List-Cleanup - do not erase obsolete packages list for
> +    #                            upstream in '/var/lib/apt/lists'
> +    apt-get update \
> +        -o Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \
> +        -o Dir::Etc::sourceparts="-" \
> +        -o APT::Get::List-Cleanup="0"
> +
> +    # Install all build deps
> +    mk-build-deps -t "${install_cmd}" -i -r debian/control
> +) 42>/dpkg.lock

BTW, what is the difference between the construction above and the 
construction you used in dpkg-base.bbclass:

8<--
sudo flock ${MOUNT_LOCKFILE} -c ' \
     if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then \
         mount --bind ${DEPLOY_DIR_APT}/${DISTRO} 
${BUILDCHROOT_DIR}/isar-apt; \
         mount -t devtmpfs -o mode=0755,nosuid devtmpfs 
${BUILDCHROOT_DIR}/dev; \
         mount -t proc none ${BUILDCHROOT_DIR}/proc; \
     fi'
8<--

If there is no difference, could we somehow unify usage of flock in Isar?

Alex

>   
>   # If autotools files have been created, update their timestamp to
>   # prevent them from being regenerated
> 

Re: [PATCH] buildchroot: Lock-protect apt/dpkg operations

[Jan Kiszka]

On 2018-02-15 09:16, Alexander Smirnov wrote:
> On 02/14/2018 10:42 PM, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> When running multiple build targets in parallel, they will compete for
>> access on the package database while the related commands do not wait
>> but fail immediately.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>
>> Fixes parallel builds (CI setup) for me.
>>
>> I just recalled that Chris was reporting the same issue a long time ago,
>> but his suggestion to lock down whole do_build was not that well
>> received...
>>
>>   meta/recipes-devtools/buildchroot/files/build.sh | 32
>> +++++++++++++++---------
>>   1 file changed, 20 insertions(+), 12 deletions(-)
>>
>> diff --git a/meta/recipes-devtools/buildchroot/files/build.sh
>> b/meta/recipes-devtools/buildchroot/files/build.sh
>> index 023a6d6..048f76f 100644
>> --- a/meta/recipes-devtools/buildchroot/files/build.sh
>> +++ b/meta/recipes-devtools/buildchroot/files/build.sh
>> @@ -2,17 +2,9 @@
>>   #
>>   # This software is a part of ISAR.
>>   # Copyright (C) 2015-2017 ilbers GmbH
>> +# Copyright (c) 2018 Siemens AG
>>   -# Make sure that we have latest isar-apt content.
>> -# Options meaning:
>> -#   Dir::Etc::sourcelist - specifies which source to be used
>> -#   Dir::Etc::sourceparts - disables looking for the other sources
>> -#   APT::Get::List-Cleanup - do not erase obsolete packages list for
>> -#                            upstream in '/var/lib/apt/lists'
>> -apt-get update \
>> -    -o Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \
>> -    -o Dir::Etc::sourceparts="-" \
>> -    -o APT::Get::List-Cleanup="0"
>> +set -e
>>     # Go to build directory
>>   cd $1
>> @@ -26,8 +18,24 @@ install_cmd="apt-get -o
>> Debug::pkgProblemResolver=yes --no-install-recommends -y
>>   # Allow unauthenticated feeds
>>   install_cmd="${install_cmd} --allow-unauthenticated"
>>   -# Install all build deps
>> -mk-build-deps -t "${install_cmd}" -i -r debian/control
>> +(
>> +    # Lock-protected because apt and dpkg do not wait in case of
>> contention
>> +    flock 42 || exit 1
>> +
>> +    # Make sure that we have latest isar-apt content.
>> +    # Options meaning:
>> +    #   Dir::Etc::sourcelist - specifies which source to be used
>> +    #   Dir::Etc::sourceparts - disables looking for the other sources
>> +    #   APT::Get::List-Cleanup - do not erase obsolete packages list for
>> +    #                            upstream in '/var/lib/apt/lists'
>> +    apt-get update \
>> +        -o
>> Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \
>> +        -o Dir::Etc::sourceparts="-" \
>> +        -o APT::Get::List-Cleanup="0"
>> +
>> +    # Install all build deps
>> +    mk-build-deps -t "${install_cmd}" -i -r debian/control
>> +) 42>/dpkg.lock
> 
> BTW, what is the difference between the construction above and the
> construction you used in dpkg-base.bbclass:
> 
> 8<--
> sudo flock ${MOUNT_LOCKFILE} -c ' \
>     if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then \
>         mount --bind ${DEPLOY_DIR_APT}/${DISTRO}
> ${BUILDCHROOT_DIR}/isar-apt; \
>         mount -t devtmpfs -o mode=0755,nosuid devtmpfs
> ${BUILDCHROOT_DIR}/dev; \
>         mount -t proc none ${BUILDCHROOT_DIR}/proc; \
>     fi'
> 8<--
> 
> If there is no difference, could we somehow unify usage of flock in Isar?
> 

See the comment in the other commit: The pattern used here is apparently
not supported in bitbake shell functions. This one here seems nicer,
though, as it does not require those line continuations.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH] buildchroot: Lock-protect apt/dpkg operations

[Alexander Smirnov]

On 02/14/2018 10:42 PM, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> When running multiple build targets in parallel, they will compete for
> access on the package database while the related commands do not wait
> but fail immediately.

Applied to next, thanks!

Alex