Re: [PATCH] buildchroot: Lock-protect apt/dpkg operations

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2018-02-15 09:16, Alexander Smirnov wrote:
> On 02/14/2018 10:42 PM, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> When running multiple build targets in parallel, they will compete for
>> access on the package database while the related commands do not wait
>> but fail immediately.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>
>> Fixes parallel builds (CI setup) for me.
>>
>> I just recalled that Chris was reporting the same issue a long time ago,
>> but his suggestion to lock down whole do_build was not that well
>> received...
>>
>>   meta/recipes-devtools/buildchroot/files/build.sh | 32
>> +++++++++++++++---------
>>   1 file changed, 20 insertions(+), 12 deletions(-)
>>
>> diff --git a/meta/recipes-devtools/buildchroot/files/build.sh
>> b/meta/recipes-devtools/buildchroot/files/build.sh
>> index 023a6d6..048f76f 100644
>> --- a/meta/recipes-devtools/buildchroot/files/build.sh
>> +++ b/meta/recipes-devtools/buildchroot/files/build.sh
>> @@ -2,17 +2,9 @@
>>   #
>>   # This software is a part of ISAR.
>>   # Copyright (C) 2015-2017 ilbers GmbH
>> +# Copyright (c) 2018 Siemens AG
>>   -# Make sure that we have latest isar-apt content.
>> -# Options meaning:
>> -#   Dir::Etc::sourcelist - specifies which source to be used
>> -#   Dir::Etc::sourceparts - disables looking for the other sources
>> -#   APT::Get::List-Cleanup - do not erase obsolete packages list for
>> -#                            upstream in '/var/lib/apt/lists'
>> -apt-get update \
>> -    -o Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \
>> -    -o Dir::Etc::sourceparts="-" \
>> -    -o APT::Get::List-Cleanup="0"
>> +set -e
>>     # Go to build directory
>>   cd $1
>> @@ -26,8 +18,24 @@ install_cmd="apt-get -o
>> Debug::pkgProblemResolver=yes --no-install-recommends -y
>>   # Allow unauthenticated feeds
>>   install_cmd="${install_cmd} --allow-unauthenticated"
>>   -# Install all build deps
>> -mk-build-deps -t "${install_cmd}" -i -r debian/control
>> +(
>> +    # Lock-protected because apt and dpkg do not wait in case of
>> contention
>> +    flock 42 || exit 1
>> +
>> +    # Make sure that we have latest isar-apt content.
>> +    # Options meaning:
>> +    #   Dir::Etc::sourcelist - specifies which source to be used
>> +    #   Dir::Etc::sourceparts - disables looking for the other sources
>> +    #   APT::Get::List-Cleanup - do not erase obsolete packages list for
>> +    #                            upstream in '/var/lib/apt/lists'
>> +    apt-get update \
>> +        -o
>> Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \
>> +        -o Dir::Etc::sourceparts="-" \
>> +        -o APT::Get::List-Cleanup="0"
>> +
>> +    # Install all build deps
>> +    mk-build-deps -t "${install_cmd}" -i -r debian/control
>> +) 42>/dpkg.lock
> 
> BTW, what is the difference between the construction above and the
> construction you used in dpkg-base.bbclass:
> 
> 8<--
> sudo flock ${MOUNT_LOCKFILE} -c ' \
>     if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then \
>         mount --bind ${DEPLOY_DIR_APT}/${DISTRO}
> ${BUILDCHROOT_DIR}/isar-apt; \
>         mount -t devtmpfs -o mode=0755,nosuid devtmpfs
> ${BUILDCHROOT_DIR}/dev; \
>         mount -t proc none ${BUILDCHROOT_DIR}/proc; \
>     fi'
> 8<--
> 
> If there is no difference, could we somehow unify usage of flock in Isar?
> 

See the comment in the other commit: The pattern used here is apparently
not supported in bitbake shell functions. This one here seems nicer,
though, as it does not require those line continuations.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux