Re: [PATCH] buildchroot: Lock-protect apt/dpkg operations

[Alexander Smirnov <asmirnov@ilbers.de>]

On 02/14/2018 10:42 PM, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> When running multiple build targets in parallel, they will compete for
> access on the package database while the related commands do not wait
> but fail immediately.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
> 
> Fixes parallel builds (CI setup) for me.
> 
> I just recalled that Chris was reporting the same issue a long time ago,
> but his suggestion to lock down whole do_build was not that well
> received...
> 
>   meta/recipes-devtools/buildchroot/files/build.sh | 32 +++++++++++++++---------
>   1 file changed, 20 insertions(+), 12 deletions(-)
> 
> diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
> index 023a6d6..048f76f 100644
> --- a/meta/recipes-devtools/buildchroot/files/build.sh
> +++ b/meta/recipes-devtools/buildchroot/files/build.sh
> @@ -2,17 +2,9 @@
>   #
>   # This software is a part of ISAR.
>   # Copyright (C) 2015-2017 ilbers GmbH
> +# Copyright (c) 2018 Siemens AG
>   
> -# Make sure that we have latest isar-apt content.
> -# Options meaning:
> -#   Dir::Etc::sourcelist - specifies which source to be used
> -#   Dir::Etc::sourceparts - disables looking for the other sources
> -#   APT::Get::List-Cleanup - do not erase obsolete packages list for
> -#                            upstream in '/var/lib/apt/lists'
> -apt-get update \
> -    -o Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \
> -    -o Dir::Etc::sourceparts="-" \
> -    -o APT::Get::List-Cleanup="0"
> +set -e
>   
>   # Go to build directory
>   cd $1
> @@ -26,8 +18,24 @@ install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y
>   # Allow unauthenticated feeds
>   install_cmd="${install_cmd} --allow-unauthenticated"
>   
> -# Install all build deps
> -mk-build-deps -t "${install_cmd}" -i -r debian/control
> +(
> +    # Lock-protected because apt and dpkg do not wait in case of contention
> +    flock 42 || exit 1
> +
> +    # Make sure that we have latest isar-apt content.
> +    # Options meaning:
> +    #   Dir::Etc::sourcelist - specifies which source to be used
> +    #   Dir::Etc::sourceparts - disables looking for the other sources
> +    #   APT::Get::List-Cleanup - do not erase obsolete packages list for
> +    #                            upstream in '/var/lib/apt/lists'
> +    apt-get update \
> +        -o Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \
> +        -o Dir::Etc::sourceparts="-" \
> +        -o APT::Get::List-Cleanup="0"
> +
> +    # Install all build deps
> +    mk-build-deps -t "${install_cmd}" -i -r debian/control
> +) 42>/dpkg.lock

BTW, what is the difference between the construction above and the 
construction you used in dpkg-base.bbclass:

8<--
sudo flock ${MOUNT_LOCKFILE} -c ' \
     if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then \
         mount --bind ${DEPLOY_DIR_APT}/${DISTRO} 
${BUILDCHROOT_DIR}/isar-apt; \
         mount -t devtmpfs -o mode=0755,nosuid devtmpfs 
${BUILDCHROOT_DIR}/dev; \
         mount -t proc none ${BUILDCHROOT_DIR}/proc; \
     fi'
8<--

If there is no difference, could we somehow unify usage of flock in Isar?

Alex

>   
>   # If autotools files have been created, update their timestamp to
>   # prevent them from being regenerated
>