Re: [PATCHv5 25/29] meta: deb-dl-dir: only export newly downloaded files

[Henning Schild <henning.schild@siemens.com>]

On Tue, 10 Mar 2020 22:56:27 +0100
Henning Schild <henning.schild@siemens.com> wrote:

> From: Henning Schild <henning.schild@siemens.com>
> 
> Remember the time we imported the previously downloaded files, and
> only copy out new ones when exporting again. That saved pointless
> copies and time.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>  meta/classes/deb-dl-dir.bbclass | 13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes/deb-dl-dir.bbclass
> b/meta/classes/deb-dl-dir.bbclass index 011dd717..55c4a839 100644
> --- a/meta/classes/deb-dl-dir.bbclass
> +++ b/meta/classes/deb-dl-dir.bbclass
> @@ -8,24 +8,31 @@ inherit repository
>  deb_dl_dir_import() {
>      export pc="${DEBDIR}/${DISTRO}/"
>      export rootfs="${1}"
> +    rm -f "${T}"/deb_dl_dir_import.stamp
> +    touch "${T}"/deb_dl_dir_import.stamp

Creating the stamp outside the flock is a race. We could create a new
stamp and get stuck on the lock while a reader would skip new packages
that have not been imported yet.

Henning

>      [ ! -d "${pc}" ] && return 0
> +    [ ! -d "${rootfs}"/var/cache/apt/archives/ ] && return 0
>      flock -s "${pc}".lock -c ' \
>          sudo find "${pc}" -type f -iname '*\.deb' -exec \
> -            cp -f --no-preserve=owner -t
> "${rootfs}"/var/cache/apt/archives/ '{}' +
> +            cp -n --no-preserve=owner -t
> "${rootfs}"/var/cache/apt/archives/ '{}' + '
>  }
>  
>  deb_dl_dir_export() {
>      export pc="${DEBDIR}/${DISTRO}/"
>      export rootfs="${1}"
> +    export T
>      mkdir -p "${pc}"
>      flock "${pc}".lock -c ' \
> -        find "${rootfs}"/var/cache/apt/archives/ -type f -iname
> '*\.deb' |\
> +        find "${rootfs}"/var/cache/apt/archives/ \
> +	    -cnewer "${T}"/deb_dl_dir_import.stamp \
> +	    -type f -iname '*\.deb' |\
>          while read p; do
>               repo_contains_package "${REPO_ISAR_DIR}"/"${DISTRO}"
> "${p}" && \ continue
> -             sudo cp -f "${p}" "${pc}"
> +             sudo cp -n "${p}" "${pc}"
>          done
>          sudo chown -R $(id -u):$(id -g) "${pc}"
>      '
> +    rm -f "${T}"/deb_dl_dir_import.stamp
>  }