Re: [PATCH v3] deb-dl-dir class rework to use faster ln -P or fallback to cp

[Uladzimir Bely <ubely@ilbers.de>]

In the email from Tuesday, 7 February 2023 10:52:22 +03 user 
roberto.foglietta@linuxteam.org wrote:
> From: "Roberto A. Foglietta" <roberto.foglietta@gmail.com>
> 
> deb-dl-dir, feature: faster when using ln -P otherwise fallback to cp
> 
> The original class functions deb_dl_dir_import/export were using cp to
> copy debian package to the target rootfs but this approach is quite slow
> while using hard link does not work if the destination and source dirs
> are not lying on the same filesystem. Thus, ln -P should fallback to cp
> when it does not work (which is different from complaining on stderr).
> 
> Moreover, these two functions have been reworked to reach a straight
> forward and more compact form. In particular, export function was using
> bashism to do some kind of comparison which after all is useless
> because copying back without overwriting just fulfills that part.
> 
> More rework using sudo in a different way plus a corner case
> addressingi, in case the spia file exists for some other reasons.
> 
> Rebased on the current next
> 
> Signed-off-by: Roberto A. Foglietta <roberto.foglietta@gmail.com>
> ---
>  meta/classes/deb-dl-dir.bbclass | 53 +++++++++++++++++++--------------
>  1 file changed, 30 insertions(+), 23 deletions(-)
> 


> diff --git a/meta/classes/deb-dl-dir.bbclass
> b/meta/classes/deb-dl-dir.bbclass index 7db25251..1fe9d40c 100644
> --- a/meta/classes/deb-dl-dir.bbclass
> +++ b/meta/classes/deb-dl-dir.bbclass
> @@ -78,39 +78,46 @@ debsrc_download() {
> 
>  deb_dl_dir_import() {
>      export pc="${DEBDIR}/${2}"
> -    export rootfs="${1}"
> -    sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
> +    export sc="${1}/var/cache/apt/archives/"
> +    sudo mkdir -p "${sc}"
>      [ ! -d "${pc}" ] && return 0
> -    flock -s "${pc}".lock -c '
> +    export tf=$(ls -1 "${pc}"/*.deb | head -n1)
> +    [ ! -e "${tf}" ] && return 0
> +    flock -Fs "${pc}".lock sudo -Es << 'EOFSUDO'
>          set -e
>          printenv | grep -q BB_VERBOSE_LOGS && set -x
> 
> -        sudo find "${pc}" -type f -iname "*\.deb" -exec \
> -            ln -Pf -t "${rootfs}"/var/cache/apt/archives/ {} +
> -    '
> +        rm -f "${sc}/"$(basename "${tf}")
> +        ln -Pf -t "${sc}" "${tf}" 2>/dev/null ||:
> +        if [ -r "${tf}" ]; then
> +            find "${pc}" -type f -iname "*\.deb" -exec \
> +                ln -Pf -t "${sc}" {} +
> +        else
> +            find "${pc}" -type f -iname "*\.deb" -exec \
> +                cp -np owner --reflink=auto -t "${sc}" {} +
> +        fi
> +EOFSUDO
>  }
> 
>  deb_dl_dir_export() {
>      export pc="${DEBDIR}/${2}"
> -    export rootfs="${1}"
> +    export sc="${1}/var/cache/apt/archives/"
>      mkdir -p "${pc}"
> -    flock "${pc}".lock -c '
> +    export tf=$(ls -1 "${sc}"/*.deb | head -n1)
> +    [ ! -e "${tf}" ] && return 0
> +    flock -F "${pc}".lock sudo -Es << 'EOFSUDO'
>          set -e
>          printenv | grep -q BB_VERBOSE_LOGS && set -x
> 
> -        find "${rootfs}"/var/cache/apt/archives/ \
> -            -maxdepth 1 -type f -iname '*\.deb' |\
> -        while read p; do
> -            # skip files from a previous export
> -            [ -f "${pc}/${p##*/}" ] && continue
> -            # can not reuse bitbake function here, this is basically
> -            # "repo_contains_package"
> -            package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name ${p##*/})
> -            if [ -n "$package" ]; then
> -                cmp --silent "$package" "$p" && continue
> -            fi
> -            sudo ln -Pf "${p}" "${pc}"
> -        done
> -        sudo chown -R $(id -u):$(id -g) "${pc}"
> -    '
> +        rm -f "${pc}/"$(basename "${tf}")
> +        ln -Pf -t "${pc}" "${tf}" 2>/dev/null ||:
> +        if [ -r "${tf}" ]; then
> +            find "${sc}" -maxdepth 1 -type f -iname '*\.deb' \
> +                -exec ln -P -t "${pc}" {} + 2>/dev/null ||:
> +        else
> +            find "${sc}" -maxdepth 1 -type f -iname '*\.deb' \
> +                -exec cp -n --reflink=auto -t "${pc}" {} +
> +        fi
> +        chown -R $(id -u):$(id -g) "${pc}"
> +EOFSUDO
>  }

I was almost ready to merge it since it passes CI (positive case), but 
patchset appeared don't work (negative case) on different filesystems - I find 
downloads/deb/*/ empty after build. At least, exports don't work.

See the example of my build for "negative" case: https://dpaste.org/
MAUKv#L1,22,23,36,37,38,39,50,52

You could also easy test it by creating "downloads.img" with ext4 inside and 
mounting it to "downloads" to make DL_DIR and BUILD_DIR be on different 
filesystems.

I still tend to apply my "[v2] deb-dl-dir: Fallback to copying when import-
export" instead. But it is worth reworking a bit, regarding sudo using in 
loops (e.g., would prefer "sudo / while <cmd>" Instead of "while / sudo 
<cmd>")...