[PATCH] deb-dl-dir: Fallback to copying when import-export

[PATCH] deb-dl-dir: Fallback to copying when import-export

[Uladzimir Bely]

By default, hardlinks are used when doing DL_DIR import and export.
This fails if DL_DIR's filesystem differs from TMPDIR's one.

Fallback to file copying (with lower performance and huge disk usage)
in this case.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/deb-dl-dir.bbclass | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 7db25251..d572e6fb 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -80,13 +80,20 @@ deb_dl_dir_import() {
     export pc="${DEBDIR}/${2}"
     export rootfs="${1}"
     sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
+
+    # Use hardlinks with fallback to `cp` if different filesystems
+    export cmd="ln -Pf -t"
+    if [ "$(stat -c %m ${pc})" != "$(stat -c %m ${rootfs})" ]; then
+        export cmd="cp -n --no-preserve=owner"
+    fi
+
     [ ! -d "${pc}" ] && return 0
     flock -s "${pc}".lock -c '
         set -e
         printenv | grep -q BB_VERBOSE_LOGS && set -x
 
         sudo find "${pc}" -type f -iname "*\.deb" -exec \
-            ln -Pf -t "${rootfs}"/var/cache/apt/archives/ {} +
+            ${cmd} "${rootfs}"/var/cache/apt/archives/ {} +
     '
 }
 
@@ -94,6 +101,13 @@ deb_dl_dir_export() {
     export pc="${DEBDIR}/${2}"
     export rootfs="${1}"
     mkdir -p "${pc}"
+
+    # Use hardlinks with fallback to `cp` if different filesystems
+    export cmd="ln -Pf"
+    if [ "$(stat -c %m ${pc})" != "$(stat -c %m ${rootfs})" ]; then
+        export cmd="cp -n"
+    fi
+
     flock "${pc}".lock -c '
         set -e
         printenv | grep -q BB_VERBOSE_LOGS && set -x
@@ -109,7 +123,7 @@ deb_dl_dir_export() {
             if [ -n "$package" ]; then
                 cmp --silent "$package" "$p" && continue
             fi
-            sudo ln -Pf "${p}" "${pc}"
+            sudo ${cmd} "${p}" "${pc}"
         done
         sudo chown -R $(id -u):$(id -g) "${pc}"
     '
-- 
2.20.1

Re: [PATCH] deb-dl-dir: Fallback to copying when import-export

[Henning Schild]

Am Tue, 31 Jan 2023 12:20:12 +0100
schrieb Uladzimir Bely <ubely@ilbers.de>:

> By default, hardlinks are used when doing DL_DIR import and export.
> This fails if DL_DIR's filesystem differs from TMPDIR's one.
> 
> Fallback to file copying (with lower performance and huge disk usage)
> in this case.
> 
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  meta/classes/deb-dl-dir.bbclass | 18 ++++++++++++++++--
>  1 file changed, 16 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/deb-dl-dir.bbclass
> b/meta/classes/deb-dl-dir.bbclass index 7db25251..d572e6fb 100644
> --- a/meta/classes/deb-dl-dir.bbclass
> +++ b/meta/classes/deb-dl-dir.bbclass
> @@ -80,13 +80,20 @@ deb_dl_dir_import() {
>      export pc="${DEBDIR}/${2}"
>      export rootfs="${1}"
>      sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
> +
> +    # Use hardlinks with fallback to `cp` if different filesystems
> +    export cmd="ln -Pf -t"
> +    if [ "$(stat -c %m ${pc})" != "$(stat -c %m ${rootfs})" ]; then

Instead of dealing only with the known problem and expecting it to
come, maybe try an actual hardlink and "catch the exception".

cmd = ln
cmd testfile || cmd = cp
rm -f testfile

Who knows which other reasons we might find where hardlinks do not
work. And who knows how that stat could get confused by bind-mounts or
funny filesystems ... where hardlinks might work after all, even if the
stat would guess not.

So instead of guessing i am all for actually trying.

Henning

> +        export cmd="cp -n --no-preserve=owner"
> +    fi
> +
>      [ ! -d "${pc}" ] && return 0
>      flock -s "${pc}".lock -c '
>          set -e
>          printenv | grep -q BB_VERBOSE_LOGS && set -x
>  
>          sudo find "${pc}" -type f -iname "*\.deb" -exec \
> -            ln -Pf -t "${rootfs}"/var/cache/apt/archives/ {} +
> +            ${cmd} "${rootfs}"/var/cache/apt/archives/ {} +
>      '
>  }
>  
> @@ -94,6 +101,13 @@ deb_dl_dir_export() {
>      export pc="${DEBDIR}/${2}"
>      export rootfs="${1}"
>      mkdir -p "${pc}"
> +
> +    # Use hardlinks with fallback to `cp` if different filesystems
> +    export cmd="ln -Pf"
> +    if [ "$(stat -c %m ${pc})" != "$(stat -c %m ${rootfs})" ]; then
> +        export cmd="cp -n"
> +    fi
> +
>      flock "${pc}".lock -c '
>          set -e
>          printenv | grep -q BB_VERBOSE_LOGS && set -x
> @@ -109,7 +123,7 @@ deb_dl_dir_export() {
>              if [ -n "$package" ]; then
>                  cmp --silent "$package" "$p" && continue
>              fi
> -            sudo ln -Pf "${p}" "${pc}"
> +            sudo ${cmd} "${p}" "${pc}"
>          done
>          sudo chown -R $(id -u):$(id -g) "${pc}"
>      '

Re: [PATCH] deb-dl-dir: Fallback to copying when import-export

[Roberto A. Foglietta]

On Wed, 1 Feb 2023 at 10:00, Henning Schild <henning.schild@siemens.com> wrote:
>
> Am Tue, 31 Jan 2023 12:20:12 +0100
> schrieb Uladzimir Bely <ubely@ilbers.de>:
>
> > By default, hardlinks are used when doing DL_DIR import and export.
> > This fails if DL_DIR's filesystem differs from TMPDIR's one.
> >
> > Fallback to file copying (with lower performance and huge disk usage)
> > in this case.
> >
> > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > ---
> >  meta/classes/deb-dl-dir.bbclass | 18 ++++++++++++++++--
> >  1 file changed, 16 insertions(+), 2 deletions(-)
> >
> > diff --git a/meta/classes/deb-dl-dir.bbclass
> > b/meta/classes/deb-dl-dir.bbclass index 7db25251..d572e6fb 100644
> > --- a/meta/classes/deb-dl-dir.bbclass
> > +++ b/meta/classes/deb-dl-dir.bbclass
> > @@ -80,13 +80,20 @@ deb_dl_dir_import() {
> >      export pc="${DEBDIR}/${2}"
> >      export rootfs="${1}"
> >      sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
> > +
> > +    # Use hardlinks with fallback to `cp` if different filesystems
> > +    export cmd="ln -Pf -t"
> > +    if [ "$(stat -c %m ${pc})" != "$(stat -c %m ${rootfs})" ]; then
>
> Instead of dealing only with the known problem and expecting it to
> come, maybe try an actual hardlink and "catch the exception".
>

Hi Henning,

> cmd = ln

cmd = "ln -Pft $dest" but this is not the issue

> cmd testfile || cmd = cp

It might not work because cp -l complains and fails but ln -P
complains but succeeds (or at least succeeds enough even in some cases
in which it complains). IMHO, if it complains it also returns an error
as well. But also this is not the issue.


> rm -f testfile
>
> Who knows which other reasons we might find where hardlinks do not
> work. And who knows how that stat could get confused by bind-mounts or
> funny filesystems ... where hardlinks might work after all, even if the
> stat would guess not.
>
> So instead of guessing i am all for actually trying.
>

The general approach you are suggesting is valid and it is good advice.

Best regards, R-