* [PATCH v3] deb-dl-dir class rework to use faster ln -P or fallback to cp @ 2023-02-07 7:52 roberto.foglietta 2023-02-10 10:07 ` Uladzimir Bely 0 siblings, 1 reply; 3+ messages in thread From: roberto.foglietta @ 2023-02-07 7:52 UTC (permalink / raw) To: isar-users; +Cc: roberto.foglietta From: "Roberto A. Foglietta" <roberto.foglietta@gmail.com> deb-dl-dir, feature: faster when using ln -P otherwise fallback to cp The original class functions deb_dl_dir_import/export were using cp to copy debian package to the target rootfs but this approach is quite slow while using hard link does not work if the destination and source dirs are not lying on the same filesystem. Thus, ln -P should fallback to cp when it does not work (which is different from complaining on stderr). Moreover, these two functions have been reworked to reach a straight forward and more compact form. In particular, export function was using bashism to do some kind of comparison which after all is useless because copying back without overwriting just fulfills that part. More rework using sudo in a different way plus a corner case addressingi, in case the spia file exists for some other reasons. Rebased on the current next Signed-off-by: Roberto A. Foglietta <roberto.foglietta@gmail.com> --- meta/classes/deb-dl-dir.bbclass | 53 +++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 23 deletions(-) diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass index 7db25251..1fe9d40c 100644 --- a/meta/classes/deb-dl-dir.bbclass +++ b/meta/classes/deb-dl-dir.bbclass @@ -78,39 +78,46 @@ debsrc_download() { deb_dl_dir_import() { export pc="${DEBDIR}/${2}" - export rootfs="${1}" - sudo mkdir -p "${rootfs}"/var/cache/apt/archives/ + export sc="${1}/var/cache/apt/archives/" + sudo mkdir -p "${sc}" [ ! -d "${pc}" ] && return 0 - flock -s "${pc}".lock -c ' + export tf=$(ls -1 "${pc}"/*.deb | head -n1) + [ ! -e "${tf}" ] && return 0 + flock -Fs "${pc}".lock sudo -Es << 'EOFSUDO' set -e printenv | grep -q BB_VERBOSE_LOGS && set -x - sudo find "${pc}" -type f -iname "*\.deb" -exec \ - ln -Pf -t "${rootfs}"/var/cache/apt/archives/ {} + - ' + rm -f "${sc}/"$(basename "${tf}") + ln -Pf -t "${sc}" "${tf}" 2>/dev/null ||: + if [ -r "${tf}" ]; then + find "${pc}" -type f -iname "*\.deb" -exec \ + ln -Pf -t "${sc}" {} + + else + find "${pc}" -type f -iname "*\.deb" -exec \ + cp -np owner --reflink=auto -t "${sc}" {} + + fi +EOFSUDO } deb_dl_dir_export() { export pc="${DEBDIR}/${2}" - export rootfs="${1}" + export sc="${1}/var/cache/apt/archives/" mkdir -p "${pc}" - flock "${pc}".lock -c ' + export tf=$(ls -1 "${sc}"/*.deb | head -n1) + [ ! -e "${tf}" ] && return 0 + flock -F "${pc}".lock sudo -Es << 'EOFSUDO' set -e printenv | grep -q BB_VERBOSE_LOGS && set -x - find "${rootfs}"/var/cache/apt/archives/ \ - -maxdepth 1 -type f -iname '*\.deb' |\ - while read p; do - # skip files from a previous export - [ -f "${pc}/${p##*/}" ] && continue - # can not reuse bitbake function here, this is basically - # "repo_contains_package" - package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name ${p##*/}) - if [ -n "$package" ]; then - cmp --silent "$package" "$p" && continue - fi - sudo ln -Pf "${p}" "${pc}" - done - sudo chown -R $(id -u):$(id -g) "${pc}" - ' + rm -f "${pc}/"$(basename "${tf}") + ln -Pf -t "${pc}" "${tf}" 2>/dev/null ||: + if [ -r "${tf}" ]; then + find "${sc}" -maxdepth 1 -type f -iname '*\.deb' \ + -exec ln -P -t "${pc}" {} + 2>/dev/null ||: + else + find "${sc}" -maxdepth 1 -type f -iname '*\.deb' \ + -exec cp -n --reflink=auto -t "${pc}" {} + + fi + chown -R $(id -u):$(id -g) "${pc}" +EOFSUDO } -- 2.34.1 ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH v3] deb-dl-dir class rework to use faster ln -P or fallback to cp 2023-02-07 7:52 [PATCH v3] deb-dl-dir class rework to use faster ln -P or fallback to cp roberto.foglietta @ 2023-02-10 10:07 ` Uladzimir Bely 2023-02-10 10:50 ` Roberto A. Foglietta 0 siblings, 1 reply; 3+ messages in thread From: Uladzimir Bely @ 2023-02-10 10:07 UTC (permalink / raw) To: isar-users, roberto.foglietta In the email from Tuesday, 7 February 2023 10:52:22 +03 user roberto.foglietta@linuxteam.org wrote: > From: "Roberto A. Foglietta" <roberto.foglietta@gmail.com> > > deb-dl-dir, feature: faster when using ln -P otherwise fallback to cp > > The original class functions deb_dl_dir_import/export were using cp to > copy debian package to the target rootfs but this approach is quite slow > while using hard link does not work if the destination and source dirs > are not lying on the same filesystem. Thus, ln -P should fallback to cp > when it does not work (which is different from complaining on stderr). > > Moreover, these two functions have been reworked to reach a straight > forward and more compact form. In particular, export function was using > bashism to do some kind of comparison which after all is useless > because copying back without overwriting just fulfills that part. > > More rework using sudo in a different way plus a corner case > addressingi, in case the spia file exists for some other reasons. > > Rebased on the current next > > Signed-off-by: Roberto A. Foglietta <roberto.foglietta@gmail.com> > --- > meta/classes/deb-dl-dir.bbclass | 53 +++++++++++++++++++-------------- > 1 file changed, 30 insertions(+), 23 deletions(-) > > diff --git a/meta/classes/deb-dl-dir.bbclass > b/meta/classes/deb-dl-dir.bbclass index 7db25251..1fe9d40c 100644 > --- a/meta/classes/deb-dl-dir.bbclass > +++ b/meta/classes/deb-dl-dir.bbclass > @@ -78,39 +78,46 @@ debsrc_download() { > > deb_dl_dir_import() { > export pc="${DEBDIR}/${2}" > - export rootfs="${1}" > - sudo mkdir -p "${rootfs}"/var/cache/apt/archives/ > + export sc="${1}/var/cache/apt/archives/" > + sudo mkdir -p "${sc}" > [ ! -d "${pc}" ] && return 0 > - flock -s "${pc}".lock -c ' > + export tf=$(ls -1 "${pc}"/*.deb | head -n1) > + [ ! -e "${tf}" ] && return 0 > + flock -Fs "${pc}".lock sudo -Es << 'EOFSUDO' > set -e > printenv | grep -q BB_VERBOSE_LOGS && set -x > > - sudo find "${pc}" -type f -iname "*\.deb" -exec \ > - ln -Pf -t "${rootfs}"/var/cache/apt/archives/ {} + > - ' > + rm -f "${sc}/"$(basename "${tf}") > + ln -Pf -t "${sc}" "${tf}" 2>/dev/null ||: > + if [ -r "${tf}" ]; then > + find "${pc}" -type f -iname "*\.deb" -exec \ > + ln -Pf -t "${sc}" {} + > + else > + find "${pc}" -type f -iname "*\.deb" -exec \ > + cp -np owner --reflink=auto -t "${sc}" {} + > + fi > +EOFSUDO > } > > deb_dl_dir_export() { > export pc="${DEBDIR}/${2}" > - export rootfs="${1}" > + export sc="${1}/var/cache/apt/archives/" > mkdir -p "${pc}" > - flock "${pc}".lock -c ' > + export tf=$(ls -1 "${sc}"/*.deb | head -n1) > + [ ! -e "${tf}" ] && return 0 > + flock -F "${pc}".lock sudo -Es << 'EOFSUDO' > set -e > printenv | grep -q BB_VERBOSE_LOGS && set -x > > - find "${rootfs}"/var/cache/apt/archives/ \ > - -maxdepth 1 -type f -iname '*\.deb' |\ > - while read p; do > - # skip files from a previous export > - [ -f "${pc}/${p##*/}" ] && continue > - # can not reuse bitbake function here, this is basically > - # "repo_contains_package" > - package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name ${p##*/}) > - if [ -n "$package" ]; then > - cmp --silent "$package" "$p" && continue > - fi > - sudo ln -Pf "${p}" "${pc}" > - done > - sudo chown -R $(id -u):$(id -g) "${pc}" > - ' > + rm -f "${pc}/"$(basename "${tf}") > + ln -Pf -t "${pc}" "${tf}" 2>/dev/null ||: > + if [ -r "${tf}" ]; then > + find "${sc}" -maxdepth 1 -type f -iname '*\.deb' \ > + -exec ln -P -t "${pc}" {} + 2>/dev/null ||: > + else > + find "${sc}" -maxdepth 1 -type f -iname '*\.deb' \ > + -exec cp -n --reflink=auto -t "${pc}" {} + > + fi > + chown -R $(id -u):$(id -g) "${pc}" > +EOFSUDO > } I was almost ready to merge it since it passes CI (positive case), but patchset appeared don't work (negative case) on different filesystems - I find downloads/deb/*/ empty after build. At least, exports don't work. See the example of my build for "negative" case: https://dpaste.org/ MAUKv#L1,22,23,36,37,38,39,50,52 You could also easy test it by creating "downloads.img" with ext4 inside and mounting it to "downloads" to make DL_DIR and BUILD_DIR be on different filesystems. I still tend to apply my "[v2] deb-dl-dir: Fallback to copying when import- export" instead. But it is worth reworking a bit, regarding sudo using in loops (e.g., would prefer "sudo / while <cmd>" Instead of "while / sudo <cmd>")... ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH v3] deb-dl-dir class rework to use faster ln -P or fallback to cp 2023-02-10 10:07 ` Uladzimir Bely @ 2023-02-10 10:50 ` Roberto A. Foglietta 0 siblings, 0 replies; 3+ messages in thread From: Roberto A. Foglietta @ 2023-02-10 10:50 UTC (permalink / raw) To: Uladzimir Bely; +Cc: isar-users, roberto.foglietta On Fri, 10 Feb 2023 at 11:07, Uladzimir Bely <ubely@ilbers.de> wrote: > > > > deb_dl_dir_import() { > > export pc="${DEBDIR}/${2}" > > - export rootfs="${1}" > > - sudo mkdir -p "${rootfs}"/var/cache/apt/archives/ > > + export sc="${1}/var/cache/apt/archives/" > > + sudo mkdir -p "${sc}" > > [ ! -d "${pc}" ] && return 0 > > - flock -s "${pc}".lock -c ' > > + export tf=$(ls -1 "${pc}"/*.deb | head -n1) > > + [ ! -e "${tf}" ] && return 0 > > + flock -Fs "${pc}".lock sudo -Es << 'EOFSUDO' > > set -e > > printenv | grep -q BB_VERBOSE_LOGS && set -x > > > > - sudo find "${pc}" -type f -iname "*\.deb" -exec \ > > - ln -Pf -t "${rootfs}"/var/cache/apt/archives/ {} + > > - ' > > + rm -f "${sc}/"$(basename "${tf}") > > + ln -Pf -t "${sc}" "${tf}" 2>/dev/null ||: > > + if [ -r "${tf}" ]; then Because we need to check the destination, not the source. if exists $sc/$(basename $tf) then > > I was almost ready to merge it since it passes CI (positive case), but > patchset appeared don't work (negative case) on different filesystems - I find > downloads/deb/*/ empty after build. At least, exports don't work. Both do not work correctly. I am going to fix it. Sorry, R- ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2023-02-10 10:51 UTC | newest] Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-02-07 7:52 [PATCH v3] deb-dl-dir class rework to use faster ln -P or fallback to cp roberto.foglietta 2023-02-10 10:07 ` Uladzimir Bely 2023-02-10 10:50 ` Roberto A. Foglietta
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox