From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7197029299818332160 X-Received: by 2002:a5d:69ce:0:b0:2c5:4c8e:3d30 with SMTP id s14-20020a5d69ce000000b002c54c8e3d30mr40893wrw.624.1676032934455; Fri, 10 Feb 2023 04:42:14 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:58cd:0:b0:2bf:ccef:53a6 with SMTP id o13-20020a5d58cd000000b002bfccef53a6ls228434wrf.0.-pod-prod-gmail; Fri, 10 Feb 2023 04:42:13 -0800 (PST) X-Google-Smtp-Source: AK7set8Gf2kIakJPRRH0KtmcylZcreU+9Q8/Arc60fP2imvwm/LmArbCWMqymm0OczH9tVJDsdBV X-Received: by 2002:a5d:4fc3:0:b0:2c5:4a89:f60b with SMTP id h3-20020a5d4fc3000000b002c54a89f60bmr1449197wrw.38.1676032933116; Fri, 10 Feb 2023 04:42:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676032933; cv=none; d=google.com; s=arc-20160816; b=wDdTWJeorkKi6HdXhzH9PRnOGdKF77SbtPgZxY3LrbJAWjKi8x1k/6LOdJPn8uF0GE fWSDxXLiusm8fm6+Z2VQUUvy86ooz2h/XKDG4cGSyDKqUCa6ROkO4JVn6qafbkqlNLxt du3YMvnqLlhuRPdhjaiMpDO6euBQNIcXBbkFXDIltOPdypcHInRDFUWym//uY5y7w3KU s1DQvRxbmWj3IUkEcxzN2PGvE7jEonPK53hy6Fm/66xMpDwKJi9g+pol5HtmEYD8MAhY hjKjkllAmDhPUTk83NIrTYOWHoRnH9fqGZfqTmXZm6FS/0+6iDXghSHx41Vbn1inBkFZ EKSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=RBXCOTC4TduH4xhi45tf+rYieAq0UrdXMg3sTsboY0s=; b=WA/WhZiTD6YXWvluU0lx+BBeLkAbN2rmhfBrB4cdQFtFa3V9JnZvUKSSpFfkrqTi5I pd+ucsb822aT9jiu0KA6nPUu9EVXYAjUG3ZZ8crof2TRfg6yH816VjBQ04lJbEpnm3L6 sfQc0QYbPCmoBbFoHmszpr87mRPk2tjioCt4EwVBQAa40ia74eH4pM1J76mfXyWu6VJL r7CA8RL+wIfkQelGPrv76hzHdNlbA077rYMXonXAw5ZUUR0SM7ahvRv7ePYCQKtDZFYs jYNAZq2JPiNlt5HHspi1TMR7yeUk+I6r6+idmbNcDCbwy5ZQGt88fk7raeXDuiFv7QeH zNkg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::165 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Return-Path: Received: from relay04.th.seeweb.it (relay04.th.seeweb.it. [2001:4b7a:2000:18::165]) by gmr-mx.google.com with ESMTPS id bg10-20020a05600c3c8a00b003db0d2c3d6esi435424wmb.0.2023.02.10.04.42.13 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Feb 2023 04:42:13 -0800 (PST) Received-SPF: pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::165 as permitted sender) client-ip=2001:4b7a:2000:18::165; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::165 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Received: from localhost.localdomain (unknown [IPv6:2a02:8071:3187:7b80:e47:5fa:2d03:1a15]) by m-r1.th.seeweb.it (Postfix) with ESMTPA id 5859C203F9; Fri, 10 Feb 2023 13:42:12 +0100 (CET) From: roberto.foglietta@linuxteam.org To: isar-users@googlegroups.com Cc: roberto.foglietta@gmail.com Subject: [PATCH v5] deb-dl-dir class rework to use faster ln -P or fallback to cp Date: Fri, 10 Feb 2023 13:42:07 +0100 Message-Id: <20230210124207.1609382-1-roberto.foglietta@linuxteam.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TUID: pFltNTIwqS8F From: "Roberto A. Foglietta" deb-dl-dir, feature: faster when using ln -P otherwise fallback to cp The original class functions deb_dl_dir_import/export were using cp to copy debian package to the target rootfs but this approach is quite slow while using hard link does not work if the destination and source dirs are not lying on the same filesystem. Thus, ln -P should fallback to cp when it does not work (which is different from complaining on stderr). Moreover, these two functions have been reworked to reach a straight forward and more compact form. In particular, export function was using bashism to do some kind of comparison which after all is useless because copying back without overwriting just fulfills that part. More rework using sudo in a different way plus a corner case addressingi, in case the spia file exists for some other reasons. Rebased on the current next and bugfix about checking the destination not the source file. It does not delete a package but touch a test file to link. Signed-off-by: Roberto A. Foglietta --- meta/classes/deb-dl-dir.bbclass | 55 +++++++++++++++++++-------------- 1 file changed, 32 insertions(+), 23 deletions(-) diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass index 7db25251..21925c9f 100644 --- a/meta/classes/deb-dl-dir.bbclass +++ b/meta/classes/deb-dl-dir.bbclass @@ -78,39 +78,48 @@ debsrc_download() { deb_dl_dir_import() { export pc="${DEBDIR}/${2}" - export rootfs="${1}" - sudo mkdir -p "${rootfs}"/var/cache/apt/archives/ + export sc="${1}/var/cache/apt/archives/" + export tf="ln-P-write-test.ok" + sudo mkdir -p "${sc}" [ ! -d "${pc}" ] && return 0 - flock -s "${pc}".lock -c ' + flock -Fs "${pc}".lock sudo -Es << 'EOFSUDO' set -e printenv | grep -q BB_VERBOSE_LOGS && set -x - sudo find "${pc}" -type f -iname "*\.deb" -exec \ - ln -Pf -t "${rootfs}"/var/cache/apt/archives/ {} + - ' + rm -f "${sc}/${tf}" + touch "${pc}/${tf}" || exit 1 + ln -Pf -t "${sc}" "${pc}/${tf}" 2>/dev/null ||: + if [ -r "${sc}/${tf}" ]; then + find "${pc}" -type f -iname "*\.deb" -exec \ + ln -Pf -t "${sc}" {} + + else + find "${pc}" -type f -iname "*\.deb" -exec \ + cp -np owner --reflink=auto -t "${sc}" {} + + fi + rm -f "${pc}/${tf}" +EOFSUDO } deb_dl_dir_export() { export pc="${DEBDIR}/${2}" - export rootfs="${1}" + export sc="${1}/var/cache/apt/archives/" + export tf="ln-P-write-test.ok" mkdir -p "${pc}" - flock "${pc}".lock -c ' + flock -F "${pc}".lock sudo -Es << 'EOFSUDO' set -e printenv | grep -q BB_VERBOSE_LOGS && set -x - find "${rootfs}"/var/cache/apt/archives/ \ - -maxdepth 1 -type f -iname '*\.deb' |\ - while read p; do - # skip files from a previous export - [ -f "${pc}/${p##*/}" ] && continue - # can not reuse bitbake function here, this is basically - # "repo_contains_package" - package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name ${p##*/}) - if [ -n "$package" ]; then - cmp --silent "$package" "$p" && continue - fi - sudo ln -Pf "${p}" "${pc}" - done - sudo chown -R $(id -u):$(id -g) "${pc}" - ' + rm -f "${pc}/${tf}" + touch "${sc}/${tf}" || exit 1 + ln -Pf -t "${pc}" "${sc}/${tf}" 2>/dev/null ||: + if [ -r "${pc}/${tf}" ]; then + find "${sc}" -maxdepth 1 -type f -iname '*\.deb' \ + -exec ln -P -t "${pc}" {} + 2>/dev/null ||: + else + find "${sc}" -maxdepth 1 -type f -iname '*\.deb' \ + -exec cp -n --reflink=auto -t "${pc}" {} + + fi + rm -f "${sc}/${tf}" + chown -R $(id -u):$(id -g) "${pc}" +EOFSUDO } -- 2.34.1