From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7197029299818332160 X-Received: by 2002:a17:902:da83:b0:199:4d8:9e71 with SMTP id j3-20020a170902da8300b0019904d89e71mr3695339plx.26.1676023652347; Fri, 10 Feb 2023 02:07:32 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:90b:3510:b0:230:f6f5:8bab with SMTP id ls16-20020a17090b351000b00230f6f58babls8742510pjb.1.-pod-canary-gmail; Fri, 10 Feb 2023 02:07:31 -0800 (PST) X-Google-Smtp-Source: AK7set9y78rAdyFP6A/z/Q4VvMDjHb1ilyf3I1Y7hdBopoHu63T6/Pek9M+gAvS2+cx2NPAlWMCW X-Received: by 2002:a17:90b:1d86:b0:22c:5b70:72b6 with SMTP id pf6-20020a17090b1d8600b0022c5b7072b6mr16239517pjb.1.1676023651344; Fri, 10 Feb 2023 02:07:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676023651; cv=none; d=google.com; s=arc-20160816; b=VG80Fod+jmuXNfgQwh7UmCyYrNq5YgIDkZalILfTBC9lMoMfV+ZhVniMGcS+k43VWZ Xe9SdbwU5ArqS47W/++/vgwCZiGg5n/Yeax300ax4OVAvz0E+i6UbHJVof5GHpwRmEwq dJKtz9okDDZSe06ksmRjSYZBacxjs+OuMYnpLeMNm6iWhPj5pvqfLFSsq9ontWkj7Exn Mkt8zf0X42Pn29B8tkXdOMoyQdyuBtfkY0PeaS/9B8ng66RFzsxUiRL7fwz+mwRnoF+7 u+1fySH4j3buI+0MXZj7X6N6CAUfmyCq4tZVBXtjpkiGHTnkTUVB4GEFQaLFQeas33ls f90A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=DhbJUrk+7JhXHq5S0DMADP3ZfjpTChCIhgScwVcv4Bc=; b=aaFNh3czzRCzpohZ0H0WB5irbeoEFdhhMZip7tdR2Y51yaODvE6xMlAEX3HQ6BAJ9Z pGAKHmN7lFFMGb3QwGtdyeY95ldM+Gd9Slw8BAeIEtLjLtzladLpOWqKLnRRji0PrOy/ 8jRY1mpax1IjbiFae7tw7X4NUN615dql+zs96s8k/JRqDz/E2sVurVVLGlD848V9gfAp kV0Dj4vsSMLKGc7QnyPUM/nFV8zViv+D2BmfefX5K3A9o2SLX7/MltBA1poACYoVyYZI Km+CcbM6CmBphpoIEit3JyuFoFEk8xfXzcRw/4P/RRQrM1JGrNYwFHwTPvNtdmuqOQFf 72Kw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id c24-20020a17090a8d1800b002309f8d0078si880402pjo.0.2023.02.10.02.07.30 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 10 Feb 2023 02:07:31 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from home.localnet (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 31AA7S4o023403 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 10 Feb 2023 11:07:28 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com, roberto.foglietta@linuxteam.org Subject: Re: [PATCH v3] deb-dl-dir class rework to use faster ln -P or fallback to cp Date: Fri, 10 Feb 2023 13:07:26 +0300 Message-ID: <15525925.Emhk5qWAgF@home> In-Reply-To: <20230207075222.390954-1-roberto.foglietta@linuxteam.org> References: <20230207075222.390954-1-roberto.foglietta@linuxteam.org> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: lHV4cYf5kwce In the email from Tuesday, 7 February 2023 10:52:22 +03 user roberto.foglietta@linuxteam.org wrote: > From: "Roberto A. Foglietta" > > deb-dl-dir, feature: faster when using ln -P otherwise fallback to cp > > The original class functions deb_dl_dir_import/export were using cp to > copy debian package to the target rootfs but this approach is quite slow > while using hard link does not work if the destination and source dirs > are not lying on the same filesystem. Thus, ln -P should fallback to cp > when it does not work (which is different from complaining on stderr). > > Moreover, these two functions have been reworked to reach a straight > forward and more compact form. In particular, export function was using > bashism to do some kind of comparison which after all is useless > because copying back without overwriting just fulfills that part. > > More rework using sudo in a different way plus a corner case > addressingi, in case the spia file exists for some other reasons. > > Rebased on the current next > > Signed-off-by: Roberto A. Foglietta > --- > meta/classes/deb-dl-dir.bbclass | 53 +++++++++++++++++++-------------- > 1 file changed, 30 insertions(+), 23 deletions(-) > > diff --git a/meta/classes/deb-dl-dir.bbclass > b/meta/classes/deb-dl-dir.bbclass index 7db25251..1fe9d40c 100644 > --- a/meta/classes/deb-dl-dir.bbclass > +++ b/meta/classes/deb-dl-dir.bbclass > @@ -78,39 +78,46 @@ debsrc_download() { > > deb_dl_dir_import() { > export pc="${DEBDIR}/${2}" > - export rootfs="${1}" > - sudo mkdir -p "${rootfs}"/var/cache/apt/archives/ > + export sc="${1}/var/cache/apt/archives/" > + sudo mkdir -p "${sc}" > [ ! -d "${pc}" ] && return 0 > - flock -s "${pc}".lock -c ' > + export tf=$(ls -1 "${pc}"/*.deb | head -n1) > + [ ! -e "${tf}" ] && return 0 > + flock -Fs "${pc}".lock sudo -Es << 'EOFSUDO' > set -e > printenv | grep -q BB_VERBOSE_LOGS && set -x > > - sudo find "${pc}" -type f -iname "*\.deb" -exec \ > - ln -Pf -t "${rootfs}"/var/cache/apt/archives/ {} + > - ' > + rm -f "${sc}/"$(basename "${tf}") > + ln -Pf -t "${sc}" "${tf}" 2>/dev/null ||: > + if [ -r "${tf}" ]; then > + find "${pc}" -type f -iname "*\.deb" -exec \ > + ln -Pf -t "${sc}" {} + > + else > + find "${pc}" -type f -iname "*\.deb" -exec \ > + cp -np owner --reflink=auto -t "${sc}" {} + > + fi > +EOFSUDO > } > > deb_dl_dir_export() { > export pc="${DEBDIR}/${2}" > - export rootfs="${1}" > + export sc="${1}/var/cache/apt/archives/" > mkdir -p "${pc}" > - flock "${pc}".lock -c ' > + export tf=$(ls -1 "${sc}"/*.deb | head -n1) > + [ ! -e "${tf}" ] && return 0 > + flock -F "${pc}".lock sudo -Es << 'EOFSUDO' > set -e > printenv | grep -q BB_VERBOSE_LOGS && set -x > > - find "${rootfs}"/var/cache/apt/archives/ \ > - -maxdepth 1 -type f -iname '*\.deb' |\ > - while read p; do > - # skip files from a previous export > - [ -f "${pc}/${p##*/}" ] && continue > - # can not reuse bitbake function here, this is basically > - # "repo_contains_package" > - package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name ${p##*/}) > - if [ -n "$package" ]; then > - cmp --silent "$package" "$p" && continue > - fi > - sudo ln -Pf "${p}" "${pc}" > - done > - sudo chown -R $(id -u):$(id -g) "${pc}" > - ' > + rm -f "${pc}/"$(basename "${tf}") > + ln -Pf -t "${pc}" "${tf}" 2>/dev/null ||: > + if [ -r "${tf}" ]; then > + find "${sc}" -maxdepth 1 -type f -iname '*\.deb' \ > + -exec ln -P -t "${pc}" {} + 2>/dev/null ||: > + else > + find "${sc}" -maxdepth 1 -type f -iname '*\.deb' \ > + -exec cp -n --reflink=auto -t "${pc}" {} + > + fi > + chown -R $(id -u):$(id -g) "${pc}" > +EOFSUDO > } I was almost ready to merge it since it passes CI (positive case), but patchset appeared don't work (negative case) on different filesystems - I find downloads/deb/*/ empty after build. At least, exports don't work. See the example of my build for "negative" case: https://dpaste.org/ MAUKv#L1,22,23,36,37,38,39,50,52 You could also easy test it by creating "downloads.img" with ext4 inside and mounting it to "downloads" to make DL_DIR and BUILD_DIR be on different filesystems. I still tend to apply my "[v2] deb-dl-dir: Fallback to copying when import- export" instead. But it is worth reworking a bit, regarding sudo using in loops (e.g., would prefer "sudo / while " Instead of "while / sudo ")...