From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Sat, 22 Mar 2025 07:15:47 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-ed1-f64.google.com (mail-ed1-f64.google.com [209.85.208.64]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 52M6Fkn5012075 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Sat, 22 Mar 2025 07:15:46 +0100 Received: by mail-ed1-f64.google.com with SMTP id 4fb4d7f45d1cf-5e6a64bd1ecsf2403842a12.2 for ; Fri, 21 Mar 2025 23:15:46 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1742624141; cv=pass; d=google.com; s=arc-20240605; b=eCczH8ov75FLAPyuQy2aD2evcb72TddkCMtlcG5Hsb+kKc5/qk9+JW1AS1PJg0g8TL 6UDcSpsZtJDSbaf0i44WqFdWPqUJwHUXmPcnlfXzPEzzDrrJrZ1eEDG7h9iTgDDD6VL+ 3AtwyxoO87+gwYLf9hOu2W4ch0QYWyiQ3lUo2YTYbc2T27OoXuZwXdESyfJRqQmAilgy oU7KeWzrQ9Yo3NneulvvmvQa3utZSO+PaewA06GszqhLA7SVRTribEaRZxfmjtq5oDcK H9655wzlI5E9mqb3GjlxgaB3o6ZjinWIF4+aKDPMbQjtaFNuHfgGTFcLcefIbxaEAcp5 ScFw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0TXHR1k2WNnQCyTEOL6CnUXKYtyjKSUYYPvKQr0cW44=; fh=r4FOa3pDMTLaTjCU8cJGgOrlJV74TH6HQPnUk70m+DI=; b=K4km53GQ+P/fsRf50EtHhVd1xP4ZLsi1/0oz837BIYgVghHPKsByI5ysjaFfDq0YjP eWbS0iw+t6J/2DCZTFewLEk8jnsL4ceYpgNcEGv+TOk/GkoF8yA294sbQqSyqfAjUI/Q 2/PwWpbh6g+5DY063cdnqarhke3/CVEs/+VVQuUY5vJrNxeDUFQsSyWWk8pkYpjlb1Px sFomsvZS0lalNjVOw0+dHBYM3JNIw1dTm8YH7m4Q8Cfnuskt4ofjaTDWJ+q9sBcAECpg 15PcAIfGyVwLk+/Ur+OKK/zu3udZ5UkJWw1Sq2NvXRTXyVWsBLnE+b4IIPSCiR/DRdY6 UTMg==; darn=ilbers.de ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=TqKX+pn8; spf=pass (google.com: domain of fm-1212295-2025032206153659e06e379c321ff20c-vp4v_p@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-1212295-2025032206153659e06e379c321ff20c-Vp4v_P@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1742624141; x=1743228941; darn=ilbers.de; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=0TXHR1k2WNnQCyTEOL6CnUXKYtyjKSUYYPvKQr0cW44=; b=CfC6K8eD35fqPL1YhceYJoosbxyvXQSD7FqzC4vDnSS+sMZS3h0vbq/OYbtinbRblM NfCG/CGM5xgXWTFqCNEhnbV4htb7LzBu12/M9fERlpkBk34V/aMVNZpkgjYb0dxrKklU If7dAYvwG+9IBBh+N5IxLTGa5XGPW+ROWBCcsEejk/pOlb0cjaqNnKko+jp1KdoyDKY5 R7eFxaPQDfV8GK8QSHltc0A4eo17XM3UCKliAYdHZNwOcgEruj/iNycuajsjAVdUNR4J p5y0Yr5HuHB+Iew18y1AekgRF7zyxj8h6piMSSAmxs76lL5S8rM2VZqX2uM5pDxT+ggW 9A/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742624141; x=1743228941; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-beenthere:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=0TXHR1k2WNnQCyTEOL6CnUXKYtyjKSUYYPvKQr0cW44=; b=dUJAH4iRD6AtyyXivsaR2VmM2tsQvn8N07+z7rLuYuzG/umT1oHRQ4w5waDSIR+jA6 AucFQFf27gePsNtrXJ/Q8C5oO/ssTpoS4XpHJ5/6f6jlzOhXNLpKR5FhTmPduXHVk5bh GC28G+k28gBkuBeZP/k10m6dt5Ag4qaFOQgsT5YPjjK5EzsB9AZ0va1+VMfOlqnZO8xP 3eMEAyGAPpvSKUGlaJYEqKrirWHb1jXPcumT79FzkIA766IbeHjk4aVBuXAspL+gKEGf lXCB3bEIEX+iCDN+o3QjGquevhvD6NJ2CHu/jOgHnpKS0dd05IrXaI1ptANvXNRH2LI/ jr0g== X-Forwarded-Encrypted: i=2; AJvYcCX2ko59z6v4g5n0EJmbdk18YKRnMWaNYPscyDdq9Q4Pa7XZOS1gX+vjmHTSGHt7v4edVfRJ@ilbers.de X-Gm-Message-State: AOJu0YzEw7fp10SN8SLU5c2H9CnEo/S6Rmo2JE5FJrmlXaE2TDyriVsj c11usDY/LCzEo7Yz6r6l//SdKtPjZLl1DhA/eiu6ItTMXuFct3nO X-Google-Smtp-Source: AGHT+IG3zbJUbWJscgRgylx1sz5CdLFusx3Nggtn/xrRMi8fduGA0VUW4bhMRmE9y60lcrseh3k1oA== X-Received: by 2002:a05:6402:42c6:b0:5e5:9a2b:167a with SMTP id 4fb4d7f45d1cf-5ebcd468ed3mr5484722a12.17.1742624140115; Fri, 21 Mar 2025 23:15:40 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h=ARLLPAI1LAJbTtEbvEoM6BHldT3xKU8YPt/NNs6anEKr5A5siA== Received: by 2002:a05:6402:184b:b0:5e5:b9a7:2c27 with SMTP id 4fb4d7f45d1cf-5eb9c4bcc0els267152a12.0.-pod-prod-02-eu; Fri, 21 Mar 2025 23:15:37 -0700 (PDT) X-Received: by 2002:a05:6402:1d4e:b0:5e7:5b30:3c4f with SMTP id 4fb4d7f45d1cf-5ebcd433b34mr5166353a12.14.1742624136720; Fri, 21 Mar 2025 23:15:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1742624136; cv=none; d=google.com; s=arc-20240605; b=H0dTlpIs1STLo/J3XjKeAraN/5udHFbw/xOb9Yu5impTMfpBM8znXvO12WfXka0uFu AIpHeugizaeofGDDriQw6Tq+alyaO8yrZLvVroY3/rduEqg5GywmBnZuTtRMgV+Ck4bq qhm06nYgJnygSH8W28GApA5QQB3v+cLpX5FEHM4l8SqMJJ2YqCODLn1gISDYVSrWGGd5 HPDSxUWii+J3iimz3h9gU/tMb9uMDgf3ZmvrDVsFV0a+e7scWC/oCO8fff6IIThgWs8y VwZ3b79jJYoflU6hzjy38VxJnWHSELYNWZF5wd6Zhm+awVTE/N2yqZsjGlA4G38dqOaq 1ZCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=lLJio5i0tNngSt1Yjd7FH08MYQ8vTvR6B+69Gmw9R74=; fh=yuBAxkaMYSnaF7r+lbEZdWO7eG7htBqkQ2A7TcD4+Dg=; b=Vk+tcPCQ8POG7ykZr7RlHew0gsIrBfJjgQKr1qde5IKUEnkUsbJyU8ujUFhoYuT00w 2EJPcofGy3jgeaOyDXbwUZLmpzb12rh+4BUkDfCNQLoEX7LYiFLE+Abx+LWmKkYwjD3v 8eiKwaDiIzCIsoXT3Ccs7EgDINH2kom1+Sh6L98r6C+CqF+KytGC1SiODwpb/4uoXPp5 PTIFFMwjSso1MKbTMQl/PSLk2G09gXYCVVITpD8fBYJbyIb0nm2mxzW8JjKTQFfsrN/c ks1LimgPLpWUC2/wU2LyaW3kO0p8KhJunu7ShG/P9DnooLUGZBWoTHjSwrxjqMI/7aEv iM/A==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=TqKX+pn8; spf=pass (google.com: domain of fm-1212295-2025032206153659e06e379c321ff20c-vp4v_p@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-1212295-2025032206153659e06e379c321ff20c-Vp4v_P@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-228.siemens.flowmailer.net (mta-64-228.siemens.flowmailer.net. [185.136.64.228]) by gmr-mx.google.com with ESMTPS id 4fb4d7f45d1cf-5ebcce05169si30181a12.0.2025.03.21.23.15.36 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Mar 2025 23:15:36 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1212295-2025032206153659e06e379c321ff20c-vp4v_p@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) client-ip=185.136.64.228; Received: by mta-64-228.siemens.flowmailer.net with ESMTPSA id 2025032206153659e06e379c321ff20c for ; Sat, 22 Mar 2025 07:15:36 +0100 From: "'Cedric Hombourger' via isar-users" To: isar-users@googlegroups.com Cc: ubely@ilbers.de, Cedric Hombourger Subject: [PATCH v2 1/1] deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download Date: Sat, 22 Mar 2025 07:15:19 +0100 Message-Id: <20250322061519.2522459-2-cedric.hombourger@siemens.com> In-Reply-To: <20250322061519.2522459-1-cedric.hombourger@siemens.com> References: <20250305131142.2717692-1-cedric.hombourger@siemens.com> <20250322061519.2522459-1-cedric.hombourger@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1212295:519-21489:flowmailer X-Original-Sender: cedric.hombourger@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=TqKX+pn8; spf=pass (google.com: domain of fm-1212295-2025032206153659e06e379c321ff20c-vp4v_p@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-1212295-2025032206153659e06e379c321ff20c-Vp4v_P@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Cedric Hombourger Reply-To: Cedric Hombourger Content-Type: text/plain; charset="UTF-8" Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: bOf1Mle0Z5++ Several calls to dpkg-deb are made for each single .deb file found in downloads to parse individual fields. This approach is terribly slow when a large amount of .deb files are found. Use apt-ftparchive to produce an index of packages that were found and a simple awk script to produce a (sorted) list of source package names and their versions. "apt-cache dumpavail" is used so we only attempt to download source packages for versions that are known to apt. Also avoid using sed to remove Epoch from the version when we are trying to determine the name of the .dsc file: we instead use a simple POSIX parameter expansion to remove everything up to the first colon Signed-off-by: Cedric Hombourger --- meta/classes/deb-dl-dir.bbclass | 78 +++++++++++++++++++++------------ 1 file changed, 51 insertions(+), 27 deletions(-) diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass index 7ebd057e..75877750 100644 --- a/meta/classes/deb-dl-dir.bbclass +++ b/meta/classes/deb-dl-dir.bbclass @@ -5,23 +5,6 @@ inherit repository -is_not_part_of_current_build() { - local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )" - local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )" - local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )" - # Since we are parsing all the debs in DEBDIR, we can to some extend - # try to eliminate some debs that are not part of the current multiconfig - # build using the below method. - local output="$( grep -xhs ".* status installed ${package}:${arch} ${version}" \ - "${IMAGE_ROOTFS}"/var/log/dpkg.log \ - "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \ - "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \ - "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \ - "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log | head -1 )" - - [ -z "${output}" ] -} - debsrc_do_mounts() { sudo -s < + # Version: + # Source: () + # + # If Source is omitted, then = and + # if is not specified then it is . + # The awk script handles these optional fields. It looks for Size: as a + # trigger to print the source,version tupple + awk '/^Package:/ { s=$2; } + /^Version:/ { v=$2; next } + /^Source:/ { s=$2; if ($3 ~ /^\(/) v=substr($3, 2, length($3)-2) } + /^Size:/ { print s, v}' \ + | sort -u +} + debsrc_download() { export rootfs="$1" export rootfs_distro="$2" @@ -54,16 +55,39 @@ debsrc_download() { ( flock 9 set -e printenv | grep -q BB_VERBOSE_LOGS && set -x - find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do - is_not_part_of_current_build "${package}" && continue - local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )" - local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )" - local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')" - local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc") - [ -n "$dscfile" ] && continue - - sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \ - sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}" + + # We need temporary files for our lists of source packages + # trap exit of this sub-shell to remove them (this script may exit abruptly + # since "set -e" is used) + avail=$(mktemp) + wanted=$(mktemp) + trap "rm -f ${avail} ${wanted}" EXIT + + # List all packages known to apt + apt-cache -o Dir=${rootfs} dumpavail | debsrc_source_version_filter > ${avail} + + # Use apt-ftparchive to scan all .deb files found in the download directory + # and get the pairs that we wish to download + apt-ftparchive --md5=no --sha1=no --sha256=no --sha512=no \ + -a "${DISTRO_ARCH}" packages \ + "${rootfs}/var/cache/apt/archives" \ + | debsrc_source_version_filter > ${wanted} + + # We now have two sorted lists: source packages we want and those known to + # apt. We will only consider source packages that may be found in both. + comm -12 ${wanted} ${avail} \ + | while read src version; do + # Name of the .dsc file does not include Epoch, remove it before checking + # if sources were already downloaded. Avoid using sed here to reduce the + # number of processes being spawned by this function: we assume that the + # version is correctly formatted and simply strip everything up to the + # first colon + dscname="${src}_${version#*:}.dsc" + [ -f "${DEBSRCDIR}"/"${rootfs_distro}"/"${src}"/"${dscname}" ] || { + # use apt-get source to download sources in DEBSRCDIR + sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \ + sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}" + } done ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock" -- 2.39.5 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250322061519.2522459-2-cedric.hombourger%40siemens.com.