From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Wed, 05 Mar 2025 14:12:12 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-ot1-f62.google.com (mail-ot1-f62.google.com [209.85.210.62]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 525DCBdT016722 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 5 Mar 2025 14:12:12 +0100 Received: by mail-ot1-f62.google.com with SMTP id 46e09a7af769-7272acdfc39sf6053588a34.3 for ; Wed, 05 Mar 2025 05:12:11 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1741180325; cv=pass; d=google.com; s=arc-20240605; b=fqWeFrVLjdQ8SgRG53SDxK0NBBMXaEMK4iggewYm6GDlJYHSj5QaVapPrgpxU4xCtd Nq4EjRTFotTgCb4drUWJY933zn38RGD0MlVlTW+qB9Qh4lQOfcrubBFcnDLXixADEzVk CCv2aiZRVa7O2ZVdZNurDVMhDMU7ATJJtQZs8IMzx/WKoUiSg4fCRNxMr9Opy0Yk/6b0 j267uaxLI4HhvPjbhln4knNy+vS9fUqW8YHvMuCmni+EILreI60wAC31SXkdLnoV/yQf lPKvcqm0pODKEPQ2FwKXHNkwCuDMvbDDvk5XsLLtVoQpBa802uTHIX77L7C9LnkrndNd WZuA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:feedback-id:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=SqJHpdua24r6VIJ5pUeZm20iLMb9elVe0uNYLfTYn1I=; fh=w1Up8dfetCyLLhePJvFfJ52xh8PReZOKeGSPF4JUy6M=; b=B7JY91nMnYZD6HwhPmKHRrrf9/sdR/fHKMtGZT24Nxg9IxwKzkWPAwrBsIrJmVpWFk joCF151bG75bqBosd2F1+HgBjqOgpqIUVUjROlOSCte4FYYAv0qdXJcsN7rR5k2sNl7V J7uqz5UPLPG2CMq8CG/9fGbG/+55hl6t+vOfYv81CZOLZMqFdxW5NHDKcYXCJmyfTyWI pmQm0yqdTjJgW3ue+wZK3FifLuDLRN9pudHWNSLxrUQ3Hou1GwgaquBqJDikc86V0/Os WyDOBHoVmhjzMZd+bBwEl3aazUYmYTUpAdH3HzFC0A0VnUTkXloULrphXHGWeLusThAv +Xog==; darn=ilbers.de ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=LjFIfQuj; spf=pass (google.com: domain of fm-1212295-2025030513115927fdf11129f6fce058-n8ap0s@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-1212295-2025030513115927fdf11129f6fce058-n8aP0s@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1741180325; x=1741785125; darn=ilbers.de; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=SqJHpdua24r6VIJ5pUeZm20iLMb9elVe0uNYLfTYn1I=; b=MDzgNgTVEdB4bYQtOkesme6ApX2RAYAegI5ZjIRKk7OCpruuToHJxGb2fgvb2dyvh0 7JXqtf8/GGaZfDUmhB4bWy02aNutBt7Qci7zcuQt06YkYXUXzlXixMSpb1NB0vQtSLPT RDgdeaIjldqmod305JD0PhW9FNCdgkpRQxtDO2Byo5Am+5Zo50f/Wky/rsewJg3huNGQ aBH9uiIXn9b80rgAzZsFFOrvblFYlFbd7anEHY9g/U73w5YxJ4nR8DLP/c5Y8RGF0Tjf 7ABuW4t0dJnso54ECmQfYsQ60jg+KDelOOZBVkY9+Qq5r6M+4Xjv/DKtE1ND8kbBr7f2 TFSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741180325; x=1741785125; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:feedback-id :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SqJHpdua24r6VIJ5pUeZm20iLMb9elVe0uNYLfTYn1I=; b=blCBEG9P0Mf68mh4kmamlf4SS1ypDIYpPPEdOxOI1RwT7yCVZQmlnLUqymdfeCWc7e /sEUKJSfzvf09vuDiP76eCmnQoRY1b/KNfBVW0lVXHA76VOwA3N5yGSVLOE06sHH/oQE 7H5oMNFqjONaby9vzzyRQfXm3o0eJbuftd0fkZoynuHoHP5ES38rY2W89BKLjh6oddvJ B2+pgfVl8a0gk9nwpRNfBh5uTYbSG2zFBpZSC+901q7fKopXSNT73TCWPK+I1VxtD1du lUn/eSyodNDBxlaUVxxSXE/OiXZgG4D8Hxzft8RnJ/aRFpnsknGOgZslzljD/6Ux3zqB W1oA== X-Forwarded-Encrypted: i=2; AJvYcCWTa5bCal4AQDJ1UbyW9ocV7AT3Vl/Zekb/FsSn5CyCY/zWfDP6DVbMH3j+x0fXATNRnymH@ilbers.de X-Gm-Message-State: AOJu0Yx5gDwTfaSGJ8Qd6fR+d1Lr7DTfWKOldYvf5UHvetVTQXbk6Gl4 a+dByrhet+VkZz4VWTHDb2oFpQlmKMQUDns+0FecsM5JoAsfWtGP X-Google-Smtp-Source: AGHT+IErmgQFnQc8I0k8MOu+dJgrWKu864A+lQWjcn6v9X8MADLPQdQ87lkiuv2q9qxze1bPLGe4PQ== X-Received: by 2002:a05:6830:7004:b0:72a:b17:6570 with SMTP id 46e09a7af769-72a1fbf8e51mr1691682a34.11.1741180325350; Wed, 05 Mar 2025 05:12:05 -0800 (PST) X-BeenThere: isar-users@googlegroups.com; h=Adn5yVGsf0OsFXk/2Nx9HLg0Ujz9E0dIIGsQGGzOxVX8YHwxuw== Received: by 2002:a4a:c70d:0:b0:5fe:b6d6:7500 with SMTP id 006d021491bc7-5feb6d676b5ls1458134eaf.2.-pod-prod-07-us; Wed, 05 Mar 2025 05:12:04 -0800 (PST) X-Received: by 2002:a05:6808:2396:b0:3f6:6d5a:8df7 with SMTP id 5614622812f47-3f6831e27c9mr1684702b6e.26.1741180324264; Wed, 05 Mar 2025 05:12:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1741180324; cv=none; d=google.com; s=arc-20240605; b=gAtsUNYf9sbk/rOEHnFtfyOvcxgylMiGhfdGR8fq8dXZ1n5sVZczZdfnKrMniaNbok t7zSyENj7HRYrwueRmIQgfkMyxkv30exXXmhz0QlfZqB5mANBUFGhiJrEiYyHtAlQ1/e cmT+Ca3ueMY477NYcYpYiqxc2M/pwhGiP7+VcJTo1ogs0UJCxcMDZhK/c3RjIz+Ris3H rQ1njkJvg3Gph8FNkXgOESMSUO9gNC1WhtMNgNGOxstdRlatdoO9ypWfgWcEBtNvC45s rbNljqp1iv+ir1AFQqNJ/fH4TtWFn+aCSfwLYzDFmu6aclHxNqI5ivv6+VMeL740RBdk EN4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=sN+xxxbr8E441BajSuylkncXPAfli8OKo7mAO4Z4URs=; fh=yuBAxkaMYSnaF7r+lbEZdWO7eG7htBqkQ2A7TcD4+Dg=; b=d1uXRaxCJe9tw+XV/kSG0Q834g/YINwpEaZBafj03eAnT5JXLgVCIGyM+yVEsqWgnR n1BLFCyrWXqauSWiHKDUrogk+Ytp4LtI15tohcxwXGn+/w1qzl/FhDwFHa26B1YOobBO EQTGZL4tWQmf+Qfmf8KaK7eTDkSlyRAPp8T5WFxwhZ/q3MJmmwW+0m1Y7GNepyUtPECw XAPcpI469kf4aeb5xmrqfWeX98xcPN75FwYJRFaSs5z3pE/MfqGMGQyFZwra1QjKRMnC yFDTzcY9ZiQZZEKVRWRBfRnWKLEBksicMxMjtDB2NQfhv+9BKNi1eFRhZxGQTvXgb1aU AF7w==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=LjFIfQuj; spf=pass (google.com: domain of fm-1212295-2025030513115927fdf11129f6fce058-n8ap0s@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-1212295-2025030513115927fdf11129f6fce058-n8aP0s@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from mta-64-226.siemens.flowmailer.net (mta-64-226.siemens.flowmailer.net. [185.136.64.226]) by gmr-mx.google.com with ESMTPS id 5614622812f47-3f5506c4cdesi754398b6e.2.2025.03.05.05.12.03 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Mar 2025 05:12:04 -0800 (PST) Received-SPF: pass (google.com: domain of fm-1212295-2025030513115927fdf11129f6fce058-n8ap0s@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) client-ip=185.136.64.226; Received: by mta-64-226.siemens.flowmailer.net with ESMTPSA id 2025030513115927fdf11129f6fce058 for ; Wed, 05 Mar 2025 14:11:59 +0100 From: "'Cedric Hombourger' via isar-users" To: isar-users@googlegroups.com Cc: ubely@ilbers.de, Cedric Hombourger Subject: [PATCH] deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download Date: Wed, 5 Mar 2025 14:11:42 +0100 Message-Id: <20250305131142.2717692-1-cedric.hombourger@siemens.com> MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1212295:519-21489:flowmailer X-Original-Sender: cedric.hombourger@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=LjFIfQuj; spf=pass (google.com: domain of fm-1212295-2025030513115927fdf11129f6fce058-n8ap0s@rts-flowmailer.siemens.com designates 185.136.64.226 as permitted sender) smtp.mailfrom=fm-1212295-2025030513115927fdf11129f6fce058-n8aP0s@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Cedric Hombourger Reply-To: Cedric Hombourger Content-Type: text/plain; charset="UTF-8" Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Status: No, score=-4.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: w5KHSy4qcVG5 Several calls to dpkg-deb are made for each single .deb file found in downloads to parse individual fields. This approach is terribly slow when a large amount of .deb files are found. Use apt-ftparchive to produce an index of packages that were found and a simple awk script to produce a (sorted) list of source package names and their versions. Also avoid using sed to remove Epoch from the version when we are trying to determine the name of the .dsc file: we instead use a simple POSIX parameter expansion to remove everything up to the first colon Signed-off-by: Cedric Hombourger --- meta/classes/deb-dl-dir.bbclass | 62 +++++++++++++++++++-------------- 1 file changed, 35 insertions(+), 27 deletions(-) diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass index 7ebd057e..53ce4538 100644 --- a/meta/classes/deb-dl-dir.bbclass +++ b/meta/classes/deb-dl-dir.bbclass @@ -5,23 +5,6 @@ inherit repository -is_not_part_of_current_build() { - local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )" - local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )" - local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )" - # Since we are parsing all the debs in DEBDIR, we can to some extend - # try to eliminate some debs that are not part of the current multiconfig - # build using the below method. - local output="$( grep -xhs ".* status installed ${package}:${arch} ${version}" \ - "${IMAGE_ROOTFS}"/var/log/dpkg.log \ - "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \ - "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \ - "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \ - "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log | head -1 )" - - [ -z "${output}" ] -} - debsrc_do_mounts() { sudo -s < + # Version: + # Source: () + # + # If Source is omitted, then = and + # if is not specified then it is . + # The awk script handles these optional fields. It looks for Size: as a + # trigger to print the source,version tupple + + apt-ftparchive --md5=no --sha1=no --sha256=no --sha512=no \ + -a "${DISTRO_ARCH}" packages \ + "${rootfs}/var/cache/apt/archives" \ + | awk '/^Package:/ { s=$2; } + /^Version:/ { v=$2; next } + /^Source:/ { s=$2; if ($3 ~ /^\(/) v=substr($3, 2, length($3)-2) } + /^Size:/ { print s, v}' \ + | sort -u \ + | while read src version; do + # Name of the .dsc file does not include Epoch, remove it before checking + # if sources were already downloaded. Avoid using sed here to reduce the + # number of processes being spawned by this function: we assume that the + # version is correctly formatted and simply strip everything up to the + # first colon + dscname="${src}_${version#*:}.dsc" + [ -f "${DEBSRCDIR}"/"${rootfs_distro}"/"${src}"/"${dscname}" ] || { + # use apt-get source to download sources in DEBSRCDIR + sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \ + sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}" + } done ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock" -- 2.39.5 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250305131142.2717692-1-cedric.hombourger%40siemens.com.