* Re: [PATCH] deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download
2025-03-05 13:11 [PATCH] deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download 'Cedric Hombourger' via isar-users
@ 2025-03-05 13:57 ` 'Jan Kiszka' via isar-users
2025-03-05 15:08 ` 'cedric.hombourger@siemens.com' via isar-users
2025-03-05 17:22 ` 'Niedermayr, BENEDIKT' via isar-users
` (2 subsequent siblings)
3 siblings, 1 reply; 9+ messages in thread
From: 'Jan Kiszka' via isar-users @ 2025-03-05 13:57 UTC (permalink / raw)
To: Cedric Hombourger, isar-users; +Cc: ubely
On 05.03.25 14:11, 'Cedric Hombourger' via isar-users wrote:
> Several calls to dpkg-deb are made for each single .deb file found in
> downloads to parse individual fields. This approach is terribly slow
> when a large amount of .deb files are found. Use apt-ftparchive to
Out of curiosity: What is roughly the amount of packages where this
inefficiency becomes visible?
> produce an index of packages that were found and a simple awk script
> to produce a (sorted) list of source package names and their versions.
> Also avoid using sed to remove Epoch from the version when we are
> trying to determine the name of the .dsc file: we instead use a simple
> POSIX parameter expansion to remove everything up to the first colon
>
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
> meta/classes/deb-dl-dir.bbclass | 62 +++++++++++++++++++--------------
> 1 file changed, 35 insertions(+), 27 deletions(-)
>
> diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
> index 7ebd057e..53ce4538 100644
> --- a/meta/classes/deb-dl-dir.bbclass
> +++ b/meta/classes/deb-dl-dir.bbclass
> @@ -5,23 +5,6 @@
>
> inherit repository
>
> -is_not_part_of_current_build() {
> - local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"
> - local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )"
> - local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )"
> - # Since we are parsing all the debs in DEBDIR, we can to some extend
> - # try to eliminate some debs that are not part of the current multiconfig
> - # build using the below method.
> - local output="$( grep -xhs ".* status installed ${package}:${arch} ${version}" \
> - "${IMAGE_ROOTFS}"/var/log/dpkg.log \
> - "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \
> - "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \
> - "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \
> - "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log | head -1 )"
> -
> - [ -z "${output}" ]
> -}
> -
> debsrc_do_mounts() {
> sudo -s <<EOSUDO
> set -e
> @@ -54,16 +37,41 @@ debsrc_download() {
> ( flock 9
> set -e
> printenv | grep -q BB_VERBOSE_LOGS && set -x
> - find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do
> - is_not_part_of_current_build "${package}" && continue
> - local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
> - local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
> - local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')"
> - local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc")
> - [ -n "$dscfile" ] && continue
> -
> - sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
> - sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
> +
> + # Use apt-ftparchive to scan all .deb files found in the download directory
> + # and produce an index that we can "parse" with awk. This is much faster
> + # than parsing each .deb file individually using dpkg-deb. Lines from the
> + # index we need are:
> + #
> + # Package: <binary-name>
> + # Version: <binary-version>
> + # Source: <source-name> (<source-version>)
> + #
> + # If Source is omitted, then <source-name>=<binary-name> and
> + # if <source-version> is not specified then it is <binary-version>.
> + # The awk script handles these optional fields. It looks for Size: as a
> + # trigger to print the source,version tupple
> +
> + apt-ftparchive --md5=no --sha1=no --sha256=no --sha512=no \
> + -a "${DISTRO_ARCH}" packages \
> + "${rootfs}/var/cache/apt/archives" \
> + | awk '/^Package:/ { s=$2; }
> + /^Version:/ { v=$2; next }
> + /^Source:/ { s=$2; if ($3 ~ /^\(/) v=substr($3, 2, length($3)-2) }
> + /^Size:/ { print s, v}' \
> + | sort -u \
> + | while read src version; do
> + # Name of the .dsc file does not include Epoch, remove it before checking
> + # if sources were already downloaded. Avoid using sed here to reduce the
> + # number of processes being spawned by this function: we assume that the
> + # version is correctly formatted and simply strip everything up to the
> + # first colon
> + dscname="${src}_${version#*:}.dsc"
> + [ -f "${DEBSRCDIR}"/"${rootfs_distro}"/"${src}"/"${dscname}" ] || {
> + # use apt-get source to download sources in DEBSRCDIR
> + sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
> + sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
Can we rewrap this horribly long line at that chance?
> + }
> done
> ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
>
Did you also consider using a python function for the content
processing? I'm not predicting that this will be faster or nicer or
whatever, just wondering if it might be while reading the above.
Jan
--
Siemens AG, Foundational Technologies
Linux Expert Center
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/7260a572-4302-4f8c-9d50-21286451b155%40siemens.com.
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH] deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download
2025-03-05 13:57 ` 'Jan Kiszka' via isar-users
@ 2025-03-05 15:08 ` 'cedric.hombourger@siemens.com' via isar-users
0 siblings, 0 replies; 9+ messages in thread
From: 'cedric.hombourger@siemens.com' via isar-users @ 2025-03-05 15:08 UTC (permalink / raw)
To: isar-users, Kiszka, Jan; +Cc: ubely
On Wed, 2025-03-05 at 14:57 +0100, Jan Kiszka wrote:
> On 05.03.25 14:11, 'Cedric Hombourger' via isar-users wrote:
> > Several calls to dpkg-deb are made for each single .deb file found
> > in
> > downloads to parse individual fields. This approach is terribly
> > slow
> > when a large amount of .deb files are found. Use apt-ftparchive to
>
> Out of curiosity: What is roughly the amount of packages where this
> inefficiency becomes visible?
That's a great question and I may not have a great answer though. I am
pleeding guilty for (1) sharing my downloads folder between builds and
(2) doing multiconfig builds. I therefore have ~4.9k .deb packages in
my downloads folder.
I would expect people start experiencing the cost of deb-src caching
with a few hundreds only. The current implementation was calling dpkg-
deb 5 times for each package + 1 call to sed for each + 1 call to find
to traverse the deb-src tree to check if the .dsc we want to download
happens to be already there.
>
> > produce an index of packages that were found and a simple awk
> > script
> > to produce a (sorted) list of source package names and their
> > versions.
> > Also avoid using sed to remove Epoch from the version when we are
> > trying to determine the name of the .dsc file: we instead use a
> > simple
> > POSIX parameter expansion to remove everything up to the first
> > colon
> >
> > Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> > ---
> > meta/classes/deb-dl-dir.bbclass | 62 +++++++++++++++++++----------
> > ----
> > 1 file changed, 35 insertions(+), 27 deletions(-)
> >
> > diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-
> > dir.bbclass
> > index 7ebd057e..53ce4538 100644
> > --- a/meta/classes/deb-dl-dir.bbclass
> > +++ b/meta/classes/deb-dl-dir.bbclass
> > @@ -5,23 +5,6 @@
> >
> > inherit repository
> >
> > -is_not_part_of_current_build() {
> > - local package="$( dpkg-deb --show --showformat '${Package}'
> > "${1}" )"
> > - local arch="$( dpkg-deb --show --showformat '${Architecture}'
> > "${1}" )"
> > - local version="$( dpkg-deb --show --showformat '${Version}'
> > "${1}" )"
> > - # Since we are parsing all the debs in DEBDIR, we can to some
> > extend
> > - # try to eliminate some debs that are not part of the current
> > multiconfig
> > - # build using the below method.
> > - local output="$( grep -xhs ".* status installed
> > ${package}:${arch} ${version}" \
> > - "${IMAGE_ROOTFS}"/var/log/dpkg.log \
> > - "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \
> > - "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \
> > - "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \
> > - "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log | head -1
> > )"
> > -
> > - [ -z "${output}" ]
> > -}
> > -
> > debsrc_do_mounts() {
> > sudo -s <<EOSUDO
> > set -e
> > @@ -54,16 +37,41 @@ debsrc_download() {
> > ( flock 9
> > set -e
> > printenv | grep -q BB_VERBOSE_LOGS && set -x
> > - find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -
> > iname '*\.deb' | while read package; do
> > - is_not_part_of_current_build "${package}" && continue
> > - local src="$( dpkg-deb --show --showformat
> > '${source:Package}' "${package}" )"
> > - local version="$( dpkg-deb --show --showformat
> > '${source:Version}' "${package}" )"
> > - local dscname="$(echo ${src}_${version} | sed -e 's/_[0-
> > 9]\+:/_/')"
> > - local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -
> > name "${dscname}.dsc")
> > - [ -n "$dscfile" ] && continue
> > -
> > - sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs}
> > \
> > - sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-
> > src/${1}/${2}" && apt-get -y --download-only --only-source source
> > "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
> > +
> > + # Use apt-ftparchive to scan all .deb files found in the
> > download directory
> > + # and produce an index that we can "parse" with awk. This is
> > much faster
> > + # than parsing each .deb file individually using dpkg-deb.
> > Lines from the
> > + # index we need are:
> > + #
> > + # Package: <binary-name>
> > + # Version: <binary-version>
> > + # Source: <source-name> (<source-version>)
> > + #
> > + # If Source is omitted, then <source-name>=<binary-name> and
> > + # if <source-version> is not specified then it is <binary-
> > version>.
> > + # The awk script handles these optional fields. It looks for
> > Size: as a
> > + # trigger to print the source,version tupple
> > +
> > + apt-ftparchive --md5=no --sha1=no --sha256=no --sha512=no \
> > + -a "${DISTRO_ARCH}" packages \
> > + "${rootfs}/var/cache/apt/archives" \
> > + | awk '/^Package:/ { s=$2; }
> > + /^Version:/ { v=$2; next }
> > + /^Source:/ { s=$2; if ($3 ~ /^\(/) v=substr($3, 2,
> > length($3)-2) }
> > + /^Size:/ { print s, v}' \
> > + | sort -u \
> > + | while read src version; do
> > + # Name of the .dsc file does not include Epoch, remove it
> > before checking
> > + # if sources were already downloaded. Avoid using sed here
> > to reduce the
> > + # number of processes being spawned by this function: we
> > assume that the
> > + # version is correctly formatted and simply strip
> > everything up to the
> > + # first colon
> > + dscname="${src}_${version#*:}.dsc"
> > + [ -f
> > "${DEBSRCDIR}"/"${rootfs_distro}"/"${src}"/"${dscname}" ] || {
> > + # use apt-get source to download sources in DEBSRCDIR
> > + sudo -E chroot --userspec=$( id -u ):$( id -g )
> > ${rootfs} \
> > + sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-
> > src/${1}/${2}" && apt-get -y --download-only --only-source source
> > "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
>
> Can we rewrap this horribly long line at that chance?
>
> > + }
> > done
> > ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
> >
>
> Did you also consider using a python function for the content
> processing? I'm not predicting that this will be faster or nicer or
> whatever, just wondering if it might be while reading the above.
>
> Jan
>
--
Cedric Hombourger
Siemens AG
www.siemens.com
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/0796523902045f26422853a246bd162bb04414fc.camel%40siemens.com.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download
2025-03-05 13:11 [PATCH] deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download 'Cedric Hombourger' via isar-users
2025-03-05 13:57 ` 'Jan Kiszka' via isar-users
@ 2025-03-05 17:22 ` 'Niedermayr, BENEDIKT' via isar-users
2025-03-05 17:24 ` 'Niedermayr, BENEDIKT' via isar-users
2025-03-10 11:06 ` Srinuvasan Arjunan
2025-03-22 6:15 ` [PATCH v2 0/1] " 'Cedric Hombourger' via isar-users
3 siblings, 1 reply; 9+ messages in thread
From: 'Niedermayr, BENEDIKT' via isar-users @ 2025-03-05 17:22 UTC (permalink / raw)
To: cedric.hombourger, isar-users; +Cc: ubely
On 05.03.25 14:11, 'Cedric Hombourger' via isar-users wrote:
> Several calls to dpkg-deb are made for each single .deb file found in
> downloads to parse individual fields. This approach is terribly slow
> when a large amount of .deb files are found. Use apt-ftparchive to > produce an index of packages that were found and a simple awk script
> to produce a (sorted) list of source package names and their versions.
> Also avoid using sed to remove Epoch from the version when we are
> trying to determine the name of the .dsc file: we instead use a simple
> POSIX parameter expansion to remove everything up to the first colon
>
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
> meta/classes/deb-dl-dir.bbclass | 62 +++++++++++++++++++--------------
> 1 file changed, 35 insertions(+), 27 deletions(-)
>
> diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
> index 7ebd057e..53ce4538 100644
> --- a/meta/classes/deb-dl-dir.bbclass
> +++ b/meta/classes/deb-dl-dir.bbclass
> @@ -5,23 +5,6 @@
>
> inherit repository
>
> -is_not_part_of_current_build() {
> - local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"
> - local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )"
> - local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )"
> - # Since we are parsing all the debs in DEBDIR, we can to some extend
> - # try to eliminate some debs that are not part of the current multiconfig
> - # build using the below method.
> - local output="$( grep -xhs ".* status installed ${package}:${arch} ${version}" \
> - "${IMAGE_ROOTFS}"/var/log/dpkg.log \
> - "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \
> - "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \
> - "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \
> - "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log | head -1 )"
> -
> - [ -z "${output}" ]
> -}
> -
> debsrc_do_mounts() {
> sudo -s <<EOSUDO
> set -e
> @@ -54,16 +37,41 @@ debsrc_download() {
> ( flock 9
> set -e
> printenv | grep -q BB_VERBOSE_LOGS && set -x
> - find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do
> - is_not_part_of_current_build "${package}" && continue
> - local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
> - local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
> - local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')"
> - local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc")
> - [ -n "$dscfile" ] && continue
> -
> - sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
> - sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
Maybe a pointer to my previous patch [1] which addresses this as well
but with a different motivation. Your patch would also fix a regression
that has been introduced with mmdebstrap.
At least my patch is causing problems.
[1] https://groups.google.com/g/isar-users/c/IeORW6eiTxI
> +
> + # Use apt-ftparchive to scan all .deb files found in the download directory
> + # and produce an index that we can "parse" with awk. This is much faster
> + # than parsing each .deb file individually using dpkg-deb. Lines from the
> + # index we need are:
> + #
> + # Package: <binary-name>
> + # Version: <binary-version>
> + # Source: <source-name> (<source-version>)
> + #
> + # If Source is omitted, then <source-name>=<binary-name> and
> + # if <source-version> is not specified then it is <binary-version>.
> + # The awk script handles these optional fields. It looks for Size: as a
> + # trigger to print the source,version tupple
> +
> + apt-ftparchive --md5=no --sha1=no --sha256=no --sha512=no \
> + -a "${DISTRO_ARCH}" packages \
> + "${rootfs}/var/cache/apt/archives" \
> + | awk '/^Package:/ { s=$2; }
> + /^Version:/ { v=$2; next }
> + /^Source:/ { s=$2; if ($3 ~ /^\(/) v=substr($3, 2, length($3)-2) }
> + /^Size:/ { print s, v}' \
> + | sort -u \
> + | while read src version; do
> + # Name of the .dsc file does not include Epoch, remove it before checking
> + # if sources were already downloaded. Avoid using sed here to reduce the
> + # number of processes being spawned by this function: we assume that the
> + # version is correctly formatted and simply strip everything up to the
> + # first colon
> + dscname="${src}_${version#*:}.dsc"
> + [ -f "${DEBSRCDIR}"/"${rootfs_distro}"/"${src}"/"${dscname}" ] || {
> + # use apt-get source to download sources in DEBSRCDIR
> + sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
> + sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
> + }
> done
> ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
>
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/7b3cd7c5-0575-4e5a-9ce9-1c0e89a47e83%40siemens.com.
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH] deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download
2025-03-05 17:22 ` 'Niedermayr, BENEDIKT' via isar-users
@ 2025-03-05 17:24 ` 'Niedermayr, BENEDIKT' via isar-users
0 siblings, 0 replies; 9+ messages in thread
From: 'Niedermayr, BENEDIKT' via isar-users @ 2025-03-05 17:24 UTC (permalink / raw)
To: cedric.hombourger, isar-users; +Cc: ubely
On 05.03.25 18:22, 'Niedermayr, BENEDIKT' via isar-users wrote:
> On 05.03.25 14:11, 'Cedric Hombourger' via isar-users wrote:
>> Several calls to dpkg-deb are made for each single .deb file found in
>> downloads to parse individual fields. This approach is terribly slow
>> when a large amount of .deb files are found. Use apt-ftparchive to > produce an index of packages that were found and a simple awk script
>> to produce a (sorted) list of source package names and their versions.
>> Also avoid using sed to remove Epoch from the version when we are
>> trying to determine the name of the .dsc file: we instead use a simple
>> POSIX parameter expansion to remove everything up to the first colon
>>
>> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
>> ---
>> meta/classes/deb-dl-dir.bbclass | 62 +++++++++++++++++++--------------
>> 1 file changed, 35 insertions(+), 27 deletions(-)
>>
>> diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
>> index 7ebd057e..53ce4538 100644
>> --- a/meta/classes/deb-dl-dir.bbclass
>> +++ b/meta/classes/deb-dl-dir.bbclass
>> @@ -5,23 +5,6 @@
>>
>> inherit repository
>>
>> -is_not_part_of_current_build() {
>> - local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"
>> - local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )"
>> - local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )"
>> - # Since we are parsing all the debs in DEBDIR, we can to some extend
>> - # try to eliminate some debs that are not part of the current multiconfig
>> - # build using the below method.
>> - local output="$( grep -xhs ".* status installed ${package}:${arch} ${version}" \
>> - "${IMAGE_ROOTFS}"/var/log/dpkg.log \
>> - "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \
>> - "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \
>> - "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \
>> - "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log | head -1 )"
>> -
>> - [ -z "${output}" ]
>> -}
>> -
>> debsrc_do_mounts() {
>> sudo -s <<EOSUDO
>> set -e
>> @@ -54,16 +37,41 @@ debsrc_download() {
>> ( flock 9
>> set -e
>> printenv | grep -q BB_VERBOSE_LOGS && set -x
>> - find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do
>> - is_not_part_of_current_build "${package}" && continue
>> - local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
>> - local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
>> - local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')"
>> - local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc")
>> - [ -n "$dscfile" ] && continue
>> -
>> - sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
>> - sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
> Maybe a pointer to my previous patch [1] which addresses this as well
> but with a different motivation. Your patch would also fix a regression
> that has been introduced with mmdebstrap.
>
> At least my patch is causing problems.
>
> [1] https://groups.google.com/g/isar-users/c/IeORW6eiTxI
Ok just saw this [1], so you might be already aware of it.
[1] https://groups.google.com/g/isar-users/c/8QstIaudyts
Regards,
Benedikt
>
>> +
>> + # Use apt-ftparchive to scan all .deb files found in the download directory
>> + # and produce an index that we can "parse" with awk. This is much faster
>> + # than parsing each .deb file individually using dpkg-deb. Lines from the
>> + # index we need are:
>> + #
>> + # Package: <binary-name>
>> + # Version: <binary-version>
>> + # Source: <source-name> (<source-version>)
>> + #
>> + # If Source is omitted, then <source-name>=<binary-name> and
>> + # if <source-version> is not specified then it is <binary-version>.
>> + # The awk script handles these optional fields. It looks for Size: as a
>> + # trigger to print the source,version tupple
>> +
>> + apt-ftparchive --md5=no --sha1=no --sha256=no --sha512=no \
>> + -a "${DISTRO_ARCH}" packages \
>> + "${rootfs}/var/cache/apt/archives" \
>> + | awk '/^Package:/ { s=$2; }
>> + /^Version:/ { v=$2; next }
>> + /^Source:/ { s=$2; if ($3 ~ /^\(/) v=substr($3, 2, length($3)-2) }
>> + /^Size:/ { print s, v}' \
>> + | sort -u \
>> + | while read src version; do
>> + # Name of the .dsc file does not include Epoch, remove it before checking
>> + # if sources were already downloaded. Avoid using sed here to reduce the
>> + # number of processes being spawned by this function: we assume that the
>> + # version is correctly formatted and simply strip everything up to the
>> + # first colon
>> + dscname="${src}_${version#*:}.dsc"
>> + [ -f "${DEBSRCDIR}"/"${rootfs_distro}"/"${src}"/"${dscname}" ] || {
>> + # use apt-get source to download sources in DEBSRCDIR
>> + sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
>> + sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
>> + }
>> done
>> ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
>>
>
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/e87858d6-d68b-4c65-9e0a-deaf2b6bb2d7%40siemens.com.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download
2025-03-05 13:11 [PATCH] deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download 'Cedric Hombourger' via isar-users
2025-03-05 13:57 ` 'Jan Kiszka' via isar-users
2025-03-05 17:22 ` 'Niedermayr, BENEDIKT' via isar-users
@ 2025-03-10 11:06 ` Srinuvasan Arjunan
2025-03-22 6:15 ` [PATCH v2 0/1] " 'Cedric Hombourger' via isar-users
3 siblings, 0 replies; 9+ messages in thread
From: Srinuvasan Arjunan @ 2025-03-10 11:06 UTC (permalink / raw)
To: isar-users
[-- Attachment #1.1: Type: text/plain, Size: 6153 bytes --]
On Wednesday, March 5, 2025 at 6:42:05 PM UTC+5:30 Cedric Hombourger wrote:
Several calls to dpkg-deb are made for each single .deb file found in
downloads to parse individual fields. This approach is terribly slow
when a large amount of .deb files are found. Use apt-ftparchive to
produce an index of packages that were found and a simple awk script
to produce a (sorted) list of source package names and their versions.
Also avoid using sed to remove Epoch from the version when we are
trying to determine the name of the .dsc file: we instead use a simple
POSIX parameter expansion to remove everything up to the first colon
Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
---
meta/classes/deb-dl-dir.bbclass | 62 +++++++++++++++++++--------------
1 file changed, 35 insertions(+), 27 deletions(-)
diff --git a/meta/classes/deb-dl-dir.bbclass
b/meta/classes/deb-dl-dir.bbclass
index 7ebd057e..53ce4538 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -5,23 +5,6 @@
inherit repository
-is_not_part_of_current_build() {
- local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"
- local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )"
- local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )"
- # Since we are parsing all the debs in DEBDIR, we can to some extend
- # try to eliminate some debs that are not part of the current multiconfig
- # build using the below method.
- local output="$( grep -xhs ".* status installed ${package}:${arch}
${version}" \
- "${IMAGE_ROOTFS}"/var/log/dpkg.log \
- "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \
- "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \
- "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \
- "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log | head -1 )"
-
- [ -z "${output}" ]
-}
-
debsrc_do_mounts() {
sudo -s <<EOSUDO
set -e
@@ -54,16 +37,41 @@ debsrc_download() {
( flock 9
set -e
printenv | grep -q BB_VERBOSE_LOGS && set -x
- find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname
'*\.deb' | while read package; do
- is_not_part_of_current_build "${package}" && continue
- local src="$( dpkg-deb --show --showformat '${source:Package}'
"${package}" )"
- local version="$( dpkg-deb --show --showformat '${source:Version}'
"${package}" )"
- local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')"
- local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name
"${dscname}.dsc")
- [ -n "$dscfile" ] && continue
-
- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
- sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" &&
apt-get -y --download-only --only-source source "$2"="$3" ' download-src
"${rootfs_distro}" "${src}" "${version}"
+
+ # Use apt-ftparchive to scan all .deb files found in the download
directory
+ # and produce an index that we can "parse" with awk. This is much faster
+ # than parsing each .deb file individually using dpkg-deb. Lines from the
+ # index we need are:
+ #
+ # Package: <binary-name>
+ # Version: <binary-version>
+ # Source: <source-name> (<source-version>)
+ #
+ # If Source is omitted, then <source-name>=<binary-name> and
+ # if <source-version> is not specified then it is <binary-version>.
+ # The awk script handles these optional fields. It looks for Size: as a
+ # trigger to print the source,version tupple
+
+ apt-ftparchive --md5=no --sha1=no --sha256=no --sha512=no \
+ -a "${DISTRO_ARCH}" packages \
Hi Cedric,
I took this patch for my deb-src-caching issue [1], now i can able to
download deb-src for bootstrap and image related packages
only missing part is imager_install related packages, going to send the
patches based on your patch.
But here i found one issue for armfh arch base-apt builds in ISAR, the
help2man and texinfo deb-src packages are missing
because when we take the index using apt-ftparchive --md5=no --sha1=no
--sha256=no --sha512=no -a "${DISTRO_ARCH}"
we uses the -a ${DISTRO_ARCH}, in this case it is armfh, but help2man and
texinfo packages are only available for amd64 arch (might
be ISAR_CROSS_COMPILE configuration) not armhf, hence the index doesn't
have those packages , due to this reason we are not able to
download src packages for those packages.
I would suggest we can remove -a "${DISTRO_ARCH}" option and anyhow we
are getting final list with sort -u.
Validated without -a option and it's working fine as expected.
[1]: https://groups.google.com/g/isar-users/c/8QstIaudyts
Please provide your thoughts?
+ "${rootfs}/var/cache/apt/archives" \
+ | awk '/^Package:/ { s=$2; }
+ /^Version:/ { v=$2; next }
+ /^Source:/ { s=$2; if ($3 ~ /^\(/) v=substr($3, 2, length($3)-2) }
+ /^Size:/ { print s, v}' \
+ | sort -u \
+ | while read src version; do
+ # Name of the .dsc file does not include Epoch, remove it before checking
+ # if sources were already downloaded. Avoid using sed here to reduce the
+ # number of processes being spawned by this function: we assume that the
+ # version is correctly formatted and simply strip everything up to the
+ # first colon
+ dscname="${src}_${version#*:}.dsc"
+ [ -f "${DEBSRCDIR}"/"${rootfs_distro}"/"${src}"/"${dscname}" ] || {
+ # use apt-get source to download sources in DEBSRCDIR
+ sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
+ sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" &&
apt-get -y --download-only --only-source source "$2"="$3" ' download-src
"${rootfs_distro}" "${src}" "${version}"
+ }
done
) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/671d33d8-1a0f-402d-8b6a-f8c56d6c3e30n%40googlegroups.com.
[-- Attachment #1.2: Type: text/html, Size: 7922 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 0/1] deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download
2025-03-05 13:11 [PATCH] deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download 'Cedric Hombourger' via isar-users
` (2 preceding siblings ...)
2025-03-10 11:06 ` Srinuvasan Arjunan
@ 2025-03-22 6:15 ` 'Cedric Hombourger' via isar-users
2025-03-22 6:15 ` [PATCH v2 1/1] " 'Cedric Hombourger' via isar-users
2025-03-27 10:34 ` [PATCH v2 0/1] " Uladzimir Bely
3 siblings, 2 replies; 9+ messages in thread
From: 'Cedric Hombourger' via isar-users @ 2025-03-22 6:15 UTC (permalink / raw)
To: isar-users; +Cc: ubely, Cedric Hombourger
Changes since v1:
* v1 had is_not_part_of_current_build() removed. It turns out that
we better check what .deb files get imported in the apt cache
of our rootfs to make sure that we only try to fetch sources for
packages we know. This is now achieved with apt-cache dumpavail
to obtain a list of known source packages.
Cedric Hombourger (1):
deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download
meta/classes/deb-dl-dir.bbclass | 78 +++++++++++++++++++++------------
1 file changed, 51 insertions(+), 27 deletions(-)
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250322061519.2522459-1-cedric.hombourger%40siemens.com.
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 1/1] deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download
2025-03-22 6:15 ` [PATCH v2 0/1] " 'Cedric Hombourger' via isar-users
@ 2025-03-22 6:15 ` 'Cedric Hombourger' via isar-users
2025-03-27 10:34 ` [PATCH v2 0/1] " Uladzimir Bely
1 sibling, 0 replies; 9+ messages in thread
From: 'Cedric Hombourger' via isar-users @ 2025-03-22 6:15 UTC (permalink / raw)
To: isar-users; +Cc: ubely, Cedric Hombourger
Several calls to dpkg-deb are made for each single .deb file found in
downloads to parse individual fields. This approach is terribly slow
when a large amount of .deb files are found. Use apt-ftparchive to
produce an index of packages that were found and a simple awk script
to produce a (sorted) list of source package names and their versions.
"apt-cache dumpavail" is used so we only attempt to download source
packages for versions that are known to apt.
Also avoid using sed to remove Epoch from the version when we are
trying to determine the name of the .dsc file: we instead use a simple
POSIX parameter expansion to remove everything up to the first colon
Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
meta/classes/deb-dl-dir.bbclass | 78 +++++++++++++++++++++------------
1 file changed, 51 insertions(+), 27 deletions(-)
diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 7ebd057e..75877750 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -5,23 +5,6 @@
inherit repository
-is_not_part_of_current_build() {
- local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"
- local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )"
- local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )"
- # Since we are parsing all the debs in DEBDIR, we can to some extend
- # try to eliminate some debs that are not part of the current multiconfig
- # build using the below method.
- local output="$( grep -xhs ".* status installed ${package}:${arch} ${version}" \
- "${IMAGE_ROOTFS}"/var/log/dpkg.log \
- "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \
- "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \
- "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \
- "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log | head -1 )"
-
- [ -z "${output}" ]
-}
-
debsrc_do_mounts() {
sudo -s <<EOSUDO
set -e
@@ -41,6 +24,24 @@ debsrc_undo_mounts() {
EOSUDO
}
+debsrc_source_version_filter() {
+ # Filter the input to only consider Package, Version and Source lines
+ #
+ # Package: <binary-name>
+ # Version: <binary-version>
+ # Source: <source-name> (<source-version>)
+ #
+ # If Source is omitted, then <source-name>=<binary-name> and
+ # if <source-version> is not specified then it is <binary-version>.
+ # The awk script handles these optional fields. It looks for Size: as a
+ # trigger to print the source,version tupple
+ awk '/^Package:/ { s=$2; }
+ /^Version:/ { v=$2; next }
+ /^Source:/ { s=$2; if ($3 ~ /^\(/) v=substr($3, 2, length($3)-2) }
+ /^Size:/ { print s, v}' \
+ | sort -u
+}
+
debsrc_download() {
export rootfs="$1"
export rootfs_distro="$2"
@@ -54,16 +55,39 @@ debsrc_download() {
( flock 9
set -e
printenv | grep -q BB_VERBOSE_LOGS && set -x
- find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do
- is_not_part_of_current_build "${package}" && continue
- local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
- local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
- local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')"
- local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc")
- [ -n "$dscfile" ] && continue
-
- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
- sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
+
+ # We need temporary files for our lists of source packages
+ # trap exit of this sub-shell to remove them (this script may exit abruptly
+ # since "set -e" is used)
+ avail=$(mktemp)
+ wanted=$(mktemp)
+ trap "rm -f ${avail} ${wanted}" EXIT
+
+ # List all packages known to apt
+ apt-cache -o Dir=${rootfs} dumpavail | debsrc_source_version_filter > ${avail}
+
+ # Use apt-ftparchive to scan all .deb files found in the download directory
+ # and get the <source> <version> pairs that we wish to download
+ apt-ftparchive --md5=no --sha1=no --sha256=no --sha512=no \
+ -a "${DISTRO_ARCH}" packages \
+ "${rootfs}/var/cache/apt/archives" \
+ | debsrc_source_version_filter > ${wanted}
+
+ # We now have two sorted lists: source packages we want and those known to
+ # apt. We will only consider source packages that may be found in both.
+ comm -12 ${wanted} ${avail} \
+ | while read src version; do
+ # Name of the .dsc file does not include Epoch, remove it before checking
+ # if sources were already downloaded. Avoid using sed here to reduce the
+ # number of processes being spawned by this function: we assume that the
+ # version is correctly formatted and simply strip everything up to the
+ # first colon
+ dscname="${src}_${version#*:}.dsc"
+ [ -f "${DEBSRCDIR}"/"${rootfs_distro}"/"${src}"/"${dscname}" ] || {
+ # use apt-get source to download sources in DEBSRCDIR
+ sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
+ sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
+ }
done
) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
--
2.39.5
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250322061519.2522459-2-cedric.hombourger%40siemens.com.
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH v2 0/1] deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download
2025-03-22 6:15 ` [PATCH v2 0/1] " 'Cedric Hombourger' via isar-users
2025-03-22 6:15 ` [PATCH v2 1/1] " 'Cedric Hombourger' via isar-users
@ 2025-03-27 10:34 ` Uladzimir Bely
1 sibling, 0 replies; 9+ messages in thread
From: Uladzimir Bely @ 2025-03-27 10:34 UTC (permalink / raw)
To: Cedric Hombourger, isar-users
On Sat, 2025-03-22 at 07:15 +0100, Cedric Hombourger wrote:
> Changes since v1:
> * v1 had is_not_part_of_current_build() removed. It turns out that
> we better check what .deb files get imported in the apt cache
> of our rootfs to make sure that we only try to fetch sources for
> packages we know. This is now achieved with apt-cache dumpavail
> to obtain a list of known source packages.
>
> Cedric Hombourger (1):
> deb-dl-dir: remove excessive calls to dpkg-deb in debsrc_download
>
> meta/classes/deb-dl-dir.bbclass | 78 +++++++++++++++++++++----------
> --
> 1 file changed, 51 insertions(+), 27 deletions(-)
Applied to next, thanks.
--
Best regards,
Uladzimir.
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/7209080ec63d6898fc04a4e413b6487b4329a74f.camel%40ilbers.de.
^ permalink raw reply [flat|nested] 9+ messages in thread