[PATCH 0/1] deb-dl-dir: fix package source download

[PATCH 0/1] deb-dl-dir: fix package source download

['Benedikt Niedermayr' via isar-users]

I recently wanted to download all source packages for a specific image and encountered
that some of the source packages were missing (e.g. adduser).

I tracked the issue down to the deb-dl-dir.bbclass and found that the
"is_not_part_of_current_build" function skipped the source package download
because the "adduser" package was not listed in the "/var/log/dpkg.log" file.

I assume that this package has been installed during early debootstrap
and therefore the package is not listed in the dpkg.log file.

I'm not sure if I got the comment in the function right:

"Since we are parsing all the debs in DEBDIR, we can to some extend
try to eliminate some debs that are not part of the current multiconfig
build using the below method"

AFAIK we could also achieve this by running dpkg-query commands on the target
filesystem and also catch the packages that have been installed during bootstrap.

I'm not sure if this patch will interfere in any way with multiconfig builds,
as mentioned in the comment above, but I think it's worth a try.

Benedikt Niedermayr (1):
  deb-dl-dir: fix package source download

 meta/classes/deb-dl-dir.bbclass | 39 ++++++++++-----------------------
 1 file changed, 11 insertions(+), 28 deletions(-)

--
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250122134907.2826754-1-benedikt.niedermayr%40siemens.com.

[PATCH 1/1] deb-dl-dir: fix package source download

['Benedikt Niedermayr' via isar-users]

Some of the packages that are part of the debootstrap base filesystem
were not downloaded when activating "cache-deb-src" via
BASE_REPO_FEATURES.

It seems that some (or all) packages are not listed in
"/var/log/dpkg.log" leading to skip the download of these files.

Package built by using the dpkg-prebuilt.bbclass tend to reference
binary packages without providing any source package reference.
To handle these currently valid cases the download function simply
skips packages that could not be downloaded and prints a warning.

Signed-off-by: Benedikt Niedermayr <benedikt.niedermayr@siemens.com>
---
 meta/classes/deb-dl-dir.bbclass | 39 ++++++++++-----------------------
 1 file changed, 11 insertions(+), 28 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 7ebd057e5504..9f6d205894dc 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -5,23 +5,6 @@
 
 inherit repository
 
-is_not_part_of_current_build() {
-    local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"
-    local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )"
-    local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )"
-    # Since we are parsing all the debs in DEBDIR, we can to some extend
-    # try to eliminate some debs that are not part of the current multiconfig
-    # build using the below method.
-    local output="$( grep -xhs ".* status installed ${package}:${arch} ${version}" \
-            "${IMAGE_ROOTFS}"/var/log/dpkg.log \
-            "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \
-            "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \
-            "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \
-            "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log | head -1 )"
-
-    [ -z "${output}" ]
-}
-
 debsrc_do_mounts() {
     sudo -s <<EOSUDO
     set -e
@@ -54,17 +37,17 @@ debsrc_download() {
     ( flock 9
     set -e
     printenv | grep -q BB_VERBOSE_LOGS && set -x
-    find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do
-        is_not_part_of_current_build "${package}" && continue
-        local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
-        local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
-        local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')"
-        local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc")
-        [ -n "$dscfile" ] && continue
-
-        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
-            sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
-    done
+    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
+        dpkg-query -f '${source:Package} ${source:Version}\n' -W | while read -r src srcver; do
+            ver_stripped=$(echo "$srcver" | sed 's/^[0-9]*://')
+            test -f "${DEBSRCDIR}/${rootfs_distro}/${src}/${src}_${ver_stripped}.dsc" && continue
+
+            # Note: package built by using dpkg-prebuilt are tend to be missing
+            sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
+                sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' \
+                    download-src "${rootfs_distro}" "${src}" "${srcver}" || \
+                    bbwarn "Failed to download source package ${src}_${srcver}"
+        done
     ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
 
     debsrc_undo_mounts "${rootfs}"
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250122134907.2826754-2-benedikt.niedermayr%40siemens.com.

Re: [PATCH 1/1] deb-dl-dir: fix package source download

['Jan Kiszka' via isar-users]

On 22.01.25 14:49, 'Benedikt Niedermayr' via isar-users wrote:
> Some of the packages that are part of the debootstrap base filesystem

Just "bootstrap", at least as long as we still have debootstrap as
legacy option beside mmdebstrap.

Is this issues possibly a fallout from the mmdebstrap migration?

Jan

> were not downloaded when activating "cache-deb-src" via
> BASE_REPO_FEATURES.
> 
> It seems that some (or all) packages are not listed in
> "/var/log/dpkg.log" leading to skip the download of these files.
> 
> Package built by using the dpkg-prebuilt.bbclass tend to reference
> binary packages without providing any source package reference.
> To handle these currently valid cases the download function simply
> skips packages that could not be downloaded and prints a warning.
> 
> Signed-off-by: Benedikt Niedermayr <benedikt.niedermayr@siemens.com>
> ---
>  meta/classes/deb-dl-dir.bbclass | 39 ++++++++++-----------------------
>  1 file changed, 11 insertions(+), 28 deletions(-)
> 
> diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
> index 7ebd057e5504..9f6d205894dc 100644
> --- a/meta/classes/deb-dl-dir.bbclass
> +++ b/meta/classes/deb-dl-dir.bbclass
> @@ -5,23 +5,6 @@
>  
>  inherit repository
>  
> -is_not_part_of_current_build() {
> -    local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"
> -    local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )"
> -    local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )"
> -    # Since we are parsing all the debs in DEBDIR, we can to some extend
> -    # try to eliminate some debs that are not part of the current multiconfig
> -    # build using the below method.
> -    local output="$( grep -xhs ".* status installed ${package}:${arch} ${version}" \
> -            "${IMAGE_ROOTFS}"/var/log/dpkg.log \
> -            "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \
> -            "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \
> -            "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \
> -            "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log | head -1 )"
> -
> -    [ -z "${output}" ]
> -}
> -
>  debsrc_do_mounts() {
>      sudo -s <<EOSUDO
>      set -e
> @@ -54,17 +37,17 @@ debsrc_download() {
>      ( flock 9
>      set -e
>      printenv | grep -q BB_VERBOSE_LOGS && set -x
> -    find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do
> -        is_not_part_of_current_build "${package}" && continue
> -        local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
> -        local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
> -        local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')"
> -        local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc")
> -        [ -n "$dscfile" ] && continue
> -
> -        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
> -            sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
> -    done
> +    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
> +        dpkg-query -f '${source:Package} ${source:Version}\n' -W | while read -r src srcver; do
> +            ver_stripped=$(echo "$srcver" | sed 's/^[0-9]*://')
> +            test -f "${DEBSRCDIR}/${rootfs_distro}/${src}/${src}_${ver_stripped}.dsc" && continue
> +
> +            # Note: package built by using dpkg-prebuilt are tend to be missing
> +            sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
> +                sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' \
> +                    download-src "${rootfs_distro}" "${src}" "${srcver}" || \
> +                    bbwarn "Failed to download source package ${src}_${srcver}"
> +        done
>      ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
>  
>      debsrc_undo_mounts "${rootfs}"


-- 
Siemens AG, Foundational Technologies
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/cc7ab13d-604c-478a-88b9-8e07067696ee%40siemens.com.

Re: [PATCH 1/1] deb-dl-dir: fix package source download

['Niedermayr, BENEDIKT' via isar-users]

On 22.01.25 15:00, Kiszka, Jan (FT RPD CED) wrote:
> On 22.01.25 14:49, 'Benedikt Niedermayr' via isar-users wrote:
>> Some of the packages that are part of the debootstrap base filesystem
> Just "bootstrap", at least as long as we still have debootstrap as
> legacy option beside mmdebstrap.
>
> Is this issues possibly a fallout from the mmdebstrap migration?

Hi Jan,

just tested it running debootstrap(PREFERRED_PROVIDER_bootstrap-*).

Using debootstrap will download all packages. Seems that mmdebstrap 
introduced some regression here.

Are there any known issues using "dpkg-query" in terms of multiconfig? I 
would like to avoid backing up and restoring

the dpkg.log file which feels more like a workaround compared to using 
dpkg-query.


>
> Jan
>
>> were not downloaded when activating "cache-deb-src" via
>> BASE_REPO_FEATURES.
>>
>> It seems that some (or all) packages are not listed in
>> "/var/log/dpkg.log" leading to skip the download of these files.
>>
>> Package built by using the dpkg-prebuilt.bbclass tend to reference
>> binary packages without providing any source package reference.
>> To handle these currently valid cases the download function simply
>> skips packages that could not be downloaded and prints a warning.
>>
>> Signed-off-by: Benedikt Niedermayr <benedikt.niedermayr@siemens.com>
>> ---
>>   meta/classes/deb-dl-dir.bbclass | 39 ++++++++++-----------------------
>>   1 file changed, 11 insertions(+), 28 deletions(-)
>>
>> diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
>> index 7ebd057e5504..9f6d205894dc 100644
>> --- a/meta/classes/deb-dl-dir.bbclass
>> +++ b/meta/classes/deb-dl-dir.bbclass
>> @@ -5,23 +5,6 @@
>>   
>>   inherit repository
>>   
>> -is_not_part_of_current_build() {
>> -    local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"
>> -    local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )"
>> -    local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )"
>> -    # Since we are parsing all the debs in DEBDIR, we can to some extend
>> -    # try to eliminate some debs that are not part of the current multiconfig
>> -    # build using the below method.
>> -    local output="$( grep -xhs ".* status installed ${package}:${arch} ${version}" \
>> -            "${IMAGE_ROOTFS}"/var/log/dpkg.log \
>> -            "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \
>> -            "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \
>> -            "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \
>> -            "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log | head -1 )"
>> -
>> -    [ -z "${output}" ]
>> -}
>> -
>>   debsrc_do_mounts() {
>>       sudo -s <<EOSUDO
>>       set -e
>> @@ -54,17 +37,17 @@ debsrc_download() {
>>       ( flock 9
>>       set -e
>>       printenv | grep -q BB_VERBOSE_LOGS && set -x
>> -    find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do
>> -        is_not_part_of_current_build "${package}" && continue
>> -        local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
>> -        local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
>> -        local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')"
>> -        local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc")
>> -        [ -n "$dscfile" ] && continue
>> -
>> -        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
>> -            sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
>> -    done
>> +    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
>> +        dpkg-query -f '${source:Package} ${source:Version}\n' -W | while read -r src srcver; do
>> +            ver_stripped=$(echo "$srcver" | sed 's/^[0-9]*://')
>> +            test -f "${DEBSRCDIR}/${rootfs_distro}/${src}/${src}_${ver_stripped}.dsc" && continue
>> +
>> +            # Note: package built by using dpkg-prebuilt are tend to be missing
>> +            sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
>> +                sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' \
>> +                    download-src "${rootfs_distro}" "${src}" "${srcver}" || \
>> +                    bbwarn "Failed to download source package ${src}_${srcver}"
>> +        done
>>       ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
>>   
>>       debsrc_undo_mounts "${rootfs}"
>

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/f47301a8-a16c-4969-8b5e-b376c5cc4dfb%40siemens.com.