* [PATCH v2 1/1] deb-dl-dir: fix package source download
2025-01-24 9:12 [PATCH v2 0/1] deb-dl-dir: fix package source download 'Benedikt Niedermayr' via isar-users
@ 2025-01-24 9:12 ` 'Benedikt Niedermayr' via isar-users
2025-02-28 13:12 ` [PATCH v2 0/1] " Uladzimir Bely
1 sibling, 0 replies; 3+ messages in thread
From: 'Benedikt Niedermayr' via isar-users @ 2025-01-24 9:12 UTC (permalink / raw)
To: isar-users
Some of the packages that are part of the bootstrapped base filesystem
were not downloaded when activating "cache-deb-src" via
BASE_REPO_FEATURES.
It seems that some (or all) packages are not listed in
"/var/log/dpkg.log" leading to skip the download of these files.
This is a regression that has been introduced with mmdebstrap, which
either not creates or just deletes the mentioned log file.
Package built by using the dpkg-prebuilt.bbclass tend to reference
binary packages without providing any source package reference.
To handle these, currently valid, cases the download function simply
skips packages that could not be downloaded and prints a warning.
Signed-off-by: Benedikt Niedermayr <benedikt.niedermayr@siemens.com>
---
meta/classes/deb-dl-dir.bbclass | 39 ++++++++++-----------------------
1 file changed, 11 insertions(+), 28 deletions(-)
diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 7ebd057e5504..9f6d205894dc 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -5,23 +5,6 @@
inherit repository
-is_not_part_of_current_build() {
- local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"
- local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )"
- local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )"
- # Since we are parsing all the debs in DEBDIR, we can to some extend
- # try to eliminate some debs that are not part of the current multiconfig
- # build using the below method.
- local output="$( grep -xhs ".* status installed ${package}:${arch} ${version}" \
- "${IMAGE_ROOTFS}"/var/log/dpkg.log \
- "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \
- "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \
- "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \
- "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log | head -1 )"
-
- [ -z "${output}" ]
-}
-
debsrc_do_mounts() {
sudo -s <<EOSUDO
set -e
@@ -54,17 +37,17 @@ debsrc_download() {
( flock 9
set -e
printenv | grep -q BB_VERBOSE_LOGS && set -x
- find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do
- is_not_part_of_current_build "${package}" && continue
- local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
- local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
- local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')"
- local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc")
- [ -n "$dscfile" ] && continue
-
- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
- sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
- done
+ sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
+ dpkg-query -f '${source:Package} ${source:Version}\n' -W | while read -r src srcver; do
+ ver_stripped=$(echo "$srcver" | sed 's/^[0-9]*://')
+ test -f "${DEBSRCDIR}/${rootfs_distro}/${src}/${src}_${ver_stripped}.dsc" && continue
+
+ # Note: package built by using dpkg-prebuilt are tend to be missing
+ sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
+ sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' \
+ download-src "${rootfs_distro}" "${src}" "${srcver}" || \
+ bbwarn "Failed to download source package ${src}_${srcver}"
+ done
) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
debsrc_undo_mounts "${rootfs}"
--
2.34.1
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250124091219.3489707-2-benedikt.niedermayr%40siemens.com.
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [PATCH v2 0/1] deb-dl-dir: fix package source download
2025-01-24 9:12 [PATCH v2 0/1] deb-dl-dir: fix package source download 'Benedikt Niedermayr' via isar-users
2025-01-24 9:12 ` [PATCH v2 1/1] " 'Benedikt Niedermayr' via isar-users
@ 2025-02-28 13:12 ` Uladzimir Bely
1 sibling, 0 replies; 3+ messages in thread
From: Uladzimir Bely @ 2025-02-28 13:12 UTC (permalink / raw)
To: Benedikt Niedermayr, isar-users; +Cc: Jan Kiszka
On Fri, 2025-01-24 at 10:12 +0100, 'Benedikt Niedermayr' via isar-users
wrote:
> I recently wanted to download all source packages for a specific
> image and encountered
> that some of the source packages were missing (e.g. adduser).
>
> I tracked the issue down to the deb-dl-dir.bbclass and found that the
> "is_not_part_of_current_build" function skipped the source package
> download
> because the "adduser" package was not listed in the
> "/var/log/dpkg.log" file.
>
> I assume that this package has been installed during early
> debootstrap
> and therefore the package is not listed in the dpkg.log file.
>
> **As discussed in V1, the issue has been introduced with mmdebstrap,
> which
> either doesn't create or deletes the dpkg.log file.**
>
> I'm not sure if I got the comment in the function right:
>
> "Since we are parsing all the debs in DEBDIR, we can to some extend
> try to eliminate some debs that are not part of the current
> multiconfig
> build using the below method"
>
> AFAIK we could also achieve this by running dpkg-query commands on
> the target
> filesystem and also catch the packages that have been installed
> during bootstrap.
>
> I'm not sure if this patch will interfere in any way with multiconfig
> builds,
> as mentioned in the comment above, but I think it's worth a try.
>
>
> Changes since v1:
> - Adapt the commit message: Make it clear that the problem
> has been introduced with mmdebstrap
>
>
> Benedikt Niedermayr (1):
> deb-dl-dir: fix package source download
>
> meta/classes/deb-dl-dir.bbclass | 39 ++++++++++---------------------
> --
> 1 file changed, 11 insertions(+), 28 deletions(-)
>
> --
> 2.34.1
>
Hello.
Sorry for delayed response on this patchset.
It was initially tested a couple of weeks ago and caused our CI to fail
with. So the patchset merging was postponed for detailed debugging.
With this patch, internal (full) CI fails in
SstateTest.test_sstate_populate with the following error:
```
[stdlog] 2025-02-27 20:06:15,150 avocado.test INFO | NOTE: recipe
hello-1.0-r0: task do_unpack: Started
[stdlog] 2025-02-27 20:06:17,363 avocado.app ERROR| ERROR:
mc:qemuamd64-bullseye:hello-1.0-r0 do_unpack: Fetcher failure: Fetch
command export PSEUDO_DISABLED=1; export
PATH="/workspace/build/isar_ub_devel/115/isar-
sstate/scripts:/workspace/build/isar_ub_devel/115/isar-
sstate/bitbake/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/g
ames:/sbin"; export HOME="/workspace";
[stdlog] 2025-02-27 20:06:17,363 avocado.test INFO | NOTE: recipe
hello-1.0-r0: task do_unpack: Failed
[stdlog] 2025-02-27 20:06:17,363 avocado.app ERROR| set
-e
[stdlog] 2025-02-27 20:06:17,363 avocado.app ERROR|
schroot -r -c isar-jenkins-ecfca206-95e9-4d1e-844d-e2033288e6a6-
2125943-83ed2888-a8fa-4485-94ed-2b16dc2cbea8 -d / -u root --
rm /etc/apt/sources.list.d/isar-apt.list /etc/apt/preferences.d/isar-
apt
[stdlog] 2025-02-27 20:06:17,363 avocado.app ERROR|
schroot -r -c isar-jenkins-ecfca206-95e9-4d1e-844d-e2033288e6a6-
2125943-83ed2888-a8fa-4485-94ed-2b16dc2cbea8 -d / --
sh -c '
[stdlog] 2025-02-27 20:06:17,363 avocado.app ERROR|
set -e
[stdlog] 2025-02-27 20:06:17,363 avocado.app ERROR|
dscfile=$(apt-get -y -qq --print-uris --only-source source hello |
cut -d " " -f2 | grep -E "\.dsc")
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR|
cp /downloads/deb-src/debian-bullseye/hello/* /home/builder/hello
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR|
cd /home/builder/hello
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR|
mv -f hello-1.0 hello-1.0.prev
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR|
dpkg-source -x "$dscfile" hello-1.0
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR|
find hello-1.0.prev -mindepth 1 -maxdepth 1 -exec mv {} hello-1.0/ \;
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR|
rmdir hello-1.0.prev
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR|
'
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR|
failed with exit code 1, output:
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR| cp: cannot stat
'/downloads/deb-src/debian-bullseye/hello/hello_2.10-
2+isar.debian.tar.xz': No such file or directory
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR| cp: cannot stat
'/downloads/deb-src/debian-bullseye/hello/hello_2.10-2+isar.dsc': No
such file or directory
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR|
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR| ERROR: Logfile of
failure stored in: /workspace/build/isar_ub_devel/115/build-
sstate/tmp/work/debian-bullseye-amd64/hello/1.0-
r0/temp/log.do_unpack.2125943
[stdlog] 2025-02-27 20:06:17,370 avocado.app ERROR| ERROR: Task
(mc:qemuamd64-bullseye:/workspace/build/isar_ub_devel/115/isar-
sstate/meta-isar/recipes-app/hello/hello.bb:do_unpack) failed with exit
code '1'
```
Recently I've got some free time to debug it and the issue is a bit
confusing:
- If run only SstateTest.test_sstate_populate, it passes OK
- If run it after ReproTest.test_repro_unsigned - it passes OK
- If run it after CcacheTest.test_ccache_rebuild - it also passes OK
But when it run with more tests ("full" CI chain) - it fails.
For me, it looks like some repo conflict between self-built
"hello_2.10-2+isar.dsc" and Debian's "hello_2.10-2.dsc", but I still
haven't figured out the exact issue.
Continue debugging...
--
Best regards,
Uladzimir.
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/71c74cfd2792b317849a12f5d55185e94e263467.camel%40ilbers.de.
^ permalink raw reply [flat|nested] 3+ messages in thread