Re: [PATCH v2 0/1] deb-dl-dir: fix package source download

[Uladzimir Bely <ubely@ilbers.de>]

On Fri, 2025-01-24 at 10:12 +0100, 'Benedikt Niedermayr' via isar-users
wrote:
> I recently wanted to download all source packages for a specific
> image and encountered
> that some of the source packages were missing (e.g. adduser).
> 
> I tracked the issue down to the deb-dl-dir.bbclass and found that the
> "is_not_part_of_current_build" function skipped the source package
> download
> because the "adduser" package was not listed in the
> "/var/log/dpkg.log" file.
> 
> I assume that this package has been installed during early
> debootstrap
> and therefore the package is not listed in the dpkg.log file.
> 
> **As discussed in V1, the issue has been introduced with mmdebstrap,
> which
> either doesn't create or deletes the dpkg.log file.**
> 
> I'm not sure if I got the comment in the function right:
> 
> "Since we are parsing all the debs in DEBDIR, we can to some extend
> try to eliminate some debs that are not part of the current
> multiconfig
> build using the below method"
> 
> AFAIK we could also achieve this by running dpkg-query commands on
> the target
> filesystem and also catch the packages that have been installed
> during bootstrap.
> 
> I'm not sure if this patch will interfere in any way with multiconfig
> builds,
> as mentioned in the comment above, but I think it's worth a try.
> 
> 
> Changes since v1:
> - Adapt the commit message: Make it clear that the problem
>   has been introduced with mmdebstrap
> 
> 
> Benedikt Niedermayr (1):
>   deb-dl-dir: fix package source download
> 
>  meta/classes/deb-dl-dir.bbclass | 39 ++++++++++---------------------
> --
>  1 file changed, 11 insertions(+), 28 deletions(-)
> 
> --
> 2.34.1
> 

Hello.

Sorry for delayed response on this patchset.

It was initially tested a couple of weeks ago and caused our CI to fail
with. So the patchset merging was postponed for detailed debugging.

With this patch, internal (full) CI fails in
SstateTest.test_sstate_populate with the following error:

```
[stdlog] 2025-02-27 20:06:15,150 avocado.test INFO | NOTE: recipe
hello-1.0-r0: task do_unpack: Started
[stdlog] 2025-02-27 20:06:17,363 avocado.app ERROR| ERROR:
mc:qemuamd64-bullseye:hello-1.0-r0 do_unpack: Fetcher failure: Fetch
command export PSEUDO_DISABLED=1; export
PATH="/workspace/build/isar_ub_devel/115/isar-
sstate/scripts:/workspace/build/isar_ub_devel/115/isar-
sstate/bitbake/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/g
ames:/sbin"; export HOME="/workspace";
[stdlog] 2025-02-27 20:06:17,363 avocado.test INFO | NOTE: recipe
hello-1.0-r0: task do_unpack: Failed
[stdlog] 2025-02-27 20:06:17,363 avocado.app ERROR|                 set
-e
[stdlog] 2025-02-27 20:06:17,363 avocado.app ERROR|                
schroot -r -c isar-jenkins-ecfca206-95e9-4d1e-844d-e2033288e6a6-
2125943-83ed2888-a8fa-4485-94ed-2b16dc2cbea8 -d / -u root --          
rm /etc/apt/sources.list.d/isar-apt.list /etc/apt/preferences.d/isar-
apt
[stdlog] 2025-02-27 20:06:17,363 avocado.app ERROR|                
schroot -r -c isar-jenkins-ecfca206-95e9-4d1e-844d-e2033288e6a6-
2125943-83ed2888-a8fa-4485-94ed-2b16dc2cbea8 -d / --                  
sh -c '
[stdlog] 2025-02-27 20:06:17,363 avocado.app ERROR|                   
set -e
[stdlog] 2025-02-27 20:06:17,363 avocado.app ERROR|                   
dscfile=$(apt-get -y -qq --print-uris --only-source source hello |    
cut -d " " -f2 | grep -E "\.dsc")
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR|                   
cp /downloads/deb-src/debian-bullseye/hello/* /home/builder/hello
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR|                   
cd /home/builder/hello
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR|                   
mv -f hello-1.0 hello-1.0.prev
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR|                   
dpkg-source -x "$dscfile" hello-1.0
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR|                   
find hello-1.0.prev -mindepth 1 -maxdepth 1 -exec mv {} hello-1.0/ \;
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR|                   
rmdir hello-1.0.prev
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR|                   
'
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR|                 
failed with exit code 1, output:
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR| cp: cannot stat
'/downloads/deb-src/debian-bullseye/hello/hello_2.10-
2+isar.debian.tar.xz': No such file or directory
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR| cp: cannot stat
'/downloads/deb-src/debian-bullseye/hello/hello_2.10-2+isar.dsc': No
such file or directory
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR| 
[stdlog] 2025-02-27 20:06:17,364 avocado.app ERROR| ERROR: Logfile of
failure stored in: /workspace/build/isar_ub_devel/115/build-
sstate/tmp/work/debian-bullseye-amd64/hello/1.0-
r0/temp/log.do_unpack.2125943
[stdlog] 2025-02-27 20:06:17,370 avocado.app ERROR| ERROR: Task
(mc:qemuamd64-bullseye:/workspace/build/isar_ub_devel/115/isar-
sstate/meta-isar/recipes-app/hello/hello.bb:do_unpack) failed with exit
code '1'
```

Recently I've got some free time to debug it and the issue is a bit
confusing:

- If run only SstateTest.test_sstate_populate, it passes OK
- If run it after ReproTest.test_repro_unsigned - it passes OK
- If run it after CcacheTest.test_ccache_rebuild - it also passes OK

But when it run with more tests ("full" CI chain) - it fails.

For me, it looks like some repo conflict between self-built
"hello_2.10-2+isar.dsc" and Debian's "hello_2.10-2.dsc", but I still
haven't figured out the exact issue.

Continue debugging...

-- 
Best regards,
Uladzimir.



-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/71c74cfd2792b317849a12f5d55185e94e263467.camel%40ilbers.de.