On Fri, Mar 21, 2025 at 7:20 PM 'cedric.hombourger@siemens.com' via isar-users wrote: > On Fri, 2025-03-21 at 11:39 +0300, Uladzimir Bely wrote: > > On Fri, 2025-03-21 at 06:32 +0000, > > cedric.hombourger@siemens.com wrote: > > > On Fri, 2025-03-21 at 09:27 +0300, Uladzimir Bely wrote: > > > > On Thu, 2025-03-20 at 07:18 +0000, 'Arjunan, Srinu' via isar- > > > > users > > > > wrote: > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Uladzimir Bely > > > > > Sent: 20 March 2025 11:02 > > > > > To: Arjunan, Srinu (FT FDS CES LX PBU 2) > > > > > ; > > > > > isar-users@googlegroups.com > > > > > Cc: Kiszka, Jan (FT RPD CED) ; > > > > > Hombourger, > > > > > Cedric (FT FDS CES LX) ; > > > > > Niedermayr, > > > > > Benedikt (FT RPD CED OES-DE) ; > > > > > Adithya.Balakumar@toshiba-tsip.com > > > > > Subject: Re: [PATCH 1/1] image: move deb-src-caching > > > > > functionality > > > > > after image creation > > > > > > > > > > On Tue, 2025-03-18 at 16:45 +0000, Arjunan, Srinu wrote: > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: Arjunan, Srinu (FT FDS CES LX PBU 2) > > > > > > > > > > > > Sent: 10 March 2025 17:22 > > > > > > To: isar-users@googlegroups.com > > > > > > Cc: Kiszka, Jan (FT RPD CED) ; > > > > > > Hombourger, > > > > > > Cedric (FT FDS CES LX) ; > > > > > > ubely@ilbers.de; Niedermayr, Benedikt (FT RPD CED OES-DE) > > > > > > ; > > > > > > Adithya.Balakumar@toshiba-tsip.com; Arjunan, Srinu (FT FDS > > > > > > CES > > > > > > LX > > > > > > PBU > > > > > > 2) > > > > > > Subject: [PATCH 1/1] image: move deb-src-caching > > > > > > functionality > > > > > > after > > > > > > image creation > > > > > > > > > > > > From: srinuvasan > > > > > > > > > > > > When we do deb-src caching, we are missing to cache imager > > > > > > related > > > > > > packages, because the deb-src-caching functionality runs in > > > > > > the > > > > > > stage > > > > > > of ROOTFS_POSTPROCESS_COMAMND, but image creation task runs > > > > > > after > > > > > > the > > > > > > rootfs creation, hence imager related packages are missing in > > > > > > rootfs's > > > > > > var/cache/apt/archives. > > > > > > > > > > > > To solve this issue run the deb-src-caching after image > > > > > > creation, > > > > > > now > > > > > > we can able to download deb-src for all the deb packages. > > > > > > > > > > > > Signed-off-by: srinuvasan > > > > > > --- > > > > > > meta/classes/image.bbclass | 8 +++++++- > > > > > > meta/classes/rootfs.bbclass > > > > > > > 3 ++- > > > > > > 2 files changed, 9 insertions(+), 2 deletions(-) > > > > > > > > > > > > diff --git a/meta/classes/image.bbclass > > > > > > b/meta/classes/image.bbclass > > > > > > index 56eca202..66b4d313 100644 > > > > > > --- a/meta/classes/image.bbclass > > > > > > +++ b/meta/classes/image.bbclass > > > > > > @@ -72,7 +72,13 @@ ROOTFS_MANIFEST_DEPLOY_DIR ?= > > > > > > "${DEPLOY_DIR_IMAGE}" > > > > > > ROOTFS_DPKGSTATUS_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" > > > > > > ROOTFS_PACKAGE_SUFFIX ?= "${PN}-${DISTRO}-${MACHINE}" > > > > > > > > > > > > -ROOTFS_POSTPROCESS_COMMAND:prepend = > > > > > > "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', > > > > > > 'cache_deb_src', '', d)} " > > > > > > +CACHE_DEB_SRC = "${@bb.utils.contains('BASE_REPO_FEATURES', > > > > > > 'cache- > > > > > > deb-src', '1', '0', d)}" > > > > > > + > > > > > > +python () { > > > > > > + if bb.utils.to_boolean(d.getVar('CACHE_DEB_SRC')): > > > > > > + bb.build.addtask('do_cache_deb_src', 'do_deploy', > > > > > > 'do_image', > > > > > > +d) } > > > > > > + > > > > > > > > > > > > inherit rootfs > > > > > > inherit sdk > > > > > > diff --git a/meta/classes/rootfs.bbclass > > > > > > b/meta/classes/rootfs.bbclass > > > > > > index 2348e269..c9e832dc 100644 > > > > > > --- a/meta/classes/rootfs.bbclass > > > > > > +++ b/meta/classes/rootfs.bbclass > > > > > > @@ -272,7 +272,8 @@ python do_rootfs_install() { } addtask > > > > > > rootfs_install before do_rootfs_postprocess after do_unpack > > > > > > > > > > > > -cache_deb_src() { > > > > > > +do_cache_deb_src[network] = "${TASK_USE_SUDO}" > > > > > > +do_cache_deb_src() { > > > > > > if [ -e "${ROOTFSDIR}"/etc/resolv.conf ] || > > > > > > [ -h "${ROOTFSDIR}"/etc/resolv.conf ]; then > > > > > > sudo mv "${ROOTFSDIR}"/etc/resolv.conf > > > > > > "${ROOTFSDIR}"/etc/resolv.conf.isar > > > > > > -- > > > > > > 2.39.5 > > > > > > > > > > > > Hi All, > > > > > > > > > > > > Can you please update the status of this patch?, or > > > > > > anything > > > > > > missing from my end? > > > > > > > > > > > > > > > > Hello. > > > > > > > > > > Sorry for delays related to "installer" currently breaking full > > > > > CI > > > > > on > > > > > "next" branch. > > > > > > > > > > Tested recently your patch, it also makes CI fail at: > > > > > > > > > > CrossTest.test_cross_debsrc: > > > > > > > > > > ``` > > > > > [stdlog] 2025-03-19 21:29:50,542 avocado.app ERROR| ERROR: Task > > > > > (mc:qemuarm64- > > > > > bookworm:/workspace/build/isar_ub_devel_fast/214/meta- > > > > > test/recipes-core/images/isar-image-ci.bb:do_cache_deb_src) > > > > > failed > > > > > with exit code '1' > > > > > [stdlog] 2025-03-19 21:30:44,921 avocado.test INFO | | DEBUG: > > > > > Executing shell function do_cache_deb_src [stdlog] 2025-03-19 > > > > > 21:30:44,921 avocado.test INFO | | Reading package lists... > > > > > [stdlog] 2025-03-19 21:30:44,921 avocado.test INFO | | E: You > > > > > must > > > > > put some 'deb-src' URIs in your sources.list ``` > > > > > > > > > > NoCrossTest.test_nocross_debsrc: > > > > > > > > > > ``` > > > > > [stdlog] 2025-03-20 01:09:35,759 avocado.app ERROR| ERROR: > > > > > Logfile > > > > > of > > > > > failure stored in: > > > > > /workspace/build/isar_ub_devel/126/build/tmp/work/debian- > > > > > bookworm- > > > > > armhf/isar-image-ci-qemuarm/1.0- > > > > > r0/temp/log.do_cache_deb_src.218291 > > > > > [stdlog] 2025-03-20 01:09:35,832 avocado.app ERROR| ERROR: Task > > > > > (mc:qemuarm-bookworm:/workspace/build/isar_ub_devel/126/meta- > > > > > test/recipes-core/images/isar-image-ci.bb:do_cache_deb_src) > > > > > failed > > > > > with exit code '1' > > > > > [stdlog] 2025-03-20 01:19:35,822 avocado.test INFO | | DEBUG: > > > > > Executing shell function do_cache_deb_src [stdlog] 2025-03-20 > > > > > 01:19:35,823 avocado.test INFO | | Reading package lists... > > > > > [stdlog] 2025-03-20 01:19:35,823 avocado.test INFO | | E: You > > > > > must > > > > > put some 'deb-src' URIs in your sources.list ``` > > > > > > > > > > > > > > > Hi Uladzimir, > > > > > > > > > > Thank you for the reply, I am not able to > > > > > reproduce the build issue what you reported , I triggered the > > > > > build > > > > > for isar-image-ci target and it's getting passed. > > > > > > > > > > This patch depend on Cedric patch [1] > > > > > [1]: > > > > > https://groups.google.com/g/isar-users/c/OwrSBmCPe0Y > > > > > > > > > > I validated my patch along with the [1], it seems working fine. > > > > > > > > > > Or Can you please provide the steps to reproduce the issue? > > > > > > > > > > > > > > > > Many thanks, > > > > > > Srinu > > > > > > > > > > > > > > > > > > > I finally found root cause and easy steps to reproduce. The patch > > > > moves > > > > debsrc caching to later stage and that results in incompatibility > > > > with > > > > DISTRO_APT_PREMIRRORS option we are globally using in CI. > > > > > > Ah thanks. Are you seeing any issues with my patch alone? (was > > > going > > > to > > > re-run a local CI build to *double* check) > > > > > > Would appreciate if my changes may be CI-checked alone and merged; > > > provided there are no issues ofcourse :) > > > > > > > Actually, found easy way to reproduce the issue with "deb-dl-dir: > > remove excessive calls to dpkg-deb in debsrc_download" on top of > > 'next'. > > Fantastic! > > I have revised my patch to combine the list of source packages we'd > like and the list of packages known to apt (apt-cache dumpavail). > > I am now going to exercise this revised approach in our downstream > layer (where we are downloading thousands of source packages) but also > with the Isar CI tests. > > This will probably take a little while... stay tuned! > and thanks again for all the help > > > > > Steps to reproduce with kas-container: > > > > 1. Place the deb we fail to download sources for to downloads > > directory. > > ``` > > mkdir -p ./build/downloads/deb/raspios-bookworm > > wget > > > https://deb.debian.org/debian/pool/main/d/debian-archive-keyring/debian-archive-keyring_2023.3+deb12u1_all.deb > > -P build/downloads/deb/raspios-bookworm/ > > ``` > > > > 2. Configure build: > > > > ./kas/kas-container menu > > - `Machine selection`: rpi-arm-v7l > > - `Distro selection`: Raspberry Pi OS Debian 12 (bookworm) > > - `Cache all source packages`: check the option > > > > Run build > > > > 3. Result: > > > > ``` > > > E: Can not find version '2023.3+deb12u1' of package 'debian- > > > archive- > > keyring' > > > E: Unable to find a source package for debian-archive-keyring > > > DEBUG: Executing shell function rootfs_do_umounts > > > DEBUG: Shell function rootfs_do_umounts finished > > > DEBUG: Python function do_rootfs_postprocess finished > > ERROR: Task (/build/../repo/meta-isar/recipes-core/images/isar-image- > > base.bb:do_rootfs_postprocess) failed with exit code '1' > > ``` > > > > The issue is not reproducable on 'next' (since we are parsing > > dpkg.log > > and it has no entry for debian-archive-keyring, since it is not a > > part > > of current build. > > > > > > > > > > Steps to reproduce: > > > > > > > > 1. Run `./kas/kas-container menu` > > > > 2. Change the configuration: > > > > - `Mirror selection => Use specific Debian mirror`: > > > > http://ftp.de.debian.org/ > > > > - `Cache all source packages`: Check the option > > > > - `Packages selection => Isar packages`: Uncheck everything to > > > > speedup > > > > build (not necessary step) > > > > 3. Run build > > > > > > > > It fails with: > > > > > > > > ``` > > > > ERROR: isar-image-base-1.0-r0 do_cache_deb_src: > > > > ExecutionError('/build/tmp/work/debian-bookworm-amd64/isar-image- > > > > base- > > > > qemuamd64/1.0-r0/temp/run.do_cache_deb_src.10979', 100, None, > > > > None) > > > > ERROR: Logfile of failure stored in: /build/tmp/work/debian- > > > > bookworm- > > > > amd64/isar-image-base-qemuamd64/1.0- > > > > r0/temp/log.do_cache_deb_src.10979 > > > > Log data follows: > > > > > DEBUG: Executing shell function do_cache_deb_src > > > > > Reading package lists... > > > > > E: You must put some 'deb-src' URIs in your sources.list > > > > ERROR: Task (/build/../repo/meta-isar/recipes-core/images/isar- > > > > image- > > > > base.bb:do_cache_deb_src) failed with exit code '1' > > > > ``` > > > > > > > > > -- > > > > > Best regards, > > > > > Uladzimir. > > > > > > Hi Uladzimir, I triggered the CI build locally with this patch; it looks like CI builds are getting passed, Here are the logs: Logs: builder@0ed36b8446c5:/work/isar/testsuite$ avocado run citest.py -t dev --max-parallel-tasks=1 JOB ID : 3ca66c44c8b33b1e2f265f46613f502fa89dbbde JOB LOG : /tmp/tmppvxevsmp/avocado/job-results/job-2025-03-27T15.58-3ca66c4/job.log (1/6) citest.py:DevTest.test_dev: STARTED (1/6) citest.py:DevTest.test_dev: PASS (1004.16 s) (2/6) citest.py:DevTest.test_dev_apps: STARTED (2/6) citest.py:DevTest.test_dev_apps: PASS (1153.27 s) (3/6) citest.py:DevTest.test_dev_rebuild: STARTED (3/6) citest.py:DevTest.test_dev_rebuild: PASS (450.10 s) (4/6) citest.py:DevTest.test_dev_run_amd64_bullseye: STARTED (4/6) citest.py:DevTest.test_dev_run_amd64_bullseye: PASS (56.52 s) (5/6) citest.py:DevTest.test_dev_run_arm64_bullseye: STARTED (5/6) citest.py:DevTest.test_dev_run_arm64_bullseye: PASS (45.19 s) (6/6) citest.py:DevTest.test_dev_run_arm_bullseye: STARTED (6/6) citest.py:DevTest.test_dev_run_arm_bullseye: PASS (49.96 s) RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 2765.02 s Many thanks, Srinu > > > > > > > > > > -- > > > Cedric Hombourger > > > Siemens AG > > > http://www.siemens.com/ > > > > -- > Cedric Hombourger > Siemens AG > http://www.siemens.com/ > > -- > You received this message because you are subscribed to the Google Groups > "isar-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to isar-users+unsubscribe@googlegroups.com. > To view this discussion visit > https://groups.google.com/d/msgid/isar-users/1c66d14280d6a27e1529659b2d651f172a32ed87.camel%40siemens.com > . > -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/CAB2Z2nPJBffOwGwMYrk7Z90Z9VcV5YbmtPuFG%3DdEd-p5hZrvUA%40mail.gmail.com.