Re: [PATCH 1/1] image: move deb-src-caching functionality after image creation

[Uladzimir Bely <ubely@ilbers.de>]

[-- Attachment #1: Type: text/plain, Size: 10911 bytes --]

On Thu, 2025-03-20 at 18:02 +0000, cedric.hombourger@siemens.com wrote:
> On Thu, 2025-03-20 at 20:56 +0300, Uladzimir Bely wrote:
> > On Thu, 2025-03-20 at 07:18 +0000, Arjunan, Srinu wrote:
> > > 
> > > 
> > > -----Original Message-----
> > > From: Uladzimir Bely <ubely@ilbers.de>
> > > Sent: 20 March 2025 11:02
> > > To: Arjunan, Srinu (FT FDS CES LX PBU 2)
> > > <srinuvasan.a@siemens.com>;
> > > isar-users@googlegroups.com
> > > Cc: Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com>;
> > > Hombourger,
> > > Cedric (FT FDS CES LX) <cedric.hombourger@siemens.com>;
> > > Niedermayr,
> > > Benedikt (FT RPD CED OES-DE) <benedikt.niedermayr@siemens.com>;
> > > Adithya.Balakumar@toshiba-tsip.com
> > > Subject: Re: [PATCH 1/1] image: move deb-src-caching
> > > functionality
> > > after image creation
> > > 
> > > On Tue, 2025-03-18 at 16:45 +0000, Arjunan, Srinu wrote:
> > > > 
> > > > 
> > > > -----Original Message-----
> > > > From: Arjunan, Srinu (FT FDS CES LX PBU 2)
> > > > <srinuvasan.a@siemens.com>
> > > > Sent: 10 March 2025 17:22
> > > > To: isar-users@googlegroups.com
> > > > Cc: Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com>;
> > > > Hombourger,
> > > > Cedric (FT FDS CES LX) <cedric.hombourger@siemens.com>;
> > > > ubely@ilbers.de; Niedermayr, Benedikt (FT RPD CED OES-DE)
> > > > <benedikt.niedermayr@siemens.com>;
> > > > Adithya.Balakumar@toshiba-tsip.com; Arjunan, Srinu (FT FDS CES
> > > > LX
> > > > PBU
> > > > 2) <srinuvasan.a@siemens.com>
> > > > Subject: [PATCH 1/1] image: move deb-src-caching functionality
> > > > after
> > > > image creation
> > > > 
> > > > From: srinuvasan <srinuvasan.a@siemens.com>
> > > > 
> > > > When we do deb-src caching, we are missing to cache imager
> > > > related
> > > > packages, because the deb-src-caching functionality runs in the
> > > > stage
> > > > of ROOTFS_POSTPROCESS_COMAMND, but image creation task runs
> > > > after
> > > > the
> > > > rootfs creation, hence imager related packages are missing in
> > > > rootfs's
> > > > var/cache/apt/archives.
> > > > 
> > > > To solve this issue run the deb-src-caching after image
> > > > creation,
> > > > now
> > > > we can able to download deb-src for all the deb packages.
> > > > 
> > > > Signed-off-by: srinuvasan <srinuvasan.a@siemens.com>
> > > > ---
> > > >  meta/classes/image.bbclass  | 8 +++++++-
> > > > meta/classes/rootfs.bbclass
> > > > > 3 ++-
> > > >  2 files changed, 9 insertions(+), 2 deletions(-)
> > > > 
> > > > diff --git a/meta/classes/image.bbclass
> > > > b/meta/classes/image.bbclass
> > > > index 56eca202..66b4d313 100644
> > > > --- a/meta/classes/image.bbclass
> > > > +++ b/meta/classes/image.bbclass
> > > > @@ -72,7 +72,13 @@ ROOTFS_MANIFEST_DEPLOY_DIR ?=
> > > > "${DEPLOY_DIR_IMAGE}"
> > > >  ROOTFS_DPKGSTATUS_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
> > > >  ROOTFS_PACKAGE_SUFFIX ?= "${PN}-${DISTRO}-${MACHINE}"
> > > > 
> > > > -ROOTFS_POSTPROCESS_COMMAND:prepend =
> > > > "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src',
> > > > 'cache_deb_src', '', d)} "
> > > > +CACHE_DEB_SRC = "${@bb.utils.contains('BASE_REPO_FEATURES',
> > > > 'cache-
> > > > deb-src', '1', '0', d)}"
> > > > +
> > > > +python () {
> > > > +    if bb.utils.to_boolean(d.getVar('CACHE_DEB_SRC')):
> > > > +        bb.build.addtask('do_cache_deb_src', 'do_deploy',
> > > > 'do_image',
> > > > +d) }
> > > > +
> > > > 
> > > >  inherit rootfs
> > > >  inherit sdk
> > > > diff --git a/meta/classes/rootfs.bbclass
> > > > b/meta/classes/rootfs.bbclass
> > > > index 2348e269..c9e832dc 100644
> > > > --- a/meta/classes/rootfs.bbclass
> > > > +++ b/meta/classes/rootfs.bbclass
> > > > @@ -272,7 +272,8 @@ python do_rootfs_install() {  }  addtask
> > > > rootfs_install before do_rootfs_postprocess after do_unpack
> > > > 
> > > > -cache_deb_src() {
> > > > +do_cache_deb_src[network] = "${TASK_USE_SUDO}"
> > > > +do_cache_deb_src() {
> > > >      if [ -e "${ROOTFSDIR}"/etc/resolv.conf ] ||
> > > >         [ -h "${ROOTFSDIR}"/etc/resolv.conf ]; then
> > > >          sudo mv "${ROOTFSDIR}"/etc/resolv.conf
> > > > "${ROOTFSDIR}"/etc/resolv.conf.isar
> > > > --
> > > > 2.39.5
> > > > 
> > > > Hi All,
> > > > 
> > > >           Can you please update the status of this patch?, or
> > > > anything
> > > > missing from my end?
> > > > 
> > > 
> > > Hello.
> > > 
> > > Sorry for delays related to "installer" currently breaking full
> > > CI
> > > on
> > > "next" branch.
> > > 
> > > Tested recently your patch, it also makes CI fail at:
> > > 
> > > CrossTest.test_cross_debsrc:
> > > 
> > > ```
> > > [stdlog] 2025-03-19 21:29:50,542 avocado.app ERROR| ERROR: Task
> > > (mc:qemuarm64-
> > > bookworm:/workspace/build/isar_ub_devel_fast/214/meta-
> > > test/recipes-core/images/isar-image-ci.bb:do_cache_deb_src)
> > > failed
> > > with exit code '1'
> > > [stdlog] 2025-03-19 21:30:44,921 avocado.test INFO | | DEBUG:
> > > Executing shell function do_cache_deb_src [stdlog] 2025-03-19
> > > 21:30:44,921 avocado.test INFO | | Reading package lists...
> > > [stdlog] 2025-03-19 21:30:44,921 avocado.test INFO | | E: You
> > > must
> > > put some 'deb-src' URIs in your sources.list ```
> > > 
> > > NoCrossTest.test_nocross_debsrc:
> > > 
> > > ```
> > > [stdlog] 2025-03-20 01:09:35,759 avocado.app ERROR| ERROR:
> > > Logfile
> > > of
> > > failure stored in:
> > > /workspace/build/isar_ub_devel/126/build/tmp/work/debian-
> > > bookworm-
> > > armhf/isar-image-ci-qemuarm/1.0-
> > > r0/temp/log.do_cache_deb_src.218291
> > > [stdlog] 2025-03-20 01:09:35,832 avocado.app ERROR| ERROR: Task
> > > (mc:qemuarm-bookworm:/workspace/build/isar_ub_devel/126/meta-
> > > test/recipes-core/images/isar-image-ci.bb:do_cache_deb_src)
> > > failed
> > > with exit code '1'
> > > [stdlog] 2025-03-20 01:19:35,822 avocado.test INFO | | DEBUG:
> > > Executing shell function do_cache_deb_src [stdlog] 2025-03-20
> > > 01:19:35,823 avocado.test INFO | | Reading package lists...
> > > [stdlog] 2025-03-20 01:19:35,823 avocado.test INFO | | E: You
> > > must
> > > put some 'deb-src' URIs in your sources.list ```
> > > 
> > > 
> > > Hi Uladzimir,
> > > 
> > >                       Thank you for the reply, I am not able to
> > > reproduce the build issue what you reported , I triggered the
> > > build
> > > for isar-image-ci target and it's getting passed.
> > > 
> > > This patch depend on Cedric patch [1]
> > > [1]:
> > > https://groups.google.com/g/isar-users/c/OwrSBmCPe0Y
> > > 
> > > I validated my patch along with the [1], it seems working fine.
> > > 
> > > Or Can you please provide the steps to reproduce the issue?
> > > 
> > 
> > I tried to find easy way to reproduce (e.g., one target case), but
> > didn't find it - build was OK. But it fails in CI - most probably,
> > due
> > to specific sequence of tests, that share sstate cache and download
> > directory between each other.
> > 
> > Similar behavior is reproduced in CI also with other patches, like
> > "deb-dl-dir: remove excessive calls to dpkg-deb in
> > debsrc_download".
> > Building some target alone is OK, but under CI it fails.
> 
> Ah damn it!
> 
> I had checked with ./scripts/ci_build.sh -T fast. I may have to try
> with the full CI (without -T fast)
> 
> Is the error exactly the same when you tried with my patch (and only
> that one)?

No, you patch I've mentioned failed in "cache-deb-src" task, but in
NoCrossTest.test_nocross_rpi_debsrc, and with a different error:

[stdlog] 2025-03-07 12:48:01,379 avocado.test INFO | | DEBUG: Executing
shell function cache_deb_src
[stdlog] 2025-03-07 12:48:01,379 avocado.test INFO | |
[stdlog] 2025-03-07 12:48:01,379 avocado.test INFO | | W: Unable to
read directory
/workspace/build/isar_ub_devel/119/build/tmp/work/raspios-bookworm-
armhf/isar-image-base-rpi-arm-v7l/1.0-
r0/rootfs/var/cache/apt/archives/partial
[stdlog] 2025-03-07 12:48:01,379 avocado.test INFO | | Reading package
lists...
[stdlog] 2025-03-07 12:48:01,379 avocado.test INFO | | E: Can not find
version '2023.3+deb12u1' of package 'debian-archive-keyring'
[stdlog] 2025-03-07 12:48:01,379 avocado.test INFO | | E: Unable to
find a source package for debian-archive-keyring


> 
> Would you be able to send a full CI log?
> 

Attached archive with both logs:

isar_ub_devel_119: full CI logs with "deb-dl-dir: remove
excessive calls to dpkg-deb in debsrc_download" applied.

isar_ub_devel_fast_214: fast CI logs with "image: move deb-src-
caching functionality after image creation" applied.


Some additional info.

For the latest patch, I tried to debug it:

- added `BASE_REPO_FEATURES = "cache-deb-src"` to local.conf (that is
the line CrossTest.test_cross_debsrc test adds; the line was removed in
the followed tests).
- chrooted to "isar-image-ci-qemuarm64" rootfs, where error happend.
- executed "apt-get -y --download-only --only-source source gcc-
12=12.2.0-14 download-src" - the same that failed in CI. 

Result:

```
# apt-get -y --download-only --only-source source gcc-12=12.2.0-14
download-src
Reading package lists... Done
E: You must put some 'deb-src' URIs in your sources.list
```

At the same time, sources.list looked fine:

```
# cat /etc/apt/sources.list.d/bootstrap.list 
deb     http://deb.debian.org/debian bookworm main contrib non-free-
firmware non-free
deb-src http://deb.debian.org/debian bookworm main contrib non-free-
firmware non-free

deb     http://deb.debian.org/debian-security bookworm-security main
contrib non-free-firmware non-free
deb-src http://deb.debian.org/debian-security bookworm-security main
contrib non-free-firmware non-free

deb     http://deb.debian.org/debian bookworm-updates main contrib non-
free-firmware non-free
deb-src http://deb.debian.org/debian bookworm-updates main contrib non-
free-firmware non-free
```

After executing "apt-get update" in this chroot, error gone.

It looks like despite proper sources list, internal apt cache was not
complete and didn't know anything about "deb-src" records. Why - I
still don't clearly understand.


> > 
> > > > Many thanks,
> > > > Srinu
> > > > 
> > > 
> > > --
> > > Best regards,
> > > Uladzimir.
> > 
> 
> --
> Cedric Hombourger
> Siemens AG
> http://www.siemens.com/


-- 
Best regards,
Uladzimir.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/def800632029c0f3f6903a77df4f7a12042badf2.camel%40ilbers.de.

[-- Attachment #2: ci_logs.tar.xz --]
[-- Type: application/x-xz-compressed-tar, Size: 7307956 bytes --]