RE: [PATCH v2 1/3] mmdebstrap: add "skip=cleanup/reproducible" option to mmdebstrap

["'Arjunan, Srinu' via isar-users" <isar-users@googlegroups.com>]

-----Original Message-----
From: Hombourger, Cedric (FT FDS CES LX) <cedric.hombourger@siemens.com>
Sent: 08 March 2025 19:15
To: isar-users@googlegroups.com; Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com>; Arjunan, Srinu (FT FDS CES LX PBU 2) <srinuvasan.a@siemens.com>
Cc: ubely@ilbers.de; Adithya.Balakumar@toshiba-tsip.com
Subject: Re: [PATCH v2 1/3] mmdebstrap: add "skip=cleanup/reproducible" option to mmdebstrap

On Fri, 2025-03-07 at 22:59 +0100, Jan Kiszka wrote:
> On 07.03.25 13:25, srinuvasan.a@siemens.com wrote:
> > From: srinuvasan <srinuvasan.a@siemens.com>
> >
> > Without this option, mmdebstrap deletes the var/log/dpkg.log and its
> > siblings by default, but this is needed to download deb-src packages
> > for base-apt creation.

I don't see where deb-src needs dpkg.log. Can you shed some light?
Both the old code and the code I recently changed parse the .deb files that were downloaded to determine which source packages should be downloaded

Ref: https://groups.google.com/g/isar-users/c/OwrSBmCPe0Y

> > Basically, we are referring to the package's status under
> > /var/log/dpkg.log file and performing the debsrc_download operation.
> >
> > Without these changes, bootstrap related deb-src files are not
> > downloaded under the /downloads/deb-src folder, due to this the repo
> > sanity check functions failed during base-apt caching stage.

What are these repo sanity checks?
Where do we see the failures?

From my base-apt workdir:

pwd && grep -ir error
<...>/build/base-apt/tmp/work/debian-bookworm-any/base-apt/1.0-r0/temp
run.repo.7287:          echo "ERROR: $*"

We probably need to be more specific if we are mentioning a failure.
Without this, people on this list will not understand what problem(s) you are fixing


Hi cedric, jan

I will explain the issue in the details below:

                To reproduce the issue ( Ref: https://github.com/ilbers/isar/blob/master/doc/user_manual.md#creation-of-local-apt-repo-caching-upstream-debian-packages ):

                Trigger the warmup build to download deb and deb-src packages ( Enable the BASE_REPO_FEATURES = "cache-deb-src" in conf/local.conf)

                e.x: bitbake mc:qemuamd64-bookworm:isar-image-base

                Once all the deb and deb-src downloaded for the particular BSP under /download folder

                Trigger the base-apt target to cache all the deb and deb-src (Enable ISAR_USE_CACHED_BASE_REPO = "1" in conf/local.conf)

                e.x: bitbake base-apt

                You will see the failures:

                Logs:

                NOTE: Resolving any missing task queue dependencies
Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 0 (0% match, 0% complete)#############################################            | ETA:  0:00:00
Initialising tasks: 100% |####################################################################################################################| Time: 0:00:00
NOTE: No setscene tasks
NOTE: Executing Tasks
ERROR: base-apt-1.0-r0 do_cache: One or more sources are missing in repo. bookworm adduser 3.134 pool/main/a/adduser/adduser_3.134_all.deb
bookworm bmap-tools 3.6-2 pool/main/b/bmap-tools/bmap-tools_3.6-2_all.deb
bookworm debconf 1.5.82 pool/main/d/debconf/debconf_1.5.82_all.deb
bookworm debian-archive-keyring 2023.3+deb12u1 pool/main/d/debian-archive-keyring/debian-archive-keyring_2023.3+deb12u1_all.deb
bookworm audit 1:3.0.9-1 pool/main/a/audit/libaudit-common_3.0.9-1_all.deb
bookworm libfile-find-rule-perl 0.34-3 pool/main/libf/libfile-find-rule-perl/libfile-find-rule-perl_0.34-3_all.deb
bookworm libnumber-compare-perl 0.03-3 pool/main/libn/libnumber-compare-perl/libnumber-compare-perl_0.03-3_all.deb
bookworm pam 1.5.2-6+deb12u1 pool/main/p/pam/libpam-runtime_1.5.2-6+deb12u1_all.deb
bookworm libsemanage 3.4-1 pool/main/libs/libsemanage/libsemanage-common_3.4-1_all.deb
bookworm libtext-glob-perl 0.11-3 pool/main/libt/libtext-glob-perl/libtext-glob-perl_0.11-3_all.deb
bookworm perl 5.36.0-7+deb12u1 pool/main/p/perl/perl-modules-5.36_5.36.0-7+deb12u1_all.deb
bookworm six 1.16.0-4 pool/main/s/six/python3-six_1.16.0-4_all.deb
bookworm tzdata 2024b-0+deb12u1 pool/main/t/tzdata/tzdata_2024b-0+deb12u1_all.deb
bookworm usrmerge 37~deb12u1 pool/main/u/usrmerge/usrmerge_37~deb12u1_all.deb
bookworm adduser 3.134 pool/main/a/adduser/adduser_3.134_all.deb

Just i captured few packages, but many of the deb packages don't have their source files

In the base-apt recipes we are calling the repo_sanity_test function finally to make sure all the deb's having
the corresponding deb-src packages should be present in the base-apt.


Yes, as you said, we are parsing deb files (from /var/cache/apt/archives) that were downloaded to determine which source packages should be downloaded.
but in the function "is_not_part_of_current_build "${package}" we are skipping the download-src if parsing deb package (status installed) is not available in the below files
```
            "${IMAGE_ROOTFS}"/var/log/dpkg.log \
            "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \
            "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \
            "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \
            "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log
```

As I mentioned above, the adduser package was installed in the stage of bootstrap, but that package's status was not captured in the bootstrap's /var/log/dpkg.log file.
because mmdebstrap deletes those files by default, hence we didn't capture those bootstrap package status.

As we know bootstrap rootfs is used for sbuildchroot creation, but here whatever packages are installed during sbuild-chroot creation,
${SCHROOT_TARGET_DIR}"/var/log/dpkg.log this file only contains the sbuildchroot related package status, but our expectation is it should have
bootstrap + sbuildchroot package status available in /var/log/dpkg.log file in sbuildchroot.

My first patch solves the above issue
with the introduction of "skip=cleanup/reproducible", the bootstrap rootfs generated with /var/log/dpkg.log files, which contain all the bootstrap package status
along with that if you use that boostrap for sbuildchroot creation , the sbuildchroot rootfs contains the existing bootstrap plus
additionally installed sbuildchroot related packages status is available in sbuldchroot /var/log/dpkg.log file


My second patch resolve the below issue:

During image creation (IMAGE_FSTYPES = "wic") few packages were installed in the session (e.g: gdisk, bmap-tools) but those packages list available
only in the sbuild environment, not outside, carry those packages status to tmp/dpkg_common.log in the rootfs to get the src packages for
imager_install related deb packages, presently the cache-deb-src function calls in the ROOTFS_POSTPROCESS_COMMAND but we need to wait till imager generation
That's why we pushed that deb-src-caching after image creation.

My third path resolve the image rootfs related deb-src missing issue:

By default we are removing the /var/log/dpkg.log file in the rootfs_postprocess_clean_log_files function, but this is needed when
We enable the BASE_REPO_FEATURES = "cache-deb-src", hence making this function as a configurable one rather than enabled by default.
to download IMAGE_ROOTFS related deb-src packages

I hope I explain the issue, presently am checking the reproducible part (for jan queries), will update that part very soon.

Many thanks,
Srinu

> >
> > Signed-off-by: srinuvasan <srinuvasan.a@siemens.com>
> > ---
> >  meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 6 ++++++
> >  1 file changed, 6 insertions(+)
> >
> > diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> > b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> > index 931f6f13..7380a658 100644
> > --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> > +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> > @@ -168,6 +168,11 @@ do_bootstrap() {
> >          arch_param="$arch_param,${COMPAT_DISTRO_ARCH}"
> >      fi
> >
> > +    cleanup=
> > +    if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-
> > src', 'yes', 'no', d)}" = "yes" ]; then
> > +        cleanup="--skip=cleanup/reproducible"
> > +    fi
> > +
> >      # Cleanup mounts if fails
> >      trap 'exit 1' INT HUP QUIT TERM ALRM USR1
> >      trap '[ -r "${WORKDIR}/mmtmpdir" ] && tmpdir=$(cat
> > "${WORKDIR}/mmtmpdir") \
> > @@ -183,6 +188,7 @@ do_bootstrap() {
> >
> >      sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \
> >                     $arch_param \
> > +                   $cleanup \
> >                     --mode=unshare \
> >                     ${MMHOOKS} \
> >                     --setup-hook='mkdir -p
> > "$1/var/cache/apt/archives/"' \
>
> What has changed in this version? I'm not seeing my concerns
> addressed, my questions answered.

+1

Please make sure new revisions of your patch series include a changelog (use --cover-letter with git format-patch)

Sure

>
> Jan
>


-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/PUZPR06MB57479349B38D83094DD2821A88D42%40PUZPR06MB5747.apcprd06.prod.outlook.com.