Re: [PATCH v2 1/3] mmdebstrap: add "skip=cleanup/reproducible" option to mmdebstrap

["'cedric.hombourger@siemens.com' via isar-users" <isar-users@googlegroups.com>]

On Sat, 2025-03-08 at 18:25 +0000, Arjunan, Srinu (FT FDS CES LX PBU 2)
wrote:
>
>
> -----Original Message-----
> From: Hombourger, Cedric (FT FDS CES LX)
> <cedric.hombourger@siemens.com>
> Sent: 08 March 2025 19:15
> To: isar-users@googlegroups.com; Kiszka, Jan (FT RPD CED)
> <jan.kiszka@siemens.com>; Arjunan, Srinu (FT FDS CES LX PBU 2)
> <srinuvasan.a@siemens.com>
> Cc: ubely@ilbers.de; Adithya.Balakumar@toshiba-tsip.com
> Subject: Re: [PATCH v2 1/3] mmdebstrap: add
> "skip=cleanup/reproducible" option to mmdebstrap
>
> On Fri, 2025-03-07 at 22:59 +0100, Jan Kiszka wrote:
> > On 07.03.25 13:25, srinuvasan.a@siemens.com wrote:
> > > From: srinuvasan <srinuvasan.a@siemens.com>
> > >
> > > Without this option, mmdebstrap deletes the var/log/dpkg.log and
> > > its
> > > siblings by default, but this is needed to download deb-src
> > > packages
> > > for base-apt creation.
>
> I don't see where deb-src needs dpkg.log. Can you shed some light?
> Both the old code and the code I recently changed parse the .deb
> files that were downloaded to determine which source packages should
> be downloaded
>
> Ref:
> https://groups.google.com/g/isar-users/c/OwrSBmCPe0Y
>
> > > Basically, we are referring to the package's status under
> > > /var/log/dpkg.log file and performing the debsrc_download
> > > operation.
> > >
> > > Without these changes, bootstrap related deb-src files are not
> > > downloaded under the /downloads/deb-src folder, due to this the
> > > repo
> > > sanity check functions failed during base-apt caching stage.
>
> What are these repo sanity checks?
> Where do we see the failures?
>
> From my base-apt workdir:
>
> pwd && grep -ir error
> <...>/build/base-apt/tmp/work/debian-bookworm-any/base-apt/1.0-
> r0/temp
> run.repo.7287:          echo "ERROR: $*"
>
> We probably need to be more specific if we are mentioning a failure.
> Without this, people on this list will not understand what problem(s)
> you are fixing
>
>
> Hi cedric, jan
>
> I will explain the issue in the details below:
>
>                 To reproduce the issue ( Ref:
> https://github.com/ilbers/isar/blob/master/doc/user_manual.md#creation-of-local-apt-repo-caching-upstream-debian-packages
>  ):
>
>                 Trigger the warmup build to download deb and deb-src
> packages ( Enable the BASE_REPO_FEATURES = "cache-deb-src" in
> conf/local.conf)
>
>                 e.x: bitbake mc:qemuamd64-bookworm:isar-image-base
>
>                 Once all the deb and deb-src downloaded for the
> particular BSP under /download folder
>
>                 Trigger the base-apt target to cache all the deb and
> deb-src (Enable ISAR_USE_CACHED_BASE_REPO = "1" in conf/local.conf)
>
>                 e.x: bitbake base-apt
>
>                 You will see the failures:
>
>                 Logs:
>
>                 NOTE: Resolving any missing task queue dependencies
> Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 0 (0%
> match, 0%
> complete)#############################################            |
> ETA:  0:00:00
> Initialising tasks: 100%
> |####################################################################
> ################################################| Time: 0:00:00
> NOTE: No setscene tasks
> NOTE: Executing Tasks
> ERROR: base-apt-1.0-r0 do_cache: One or more sources are missing in
> repo. bookworm adduser 3.134
> pool/main/a/adduser/adduser_3.134_all.deb

.dsc found in our downstream project

> bookworm bmap-tools 3.6-2 pool/main/b/bmap-tools/bmap-tools_3.6-
> 2_all.deb
> bookworm debconf 1.5.82 pool/main/d/debconf/debconf_1.5.82_all.deb

.dsc found in our downstream project

> bookworm debian-archive-keyring 2023.3+deb12u1 pool/main/d/debian-
> archive-keyring/debian-archive-keyring_2023.3+deb12u1_all.deb
> bookworm audit 1:3.0.9-1 pool/main/a/audit/libaudit-common_3.0.9-
> 1_all.deb
> bookworm libfile-find-rule-perl 0.34-3 pool/main/libf/libfile-find-
> rule-perl/libfile-find-rule-perl_0.34-3_all.deb
> bookworm libnumber-compare-perl 0.03-3 pool/main/libn/libnumber-
> compare-perl/libnumber-compare-perl_0.03-3_all.deb
> bookworm pam 1.5.2-6+deb12u1 pool/main/p/pam/libpam-runtime_1.5.2-
> 6+deb12u1_all.deb


> bookworm libsemanage 3.4-1 pool/main/libs/libsemanage/libsemanage-
> common_3.4-1_all.deb
> bookworm libtext-glob-perl 0.11-3 pool/main/libt/libtext-glob-
> perl/libtext-glob-perl_0.11-3_all.deb
> bookworm perl 5.36.0-7+deb12u1 pool/main/p/perl/perl-modules-
> 5.36_5.36.0-7+deb12u1_all.deb
> bookworm six 1.16.0-4 pool/main/s/six/python3-six_1.16.0-4_all.deb
> bookworm tzdata 2024b-0+deb12u1 pool/main/t/tzdata/tzdata_2024b-
> 0+deb12u1_all.deb
> bookworm usrmerge 37~deb12u1
> pool/main/u/usrmerge/usrmerge_37~deb12u1_all.deb
> bookworm adduser 3.134 pool/main/a/adduser/adduser_3.134_all.deb
>
> Just i captured few packages, but many of the deb packages don't have
> their source files
>
> In the base-apt recipes we are calling the repo_sanity_test function
> finally to make sure all the deb's having
> the corresponding deb-src packages should be present in the base-apt.
>
>
> Yes, as you said, we are parsing deb files (from
> /var/cache/apt/archives) that were downloaded to determine which
> source packages should be downloaded.
> but in the function "is_not_part_of_current_build "${package}" we are
> skipping the download-src if parsing deb package (status installed)
> is not available in the below files
> ```
>             "${IMAGE_ROOTFS}"/var/log/dpkg.log \
>             "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \
>             "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \
>             "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \
>             "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log
> ```
>
> As I mentioned above, the adduser package was installed in the stage
> of bootstrap, but that package's status was not captured in the
> bootstrap's /var/log/dpkg.log file.
> because mmdebstrap deletes those files by default, hence we didn't
> capture those bootstrap package status.
>
> As we know bootstrap rootfs is used for sbuildchroot creation, but
> here whatever packages are installed during sbuild-chroot creation,
> ${SCHROOT_TARGET_DIR}"/var/log/dpkg.log this file only contains the
> sbuildchroot related package status, but our expectation is it should
> have
> bootstrap + sbuildchroot package status available in
> /var/log/dpkg.log file in sbuildchroot.
>
> My first patch solves the above issue
> with the introduction of "skip=cleanup/reproducible", the bootstrap
> rootfs generated with /var/log/dpkg.log files, which contain all the
> bootstrap package status
> along with that if you use that boostrap for sbuildchroot creation ,
> the sbuildchroot rootfs contains the existing bootstrap plus
> additionally installed sbuildchroot related packages status is
> available in sbuldchroot /var/log/dpkg.log file
>
>
> My second patch resolve the below issue:
>
> During image creation (IMAGE_FSTYPES = "wic") few packages were
> installed in the session (e.g: gdisk, bmap-tools) but those packages
> list available
> only in the sbuild environment, not outside, carry those packages
> status to tmp/dpkg_common.log in the rootfs to get the src packages
> for
> imager_install related deb packages, presently the cache-deb-src
> function calls in the ROOTFS_POSTPROCESS_COMMAND but we need to wait
> till imager generation
> That's why we pushed that deb-src-caching after image creation.
>
> My third path resolve the image rootfs related deb-src missing issue:
>
> By default we are removing the /var/log/dpkg.log file in the
> rootfs_postprocess_clean_log_files function, but this is needed when
> We enable the BASE_REPO_FEATURES = "cache-deb-src", hence making this
> function as a configurable one rather than enabled by default.
> to download IMAGE_ROOTFS related deb-src packages
>
> I hope I explain the issue, presently am checking the reproducible
> part (for jan queries), will update that part very soon.
>
> Many thanks,
> Srinu
>
> > >
> > > Signed-off-by: srinuvasan <srinuvasan.a@siemens.com>
> > > ---
> > >  meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 6 ++++++
> > >  1 file changed, 6 insertions(+)
> > >
> > > diff --git a/meta/recipes-core/isar-mmdebstrap/isar-
> > > mmdebstrap.inc
> > > b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> > > index 931f6f13..7380a658 100644
> > > --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> > > +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> > > @@ -168,6 +168,11 @@ do_bootstrap() {
> > >          arch_param="$arch_param,${COMPAT_DISTRO_ARCH}"
> > >      fi
> > >
> > > +    cleanup=
> > > +    if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-
> > > src', 'yes', 'no', d)}" = "yes" ]; then
> > > +        cleanup="--skip=cleanup/reproducible"
> > > +    fi
> > > +
> > >      # Cleanup mounts if fails
> > >      trap 'exit 1' INT HUP QUIT TERM ALRM USR1
> > >      trap '[ -r "${WORKDIR}/mmtmpdir" ] && tmpdir=$(cat
> > > "${WORKDIR}/mmtmpdir") \
> > > @@ -183,6 +188,7 @@ do_bootstrap() {
> > >
> > >      sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args
> > > \
> > >                     $arch_param \
> > > +                   $cleanup \
> > >                     --mode=unshare \
> > >                     ${MMHOOKS} \
> > >                     --setup-hook='mkdir -p
> > > "$1/var/cache/apt/archives/"' \
> >
> > What has changed in this version? I'm not seeing my concerns
> > addressed, my questions answered.
>
> +1
>
> Please make sure new revisions of your patch series include a
> changelog (use --cover-letter with git format-patch)
>
> Sure
>
> >
> > Jan
> >
>
>

--
Cedric Hombourger
Siemens AG
http://www.siemens.com/

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/64884fdb07b7d44ed1f676bdfeddc9b5135be417.camel%40siemens.com.