Re: [PATCH v2 1/3] mmdebstrap: add "skip=cleanup/reproducible" option to mmdebstrap

["'cedric.hombourger@siemens.com' via isar-users" <isar-users@googlegroups.com>]

On Sun, 2025-03-09 at 08:46 +0000, Arjunan, Srinu (FT FDS CES LX PBU 2)
wrote:
> 
> 
> -----Original Message-----
> From: Hombourger, Cedric (FT FDS CES LX)
> <cedric.hombourger@siemens.com> 
> Sent: 09 March 2025 00:46
> To: isar-users@googlegroups.com; Kiszka, Jan (FT RPD CED)
> <jan.kiszka@siemens.com>; Arjunan, Srinu (FT FDS CES LX PBU 2)
> <srinuvasan.a@siemens.com>
> Cc: ubely@ilbers.de; Adithya.Balakumar@toshiba-tsip.com
> Subject: Re: [PATCH v2 1/3] mmdebstrap: add
> "skip=cleanup/reproducible" option to mmdebstrap
> 
> On Sat, 2025-03-08 at 18:25 +0000, Arjunan, Srinu (FT FDS CES LX PBU
> 2)
> wrote:
> > 
> > 
> > -----Original Message-----
> > From: Hombourger, Cedric (FT FDS CES LX) 
> > <cedric.hombourger@siemens.com>
> > Sent: 08 March 2025 19:15
> > To: isar-users@googlegroups.com; Kiszka, Jan (FT RPD CED) 
> > <jan.kiszka@siemens.com>; Arjunan, Srinu (FT FDS CES LX PBU 2) 
> > <srinuvasan.a@siemens.com>
> > Cc: ubely@ilbers.de; Adithya.Balakumar@toshiba-tsip.com
> > Subject: Re: [PATCH v2 1/3] mmdebstrap: add 
> > "skip=cleanup/reproducible" option to mmdebstrap
> > 
> > On Fri, 2025-03-07 at 22:59 +0100, Jan Kiszka wrote:
> > > On 07.03.25 13:25, srinuvasan.a@siemens.com wrote:
> > > > From: srinuvasan <srinuvasan.a@siemens.com>
> > > > 
> > > > Without this option, mmdebstrap deletes the var/log/dpkg.log
> > > > and 
> > > > its siblings by default, but this is needed to download deb-src
> > > > packages for base-apt creation.
> > 
> > I don't see where deb-src needs dpkg.log. Can you shed some light?
> > Both the old code and the code I recently changed parse the .deb
> > files 
> > that were downloaded to determine which source packages should be 
> > downloaded
> > 
> > Ref:
> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgrou%2F&data=05%7C02%7Ccedric.hombourger%40siemens.com%7C42bcccafcd8f42e5f6d308dd5ee6f2b3%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C638771068307563689%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kPyQeXBw6pYTZng1OieFPozM1xn9YgRdyuC%2FSfGK7o0%3D&reserved=0
> > ps.google.com%2Fg%2Fisar-
> > users%2Fc%2FOwrSBmCPe0Y&data=05%7C02%7Csrinuv
> > asan.a%40siemens.com%7C9492cda997b94dc4e81708dd5e759d85%7C38ae3bcd9
> > 579
> > 4fd4addab42e1495d55a%7C1%7C0%7C638770581452433346%7CUnknown%7CTWFpb
> > GZs
> > b3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkF
> > OIj
> > oiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=b49oZTatP9h6CxEys%2BP
> > v%2
> > F%2F%2F5v2BmE%2FX1%2FdhDPs%2F1qOY%3D&reserved=0
> > 
> > > > Basically, we are referring to the package's status under 
> > > > /var/log/dpkg.log file and performing the debsrc_download 
> > > > operation.
> > > > 
> > > > Without these changes, bootstrap related deb-src files are not 
> > > > downloaded under the /downloads/deb-src folder, due to this the
> > > > repo sanity check functions failed during base-apt caching
> > > > stage.
> > 
> > What are these repo sanity checks?
> > Where do we see the failures?
> > 
> > From my base-apt workdir:
> > 
> > pwd && grep -ir error
> > <...>/build/base-apt/tmp/work/debian-bookworm-any/base-apt/1.0-
> > r0/temp
> > run.repo.7287:          echo "ERROR: $*"
> > 
> > We probably need to be more specific if we are mentioning a
> > failure.
> > Without this, people on this list will not understand what
> > problem(s) 
> > you are fixing
> > 
> > 
> > Hi cedric, jan
> > 
> > I will explain the issue in the details below:
> > 
> >                 To reproduce the issue ( Ref:
> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith%2F&data=05%7C02%7Ccedric.hombourger%40siemens.com%7C42bcccafcd8f42e5f6d308dd5ee6f2b3%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C638771068307586848%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=AUH81pXJwMNzrIIkLA3GkJ2fLWwL9jBFGeltINWR41g%3D&reserved=0
> > ub.com%2Filbers%2Fisar%2Fblob%2Fmaster%2Fdoc%2Fuser_manual.md%23cre
> > ati
> > on-of-local-apt-repo-caching-upstream-debian-
> > packages&data=05%7C02%7Cs
> > rinuvasan.a%40siemens.com%7C9492cda997b94dc4e81708dd5e759d85%7C38ae
> > 3bc
> > d95794fd4addab42e1495d55a%7C1%7C0%7C638770581452457810%7CUnknown%7C
> > TWF
> > pbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMi
> > IsI
> > kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=3XId9a7IuBK9xyzY
> > Wnu
> > dmeG0aKe3awlNSGSmwId7D%2BU%3D&reserved=0
> >  ):
> > 
> >                 Trigger the warmup build to download deb and deb-
> > src 
> > packages ( Enable the BASE_REPO_FEATURES = "cache-deb-src" in
> > conf/local.conf)
> > 
> >                 e.x: bitbake mc:qemuamd64-bookworm:isar-image-base
> > 
> >                 Once all the deb and deb-src downloaded for the 
> > particular BSP under /download folder
> > 
> >                 Trigger the base-apt target to cache all the deb
> > and 
> > deb-src (Enable ISAR_USE_CACHED_BASE_REPO = "1" in conf/local.conf)
> > 
> >                 e.x: bitbake base-apt
> > 
> >                 You will see the failures:
> >                 
> >                 Logs: 
> >                 
> >                 NOTE: Resolving any missing task queue dependencies
> > Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 0 (0% 
> > match, 0%
> > complete)#############################################            
> > > 
> > ETA:  0:00:00
> > Initialising tasks: 100%
> > > #################################################################
> > > ###
> > ################################################| Time: 0:00:00
> > NOTE: No setscene tasks
> > NOTE: Executing Tasks
> > ERROR: base-apt-1.0-r0 do_cache: One or more sources are missing in
> > repo. bookworm adduser 3.134
> > pool/main/a/adduser/adduser_3.134_all.deb
> > bookworm bmap-tools 3.6-2 pool/main/b/bmap-tools/bmap-tools_3.6-
> > 2_all.deb
> > bookworm debconf 1.5.82 pool/main/d/debconf/debconf_1.5.82_all.deb
> > bookworm debian-archive-keyring 2023.3+deb12u1 pool/main/d/debian- 
> > archive-keyring/debian-archive-keyring_2023.3+deb12u1_all.deb
> > bookworm audit 1:3.0.9-1 pool/main/a/audit/libaudit-common_3.0.9-
> > 1_all.deb
> > bookworm libfile-find-rule-perl 0.34-3 pool/main/libf/libfile-find-
> > rule-perl/libfile-find-rule-perl_0.34-3_all.deb
> > bookworm libnumber-compare-perl 0.03-3 pool/main/libn/libnumber- 
> > compare-perl/libnumber-compare-perl_0.03-3_all.deb
> > bookworm pam 1.5.2-6+deb12u1 pool/main/p/pam/libpam-runtime_1.5.2-
> > 6+deb12u1_all.deb
> 
> .dsc for pam packages found in our downstream project
> 
> Yes cedric, validated from my end, it is available in downstream
> project because we have your changes in isar-patches
> (https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgr
> oups.google.com%2Fg%2Fisar-
> users%2Fc%2FOwrSBmCPe0Y&data=05%7C02%7Ccedric.hombourger%40siemens.co
> m%7C42bcccafcd8f42e5f6d308dd5ee6f2b3%7C38ae3bcd95794fd4addab42e1495d5
> 5a%7C1%7C0%7C638771068307601863%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hc
> GkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIj
> oyfQ%3D%3D%7C0%7C%7C%7C&sdata=OoUVTbJLnDaPdXKYmZmOh0Wxn9HpiRis%2FYRaL
> YL2Khs%3D&reserved=0)
> 
> But still we can reproduce the issue in vanilla ISAR without your
> patch, I started working on these changes before you proposed these
> changes
> (https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgr
> oups.google.com%2Fg%2Fisar-
> users%2Fc%2FOwrSBmCPe0Y&data=05%7C02%7Ccedric.hombourger%40siemens.co
> m%7C42bcccafcd8f42e5f6d308dd5ee6f2b3%7C38ae3bcd95794fd4addab42e1495d5
> 5a%7C1%7C0%7C638771068307616236%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hc
> GkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIj
> oyfQ%3D%3D%7C0%7C%7C%7C&sdata=IknRwnuSOZiNUPnx43d4W4dJd2vmylKoUyp%2Bp
> T4W6OE%3D&reserved=0)
> 
> Basically, your patches remove the is_not_part_of_current_build()
> function
> And refactor the debsrc_download functionality.
> 
>  My patch becomes obsoleted if your
> (https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgr
> oups.google.com%2Fg%2Fisar-
> users%2Fc%2FOwrSBmCPe0Y&data=05%7C02%7Ccedric.hombourger%40siemens.co
> m%7C42bcccafcd8f42e5f6d308dd5ee6f2b3%7C38ae3bcd95794fd4addab42e1495d5
> 5a%7C1%7C0%7C638771068307630537%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hc
> GkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIj
> oyfQ%3D%3D%7C0%7C%7C%7C&sdata=vkjpy98ZFZZo4WgKjtwj7UREcd%2B2J4EBcWliJ
> eIdHbU%3D&reserved=0) changes are going to merge in ISAR.
> 
> Please provide your thoughts?


Well the future of my patch and yours are more for the Isar maintainers
to decide. They definitely conflict with each other.

> 
> > bookworm libsemanage 3.4-1 pool/main/libs/libsemanage/libsemanage-
> > common_3.4-1_all.deb
> > bookworm libtext-glob-perl 0.11-3 pool/main/libt/libtext-glob- 
> > perl/libtext-glob-perl_0.11-3_all.deb
> > bookworm perl 5.36.0-7+deb12u1 pool/main/p/perl/perl-modules- 
> > 5.36_5.36.0-7+deb12u1_all.deb bookworm six 1.16.0-4 
> > pool/main/s/six/python3-six_1.16.0-4_all.deb
> > bookworm tzdata 2024b-0+deb12u1 pool/main/t/tzdata/tzdata_2024b-
> > 0+deb12u1_all.deb
> > bookworm usrmerge 37~deb12u1
> > pool/main/u/usrmerge/usrmerge_37~deb12u1_all.deb
> > bookworm adduser 3.134 pool/main/a/adduser/adduser_3.134_all.deb
> > 
> > Just i captured few packages, but many of the deb packages don't
> > have 
> > their source files
> > 
> > In the base-apt recipes we are calling the repo_sanity_test
> > function 
> > finally to make sure all the deb's having the corresponding deb-src
> > packages should be present in the base-apt.
> 
> I checked again log.do_cache in our downstream project and I did not
> see any of the mentioned errors.
> 
> In downstream project we are not at all calling the repo_sanity_test
> function due to enabled the cache-deb-src in
> ROOTFS_POSTPROCESS_COMMAND rather than enabled in conf/local.conf
> 
> The below condition fails:
> if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src',
> 'yes', 'no', d)}" = "yes" ]
> 
> that is the reason we are not calling this statement at all "
> reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing "
> 
> because we are not enabled the cache-deb-src via BASE_REPO_FEATURES
> 
> 
> $ grep -v 'Export' ./tmp/work/debian-bookworm-any/base-apt/1.0-
> r0/temp/log.do_cache
> DEBUG: Executing python function do_cache
> DEBUG: Using absolute /build/../repo/gnupg/205D601E.pub
> DEBUG: Executing shell function repo
> DEBUG: Shell function repo finished
> DEBUG: Python function do_cache finished
> 
> That build was however done with the patch I have submitted earlier
> to this list. It also includes our own layer. I will do a plain Isar
> build (with my patch) to check if the issue is reproducible.
> 
> I took a few error cases from your example above and verified that
> the .dsc files are present in the generated base-apt build.
> 
> It should be noted that my patch removes
> "is_not_part_of_current_build()"
> 
> parsing dpkg.log was IMO not a great idea as I don't think its format
> is officially documented (I could be wrong though)
> 
> > 
> >                 
> > Yes, as you said, we are parsing deb files (from
> > /var/cache/apt/archives) that were downloaded to determine which 
> > source packages should be downloaded.
> > but in the function "is_not_part_of_current_build "${package}" we
> > are 
> > skipping the download-src if parsing deb package (status installed)
> > is 
> > not available in the below files ```
> >             "${IMAGE_ROOTFS}"/var/log/dpkg.log \
> >             "${SCHROOT_HOST_DIR}"/var/log/dpkg.log \
> >             "${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \
> >             "${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \
> >             "${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log
> > ```
> > 
> > As I mentioned above, the adduser package was installed in the
> > stage 
> > of bootstrap, but that package's status was not captured in the 
> > bootstrap's /var/log/dpkg.log file.
> > because mmdebstrap deletes those files by default, hence we didn't 
> > capture those bootstrap package status.
> > 
> > As we know bootstrap rootfs is used for sbuildchroot creation, but 
> > here whatever packages are installed during sbuild-chroot creation,
> > ${SCHROOT_TARGET_DIR}"/var/log/dpkg.log this file only contains the
> > sbuildchroot related package status, but our expectation is it
> > should 
> > have bootstrap + sbuildchroot package status available in 
> > /var/log/dpkg.log file in sbuildchroot.
> > 
> > My first patch solves the above issue
> > with the introduction of "skip=cleanup/reproducible", the bootstrap
> > rootfs generated with /var/log/dpkg.log files, which contain all
> > the 
> > bootstrap package status along with that if you use that boostrap
> > for 
> > sbuildchroot creation , the sbuildchroot rootfs contains the
> > existing 
> > bootstrap plus additionally installed sbuildchroot related packages
> > status is available in sbuldchroot /var/log/dpkg.log file
> > 
> > 
> > My second patch resolve the below issue:
> > 
> > During image creation (IMAGE_FSTYPES = "wic") few packages were 
> > installed in the session (e.g: gdisk, bmap-tools) but those
> > packages 
> > list available only in the sbuild environment, not outside, carry 
> > those packages status to tmp/dpkg_common.log in the rootfs to get
> > the 
> > src packages for imager_install related deb packages, presently the
> > cache-deb-src function calls in the ROOTFS_POSTPROCESS_COMMAND but
> > we 
> > need to wait till imager generation That's why we pushed that 
> > deb-src-caching after image creation.
> > 
> > My third path resolve the image rootfs related deb-src missing
> > issue:
> > 
> > By default we are removing the /var/log/dpkg.log file in the 
> > rootfs_postprocess_clean_log_files function, but this is needed
> > when 
> > We enable the BASE_REPO_FEATURES = "cache-deb-src", hence making
> > this 
> > function as a configurable one rather than enabled by default.
> > to download IMAGE_ROOTFS related deb-src packages
> > 
> > I hope I explain the issue, presently am checking the reproducible 
> > part (for jan queries), will update that part very soon.
> > 
> > Many thanks,
> > Srinu
> > 
> > > > 
> > > > Signed-off-by: srinuvasan <srinuvasan.a@siemens.com>
> > > > ---
> > > >  meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 6
> > > > ++++++
> > > >  1 file changed, 6 insertions(+)
> > > > 
> > > > diff --git a/meta/recipes-core/isar-mmdebstrap/isar-
> > > > mmdebstrap.inc
> > > > b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> > > > index 931f6f13..7380a658 100644
> > > > --- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> > > > +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
> > > > @@ -168,6 +168,11 @@ do_bootstrap() {
> > > >          arch_param="$arch_param,${COMPAT_DISTRO_ARCH}"
> > > >      fi
> > > >  
> > > > +    cleanup=
> > > > +    if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-
> > > > deb-
> > > > src', 'yes', 'no', d)}" = "yes" ]; then
> > > > +        cleanup="--skip=cleanup/reproducible"
> > > > +    fi
> > > > +
> > > >      # Cleanup mounts if fails
> > > >      trap 'exit 1' INT HUP QUIT TERM ALRM USR1
> > > >      trap '[ -r "${WORKDIR}/mmtmpdir" ] && tmpdir=$(cat
> > > > "${WORKDIR}/mmtmpdir") \
> > > > @@ -183,6 +188,7 @@ do_bootstrap() {
> > > >  
> > > >      sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap
> > > > $bootstrap_args 
> > > > \
> > > >                     $arch_param \
> > > > +                   $cleanup \
> > > >                     --mode=unshare \
> > > >                     ${MMHOOKS} \
> > > >                     --setup-hook='mkdir -p 
> > > > "$1/var/cache/apt/archives/"' \
> > > 
> > > What has changed in this version? I'm not seeing my concerns 
> > > addressed, my questions answered.
> > 
> > +1
> > 
> > Please make sure new revisions of your patch series include a 
> > changelog (use --cover-letter with git format-patch)
> > 
> > Sure
> > 
> > > 
> > > Jan
> > > 
> > 
> > 
> 
> --
> Cedric Hombourger
> Siemens AG
> www.siemens.com

-- 
Cedric Hombourger
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/ac437554de180e0465e53fd3266684e822875bbd.camel%40siemens.com.