Re: [PATCH] image.bbclass: fix non-reproducible file time-stamps inside rootfs image

[Henning Schild <henning.schild@siemens.com>]

Am Tue, 3 Jan 2023 14:10:14 +0000
schrieb <Venkata.Pyla@toshiba-tsip.com>:

> >-----Original Message-----
> >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
> >Behalf Of Henning Schild
> >Sent: 02 January 2023 22:14
> >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-  
> >tsip.com>  
> >Cc: isar-users@googlegroups.com; amikan@ilbers.de;
> >jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ)
> ><kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG
> >Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH]
> >image.bbclass: fix non-reproducible file time-stamps inside rootfs
> >image
> >
> >Am Mon,  2 Jan 2023 20:28:28 +0530
> >schrieb venkata.pyla@toshiba-tsip.com:
> >  
> >> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> >>
> >> As part of reproducible-build work, the rootfs images generated on
> >> same source should be identical between two builds.
> >>
> >> In this commit it tries to solve one of the non-reproducible
> >> problem i.e. the rootfs file time-stamps generated during build
> >> time are not reproducible, it uses one of the solution provided in
> >> the debian live-build image project (refer [1]), it fixes by
> >> finding all the files/folders that are gernerated newly and set
> >> the time-stamp provided by `SOURCE_DATE_EPOCH` environment
> >> variable.
> >>
> >> [1]
> >> https://salsa.debian.org/live-team/live-build/-/merge_requests/218
> >>
> >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> >> ---
> >>  meta/classes/image.bbclass | 9 +++++++++
> >>  1 file changed, 9 insertions(+)
> >>
> >> diff --git a/meta/classes/image.bbclass
> >> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644
> >> --- a/meta/classes/image.bbclass
> >> +++ b/meta/classes/image.bbclass
> >> @@ -430,6 +430,15 @@ do_rootfs_finalize() {
> >>              "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> >>
> >>          rm -f "${ROOTFSDIR}/etc/apt/sources-list"
> >> +
> >> +        # Set same time-stamps to the newly generated file/folders
> >> in the
> >> +        # rootfs image for the purpose of reproducible builds.
> >> +        test ! -z "${SOURCE_DATE_EPOCH}" && \
> >> +            find ${ROOTFSDIR} -newermt \
> >> +                "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d
> >> %H:%M:%S')" \
> >> +                -printf "%y %p\n" \
> >> +                -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';'
> >> +  
> >
> >This looks like i have seen it before. For me that is _way_ too
> >generic and something that is not a package touches files all over
> >the place. If some package now wants to intentionally bring a file
> >that is from a far away future?
> >
> >Which files are we talking about here? It can basically only be
> >metadata and other little places where we violate our "everything
> >comes from a package" rule.  
> 
> files/folder/symbolic-link that are modified or generated during
> build time like /etc/os-release
> /etc/hostname
> .
> .
> .
> /var/lib/dpkg/info/*
> /var/cache/*
> ---
> 
> I have printed all the files that modified during build by executing
> below command find ${ROOTFSDIR} -newermt "$(date
> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t %y %p\n" >
> modified_files_times.txt
> 
> attached modified_files_times.txt for your reference, all these
> files/folders/symbolic-link are generated or modified during build
> time.

So i am guessing it is about anything that comes out of postprocess
functions and maintainer scripts like postinst ?

> >
> >Has this ever been tested against a complex layer, has any of the
> >repro work ever looked at something bigger than the very artificial
> >isar base image?  
> Similar implementations were used in the projects like
> Debian/live-build[1] and poky[2]
> 
> [1]
> https://salsa.debian.org/live-team/live-build/-/blob/master/scripts/build/efi-image#L57
> [2]
> https://github.com/yoctoproject/poky/blob/master/meta/classes-recipe/image.bbclass#L673 

I am talking about isar layers not other projects.

> >
> >I think a better start would be to bbwarn and only much later move
> >to "-exec touch".
> >
> >We recently added CI tests for reproducible building. Would be nice
> >to the two patches. p1 writes a test-case that goes red, p2 (this
> >one) makes it go green  
> The patch (p1) already been sent [1], for writing test-case to verify
> reproducible build problems, the test-case is getting failed (shows
> non-reproducible time-stamp problem in build/diffoscope-output.txt)
> when executed without this patch, and after applying this patch it
> will still show fail (because there are other reproducible problems
> to fix) but the non-reproducible timestamp problem in rootfs shows
> fixed.

So the code is there but not used because it goes failed anyhow? In
that case i would suggest to rewrite it in a way that it does not fail
and can run in CI. It could for example ignore "known issues" and be
enabled, where then this very series would remove a few "known issues"
in p2 (p1 would be to introduce all currently known issues and enable
that thing in CI) and p3 would be what we look at here

Not sure i understood but it sounds like we have a test that we only
run manually, and i want it to run in CI. In fact a script to check for
repro issues should be so generic that any layer can use it.

So your vision would be that isar core brings that script and i.e.
jailhouse-images can run it on all its images. And a few calls to the
script will also be in testsuite/ and be called in pipelines regularly.

Henning

> [1]
> https://github.com/ilbers/isar/commit/9698445eade76fab81e2dc16d4cb58b24e953cb9 
>
> >
> >Henning
> >  
> >>  EOSUDO
> >>  }
> >>  addtask rootfs_finalize before do_rootfs after
> >> do_rootfs_postprocess  
> >
> >--
> >You received this message because you are subscribed to the Google
> >Groups "isar-users" group.
> >To unsubscribe from this group and stop receiving emails from it,
> >send an email to isar-users+unsubscribe@googlegroups.com.
> >To view this discussion on the web visit
> >https://groups.google.com/d/msgid/isar-
> >users/20230102174418.686715cf%40md1za8fc.ad001.siemens.net.