RE: [PATCH] image.bbclass: fix non-reproducible file time-stamps inside rootfs image

[<Venkata.Pyla@toshiba-tsip.com>]

>-----Original Message-----
>From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf
>Of Henning Schild
>Sent: 04 January 2023 00:36
>To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-
>tsip.com>
>Cc: isar-users@googlegroups.com; amikan@ilbers.de; jan.kiszka@siemens.com;
>hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>;
>dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>
>Subject: Re: [PATCH] image.bbclass: fix non-reproducible file time-stamps inside
>rootfs image
>
>Am Tue, 3 Jan 2023 14:10:14 +0000
>schrieb <Venkata.Pyla@toshiba-tsip.com>:
>
>> >-----Original Message-----
>> >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
>> >Behalf Of Henning Schild
>> >Sent: 02 January 2023 22:14
>> >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-
>> >tsip.com>
>> >Cc: isar-users@googlegroups.com; amikan@ilbers.de;
>> >jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ)
>> ><kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>> >Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH]
>> >image.bbclass: fix non-reproducible file time-stamps inside rootfs
>> >image
>> >
>> >Am Mon,  2 Jan 2023 20:28:28 +0530
>> >schrieb venkata.pyla@toshiba-tsip.com:
>> >
>> >> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
>> >>
>> >> As part of reproducible-build work, the rootfs images generated on
>> >> same source should be identical between two builds.
>> >>
>> >> In this commit it tries to solve one of the non-reproducible
>> >> problem i.e. the rootfs file time-stamps generated during build
>> >> time are not reproducible, it uses one of the solution provided in
>> >> the debian live-build image project (refer [1]), it fixes by
>> >> finding all the files/folders that are gernerated newly and set the
>> >> time-stamp provided by `SOURCE_DATE_EPOCH` environment variable.
>> >>
>> >> [1]
>> >> https://salsa.debian.org/live-team/live-build/-/merge_requests/218
>> >>
>> >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
>> >> ---
>> >>  meta/classes/image.bbclass | 9 +++++++++
>> >>  1 file changed, 9 insertions(+)
>> >>
>> >> diff --git a/meta/classes/image.bbclass
>> >> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644
>> >> --- a/meta/classes/image.bbclass
>> >> +++ b/meta/classes/image.bbclass
>> >> @@ -430,6 +430,15 @@ do_rootfs_finalize() {
>> >>              "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
>> >>
>> >>          rm -f "${ROOTFSDIR}/etc/apt/sources-list"
>> >> +
>> >> +        # Set same time-stamps to the newly generated file/folders
>> >> in the
>> >> +        # rootfs image for the purpose of reproducible builds.
>> >> +        test ! -z "${SOURCE_DATE_EPOCH}" && \
>> >> +            find ${ROOTFSDIR} -newermt \
>> >> +                "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d
>> >> %H:%M:%S')" \
>> >> +                -printf "%y %p\n" \
>> >> +                -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';'
>> >> +
>> >
>> >This looks like i have seen it before. For me that is _way_ too
>> >generic and something that is not a package touches files all over
>> >the place. If some package now wants to intentionally bring a file
>> >that is from a far away future?
>> >
>> >Which files are we talking about here? It can basically only be
>> >metadata and other little places where we violate our "everything
>> >comes from a package" rule.
>>
>> files/folder/symbolic-link that are modified or generated during build
>> time like /etc/os-release /etc/hostname .
>> .
>> .
>> /var/lib/dpkg/info/*
>> /var/cache/*
>> ---
>>
>> I have printed all the files that modified during build by executing
>> below command find ${ROOTFSDIR} -newermt "$(date
>> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t %y %p\n"
>>
>> modified_files_times.txt
>>
>> attached modified_files_times.txt for your reference, all these
>> files/folders/symbolic-link are generated or modified during build
>> time.
>
>So i am guessing it is about anything that comes out of postprocess functions
>and maintainer scripts like postinst ?

Yes, those are the files that are modified by the packages and have different timestamp on each build.
I think it is okay to set reproducible times-tamp for such files, as they are not coming from the packages but are modified at build time.

>
>> >
>> >Has this ever been tested against a complex layer, has any of the
>> >repro work ever looked at something bigger than the very artificial
>> >isar base image?
>> Similar implementations were used in the projects like
>> Debian/live-build[1] and poky[2]
>>
>> [1]
>> https://salsa.debian.org/live-team/live-build/-/blob/master/scripts/bu
>> ild/efi-image#L57
>> [2]
>> https://github.com/yoctoproject/poky/blob/master/meta/classes-recipe/i
>> mage.bbclass#L673
>
>I am talking about isar layers not other projects.
>
>> >
>> >I think a better start would be to bbwarn and only much later move to
>> >"-exec touch".
>> >
>> >We recently added CI tests for reproducible building. Would be nice
>> >to the two patches. p1 writes a test-case that goes red, p2 (this
>> >one) makes it go green
>> The patch (p1) already been sent [1], for writing test-case to verify
>> reproducible build problems, the test-case is getting failed (shows
>> non-reproducible time-stamp problem in build/diffoscope-output.txt)
>> when executed without this patch, and after applying this patch it
>> will still show fail (because there are other reproducible problems to
>> fix) but the non-reproducible timestamp problem in rootfs shows fixed.
>
>So the code is there but not used because it goes failed anyhow? In that case i
>would suggest to rewrite it in a way that it does not fail and can run in CI. It
>could for example ignore "known issues" and be enabled, where then this very
>series would remove a few "known issues"
>in p2 (p1 would be to introduce all currently known issues and enable that thing
>in CI) and p3 would be what we look at here

You mean run the test and just ignore the results?

>
>Not sure i understood but it sounds like we have a test that we only run
>manually, and i want it to run in CI. In fact a script to check for repro issues
>should be so generic that any layer can use it.

Eventually the repro-build test-case should go to CI, I will plan to do it.

Can I know where the isar CI pipe lines are running? I couldn't see in isar-github may be running some other place?

>
>So your vision would be that isar core brings that script and i.e.
>jailhouse-images can run it on all its images. And a few calls to the script will
>also be in testsuite/ and be called in pipelines regularly.
>
>Henning
>
>> [1]
>> https://github.com/ilbers/isar/commit/9698445eade76fab81e2dc16d4cb58b2
>> 4e953cb9
>>
>> >
>> >Henning
>> >
>> >>  EOSUDO
>> >>  }
>> >>  addtask rootfs_finalize before do_rootfs after
>> >> do_rootfs_postprocess
>> >
>> >--
>> >You received this message because you are subscribed to the Google
>> >Groups "isar-users" group.
>> >To unsubscribe from this group and stop receiving emails from it,
>> >send an email to isar-users+unsubscribe@googlegroups.com.
>> >To view this discussion on the web visit
>> >https://groups.google.com/d/msgid/isar-
>> >users/20230102174418.686715cf%40md1za8fc.ad001.siemens.net.
>
>--
>You received this message because you are subscribed to the Google Groups
>"isar-users" group.
>To unsubscribe from this group and stop receiving emails from it, send an email
>to isar-users+unsubscribe@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/isar-
>users/20230103200543.07e987ba%40md1za8fc.ad001.siemens.net.