RE: [PATCH] image.bbclass: fix non-reproducible file time-stamps inside rootfs image

[<Venkata.Pyla@toshiba-tsip.com>]

>-----Original Message-----
>From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf
>Of Henning Schild
>Sent: 04 January 2023 15:00
>To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-
>tsip.com>
>Cc: isar-users@googlegroups.com; amikan@ilbers.de; jan.kiszka@siemens.com;
>hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>;
>dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>;
>Baurzhan Ismagulov <ibr@radix50.net>
>Subject: Re: [PATCH] image.bbclass: fix non-reproducible file time-stamps inside
>rootfs image
>
>Am Wed, 4 Jan 2023 07:54:44 +0000
>schrieb <Venkata.Pyla@toshiba-tsip.com>:
>
>> >-----Original Message-----
>> >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
>> >Behalf Of Henning Schild
>> >Sent: 04 January 2023 00:36
>> >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-
>> >tsip.com>
>> >Cc: isar-users@googlegroups.com; amikan@ilbers.de;
>> >jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ)
>> ><kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>> >Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH]
>> >image.bbclass: fix non-reproducible file time-stamps inside rootfs
>> >image
>> >
>> >Am Tue, 3 Jan 2023 14:10:14 +0000
>> >schrieb <Venkata.Pyla@toshiba-tsip.com>:
>> >
>> >> >-----Original Message-----
>> >> >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
>> >> >Behalf Of Henning Schild
>> >> >Sent: 02 January 2023 22:14
>> >> >To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting)
>> >> ><Venkata.Pyla@toshiba- tsip.com>
>> >> >Cc: isar-users@googlegroups.com; amikan@ilbers.de;
>> >> >jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ)
>> >> ><kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>> >> >Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH]
>> >> >image.bbclass: fix non-reproducible file time-stamps inside rootfs
>> >> >image
>> >> >
>> >> >Am Mon,  2 Jan 2023 20:28:28 +0530 schrieb
>> >> >venkata.pyla@toshiba-tsip.com:
>> >> >
>> >> >> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
>> >> >>
>> >> >> As part of reproducible-build work, the rootfs images generated
>> >> >> on same source should be identical between two builds.
>> >> >>
>> >> >> In this commit it tries to solve one of the non-reproducible
>> >> >> problem i.e. the rootfs file time-stamps generated during build
>> >> >> time are not reproducible, it uses one of the solution provided
>> >> >> in the debian live-build image project (refer [1]), it fixes by
>> >> >> finding all the files/folders that are gernerated newly and set
>> >> >> the time-stamp provided by `SOURCE_DATE_EPOCH` environment
>> >> >> variable.
>> >> >>
>> >> >> [1]
>> >> >> https://salsa.debian.org/live-team/live-build/-/merge_requests/2
>> >> >> 18
>> >> >>
>> >> >> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
>> >> >> ---
>> >> >>  meta/classes/image.bbclass | 9 +++++++++
>> >> >>  1 file changed, 9 insertions(+)
>> >> >>
>> >> >> diff --git a/meta/classes/image.bbclass
>> >> >> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644
>> >> >> --- a/meta/classes/image.bbclass
>> >> >> +++ b/meta/classes/image.bbclass
>> >> >> @@ -430,6 +430,15 @@ do_rootfs_finalize() {
>> >> >>              "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
>> >> >>
>> >> >>          rm -f "${ROOTFSDIR}/etc/apt/sources-list"
>> >> >> +
>> >> >> +        # Set same time-stamps to the newly generated
>> >> >> file/folders in the
>> >> >> +        # rootfs image for the purpose of reproducible builds.
>> >> >> +        test ! -z "${SOURCE_DATE_EPOCH}" && \
>> >> >> +            find ${ROOTFSDIR} -newermt \
>> >> >> +                "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d
>> >> >> %H:%M:%S')" \
>> >> >> +                -printf "%y %p\n" \
>> >> >> +                -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';'
>> >> >> +
>> >> >
>> >> >This looks like i have seen it before. For me that is _way_ too
>> >> >generic and something that is not a package touches files all over
>> >> >the place. If some package now wants to intentionally bring a file
>> >> >that is from a far away future?
>> >> >
>> >> >Which files are we talking about here? It can basically only be
>> >> >metadata and other little places where we violate our "everything
>> >> >comes from a package" rule.
>> >>
>> >> files/folder/symbolic-link that are modified or generated during
>> >> build time like /etc/os-release /etc/hostname .
>> >> .
>> >> .
>> >> /var/lib/dpkg/info/*
>> >> /var/cache/*
>> >> ---
>> >>
>> >> I have printed all the files that modified during build by
>> >> executing below command find ${ROOTFSDIR} -newermt "$(date
>> >> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t %y
>%p\n"
>> >>
>> >> modified_files_times.txt
>> >>
>> >> attached modified_files_times.txt for your reference, all these
>> >> files/folders/symbolic-link are generated or modified during build
>> >> time.
>> >
>> >So i am guessing it is about anything that comes out of postprocess
>> >functions and maintainer scripts like postinst ?
>>
>> Yes, those are the files that are modified by the packages and have
>> different timestamp on each build. I think it is okay to set
>> reproducible times-tamp for such files, as they are not coming from
>> the packages but are modified at build time.
>
>Sure they are OK. But if i read your patch correctly you will potentially adjust
>times on ALL files, not just some for which it might be OK.
>
>And it uses a variable which does not have a default, should we not set that to
>something? Or help users to choose a good value for it.

Setting default value, I didn't think of it because reproducible builds may not be required for regular builds, however when someone wants it can be enable by setting the value to it.
So as you mentioned it is good to mention somewhere to help users to choose good value for it,
I am thinking to add it in doc/user_manual.md

>
>Without this variable and a clue on a value to pick, this is all dead code.
>
>> >
>> >> >
>> >> >Has this ever been tested against a complex layer, has any of the
>> >> >repro work ever looked at something bigger than the very
>> >> >artificial isar base image?
>> >> Similar implementations were used in the projects like
>> >> Debian/live-build[1] and poky[2]
>> >>
>> >> [1]
>> >> https://salsa.debian.org/live-team/live-build/-/blob/master/scripts
>> >> /bu
>> >> ild/efi-image#L57
>> >> [2]
>> >> https://github.com/yoctoproject/poky/blob/master/meta/classes-recip
>> >> e/i
>> >> mage.bbclass#L673
>> >
>> >I am talking about isar layers not other projects.
>> >
>> >> >
>> >> >I think a better start would be to bbwarn and only much later move
>> >> >to "-exec touch".
>> >> >
>> >> >We recently added CI tests for reproducible building. Would be
>> >> >nice to the two patches. p1 writes a test-case that goes red, p2
>> >> >(this one) makes it go green
>> >> The patch (p1) already been sent [1], for writing test-case to
>> >> verify reproducible build problems, the test-case is getting failed
>> >> (shows non-reproducible time-stamp problem in
>> >> build/diffoscope-output.txt) when executed without this patch, and
>> >> after applying this patch it will still show fail (because there
>> >> are other reproducible problems to fix) but the non-reproducible
>> >> timestamp problem in rootfs shows fixed.
>> >
>> >So the code is there but not used because it goes failed anyhow? In
>> >that case i would suggest to rewrite it in a way that it does not
>> >fail and can run in CI. It could for example ignore "known issues"
>> >and be enabled, where then this very series would remove a few "known
>> >issues" in p2 (p1 would be to introduce all currently known issues
>> >and enable that thing in CI) and p3 would be what we look at here
>>
>> You mean run the test and just ignore the results?
>
>Ignore some, the ones we know as violators that we still need to work on. But
>do not ignore anything coming up as new problems, maybe/likely in layers.
>
>Again, has this ever been tested on anything but the very boring isar core? isar-
>cip-core could be interesting as well, especially the image cip-core-image-
>security
>
I have executed reproducible build tests in isar-cip-core image manually (not with the test-suite)
and reported the problem as well here [1]
[1] https://gitlab.com/cip-project/cip-core/isar-cip-core/-/issues/31

>> >
>> >Not sure i understood but it sounds like we have a test that we only
>> >run manually, and i want it to run in CI. In fact a script to check
>> >for repro issues should be so generic that any layer can use it.
>>
>> Eventually the repro-build test-case should go to CI, I will plan to
>> do it.
>>
>> Can I know where the isar CI pipe lines are running? I couldn't see in
>> isar-github may be running some other place?
>
>Unfortunately the two places that run it are not public and would require you
>to receive an account. You could ask Baurzhan if you can get one, or you just
>run it yourself manually while working on the patches. If it is enabled and will
>ever fail, you will likely get reports from CI sent to you.

Got it, then I will test them manually as usual.

>
>Henning
>
>> >
>> >So your vision would be that isar core brings that script and i.e.
>> >jailhouse-images can run it on all its images. And a few calls to the
>> >script will also be in testsuite/ and be called in pipelines
>> >regularly.
>> >
>> >Henning
>> >
>> >> [1]
>> >> https://github.com/ilbers/isar/commit/9698445eade76fab81e2dc16d4cb5
>> >> 8b2
>> >> 4e953cb9
>> >>
>> >> >
>> >> >Henning
>> >> >
>> >> >>  EOSUDO
>> >> >>  }
>> >> >>  addtask rootfs_finalize before do_rootfs after
>> >> >> do_rootfs_postprocess
>> >> >
>> >> >--
>> >> >You received this message because you are subscribed to the Google
>> >> >Groups "isar-users" group.
>> >> >To unsubscribe from this group and stop receiving emails from it,
>> >> >send an email to isar-users+unsubscribe@googlegroups.com.
>> >> >To view this discussion on the web visit
>> >> >https://groups.google.com/d/msgid/isar-
>> >> >users/20230102174418.686715cf%40md1za8fc.ad001.siemens.net.
>> >
>> >--
>> >You received this message because you are subscribed to the Google
>> >Groups "isar-users" group.
>> >To unsubscribe from this group and stop receiving emails from it,
>> >send an email to isar-users+unsubscribe@googlegroups.com.
>> >To view this discussion on the web visit
>> >https://groups.google.com/d/msgid/isar-
>> >users/20230103200543.07e987ba%40md1za8fc.ad001.siemens.net.
>
>--
>You received this message because you are subscribed to the Google Groups
>"isar-users" group.
>To unsubscribe from this group and stop receiving emails from it, send an email
>to isar-users+unsubscribe@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/isar-
>users/20230104102934.042477a8%40md1za8fc.ad001.siemens.net.