Re: [PATCH] image.bbclass: fix non-reproducible file time-stamps inside rootfs image

[Henning Schild <henning.schild@siemens.com>]

Am Wed, 4 Jan 2023 14:50:44 +0000
schrieb <Venkata.Pyla@toshiba-tsip.com>:

> >-----Original Message-----
> >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
> >Behalf Of Jan Kiszka
> >Sent: 04 January 2023 20:06
> >To: Henning Schild <henning.schild@siemens.com>; pyla
> >venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-tsip.com>
> >Cc: isar-users@googlegroups.com; amikan@ilbers.de; hayashi
> >kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>;
> >dinesh kumar(ＴＳＩＰ TMIEC ODG Porting)
> ><dinesh.kumar@toshiba-tsip.com>; ibr@radix50.net Subject: Re:
> >[PATCH] image.bbclass: fix non-reproducible file time-stamps inside
> >rootfs image
> >
> >On 04.01.23 14:53, Henning Schild wrote:  
> >> Am Wed, 4 Jan 2023 13:48:10 +0000
> >> schrieb <Venkata.Pyla@toshiba-tsip.com>:
> >>  
> >>>> -----Original Message-----
> >>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com>
> >>>> On Behalf Of Henning Schild
> >>>> Sent: 04 January 2023 15:00
> >>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting)
> >>>> <Venkata.Pyla@toshiba- tsip.com>  
> >>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de;
> >>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ)
> >>>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC
> >>>> ODG Porting) <dinesh.kumar@toshiba-tsip.com>; Baurzhan Ismagulov
> >>>> <ibr@radix50.net> Subject: Re: [PATCH] image.bbclass: fix
> >>>> non-reproducible file time-stamps inside rootfs image
> >>>>
> >>>> Am Wed, 4 Jan 2023 07:54:44 +0000
> >>>> schrieb <Venkata.Pyla@toshiba-tsip.com>:
> >>>>  
> >>>>>> -----Original Message-----
> >>>>>> From: isar-users@googlegroups.com
> >>>>>> <isar-users@googlegroups.com> On Behalf Of Henning Schild
> >>>>>> Sent: 04 January 2023 00:36
> >>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting)
> >>>>>> <Venkata.Pyla@toshiba- tsip.com>
> >>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de;
> >>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏
> >>>>>> □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh
> >>>>>> kumar(ＴＳＩＰ TMIEC ODG Porting)
> >>>>>> <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH]
> >>>>>> image.bbclass: fix non-reproducible file time-stamps inside
> >>>>>> rootfs image
> >>>>>>
> >>>>>> Am Tue, 3 Jan 2023 14:10:14 +0000
> >>>>>> schrieb <Venkata.Pyla@toshiba-tsip.com>:
> >>>>>>  
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: isar-users@googlegroups.com
> >>>>>>>> <isar-users@googlegroups.com> On Behalf Of Henning Schild
> >>>>>>>> Sent: 02 January 2023 22:14
> >>>>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting)
> >>>>>>>> <Venkata.Pyla@toshiba- tsip.com>
> >>>>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de;
> >>>>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏
> >>>>>>>> □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh
> >>>>>>>> kumar(ＴＳＩＰ TMIEC ODG Porting)
> >>>>>>>> <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH]
> >>>>>>>> image.bbclass: fix non-reproducible file time-stamps inside
> >>>>>>>> rootfs image
> >>>>>>>>
> >>>>>>>> Am Mon,  2 Jan 2023 20:28:28 +0530 schrieb
> >>>>>>>> venkata.pyla@toshiba-tsip.com:
> >>>>>>>>  
> >>>>>>>>> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> >>>>>>>>>
> >>>>>>>>> As part of reproducible-build work, the rootfs images
> >>>>>>>>> generated on same source should be identical between two
> >>>>>>>>> builds.
> >>>>>>>>>
> >>>>>>>>> In this commit it tries to solve one of the non-reproducible
> >>>>>>>>> problem i.e. the rootfs file time-stamps generated during
> >>>>>>>>> build time are not reproducible, it uses one of the
> >>>>>>>>> solution provided in the debian live-build image project
> >>>>>>>>> (refer [1]), it fixes by finding all the files/folders that
> >>>>>>>>> are gernerated newly and set the time-stamp provided by
> >>>>>>>>> `SOURCE_DATE_EPOCH` environment variable.
> >>>>>>>>>
> >>>>>>>>> [1]
> >>>>>>>>> https://salsa.debian.org/live-team/live-build/-/merge_requests/
> >>>>>>>>> 2
> >>>>>>>>> 18
> >>>>>>>>>
> >>>>>>>>> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> >>>>>>>>> ---
> >>>>>>>>>  meta/classes/image.bbclass | 9 +++++++++
> >>>>>>>>>  1 file changed, 9 insertions(+)
> >>>>>>>>>
> >>>>>>>>> diff --git a/meta/classes/image.bbclass
> >>>>>>>>> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644
> >>>>>>>>> --- a/meta/classes/image.bbclass
> >>>>>>>>> +++ b/meta/classes/image.bbclass
> >>>>>>>>> @@ -430,6 +430,15 @@ do_rootfs_finalize() {
> >>>>>>>>>              "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> >>>>>>>>>
> >>>>>>>>>          rm -f "${ROOTFSDIR}/etc/apt/sources-list"
> >>>>>>>>> +
> >>>>>>>>> +        # Set same time-stamps to the newly generated
> >>>>>>>>> file/folders in the
> >>>>>>>>> +        # rootfs image for the purpose of reproducible
> >>>>>>>>> builds.
> >>>>>>>>> +        test ! -z "${SOURCE_DATE_EPOCH}" && \
> >>>>>>>>> +            find ${ROOTFSDIR} -newermt \
> >>>>>>>>> +                "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d
> >>>>>>>>> %H:%M:%S')" \
> >>>>>>>>> +                -printf "%y %p\n" \
> >>>>>>>>> +                -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH}
> >>>>>>>>> ';'
> >>>>>>>>> +  
> >>>>>>>>
> >>>>>>>> This looks like i have seen it before. For me that is _way_
> >>>>>>>> too generic and something that is not a package touches
> >>>>>>>> files all over the place. If some package now wants to
> >>>>>>>> intentionally bring a file that is from a far away future?
> >>>>>>>>
> >>>>>>>> Which files are we talking about here? It can basically only
> >>>>>>>> be metadata and other little places where we violate our
> >>>>>>>> "everything comes from a package" rule.  
> >>>>>>>
> >>>>>>> files/folder/symbolic-link that are modified or generated
> >>>>>>> during build time like /etc/os-release /etc/hostname .
> >>>>>>> .
> >>>>>>> .
> >>>>>>> /var/lib/dpkg/info/*
> >>>>>>> /var/cache/*
> >>>>>>> ---
> >>>>>>>
> >>>>>>> I have printed all the files that modified during build by
> >>>>>>> executing below command find ${ROOTFSDIR} -newermt "$(date
> >>>>>>> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t %y
> >>>>>>>  
> >>>> %p\n"  
> >>>>>>>
> >>>>>>> modified_files_times.txt
> >>>>>>>
> >>>>>>> attached modified_files_times.txt for your reference, all
> >>>>>>> these files/folders/symbolic-link are generated or modified
> >>>>>>> during build time.  
> >>>>>>
> >>>>>> So i am guessing it is about anything that comes out of
> >>>>>> postprocess functions and maintainer scripts like postinst ?  
> >>>>>
> >>>>> Yes, those are the files that are modified by the packages and
> >>>>> have different timestamp on each build. I think it is okay to
> >>>>> set reproducible times-tamp for such files, as they are not
> >>>>> coming from the packages but are modified at build time.  
> >>>>
> >>>> Sure they are OK. But if i read your patch correctly you will
> >>>> potentially adjust times on ALL files, not just some for which it
> >>>> might be OK.
> >>>>
> >>>> And it uses a variable which does not have a default, should we
> >>>> not set that to something? Or help users to choose a good value
> >>>> for it.  
> >>>
> >>> Setting default value, I didn't think of it because reproducible
> >>> builds may not be required for regular builds, however when
> >>> someone wants it can be enable by setting the value to it. So as
> >>> you mentioned it is good to mention somewhere to help users to
> >>> choose good value for it, I am thinking to add it in
> >>> doc/user_manual.md  
> >>
> >> Yes, user manual or maybe local.conf.sample but commented out. But
> >> some value that works and makes sense needs to be given to people.
> >> Otherwise i would set
> >>
> >> SOURCE_DATE_EPOCH="just a guess 42"
> >>
> >> which might not work  
> >
> >If I understand this patch correctly, any file older than
> >SOURCE_DATE_EPOCH will not be touched, any newer one will get that
> >date. So proposing some concrete date here could eventually
> >overshoot. What would be better is to document the format along the
> >commented-out variable.  
> 
> The good value for it is the date of the last change to the source
> (git log -1 --pretty=%ct)

And i assume if one wanted to release tarballs one would have to
include a value in such a release, derived from version control.

I am not sure anyone really puts isar layers into tarballs but wanted
to remind that git is not all, the documentation can and should mention
git first though.
Maybe with a line that actually calls that very git command once
comment marks have been removed.

Henning

> >
> >Jan
> >
> >--
> >Siemens AG, Technology
> >Competence Center Embedded Linux
> >
> >--
> >You received this message because you are subscribed to the Google
> >Groups "isar-users" group.
> >To unsubscribe from this group and stop receiving emails from it,
> >send an email to isar-users+unsubscribe@googlegroups.com.
> >To view this discussion on the web visit
> >https://groups.google.com/d/msgid/isar-users/832199ad-53c9-dcd2-a7e7-
> >30c61fe92c92%40siemens.com.  
>