RE: [PATCH] image.bbclass: fix non-reproducible file time-stamps inside rootfs image

[<Venkata.Pyla@toshiba-tsip.com>]

>-----Original Message-----
>From: Henning Schild <henning.schild@siemens.com>
>Sent: 04 January 2023 20:37
>To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-
>tsip.com>
>Cc: jan.kiszka@siemens.com; isar-users@googlegroups.com; amikan@ilbers.de;
>hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>;
>dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>;
>ibr@radix50.net
>Subject: Re: [PATCH] image.bbclass: fix non-reproducible file time-stamps inside
>rootfs image
>
>Am Wed, 4 Jan 2023 14:50:44 +0000
>schrieb <Venkata.Pyla@toshiba-tsip.com>:
>
>> >-----Original Message-----
>> >From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
>> >Behalf Of Jan Kiszka
>> >Sent: 04 January 2023 20:06
>> >To: Henning Schild <henning.schild@siemens.com>; pyla
>> >venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-tsip.com>
>> >Cc: isar-users@googlegroups.com; amikan@ilbers.de; hayashi
>> >kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>;
>> >dinesh kumar(ＴＳＩＰ TMIEC ODG Porting)
>> ><dinesh.kumar@toshiba-tsip.com>; ibr@radix50.net Subject: Re:
>> >[PATCH] image.bbclass: fix non-reproducible file time-stamps inside
>> >rootfs image
>> >
>> >On 04.01.23 14:53, Henning Schild wrote:
>> >> Am Wed, 4 Jan 2023 13:48:10 +0000
>> >> schrieb <Venkata.Pyla@toshiba-tsip.com>:
>> >>
>> >>>> -----Original Message-----
>> >>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com>
>> >>>> On Behalf Of Henning Schild
>> >>>> Sent: 04 January 2023 15:00
>> >>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting)
>> >>>> <Venkata.Pyla@toshiba- tsip.com>
>> >>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de;
>> >>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ)
>> >>>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>> >>>> Porting) <dinesh.kumar@toshiba-tsip.com>; Baurzhan Ismagulov
>> >>>> <ibr@radix50.net> Subject: Re: [PATCH] image.bbclass: fix
>> >>>> non-reproducible file time-stamps inside rootfs image
>> >>>>
>> >>>> Am Wed, 4 Jan 2023 07:54:44 +0000 schrieb
>> >>>> <Venkata.Pyla@toshiba-tsip.com>:
>> >>>>
>> >>>>>> -----Original Message-----
>> >>>>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com>
>> >>>>>> On Behalf Of Henning Schild
>> >>>>>> Sent: 04 January 2023 00:36
>> >>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting)
>> >>>>>> <Venkata.Pyla@toshiba- tsip.com>
>> >>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de;
>> >>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏
>> >>>>>> □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh
>> >>>>>> kumar(ＴＳＩＰ TMIEC ODG Porting)
>> >>>>>> <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH]
>> >>>>>> image.bbclass: fix non-reproducible file time-stamps inside
>> >>>>>> rootfs image
>> >>>>>>
>> >>>>>> Am Tue, 3 Jan 2023 14:10:14 +0000 schrieb
>> >>>>>> <Venkata.Pyla@toshiba-tsip.com>:
>> >>>>>>
>> >>>>>>>> -----Original Message-----
>> >>>>>>>> From: isar-users@googlegroups.com
>> >>>>>>>> <isar-users@googlegroups.com> On Behalf Of Henning Schild
>> >>>>>>>> Sent: 02 January 2023 22:14
>> >>>>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting)
>> >>>>>>>> <Venkata.Pyla@toshiba- tsip.com>
>> >>>>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de;
>> >>>>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏
>> >>>>>>>> □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh
>> >>>>>>>> kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-
>tsip.com>
>> >>>>>>>> Subject: Re: [PATCH]
>> >>>>>>>> image.bbclass: fix non-reproducible file time-stamps inside
>> >>>>>>>> rootfs image
>> >>>>>>>>
>> >>>>>>>> Am Mon,  2 Jan 2023 20:28:28 +0530 schrieb
>> >>>>>>>> venkata.pyla@toshiba-tsip.com:
>> >>>>>>>>
>> >>>>>>>>> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
>> >>>>>>>>>
>> >>>>>>>>> As part of reproducible-build work, the rootfs images
>> >>>>>>>>> generated on same source should be identical between two
>> >>>>>>>>> builds.
>> >>>>>>>>>
>> >>>>>>>>> In this commit it tries to solve one of the non-reproducible
>> >>>>>>>>> problem i.e. the rootfs file time-stamps generated during
>> >>>>>>>>> build time are not reproducible, it uses one of the solution
>> >>>>>>>>> provided in the debian live-build image project (refer [1]),
>> >>>>>>>>> it fixes by finding all the files/folders that are
>> >>>>>>>>> gernerated newly and set the time-stamp provided by
>> >>>>>>>>> `SOURCE_DATE_EPOCH` environment variable.
>> >>>>>>>>>
>> >>>>>>>>> [1]
>> >>>>>>>>> https://salsa.debian.org/live-team/live-build/-/merge_reques
>> >>>>>>>>> ts/
>> >>>>>>>>> 2
>> >>>>>>>>> 18
>> >>>>>>>>>
>> >>>>>>>>> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
>> >>>>>>>>> ---
>> >>>>>>>>>  meta/classes/image.bbclass | 9 +++++++++
>> >>>>>>>>>  1 file changed, 9 insertions(+)
>> >>>>>>>>>
>> >>>>>>>>> diff --git a/meta/classes/image.bbclass
>> >>>>>>>>> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644
>> >>>>>>>>> --- a/meta/classes/image.bbclass
>> >>>>>>>>> +++ b/meta/classes/image.bbclass
>> >>>>>>>>> @@ -430,6 +430,15 @@ do_rootfs_finalize() {
>> >>>>>>>>>              "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
>> >>>>>>>>>
>> >>>>>>>>>          rm -f "${ROOTFSDIR}/etc/apt/sources-list"
>> >>>>>>>>> +
>> >>>>>>>>> +        # Set same time-stamps to the newly generated
>> >>>>>>>>> file/folders in the
>> >>>>>>>>> +        # rootfs image for the purpose of reproducible
>> >>>>>>>>> builds.
>> >>>>>>>>> +        test ! -z "${SOURCE_DATE_EPOCH}" && \
>> >>>>>>>>> +            find ${ROOTFSDIR} -newermt \
>> >>>>>>>>> +                "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d
>> >>>>>>>>> %H:%M:%S')" \
>> >>>>>>>>> +                -printf "%y %p\n" \
>> >>>>>>>>> +                -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH}
>> >>>>>>>>> ';'
>> >>>>>>>>> +
>> >>>>>>>>
>> >>>>>>>> This looks like i have seen it before. For me that is _way_
>> >>>>>>>> too generic and something that is not a package touches files
>> >>>>>>>> all over the place. If some package now wants to
>> >>>>>>>> intentionally bring a file that is from a far away future?
>> >>>>>>>>
>> >>>>>>>> Which files are we talking about here? It can basically only
>> >>>>>>>> be metadata and other little places where we violate our
>> >>>>>>>> "everything comes from a package" rule.
>> >>>>>>>
>> >>>>>>> files/folder/symbolic-link that are modified or generated
>> >>>>>>> during build time like /etc/os-release /etc/hostname .
>> >>>>>>> .
>> >>>>>>> .
>> >>>>>>> /var/lib/dpkg/info/*
>> >>>>>>> /var/cache/*
>> >>>>>>> ---
>> >>>>>>>
>> >>>>>>> I have printed all the files that modified during build by
>> >>>>>>> executing below command find ${ROOTFSDIR} -newermt "$(date
>> >>>>>>> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t
>%y
>> >>>>>>>
>> >>>> %p\n"
>> >>>>>>>
>> >>>>>>> modified_files_times.txt
>> >>>>>>>
>> >>>>>>> attached modified_files_times.txt for your reference, all
>> >>>>>>> these files/folders/symbolic-link are generated or modified
>> >>>>>>> during build time.
>> >>>>>>
>> >>>>>> So i am guessing it is about anything that comes out of
>> >>>>>> postprocess functions and maintainer scripts like postinst ?
>> >>>>>
>> >>>>> Yes, those are the files that are modified by the packages and
>> >>>>> have different timestamp on each build. I think it is okay to
>> >>>>> set reproducible times-tamp for such files, as they are not
>> >>>>> coming from the packages but are modified at build time.
>> >>>>
>> >>>> Sure they are OK. But if i read your patch correctly you will
>> >>>> potentially adjust times on ALL files, not just some for which it
>> >>>> might be OK.
>> >>>>
>> >>>> And it uses a variable which does not have a default, should we
>> >>>> not set that to something? Or help users to choose a good value
>> >>>> for it.
>> >>>
>> >>> Setting default value, I didn't think of it because reproducible
>> >>> builds may not be required for regular builds, however when
>> >>> someone wants it can be enable by setting the value to it. So as
>> >>> you mentioned it is good to mention somewhere to help users to
>> >>> choose good value for it, I am thinking to add it in
>> >>> doc/user_manual.md
>> >>
>> >> Yes, user manual or maybe local.conf.sample but commented out. But
>> >> some value that works and makes sense needs to be given to people.
>> >> Otherwise i would set
>> >>
>> >> SOURCE_DATE_EPOCH="just a guess 42"
>> >>
>> >> which might not work
>> >
>> >If I understand this patch correctly, any file older than
>> >SOURCE_DATE_EPOCH will not be touched, any newer one will get that
>> >date. So proposing some concrete date here could eventually
>> >overshoot. What would be better is to document the format along the
>> >commented-out variable.
>>
>> The good value for it is the date of the last change to the source
>> (git log -1 --pretty=%ct)
>
>And i assume if one wanted to release tarballs one would have to include a
>value in such a release, derived from version control.
>
>I am not sure anyone really puts isar layers into tarballs but wanted to remind
>that git is not all, the documentation can and should mention git first though.
>Maybe with a line that actually calls that very git command once comment
>marks have been removed.

In that case in the comment, we can suggest to the users for the repository users
Can use git command or the 

#
# Uncomment for reproducible builds
# Git repository users can use value from 'git log -1 --pretty=%ct'
# Not repository users can use value from 'stat -c%Y ChangeLog'
#SOURCE_DATE_EPOCH = 


>
>Henning
>
>> >
>> >Jan
>> >
>> >--
>> >Siemens AG, Technology
>> >Competence Center Embedded Linux
>> >
>> >--
>> >You received this message because you are subscribed to the Google
>> >Groups "isar-users" group.
>> >To unsubscribe from this group and stop receiving emails from it,
>> >send an email to isar-users+unsubscribe@googlegroups.com.
>> >To view this discussion on the web visit
>> >https://groups.google.com/d/msgid/isar-users/832199ad-53c9-dcd2-a7e7-
>> >30c61fe92c92%40siemens.com.
>>