RE: [PATCH] image.bbclass: fix non-reproducible file time-stamps inside rootfs image

[<Venkata.Pyla@toshiba-tsip.com>]

>-----Original Message-----
>From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf
>Of Jan Kiszka
>Sent: 04 January 2023 20:06
>To: Henning Schild <henning.schild@siemens.com>; pyla venkata(ＴＳＩＰ
>TMIEC ODG Porting) <Venkata.Pyla@toshiba-tsip.com>
>Cc: isar-users@googlegroups.com; amikan@ilbers.de; hayashi kazuhiro(林 和宏
>□ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ
>TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; ibr@radix50.net
>Subject: Re: [PATCH] image.bbclass: fix non-reproducible file time-stamps inside
>rootfs image
>
>On 04.01.23 14:53, Henning Schild wrote:
>> Am Wed, 4 Jan 2023 13:48:10 +0000
>> schrieb <Venkata.Pyla@toshiba-tsip.com>:
>>
>>>> -----Original Message-----
>>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
>>>> Behalf Of Henning Schild
>>>> Sent: 04 January 2023 15:00
>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-
>>>> tsip.com>
>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de;
>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ)
>>>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>>>> Porting) <dinesh.kumar@toshiba-tsip.com>; Baurzhan Ismagulov
>>>> <ibr@radix50.net> Subject: Re: [PATCH] image.bbclass: fix
>>>> non-reproducible file time-stamps inside rootfs image
>>>>
>>>> Am Wed, 4 Jan 2023 07:54:44 +0000
>>>> schrieb <Venkata.Pyla@toshiba-tsip.com>:
>>>>
>>>>>> -----Original Message-----
>>>>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
>>>>>> Behalf Of Henning Schild
>>>>>> Sent: 04 January 2023 00:36
>>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting)
>>>>>> <Venkata.Pyla@toshiba- tsip.com>
>>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de;
>>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ)
>>>>>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>>>>>> Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH]
>>>>>> image.bbclass: fix non-reproducible file time-stamps inside rootfs
>>>>>> image
>>>>>>
>>>>>> Am Tue, 3 Jan 2023 14:10:14 +0000
>>>>>> schrieb <Venkata.Pyla@toshiba-tsip.com>:
>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com>
>>>>>>>> On Behalf Of Henning Schild
>>>>>>>> Sent: 02 January 2023 22:14
>>>>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting)
>>>>>>>> <Venkata.Pyla@toshiba- tsip.com>
>>>>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de;
>>>>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏
>>>>>>>> □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh
>>>>>>>> kumar(ＴＳＩＰ TMIEC ODG Porting)
>>>>>>>> <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH]
>>>>>>>> image.bbclass: fix non-reproducible file time-stamps inside
>>>>>>>> rootfs image
>>>>>>>>
>>>>>>>> Am Mon,  2 Jan 2023 20:28:28 +0530 schrieb
>>>>>>>> venkata.pyla@toshiba-tsip.com:
>>>>>>>>
>>>>>>>>> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
>>>>>>>>>
>>>>>>>>> As part of reproducible-build work, the rootfs images generated
>>>>>>>>> on same source should be identical between two builds.
>>>>>>>>>
>>>>>>>>> In this commit it tries to solve one of the non-reproducible
>>>>>>>>> problem i.e. the rootfs file time-stamps generated during build
>>>>>>>>> time are not reproducible, it uses one of the solution provided
>>>>>>>>> in the debian live-build image project (refer [1]), it fixes by
>>>>>>>>> finding all the files/folders that are gernerated newly and set
>>>>>>>>> the time-stamp provided by `SOURCE_DATE_EPOCH` environment
>>>>>>>>> variable.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://salsa.debian.org/live-team/live-build/-/merge_requests/
>>>>>>>>> 2
>>>>>>>>> 18
>>>>>>>>>
>>>>>>>>> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
>>>>>>>>> ---
>>>>>>>>>  meta/classes/image.bbclass | 9 +++++++++
>>>>>>>>>  1 file changed, 9 insertions(+)
>>>>>>>>>
>>>>>>>>> diff --git a/meta/classes/image.bbclass
>>>>>>>>> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644
>>>>>>>>> --- a/meta/classes/image.bbclass
>>>>>>>>> +++ b/meta/classes/image.bbclass
>>>>>>>>> @@ -430,6 +430,15 @@ do_rootfs_finalize() {
>>>>>>>>>              "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
>>>>>>>>>
>>>>>>>>>          rm -f "${ROOTFSDIR}/etc/apt/sources-list"
>>>>>>>>> +
>>>>>>>>> +        # Set same time-stamps to the newly generated
>>>>>>>>> file/folders in the
>>>>>>>>> +        # rootfs image for the purpose of reproducible
>>>>>>>>> builds.
>>>>>>>>> +        test ! -z "${SOURCE_DATE_EPOCH}" && \
>>>>>>>>> +            find ${ROOTFSDIR} -newermt \
>>>>>>>>> +                "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d
>>>>>>>>> %H:%M:%S')" \
>>>>>>>>> +                -printf "%y %p\n" \
>>>>>>>>> +                -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH}
>>>>>>>>> ';'
>>>>>>>>> +
>>>>>>>>
>>>>>>>> This looks like i have seen it before. For me that is _way_ too
>>>>>>>> generic and something that is not a package touches files all
>>>>>>>> over the place. If some package now wants to intentionally bring
>>>>>>>> a file that is from a far away future?
>>>>>>>>
>>>>>>>> Which files are we talking about here? It can basically only be
>>>>>>>> metadata and other little places where we violate our
>>>>>>>> "everything comes from a package" rule.
>>>>>>>
>>>>>>> files/folder/symbolic-link that are modified or generated during
>>>>>>> build time like /etc/os-release /etc/hostname .
>>>>>>> .
>>>>>>> .
>>>>>>> /var/lib/dpkg/info/*
>>>>>>> /var/cache/*
>>>>>>> ---
>>>>>>>
>>>>>>> I have printed all the files that modified during build by
>>>>>>> executing below command find ${ROOTFSDIR} -newermt "$(date
>>>>>>> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t %y
>>>> %p\n"
>>>>>>>
>>>>>>> modified_files_times.txt
>>>>>>>
>>>>>>> attached modified_files_times.txt for your reference, all these
>>>>>>> files/folders/symbolic-link are generated or modified during
>>>>>>> build time.
>>>>>>
>>>>>> So i am guessing it is about anything that comes out of
>>>>>> postprocess functions and maintainer scripts like postinst ?
>>>>>
>>>>> Yes, those are the files that are modified by the packages and have
>>>>> different timestamp on each build. I think it is okay to set
>>>>> reproducible times-tamp for such files, as they are not coming from
>>>>> the packages but are modified at build time.
>>>>
>>>> Sure they are OK. But if i read your patch correctly you will
>>>> potentially adjust times on ALL files, not just some for which it
>>>> might be OK.
>>>>
>>>> And it uses a variable which does not have a default, should we not
>>>> set that to something? Or help users to choose a good value for it.
>>>
>>> Setting default value, I didn't think of it because reproducible
>>> builds may not be required for regular builds, however when someone
>>> wants it can be enable by setting the value to it. So as you
>>> mentioned it is good to mention somewhere to help users to choose
>>> good value for it, I am thinking to add it in doc/user_manual.md
>>
>> Yes, user manual or maybe local.conf.sample but commented out. But
>> some value that works and makes sense needs to be given to people.
>> Otherwise i would set
>>
>> SOURCE_DATE_EPOCH="just a guess 42"
>>
>> which might not work
>
>If I understand this patch correctly, any file older than SOURCE_DATE_EPOCH
>will not be touched, any newer one will get that date.
>So proposing some concrete date here could eventually overshoot. What would
>be better is to document the format along the commented-out variable.

The good value for it is the date of the last change to the source (git log -1 --pretty=%ct)

>
>Jan
>
>--
>Siemens AG, Technology
>Competence Center Embedded Linux
>
>--
>You received this message because you are subscribed to the Google Groups
>"isar-users" group.
>To unsubscribe from this group and stop receiving emails from it, send an email
>to isar-users+unsubscribe@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/isar-users/832199ad-53c9-dcd2-a7e7-
>30c61fe92c92%40siemens.com.