Re: [PATCH] image.bbclass: fix non-reproducible file time-stamps inside rootfs image

[Jan Kiszka <jan.kiszka@siemens.com>]

On 04.01.23 14:53, Henning Schild wrote:
> Am Wed, 4 Jan 2023 13:48:10 +0000
> schrieb <Venkata.Pyla@toshiba-tsip.com>:
> 
>>> -----Original Message-----
>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
>>> Behalf Of Henning Schild
>>> Sent: 04 January 2023 15:00
>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-  
>>> tsip.com>  
>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de;
>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ)
>>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>>> Porting) <dinesh.kumar@toshiba-tsip.com>; Baurzhan Ismagulov
>>> <ibr@radix50.net> Subject: Re: [PATCH] image.bbclass: fix
>>> non-reproducible file time-stamps inside rootfs image
>>>
>>> Am Wed, 4 Jan 2023 07:54:44 +0000
>>> schrieb <Venkata.Pyla@toshiba-tsip.com>:
>>>  
>>>>> -----Original Message-----
>>>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
>>>>> Behalf Of Henning Schild
>>>>> Sent: 04 January 2023 00:36
>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting)
>>>>> <Venkata.Pyla@toshiba- tsip.com>  
>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de;
>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ)
>>>>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>>>>> Porting) <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH]
>>>>> image.bbclass: fix non-reproducible file time-stamps inside rootfs
>>>>> image
>>>>>
>>>>> Am Tue, 3 Jan 2023 14:10:14 +0000
>>>>> schrieb <Venkata.Pyla@toshiba-tsip.com>:
>>>>>  
>>>>>>> -----Original Message-----
>>>>>>> From: isar-users@googlegroups.com
>>>>>>> <isar-users@googlegroups.com> On Behalf Of Henning Schild
>>>>>>> Sent: 02 January 2023 22:14
>>>>>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting)
>>>>>>> <Venkata.Pyla@toshiba- tsip.com>
>>>>>>> Cc: isar-users@googlegroups.com; amikan@ilbers.de;
>>>>>>> jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏
>>>>>>> □ＳＷＣ◯ＡＣＴ) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh
>>>>>>> kumar(ＴＳＩＰ TMIEC ODG Porting)
>>>>>>> <dinesh.kumar@toshiba-tsip.com> Subject: Re: [PATCH]
>>>>>>> image.bbclass: fix non-reproducible file time-stamps inside
>>>>>>> rootfs image
>>>>>>>
>>>>>>> Am Mon,  2 Jan 2023 20:28:28 +0530 schrieb
>>>>>>> venkata.pyla@toshiba-tsip.com:
>>>>>>>  
>>>>>>>> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
>>>>>>>>
>>>>>>>> As part of reproducible-build work, the rootfs images
>>>>>>>> generated on same source should be identical between two
>>>>>>>> builds.
>>>>>>>>
>>>>>>>> In this commit it tries to solve one of the non-reproducible
>>>>>>>> problem i.e. the rootfs file time-stamps generated during
>>>>>>>> build time are not reproducible, it uses one of the solution
>>>>>>>> provided in the debian live-build image project (refer [1]),
>>>>>>>> it fixes by finding all the files/folders that are
>>>>>>>> gernerated newly and set the time-stamp provided by
>>>>>>>> `SOURCE_DATE_EPOCH` environment variable.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://salsa.debian.org/live-team/live-build/-/merge_requests/2
>>>>>>>> 18
>>>>>>>>
>>>>>>>> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
>>>>>>>> ---
>>>>>>>>  meta/classes/image.bbclass | 9 +++++++++
>>>>>>>>  1 file changed, 9 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/meta/classes/image.bbclass
>>>>>>>> b/meta/classes/image.bbclass index 813e1f3..f592a12 100644
>>>>>>>> --- a/meta/classes/image.bbclass
>>>>>>>> +++ b/meta/classes/image.bbclass
>>>>>>>> @@ -430,6 +430,15 @@ do_rootfs_finalize() {
>>>>>>>>              "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
>>>>>>>>
>>>>>>>>          rm -f "${ROOTFSDIR}/etc/apt/sources-list"
>>>>>>>> +
>>>>>>>> +        # Set same time-stamps to the newly generated
>>>>>>>> file/folders in the
>>>>>>>> +        # rootfs image for the purpose of reproducible
>>>>>>>> builds.
>>>>>>>> +        test ! -z "${SOURCE_DATE_EPOCH}" && \
>>>>>>>> +            find ${ROOTFSDIR} -newermt \
>>>>>>>> +                "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d
>>>>>>>> %H:%M:%S')" \
>>>>>>>> +                -printf "%y %p\n" \
>>>>>>>> +                -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH}
>>>>>>>> ';'
>>>>>>>> +  
>>>>>>>
>>>>>>> This looks like i have seen it before. For me that is _way_ too
>>>>>>> generic and something that is not a package touches files all
>>>>>>> over the place. If some package now wants to intentionally
>>>>>>> bring a file that is from a far away future?
>>>>>>>
>>>>>>> Which files are we talking about here? It can basically only be
>>>>>>> metadata and other little places where we violate our
>>>>>>> "everything comes from a package" rule.  
>>>>>>
>>>>>> files/folder/symbolic-link that are modified or generated during
>>>>>> build time like /etc/os-release /etc/hostname .
>>>>>> .
>>>>>> .
>>>>>> /var/lib/dpkg/info/*
>>>>>> /var/cache/*
>>>>>> ---
>>>>>>
>>>>>> I have printed all the files that modified during build by
>>>>>> executing below command find ${ROOTFSDIR} -newermt "$(date
>>>>>> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" -printf "%t %y  
>>> %p\n"  
>>>>>>
>>>>>> modified_files_times.txt
>>>>>>
>>>>>> attached modified_files_times.txt for your reference, all these
>>>>>> files/folders/symbolic-link are generated or modified during
>>>>>> build time.  
>>>>>
>>>>> So i am guessing it is about anything that comes out of
>>>>> postprocess functions and maintainer scripts like postinst ?  
>>>>
>>>> Yes, those are the files that are modified by the packages and have
>>>> different timestamp on each build. I think it is okay to set
>>>> reproducible times-tamp for such files, as they are not coming from
>>>> the packages but are modified at build time.  
>>>
>>> Sure they are OK. But if i read your patch correctly you will
>>> potentially adjust times on ALL files, not just some for which it
>>> might be OK.
>>>
>>> And it uses a variable which does not have a default, should we not
>>> set that to something? Or help users to choose a good value for it.  
>>
>> Setting default value, I didn't think of it because reproducible
>> builds may not be required for regular builds, however when someone
>> wants it can be enable by setting the value to it. So as you
>> mentioned it is good to mention somewhere to help users to choose
>> good value for it, I am thinking to add it in doc/user_manual.md
> 
> Yes, user manual or maybe local.conf.sample but commented out. But some
> value that works and makes sense needs to be given to people. Otherwise
> i would set
> 
> SOURCE_DATE_EPOCH="just a guess 42"
> 
> which might not work

If I understand this patch correctly, any file older than
SOURCE_DATE_EPOCH will not be touched, any newer one will get that date.
So proposing some concrete date here could eventually overshoot. What
would be better is to document the format along the commented-out variable.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux