[RFC PATCH] image.bbclass: create separate task for creating reproducible image

[RFC PATCH] image.bbclass: create separate task for creating reproducible image

[venkata.pyla]

From: venkata pyla <venkata.pyla@toshiba-tsip.com>

Currently the reproducible fix for setting same file timestamps across
builds is present in the function `do_rootfs_finalize` and this fix may
be skipped for the tasks that are added after this task or `do_rootfs`
 for e.g. in one of the child project(CIP) for swupdate target it adds
 additional task `do_generate_image_uuid` that is added after `do_rootfs`
 this task is updating some contents in the rootfs which is skipping the
 reproducible fix was applied in `do_rootfs_finalize`.

For this reason a separate task `do_image_make_reproducible` is created
which should call before the final target image is created [tar, wic,
squashfs] so that it applies reproducible fixes just before image
creation.

RFC:
In this patch I tried below dependency[1] for the function to solve the
above problem, it works now but I am not sure whether it guarantee
always this task `do_image_make_reproducible` will run just before
`do_image_[tar,wic,squashfs]` tasks or does it need more changes to make
it guarantee, any suggestions?

[1] addtask image_make_reprodcible after do_rootfs before do_image

Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 meta/classes/image.bbclass | 33 ++++++++++++++++++++-------------
 1 file changed, 20 insertions(+), 13 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index ef7d5a2..c3a3d45 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -446,19 +446,6 @@ do_rootfs_finalize() {
         fi
 EOSUDO
 
-    # Set same time-stamps to the newly generated file/folders in the
-    # rootfs image for the purpose of reproducible builds.
-    if [ -n "${SOURCE_DATE_EPOCH}" ]; then
-        fn="${DEPLOY_DIR_IMAGE}/files.modified_timestamps"
-        if sudo find ${ROOTFSDIR} -newermt "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \
-            -printf "%y %p\n" -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' | egrep ^f >"$fn"; then
-            if [ -e "$fn" ]; then
-                bbwarn "modified timestamp (${SOURCE_DATE_EPOCH}) of $(cat "$fn" | wc -l) files for image reproducibly." \
-                       "List of files modified can be found in: .${DEPLOY_DIR_IMAGE}/files.modified_timestamps"
-            fi
-        fi
-    fi
-
 }
 do_rootfs_finalize[network] = "${TASK_USE_SUDO}"
 addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess
@@ -502,3 +489,23 @@ do_rootfs_quality_check() {
 do_rootfs_quality_check[network] = "${TASK_USE_SUDO}"
 
 addtask rootfs_quality_check after do_rootfs_finalize before do_rootfs
+
+# Run this task just before final image creation of different image types
+#     i.e do_image_tar, do_image_squashfs, do_image_wic etc.
+#     to avoid modification of image contents that leads to non-reproducible
+#     image
+do_image_make_reproducible() {
+    # Set same time-stamps to the newly generated file/folders in the
+    # rootfs image for the purpose of reproducible builds.
+    if [ -n "${SOURCE_DATE_EPOCH}" ]; then
+        fn="${DEPLOY_DIR_IMAGE}/files.modified_timestamps"
+        if sudo find ${ROOTFSDIR} -newermt "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \
+            -printf "%y %p\n" -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' | egrep ^f >"$fn"; then
+            if [ -e "$fn" ]; then
+                bbwarn "modified timestamp (${SOURCE_DATE_EPOCH}) of $(cat "$fn" | wc -l) files for image reproducibly." \
+                       "List of files modified can be found in: .${DEPLOY_DIR_IMAGE}/files.modified_timestamps"
+            fi
+        fi
+    fi
+}
+addtask image_make_reprodcible after do_rootfs before do_image
-- 
2.20.1

Re: [RFC PATCH] image.bbclass: create separate task for creating reproducible image

[Jan Kiszka]

On 27.02.23 08:50, venkata.pyla@toshiba-tsip.com wrote:
> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> 
> Currently the reproducible fix for setting same file timestamps across
> builds is present in the function `do_rootfs_finalize` and this fix may
> be skipped for the tasks that are added after this task or `do_rootfs`
>  for e.g. in one of the child project(CIP) for swupdate target it adds
>  additional task `do_generate_image_uuid` that is added after `do_rootfs`
>  this task is updating some contents in the rootfs which is skipping the
>  reproducible fix was applied in `do_rootfs_finalize`.
> 
> For this reason a separate task `do_image_make_reproducible` is created
> which should call before the final target image is created [tar, wic,
> squashfs] so that it applies reproducible fixes just before image
> creation.
> 
> RFC:
> In this patch I tried below dependency[1] for the function to solve the
> above problem, it works now but I am not sure whether it guarantee
> always this task `do_image_make_reproducible` will run just before
> `do_image_[tar,wic,squashfs]` tasks or does it need more changes to make
> it guarantee, any suggestions?
> 
> [1] addtask image_make_reprodcible after do_rootfs before do_image
> 

Isn't it possible to adjust isar-cip-core to perform
do_generate_image_uuid before do_rootfs_finalize? Didn't check all
details yet, just wondering.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

RE: [RFC PATCH] image.bbclass: create separate task for creating reproducible image

[Venkata.Pyla]

>-----Original Message-----
>From: Jan Kiszka <jan.kiszka@siemens.com>
>Sent: 27 February 2023 14:28
>To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-
>tsip.com>; isar-users@googlegroups.com
>Cc: felix.moessbauer@siemens.com; roberto.foglietta@gmail.com;
>henning.schild@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ)
><kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>Porting) <dinesh.kumar@toshiba-tsip.com>
>Subject: Re: [RFC PATCH] image.bbclass: create separate task for creating
>reproducible image
>
>On 27.02.23 08:50, venkata.pyla@toshiba-tsip.com wrote:
>> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
>>
>> Currently the reproducible fix for setting same file timestamps across
>> builds is present in the function `do_rootfs_finalize` and this fix
>> may be skipped for the tasks that are added after this task or
>> `do_rootfs`  for e.g. in one of the child project(CIP) for swupdate
>> target it adds  additional task `do_generate_image_uuid` that is added
>> after `do_rootfs`  this task is updating some contents in the rootfs
>> which is skipping the  reproducible fix was applied in `do_rootfs_finalize`.
>>
>> For this reason a separate task `do_image_make_reproducible` is
>> created which should call before the final target image is created
>> [tar, wic, squashfs] so that it applies reproducible fixes just before
>> image creation.
>>
>> RFC:
>> In this patch I tried below dependency[1] for the function to solve
>> the above problem, it works now but I am not sure whether it guarantee
>> always this task `do_image_make_reproducible` will run just before
>> `do_image_[tar,wic,squashfs]` tasks or does it need more changes to
>> make it guarantee, any suggestions?
>>
>> [1] addtask image_make_reprodcible after do_rootfs before do_image
>>
>
>Isn't it possible to adjust isar-cip-core to perform do_generate_image_uuid
>before do_rootfs_finalize? Didn't check all details yet, just wondering.
>

Yeah that should be possible by changing its dependencies order,
but I just tried if it can be fixed in base recipes to address this problem commonly.

>Jan
>
>--
>Siemens AG, Technology
>Competence Center Embedded Linux

Re: [RFC PATCH] image.bbclass: create separate task for creating reproducible image

[Jan Kiszka]

On 27.02.23 10:57, Venkata.Pyla@toshiba-tsip.com wrote:
> 
> 
>> -----Original Message-----
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>> Sent: 27 February 2023 14:28
>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-
>> tsip.com>; isar-users@googlegroups.com
>> Cc: felix.moessbauer@siemens.com; roberto.foglietta@gmail.com;
>> henning.schild@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ)
>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>> Porting) <dinesh.kumar@toshiba-tsip.com>
>> Subject: Re: [RFC PATCH] image.bbclass: create separate task for creating
>> reproducible image
>>
>> On 27.02.23 08:50, venkata.pyla@toshiba-tsip.com wrote:
>>> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
>>>
>>> Currently the reproducible fix for setting same file timestamps across
>>> builds is present in the function `do_rootfs_finalize` and this fix
>>> may be skipped for the tasks that are added after this task or
>>> `do_rootfs`  for e.g. in one of the child project(CIP) for swupdate
>>> target it adds  additional task `do_generate_image_uuid` that is added
>>> after `do_rootfs`  this task is updating some contents in the rootfs
>>> which is skipping the  reproducible fix was applied in `do_rootfs_finalize`.
>>>
>>> For this reason a separate task `do_image_make_reproducible` is
>>> created which should call before the final target image is created
>>> [tar, wic, squashfs] so that it applies reproducible fixes just before
>>> image creation.
>>>
>>> RFC:
>>> In this patch I tried below dependency[1] for the function to solve
>>> the above problem, it works now but I am not sure whether it guarantee
>>> always this task `do_image_make_reproducible` will run just before
>>> `do_image_[tar,wic,squashfs]` tasks or does it need more changes to
>>> make it guarantee, any suggestions?
>>>
>>> [1] addtask image_make_reprodcible after do_rootfs before do_image
>>>
>>
>> Isn't it possible to adjust isar-cip-core to perform do_generate_image_uuid
>> before do_rootfs_finalize? Didn't check all details yet, just wondering.
>>
> 
> Yeah that should be possible by changing its dependencies order,
> but I just tried if it can be fixed in base recipes to address this problem commonly.

Well, the question is if the rather imprecise ordering in downstream
isn't the actual issue. Or do you see other values in factoring this
thing here out into an own task?

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

RE: [RFC PATCH] image.bbclass: create separate task for creating reproducible image

[Venkata.Pyla]

>-----Original Message-----
>From: Jan Kiszka <jan.kiszka@siemens.com>
>Sent: 27 February 2023 15:33
>To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-
>tsip.com>; isar-users@googlegroups.com
>Cc: felix.moessbauer@siemens.com; roberto.foglietta@gmail.com;
>henning.schild@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ)
><kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>Porting) <dinesh.kumar@toshiba-tsip.com>
>Subject: Re: [RFC PATCH] image.bbclass: create separate task for creating
>reproducible image
>
>On 27.02.23 10:57, Venkata.Pyla@toshiba-tsip.com wrote:
>>
>>
>>> -----Original Message-----
>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>> Sent: 27 February 2023 14:28
>>> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-
>>> tsip.com>; isar-users@googlegroups.com
>>> Cc: felix.moessbauer@siemens.com; roberto.foglietta@gmail.com;
>>> henning.schild@siemens.com; hayashi kazuhiro(林 和宏 □ＳＷＣ◯ＡＣＴ)
>>> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>>> Porting) <dinesh.kumar@toshiba-tsip.com>
>>> Subject: Re: [RFC PATCH] image.bbclass: create separate task for
>>> creating reproducible image
>>>
>>> On 27.02.23 08:50, venkata.pyla@toshiba-tsip.com wrote:
>>>> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
>>>>
>>>> Currently the reproducible fix for setting same file timestamps
>>>> across builds is present in the function `do_rootfs_finalize` and
>>>> this fix may be skipped for the tasks that are added after this task
>>>> or `do_rootfs`  for e.g. in one of the child project(CIP) for
>>>> swupdate target it adds  additional task `do_generate_image_uuid`
>>>> that is added after `do_rootfs`  this task is updating some contents
>>>> in the rootfs which is skipping the  reproducible fix was applied in
>`do_rootfs_finalize`.
>>>>
>>>> For this reason a separate task `do_image_make_reproducible` is
>>>> created which should call before the final target image is created
>>>> [tar, wic, squashfs] so that it applies reproducible fixes just
>>>> before image creation.
>>>>
>>>> RFC:
>>>> In this patch I tried below dependency[1] for the function to solve
>>>> the above problem, it works now but I am not sure whether it
>>>> guarantee always this task `do_image_make_reproducible` will run
>>>> just before `do_image_[tar,wic,squashfs]` tasks or does it need more
>>>> changes to make it guarantee, any suggestions?
>>>>
>>>> [1] addtask image_make_reprodcible after do_rootfs before do_image
>>>>
>>>
>>> Isn't it possible to adjust isar-cip-core to perform
>>> do_generate_image_uuid before do_rootfs_finalize? Didn't check all details
>yet, just wondering.
>>>
>>
>> Yeah that should be possible by changing its dependencies order, but I
>> just tried if it can be fixed in base recipes to address this problem commonly.
>
>Well, the question is if the rather imprecise ordering in downstream isn't the
>actual issue. Or do you see other values in factoring this thing here out into an
>own task?

Fixing this ordering problem is main goal for adding this new task, other is we can extend in future to add any other reproducible fixes in to the same new task.

The main reason for creating this new task is to set this task run just before the image creation (not sure how to guarantee it in bitbake or it is already using before and after clauses)
So that in any such cases like `do_generate_image_uuid ` task which is created after do_rootfs and skiping the reproducible fix should not happen with other task.

Do you think creating new task is not required here and fixing the imprecise ordering in downstream or with any other task is suffice?

>
>Jan
>
>--
>Siemens AG, Technology
>Competence Center Embedded Linux

Re: [RFC PATCH] image.bbclass: create separate task for creating reproducible image

[Henning Schild]

Please do not default cc me on reproducible image topics. I do not care
enough and will see it coming or see it on the list.

Henning

Am Mon, 27 Feb 2023 13:20:32 +0530
schrieb venkata.pyla@toshiba-tsip.com:

> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> 
> Currently the reproducible fix for setting same file timestamps across
> builds is present in the function `do_rootfs_finalize` and this fix
> may be skipped for the tasks that are added after this task or
> `do_rootfs` for e.g. in one of the child project(CIP) for swupdate
> target it adds additional task `do_generate_image_uuid` that is added
> after `do_rootfs` this task is updating some contents in the rootfs
> which is skipping the reproducible fix was applied in
> `do_rootfs_finalize`.
> 
> For this reason a separate task `do_image_make_reproducible` is
> created which should call before the final target image is created
> [tar, wic, squashfs] so that it applies reproducible fixes just
> before image creation.
> 
> RFC:
> In this patch I tried below dependency[1] for the function to solve
> the above problem, it works now but I am not sure whether it guarantee
> always this task `do_image_make_reproducible` will run just before
> `do_image_[tar,wic,squashfs]` tasks or does it need more changes to
> make it guarantee, any suggestions?
> 
> [1] addtask image_make_reprodcible after do_rootfs before do_image
> 
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  meta/classes/image.bbclass | 33 ++++++++++++++++++++-------------
>  1 file changed, 20 insertions(+), 13 deletions(-)
> 
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index ef7d5a2..c3a3d45 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -446,19 +446,6 @@ do_rootfs_finalize() {
>          fi
>  EOSUDO
>  
> -    # Set same time-stamps to the newly generated file/folders in the
> -    # rootfs image for the purpose of reproducible builds.
> -    if [ -n "${SOURCE_DATE_EPOCH}" ]; then
> -        fn="${DEPLOY_DIR_IMAGE}/files.modified_timestamps"
> -        if sudo find ${ROOTFSDIR} -newermt "$(date
> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \
> -            -printf "%y %p\n" -exec touch '{}' -h
> -d@${SOURCE_DATE_EPOCH} ';' | egrep ^f >"$fn"; then
> -            if [ -e "$fn" ]; then
> -                bbwarn "modified timestamp (${SOURCE_DATE_EPOCH}) of
> $(cat "$fn" | wc -l) files for image reproducibly." \
> -                       "List of files modified can be found in:
> .${DEPLOY_DIR_IMAGE}/files.modified_timestamps"
> -            fi
> -        fi
> -    fi
> -
>  }
>  do_rootfs_finalize[network] = "${TASK_USE_SUDO}"
>  addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess
> @@ -502,3 +489,23 @@ do_rootfs_quality_check() {
>  do_rootfs_quality_check[network] = "${TASK_USE_SUDO}"
>  
>  addtask rootfs_quality_check after do_rootfs_finalize before
> do_rootfs +
> +# Run this task just before final image creation of different image
> types +#     i.e do_image_tar, do_image_squashfs, do_image_wic etc.
> +#     to avoid modification of image contents that leads to
> non-reproducible +#     image
> +do_image_make_reproducible() {
> +    # Set same time-stamps to the newly generated file/folders in the
> +    # rootfs image for the purpose of reproducible builds.
> +    if [ -n "${SOURCE_DATE_EPOCH}" ]; then
> +        fn="${DEPLOY_DIR_IMAGE}/files.modified_timestamps"
> +        if sudo find ${ROOTFSDIR} -newermt "$(date
> -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \
> +            -printf "%y %p\n" -exec touch '{}' -h
> -d@${SOURCE_DATE_EPOCH} ';' | egrep ^f >"$fn"; then
> +            if [ -e "$fn" ]; then
> +                bbwarn "modified timestamp (${SOURCE_DATE_EPOCH}) of
> $(cat "$fn" | wc -l) files for image reproducibly." \
> +                       "List of files modified can be found in:
> .${DEPLOY_DIR_IMAGE}/files.modified_timestamps"
> +            fi
> +        fi
> +    fi
> +}
> +addtask image_make_reprodcible after do_rootfs before do_image