[PATCH v2 0/1] Set hash-seed for empty ext partition

[PATCH v2 0/1] Set hash-seed for empty ext partition

[Adithya Balakumar]

ext filesystems require a hash_seed to generate deterministic directory indexes 
for reproducible builds. This is handled by wic in the case where the rootfs 
plugin is used but this not handled in the case when an empty ext4 partition 
is deployed.

This patch is also shared with openembedded-core and also accepted. 
master branch commit: 0202fb594fb05098cb8d8b6088e63beb40b5906e
scarthgap branch commit: a4e7334a4b87f1bb1947f6b10f71ddb445c91d0c

Changes since v1:
There is no change in the patch from v1. Just that this patch is also applied
to the scarthgap branch in OE-core. 

Adithya Balakumar (1):
  wic/partition.py: Set hash_seed for empty ext partition

 scripts/lib/wic/partition.py | 37 ++++++++++++++++++++++--------------
 1 file changed, 23 insertions(+), 14 deletions(-)

-- 
2.39.2


-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240704111053.1647620-1-Adithya.Balakumar%40toshiba-tsip.com.

[PATCH v2 1/1] wic/partition.py: Set hash_seed for empty ext partition

[Adithya Balakumar]

From: Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>

Although setting hash_seed is handled for the rootfs plugin case, but
this is missed when deploying an empty ext partition.

Signed-off-by: Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
---
 scripts/lib/wic/partition.py | 37 ++++++++++++++++++++++--------------
 1 file changed, 23 insertions(+), 14 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 795707ec..bf2c34d5 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -284,19 +284,8 @@ class Partition():
 
         extraopts = self.mkfs_extraopts or "-F -i 8192"
 
-        if os.getenv('SOURCE_DATE_EPOCH'):
-            sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
-            if pseudo:
-                pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s " % (sde_time, pseudo)
-            else:
-                pseudo = "export E2FSPROGS_FAKE_TIME=%s; " % sde_time
-
-            # Set hash_seed to generate deterministic directory indexes
-            namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-2f2fdf33d460")
-            if self.fsuuid:
-                namespace = uuid.UUID(self.fsuuid)
-            hash_seed = str(uuid.uuid5(namespace, str(sde_time)))
-            extraopts += " -E hash_seed=%s" % hash_seed
+        # use hash_seed to generate reproducible ext4 images
+        (extraopts, pseudo) = self.get_hash_seed_ext4(extraopts, pseudo)
 
         label_str = ""
         if self.label:
@@ -344,6 +333,23 @@ class Partition():
 
         self.check_for_Y2038_problem(rootfs, native_sysroot)
 
+    def get_hash_seed_ext4(self, extraopts, pseudo):
+        if os.getenv('SOURCE_DATE_EPOCH'):
+            sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
+            if pseudo:
+                pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s " % (sde_time, pseudo)
+            else:
+                pseudo = "export E2FSPROGS_FAKE_TIME=%s; " % sde_time
+
+            # Set hash_seed to generate deterministic directory indexes
+            namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-2f2fdf33d460")
+            if self.fsuuid:
+                namespace = uuid.UUID(self.fsuuid)
+            hash_seed = str(uuid.uuid5(namespace, str(sde_time)))
+            extraopts += " -E hash_seed=%s" % hash_seed
+
+        return (extraopts, pseudo)
+
     def prepare_rootfs_btrfs(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
                              native_sysroot, pseudo):
         """
@@ -437,13 +443,16 @@ class Partition():
 
         extraopts = self.mkfs_extraopts or "-i 8192"
 
+        # use hash_seed to generate reproducible ext4 images
+        (extraopts, pseudo) = self.get_hash_seed_ext4(extraopts, None)
+
         label_str = ""
         if self.label:
             label_str = "-L %s" % self.label
 
         mkfs_cmd = "mkfs.%s -F %s %s -U %s %s" % \
             (self.fstype, extraopts, label_str, self.fsuuid, rootfs)
-        exec_native_cmd(mkfs_cmd, native_sysroot)
+        exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
 
         self.check_for_Y2038_problem(rootfs, native_sysroot)
 
-- 
2.39.2


-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240704111053.1647620-2-Adithya.Balakumar%40toshiba-tsip.com.

Re: [PATCH v2 1/1] wic/partition.py: Set hash_seed for empty ext partition

['MOESSBAUER, Felix' via isar-users]

On Thu, 2024-07-04 at 16:40 +0530, Adithya Balakumar wrote:
> From: Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
> 
> Although setting hash_seed is handled for the rootfs plugin case, but
> this is missed when deploying an empty ext partition.
> 
> Signed-off-by: Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
> ---
>  scripts/lib/wic/partition.py | 37 ++++++++++++++++++++++------------
> --
>  1 file changed, 23 insertions(+), 14 deletions(-)
> 
> diff --git a/scripts/lib/wic/partition.py
> b/scripts/lib/wic/partition.py
> index 795707ec..bf2c34d5 100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -284,19 +284,8 @@ class Partition():
>  
>          extraopts = self.mkfs_extraopts or "-F -i 8192"
>  
> -        if os.getenv('SOURCE_DATE_EPOCH'):
> -            sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
> -            if pseudo:
> -                pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s " %
> (sde_time, pseudo)
> -            else:
> -                pseudo = "export E2FSPROGS_FAKE_TIME=%s; " %
> sde_time
> -
> -            # Set hash_seed to generate deterministic directory
> indexes
> -            namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-
> 2f2fdf33d460")
> -            if self.fsuuid:
> -                namespace = uuid.UUID(self.fsuuid)
> -            hash_seed = str(uuid.uuid5(namespace, str(sde_time)))
> -            extraopts += " -E hash_seed=%s" % hash_seed
> +        # use hash_seed to generate reproducible ext4 images
> +        (extraopts, pseudo) = self.get_hash_seed_ext4(extraopts,
> pseudo)
>  
>          label_str = ""
>          if self.label:
> @@ -344,6 +333,23 @@ class Partition():
>  
>          self.check_for_Y2038_problem(rootfs, native_sysroot)
>  
> +    def get_hash_seed_ext4(self, extraopts, pseudo):
> +        if os.getenv('SOURCE_DATE_EPOCH'):

Hi, isn't that always true since we (and OE) nowadays always set this
variable? Why do we need special handling in this case?

Felix

> +            sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
> +            if pseudo:
> +                pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s " %
> (sde_time, pseudo)
> +            else:
> +                pseudo = "export E2FSPROGS_FAKE_TIME=%s; " %
> sde_time
> +
> +            # Set hash_seed to generate deterministic directory
> indexes
> +            namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-
> 2f2fdf33d460")
> +            if self.fsuuid:
> +                namespace = uuid.UUID(self.fsuuid)
> +            hash_seed = str(uuid.uuid5(namespace, str(sde_time)))
> +            extraopts += " -E hash_seed=%s" % hash_seed
> +
> +        return (extraopts, pseudo)
> +
>      def prepare_rootfs_btrfs(self, rootfs, cr_workdir, oe_builddir,
> rootfs_dir,
>                               native_sysroot, pseudo):
>          """
> @@ -437,13 +443,16 @@ class Partition():
>  
>          extraopts = self.mkfs_extraopts or "-i 8192"
>  
> +        # use hash_seed to generate reproducible ext4 images
> +        (extraopts, pseudo) = self.get_hash_seed_ext4(extraopts,
> None)
> +
>          label_str = ""
>          if self.label:
>              label_str = "-L %s" % self.label
>  
>          mkfs_cmd = "mkfs.%s -F %s %s -U %s %s" % \
>              (self.fstype, extraopts, label_str, self.fsuuid, rootfs)
> -        exec_native_cmd(mkfs_cmd, native_sysroot)
> +        exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
>  
>          self.check_for_Y2038_problem(rootfs, native_sysroot)
>  
> -- 
> 2.39.2
> 
> 

-- 
Siemens AG, Technology
Linux Expert Center


-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/f6682f087e1bb9c9324a508700da4b2f8bc78411.camel%40siemens.com.

Re: [PATCH v2 0/1] Set hash-seed for empty ext partition

['Florian Bezdeka' via isar-users]

On Thu, 2024-07-04 at 16:40 +0530, Adithya Balakumar wrote:
> ext filesystems require a hash_seed to generate deterministic directory indexes 
> for reproducible builds. This is handled by wic in the case where the rootfs 
> plugin is used but this not handled in the case when an empty ext4 partition 
> is deployed.
> 
> This patch is also shared with openembedded-core and also accepted. 
> master branch commit: 0202fb594fb05098cb8d8b6088e63beb40b5906e
> scarthgap branch commit: a4e7334a4b87f1bb1947f6b10f71ddb445c91d0c

Thanks for upstreaming!

Instead of applying a single patch I would vote for triggering a
complete wic update.

> 
> Changes since v1:
> There is no change in the patch from v1. Just that this patch is also applied
> to the scarthgap branch in OE-core. 
> 
> Adithya Balakumar (1):
>   wic/partition.py: Set hash_seed for empty ext partition
> 
>  scripts/lib/wic/partition.py | 37 ++++++++++++++++++++++--------------
>  1 file changed, 23 insertions(+), 14 deletions(-)
> 
> -- 
> 2.39.2

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/e747bf3630ef891b04d8d8ddf916eb8a5f72c09c.camel%40siemens.com.

RE: [PATCH v2 1/1] wic/partition.py: Set hash_seed for empty ext partition

[Adithya.Balakumar]

Hi Felix,

Thanks for the feedback. You are right, there is no need for a check for existence of the SOURCE_DATE_EPOCH environment variable since it is always available. I see this check in quite a few places in the wic source code upstream (OE) as well. 

Probably this could be addressed in a separate patch to upstream and then subsequently to isar as well. What do you think?

Thanks and Regards,
Adithya Balakumar

-----Original Message-----
From: MOESSBAUER, Felix <felix.moessbauer@siemens.com> 
Sent: Thursday, July 4, 2024 5:19 PM
To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; isar-users@googlegroups.com; amikan@ilbers.de
Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; Kiszka, Jan <jan.kiszka@siemens.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>
Subject: Re: [PATCH v2 1/1] wic/partition.py: Set hash_seed for empty ext partition

On Thu, 2024-07-04 at 16:40 +0530, Adithya Balakumar wrote:
> From: Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
> 
> Although setting hash_seed is handled for the rootfs plugin case, but 
> this is missed when deploying an empty ext partition.
> 
> Signed-off-by: Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
> ---
>  scripts/lib/wic/partition.py | 37 ++++++++++++++++++++++------------
> --
>  1 file changed, 23 insertions(+), 14 deletions(-)
> 
> diff --git a/scripts/lib/wic/partition.py 
> b/scripts/lib/wic/partition.py index 795707ec..bf2c34d5 100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -284,19 +284,8 @@ class Partition():
>  
>          extraopts = self.mkfs_extraopts or "-F -i 8192"
>  
> -        if os.getenv('SOURCE_DATE_EPOCH'):
> -            sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
> -            if pseudo:
> -                pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s " % 
> (sde_time, pseudo)
> -            else:
> -                pseudo = "export E2FSPROGS_FAKE_TIME=%s; " % sde_time
> -
> -            # Set hash_seed to generate deterministic directory 
> indexes
> -            namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-
> 2f2fdf33d460")
> -            if self.fsuuid:
> -                namespace = uuid.UUID(self.fsuuid)
> -            hash_seed = str(uuid.uuid5(namespace, str(sde_time)))
> -            extraopts += " -E hash_seed=%s" % hash_seed
> +        # use hash_seed to generate reproducible ext4 images
> +        (extraopts, pseudo) = self.get_hash_seed_ext4(extraopts,
> pseudo)
>  
>          label_str = ""
>          if self.label:
> @@ -344,6 +333,23 @@ class Partition():
>  
>          self.check_for_Y2038_problem(rootfs, native_sysroot)
>  
> +    def get_hash_seed_ext4(self, extraopts, pseudo):
> +        if os.getenv('SOURCE_DATE_EPOCH'):

Hi, isn't that always true since we (and OE) nowadays always set this variable? Why do we need special handling in this case?

Felix

> +            sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
> +            if pseudo:
> +                pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s " %
> (sde_time, pseudo)
> +            else:
> +                pseudo = "export E2FSPROGS_FAKE_TIME=%s; " %
> sde_time
> +
> +            # Set hash_seed to generate deterministic directory
> indexes
> +            namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-
> 2f2fdf33d460")
> +            if self.fsuuid:
> +                namespace = uuid.UUID(self.fsuuid)
> +            hash_seed = str(uuid.uuid5(namespace, str(sde_time)))
> +            extraopts += " -E hash_seed=%s" % hash_seed
> +
> +        return (extraopts, pseudo)
> +
>      def prepare_rootfs_btrfs(self, rootfs, cr_workdir, oe_builddir, 
> rootfs_dir,
>                               native_sysroot, pseudo):
>          """
> @@ -437,13 +443,16 @@ class Partition():
>  
>          extraopts = self.mkfs_extraopts or "-i 8192"
>  
> +        # use hash_seed to generate reproducible ext4 images
> +        (extraopts, pseudo) = self.get_hash_seed_ext4(extraopts,
> None)
> +
>          label_str = ""
>          if self.label:
>              label_str = "-L %s" % self.label
>  
>          mkfs_cmd = "mkfs.%s -F %s %s -U %s %s" % \
>              (self.fstype, extraopts, label_str, self.fsuuid, rootfs)
> -        exec_native_cmd(mkfs_cmd, native_sysroot)
> +        exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
>  
>          self.check_for_Y2038_problem(rootfs, native_sysroot)
>  
> --
> 2.39.2
> 
> 

--
Siemens AG, Technology
Linux Expert Center


-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/TYCPR01MB9669BCB41F05029069596503C4DF2%40TYCPR01MB9669.jpnprd01.prod.outlook.com.

RE: [PATCH v2 0/1] Set hash-seed for empty ext partition

[Adithya.Balakumar]

Hi Florian,

As I remember, ISAR just updated wic for the latest LTS release of OE (scarthgap) back in May, and this is the only patch to the scarthgap branch that is wic related since then. 

If I understand correctly, ISAR updates OE related libraries / dependencies for every LTS release in OE. Please correct me if I am mistaken.

Thanks and Regards,
Adithya Balakumar

-----Original Message-----
From: Florian Bezdeka <florian.bezdeka@siemens.com> 
Sent: Thursday, July 4, 2024 9:14 PM
To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; isar-users@googlegroups.com; amikan@ilbers.de
Cc: jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>
Subject: Re: [PATCH v2 0/1] Set hash-seed for empty ext partition

On Thu, 2024-07-04 at 16:40 +0530, Adithya Balakumar wrote:
> ext filesystems require a hash_seed to generate deterministic 
> directory indexes for reproducible builds. This is handled by wic in 
> the case where the rootfs plugin is used but this not handled in the 
> case when an empty ext4 partition is deployed.
> 
> This patch is also shared with openembedded-core and also accepted. 
> master branch commit: 0202fb594fb05098cb8d8b6088e63beb40b5906e
> scarthgap branch commit: a4e7334a4b87f1bb1947f6b10f71ddb445c91d0c

Thanks for upstreaming!

Instead of applying a single patch I would vote for triggering a complete wic update.

> 
> Changes since v1:
> There is no change in the patch from v1. Just that this patch is also 
> applied to the scarthgap branch in OE-core.
> 
> Adithya Balakumar (1):
>   wic/partition.py: Set hash_seed for empty ext partition
> 
>  scripts/lib/wic/partition.py | 37 
> ++++++++++++++++++++++--------------
>  1 file changed, 23 insertions(+), 14 deletions(-)
> 
> --
> 2.39.2

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/TYCPR01MB9669173E8E3748D386C8DC1CC4DF2%40TYCPR01MB9669.jpnprd01.prod.outlook.com.

Re: [PATCH v2 1/1] wic/partition.py: Set hash_seed for empty ext partition

['MOESSBAUER, Felix' via isar-users]

On Fri, 2024-07-05 at 07:34 +0000, Adithya.Balakumar@toshiba-tsip.com
wrote:
> Hi Felix,
> 
> Thanks for the feedback. You are right, there is no need for a check
> for existence of the SOURCE_DATE_EPOCH environment variable since it
> is always available. I see this check in quite a few places in the
> wic source code upstream (OE) as well. 
> 
> Probably this could be addressed in a separate patch to upstream and
> then subsequently to isar as well. What do you think?

Ok, fine for me.

Felix

> 
> Thanks and Regards,
> Adithya Balakumar
> 
> -----Original Message-----
> From: MOESSBAUER, Felix <felix.moessbauer@siemens.com> 
> Sent: Thursday, July 4, 2024 5:19 PM
> To: balakumar adithya(ＴＳＩＰ TEUR)
> <Adithya.Balakumar@toshiba-tsip.com>; isar-users@googlegroups.com;
> amikan@ilbers.de
> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting)
> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
> <Sai.Sathujoda@toshiba-tsip.com>; Kiszka, Jan
> <jan.kiszka@siemens.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４)
> <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG
> Porting) <dinesh.kumar@toshiba-tsip.com>
> Subject: Re: [PATCH v2 1/1] wic/partition.py: Set hash_seed for empty
> ext partition
> 
> On Thu, 2024-07-04 at 16:40 +0530, Adithya Balakumar wrote:
> > From: Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
> > 
> > Although setting hash_seed is handled for the rootfs plugin case,
> > but 
> > this is missed when deploying an empty ext partition.
> > 
> > Signed-off-by: Adithya Balakumar
> > <adithya.balakumar@toshiba-tsip.com>
> > ---
> >  scripts/lib/wic/partition.py | 37 ++++++++++++++++++++++----------
> > --
> > --
> >  1 file changed, 23 insertions(+), 14 deletions(-)
> > 
> > diff --git a/scripts/lib/wic/partition.py 
> > b/scripts/lib/wic/partition.py index 795707ec..bf2c34d5 100644
> > --- a/scripts/lib/wic/partition.py
> > +++ b/scripts/lib/wic/partition.py
> > @@ -284,19 +284,8 @@ class Partition():
> >  
> >          extraopts = self.mkfs_extraopts or "-F -i 8192"
> >  
> > -        if os.getenv('SOURCE_DATE_EPOCH'):
> > -            sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
> > -            if pseudo:
> > -                pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s " % 
> > (sde_time, pseudo)
> > -            else:
> > -                pseudo = "export E2FSPROGS_FAKE_TIME=%s; " %
> > sde_time
> > -
> > -            # Set hash_seed to generate deterministic directory 
> > indexes
> > -            namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-
> > 2f2fdf33d460")
> > -            if self.fsuuid:
> > -                namespace = uuid.UUID(self.fsuuid)
> > -            hash_seed = str(uuid.uuid5(namespace, str(sde_time)))
> > -            extraopts += " -E hash_seed=%s" % hash_seed
> > +        # use hash_seed to generate reproducible ext4 images
> > +        (extraopts, pseudo) = self.get_hash_seed_ext4(extraopts,
> > pseudo)
> >  
> >          label_str = ""
> >          if self.label:
> > @@ -344,6 +333,23 @@ class Partition():
> >  
> >          self.check_for_Y2038_problem(rootfs, native_sysroot)
> >  
> > +    def get_hash_seed_ext4(self, extraopts, pseudo):
> > +        if os.getenv('SOURCE_DATE_EPOCH'):
> 
> Hi, isn't that always true since we (and OE) nowadays always set this
> variable? Why do we need special handling in this case?
> 
> Felix
> 
> > +            sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
> > +            if pseudo:
> > +                pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s " %
> > (sde_time, pseudo)
> > +            else:
> > +                pseudo = "export E2FSPROGS_FAKE_TIME=%s; " %
> > sde_time
> > +
> > +            # Set hash_seed to generate deterministic directory
> > indexes
> > +            namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-
> > 2f2fdf33d460")
> > +            if self.fsuuid:
> > +                namespace = uuid.UUID(self.fsuuid)
> > +            hash_seed = str(uuid.uuid5(namespace, str(sde_time)))
> > +            extraopts += " -E hash_seed=%s" % hash_seed
> > +
> > +        return (extraopts, pseudo)
> > +
> >      def prepare_rootfs_btrfs(self, rootfs, cr_workdir,
> > oe_builddir, 
> > rootfs_dir,
> >                               native_sysroot, pseudo):
> >          """
> > @@ -437,13 +443,16 @@ class Partition():
> >  
> >          extraopts = self.mkfs_extraopts or "-i 8192"
> >  
> > +        # use hash_seed to generate reproducible ext4 images
> > +        (extraopts, pseudo) = self.get_hash_seed_ext4(extraopts,
> > None)
> > +
> >          label_str = ""
> >          if self.label:
> >              label_str = "-L %s" % self.label
> >  
> >          mkfs_cmd = "mkfs.%s -F %s %s -U %s %s" % \
> >              (self.fstype, extraopts, label_str, self.fsuuid,
> > rootfs)
> > -        exec_native_cmd(mkfs_cmd, native_sysroot)
> > +        exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
> >  
> >          self.check_for_Y2038_problem(rootfs, native_sysroot)
> >  
> > --
> > 2.39.2
> > 
> > 
> 
> --
> Siemens AG, Technology
> Linux Expert Center
> 
> 

-- 
Siemens AG, Technology
Linux Expert Center


-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/0bc79c69d955aa1fcc5d78b185fd4b0d9478a52b.camel%40siemens.com.

RE: [PATCH v2 1/1] wic/partition.py: Set hash_seed for empty ext partition

[Adithya.Balakumar]

Hi All,

Any update on this patch?

Thanks and Regards,
Adithya

-----Original Message-----
From: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com> 
Sent: Thursday, July 4, 2024 4:41 PM
To: isar-users@googlegroups.com; amikan@ilbers.de
Cc: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>
Subject: [PATCH v2 1/1] wic/partition.py: Set hash_seed for empty ext partition

From: Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>

Although setting hash_seed is handled for the rootfs plugin case, but this is missed when deploying an empty ext partition.

Signed-off-by: Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
---
 scripts/lib/wic/partition.py | 37 ++++++++++++++++++++++--------------
 1 file changed, 23 insertions(+), 14 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py index 795707ec..bf2c34d5 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -284,19 +284,8 @@ class Partition():
 
         extraopts = self.mkfs_extraopts or "-F -i 8192"
 
-        if os.getenv('SOURCE_DATE_EPOCH'):
-            sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
-            if pseudo:
-                pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s " % (sde_time, pseudo)
-            else:
-                pseudo = "export E2FSPROGS_FAKE_TIME=%s; " % sde_time
-
-            # Set hash_seed to generate deterministic directory indexes
-            namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-2f2fdf33d460")
-            if self.fsuuid:
-                namespace = uuid.UUID(self.fsuuid)
-            hash_seed = str(uuid.uuid5(namespace, str(sde_time)))
-            extraopts += " -E hash_seed=%s" % hash_seed
+        # use hash_seed to generate reproducible ext4 images
+        (extraopts, pseudo) = self.get_hash_seed_ext4(extraopts, 
+ pseudo)
 
         label_str = ""
         if self.label:
@@ -344,6 +333,23 @@ class Partition():
 
         self.check_for_Y2038_problem(rootfs, native_sysroot)
 
+    def get_hash_seed_ext4(self, extraopts, pseudo):
+        if os.getenv('SOURCE_DATE_EPOCH'):
+            sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
+            if pseudo:
+                pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s " % (sde_time, pseudo)
+            else:
+                pseudo = "export E2FSPROGS_FAKE_TIME=%s; " % sde_time
+
+            # Set hash_seed to generate deterministic directory indexes
+            namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-2f2fdf33d460")
+            if self.fsuuid:
+                namespace = uuid.UUID(self.fsuuid)
+            hash_seed = str(uuid.uuid5(namespace, str(sde_time)))
+            extraopts += " -E hash_seed=%s" % hash_seed
+
+        return (extraopts, pseudo)
+
     def prepare_rootfs_btrfs(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
                              native_sysroot, pseudo):
         """
@@ -437,13 +443,16 @@ class Partition():
 
         extraopts = self.mkfs_extraopts or "-i 8192"
 
+        # use hash_seed to generate reproducible ext4 images
+        (extraopts, pseudo) = self.get_hash_seed_ext4(extraopts, None)
+
         label_str = ""
         if self.label:
             label_str = "-L %s" % self.label
 
         mkfs_cmd = "mkfs.%s -F %s %s -U %s %s" % \
             (self.fstype, extraopts, label_str, self.fsuuid, rootfs)
-        exec_native_cmd(mkfs_cmd, native_sysroot)
+        exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
 
         self.check_for_Y2038_problem(rootfs, native_sysroot)
 
--
2.39.2


-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/TYCPR01MB966937A9AF18EEC1DF289492C4A12%40TYCPR01MB9669.jpnprd01.prod.outlook.com.

Re: [PATCH v2 0/1] Set hash-seed for empty ext partition

['Jan Kiszka' via isar-users]

On 04.07.24 13:10, Adithya Balakumar wrote:
> ext filesystems require a hash_seed to generate deterministic directory indexes 
> for reproducible builds. This is handled by wic in the case where the rootfs 
> plugin is used but this not handled in the case when an empty ext4 partition 
> is deployed.
> 
> This patch is also shared with openembedded-core and also accepted. 
> master branch commit: 0202fb594fb05098cb8d8b6088e63beb40b5906e
> scarthgap branch commit: a4e7334a4b87f1bb1947f6b10f71ddb445c91d0c
> 

So, your change lifts wic to commit a4e7334a4b? This remains unclear
from your patch but, given that your patch is accepted to stable now, it
is the way to go for isar now.

Jan

> Changes since v1:
> There is no change in the patch from v1. Just that this patch is also applied
> to the scarthgap branch in OE-core. 
> 
> Adithya Balakumar (1):
>   wic/partition.py: Set hash_seed for empty ext partition
> 
>  scripts/lib/wic/partition.py | 37 ++++++++++++++++++++++--------------
>  1 file changed, 23 insertions(+), 14 deletions(-)
> 

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/b69f7036-6537-4e96-8c89-7e4cb1a9fe7d%40siemens.com.

RE: [PATCH v2 0/1] Set hash-seed for empty ext partition

[Adithya.Balakumar]

-----Original Message-----
From: Jan Kiszka <jan.kiszka@siemens.com> 
Sent: Tuesday, July 16, 2024 2:30 PM
To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; isar-users@googlegroups.com; amikan@ilbers.de
Cc: hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>
Subject: Re: [PATCH v2 0/1] Set hash-seed for empty ext partition

On 04.07.24 13:10, Adithya Balakumar wrote:
> ext filesystems require a hash_seed to generate deterministic 
> directory indexes for reproducible builds. This is handled by wic in 
> the case where the rootfs plugin is used but this not handled in the 
> case when an empty ext4 partition is deployed.
> 
> This patch is also shared with openembedded-core and also accepted. 
> master branch commit: 0202fb594fb05098cb8d8b6088e63beb40b5906e
> scarthgap branch commit: a4e7334a4b87f1bb1947f6b10f71ddb445c91d0c
> 

So, your change lifts wic to commit a4e7334a4b? This remains unclear from your patch but, given that your patch is accepted to stable now, it is the way to go for isar now.

Jan

[Adithya Balakumar

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/TYCPR01MB96693435BE9F69BB61054B0DC4A22%40TYCPR01MB9669.jpnprd01.prod.outlook.com.

RE: [PATCH v2 0/1] Set hash-seed for empty ext partition

[Adithya.Balakumar]

Yes Jan, My patch lifts wic to commit a4e7334a4b87f1bb1947f6b10f71ddb445c91d0c in the stable branch (scarthgap) of oe-core. Apologies for not making that clear in my patch.

Regards,
Adithya

-----Original Message-----
From: Jan Kiszka <jan.kiszka@siemens.com> 
Sent: Tuesday, July 16, 2024 2:30 PM
To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; isar-users@googlegroups.com; amikan@ilbers.de
Cc: hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>
Subject: Re: [PATCH v2 0/1] Set hash-seed for empty ext partition

On 04.07.24 13:10, Adithya Balakumar wrote:
> ext filesystems require a hash_seed to generate deterministic 
> directory indexes for reproducible builds. This is handled by wic in 
> the case where the rootfs plugin is used but this not handled in the 
> case when an empty ext4 partition is deployed.
> 
> This patch is also shared with openembedded-core and also accepted. 
> master branch commit: 0202fb594fb05098cb8d8b6088e63beb40b5906e
> scarthgap branch commit: a4e7334a4b87f1bb1947f6b10f71ddb445c91d0c
> 

So, your change lifts wic to commit a4e7334a4b? This remains unclear from your patch but, given that your patch is accepted to stable now, it is the way to go for isar now.

Jan

> Changes since v1:
> There is no change in the patch from v1. Just that this patch is also 
> applied to the scarthgap branch in OE-core.
> 
> Adithya Balakumar (1):
>   wic/partition.py: Set hash_seed for empty ext partition
> 
>  scripts/lib/wic/partition.py | 37 
> ++++++++++++++++++++++--------------
>  1 file changed, 23 insertions(+), 14 deletions(-)
> 

--
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/TYCPR01MB96695077078EE5A6D0D9E84AC4A22%40TYCPR01MB9669.jpnprd01.prod.outlook.com.

Re: [PATCH v2 0/1] Set hash-seed for empty ext partition

['Jan Kiszka' via isar-users]

On 16.07.24 11:49, Adithya.Balakumar@toshiba-tsip.com wrote:
> Yes Jan, My patch lifts wic to commit a4e7334a4b87f1bb1947f6b10f71ddb445c91d0c in the stable branch (scarthgap) of oe-core. Apologies for not making that clear in my patch.
> 

Then please update the commit message accordingly, see e.g.
07fb09747f60a99ffece335404f5096977af4f4e. The commit message can
additionally refer to the only change this brings, but most important is
recording the wic sha in isar's git.

Thanks,
Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/e3c8e8ac-a3b0-4ce4-a443-20329ed4c7c0%40siemens.com.