[PATCH 0/2] Fix reproducibility of /etc/shadow file when image built on different days

[PATCH 0/2] Fix reproducibility of /etc/shadow file when image built on different days

[Adithya Balakumar]

The third field in the /etc/shadow file (sp_lstchg) contains the date of
the last password change expressed as the number of days since Jan 1, 1970.
As this is a relative time, creating a user today will result in:

username:17238:0:99999:7:::
whilst creating the same user tomorrow will result in:

username:17239:0:99999:7:::

This impacts reproducibility of images when built on different days.

Since /etc/shadow honours SOURCE_DATE_EPOCH, this patchset makes the
SOURCE_DATE_EPOCH variable available when the /etc/shadow file is
created/modified.

Adithya Balakumar (2):
  isar-bootstrap.inc: Export SOURCE_DATE_EPOCH variable
  image-account-extension.bbclass: Export SOURCE_DATE_EPOCH variable

 meta/classes/image-account-extension.bbclass        | 5 +++++
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 4 ++++
 2 files changed, 9 insertions(+)

-- 
2.39.2

[PATCH 1/2] isar-bootstrap.inc: Export SOURCE_DATE_EPOCH variable

[Adithya Balakumar]

The third field in the /etc/shadow file (sp_lstchg) contains the date of
the last password change expressed as the number of days since Jan 1,
1970. The sp_lstchg value changes when the images are built on
different days making the image not reproducible.

Exporting the SOURCE_DATE_EPOCH variable ensures that sp_lstchg value is
set with respect to the SOURCE_DATE_EPOCH value while debootstrap creates
the minimal rootfile system.

Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 3477c2fb..89746e85 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -292,6 +292,10 @@ do_bootstrap() {
     sudo rm -rf --one-file-system "${ROOTFSDIR}"
     deb_dl_dir_import "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
 
+    if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
+        export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
+    fi
+
     sudo -E -s <<'EOSUDO'
         set -e
         if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
-- 
2.39.2

[PATCH 2/2] image-account-extension.bbclass: Export SOURCE_DATE_EPOCH variable

[Adithya Balakumar]

When a user account is created / updated, the third field (sp_lstchg)
in the /etc/shadow file could be modified. The third field contains
the date of the last password change expressed as the number of days
since Jan 1,1970. When the images are built on different days, the
value of sp_lstchg will be different making the image not reproducible.

Adding the SOURCE_DATE_EPOCH variable to environment ensures that
sp_lstchg value is set with respect to the SOURCE_DATE_EPOCH value.

Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
---
 meta/classes/image-account-extension.bbclass | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/classes/image-account-extension.bbclass b/meta/classes/image-account-extension.bbclass
index 9e67cb91..5fe67106 100644
--- a/meta/classes/image-account-extension.bbclass
+++ b/meta/classes/image-account-extension.bbclass
@@ -131,6 +131,11 @@ def image_create_users(d: "DataSmart") -> None:
 
 ROOTFS_POSTPROCESS_COMMAND += "image_postprocess_accounts"
 python image_postprocess_accounts() {
+    import os
+    if os.getenv("SOURCE_DATE_EPOCH") == None:
+         source_date_epoch = d.getVar("SOURCE_DATE_EPOCH")
+         os.environ["SOURCE_DATE_EPOCH"] = source_date_epoch
+ 
     image_create_groups(d)
     image_create_users(d)
 }
-- 
2.39.2

Re: [PATCH 0/2] Fix reproducibility of /etc/shadow file when image built on different days

[Jan Kiszka]

On 04.03.24 11:37, Adithya Balakumar wrote:
> The third field in the /etc/shadow file (sp_lstchg) contains the date of
> the last password change expressed as the number of days since Jan 1, 1970.
> As this is a relative time, creating a user today will result in:
> 
> username:17238:0:99999:7:::
> whilst creating the same user tomorrow will result in:
> 
> username:17239:0:99999:7:::
> 
> This impacts reproducibility of images when built on different days.
> 
> Since /etc/shadow honours SOURCE_DATE_EPOCH, this patchset makes the
> SOURCE_DATE_EPOCH variable available when the /etc/shadow file is
> created/modified.
> 

Yeah, this is what I learned the hard way these days as well while
trying to make the kas container images bit-identical reproducible.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 0/2] Fix reproducibility of /etc/shadow file when image built on different days

[MOESSBAUER, Felix]

On Mon, 2024-03-04 at 16:07 +0530, Adithya Balakumar wrote:
> The third field in the /etc/shadow file (sp_lstchg) contains the date
> of
> the last password change expressed as the number of days since Jan 1,
> 1970.
> As this is a relative time, creating a user today will result in:
> 
> username:17238:0:99999:7:::
> whilst creating the same user tomorrow will result in:
> 
> username:17239:0:99999:7:::
> 
> This impacts reproducibility of images when built on different days.
> 
> Since /etc/shadow honours SOURCE_DATE_EPOCH, this patchset makes the
> SOURCE_DATE_EPOCH variable available when the /etc/shadow file is
> created/modified.

Good catch! Thanks for finding and fixing.

Acked-by: Felix Moessbauer <felix.moessbauer@siemens.com>

Felix

> 
> Adithya Balakumar (2):
>   isar-bootstrap.inc: Export SOURCE_DATE_EPOCH variable
>   image-account-extension.bbclass: Export SOURCE_DATE_EPOCH variable
> 
>  meta/classes/image-account-extension.bbclass        | 5 +++++
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 4 ++++
>  2 files changed, 9 insertions(+)
> 
> -- 
> 2.39.2
> 
> 

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 1/2] isar-bootstrap.inc: Export SOURCE_DATE_EPOCH variable

[Gylstorff Quirin]

On 3/4/24 11:37 AM, Adithya Balakumar wrote:
> The third field in the /etc/shadow file (sp_lstchg) contains the date of
> the last password change expressed as the number of days since Jan 1,
> 1970. The sp_lstchg value changes when the images are built on
> different days making the image not reproducible.
> 
> Exporting the SOURCE_DATE_EPOCH variable ensures that sp_lstchg value is
> set with respect to the SOURCE_DATE_EPOCH value while debootstrap creates
> the minimal rootfile system.
> 
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
>   meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 4 ++++
>   1 file changed, 4 insertions(+)
> 
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index 3477c2fb..89746e85 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -292,6 +292,10 @@ do_bootstrap() {
>       sudo rm -rf --one-file-system "${ROOTFSDIR}"
>       deb_dl_dir_import "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
>   
> +    if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
' ! -z ' is the same as ' -n ' .

Quirin

> +        export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
> +    fi
> +
>       sudo -E -s <<'EOSUDO'
>           set -e
>           if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then

Re: [PATCH 2/2] image-account-extension.bbclass: Export SOURCE_DATE_EPOCH variable

[Uladzimir Bely]

On Mon, 2024-03-04 at 16:07 +0530, Adithya Balakumar wrote:

> > When a user account is created / updated, the third field
(sp_lstchg)
> > in the /etc/shadow file could be modified. The third field contains
> > the date of the last password change expressed as the number of
days
> > since Jan 1,1970. When the images are built on different days, the
> > value of sp_lstchg will be different making the image not  
> > reproducible.
> > 
> > Adding the SOURCE_DATE_EPOCH variable to environment ensures that
> > sp_lstchg value is set with respect to the SOURCE_DATE_EPOCH value.
> > 
> > Signed-off-by: Adithya Balakumar
<[Adithya.Balakumar@toshiba-tsip.com](mailto:Adithya.Balakumar@toshiba-tsip.com
)>
> > ---
> >  meta/classes/image-account-extension.bbclass | 5 +++++
> >  1 file changed, 5 insertions(+)
> > 
> > diff --git a/meta/classes/image-account-extension.bbclass  
> > b/meta/classes/image-account-extension.bbclass
> > index 9e67cb91..5fe67106 100644
> > --- a/meta/classes/image-account-extension.bbclass
> > +++ b/meta/classes/image-account-extension.bbclass
> > @@ -131,6 +131,11 @@ def image_create_users(d: "DataSmart") ->
None:
> >  
> >  ROOTFS_POSTPROCESS_COMMAND += "image_postprocess_accounts"
> >  python image_postprocess_accounts() {
> > +    import os
> > +    if os.getenv("SOURCE_DATE_EPOCH") == None:
> > +         source_date_epoch = d.getVar("SOURCE_DATE_EPOCH")
> > +         os.environ["SOURCE_DATE_EPOCH"] = source_date_epoch
> > +


If bitbake var is empty, this fails the following way:

ERROR: mc:qemuamd64-bullseye:isar-image-ci-1.0-r0 do_rootfs_postprocess: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:  
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>  
0001:  
*** 0002:image_postprocess_accounts(d)  
0003:  
File: '/workspace/build/isar_ub_devel_fast/552/meta/classes/image-account-extension.bbclass', lineno: 137, function: image_postprocess_accounts  
0133:python image_postprocess_accounts() {  
0134:    import os  
0135:    if os.getenv("SOURCE_DATE_EPOCH") == None:  
0136:         source_date_epoch = d.getVar("SOURCE_DATE_EPOCH")  
*** 0137:         os.environ["SOURCE_DATE_EPOCH"] = source_date_epoch  
0138:  
0139:    image_create_groups(d)  
0140:    image_create_users(d)  
0141:}  
File: '<frozen os>', lineno: 684, function: __setitem__  
File "<frozen os>", line 684, in __setitem__

File: '<frozen os>', lineno: 758, function: encode  
File "<frozen os>", line 758, in encode

Exception: TypeError: str expected, not NoneType


> >      image_create_groups(d)
> >      image_create_users(d)
> >  }
> > -- 
> > 2.39.2
> > 
> >

RE: [PATCH 1/2] isar-bootstrap.inc: Export SOURCE_DATE_EPOCH variable

[Adithya.Balakumar]

Thanks Quirin, Will update it in v2.

Thanks and Regards,
Adithya Balakumar

-----Original Message-----
From: Gylstorff Quirin <quirin.gylstorff@siemens.com> 
Sent: Monday, March 4, 2024 9:36 PM
To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; isar-users@googlegroups.com; amikan@ilbers.de
Cc: jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>
Subject: Re: [PATCH 1/2] isar-bootstrap.inc: Export SOURCE_DATE_EPOCH variable



On 3/4/24 11:37 AM, Adithya Balakumar wrote:
> The third field in the /etc/shadow file (sp_lstchg) contains the date 
> of the last password change expressed as the number of days since Jan 
> 1, 1970. The sp_lstchg value changes when the images are built on 
> different days making the image not reproducible.
> 
> Exporting the SOURCE_DATE_EPOCH variable ensures that sp_lstchg value 
> is set with respect to the SOURCE_DATE_EPOCH value while debootstrap 
> creates the minimal rootfile system.
> 
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
>   meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 4 ++++
>   1 file changed, 4 insertions(+)
> 
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc 
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index 3477c2fb..89746e85 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -292,6 +292,10 @@ do_bootstrap() {
>       sudo rm -rf --one-file-system "${ROOTFSDIR}"
>       deb_dl_dir_import "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
>   
> +    if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
' ! -z ' is the same as ' -n ' .

Quirin

> +        export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
> +    fi
> +
>       sudo -E -s <<'EOSUDO'
>           set -e
>           if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then

RE: [PATCH 2/2] image-account-extension.bbclass: Export SOURCE_DATE_EPOCH variable

[Adithya.Balakumar]

I Apologize. Will fix this is in v2.

Thanks and Regards,
Adithya Balakumar

-----Original Message-----
From: Uladzimir Bely <ubely@ilbers.de> 
Sent: Tuesday, March 5, 2024 9:49 AM
To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; isar-users@googlegroups.com
Subject: Re: [PATCH 2/2] image-account-extension.bbclass: Export SOURCE_DATE_EPOCH variable

On Mon, 2024-03-04 at 16:07 +0530, Adithya Balakumar wrote:

> > When a user account is created / updated, the third field
(sp_lstchg)
> > in the /etc/shadow file could be modified. The third field contains 
> > the date of the last password change expressed as the number of
days
> > since Jan 1,1970. When the images are built on different days, the 
> > value of sp_lstchg will be different making the image not 
> > reproducible.
> > 
> > Adding the SOURCE_DATE_EPOCH variable to environment ensures that 
> > sp_lstchg value is set with respect to the SOURCE_DATE_EPOCH value.
> > 
> > Signed-off-by: Adithya Balakumar
<[Adithya.Balakumar@toshiba-tsip.com](mailto:Adithya.Balakumar@toshiba-tsip.com
)>
> > ---
> >  meta/classes/image-account-extension.bbclass | 5 +++++
> >  1 file changed, 5 insertions(+)
> > 
> > diff --git a/meta/classes/image-account-extension.bbclass
> > b/meta/classes/image-account-extension.bbclass
> > index 9e67cb91..5fe67106 100644
> > --- a/meta/classes/image-account-extension.bbclass
> > +++ b/meta/classes/image-account-extension.bbclass
> > @@ -131,6 +131,11 @@ def image_create_users(d: "DataSmart") ->
None:
> >  
> >  ROOTFS_POSTPROCESS_COMMAND += "image_postprocess_accounts"
> >  python image_postprocess_accounts() {
> > +    import os
> > +    if os.getenv("SOURCE_DATE_EPOCH") == None:
> > +         source_date_epoch = d.getVar("SOURCE_DATE_EPOCH")
> > +         os.environ["SOURCE_DATE_EPOCH"] = source_date_epoch
> > +


If bitbake var is empty, this fails the following way:

ERROR: mc:qemuamd64-bullseye:isar-image-ci-1.0-r0 do_rootfs_postprocess: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:  
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:  
*** 0002:image_postprocess_accounts(d)
0003:  
File: '/workspace/build/isar_ub_devel_fast/552/meta/classes/image-account-extension.bbclass', lineno: 137, function: image_postprocess_accounts 0133:python image_postprocess_accounts() {  
0134:    import os  
0135:    if os.getenv("SOURCE_DATE_EPOCH") == None:  
0136:         source_date_epoch = d.getVar("SOURCE_DATE_EPOCH")  
*** 0137:         os.environ["SOURCE_DATE_EPOCH"] = source_date_epoch  
0138:  
0139:    image_create_groups(d)  
0140:    image_create_users(d)  
0141:}
File: '<frozen os>', lineno: 684, function: __setitem__ File "<frozen os>", line 684, in __setitem__

File: '<frozen os>', lineno: 758, function: encode File "<frozen os>", line 758, in encode

Exception: TypeError: str expected, not NoneType


> >      image_create_groups(d)
> >      image_create_users(d)
> >  }
> > --
> > 2.39.2
> > 
> >