[PATCH 0/5] Make ext4 images reproducible

[PATCH 0/5] Make ext4 images reproducible

[venkata.pyla]

From: venkata pyla <venkata.pyla@toshiba-tsip.com>

Hi,

The following series of patches will help to fix reproducible build
issues in ext4 file systems.

Verified with following steps:

```

$ git clone http://github.com/ilbers/isar.git
$ cd isar
$ . isar-init-build-env ../build

configure image parameters
$ cd isar
$ cat <<EOT >> conf/local.conf
SOURCE_DATE_EPOCH = "$(git log -1 --pretty=%ct)"
IMAGE_FSTYPES = "wic"
WIC_DEPLOY_PARTITIONS = "1"
EOT

# First Build
$ cd ../build
$ sudo rm -rf tmp sstate-cache
$ bitbake mc:qemuamd64-bookworm:isar-image-base
$ mv tmp/deploy/images/qemuamd64/ ../qemuamd64-1

# Second Build
$ sudo rm -rf tmp sstate-cache
$ bitbake mc:qemuamd64-bookworm:isar-image-base
$ mv tmp/deploy/images/qemuamd64/ ../qemuamd64-2

# compare ext4 images with diffoscope
$ diffoscope qemuamd64-1/isar-image-base-debian-bookworm-qemuamd64.wic.p2 \
	qemuamd64-2/isar-image-base-debian-bookworm-qemuamd64.wic.p2

```

venkata pyla (5):
  linux-module: Use debug-prefix-map to build modules deterministically
  wic: Set file system uuid to ext4 partitions
  wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible
    ext4 images
  wic: honor the SOURCE_DATE_EPOCH in case of updated fstab
  wic: make ext2/3/4 images reproducible

 .../lib/wic/canned-wks/common-isar.wks.inc    |  2 +-
 .../lib/wic/canned-wks/de0-nano-soc.wks.in    |  2 +-
 .../lib/wic/canned-wks/efi-plus-pcbios.wks    |  2 +-
 .../scripts/lib/wic/canned-wks/hikey.wks      |  2 +-
 .../lib/wic/canned-wks/multipart-efi.wks      |  4 +-
 .../lib/wic/canned-wks/nanopi-neo-efi.wks.in  |  2 +-
 .../lib/wic/canned-wks/nanopi-neo.wks.in      |  2 +-
 .../scripts/lib/wic/canned-wks/rpi-sdimg.wks  |  2 +-
 .../wic/canned-wks/sdimage-efi-sb-debian.wks  |  2 +-
 .../lib/wic/canned-wks/sdimage-efi-sd.wks     |  2 +-
 .../lib/wic/canned-wks/sdimage-efi.wks        |  2 +-
 .../lib/wic/canned-wks/sifive-fu540.wks       |  2 +-
 .../canned-wks/starfive-visionfive2.wks.in    |  2 +-
 .../lib/wic/canned-wks/stm32mp15x.wks.in      |  2 +-
 .../linux-module/files/debian/rules.tmpl      |  5 ++-
 scripts/lib/wic/partition.py                  | 37 ++++++++++++++++++-
 scripts/lib/wic/plugins/imager/direct.py      |  3 ++
 scripts/lib/wic/plugins/source/rootfs.py      |  2 +-
 18 files changed, 59 insertions(+), 18 deletions(-)

-- 
2.20.1

[PATCH 1/5] linux-module: Use debug-prefix-map to build modules deterministically

[venkata.pyla]

From: venkata pyla <venkata.pyla@toshiba-tsip.com>

The custom linux modules are built non-determinstically due to the
reason that the kernel build includes absolute paths in the debug
information which is not necessary and can be trimmed using
`debug-prefix-map` option.

Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 meta/recipes-kernel/linux-module/files/debian/rules.tmpl | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-kernel/linux-module/files/debian/rules.tmpl b/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
index 7d950e38..20d5f670 100755
--- a/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
+++ b/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
@@ -45,11 +45,14 @@ KERNEL_DEP := $(shell dpkg-query -W -f '$${Depends}' ${KERNEL_HEADERS_PKG} | sed
 KDIR := $(shell dpkg -L $(KERNEL_DEP) | grep "/lib/modules/.*/build")
 endif
 
+KCFLAGS := "-fdebug-prefix-map=$(PWD)=."
+KAFLAGS := "-fdebug-prefix-map=$(PWD)=."
+
 override_dh_auto_clean:
 	$(MAKE) -C $(KDIR) M=$(PWD) clean
 
 override_dh_auto_build:
-	$(MAKE) -C $(KDIR) M=$(PWD) $(PARALLEL_MAKE) modules
+	$(MAKE) -C $(KDIR) M=$(PWD) $(PARALLEL_MAKE) KCFLAGS="$(KCFLAGS)" KAFLAGS="$(KAFLAGS)" modules
 ifneq ($(filter pkg.sign,$(DEB_BUILD_PROFILES)),)
 	find . -name "*.ko" -print -exec $(KDIR)/scripts/sign-file ${SIGNATURE_HASHFN} ${SIGNATURE_KEYFILE} ${SIGNATURE_CERTFILE} {} \;
 endif
-- 
2.20.1

[PATCH 2/5] wic: Set file system uuid to ext4 partitions

[venkata.pyla]

From: venkata pyla <venkata.pyla@toshiba-tsip.com>

Adding pre configured UUID's to the file systems will helps to build
reproducible images.

Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc      | 2 +-
 meta-isar/scripts/lib/wic/canned-wks/de0-nano-soc.wks.in      | 2 +-
 meta-isar/scripts/lib/wic/canned-wks/efi-plus-pcbios.wks      | 2 +-
 meta-isar/scripts/lib/wic/canned-wks/hikey.wks                | 2 +-
 meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks        | 4 ++--
 meta-isar/scripts/lib/wic/canned-wks/nanopi-neo-efi.wks.in    | 2 +-
 meta-isar/scripts/lib/wic/canned-wks/nanopi-neo.wks.in        | 2 +-
 meta-isar/scripts/lib/wic/canned-wks/rpi-sdimg.wks            | 2 +-
 .../scripts/lib/wic/canned-wks/sdimage-efi-sb-debian.wks      | 2 +-
 meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks       | 2 +-
 meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks          | 2 +-
 meta-isar/scripts/lib/wic/canned-wks/sifive-fu540.wks         | 2 +-
 .../scripts/lib/wic/canned-wks/starfive-visionfive2.wks.in    | 2 +-
 meta-isar/scripts/lib/wic/canned-wks/stm32mp15x.wks.in        | 2 +-
 14 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc b/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc
index fe028b50..71885cb4 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc
+++ b/meta-isar/scripts/lib/wic/canned-wks/common-isar.wks.inc
@@ -1,3 +1,3 @@
 # This file is included into 3 canned wks files from this directory
 part /boot --source bootimg-pcbios-isar --ondisk sda --label boot --active --align 1024
-part / --source rootfs --use-uuid --fstype=ext4 --label platform --align 1024 --exclude-path=boot --mkfs-extraopts "-T default"
+part / --source rootfs --use-uuid --fstype=ext4 --label platform --align 1024 --exclude-path=boot --mkfs-extraopts "-T default" --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002
diff --git a/meta-isar/scripts/lib/wic/canned-wks/de0-nano-soc.wks.in b/meta-isar/scripts/lib/wic/canned-wks/de0-nano-soc.wks.in
index 0794a2f3..06154e01 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/de0-nano-soc.wks.in
+++ b/meta-isar/scripts/lib/wic/canned-wks/de0-nano-soc.wks.in
@@ -5,6 +5,6 @@
 
 part --source rawcopy --sourceparams "file=/usr/lib/u-boot/${MACHINE}/${U_BOOT_BIN}" --system-id 0xa2 --align 1
 
-part / --source rootfs-u-boot --ondisk mmcblk0 --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --active
+part / --source rootfs-u-boot --ondisk mmcblk0 --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --active --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002
 
 bootloader --append "rw rootwait"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/efi-plus-pcbios.wks b/meta-isar/scripts/lib/wic/canned-wks/efi-plus-pcbios.wks
index 03928915..321be10b 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/efi-plus-pcbios.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/efi-plus-pcbios.wks
@@ -1,6 +1,6 @@
 # Example to show how to create an efi + pcbios image
 # Note, that the loader argument is mandatory. But systemd-boot also works.
 part /boot --source bootimg-biosplusefi-isar --sourceparams="loader=grub-efi" --label boot --active --align 1024
-part / --source rootfs --ondisk sda --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --exclude-path=boot
+part / --source rootfs --ondisk sda --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --exclude-path=boot --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002
 
 bootloader --ptable gpt --timeout 3 --append "rootwait console=ttyS0,115200 console=tty0"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks
index 0c966e65..27447e2d 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/hikey.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/hikey.wks
@@ -5,6 +5,6 @@
 
 part /boot --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk mmcblk1 --label efi --part-type EF00 --align 1024
 
-part / --source rootfs --ondisk mmcblk1 --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/
+part / --source rootfs --ondisk mmcblk1 --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/ --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002
 
 bootloader --ptable gpt --timeout=3
diff --git a/meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks b/meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks
index b3abb3d0..ac8ebea9 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/multipart-efi.wks
@@ -1,8 +1,8 @@
 # Example showing how to split a single rootfs across partitions
 part /boot --ondisk sda --source bootimg-efi-isar --sourceparams="loader=grub-efi" --label boot --active --align 1024
 # when excluding paths, just specify --exclude-path once and separate paths using spaces
-part /     --ondisk sda --source rootfs --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --exclude-path boot home --extra-space 1G
+part /     --ondisk sda --source rootfs --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --exclude-path boot home --extra-space 1G --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002
 # put home last to support auto-expand of this partition
-part /home --ondisk sda --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/home --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024
+part /home --ondisk sda --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/home --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120004
 
 bootloader --ptable gpt --timeout 3 --append "rootwait console=ttyS0,115200 console=tty0"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo-efi.wks.in b/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo-efi.wks.in
index 7171a23e..4baf0642 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo-efi.wks.in
+++ b/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo-efi.wks.in
@@ -10,6 +10,6 @@
 part u-boot --source rawcopy --sourceparams "file=/usr/lib/u-boot/nanopi_neo/u-boot-sunxi-with-spl.bin" --no-table --align 128
 
 part /boot --use-uuid --source bootimg-efi-isar --sourceparams="loader=systemd-boot" --label efi --part-type EF00 --align 1024
-part /     --use-uuid --source rootfs --fstype=ext4 --mkfs-extraopts "-T default" --label platform --active --align 1024 --exclude-path boot
+part /     --use-uuid --source rootfs --fstype=ext4 --mkfs-extraopts "-T default" --label platform --active --align 1024 --exclude-path boot --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002
 
 bootloader --ptable gpt --timeout 0 --append "rootwait"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo.wks.in b/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo.wks.in
index af5b6f08..cfad430e 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo.wks.in
+++ b/meta-isar/scripts/lib/wic/canned-wks/nanopi-neo.wks.in
@@ -5,6 +5,6 @@
 
 part u-boot --source rawcopy --sourceparams "file=/usr/lib/u-boot/nanopi_neo/u-boot-sunxi-with-spl.bin" --no-table --align 8
 
-part / --source rootfs-u-boot --ondisk mmcblk2 --fstype ext4 --mkfs-extraopts "-T default" --sourceparams "builtin_dt=yes" --label platform --align 1024 --active
+part / --source rootfs-u-boot --ondisk mmcblk2 --fstype ext4 --mkfs-extraopts "-T default" --sourceparams "builtin_dt=yes" --label platform --align 1024 --active --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002
 
 bootloader --append "rw rootwait"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/rpi-sdimg.wks b/meta-isar/scripts/lib/wic/canned-wks/rpi-sdimg.wks
index e5a3668b..de1b742e 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/rpi-sdimg.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/rpi-sdimg.wks
@@ -1,5 +1,5 @@
 part /boot --source bootimg-partition --ondisk mmcblk0 --fstype=vfat --label boot --active --align 4096 --size 10
 
-part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --mkfs-extraopts "-T default" --label root --align 4096 --exclude-path=boot
+part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --mkfs-extraopts "-T default" --label root --align 4096 --exclude-path=boot --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002
 
 bootloader
diff --git a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sb-debian.wks b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sb-debian.wks
index cd99ebca..bb2da5ac 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sb-debian.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sb-debian.wks
@@ -5,6 +5,6 @@
 
 part /boot --source bootimg-efi-isar --sourceparams "loader=grub-efi,use-debian-sb-stub=true" --ondisk sda --label efi --part-type EF00 --align 1024
 
-part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/
+part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/ --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002
 
 bootloader --ptable gpt --timeout 3 --append "rootwait console=ttyS0,115200 console=tty0"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks
index 7fe2953f..6dc9c6c4 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-sd.wks
@@ -4,7 +4,7 @@
 
 part /boot --source bootimg-efi-isar --sourceparams "loader=systemd-boot" --ondisk sda --label efi --part-type EF00 --align 1024
 
-part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/
+part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/ --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002
 
 include expand-padding.wks.inc
 
diff --git a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
index 80c3a535..8c864f2f 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/sdimage-efi.wks
@@ -4,7 +4,7 @@
 
 part /boot --source bootimg-efi-isar --sourceparams "loader=grub-efi" --ondisk sda --label efi --part-type EF00 --align 1024
 
-part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/
+part / --source rootfs --ondisk sda --fstype ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid --exclude-path boot/ --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002
 
 include expand-padding.wks.inc
 
diff --git a/meta-isar/scripts/lib/wic/canned-wks/sifive-fu540.wks b/meta-isar/scripts/lib/wic/canned-wks/sifive-fu540.wks
index 9cd0e662..ef99edd7 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/sifive-fu540.wks
+++ b/meta-isar/scripts/lib/wic/canned-wks/sifive-fu540.wks
@@ -5,6 +5,6 @@
 
 part --source rawcopy --sourceparams "file=/usr/lib/opensbi/sifive-fu540/fw_payload.bin" --ondisk mmcblk0 --fixed-size 8M --align 1 --part-type 2e54b353-1271-4842-806f-e436d6af6985
 
-part / --source rootfs-u-boot --ondisk mmcblk0 --fstype ext4 --mkfs-extraopts "-T default" --label platform --active --align 1024 --sourceparams "no_initrd=yes,script_prepend=setenv fdtfile sifive/hifive-unleashed-a00.dtb"
+part / --source rootfs-u-boot --ondisk mmcblk0 --fstype ext4 --mkfs-extraopts "-T default" --label platform --active --align 1024 --sourceparams "no_initrd=yes,script_prepend=setenv fdtfile sifive/hifive-unleashed-a00.dtb" --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002
 
 bootloader --ptable gpt --append "rootwait"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/starfive-visionfive2.wks.in b/meta-isar/scripts/lib/wic/canned-wks/starfive-visionfive2.wks.in
index 0f3d76a9..c265c752 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/starfive-visionfive2.wks.in
+++ b/meta-isar/scripts/lib/wic/canned-wks/starfive-visionfive2.wks.in
@@ -18,6 +18,6 @@ part --source rawcopy --sourceparams "file=/usr/share/visionfive2-u-boot-firmwar
 # EFI
 part /boot --source bootimg-efi-isar --sourceparams "loader=systemd-boot" --use-uuid --label efi --part-type EF00 --align 1024
 # rootfs
-part / --source rootfs --use-uuid --fstype ext4 --mkfs-extraopts "-T default" --label image --active --align 1024 --exclude-path=boot/
+part / --source rootfs --use-uuid --fstype ext4 --mkfs-extraopts "-T default" --label image --active --align 1024 --exclude-path=boot/ --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002
 
 bootloader --ptable gpt --timeout=0 --append "console=tty0 console=ttyS0,115200 earlycon rootwait"
diff --git a/meta-isar/scripts/lib/wic/canned-wks/stm32mp15x.wks.in b/meta-isar/scripts/lib/wic/canned-wks/stm32mp15x.wks.in
index 5d96f65f..6e651292 100644
--- a/meta-isar/scripts/lib/wic/canned-wks/stm32mp15x.wks.in
+++ b/meta-isar/scripts/lib/wic/canned-wks/stm32mp15x.wks.in
@@ -11,6 +11,6 @@ part teeh --part-name teeh --source rawcopy --sourceparams "file=/usr/lib/optee-
 part teed --part-name teed --source rawcopy --sourceparams "file=/usr/lib/optee-os/${MACHINE}/tee-pageable_v2.stm32" --fstype=ext4 --fsoptions "noauto" --part-type 0x8301 --fixed-size 1024K
 part teex --part-name teex --source rawcopy --sourceparams "file=/usr/lib/optee-os/${MACHINE}/tee-pager_v2.stm32" --fstype=ext4 --fsoptions "noauto" --part-type 0x8301 --fixed-size 256K
 
-part / --source rootfs-u-boot --fstype ext4 --mkfs-extraopts "-T default" --label root --align 1024 --active --use-uuid
+part / --source rootfs-u-boot --fstype ext4 --mkfs-extraopts "-T default" --label root --align 1024 --active --use-uuid --fsuuid 1f55d66a-40d8-11ee-be56-0242ac120002
 
 bootloader --ptable gpt --append="rootwait rw console=ttySTM0,115200"
-- 
2.20.1

[PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[venkata.pyla]

From: venkata pyla <venkata.pyla@toshiba-tsip.com>

E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file system.
hash_seed: creates reproducible directory indexes in the file system.

Reference commit in e2fsprogs: e1f7100643a46456be107b33098f6034b0835e6d

Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 scripts/lib/wic/partition.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index e50871b8..90b2c037 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -280,6 +280,17 @@ class Partition():
 
         extraopts = self.mkfs_extraopts or "-F -i 8192"
 
+        if os.getenv('SOURCE_DATE_EPOCH'):
+            sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
+            pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s" % (sde_time, pseudo)
+
+            # Set hash_seed to generate deterministic directory indexes
+            namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-2f2fdf33d460")
+            if self.fsuuid:
+                namespace = uuid.UUID(self.fsuuid)
+            hash_seed = str(uuid.uuid5(namespace, str(sde_time)))
+            extraopts += " -E hash_seed=%s" % hash_seed
+
         label_str = ""
         if self.label:
             label_str = "-L %s" % self.label
-- 
2.20.1

[PATCH 4/5] wic: honor the SOURCE_DATE_EPOCH in case of updated fstab

[venkata.pyla]

From: venkata pyla <venkata.pyla@toshiba-tsip.com>

In case user requested to build a binary repeatable package,
it's required to honor the SOURCE_DATE_EPOCH environment
variable. So forcefully set mtime inside all the routines
which modify fstab in case it is updated.

Taken from openembedded-core: 99719a3712a88dce8450994d995803e126e49115

Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 scripts/lib/wic/partition.py             | 7 ++++++-
 scripts/lib/wic/plugins/imager/direct.py | 3 +++
 scripts/lib/wic/plugins/source/rootfs.py | 2 +-
 3 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 90b2c037..44b1277e 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -305,6 +305,11 @@ class Partition():
                 f.write("cd etc\n")
                 f.write("rm fstab\n")
                 f.write("write %s fstab\n" % (self.updated_fstab_path))
+                if os.getenv('SOURCE_DATE_EPOCH'):
+                    fstab_time = int(os.getenv('SOURCE_DATE_EPOCH'))
+                    for time in ["atime", "mtime", "ctime"]:
+                        f.write("set_inode_field fstab %s %s\n" % (time, hex(fstab_time)))
+                        f.write("set_inode_field fstab %s_extra 0\n" % (time))
             debugfs_cmd = "debugfs -w -f %s %s" % (debugfs_script_path, rootfs)
             exec_native_cmd(debugfs_cmd, native_sysroot)
 
@@ -364,7 +369,7 @@ class Partition():
         exec_native_cmd(mcopy_cmd, native_sysroot)
 
         if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update:
-            mcopy_cmd = "mcopy -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path)
+            mcopy_cmd = "mcopy -m -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path)
             exec_native_cmd(mcopy_cmd, native_sysroot)
 
         chmod_cmd = "chmod 644 %s" % rootfs
diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index c44159b2..4d8f9216 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -149,6 +149,9 @@ class DirectPlugin(ImagerPlugin):
             self.updated_fstab_path = os.path.join(self.workdir, "fstab")
             with open(self.updated_fstab_path, "w") as f:
                 f.writelines(fstab_lines)
+            if os.getenv('SOURCE_DATE_EPOCH'):
+                fstab_time = int(os.getenv('SOURCE_DATE_EPOCH'))
+                os.utime(self.updated_fstab_path, (fstab_time, fstab_time))
 
     def _full_path(self, path, name, extention):
         """ Construct full file path to a file we generate. """
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index fc06312e..e29f3a4c 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -224,7 +224,7 @@ class RootfsPlugin(SourcePlugin):
             if part.update_fstab_in_rootfs and part.has_fstab and not part.no_fstab_update:
                 fstab_path = os.path.join(new_rootfs, "etc/fstab")
                 # Assume that fstab should always be owned by root with fixed permissions
-                install_cmd = "install -m 0644 %s %s" % (part.updated_fstab_path, fstab_path)
+                install_cmd = "install -m 0644 -p %s %s" % (part.updated_fstab_path, fstab_path)
                 if new_pseudo:
                     pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo)
                 else:
-- 
2.20.1

[PATCH 5/5] wic: make ext2/3/4 images reproducible

[venkata.pyla]

From: venkata pyla <venkata.pyla@toshiba-tsip.com>

Ext2/3/4 FS contains not only mtime, but also ctime, atime and crtime.
Currently, all the files are being added into the rootfs image using
mkfs -d functionality which affects all the timestamps excluding mtime.
This patch ensures these timestamps inside the FS image equal to
the SOURCE_DATE_EPOCH if it is set.

taken from openembedded-core: 75d2dd0ea7790db2e8ee921784ca373abff2df65

Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 scripts/lib/wic/partition.py | 29 ++++++++++++++++++++++++-----
 1 file changed, 24 insertions(+), 5 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 44b1277e..e7641c9e 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -305,17 +305,36 @@ class Partition():
                 f.write("cd etc\n")
                 f.write("rm fstab\n")
                 f.write("write %s fstab\n" % (self.updated_fstab_path))
-                if os.getenv('SOURCE_DATE_EPOCH'):
-                    fstab_time = int(os.getenv('SOURCE_DATE_EPOCH'))
-                    for time in ["atime", "mtime", "ctime"]:
-                        f.write("set_inode_field fstab %s %s\n" % (time, hex(fstab_time)))
-                        f.write("set_inode_field fstab %s_extra 0\n" % (time))
             debugfs_cmd = "debugfs -w -f %s %s" % (debugfs_script_path, rootfs)
             exec_native_cmd(debugfs_cmd, native_sysroot)
 
         mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs)
         exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo)
 
+        if os.getenv('SOURCE_DATE_EPOCH'):
+            sde_time = hex(int(os.getenv('SOURCE_DATE_EPOCH')))
+            debugfs_script_path = os.path.join(cr_workdir, "debugfs_script")
+            files = []
+            for root, dirs, others in os.walk(rootfs_dir):
+                base = root.replace(rootfs_dir, "").rstrip(os.sep)
+                files += [ "/" if base == "" else base ]
+                files += [ base + "/" + n for n in dirs + others ]
+            with open(debugfs_script_path, "w") as f:
+                f.write("set_current_time %s\n" % (sde_time))
+                if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update:
+                    f.write("set_inode_field /etc/fstab mtime %s\n" % (sde_time))
+                    f.write("set_inode_field /etc/fstab mtime_extra 0\n")
+                for file in set(files):
+                    for time in ["atime", "ctime", "crtime"]:
+                        f.write("set_inode_field \"%s\" %s %s\n" % (file, time, sde_time))
+                        f.write("set_inode_field \"%s\" %s_extra 0\n" % (file, time))
+                for time in ["wtime", "mkfs_time", "lastcheck"]:
+                    f.write("set_super_value %s %s\n" % (time, sde_time))
+                for time in ["mtime", "first_error_time", "last_error_time", "kbytes_written"]:
+                    f.write("set_super_value %s 0\n" % (time))
+            debugfs_cmd = "debugfs -w -f %s %s" % (debugfs_script_path, rootfs)
+            exec_native_cmd(debugfs_cmd, native_sysroot)
+
         self.check_for_Y2038_problem(rootfs, native_sysroot)
 
     def prepare_rootfs_btrfs(self, rootfs, cr_workdir, oe_builddir, rootfs_dir,
-- 
2.20.1

Re: [PATCH 1/5] linux-module: Use debug-prefix-map to build modules deterministically

[MOESSBAUER, Felix]

On Thu, 2023-12-07 at 21:11 +0530, venkata.pyla@toshiba-tsip.com wrote:
> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> 
> The custom linux modules are built non-determinstically due to the
> reason that the kernel build includes absolute paths in the debug
> information which is not necessary and can be trimmed using
> `debug-prefix-map` option.
> 
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>

Hi, 

thanks for this. It is basically the same patch I send a year ago for
the kernel recipe.

@Maintainers: This patch needs to be serialized with this one:
"add support for non-default modules dir in linux-module".

Acked-by: Felix Moessbauer <felix.moessbauer@siemens.com>

Best regards,
Felix

> ---
>  meta/recipes-kernel/linux-module/files/debian/rules.tmpl | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
> b/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
> index 7d950e38..20d5f670 100755
> --- a/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
> +++ b/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
> @@ -45,11 +45,14 @@ KERNEL_DEP := $(shell dpkg-query -W -f
> '$${Depends}' ${KERNEL_HEADERS_PKG} | sed
>  KDIR := $(shell dpkg -L $(KERNEL_DEP) | grep
> "/lib/modules/.*/build")
>  endif
>  
> +KCFLAGS := "-fdebug-prefix-map=$(PWD)=."
> +KAFLAGS := "-fdebug-prefix-map=$(PWD)=."
> +
>  override_dh_auto_clean:
>         $(MAKE) -C $(KDIR) M=$(PWD) clean
>  
>  override_dh_auto_build:
> -       $(MAKE) -C $(KDIR) M=$(PWD) $(PARALLEL_MAKE) modules
> +       $(MAKE) -C $(KDIR) M=$(PWD) $(PARALLEL_MAKE)
> KCFLAGS="$(KCFLAGS)" KAFLAGS="$(KAFLAGS)" modules
>  ifneq ($(filter pkg.sign,$(DEB_BUILD_PROFILES)),)
>         find . -name "*.ko" -print -exec $(KDIR)/scripts/sign-file
> ${SIGNATURE_HASHFN} ${SIGNATURE_KEYFILE} ${SIGNATURE_CERTFILE} {} \;
>  endif

Re: [PATCH 0/5] Make ext4 images reproducible

[MOESSBAUER, Felix]

On Thu, 2023-12-07 at 21:11 +0530, venkata.pyla@toshiba-tsip.com wrote:
> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> 
> Hi,
> 
> The following series of patches will help to fix reproducible build
> issues in ext4 file systems.
> 
> Verified with following steps:
> 
> ```
> 
> $ git clone http://github.com/ilbers/isar.git
> $ cd isar
> $ . isar-init-build-env ../build
> 
> configure image parameters
> $ cd isar
> $ cat <<EOT >> conf/local.conf
> SOURCE_DATE_EPOCH = "$(git log -1 --pretty=%ct)"
> IMAGE_FSTYPES = "wic"
> WIC_DEPLOY_PARTITIONS = "1"
> EOT
> 
> # First Build
> $ cd ../build
> $ sudo rm -rf tmp sstate-cache
> $ bitbake mc:qemuamd64-bookworm:isar-image-base
> $ mv tmp/deploy/images/qemuamd64/ ../qemuamd64-1
> 
> # Second Build
> $ sudo rm -rf tmp sstate-cache

Hi,

just out of curiosity: Did you also have a look a building with sstate
cache? Are there reproducibility issues?

In the past I fixed a couple of these, but I'm not 100% sure if we
still have some (esp. around the rootfs caching).

Happy Coding!
Felix

> $ bitbake mc:qemuamd64-bookworm:isar-image-base
> $ mv tmp/deploy/images/qemuamd64/ ../qemuamd64-2
> 
> # compare ext4 images with diffoscope
> $ diffoscope qemuamd64-1/isar-image-base-debian-bookworm-
> qemuamd64.wic.p2 \
>         qemuamd64-2/isar-image-base-debian-bookworm-qemuamd64.wic.p2
> 
> ```
> 
> venkata pyla (5):
>   linux-module: Use debug-prefix-map to build modules
> deterministically
>   wic: Set file system uuid to ext4 partitions
>   wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible
>     ext4 images
>   wic: honor the SOURCE_DATE_EPOCH in case of updated fstab
>   wic: make ext2/3/4 images reproducible
> 
>  .../lib/wic/canned-wks/common-isar.wks.inc    |  2 +-
>  .../lib/wic/canned-wks/de0-nano-soc.wks.in    |  2 +-
>  .../lib/wic/canned-wks/efi-plus-pcbios.wks    |  2 +-
>  .../scripts/lib/wic/canned-wks/hikey.wks      |  2 +-
>  .../lib/wic/canned-wks/multipart-efi.wks      |  4 +-
>  .../lib/wic/canned-wks/nanopi-neo-efi.wks.in  |  2 +-
>  .../lib/wic/canned-wks/nanopi-neo.wks.in      |  2 +-
>  .../scripts/lib/wic/canned-wks/rpi-sdimg.wks  |  2 +-
>  .../wic/canned-wks/sdimage-efi-sb-debian.wks  |  2 +-
>  .../lib/wic/canned-wks/sdimage-efi-sd.wks     |  2 +-
>  .../lib/wic/canned-wks/sdimage-efi.wks        |  2 +-
>  .../lib/wic/canned-wks/sifive-fu540.wks       |  2 +-
>  .../canned-wks/starfive-visionfive2.wks.in    |  2 +-
>  .../lib/wic/canned-wks/stm32mp15x.wks.in      |  2 +-
>  .../linux-module/files/debian/rules.tmpl      |  5 ++-
>  scripts/lib/wic/partition.py                  | 37
> ++++++++++++++++++-
>  scripts/lib/wic/plugins/imager/direct.py      |  3 ++
>  scripts/lib/wic/plugins/source/rootfs.py      |  2 +-
>  18 files changed, 59 insertions(+), 18 deletions(-)
> 

Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[MOESSBAUER, Felix]

On Thu, 2023-12-07 at 21:11 +0530, venkata.pyla@toshiba-tsip.com wrote:
> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> 
> E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file
> system.
> hash_seed: creates reproducible directory indexes in the file system.
> 
> Reference commit in e2fsprogs:
> e1f7100643a46456be107b33098f6034b0835e6d

Hi!

Did you also send this patch to OE?
We need to integrate it there as well, as this is the basis for the
ISAR plugins.

Best regards,
Felix

> 
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  scripts/lib/wic/partition.py | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/scripts/lib/wic/partition.py
> b/scripts/lib/wic/partition.py
> index e50871b8..90b2c037 100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -280,6 +280,17 @@ class Partition():
>  
>          extraopts = self.mkfs_extraopts or "-F -i 8192"
>  
> +        if os.getenv('SOURCE_DATE_EPOCH'):
> +            sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
> +            pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s" % (sde_time,
> pseudo)
> +
> +            # Set hash_seed to generate deterministic directory
> indexes
> +            namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-
> 2f2fdf33d460")
> +            if self.fsuuid:
> +                namespace = uuid.UUID(self.fsuuid)
> +            hash_seed = str(uuid.uuid5(namespace, str(sde_time)))
> +            extraopts += " -E hash_seed=%s" % hash_seed
> +
>          label_str = ""
>          if self.label:
>              label_str = "-L %s" % self.label

RE: [PATCH 0/5] Make ext4 images reproducible

[Venkata.Pyla]

> -----Original Message-----
> From: 'MOESSBAUER, Felix' via isar-users <isar-users@googlegroups.com>
> Sent: Friday, December 8, 2023 4:00 PM
> To: isar-users@googlegroups.com; pyla venkata(ＴＳＩＰ TMIEC ODG
> Porting) <Venkata.Pyla@toshiba-tsip.com>
> Cc: Kiszka, Jan <jan.kiszka@siemens.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh
> kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>
> Subject: Re: [PATCH 0/5] Make ext4 images reproducible
> 
> On Thu, 2023-12-07 at 21:11 +0530, venkata.pyla@toshiba-tsip.com wrote:
> > From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> >
> > Hi,
> >
> > The following series of patches will help to fix reproducible build
> > issues in ext4 file systems.
> >
> > Verified with following steps:
> >
> > ```
> >
> > $ git clone http://github.com/ilbers/isar.git $ cd isar $ .
> > isar-init-build-env ../build
> >
> > configure image parameters
> > $ cd isar
> > $ cat <<EOT >> conf/local.conf
> > SOURCE_DATE_EPOCH = "$(git log -1 --pretty=%ct)"
> > IMAGE_FSTYPES = "wic"
> > WIC_DEPLOY_PARTITIONS = "1"
> > EOT
> >
> > # First Build
> > $ cd ../build
> > $ sudo rm -rf tmp sstate-cache
> > $ bitbake mc:qemuamd64-bookworm:isar-image-base
> > $ mv tmp/deploy/images/qemuamd64/ ../qemuamd64-1
> >
> > # Second Build
> > $ sudo rm -rf tmp sstate-cache
> 
> Hi,
> 
> just out of curiosity: Did you also have a look a building with sstate cache? Are
> there reproducibility issues?

I just checked, with sstate cache the images are reproducible as well.
In my steps I have cleaned the sstate cache, because it should not hide any issues due to caching.

> 
> In the past I fixed a couple of these, but I'm not 100% sure if we still have
> some (esp. around the rootfs caching).
> 
> Happy Coding!
> Felix
> 
> > $ bitbake mc:qemuamd64-bookworm:isar-image-base
> > $ mv tmp/deploy/images/qemuamd64/ ../qemuamd64-2
> >
> > # compare ext4 images with diffoscope
> > $ diffoscope qemuamd64-1/isar-image-base-debian-bookworm-
> > qemuamd64.wic.p2 \
> >         qemuamd64-2/isar-image-base-debian-bookworm-qemuamd64.wic.p2
> >
> > ```
> >
> > venkata pyla (5):
> >   linux-module: Use debug-prefix-map to build modules
> > deterministically
> >   wic: Set file system uuid to ext4 partitions
> >   wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible
> >     ext4 images
> >   wic: honor the SOURCE_DATE_EPOCH in case of updated fstab
> >   wic: make ext2/3/4 images reproducible
> >
> >  .../lib/wic/canned-wks/common-isar.wks.inc    |  2 +-
> >  .../lib/wic/canned-wks/de0-nano-soc.wks.in    |  2 +-
> >  .../lib/wic/canned-wks/efi-plus-pcbios.wks    |  2 +-
> >  .../scripts/lib/wic/canned-wks/hikey.wks      |  2 +-
> >  .../lib/wic/canned-wks/multipart-efi.wks      |  4 +-
> >  .../lib/wic/canned-wks/nanopi-neo-efi.wks.in  |  2 +-
> >  .../lib/wic/canned-wks/nanopi-neo.wks.in      |  2 +-
> >  .../scripts/lib/wic/canned-wks/rpi-sdimg.wks  |  2 +-
> >  .../wic/canned-wks/sdimage-efi-sb-debian.wks  |  2 +-
> >  .../lib/wic/canned-wks/sdimage-efi-sd.wks     |  2 +-
> >  .../lib/wic/canned-wks/sdimage-efi.wks        |  2 +-
> >  .../lib/wic/canned-wks/sifive-fu540.wks       |  2 +-
> >  .../canned-wks/starfive-visionfive2.wks.in    |  2 +-
> >  .../lib/wic/canned-wks/stm32mp15x.wks.in      |  2 +-
> >  .../linux-module/files/debian/rules.tmpl      |  5 ++-
> >  scripts/lib/wic/partition.py                  | 37
> > ++++++++++++++++++-
> >  scripts/lib/wic/plugins/imager/direct.py      |  3 ++
> >  scripts/lib/wic/plugins/source/rootfs.py      |  2 +-
> >  18 files changed, 59 insertions(+), 18 deletions(-)
> >
> 
> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-
> users/3397cb2d254bc8a2e45c47b5a4094e34c85bec32.camel%40siemens.c
> om.

RE: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[Venkata.Pyla]

> -----Original Message-----
> From: 'MOESSBAUER, Felix' via isar-users <isar-users@googlegroups.com>
> Sent: Friday, December 8, 2023 4:04 PM
> To: isar-users@googlegroups.com; pyla venkata(ＴＳＩＰ TMIEC ODG Porting)
> <Venkata.Pyla@toshiba-tsip.com>
> Cc: Kiszka, Jan <jan.kiszka@siemens.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh
> kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>
> Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to
> generate reproducible ext4 images
> 
> On Thu, 2023-12-07 at 21:11 +0530, venkata.pyla@toshiba-tsip.com wrote:
> > From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> >
> > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file
> > system.
> > hash_seed: creates reproducible directory indexes in the file system.
> >
> > Reference commit in e2fsprogs:
> > e1f7100643a46456be107b33098f6034b0835e6d
> 
> Hi!
> 
> Did you also send this patch to OE?
> We need to integrate it there as well, as this is the basis for the ISAR plugins.

Not shared yet, I will send to OE as well.

> 
> Best regards,
> Felix
> 
> >
> > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> > ---
> >  scripts/lib/wic/partition.py | 11 +++++++++++
> >  1 file changed, 11 insertions(+)
> >
> > diff --git a/scripts/lib/wic/partition.py
> > b/scripts/lib/wic/partition.py index e50871b8..90b2c037 100644
> > --- a/scripts/lib/wic/partition.py
> > +++ b/scripts/lib/wic/partition.py
> > @@ -280,6 +280,17 @@ class Partition():
> >
> >          extraopts = self.mkfs_extraopts or "-F -i 8192"
> >
> > +        if os.getenv('SOURCE_DATE_EPOCH'):
> > +            sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
> > +            pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s" % (sde_time,
> > pseudo)
> > +
> > +            # Set hash_seed to generate deterministic directory
> > indexes
> > +            namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-
> > 2f2fdf33d460")
> > +            if self.fsuuid:
> > +                namespace = uuid.UUID(self.fsuuid)
> > +            hash_seed = str(uuid.uuid5(namespace, str(sde_time)))
> > +            extraopts += " -E hash_seed=%s" % hash_seed
> > +
> >          label_str = ""
> >          if self.label:
> >              label_str = "-L %s" % self.label
> 
> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-
> users/813c16c90a28fd387d5dbdb51aa02f98b1ebfa57.camel%40siemens.com.

Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[Florian Bezdeka]

On Fri, 2023-12-08 at 18:14 +0000, Venkata.Pyla@toshiba-tsip.com wrote:
> > -----Original Message-----
> > From: 'MOESSBAUER, Felix' via isar-users <isar-users@googlegroups.com>
> > Sent: Friday, December 8, 2023 4:04 PM
> > To: isar-users@googlegroups.com; pyla venkata(ＴＳＩＰ TMIEC ODG Porting)
> > <Venkata.Pyla@toshiba-tsip.com>
> > Cc: Kiszka, Jan <jan.kiszka@siemens.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
> > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh
> > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>
> > Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to
> > generate reproducible ext4 images
> > 
> > On Thu, 2023-12-07 at 21:11 +0530, venkata.pyla@toshiba-tsip.com wrote:
> > > From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> > > 
> > > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file
> > > system.
> > > hash_seed: creates reproducible directory indexes in the file system.
> > > 
> > > Reference commit in e2fsprogs:
> > > e1f7100643a46456be107b33098f6034b0835e6d
> > 
> > Hi!
> > 
> > Did you also send this patch to OE?
> > We need to integrate it there as well, as this is the basis for the ISAR plugins.
> 
> Not shared yet, I will send to OE as well.

We should follow "upstream first" here. Once OE has accepted the patch
we will inherit it during next OE -> Isar update cycle.

Maintainers: Do not apply it to Isar directly.

> 
> > 
> > Best regards,
> > Felix
> > 
> > > 

RE: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[Venkata.Pyla]

> -----Original Message-----
> From: 'Florian Bezdeka' via isar-users <isar-users@googlegroups.com>
> Sent: Saturday, December 9, 2023 4:38 PM
> To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-
> tsip.com>; felix.moessbauer@siemens.com; isar-users@googlegroups.com
> Cc: jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ
> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh
> kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>
> Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to
> generate reproducible ext4 images
> 
> On Fri, 2023-12-08 at 18:14 +0000, Venkata.Pyla@toshiba-tsip.com wrote:
> > > -----Original Message-----
> > > From: 'MOESSBAUER, Felix' via isar-users
> > > <isar-users@googlegroups.com>
> > > Sent: Friday, December 8, 2023 4:04 PM
> > > To: isar-users@googlegroups.com; pyla venkata(ＴＳＩＰ TMIEC ODG
> > > Porting) <Venkata.Pyla@toshiba-tsip.com>
> > > Cc: Kiszka, Jan <jan.kiszka@siemens.com>; hayashi kazuhiro(林 和宏
> ＤＭＥ
> > > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh
> > > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>
> > > Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed
> > > to generate reproducible ext4 images
> > >
> > > On Thu, 2023-12-07 at 21:11 +0530, venkata.pyla@toshiba-tsip.com
> wrote:
> > > > From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> > > >
> > > > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file
> > > > system.
> > > > hash_seed: creates reproducible directory indexes in the file system.
> > > >
> > > > Reference commit in e2fsprogs:
> > > > e1f7100643a46456be107b33098f6034b0835e6d
> > >
> > > Hi!
> > >
> > > Did you also send this patch to OE?
> > > We need to integrate it there as well, as this is the basis for the ISAR
> plugins.
> >
> > Not shared yet, I will send to OE as well.
> 
> We should follow "upstream first" here. Once OE has accepted the patch we
> will inherit it during next OE -> Isar update cycle.

I have shared this patch to OE [1], hope I will get feedback from the community and will support to fix this issue in the OE itself.

[1] https://lists.openembedded.org/g/openembedded-core/message/192192
> 
> Maintainers: Do not apply it to Isar directly.
> 
> >
> > >
> > > Best regards,
> > > Felix
> > >
> > > >
> 
> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-
> users/f2563bb179d011f22de10d212600f6ef4c0cb284.camel%40siemens.com
> .

RE: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[Venkata.Pyla]

> -----Original Message-----
> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf
> Of Venkata.Pyla@toshiba-tsip.com
> Sent: Tuesday, December 12, 2023 7:37 PM
> To: florian.bezdeka@siemens.com; felix.moessbauer@siemens.com; isar-
> users@googlegroups.com
> Cc: jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ
> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh
> kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>
> Subject: RE: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to
> generate reproducible ext4 images
> 
> 
> 
> > -----Original Message-----
> > From: 'Florian Bezdeka' via isar-users <isar-users@googlegroups.com>
> > Sent: Saturday, December 9, 2023 4:38 PM
> > To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-
> > tsip.com>; felix.moessbauer@siemens.com; isar-users@googlegroups.com
> > Cc: jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ
> > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh
> > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>
> > Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to
> > generate reproducible ext4 images
> >
> > On Fri, 2023-12-08 at 18:14 +0000, Venkata.Pyla@toshiba-tsip.com wrote:
> > > > -----Original Message-----
> > > > From: 'MOESSBAUER, Felix' via isar-users
> > > > <isar-users@googlegroups.com>
> > > > Sent: Friday, December 8, 2023 4:04 PM
> > > > To: isar-users@googlegroups.com; pyla venkata(ＴＳＩＰ TMIEC ODG
> > > > Porting) <Venkata.Pyla@toshiba-tsip.com>
> > > > Cc: Kiszka, Jan <jan.kiszka@siemens.com>; hayashi kazuhiro(林 和宏
> > ＤＭＥ
> > > > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh
> > > > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>
> > > > Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and
> > > > hash_seed to generate reproducible ext4 images
> > > >
> > > > On Thu, 2023-12-07 at 21:11 +0530, venkata.pyla@toshiba-tsip.com
> > wrote:
> > > > > From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> > > > >
> > > > > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file
> > > > > system.
> > > > > hash_seed: creates reproducible directory indexes in the file system.
> > > > >
> > > > > Reference commit in e2fsprogs:
> > > > > e1f7100643a46456be107b33098f6034b0835e6d
> > > >
> > > > Hi!
> > > >
> > > > Did you also send this patch to OE?
> > > > We need to integrate it there as well, as this is the basis for
> > > > the ISAR
> > plugins.
> > >
> > > Not shared yet, I will send to OE as well.
> >
> > We should follow "upstream first" here. Once OE has accepted the patch
> > we will inherit it during next OE -> Isar update cycle.
> 
> I have shared this patch to OE [1], hope I will get feedback from the community
> and will support to fix this issue in the OE itself.
> 
> [1] https://lists.openembedded.org/g/openembedded-core/message/192192

This patch [1] has been merged now in upstream OE-core,
 now the required patches should be taken to ISAR, is there any timeline to bring the OE-core changes to ISAR or should we apply the required changes to ISAR manually? 

[1] https://github.com/openembedded/openembedded-core/commit/bb822ab75de0020572058090439b93cc56bbf7e0

> >
> > Maintainers: Do not apply it to Isar directly.
> >
> > >
> > > >
> > > > Best regards,
> > > > Felix
> > > >
> > > > >
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "isar-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to isar-users+unsubscribe@googlegroups.com.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/isar-
> > users/f2563bb179d011f22de10d212600f6ef4c0cb284.camel%40siemens.com
> > .
> 
> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-
> users/OS3PR01MB68566626DF8F78E4F73D2A45A48EA%40OS3PR01MB6856.jpn
> prd01.prod.outlook.com.

Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[Florian Bezdeka]

On Thu, 2023-12-28 at 06:02 +0000, Venkata.Pyla@toshiba-tsip.com wrote:
> > -----Original Message-----
> > From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf
> > Of Venkata.Pyla@toshiba-tsip.com
> > Sent: Tuesday, December 12, 2023 7:37 PM
> > To: florian.bezdeka@siemens.com; felix.moessbauer@siemens.com; isar-
> > users@googlegroups.com
> > Cc: jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ
> > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh
> > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>
> > Subject: RE: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to
> > generate reproducible ext4 images
> > 
> > 
> > 
> > > -----Original Message-----
> > > From: 'Florian Bezdeka' via isar-users <isar-users@googlegroups.com>
> > > Sent: Saturday, December 9, 2023 4:38 PM
> > > To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-
> > > tsip.com>; felix.moessbauer@siemens.com; isar-users@googlegroups.com
> > > Cc: jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ
> > > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh
> > > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>
> > > Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to
> > > generate reproducible ext4 images
> > > 
> > > On Fri, 2023-12-08 at 18:14 +0000, Venkata.Pyla@toshiba-tsip.com wrote:
> > > > > -----Original Message-----
> > > > > From: 'MOESSBAUER, Felix' via isar-users
> > > > > <isar-users@googlegroups.com>
> > > > > Sent: Friday, December 8, 2023 4:04 PM
> > > > > To: isar-users@googlegroups.com; pyla venkata(ＴＳＩＰ TMIEC ODG
> > > > > Porting) <Venkata.Pyla@toshiba-tsip.com>
> > > > > Cc: Kiszka, Jan <jan.kiszka@siemens.com>; hayashi kazuhiro(林 和宏
> > > ＤＭＥ
> > > > > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh
> > > > > kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>
> > > > > Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and
> > > > > hash_seed to generate reproducible ext4 images
> > > > > 
> > > > > On Thu, 2023-12-07 at 21:11 +0530, venkata.pyla@toshiba-tsip.com
> > > wrote:
> > > > > > From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> > > > > > 
> > > > > > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file
> > > > > > system.
> > > > > > hash_seed: creates reproducible directory indexes in the file system.
> > > > > > 
> > > > > > Reference commit in e2fsprogs:
> > > > > > e1f7100643a46456be107b33098f6034b0835e6d
> > > > > 
> > > > > Hi!
> > > > > 
> > > > > Did you also send this patch to OE?
> > > > > We need to integrate it there as well, as this is the basis for
> > > > > the ISAR
> > > plugins.
> > > > 
> > > > Not shared yet, I will send to OE as well.
> > > 
> > > We should follow "upstream first" here. Once OE has accepted the patch
> > > we will inherit it during next OE -> Isar update cycle.
> > 
> > I have shared this patch to OE [1], hope I will get feedback from the community
> > and will support to fix this issue in the OE itself.
> > 
> > [1] https://lists.openembedded.org/g/openembedded-core/message/192192
> 
> This patch [1] has been merged now in upstream OE-core,
>  now the required patches should be taken to ISAR, is there any timeline to bring the OE-core changes to ISAR or should we apply the required changes to ISAR manually? 
> 
> [1] https://github.com/openembedded/openembedded-core/commit/bb822ab75de0020572058090439b93cc56bbf7e0

That's great! Thanks for taking care.

I added two guys from Ilbers, maybe they have an update pending already
or can comment on the timeline. Anton, Uladzimir any comments?

In general it's a good idea to come up with a patch to speed things up.

The last update was done in 64af29bcc4b0 ("wic: Update to the latest
revision"). Might be used as inspiration.

Florian

> 
> > > 
> > > Maintainers: Do not apply it to Isar directly.
> > > 
> > > > 
> > > > > 
> > > > > Best regards,
> > > > > Felix
> > > > > 
> > > > > > 
> > > 
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "isar-users" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to isar-users+unsubscribe@googlegroups.com.
> > > To view this discussion on the web visit
> > > https://groups.google.com/d/msgid/isar-
> > > users/f2563bb179d011f22de10d212600f6ef4c0cb284.camel%40siemens.com
> > > .
> > 
> > --
> > You received this message because you are subscribed to the Google Groups
> > "isar-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to
> > isar-users+unsubscribe@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/isar-
> > users/OS3PR01MB68566626DF8F78E4F73D2A45A48EA%40OS3PR01MB6856.jpn
> > prd01.prod.outlook.com.

Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[Anton Mikanovich]

28/12/2023 10:47, Florian Bezdeka wrote:
> On Thu, 2023-12-28 at 06:02 +0000, Venkata.Pyla@toshiba-tsip.com wrote:
>> This patch [1] has been merged now in upstream OE-core,
>>   now the required patches should be taken to ISAR, is there any timeline to bring the OE-core changes to ISAR or should we apply the required changes to ISAR manually?
>>
>> [1] https://github.com/openembedded/openembedded-core/commit/bb822ab75de0020572058090439b93cc56bbf7e0
> That's great! Thanks for taking care.
>
> I added two guys from Ilbers, maybe they have an update pending already
> or can comment on the timeline. Anton, Uladzimir any comments?
>
> In general it's a good idea to come up with a patch to speed things up.
>
> The last update was done in 64af29bcc4b0 ("wic: Update to the latest
> revision"). Might be used as inspiration.
>
> Florian
Hello,

We are going to make v0.10 release soon and probably can take care of 
externals
(like WIC, OE libs and Bitbake) update right after that.

RE: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[Adithya.Balakumar]

Hello Anton,

Hope this message finds you well.

Just wanted to follow up on a previous mail regarding wic and other related updates from openembedded-core. You had mentioned about the updates from openembedded-core to be handled right after the release of ISAR v0.10

Would like to understand if there are any plans for wic and other related updates from openembedded-core.

Thanks and Regards,
Adithya Balakumar

-----Original Message-----
From: Anton Mikanovich <amikan@ilbers.de> 
Sent: Thursday, December 28, 2023 3:40 PM
To: Florian Bezdeka <florian.bezdeka@siemens.com>; pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba-tsip.com>; felix.moessbauer@siemens.com; isar-users@googlegroups.com; Uladzimir Bely <ubely@ilbers.de>
Cc: jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; venkatapyla4@gmail.com; balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>
Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

28/12/2023 10:47, Florian Bezdeka wrote:
> On Thu, 2023-12-28 at 06:02 +0000, Venkata.Pyla@toshiba-tsip.com wrote:
>> This patch [1] has been merged now in upstream OE-core,
>>   now the required patches should be taken to ISAR, is there any timeline to bring the OE-core changes to ISAR or should we apply the required changes to ISAR manually?
>>
>> [1] 
>> https://github.com/openembedded/openembedded-core/commit/bb822ab75de0
>> 020572058090439b93cc56bbf7e0
> That's great! Thanks for taking care.
>
> I added two guys from Ilbers, maybe they have an update pending 
> already or can comment on the timeline. Anton, Uladzimir any comments?
>
> In general it's a good idea to come up with a patch to speed things up.
>
> The last update was done in 64af29bcc4b0 ("wic: Update to the latest 
> revision"). Might be used as inspiration.
>
> Florian
Hello,

We are going to make v0.10 release soon and probably can take care of externals (like WIC, OE libs and Bitbake) update right after that.

Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[Anton Mikanovich]

13/03/2024 12:01, Adithya.Balakumar@toshiba-tsip.com wrote:
> Hello Anton,
>
> Hope this message finds you well.
>
> Just wanted to follow up on a previous mail regarding wic and other related updates from openembedded-core. You had mentioned about the updates from openembedded-core to be handled right after the release of ISAR v0.10
>
> Would like to understand if there are any plans for wic and other related updates from openembedded-core.
>
> Thanks and Regards,
> Adithya Balakumar

Hello Adithya,

We've checked and the next OE LTS using bitbake 2.8 should be released 
in April.
We could start preparations for the bitbake upgrade and update OE after 
that.
Would that suit your needs?

RE: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[Adithya.Balakumar]

Hello Anton,

Thanks for the quick update. Updating OE after the update to bitbake 2.8 should be fine.

Thanks and Regards,
Adithya Balakumar

-----Original Message-----
From: Anton Mikanovich <amikan@ilbers.de> 
Sent: Wednesday, March 13, 2024 9:30 PM
To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; florian.bezdeka@siemens.com; venkata1.pyla <venkata1.pyla@toshibaap.onmicrosoft.com>; felix.moessbauer@siemens.com; isar-users@googlegroups.com; ubely@ilbers.de
Cc: jan.kiszka@siemens.com; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; venkatapyla4@gmail.com
Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

13/03/2024 12:01, Adithya.Balakumar@toshiba-tsip.com wrote:
> Hello Anton,
>
> Hope this message finds you well.
>
> Just wanted to follow up on a previous mail regarding wic and other 
> related updates from openembedded-core. You had mentioned about the 
> updates from openembedded-core to be handled right after the release 
> of ISAR v0.10
>
> Would like to understand if there are any plans for wic and other related updates from openembedded-core.
>
> Thanks and Regards,
> Adithya Balakumar

Hello Adithya,

We've checked and the next OE LTS using bitbake 2.8 should be released in April.
We could start preparations for the bitbake upgrade and update OE after that.
Would that suit your needs?

Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[MOESSBAUER, Felix]

On Thu, 2023-12-07 at 21:11 +0530, venkata.pyla@toshiba-tsip.com wrote:
> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> 
> E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file
> system.
> hash_seed: creates reproducible directory indexes in the file system.
> 
> Reference commit in e2fsprogs:
> e1f7100643a46456be107b33098f6034b0835e6d
> 
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  scripts/lib/wic/partition.py | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/scripts/lib/wic/partition.py
> b/scripts/lib/wic/partition.py
> index e50871b8..90b2c037 100644
> --- a/scripts/lib/wic/partition.py
> +++ b/scripts/lib/wic/partition.py
> @@ -280,6 +280,17 @@ class Partition():
>  
>          extraopts = self.mkfs_extraopts or "-F -i 8192"
>  
> +        if os.getenv('SOURCE_DATE_EPOCH'):
> +            sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
> +            pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s" % (sde_time,
> pseudo)
> +
> +            # Set hash_seed to generate deterministic directory
> indexes
> +            namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-
> 2f2fdf33d460")
> +            if self.fsuuid:
> +                namespace = uuid.UUID(self.fsuuid)
> +            hash_seed = str(uuid.uuid5(namespace, str(sde_time)))
> +            extraopts += " -E hash_seed=%s" % hash_seed
> +

Hi, while reworking the SDE in ISAR, I stumbled upon this as well.
This patch only covers the .wic part, but we need a similar patch for
the IMAGE_CMD:ext4 as well. I tried to mimic the pattern there, but I
was not able to make the .ext4 build reproducible (the diff indicated
that the inodes are still shuffled around). This makes me wonder if
mke2fs.ext4 even supports producing a reproducible rootfs.

Have you been able to create a bit-by-bit identical .wic of an ext4
partition?

Best regards,
Felix

>          label_str = ""
>          if self.label:
>              label_str = "-L %s" % self.label

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[MOESSBAUER, Felix]

On Wed, 2024-03-13 at 18:00 +0200, Anton Mikanovich wrote:
> 13/03/2024 12:01, Adithya.Balakumar@toshiba-tsip.com wrote:
> > Hello Anton,
> > 
> > Hope this message finds you well.
> > 
> > Just wanted to follow up on a previous mail regarding wic and other
> > related updates from openembedded-core. You had mentioned about the
> > updates from openembedded-core to be handled right after the
> > release of ISAR v0.10
> > 
> > Would like to understand if there are any plans for wic and other
> > related updates from openembedded-core.
> > 
> > Thanks and Regards,
> > Adithya Balakumar
> 
> Hello Adithya,
> 
> We've checked and the next OE LTS using bitbake 2.8 should be
> released 
> in April.

Hi, bitbake 2.8 has been released last week.

> We could start preparations for the bitbake upgrade and update OE
> after 
> that.

It would be great if we could have bitbake 2.8 in a timely manner. This
release contains many of our contributed fixes (e.g. for reproducible
builds, better estimation of available CPUs on large servers).

Felix

> Would that suit your needs?
> 

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[Jan Kiszka]

On 25.04.24 08:24, Moessbauer, Felix (T CED OES-DE) wrote:
> On Wed, 2024-03-13 at 18:00 +0200, Anton Mikanovich wrote:
>> 13/03/2024 12:01, Adithya.Balakumar@toshiba-tsip.com wrote:
>>> Hello Anton,
>>>
>>> Hope this message finds you well.
>>>
>>> Just wanted to follow up on a previous mail regarding wic and other
>>> related updates from openembedded-core. You had mentioned about the
>>> updates from openembedded-core to be handled right after the
>>> release of ISAR v0.10
>>>
>>> Would like to understand if there are any plans for wic and other
>>> related updates from openembedded-core.
>>>
>>> Thanks and Regards,
>>> Adithya Balakumar
>>
>> Hello Adithya,
>>
>> We've checked and the next OE LTS using bitbake 2.8 should be
>> released 
>> in April.
> 
> Hi, bitbake 2.8 has been released last week.
> 
>> We could start preparations for the bitbake upgrade and update OE
>> after 
>> that.
> 
> It would be great if we could have bitbake 2.8 in a timely manner. This
> release contains many of our contributed fixes (e.g. for reproducible
> builds, better estimation of available CPUs on large servers).
> 

Any impact on recipe syntax etc. again?

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[MOESSBAUER, Felix]

On Thu, 2024-04-25 at 08:30 +0200, Jan Kiszka wrote:
> On 25.04.24 08:24, Moessbauer, Felix (T CED OES-DE) wrote:
> > On Wed, 2024-03-13 at 18:00 +0200, Anton Mikanovich wrote:
> > > 13/03/2024 12:01, Adithya.Balakumar@toshiba-tsip.com wrote:
> > > > Hello Anton,
> > > > 
> > > > Hope this message finds you well.
> > > > 
> > > > Just wanted to follow up on a previous mail regarding wic and
> > > > other
> > > > related updates from openembedded-core. You had mentioned about
> > > > the
> > > > updates from openembedded-core to be handled right after the
> > > > release of ISAR v0.10
> > > > 
> > > > Would like to understand if there are any plans for wic and
> > > > other
> > > > related updates from openembedded-core.
> > > > 
> > > > Thanks and Regards,
> > > > Adithya Balakumar
> > > 
> > > Hello Adithya,
> > > 
> > > We've checked and the next OE LTS using bitbake 2.8 should be
> > > released 
> > > in April.
> > 
> > Hi, bitbake 2.8 has been released last week.
> > 
> > > We could start preparations for the bitbake upgrade and update OE
> > > after 
> > > that.
> > 
> > It would be great if we could have bitbake 2.8 in a timely manner.
> > This
> > release contains many of our contributed fixes (e.g. for
> > reproducible
> > builds, better estimation of available CPUs on large servers).
> > 
> 
> Any impact on recipe syntax etc. again?

I just did a quick test of bitbake 2.8 in ISAR and that (almost) worked
out of the box. Only required change was to move the
kbuildtarget.bbclass into the meta/classes dir.

Probably the only relevant changes are around the handling of PR:

- The PR value can no longer be set from the recipe file name - this
was rarely used, but in any case is no longer supported.
- PE and PR are no longer included in the work directory path
(WORKDIR). This may break some tool assumptions about directory paths,
but those should really be querying paths from the build system (or not
poking into WORKDIR externally).

https://docs.yoctoproject.org/4.3.4/migration-guides/migration-4.3.html#versioning-changes

Felix

> 
> Jan
> 

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[Anton Mikanovich]

25/04/2024 09:24, MOESSBAUER, Felix wrote:
> On Wed, 2024-03-13 at 18:00 +0200, Anton Mikanovich wrote:
>> Hello Adithya,
>>
>> We've checked and the next OE LTS using bitbake 2.8 should be
>> released
>> in April.
> Hi, bitbake 2.8 has been released last week.
>
>> We could start preparations for the bitbake upgrade and update OE
>> after
>> that.
> It would be great if we could have bitbake 2.8 in a timely manner. This
> release contains many of our contributed fixes (e.g. for reproducible
> builds, better estimation of available CPUs on large servers).
>
> Felix
>
>> Would that suit your needs?
>>
Hello,

I've already started working on Bitbake and OE libs update.
As you've already mentioned there are no much changes, so hope will send 
it soon.

Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[Adithya Balakumar]

[-- Attachment #1.1: Type: text/plain, Size: 2911 bytes --]

Hi Felix,

I saw your mail regarding your attempt to make ext4 filesystem images from 
IMAGE_CMD:ext4 reproducible. 
If you don't mind, could you briefly explain what was the problem you faced 
in achieving this? 
I am also trying to understand on how to achieve the same.

Thanks and Regards,
Adithya Balakumar


On Tuesday, April 23, 2024 at 2:47:11 PM UTC+5:30 MOESSBAUER, Felix wrote:

On Thu, 2023-12-07 at 21:11 +0530, venkat...@toshiba-tsip.com wrote: 
> From: venkata pyla <venkat...@toshiba-tsip.com> 
> 
> E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file 
> system. 
> hash_seed: creates reproducible directory indexes in the file system. 
> 
> Reference commit in e2fsprogs: 
> e1f7100643a46456be107b33098f6034b0835e6d 
> 
> Signed-off-by: venkata pyla <venkat...@toshiba-tsip.com> 
> --- 
>  scripts/lib/wic/partition.py | 11 +++++++++++ 
>  1 file changed, 11 insertions(+) 
> 
> diff --git a/scripts/lib/wic/partition.py 
> b/scripts/lib/wic/partition.py 
> index e50871b8..90b2c037 100644 
> --- a/scripts/lib/wic/partition.py 
> +++ b/scripts/lib/wic/partition.py 
> @@ -280,6 +280,17 @@ class Partition(): 
>   
>          extraopts = self.mkfs_extraopts or "-F -i 8192" 
>   
> +        if os.getenv('SOURCE_DATE_EPOCH'): 
> +            sde_time = int(os.getenv('SOURCE_DATE_EPOCH')) 
> +            pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s" % (sde_time, 
> pseudo) 
> + 
> +            # Set hash_seed to generate deterministic directory 
> indexes 
> +            namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9- 
> 2f2fdf33d460") 
> +            if self.fsuuid: 
> +                namespace = uuid.UUID(self.fsuuid) 
> +            hash_seed = str(uuid.uuid5(namespace, str(sde_time))) 
> +            extraopts += " -E hash_seed=%s" % hash_seed 
> + 

Hi, while reworking the SDE in ISAR, I stumbled upon this as well. 
This patch only covers the .wic part, but we need a similar patch for 
the IMAGE_CMD:ext4 as well. I tried to mimic the pattern there, but I 
was not able to make the .ext4 build reproducible (the diff indicated 
that the inodes are still shuffled around). This makes me wonder if 
mke2fs.ext4 even supports producing a reproducible rootfs. 

Have you been able to create a bit-by-bit identical .wic of an ext4 
partition? 

Best regards, 
Felix 

>          label_str = "" 
>          if self.label: 
>              label_str = "-L %s" % self.label 

-- 
Siemens AG, Technology 
Linux Expert Center 


-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/3d9226f6-f046-445e-9edc-e107111c2437n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 4108 bytes --]

Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

['MOESSBAUER, Felix' via isar-users]

On Fri, 2024-07-05 at 05:13 -0700, Adithya Balakumar wrote:
> Hi Felix,
> 
> I saw your mail regarding your attempt to make ext4 filesystem images
> from IMAGE_CMD:ext4 reproducible. 
> If you don't mind, could you briefly explain what was the problem you
> faced in achieving this? 

Hi, the problem is stated below:

> the diff indicated that the inodes are still shuffled around). This
makes me wonder if mke2fs.ext4 even supports producing a reproducible
rootfs.

I just copied the pattern from wic, but for whatever reason the inodes
still were not deterministic.

Are you sure, that this patch is sufficient to make the ext4
reproducible?

Felix

> I am also trying to understand on how to achieve the same.
> 
> Thanks and Regards,
> Adithya Balakumar
> 
> 
> On Tuesday, April 23, 2024 at 2:47:11 PM UTC+5:30 MOESSBAUER, Felix
> wrote:
> > On Thu, 2023-12-07 at 21:11 +0530, venkat...@toshiba-tsip.com
> > wrote:
> > > From: venkata pyla <venkat...@toshiba-tsip.com>
> > > 
> > > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file
> > > system.
> > > hash_seed: creates reproducible directory indexes in the file
> > > system.
> > > 
> > > Reference commit in e2fsprogs:
> > > e1f7100643a46456be107b33098f6034b0835e6d
> > > 
> > > Signed-off-by: venkata pyla <venkat...@toshiba-tsip.com>
> > > ---
> > >  scripts/lib/wic/partition.py | 11 +++++++++++
> > >  1 file changed, 11 insertions(+)
> > > 
> > > diff --git a/scripts/lib/wic/partition.py
> > > b/scripts/lib/wic/partition.py
> > > index e50871b8..90b2c037 100644
> > > --- a/scripts/lib/wic/partition.py
> > > +++ b/scripts/lib/wic/partition.py
> > > @@ -280,6 +280,17 @@ class Partition():
> > >  
> > >          extraopts = self.mkfs_extraopts or "-F -i 8192"
> > >  
> > > +        if os.getenv('SOURCE_DATE_EPOCH'):
> > > +            sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
> > > +            pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s" %
> > > (sde_time,
> > > pseudo)
> > > +
> > > +            # Set hash_seed to generate deterministic directory
> > > indexes
> > > +            namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-
> > > 2f2fdf33d460")
> > > +            if self.fsuuid:
> > > +                namespace = uuid.UUID(self.fsuuid)
> > > +            hash_seed = str(uuid.uuid5(namespace,
> > > str(sde_time)))
> > > +            extraopts += " -E hash_seed=%s" % hash_seed
> > > +
> > 
> > Hi, while reworking the SDE in ISAR, I stumbled upon this as well.
> > This patch only covers the .wic part, but we need a similar patch
> > for
> > the IMAGE_CMD:ext4 as well. I tried to mimic the pattern there, but
> > I
> > was not able to make the .ext4 build reproducible (the diff
> > indicated
> > that the inodes are still shuffled around). This makes me wonder if
> > mke2fs.ext4 even supports producing a reproducible rootfs.
> > 
> > Have you been able to create a bit-by-bit identical .wic of an ext4
> > partition?
> > 
> > Best regards,
> > Felix
> > 
> > >          label_str = ""
> > >          if self.label:
> > >              label_str = "-L %s" % self.label
> > 
> > -- 
> > Siemens AG, Technology
> > Linux Expert Center
> > 
> > 

-- 
Siemens AG, Technology
Linux Expert Center


-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/356aaf0ad95d69f5c1d023d62cfe6d8a0ac0d024.camel%40siemens.com.

RE: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images

[Adithya.Balakumar]

[-- Attachment #1: Type: text/plain, Size: 5308 bytes --]

Hi Felix,

To answer your question, No, these patches alone are not enough to make ext4 filesystem images reproducible.
These patches were mainly tested with isar-cip-core security images which has /home ad /var mounted as ext4 filesystem images.

With these patches I noticed the /home ext4 partition is reproducible but /var required an extra fix (https://groups.google.com/g/isar-users/c/RsjRjzigLOE)

But, I see reproducibility problems when entire rootfs is deployed in an ext4 filesystem (in the case of wic and with IMAGE_CMD:ext4). I even raised this point in the ISAR ML (https://groups.google.com/g/isar-users/c/Ll7t4G41Lfo)

That’s when I saw your previous mail.

Thanks and Regards,
Adithya Balakumar

From: Adithya Balakumar <adithya190298@gmail.com>
Sent: Monday, July 8, 2024 4:28 PM
To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>
Subject: Fwd: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images


---------- Forwarded message ---------
From: MOESSBAUER, Felix <felix.moessbauer@siemens.com<mailto:felix.moessbauer@siemens.com>>
Date: Mon, 8 Jul, 2024, 16:13
Subject: Re: [PATCH 3/5] wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images
To: isar-users@googlegroups.com<mailto:isar-users@googlegroups.com> <isar-users@googlegroups.com<mailto:isar-users@googlegroups.com>>, adithya190298@gmail.com<mailto:adithya190298@gmail.com> <adithya190298@gmail.com<mailto:adithya190298@gmail.com>>


On Fri, 2024-07-05 at 05:13 -0700, Adithya Balakumar wrote:
> Hi Felix,
>
> I saw your mail regarding your attempt to make ext4 filesystem images
> from IMAGE_CMD:ext4 reproducible.
> If you don't mind, could you briefly explain what was the problem you
> faced in achieving this?

Hi, the problem is stated below:

> the diff indicated that the inodes are still shuffled around). This
makes me wonder if mke2fs.ext4 even supports producing a reproducible
rootfs.

I just copied the pattern from wic, but for whatever reason the inodes
still were not deterministic.

Are you sure, that this patch is sufficient to make the ext4
reproducible?

Felix

> I am also trying to understand on how to achieve the same.
>
> Thanks and Regards,
> Adithya Balakumar
>
>
> On Tuesday, April 23, 2024 at 2:47:11 PM UTC+5:30 MOESSBAUER, Felix
> wrote:
> > On Thu, 2023-12-07 at 21:11 +0530, venkat...@toshiba-tsip.com<mailto:venkat...@toshiba-tsip.com>
> > wrote:
> > > From: venkata pyla <venkat...@toshiba-tsip.com<mailto:venkat...@toshiba-tsip.com>>
> > >
> > > E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file
> > > system.
> > > hash_seed: creates reproducible directory indexes in the file
> > > system.
> > >
> > > Reference commit in e2fsprogs:
> > > e1f7100643a46456be107b33098f6034b0835e6d
> > >
> > > Signed-off-by: venkata pyla <venkat...@toshiba-tsip.com<mailto:venkat...@toshiba-tsip.com>>
> > > ---
> > >  scripts/lib/wic/partition.py | 11 +++++++++++
> > >  1 file changed, 11 insertions(+)
> > >
> > > diff --git a/scripts/lib/wic/partition.py
> > > b/scripts/lib/wic/partition.py
> > > index e50871b8..90b2c037 100644
> > > --- a/scripts/lib/wic/partition.py
> > > +++ b/scripts/lib/wic/partition.py
> > > @@ -280,6 +280,17 @@ class Partition():
> > >
> > >          extraopts = self.mkfs_extraopts or "-F -i 8192"
> > >
> > > +        if os.getenv('SOURCE_DATE_EPOCH'):
> > > +            sde_time = int(os.getenv('SOURCE_DATE_EPOCH'))
> > > +            pseudo = "export E2FSPROGS_FAKE_TIME=%s;%s" %
> > > (sde_time,
> > > pseudo)
> > > +
> > > +            # Set hash_seed to generate deterministic directory
> > > indexes
> > > +            namespace = uuid.UUID("e7429877-e7b3-4a68-a5c9-
> > > 2f2fdf33d460")
> > > +            if self.fsuuid:
> > > +                namespace = uuid.UUID(self.fsuuid)
> > > +            hash_seed = str(uuid.uuid5(namespace,
> > > str(sde_time)))
> > > +            extraopts += " -E hash_seed=%s" % hash_seed
> > > +
> >
> > Hi, while reworking the SDE in ISAR, I stumbled upon this as well.
> > This patch only covers the .wic part, but we need a similar patch
> > for
> > the IMAGE_CMD:ext4 as well. I tried to mimic the pattern there, but
> > I
> > was not able to make the .ext4 build reproducible (the diff
> > indicated
> > that the inodes are still shuffled around). This makes me wonder if
> > mke2fs.ext4 even supports producing a reproducible rootfs.
> >
> > Have you been able to create a bit-by-bit identical .wic of an ext4
> > partition?
> >
> > Best regards,
> > Felix
> >
> > >          label_str = ""
> > >          if self.label:
> > >              label_str = "-L %s" % self.label
> >
> > --
> > Siemens AG, Technology
> > Linux Expert Center
> >
> >

--
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/TYCPR01MB96695360B889AA307AC0FBEBC4DA2%40TYCPR01MB9669.jpnprd01.prod.outlook.com.

[-- Attachment #2: Type: text/html, Size: 11466 bytes --]