[PATCH 1/4] set SOURCE_DATE_EPOCH in wic environment

[PATCH 1/4] set SOURCE_DATE_EPOCH in wic environment

[Felix Moessbauer]

By setting the variable in the generate_wic_image task, all wic
utilities that support reproducible builds pick it up and operate
accordingly.

This also makes the boot.scr u-boot script reproducible, as the script
generation is triggered by the wic plugin rootfs-u-boot.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/imagetypes_wic.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/imagetypes_wic.bbclass b/meta/classes/imagetypes_wic.bbclass
index 3869525b..6fadc3ce 100644
--- a/meta/classes/imagetypes_wic.bbclass
+++ b/meta/classes/imagetypes_wic.bbclass
@@ -170,6 +170,9 @@ generate_wic_image() {
     export FAKEROOTCMD=${FAKEROOTCMD}
     export BUILDDIR=${TOPDIR}
     export MTOOLS_SKIP_CHECK=1
+    if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
+        export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
+    fi
     mkdir -p ${IMAGE_ROOTFS}/../pseudo
     touch ${IMAGE_ROOTFS}/../pseudo/files.db
 
-- 
2.34.1

[PATCH 2/4] remove apt cache dirs from final rootfs

[Felix Moessbauer]

In 0c177a4b an improvement was added to not add the apt cache to the
sstate cache. This lead to reproducibility issues in case the rootfs is
provided by the sstate cache, compared to a newly assembled one.

As these folders are created on-the-fly when running apt, we simply
remove them from the final image.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/rootfs.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 86f228d0..870fcf2c 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -235,6 +235,8 @@ rootfs_postprocess_clean_package_cache() {
     sudo -E chroot '${ROOTFSDIR}' \
         /usr/bin/apt-get clean
     sudo rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"*
+    # remove apt-cache folder itself (required in case rootfs is provided by sstate cache)
+    sudo rm -rf "${ROOTFSDIR}/var/cache/apt/archives"
 }
 
 ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-log-files', 'rootfs_postprocess_clean_log_files', '', d)}"
-- 
2.34.1

[PATCH 3/4] clean /tmp in rootfs postprocess

[Felix Moessbauer]

This patch cleans all files below /tmp to ensure the rootfs is
reproducible, independent if provided by the sstate cache or rebuild.
In general, /tmp has to be considered volatile across boots, so
always cleaning it is not harmful.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/rootfs.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 870fcf2c..1c253616 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -267,6 +267,12 @@ rootfs_postprocess_clean_ldconfig_cache() {
     sudo rm -f ${ROOTFSDIR}/var/cache/ldconfig/aux-cache
 }
 
+ROOTFS_POSTPROCESS_COMMAND += "rootfs_postprocess_clean_tmp"
+rootfs_postprocess_clean_tmp() {
+    # /tmp is by definition non persistent across boots
+    sudo rm -rf "${ROOTFSDIR}/tmp/"*
+}
+
 ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'generate-manifest', 'rootfs_generate_manifest', '', d)}"
 rootfs_generate_manifest () {
     mkdir -p ${ROOTFS_MANIFEST_DEPLOY_DIR}
-- 
2.34.1

[PATCH 4/4] warn only once in case file timestamps are updated

[Felix Moessbauer]

When creating a reproducible image, file timestamps of too-new files are
updated in a postprocess step. This includes potentially many files.

Previously, a bitbake warning was filed per single file, which
overflooded the terminal. The new implementation simply prints a summary
and gives a hint where the full list can be found.

Suggested-by: Roberto A. Foglietta <roberto.foglietta@gmail.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/image.bbclass | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index e59767e7..eb546084 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -447,11 +447,14 @@ EOSUDO
     # Set same time-stamps to the newly generated file/folders in the
     # rootfs image for the purpose of reproducible builds.
     if [ -n "${SOURCE_DATE_EPOCH}" ]; then
-        sudo find ${ROOTFSDIR} -newermt \
-            "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \
-            -printf "%y %p\n" \
-            -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > ${DEPLOY_DIR_IMAGE}/files.modified_timestamps && \
-            bbwarn "$(grep ^f ${DEPLOY_DIR_IMAGE}/files.modified_timestamps) \nModified above file timestamps to build image reproducibly"
+        fn="${DEPLOY_DIR_IMAGE}/files.modified_timestamps"
+        if sudo find ${ROOTFSDIR} -newermt "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \
+            -printf "%y %p\n" -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' | egrep ^f >"$fn"; then
+            if [ -e "$fn" ]; then
+                bbwarn "modified timestamp (${SOURCE_DATE_EPOCH}) of $(cat "$fn" | wc -l) files for image reproducibly." \
+                       "List of files modified can be found in: .${DEPLOY_DIR_IMAGE}/files.modified_timestamps"
+            fi
+        fi
     fi
 
 }
-- 
2.34.1

Re: [PATCH 1/4] set SOURCE_DATE_EPOCH in wic environment

[Uladzimir Bely]

In mail from Wednesday, 25 January 2023 17:46:29 +03 user Felix Moessbauer 
wrote:
> By setting the variable in the generate_wic_image task, all wic
> utilities that support reproducible builds pick it up and operate
> accordingly.
> 
> This also makes the boot.scr u-boot script reproducible, as the script
> generation is triggered by the wic plugin rootfs-u-boot.
> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---

Series applied to next, thanks.