Re: [PATCH v3 1/1] image.bbclass: fix non-reproducible file time-stamps inside rootfs

["Moessbauer, Felix" <felix.moessbauer@siemens.com>]

On Thu, 2023-01-05 at 20:42 +0530, venkata.pyla@toshiba-tsip.com wrote:
> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> 
> As part of reproducible-build work, the rootfs images generated on
> same
> source should be identical between two builds.
> 
> In this commit it tries to solve one of the non-reproducible problem
> i.e. the rootfs file time-stamps generated during build time are not
> reproducible, it uses one of the solution provided in the debian
> live-build image project (refer [1]), it fixes by finding all the
> files/folders that are gernerated newly and set the time-stamp
> provided
> by `SOURCE_DATE_EPOCH` environment variable.
> 
> [1] 
> https://salsa.debian.org/live-team/live-build/-/merge_requests/218
> 
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  meta-isar/conf/local.conf.sample | 10 ++++++++++
>  meta/classes/image.bbclass       | 10 ++++++++++
>  2 files changed, 20 insertions(+)
> 
> diff --git a/meta-isar/conf/local.conf.sample b/meta-
> isar/conf/local.conf.sample
> index 57d0620..3c4a473 100644
> --- a/meta-isar/conf/local.conf.sample
> +++ b/meta-isar/conf/local.conf.sample
> @@ -255,3 +255,13 @@ USER_isar[flags] += "clear-text-password"
>  #CCACHE_TOP_DIR ?= "${TMPDIR}/ccache"
>  # Enable ccache debug mode
>  #CCACHE_DEBUG = "1"
> +
> +# Uncommnet and add value to it to build images reproducibly
> +#
> +# The value for `SOURCE_DATE_EPOCH` should be latest source change
> time in
> +# seconds since the Epoch.
> +# Git repository users can use value from 'git log -1 --pretty=%ct'
> +# Non git repository users can use value from 'stat -c%Y ChangeLog'
> +# To know more details about this variable and how to set the value
> refer below
> +# https://reproducible-builds.org/docs/source-date-epoch/
> +#SOURCE_DATE_EPOCH =
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 813e1f3..8371ecd 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -431,6 +431,16 @@ do_rootfs_finalize() {
>  
>          rm -f "${ROOTFSDIR}/etc/apt/sources-list"
>  EOSUDO
> +
> +    # Set same time-stamps to the newly generated file/folders in
> the
> +    # rootfs image for the purpose of reproducible builds.
> +    test ! -z "${SOURCE_DATE_EPOCH}" && \
> +        sudo find ${ROOTFSDIR} -newermt \
> +            "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \
> +            -printf "%y %p\n" \
> +            -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' >
> ${DEPLOY_DIR_IMAGE}/files.modified_timestamps && \
> +            bbwarn "$(cat
> ${DEPLOY_DIR_IMAGE}/files.modified_timestamps) \nModified above file
> timestamps to build image reproducibly"
> +

Hi, I just tested this code and found the following issues:

This does not rebuild cleanly, as in general the do_rootfs_finalize
cannot be re-executed. IMHO this is an ISAR bug and not a problem of
your patch:

do_rootfs_finalize
mv: cannot stat '/build/tmp/work/debian-bookworm-amd64/img-<...>/1.0-
r0/rootfs/etc/apt/sources-list': No such file or directory

Second, it is a bit hard to use with automatic data from git.
I solved it the following way, which works so far:

SOURCE_DATE_EPOCH := "${@ bb.process.run('git -C ${LAYERDIR_project}
log -1 --pretty=\%ct')[0].strip() }"

One problem here is, that this can hardly be generalized as we have to
feed in the path of the main layer - which is not known by ISAR.

The list of touched files is quite long, but basically most of it falls
in the following classes:

- PKG info: /var/lib/dpkg/info/
- Pycache: /usr/lib/python3.10/curses/__pycache__ This anyways needs to
be further investigated as my understanding is that we do not want to
distribute the pycache
- directories and symlinks:  These always have to be fixed as the
creation date depends on the install order of packages. I would not
warn on these

Best regards,
Felix

>  }
>  addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess
>  
> -- 
> 2.20.1
> 
>