Re: [PATCH v2 1/1] image.bbclass: fix non-reproducible file time-stamps inside rootfs

["Moessbauer, Felix" <felix.moessbauer@siemens.com>]

On Thu, 2023-01-05 at 09:19 +0100, Henning Schild wrote:
> Am Thu,  5 Jan 2023 11:48:57 +0530
> schrieb venkata.pyla@toshiba-tsip.com:
> 
> > From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> > 
> > As part of reproducible-build work, the rootfs images generated on
> > same source should be identical between two builds.
> > 
> > In this commit it tries to solve one of the non-reproducible
> > problem
> > i.e. the rootfs file time-stamps generated during build time are
> > not
> > reproducible, it uses one of the solution provided in the debian
> > live-build image project (refer [1]), it fixes by finding all the
> > files/folders that are gernerated newly and set the time-stamp
> > provided by `SOURCE_DATE_EPOCH` environment variable.
> > 
> > [1] 
> > https://salsa.debian.org/live-team/live-build/-/merge_requests/218
> > 
> > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> > ---
> >  meta-isar/conf/local.conf.sample | 10 ++++++++++
> >  meta/classes/image.bbclass       |  9 +++++++++
> >  2 files changed, 19 insertions(+)
> > 
> > diff --git a/meta-isar/conf/local.conf.sample
> > b/meta-isar/conf/local.conf.sample index 57d0620..3c4a473 100644
> > --- a/meta-isar/conf/local.conf.sample
> > +++ b/meta-isar/conf/local.conf.sample
> > @@ -255,3 +255,13 @@ USER_isar[flags] += "clear-text-password"
> >  #CCACHE_TOP_DIR ?= "${TMPDIR}/ccache"
> >  # Enable ccache debug mode
> >  #CCACHE_DEBUG = "1"
> > +
> > +# Uncommnet and add value to it to build images reproducibly
> > +#
> > +# The value for `SOURCE_DATE_EPOCH` should be latest source change
> > time in +# seconds since the Epoch.
> > +# Git repository users can use value from 'git log -1 --
> > pretty=%ct'
> > +# Non git repository users can use value from 'stat -c%Y
> > ChangeLog'
> > +# To know more details about this variable and how to set the
> > value
> > refer below +# 
> > https://reproducible-builds.org/docs/source-date-epoch/
> > +#SOURCE_DATE_EPOCH =
> 
> ${@bb.process.run(git log ...)}
> 
> would be nice here. So once uncommented it will keep moving as people
> commit.

Be careful with this pattern as it bypasses the Bitbake signatures,
leading to non-deterministic signatures errors. We had the same problem
with the ISAR release command. Maybe the immediate expansion operator
works in the local conf, but I did not check that.

It would be better to inject that from the outside, e.g. via KAS or an
environment variable. And in addition, we need to make sure that we
only forward this variable to non-sbuild tasks. The sbuild tasks should
get the SOURCE_DATE_EPOCH from the latest entry in the changelog.
However, for auto-generated changelogs, the date should be the value of
SOURCE_DATE_EPOCH.

Felix

> 
> > diff --git a/meta/classes/image.bbclass
> > b/meta/classes/image.bbclass
> > index 813e1f3..38a9adf 100644
> > --- a/meta/classes/image.bbclass
> > +++ b/meta/classes/image.bbclass
> > @@ -430,6 +430,15 @@ do_rootfs_finalize() {
> >              "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> >  
> >          rm -f "${ROOTFSDIR}/etc/apt/sources-list"
> > +
> > +        # Set same time-stamps to the newly generated file/folders
> > in the
> > +        # rootfs image for the purpose of reproducible builds.
> > +        test ! -z "${SOURCE_DATE_EPOCH}" && \
> > +            find ${ROOTFSDIR} -newermt \
> > +                "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d
> > %H:%M:%S')" \
> > +                -printf "%y %p\n" \
> > +                -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' >
> > ${DEPLOY_DIR_IMAGE}/files.modified_timestamps +
> >  EOSUDO
> 
> I would suggest to at least display a bbwarn if "wc -l" of that file
> exceeds some number ... say 50. I guess if SOURCE_DATE_EPOCH was too
> old, say 01.01.1990 the whole filesystem would be touched which might
> indicate a problem.
> 
> Not sure what a good number would be. We could also check for certain
> files to _not_ be in there for sure.
> 
> I might give that patch a try and see for myself what a too old value
> would do. But right now i will keep going with the expectation that
> it
> would "touch all files without big warning" and the thing might still
> boot but the broken metadata could cause any kind of problems in
> applications that can get confused by that big change.
> 
> Henning
> 
> >  }
> >  addtask rootfs_finalize before do_rootfs after
> > do_rootfs_postprocess
>