Re: [PATCH v2 1/1] image.bbclass: fix non-reproducible file time-stamps inside rootfs

[Henning Schild <henning.schild@siemens.com>]

Am Thu, 5 Jan 2023 10:50:36 +0100
schrieb "Moessbauer, Felix (T CED INW-CN)"
<felix.moessbauer@siemens.com>:

> On Thu, 2023-01-05 at 09:19 +0100, Henning Schild wrote:
> > Am Thu,  5 Jan 2023 11:48:57 +0530
> > schrieb venkata.pyla@toshiba-tsip.com:
> >  
> > > From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> > >
> > > As part of reproducible-build work, the rootfs images generated on
> > > same source should be identical between two builds.
> > >
> > > In this commit it tries to solve one of the non-reproducible
> > > problem
> > > i.e. the rootfs file time-stamps generated during build time are
> > > not
> > > reproducible, it uses one of the solution provided in the debian
> > > live-build image project (refer [1]), it fixes by finding all the
> > > files/folders that are gernerated newly and set the time-stamp
> > > provided by `SOURCE_DATE_EPOCH` environment variable.
> > >
> > > [1]
> > > https://salsa.debian.org/live-team/live-build/-/merge_requests/218
> > >
> > > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> > > ---
> > >  meta-isar/conf/local.conf.sample | 10 ++++++++++
> > >  meta/classes/image.bbclass       |  9 +++++++++
> > >  2 files changed, 19 insertions(+)
> > >
> > > diff --git a/meta-isar/conf/local.conf.sample
> > > b/meta-isar/conf/local.conf.sample index 57d0620..3c4a473 100644
> > > --- a/meta-isar/conf/local.conf.sample
> > > +++ b/meta-isar/conf/local.conf.sample
> > > @@ -255,3 +255,13 @@ USER_isar[flags] += "clear-text-password"
> > >  #CCACHE_TOP_DIR ?= "${TMPDIR}/ccache"
> > >  # Enable ccache debug mode
> > >  #CCACHE_DEBUG = "1"
> > > +
> > > +# Uncommnet and add value to it to build images reproducibly
> > > +#
> > > +# The value for `SOURCE_DATE_EPOCH` should be latest source
> > > change time in +# seconds since the Epoch.
> > > +# Git repository users can use value from 'git log -1 --
> > > pretty=%ct'
> > > +# Non git repository users can use value from 'stat -c%Y
> > > ChangeLog'
> > > +# To know more details about this variable and how to set the
> > > value
> > > refer below +#
> > > https://reproducible-builds.org/docs/source-date-epoch/
> > > +#SOURCE_DATE_EPOCH =  
> >
> > ${@bb.process.run(git log ...)}
> >
> > would be nice here. So once uncommented it will keep moving as
> > people commit.  
> 
> Be careful with this pattern as it bypasses the Bitbake signatures,
> leading to non-deterministic signatures errors. We had the same
> problem with the ISAR release command. Maybe the immediate expansion
> operator works in the local conf, but I did not check that.

The value is only ever used in image.bbclass in the finalize function.
So maybe not a problem to expect with sstate.

> It would be better to inject that from the outside, e.g. via KAS or an
> environment variable.

Yes but we also need to document sane values for it, maybe in the user
doc after all and not the local.conf.sample.

> And in addition, we need to make sure that we
> only forward this variable to non-sbuild tasks. The sbuild tasks
> should get the SOURCE_DATE_EPOCH from the latest entry in the
> changelog. However, for auto-generated changelogs, the date should be
> the value of SOURCE_DATE_EPOCH.

I sure hope this one does not go into the env of the package building,
but it should be double checked.

For auto-generated changelogs we have another solution based on last
debian changelog entry or 1.1.1970 if there is none.
https://github.com/ilbers/isar/commit/53d315fdd3f3dcf70ac3f257d9431ba522e86eb4

Henning

> Felix
> 
> >  
> > > diff --git a/meta/classes/image.bbclass
> > > b/meta/classes/image.bbclass
> > > index 813e1f3..38a9adf 100644
> > > --- a/meta/classes/image.bbclass
> > > +++ b/meta/classes/image.bbclass
> > > @@ -430,6 +430,15 @@ do_rootfs_finalize() {
> > >              "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> > >
> > >          rm -f "${ROOTFSDIR}/etc/apt/sources-list"
> > > +
> > > +        # Set same time-stamps to the newly generated
> > > file/folders in the
> > > +        # rootfs image for the purpose of reproducible builds.
> > > +        test ! -z "${SOURCE_DATE_EPOCH}" && \
> > > +            find ${ROOTFSDIR} -newermt \
> > > +                "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d
> > > %H:%M:%S')" \
> > > +                -printf "%y %p\n" \
> > > +                -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' >
> > > ${DEPLOY_DIR_IMAGE}/files.modified_timestamps +
> > >  EOSUDO  
> >
> > I would suggest to at least display a bbwarn if "wc -l" of that file
> > exceeds some number ... say 50. I guess if SOURCE_DATE_EPOCH was too
> > old, say 01.01.1990 the whole filesystem would be touched which
> > might indicate a problem.
> >
> > Not sure what a good number would be. We could also check for
> > certain files to _not_ be in there for sure.
> >
> > I might give that patch a try and see for myself what a too old
> > value would do. But right now i will keep going with the
> > expectation that it
> > would "touch all files without big warning" and the thing might
> > still boot but the broken metadata could cause any kind of problems
> > in applications that can get confused by that big change.
> >
> > Henning
> >  
> > >  }
> > >  addtask rootfs_finalize before do_rootfs after
> > > do_rootfs_postprocess  
> >  
>