[PATCH v2 00/10] Make rootfs build reproducible

[PATCH v2 00/10] Make rootfs build reproducible

[Felix Moessbauer]

This series finally makes the rootfs generation bit-reproducible
from debian bullseye on. Parts of it have already been sent
as individual patches. However, image reproducibility can only
be achived once all parts are reproducible itself. By that,
these patches are included in this series as well.

With this series, the following parts are now fully reproducible.
This has been tested on the isar-image-base target.

- custom initramfs (creation and updates)
- debian initramfs (only updates are relevant)
- custom kernel (debian kernel is reproducible itself)
- rootfs itself
- tar file generation (<image>.tar)
- ext4 generation (only from bookworm on, more tests needed)

Other parts that are still not reproducible are:

- WIC (should be solved in OE already)
- containers (untested yet)

Changes since v1:

- dropped patch "deb_add_changelog: use SOURCE_DATE_EPOCH"
- fixed typo in "generate deterministic clear-text password hash"
- added comment about why SOURCE_DATE_EPOCH must only be set for
  image rootfs but not for other rootfs'.

Best regards,
Felix Moessbauer
Siemens AG

Felix Moessbauer (9):
  fix rebuild of rootfs_finalize task
  rootfs postprocess: clean python cache
  remove non-portable ldconfig aux-cache
  generate deterministic clear-text password hash
  update debian initramfs in deterministic mode
  create custom initramfs in deterministic mode
  make deb_add_changelog idempotent
  deb_add_changelog: set timestamp to valid epoch
  make custom linux-image bit-by-bit reproducible

venkata pyla (1):
  image.bbclass: fix non-reproducible file time-stamps inside rootfs

 meta-isar/conf/local.conf.sample              | 10 ++++++++
 meta/classes/debianize.bbclass                | 20 +++++++++------
 meta/classes/image-account-extension.bbclass  | 10 +++++++-
 meta/classes/image.bbclass                    | 25 +++++++++++++++++--
 meta/classes/initramfs.bbclass                |  5 ++++
 meta/classes/rootfs.bbclass                   | 13 ++++++++++
 .../linux/files/debian/isar/build.tmpl        |  1 +
 .../linux/files/debian/rules.tmpl             | 14 ++++++++++-
 meta/recipes-kernel/linux/linux-custom.inc    |  2 ++
 9 files changed, 89 insertions(+), 11 deletions(-)

-- 
2.34.1

[PATCH v2 01/10] fix rebuild of rootfs_finalize task

[Felix Moessbauer]

The rootfs_finalize task currently cannot be re-executed, as it moves
the sources-list into bootstrap.list. As this only has to be done once,
it is not required on subsequent executions. To fix the rebuild issue,
we simply ignore the return code of the mv statement.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/image.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 629a0c1..6f0607e 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -425,7 +425,7 @@ do_rootfs_finalize() {
         rm -f "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar"
 
         mv "${ROOTFSDIR}/etc/apt/sources-list" \
-            "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
+            "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" || true
 
         rm -f "${ROOTFSDIR}/etc/apt/sources-list"
 EOSUDO
-- 
2.34.1

Re: [PATCH v2 01/10] fix rebuild of rootfs_finalize task

[Uladzimir Bely]

In the email from Thursday, 12 January 2023 08:56:10 +03 user Felix Moessbauer 
wrote:
> [PATCH v2 01/10] fix rebuild of rootfs_finalize task

Again, we had 4 different solutions for this (as single patches)

Felix: [PATCH] fix rebuild of rootfs_finalize task
Roberto: [PATCH v4] image: make sure do_rootfs_finalize can run multiple 
times, v4
Roberto: [PATCH] image class bugfix: interuption does not break the rebuild
Henning: [PATCH] image: make sure do_rootfs_finalize can run multiple times

Now, the first one comes as part of series... I think, we should pick 
Henning's solution instead, since it acts better ("check first than do stuff" 
instead of "try stuff and 'true' if fails), and drop the alternative patch 
from the series.

Re: [PATCH v2 01/10] fix rebuild of rootfs_finalize task

[Roberto A. Foglietta]

On Sat, 14 Jan 2023 at 21:48, Uladzimir Bely <ubely@ilbers.de> wrote:
>
> In the email from Thursday, 12 January 2023 08:56:10 +03 user Felix Moessbauer
> wrote:
> > [PATCH v2 01/10] fix rebuild of rootfs_finalize task
>
> Again, we had 4 different solutions for this (as single patches)
>
> Felix: [PATCH] fix rebuild of rootfs_finalize task
> Roberto: [PATCH v4] image: make sure do_rootfs_finalize can run multiple
> times, v4

Hi Felix,

if you do not like to reorganize the code above you can just use -f
like in my patch

+        mv -f "${aptdir}/sources-list" \
+            "${aptdir}/sources.list.d/bootstrap.list" 2>/dev/null || :

Best regards, R-

Re: [PATCH v2 01/10] fix rebuild of rootfs_finalize task

[Roberto A. Foglietta]

On Sat, 14 Jan 2023 at 23:16, Roberto A. Foglietta
<roberto.foglietta@gmail.com> wrote:
>
> On Sat, 14 Jan 2023 at 21:48, Uladzimir Bely <ubely@ilbers.de> wrote:
> >
> > In the email from Thursday, 12 January 2023 08:56:10 +03 user Felix Moessbauer
> > wrote:
> > > [PATCH v2 01/10] fix rebuild of rootfs_finalize task
> >
> > Again, we had 4 different solutions for this (as single patches)

Now, 5! LOL

v5-0001-image-make-sure-do_rootfs_finalize-can-run-multip.patch

> >
> > Felix: [PATCH] fix rebuild of rootfs_finalize task
> > Roberto: [PATCH v4] image: make sure do_rootfs_finalize can run multiple
> > times, v4
>
> Hi Felix,
>
> if you do not like to reorganize the code above you can just use -f
> like in my patch
>
> +        mv -f "${aptdir}/sources-list" \
> +            "${aptdir}/sources.list.d/bootstrap.list" 2>/dev/null || :
>

We can also use change directory for re-organise the code above the fix

+        cd "${ROOTFSDIR}/etc/apt"
+        rm -f "apt.conf.d/50isar"
+        rm -f "preferences.d/isar-apt"
+        rm -f "sources.list.d/isar-apt.list"
+        rm -f "sources.list.d/base-apt.list"
+        mv -f "sources-list" "sources.list.d/bootstrap.list" 2>/dev/null ||:
+        cd -

cd - is not necessary at the end of SUDO but in future those add code
might not notice that the working path is changed and got crazy or
broken, so it is nice to have.

Best

Re: [PATCH v2 01/10] fix rebuild of rootfs_finalize task

[Moessbauer, Felix]

On Sat, 2023-01-14 at 23:47 +0300, Uladzimir Bely wrote:
> In the email from Thursday, 12 January 2023 08:56:10 +03 user Felix
> Moessbauer 
> wrote:
> > [PATCH v2 01/10] fix rebuild of rootfs_finalize task
> 
> Again, we had 4 different solutions for this (as single patches)

I'm fully aware of that. In the end it is the maintainers decision
which patch to integrate. We just need the fix here as otherwise the
whole reproducibility story cannot be tested.

I'm perfectly fine with taking Hennings patch. It anyways looks like we
will need a v3 of this series. Then, I can either add Hennings patch or
simply rebase against next in case one of the patches will already be
integrated then.

PS: Please do not discuss details of this particular patch as part of
this series. It is really only a necessary enabler, but has been sent
as dedicated patches already. Please discuss it on the original patch
series.

Felix

> 
> Felix: [PATCH] fix rebuild of rootfs_finalize task
> Roberto: [PATCH v4] image: make sure do_rootfs_finalize can run
> multiple 
> times, v4
> Roberto: [PATCH] image class bugfix: interuption does not break the
> rebuild
> Henning: [PATCH] image: make sure do_rootfs_finalize can run multiple
> times
> 
> Now, the first one comes as part of series... I think, we should pick
> Henning's solution instead, since it acts better ("check first than
> do stuff" 
> instead of "try stuff and 'true' if fails), and drop the alternative
> patch 
> from the series.
> 
> 

Re: [PATCH v2 01/10] fix rebuild of rootfs_finalize task

[Henning Schild]

Am Sun, 15 Jan 2023 14:31:46 +0100
schrieb "Moessbauer, Felix (T CED INW-CN)"
<felix.moessbauer@siemens.com>:

> On Sat, 2023-01-14 at 23:47 +0300, Uladzimir Bely wrote:
> > In the email from Thursday, 12 January 2023 08:56:10 +03 user Felix
> > Moessbauer
> > wrote:  
> > > [PATCH v2 01/10] fix rebuild of rootfs_finalize task  
> >
> > Again, we had 4 different solutions for this (as single patches)  
> 
> I'm fully aware of that. In the end it is the maintainers decision
> which patch to integrate. We just need the fix here as otherwise the
> whole reproducibility story cannot be tested.

That was all a mess. My patch was first and later people wrote
conflicting patches without saying so. We can do that but it would help
the maintainer a lot if the cover-letters would say which other patches
become obsolete, or if one comments on those other patches.

I also do not care which one gets picked. Just try to point out
conflicts to help the maintainer.

Henning

> I'm perfectly fine with taking Hennings patch. It anyways looks like
> we will need a v3 of this series. Then, I can either add Hennings
> patch or simply rebase against next in case one of the patches will
> already be integrated then.
> 
> PS: Please do not discuss details of this particular patch as part of
> this series. It is really only a necessary enabler, but has been sent
> as dedicated patches already. Please discuss it on the original patch
> series.
> 
> Felix
> 
> >
> > Felix: [PATCH] fix rebuild of rootfs_finalize task
> > Roberto: [PATCH v4] image: make sure do_rootfs_finalize can run
> > multiple
> > times, v4
> > Roberto: [PATCH] image class bugfix: interuption does not break the
> > rebuild
> > Henning: [PATCH] image: make sure do_rootfs_finalize can run
> > multiple times
> >
> > Now, the first one comes as part of series... I think, we should
> > pick Henning's solution instead, since it acts better ("check first
> > than do stuff"
> > instead of "try stuff and 'true' if fails), and drop the alternative
> > patch
> > from the series.
> >
> >  
> 

[PATCH v2 02/10] image.bbclass: fix non-reproducible file time-stamps inside rootfs

[Felix Moessbauer]

From: venkata pyla <venkata.pyla@toshiba-tsip.com>

As part of reproducible-build work, the rootfs images generated on same
source should be identical between two builds.

In this commit it tries to solve one of the non-reproducible problem
i.e. the rootfs file time-stamps generated during build time are not
reproducible, it uses one of the solution provided in the debian
live-build image project (refer [1]), it fixes by finding all the
files/folders that are gernerated newly and set the time-stamp provided
by `SOURCE_DATE_EPOCH` environment variable.

[1] https://salsa.debian.org/live-team/live-build/-/merge_requests/218

Acked-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 meta-isar/conf/local.conf.sample | 10 ++++++++++
 meta/classes/image.bbclass       | 10 ++++++++++
 2 files changed, 20 insertions(+)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index e1cd66e..6208623 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -248,3 +248,13 @@ USER_isar[flags] += "clear-text-password"
 #CCACHE_TOP_DIR ?= "${TMPDIR}/ccache"
 # Enable ccache debug mode
 #CCACHE_DEBUG = "1"
+
+# Uncommnet and add value to it to build images reproducibly
+#
+# The value for `SOURCE_DATE_EPOCH` should be latest source change time in
+# seconds since the Epoch.
+# Git repository users can use value from 'git log -1 --pretty=%ct'
+# Non git repository users can use value from 'stat -c%Y ChangeLog'
+# To know more details about this variable and how to set the value refer below
+# https://reproducible-builds.org/docs/source-date-epoch/
+#SOURCE_DATE_EPOCH =
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 6f0607e..519a2e5 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -429,6 +429,16 @@ do_rootfs_finalize() {
 
         rm -f "${ROOTFSDIR}/etc/apt/sources-list"
 EOSUDO
+
+    # Set same time-stamps to the newly generated file/folders in the
+    # rootfs image for the purpose of reproducible builds.
+    test ! -z "${SOURCE_DATE_EPOCH}" && \
+        sudo find ${ROOTFSDIR} -newermt \
+            "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \
+            -printf "%y %p\n" \
+            -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' > ${DEPLOY_DIR_IMAGE}/files.modified_timestamps && \
+            bbwarn "$(grep ^f ${DEPLOY_DIR_IMAGE}/files.modified_timestamps) \nModified above file timestamps to build image reproducibly"
+
 }
 addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess
 
-- 
2.34.1

Re: [PATCH v2 02/10] image.bbclass: fix non-reproducible file time-stamps inside rootfs

[Uladzimir Bely]

In the email from Thursday, 12 January 2023 08:56:11 +03 user Felix Moessbauer 
wrote:
> From: venkata pyla <venkata.pyla@toshiba-tsip.com>
> 
> As part of reproducible-build work, the rootfs images generated on same
> source should be identical between two builds.
> 
> In this commit it tries to solve one of the non-reproducible problem
> i.e. the rootfs file time-stamps generated during build time are not
> reproducible, it uses one of the solution provided in the debian
> live-build image project (refer [1]), it fixes by finding all the
> files/folders that are gernerated newly and set the time-stamp provided
> by `SOURCE_DATE_EPOCH` environment variable.
> 
> [1] https://salsa.debian.org/live-team/live-build/-/merge_requests/218
> 
> Acked-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  meta-isar/conf/local.conf.sample | 10 ++++++++++
>  meta/classes/image.bbclass       | 10 ++++++++++
>  2 files changed, 20 insertions(+)
> 
> diff --git a/meta-isar/conf/local.conf.sample
> b/meta-isar/conf/local.conf.sample index e1cd66e..6208623 100644
> --- a/meta-isar/conf/local.conf.sample
> +++ b/meta-isar/conf/local.conf.sample
> @@ -248,3 +248,13 @@ USER_isar[flags] += "clear-text-password"
>  #CCACHE_TOP_DIR ?= "${TMPDIR}/ccache"
>  # Enable ccache debug mode
>  #CCACHE_DEBUG = "1"
> +
> +# Uncommnet and add value to it to build images reproducibly
> +#
> +# The value for `SOURCE_DATE_EPOCH` should be latest source change time in
> +# seconds since the Epoch.
> +# Git repository users can use value from 'git log -1 --pretty=%ct'
> +# Non git repository users can use value from 'stat -c%Y ChangeLog'
> +# To know more details about this variable and how to set the value refer
> below +# https://reproducible-builds.org/docs/source-date-epoch/
> +#SOURCE_DATE_EPOCH =
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 6f0607e..519a2e5 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -429,6 +429,16 @@ do_rootfs_finalize() {
> 
>          rm -f "${ROOTFSDIR}/etc/apt/sources-list"
>  EOSUDO
> +
> +    # Set same time-stamps to the newly generated file/folders in the
> +    # rootfs image for the purpose of reproducible builds.
> +    test ! -z "${SOURCE_DATE_EPOCH}" && \

This failed in CI with empty variable with the following log:

DEBUG: Executing shell function do_rootfs_finalize
+ do_rootfs_finalize
+ sudo -s
+ test ! -z 
+ bb_sh_exit_handler
+ ret=1
+ [ 1 != 0 ]
+ echo WARNING: exit code 1 from a shell command.
+ exit 1
WARNING: exit code 1 from a shell command.
ERROR: ExecutionError('/workspace/build/isar_ub_devel_fast/318/build/tmp/work/
debian-buster-armhf/isar-image-base-stm32mp15x/1.0-r0/temp/
run.do_rootfs_finalize.2265252', 1, None, None)

> +        sudo find ${ROOTFSDIR} -newermt \
> +            "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \
> +            -printf "%y %p\n" \
> +            -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';' >
> ${DEPLOY_DIR_IMAGE}/files.modified_timestamps && \ +            bbwarn
> "$(grep ^f ${DEPLOY_DIR_IMAGE}/files.modified_timestamps) \nModified above
> file timestamps to build image reproducibly" +
>  }
>  addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess

Re: [PATCH v2 02/10] image.bbclass: fix non-reproducible file time-stamps inside rootfs

[Roberto A. Foglietta]

On Sat, 14 Jan 2023 at 21:26, Uladzimir Bely <ubely@ilbers.de> wrote:

>
> This failed in CI with empty variable with the following log:

Hi Uladzimir, try again using this

https://patchwork.isar-build.org/project/isar/patch/20230113174737.281104-1-roberto.foglietta@linuxteam.org/

so not focus on my patch (goor or less good, after all it is just a
suggestion that Felix may elaborate in the next future) but just try
to see if it solves the problem. I faced some failure that I solved
with the mods in the image.bbclass

Best regards, R-

Re: [PATCH v2 02/10] image.bbclass: fix non-reproducible file time-stamps inside rootfs

[Uladzimir Bely]

In the email from Saturday, 14 January 2023 23:31:47 +03 user Roberto A. 
Foglietta wrote:
> On Sat, 14 Jan 2023 at 21:26, Uladzimir Bely <ubely@ilbers.de> wrote:
> > This failed in CI with empty variable with the following log:
> Hi Uladzimir, try again using this
> 
> https://patchwork.isar-build.org/project/isar/patch/20230113174737.281104-1-> roberto.foglietta@linuxteam.org/
> 

Yes, this check with "if" should fix the issue. 

> so not focus on my patch (goor or less good, after all it is just a
> suggestion that Felix may elaborate in the next future) but just try
> to see if it solves the problem. I faced some failure that I solved
> with the mods in the image.bbclass

Let's wait v3 with the fix.

> 
> Best regards, R-

Re: [PATCH v2 02/10] image.bbclass: fix non-reproducible file time-stamps inside rootfs

[Moessbauer, Felix]

On Sat, 2023-01-14 at 23:39 +0300, Uladzimir Bely wrote:
> In the email from Saturday, 14 January 2023 23:31:47 +03 user Roberto
> A. 
> Foglietta wrote:
> > On Sat, 14 Jan 2023 at 21:26, Uladzimir Bely <ubely@ilbers.de>
> > wrote:
> > > This failed in CI with empty variable with the following log:
> > Hi Uladzimir, try again using this
> > 
> > https://patchwork.isar-build.org/project/isar/patch/20230113174737.281104-1
> > -> roberto.foglietta@linuxteam.org/
> > 
> 
> Yes, this check with "if" should fix the issue. 

Agree. But I don't want to mix up the ownership of the patch too much
and Roberto's solution changes more of the logic. By that, I will only
fix the build failure (by adding the if) and keep the original author
(Venkata).

@Roberto: Once this is integrated into next, feel free to send your
improvement (please ONLY that) as a new patch on top. This also helps
to review things.

Felix

> 
> > so not focus on my patch (goor or less good, after all it is just a
> > suggestion that Felix may elaborate in the next future) but just
> > try
> > to see if it solves the problem. I faced some failure that I solved
> > with the mods in the image.bbclass
> 
> Let's wait v3 with the fix.
> 
> > 
> > Best regards, R-
> 
> 
> 
> 

Re: [PATCH v2 02/10] image.bbclass: fix non-reproducible file time-stamps inside rootfs

[Roberto A. Foglietta]

On Sun, 15 Jan 2023 at 14:43, Moessbauer, Felix
<felix.moessbauer@siemens.com> wrote:
>
> On Sat, 2023-01-14 at 23:39 +0300, Uladzimir Bely wrote:
> > In the email from Saturday, 14 January 2023 23:31:47 +03 user Roberto
> > A.
> > Foglietta wrote:
> > > On Sat, 14 Jan 2023 at 21:26, Uladzimir Bely <ubely@ilbers.de>
> > > wrote:
> > > > This failed in CI with empty variable with the following log:
> > > Hi Uladzimir, try again using this
> > >
> > > https://patchwork.isar-build.org/project/isar/patch/20230113174737.281104-1
> > > -> roberto.foglietta@linuxteam.org/
> > >
> >
> > Yes, this check with "if" should fix the issue.
>

>
> @Roberto: Once this is integrated into next, feel free to send your
> improvement (please ONLY that) as a new patch on top. This also helps
> to review things.

NEW - suggested changes for reproducibility patchset v6

In my opinion increases greatly the way in which the effect is applied.
The warnings have just two lines, very informative and the 2nd line
appears only when there are some files changed.

Best regards, R-

[PATCH v2 03/10] rootfs postprocess: clean python cache

[Felix Moessbauer]

When calling python scripts, python automatically creates cache files
to speedup future invocations of the same sources. This often happens
in postinst scripts, that directly run in the image chroot. The
created debian packages do not ship these files, as the debheper
scripts remove them before installing.

For the rootfs part, we manually have to do it to also not
include these in the final image. This patch implements this logic in
a custom cleanup postprocess step. As there might be situations where
shipping of a subset of the caches is desireable (e.g. readonly rootfs
images), we add support to control this logic using ROOTFS_FEATURES.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/image.bbclass  | 2 +-
 meta/classes/rootfs.bbclass | 6 ++++++
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 519a2e5..b86a428 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -80,7 +80,7 @@ image_do_mounts() {
 }
 
 ROOTFSDIR = "${IMAGE_ROOTFS}"
-ROOTFS_FEATURES += "clean-package-cache generate-manifest export-dpkg-status clean-log-files clean-debconf-cache"
+ROOTFS_FEATURES += "clean-package-cache clean-pycache generate-manifest export-dpkg-status clean-log-files clean-debconf-cache"
 ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
 ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
 ROOTFS_DPKGSTATUS_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 786682d..325e7ae 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -252,6 +252,12 @@ rootfs_postprocess_clean_debconf_cache() {
     sudo rm -rf "${ROOTFSDIR}/var/cache/debconf/"*
 }
 
+ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-pycache', 'rootfs_postprocess_clean_pycache', '', d)}"
+rootfs_postprocess_clean_pycache() {
+    sudo find ${ROOTFSDIR}/usr -type f -name '*.pyc'       -delete -print
+    sudo find ${ROOTFSDIR}/usr -type d -name '__pycache__' -delete -print
+}
+
 ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'generate-manifest', 'rootfs_generate_manifest', '', d)}"
 rootfs_generate_manifest () {
     mkdir -p ${ROOTFS_MANIFEST_DEPLOY_DIR}
-- 
2.34.1

[PATCH v2 04/10] remove non-portable ldconfig aux-cache

[Felix Moessbauer]

This patch removes the ldconfig aux-cache from the final rootfs.
The cache is both not portable across file systems, as well as non
reproducible.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/rootfs.bbclass | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 325e7ae..226fa8b 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -258,6 +258,13 @@ rootfs_postprocess_clean_pycache() {
     sudo find ${ROOTFSDIR}/usr -type d -name '__pycache__' -delete -print
 }
 
+ROOTFS_POSTPROCESS_COMMAND += "rootfs_postprocess_clean_ldconfig_cache"
+rootfs_postprocess_clean_ldconfig_cache() {
+    # the ldconfig aux-cache is not portable and breaks reproducability
+    # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=845034#49
+    sudo rm -f ${ROOTFSDIR}/var/cache/ldconfig/aux-cache
+}
+
 ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'generate-manifest', 'rootfs_generate_manifest', '', d)}"
 rootfs_generate_manifest () {
     mkdir -p ${ROOTFS_MANIFEST_DEPLOY_DIR}
-- 
2.34.1

[PATCH v2 05/10] generate deterministic clear-text password hash

[Felix Moessbauer]

This patch changes how we derive the hashed password of a user that is
created using the clear-text-password flag. Previously, the clear-text
password was directly input into chpasswd. However, chpasswd internally
creates a 16-character random salt. This breaks the reproducability.

Instead of letting chpasswd create the hashed password string, we now
create it manually by deriving the salt from the SOURCE_DATE_EPOCH
variable. This is technically done using the host openssl tool. As
openssl is a transitive dependency of sbuild, we do not need to add
it explicitly to the host-tools.

In case SOURCE_DATE_EPOCH is not set, chpasswd is used
directly to create the salt.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/image-account-extension.bbclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image-account-extension.bbclass b/meta/classes/image-account-extension.bbclass
index 70950a7..bb173b1 100644
--- a/meta/classes/image-account-extension.bbclass
+++ b/meta/classes/image-account-extension.bbclass
@@ -253,7 +253,15 @@ image_postprocess_accounts() {
         if [ -n "$password" -o "${flags}" != "${flags%*,allow-empty-password,*}" ]; then
             chpasswd_args="-e"
             if [ "${flags}" != "${flags%*,clear-text-password,*}" ]; then
-                chpasswd_args=""
+                # chpasswd adds a random salt when running against a clear-text password.
+                # For reproducible images, we manually generate the password and use the
+                # SOURCE_DATE_EPOCH to generate the salt in a deterministic way.
+                if [ -z "${SOURCE_DATE_EPOCH}"]; then
+                    chpasswd_args=""
+                else
+                    salt="$(echo "${SOURCE_DATE_EPOCH}" | sha256sum -z | cut -c 1-15)"
+                    password="$(openssl passwd -6 -salt $salt "$password")"
+                fi
             fi
             printf '%s:%s' "$name" "$password" | sudo chroot '${ROOTFSDIR}' \
                 /usr/sbin/chpasswd $chpasswd_args
-- 
2.34.1

[PATCH v2 06/10] update debian initramfs in deterministic mode

[Felix Moessbauer]

This patch exports the SOURCE_DATE_EPOCH variable in the image install
task. By that, update-initramfs is switched into reproducible mode.
Before this patch, each trigger of update-initramfs created a new
non-deterministic version of the initramfs.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/image.bbclass | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index b86a428..063b9a3 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -304,6 +304,17 @@ python() {
 }
 
 
+# make generation of initramfs reproducible
+# note: this function is shared across multiple rootfs, but we only want to make the
+#       image rootfs reproducible. Otherwise changes of SOURCE_DATE_EPOCH would
+#       invalidate the SSTATE entries for most packages, even if they don't use the
+#       global SOURCE_DATE_EPOCH variable.
+rootfs_install_pkgs_install_prepend() {
+    if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
+        export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
+    fi
+}
+
 # here we call a command that should describe your whole build system,
 # this could be "git describe" or something similar.
 # set ISAR_RELEASE_CMD to customize, or override do_mark_rootfs to do something
-- 
2.34.1

[PATCH v2 07/10] create custom initramfs in deterministic mode

[Felix Moessbauer]

This patch enables the deterministic mode of update-initramfs for custom
initramfs versions, in case SOURCE_DATE_EPOCH is defined.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/initramfs.bbclass | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/classes/initramfs.bbclass b/meta/classes/initramfs.bbclass
index 2cec85d..db28334 100644
--- a/meta/classes/initramfs.bbclass
+++ b/meta/classes/initramfs.bbclass
@@ -32,6 +32,11 @@ do_generate_initramfs() {
     rootfs_do_mounts
     rootfs_do_qemu
 
+    # generate reproducible initrd if requested
+    if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
+        export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
+    fi
+
     sudo -E chroot "${INITRAMFS_ROOTFS}" \
         update-initramfs -u -v
 
-- 
2.34.1

[PATCH v2 08/10] make deb_add_changelog idempotent

[Felix Moessbauer]

Previously, the deb_add_changelog function considered an auto-generated
changelog as a base to add changes on top. This behavior is not
idempontent on subsequent invocations of the function (e.g. on partial
rebuilds). This lead to both reproducability issues, as well as unclean
changelog files having multiple "generated by ISAR" entries.

This patch changes this implementation in a way to always create a
(possibly empty) orig changelog on the first invocation. On subequent
invocations, the orig changelog is only considered as provided by the
user, if it is not empty.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/debianize.bbclass | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/meta/classes/debianize.bbclass b/meta/classes/debianize.bbclass
index d125256..ca7b520 100644
--- a/meta/classes/debianize.bbclass
+++ b/meta/classes/debianize.bbclass
@@ -19,11 +19,14 @@ deb_add_changelog() {
 		if [ ! -f ${WORKDIR}/changelog.orig ]; then
 			cp ${S}/debian/changelog ${WORKDIR}/changelog.orig
 		fi
-		orig_version=$(dpkg-parsechangelog -l ${WORKDIR}/changelog.orig -S Version)
-		changelog_v=$(echo "${changelog_v}" | sed 's/<orig-version>/'${orig_version}'/')
-		orig_date=$(dpkg-parsechangelog -l ${WORKDIR}/changelog.orig -S Date)
-		orig_seconds=$(date --date="${orig_date}" +'%s')
-		timestamp=$(expr ${orig_seconds} + 42)
+		# we have a non auto-generated original changelog
+		if [ -s ${WORKDIR}/changelog.orig ]; then
+			orig_version=$(dpkg-parsechangelog -l ${WORKDIR}/changelog.orig -S Version)
+			changelog_v=$(echo "${changelog_v}" | sed 's/<orig-version>/'${orig_version}'/')
+			orig_date=$(dpkg-parsechangelog -l ${WORKDIR}/changelog.orig -S Date)
+			orig_seconds=$(date --date="${orig_date}" +'%s')
+			timestamp=$(expr ${orig_seconds} + 42)
+		fi
 	fi
 
 	date=$(LANG=C date -R -d @${timestamp})
@@ -34,7 +37,10 @@ ${PN} (${changelog_v}) UNRELEASED; urgency=low
 
  -- ${MAINTAINER}  ${date}
 EOF
-	if [ -f ${WORKDIR}/changelog.orig ]; then
+	# ensure that we always start with the orig version of the
+	# changelog on repeated invocations (e.g. on partial rebuilds)
+	touch ${WORKDIR}/changelog.orig
+	if [ -s ${WORKDIR}/changelog.orig ]; then
 		# prepend our entry to the original changelog
 		echo >> ${S}/debian/changelog
 		cat ${WORKDIR}/changelog.orig >> ${S}/debian/changelog
-- 
2.34.1

[PATCH v2 09/10] deb_add_changelog: set timestamp to valid epoch

[Felix Moessbauer]

A changelog date of 0 (unix timestamp) is not considered a valid
timestamp for the SOURCE_DATE_EPOCH. By that, the debhelper scripts
set the SOURCE_DATE_EPOCH variable to the current time of the build,
breaking reproducability. By that, we get an inconsistency between the
debian changelog timestamp and the timestamp that the build tools encode
into the binary and the file timestamps.

Without having support to control the SOURCE_DATE_EPOCH variable
externally via bitbake, this always led to non-reproducible packages.
To fix this, we simply set the default timestamp to 1h later.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/debianize.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/debianize.bbclass b/meta/classes/debianize.bbclass
index ca7b520..a6694a0 100644
--- a/meta/classes/debianize.bbclass
+++ b/meta/classes/debianize.bbclass
@@ -14,7 +14,7 @@ MAINTAINER ??= "Unknown maintainer <unknown@example.com>"
 
 deb_add_changelog() {
 	changelog_v="${CHANGELOG_V}"
-	timestamp=0
+	timestamp=3600
 	if [ -f ${S}/debian/changelog ]; then
 		if [ ! -f ${WORKDIR}/changelog.orig ]; then
 			cp ${S}/debian/changelog ${WORKDIR}/changelog.orig
-- 
2.34.1

[PATCH v2 10/10] make custom linux-image bit-by-bit reproducible

[Felix Moessbauer]

This patch makes the build of custom linux kernels bit-by-bit
reproducible. By that, we can remove the dh_strip_nondeterminism step,
which significantly reduces the kernel build time.

The implementation is similar to how upstream debian builds their kernel
images and extracts all information from the changelog. As the
DISTRIBUTOR field is not part of the changelog, we inject it via a bb
variable which is defaulted to ISAR.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 .../linux/files/debian/isar/build.tmpl             |  1 +
 meta/recipes-kernel/linux/files/debian/rules.tmpl  | 14 +++++++++++++-
 meta/recipes-kernel/linux/linux-custom.inc         |  2 ++
 3 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
index 94cfbe0..e7e0479 100644
--- a/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/isar/build.tmpl
@@ -34,6 +34,7 @@ print_settings() {
 # ---------------
 # ARCH=${ARCH}
 # CROSS_COMPILE=${CROSS_COMPILE}
+# KBUILD_BUILD_TIMESTAMP=${KBUILD_BUILD_TIMESTAMP}
 EOF
 }
 
diff --git a/meta/recipes-kernel/linux/files/debian/rules.tmpl b/meta/recipes-kernel/linux/files/debian/rules.tmpl
index 8063c49..e8ae3da 100755
--- a/meta/recipes-kernel/linux/files/debian/rules.tmpl
+++ b/meta/recipes-kernel/linux/files/debian/rules.tmpl
@@ -2,6 +2,11 @@
 
 CROSS_COMPILE:=$(DEB_HOST_GNU_TYPE)-
 
+MAINTAINER := $(shell sed -ne 's,^Maintainer: .[^<]*<\([^>]*\)>,\1,p' debian/control)
+DISTRIBUTOR := ${DISTRIBUTOR}
+SOURCE_DATE := $(shell dpkg-parsechangelog -SDate)
+SOURCE_DATE_UTC_ISO := $(shell date -u -d '$(SOURCE_DATE)' +%Y-%m-%d)
+
 O:=$(CURDIR)/${KERNEL_BUILD_DIR}
 S:=$(CURDIR)
 deb_top_dir:=$(S)/debian
@@ -14,7 +19,11 @@ isar_env=$(strip \
 	export MAKE='$(MAKE)' && \
 	export O='${O}' && \
 	export S='${S}' && \
-	export CURDIR='$(CURDIR)' \
+	export CURDIR='$(CURDIR)' && \
+	export KBUILD_BUILD_TIMESTAMP='$(SOURCE_DATE)' && \
+	export KBUILD_BUILD_VERSION_TIMESTAMP='$(DISTRIBUTOR) $(DEB_VERSION_UPSTREAM) ($(SOURCE_DATE_UTC_ISO))' && \
+	export KBUILD_BUILD_USER='$(word 1,$(subst @, ,$(MAINTAINER)))' && \
+	export KBUILD_BUILD_HOST='$(word 2,$(subst @, ,$(MAINTAINER)))' \
 )
 
 %:
@@ -35,5 +44,8 @@ override_dh_auto_install:
 override_dh_auto_test:
 	true
 
+override_dh_strip_nondeterminism:
+	true
+
 override_dh_strip:
 	unset DEB_HOST_GNU_TYPE && dh_strip -Xvmlinu --no-automatic-dbgsym
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 447d4e8..6c539c0 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -12,6 +12,7 @@
 CHANGELOG_V = "${PV}+${PR}"
 DESCRIPTION ?= "Custom kernel"
 MAINTAINER ?= "isar-users <isar-users@googlegroups.com>"
+DISTRIBUTOR ?= "ISAR"
 
 KBUILD_DEPENDS ?= "build-essential:native, \
                    libelf-dev:native, \
@@ -79,6 +80,7 @@ TEMPLATE_VARS += "                \
     KERNEL_NAME_PROVIDED          \
     KERNEL_CONFIG_FRAGMENTS       \
     KCFLAGS                       \
+    DISTRIBUTOR                   \
 "
 
 inherit dpkg
-- 
2.34.1

Re: [PATCH v2 00/10] Make rootfs build reproducible

[Henning Schild]

Good effort, Thanks. I will not review again.

Henning

Am Thu, 12 Jan 2023 05:56:09 +0000
schrieb Felix Moessbauer <felix.moessbauer@siemens.com>:

> This series finally makes the rootfs generation bit-reproducible
> from debian bullseye on. Parts of it have already been sent
> as individual patches. However, image reproducibility can only
> be achived once all parts are reproducible itself. By that,
> these patches are included in this series as well.
> 
> With this series, the following parts are now fully reproducible.
> This has been tested on the isar-image-base target.
> 
> - custom initramfs (creation and updates)
> - debian initramfs (only updates are relevant)
> - custom kernel (debian kernel is reproducible itself)
> - rootfs itself
> - tar file generation (<image>.tar)
> - ext4 generation (only from bookworm on, more tests needed)
> 
> Other parts that are still not reproducible are:
> 
> - WIC (should be solved in OE already)
> - containers (untested yet)
> 
> Changes since v1:
> 
> - dropped patch "deb_add_changelog: use SOURCE_DATE_EPOCH"
> - fixed typo in "generate deterministic clear-text password hash"
> - added comment about why SOURCE_DATE_EPOCH must only be set for
>   image rootfs but not for other rootfs'.
> 
> Best regards,
> Felix Moessbauer
> Siemens AG
> 
> Felix Moessbauer (9):
>   fix rebuild of rootfs_finalize task
>   rootfs postprocess: clean python cache
>   remove non-portable ldconfig aux-cache
>   generate deterministic clear-text password hash
>   update debian initramfs in deterministic mode
>   create custom initramfs in deterministic mode
>   make deb_add_changelog idempotent
>   deb_add_changelog: set timestamp to valid epoch
>   make custom linux-image bit-by-bit reproducible
> 
> venkata pyla (1):
>   image.bbclass: fix non-reproducible file time-stamps inside rootfs
> 
>  meta-isar/conf/local.conf.sample              | 10 ++++++++
>  meta/classes/debianize.bbclass                | 20 +++++++++------
>  meta/classes/image-account-extension.bbclass  | 10 +++++++-
>  meta/classes/image.bbclass                    | 25
> +++++++++++++++++-- meta/classes/initramfs.bbclass                |
> 5 ++++ meta/classes/rootfs.bbclass                   | 13 ++++++++++
>  .../linux/files/debian/isar/build.tmpl        |  1 +
>  .../linux/files/debian/rules.tmpl             | 14 ++++++++++-
>  meta/recipes-kernel/linux/linux-custom.inc    |  2 ++
>  9 files changed, 89 insertions(+), 11 deletions(-)
> 

Re: [PATCH v2 00/10] Make rootfs build reproducible

[Moessbauer, Felix]

On Thu, 2023-01-12 at 15:57 +0100, Roberto A. Foglietta wrote:
> 
> 
> On Thu, 12 Jan 2023 at 06:56, Felix Moessbauer <
> felix.moessbauer@siemens.com> wrote:
> > This series finally makes the rootfs generation bit-reproducible
> > from debian bullseye on. Parts of it have already been sent
> > as individual patches. However, image reproducibility can only
> > be achived once all parts are reproducible itself. By that,
> > these patches are included in this series as well.
> > 
> 
> 
> Hi Felix,
> 
>  is there a repository / branch from which I can fetch/import these
> patches?
>  I wish to test them.
>  
>  Thanks,

Hi Roberto,

testers are always welcome.
I just created such a branch on github:
https://github.com/fmoessbauer/isar/tree/fm/reproducible-v2

PS: putting the list in CC so that others can test it as well

Felix