[PATCH 0/8] Make all sstate artifacts reproducible

[PATCH 0/8] Make all sstate artifacts reproducible

['Felix Moessbauer' via isar-users]

Currently many artifacts we deploy to the sstate cache are not reproducible
by itself. By that, rebuilds of the same artifact result in different sstate hashes,
which leads to cache trashing.

As a preparation to use the hashequiv signature handler, we have to make all artifacts
reproducible, as this handler tracks which input hashes influence the output hash (hash
of the artifact).

As a result of this requirement, we can no longer split the do_rootfs_install task
(which is cached) and the do_rootfs_postprocess task which makes the rootfs reproducible.
Instead, we now fold the postprocess task into the install task, similar to how OE does it.

Best regards,
Felix Moessbauer
Siemens AG

Felix Moessbauer (8):
  make bootstrap output reproducible
  rootfs: fold do_rootfs_postprocess task into do_rootfs_install
  rootfs: make tarring of sstate artifact reproducible
  rootfs: split clean_package_cache command
  rootfs: make file timestamps reproducible
  repository: create isar-apt with a reproducible timestamp
  make our internal chroots bit-by-bit reproducible
  rootfs: make capture of apt state bit-by-bit reproducible

 RECIPE-API-CHANGELOG.md                       |  6 ++
 .../recipes-core/images/isar-rootfs-ci.bb     |  2 +-
 .../image-account-extension.bbclass           |  3 -
 meta/classes-recipe/image.bbclass             | 15 ++--
 meta/classes-recipe/initramfs.bbclass         |  5 +-
 meta/classes-recipe/repository.bbclass        | 19 +++++
 meta/classes-recipe/rootfs.bbclass            | 72 ++++++++++---------
 meta/classes-recipe/sdk.bbclass               |  2 +-
 .../isar-mmdebstrap/isar-mmdebstrap.inc       |  2 +
 9 files changed, 74 insertions(+), 52 deletions(-)

-- 
2.55.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260820083014.3377283-1-felix.moessbauer%40siemens.com.

[PATCH 1/8] make bootstrap output reproducible

['Felix Moessbauer' via isar-users]

As a preparation to benefit from the hashequiv server, we need to ensure
the sstate artifacts themselves are reproducible (i.e. if no input
parameter changes, the output does not change).

For that, we follow the mmdebstrap recommendation from the manpage and
pin the hostname and resolv.conf.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
index 994da174..ee6eda3c 100644
--- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
+++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
@@ -255,6 +255,8 @@ do_bootstrap() {
                    --customize-hook='sed -i "/en_US.UTF-8 UTF-8/s/^#//g" "$1/etc/locale.gen"' \
                    --customize-hook='chroot "$1" /usr/sbin/locale-gen' \
                    --customize-hook='chroot "$1" /usr/bin/apt-get -y clean' \
+                   --customize-hook='echo nameserver 127.0.0.1 > "$1"/etc/resolv.conf' \
+                   --customize-hook='echo isar > "$1"/etc/hostname' \
                    ${@'--skip=output/dev' if d.getVar('ISAR_CHROOT_MODE') == 'unshare' else ''} \
                    --skip=cleanup/apt \
                    --skip=download/empty \
-- 
2.55.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260820083014.3377283-2-felix.moessbauer%40siemens.com.

[PATCH 2/8] rootfs: fold do_rootfs_postprocess task into do_rootfs_install

['Felix Moessbauer' via isar-users]

The do_rootfs_postprocess task is not idempotent and contains steps
which can only run on an unmodified rootfs from do_rootfs_install (e.g.
user-creation, steps that need the apt cache). To make both steps
atomic, we move them into the same task, similar to how OE models
it.

This further improves build performance on rebuilds from the sstate
cache, as the whole artifact can be taken and the postprocess steps do
not need to be re-executed. The sstate artifact is also smaller, as the
rootfs is stored in clean state.

By that, we further drop the manual registration of the USERS / GROUPS
vardeps on the do_rootfs_install task. These are now propagated via the
corresponding ROOTFS_POSTPROCESS_COMMAND.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 RECIPE-API-CHANGELOG.md                       |  6 +++
 .../image-account-extension.bbclass           |  3 --
 meta/classes-recipe/image.bbclass             |  4 +-
 meta/classes-recipe/initramfs.bbclass         |  2 +-
 meta/classes-recipe/rootfs.bbclass            | 42 +++++--------------
 5 files changed, 20 insertions(+), 37 deletions(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 2db05169..f5630762 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -1201,3 +1201,9 @@ To opt out, remove the feature explicitly:
 ```
 ROOTFS_FEATURES:remove = "clean-apt-credentials"
 ```
+
+### do_rootfs_postprocess is folded into do_rootfs_install
+
+All `ROOTFS_POSTPROCESS_COMMAND` steps are now executed within the `do_rootfs_install`
+task to avoid idempotency issues on partial rebuilds. Task that previously had a
+dependency to `do_rootfs_postprocess` shall now be changed to run after `do_rootfs_install`.
diff --git a/meta/classes-recipe/image-account-extension.bbclass b/meta/classes-recipe/image-account-extension.bbclass
index de01484c..dd70f2a3 100644
--- a/meta/classes-recipe/image-account-extension.bbclass
+++ b/meta/classes-recipe/image-account-extension.bbclass
@@ -14,14 +14,11 @@ python() {
     for entry in (d.getVar("GROUPS") or "").split():
         group_entry = "GROUP_{}".format(entry)
         d.appendVarFlag("image_postprocess_accounts", "vardeps", " {}".format(group_entry))
-        d.appendVarFlag("do_rootfs_install", "vardeps", " {}".format(group_entry))
 
     for entry in (d.getVar("USERS") or "").split():
         user_entry = "USER_{}".format(entry)
         d.appendVarFlag("image_postprocess_accounts", "vardeps", " {}".format(user_entry))
-        d.appendVarFlag("do_rootfs_install", "vardeps", " {}".format(user_entry))
 }
-do_rootfs_install[vardeps] += "GROUPS USERS"
 
 def image_create_groups(d: "DataSmart") -> None:
     """Creates the groups defined in the ``GROUPS`` bitbake variable.
diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass
index d02f99da..a749441f 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -408,7 +408,7 @@ do_copy_boot_files() {
         cp -f "$dtb" "${DEPLOYDIR}/"
     done
 }
-addtask copy_boot_files before do_rootfs_postprocess after do_rootfs_install
+addtask copy_boot_files before do_rootfs do_generate_initramfs after do_rootfs_install
 
 python do_copy_boot_files_setscene () {
     sstate_setscene(d)
@@ -492,7 +492,7 @@ EOSUDO
         -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';'
 }
 do_rootfs_finalize[network] = "${TASK_USE_SUDO}"
-addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess
+addtask rootfs_finalize before do_rootfs after do_rootfs_install
 
 ROOTFS_QA_FIND_ARGS ?= ""
 
diff --git a/meta/classes-recipe/initramfs.bbclass b/meta/classes-recipe/initramfs.bbclass
index 5a98e337..a647b6ea 100644
--- a/meta/classes-recipe/initramfs.bbclass
+++ b/meta/classes-recipe/initramfs.bbclass
@@ -78,7 +78,7 @@ rootfs_purge_initramfs() {
 
 # If an external initrd shall be used, run it as a task
 # instead of as part of the rootfs install.
-addtask do_generate_initramfs before do_rootfs_postprocess after do_rootfs_install
+addtask do_generate_initramfs before do_rootfs after do_rootfs_install
 addtask do_generate_initramfs_setscene
 ROOTFS_INSTALL_COMMAND:remove = "rootfs_generate_initramfs"
 
diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index 8d394b05..b85cb59a 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -430,19 +430,21 @@ do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}"
 do_rootfs_install[cleandirs] += "${DEPLOYDIR}"
 do_rootfs_install[sstate-inputdirs] = "${DEPLOYDIR}"
 do_rootfs_install[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
-do_rootfs_install[vardeps] += "${ROOTFS_CONFIGURE_COMMAND} ${ROOTFS_INSTALL_COMMAND} ${ROOTFS_VARDEPS}"
+do_rootfs_install[vardeps] += "${ROOTFS_CONFIGURE_COMMAND} ${ROOTFS_INSTALL_COMMAND} ${ROOTFS_POSTPROCESS_COMMAND} ${ROOTFS_VARDEPS}"
 do_rootfs_install[vardepsexclude] += "IMAGE_ROOTFS"
 do_rootfs_install[depends] = "bootstrap-${@'target' if d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build"
+do_rootfs_install[depends] += "base-apt:do_cache isar-apt:do_cache_config"
 do_rootfs_install[deptask] = "do_deploy_deb"
 do_rootfs_install[rdeptask] = "do_deploy_deb"
 do_rootfs_install[network] = "${TASK_USE_SUDO}"
 python do_rootfs_install() {
     configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND") or "").split()
     install_cmds = (d.getVar("ROOTFS_INSTALL_COMMAND") or "").split()
+    postprocess_cmds = (d.getVar("ROOTFS_POSTPROCESS_COMMAND") or "").split()
 
     # Mount after configure commands, so that they have time to copy
     # 'isar-apt' (sdkchroot):
-    cmds = ['rootfs_prepare'] + configure_cmds + ['rootfs_do_mounts'] + install_cmds
+    cmds = ['rootfs_prepare'] + configure_cmds + ['rootfs_do_mounts'] + install_cmds + postprocess_cmds
 
     # NOTE: The weights specify how long each task takes in seconds and are used
     # by the MultiStageProgressReporter to render a progress bar for this task.
@@ -470,7 +472,7 @@ python do_rootfs_install() {
         progress_reporter.finish()
         bb.build.exec_func('rootfs_do_umounts', d)
 }
-addtask rootfs_install before do_rootfs_postprocess after do_unpack
+addtask rootfs_install before do_rootfs after do_unpack
 
 do_cache_deb_src[network] = "${TASK_USE_SUDO}"
 do_cache_deb_src() {
@@ -611,33 +613,6 @@ image_postprocess_populate_systemd_preset() {
 EOSH
 }
 
-do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}"
-do_rootfs_postprocess[network] = "${TASK_USE_SUDO}"
-do_rootfs_postprocess[depends] = "base-apt:do_cache isar-apt:do_cache_config"
-python do_rootfs_postprocess() {
-    # Take care that its correctly mounted:
-    bb.build.exec_func('rootfs_do_mounts', d)
-    # Take care that qemu-*-static is available, since it could have been
-    # removed on a previous execution of this task:
-    bb.build.exec_func('rootfs_do_qemu', d)
-
-    progress_reporter = bb.progress.ProgressHandler(d)
-    progress_reporter.update(0)
-
-    cmds = d.getVar("ROOTFS_POSTPROCESS_COMMAND")
-    if cmds is None or not cmds.strip():
-        return
-    cmds = cmds.split()
-
-    try:
-        for i, cmd in enumerate(cmds):
-            bb.build.exec_func(cmd, d)
-            progress_reporter.update(int(i / len(cmds) * 100))
-    finally:
-        bb.build.exec_func('rootfs_do_umounts', d)
-}
-addtask rootfs_postprocess before do_rootfs after do_unpack
-
 ROOTFS_INSTALL_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'generate-initrd', 'rootfs_generate_initramfs', '', d)}"
 rootfs_generate_initramfs[weight] = "1000"
 rootfs_generate_initramfs[progress] = "custom:rootfs_progress.InitrdProgressHandler"
@@ -682,7 +657,7 @@ rootfs_install_sstate_prepare() {
 
     run_privileged_heredoc <<'EOF'
         mount -o bind,private '${ROOTFSDIR}' '${WORKDIR}/mnt/rootfs' -o ro
-        lopts="--one-file-system --exclude=var/cache/apt/archives"
+        lopts="--one-file-system"
         tar -C ${WORKDIR}/mnt/rootfs -cpSf rootfs.tar $lopts ${SSTATE_TAR_ATTR_FLAGS} .
         umount -q ${WORKDIR}/mnt/rootfs
 EOF
@@ -715,3 +690,8 @@ python do_rootfs_install_setscene() {
     sstate_setscene(d)
 }
 addtask do_rootfs_install_setscene
+
+python do_rootfs_postprocess() {
+    bb.warn("task do_rootfs_postprocess was folded into do_rootfs_install, please order your task after do_rootfs_install instead")
+}
+addtask do_rootfs_postprocess after do_rootfs_install
-- 
2.55.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260820083014.3377283-3-felix.moessbauer%40siemens.com.

[PATCH 3/8] rootfs: make tarring of sstate artifact reproducible

['Felix Moessbauer' via isar-users]

To reduce the sstate cache size and profit from the hashequiv server, we
need to make the sstate artifacts reproducible. For that, we remove the
-S/--sparse parameter, as the sparse maps itself are not reproducible.
Further, we need to drop the atime and ctime of the extended
attributes which, which point to GNU tar's volatile atime/ctime entries
rather than an actual xattr value.

With these changes, the tarring itself is bit-by-bit reproducible.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes-recipe/rootfs.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index b85cb59a..e67a0214 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -643,7 +643,7 @@ SSTATETASKS += "do_rootfs_install"
 SSTATECREATEFUNCS += "rootfs_install_sstate_prepare"
 SSTATEPOSTINSTFUNCS += "rootfs_install_sstate_finalize"
 
-SSTATE_TAR_ATTR_FLAGS ?= "--xattrs --xattrs-include='*'"
+SSTATE_TAR_ATTR_FLAGS ?= "--xattrs --xattrs-include='*' --pax-option=delete=atime,delete=ctime"
 
 # the rootfs is owned by root, so we need some sudoing to pack and unpack
 rootfs_install_sstate_prepare() {
@@ -657,8 +657,8 @@ rootfs_install_sstate_prepare() {
 
     run_privileged_heredoc <<'EOF'
         mount -o bind,private '${ROOTFSDIR}' '${WORKDIR}/mnt/rootfs' -o ro
-        lopts="--one-file-system"
-        tar -C ${WORKDIR}/mnt/rootfs -cpSf rootfs.tar $lopts ${SSTATE_TAR_ATTR_FLAGS} .
+        lopts="--one-file-system --sort=name"
+        tar -C ${WORKDIR}/mnt/rootfs -cpf rootfs.tar $lopts ${SSTATE_TAR_ATTR_FLAGS} .
         umount -q ${WORKDIR}/mnt/rootfs
 EOF
     if [ -f ${WORKDIR}/${ROOTFS_APT_STATE} ]; then
-- 
2.55.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260820083014.3377283-4-felix.moessbauer%40siemens.com.

[PATCH 4/8] rootfs: split clean_package_cache command

['Felix Moessbauer' via isar-users]

We currently also delete the apt lists in /var/lib/apt in that command,
but this requires a later user of the chroot to run apt-update again
(with all its implications). To avoid that, but still be able to clean
the apt cache for reproducability reasons, we split the command into the
purging of the cache and a new one "clean_apt_lists" that remove the apt
lists.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes-recipe/image.bbclass  |  1 +
 meta/classes-recipe/rootfs.bbclass | 10 ++++++++--
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass
index a749441f..f3a27782 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -95,6 +95,7 @@ inherit essential
 ROOTFSDIR = "${IMAGE_ROOTFS}"
 ROOTFS_FEATURES += "\
     clean-package-cache \
+    clean-apt-lists \
     clean-pycache \
     generate-manifest \
     export-dpkg-status \
diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index e67a0214..79dec69f 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -31,6 +31,7 @@ ROOTFS_BASE_DISTRO ?= "${BASE_DISTRO}"
 # available features are:
 # 'clean-package-cache' - delete package cache from rootfs
 # 'clean-debconf-cache' - delete debconf cache files
+# 'clean-apt-lists' - delete apt lists files (after execution, apt-get update needs to be run again)
 # 'clean-pycache' - delete python bytecode cache files
 # 'generate-manifest' - generate a package manifest of the rootfs into ${ROOTFS_MANIFEST_DEPLOY_DIR}
 # 'generate-sbom' - generate a SBOM of the rootfs into ${DEPLOY_DIR_SBOM}
@@ -522,9 +523,14 @@ ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-pa
 rootfs_postprocess_clean_package_cache() {
     run_in_chroot '${ROOTFSDIR}' \
         /usr/bin/apt-get clean
-    run_privileged rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"*
     # remove apt-cache folder itself (required in case rootfs is provided by sstate cache)
-    run_privileged rm -rf "${ROOTFSDIR}/var/cache/apt/archives"
+    run_privileged find "${ROOTFSDIR}/var/cache/apt" -type f \
+        \( -name '*.deb' -o -name '*.bin' \) -delete
+}
+
+ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-apt-lists', 'rootfs_postprocess_clean_apt_lists', '', d)}"
+rootfs_postprocess_clean_apt_lists() {
+    run_privileged find "${ROOTFSDIR}/var/lib/apt/lists" -type f -delete
 }
 
 ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-log-files', 'rootfs_postprocess_clean_log_files', '', d)}"
-- 
2.55.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260820083014.3377283-5-felix.moessbauer%40siemens.com.

[PATCH 5/8] rootfs: make file timestamps reproducible

['Felix Moessbauer' via isar-users]

As a preparation to get a bit-by-bit reproducible sstate artifact of the
rootfs, we make the file timestamps of do_rootfs_install reproducible.
For that, we move the corresponding logic from the image class to the
rootfs class. In image, we still do one final pass, as later tasks might
have changed the timestamps again.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes-recipe/image.bbclass  | 6 ++----
 meta/classes-recipe/rootfs.bbclass | 8 ++++++++
 2 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass
index f3a27782..7abda3de 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -487,10 +487,8 @@ EOSUDO
         bbwarn "found core dump in rootfs, check it in ${WORKDIR}/temp/${f##*/}"
     done
 
-    # Set same time-stamps to the newly generated file/folders in the
-    # rootfs image for the purpose of reproducible builds.
-    run_privileged find ${ROOTFSDIR} -newermt "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \
-        -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';'
+    # Final pass to set timestamps of files in rootfs to SOURCE_DATE_EPOCH for reproducible builds
+    rootfs_set_timestamps
 }
 do_rootfs_finalize[network] = "${TASK_USE_SUDO}"
 addtask rootfs_finalize before do_rootfs after do_rootfs_install
diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index 79dec69f..9a08312e 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -438,6 +438,7 @@ do_rootfs_install[depends] += "base-apt:do_cache isar-apt:do_cache_config"
 do_rootfs_install[deptask] = "do_deploy_deb"
 do_rootfs_install[rdeptask] = "do_deploy_deb"
 do_rootfs_install[network] = "${TASK_USE_SUDO}"
+do_rootfs_install[postfuncs] += "rootfs_set_timestamps"
 python do_rootfs_install() {
     configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND") or "").split()
     install_cmds = (d.getVar("ROOTFS_INSTALL_COMMAND") or "").split()
@@ -619,6 +620,13 @@ image_postprocess_populate_systemd_preset() {
 EOSH
 }
 
+# Set same time-stamps to the newly generated file/folders in the rootfs image for the purpose of reproducible builds.
+# This has to be done after unmounting
+rootfs_set_timestamps() {
+    run_privileged find ${ROOTFSDIR} -newermt "$(date -d@${SOURCE_DATE_EPOCH} '+%Y-%m-%d %H:%M:%S')" \
+        -exec touch '{}' -h -d@${SOURCE_DATE_EPOCH} ';'
+}
+
 ROOTFS_INSTALL_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'generate-initrd', 'rootfs_generate_initramfs', '', d)}"
 rootfs_generate_initramfs[weight] = "1000"
 rootfs_generate_initramfs[progress] = "custom:rootfs_progress.InitrdProgressHandler"
-- 
2.55.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260820083014.3377283-6-felix.moessbauer%40siemens.com.

[PATCH 6/8] repository: create isar-apt with a reproducible timestamp

['Felix Moessbauer' via isar-users]

As this timestamp ends up in the "Date:" field of the release file, it
also ends up in our internal rootfs'. To make these reproducible, we
have to set this to a fixed date. As the date itself is completely
irrelevant and not part of the generated images, we just use 1h after
epoch (similar to our default changelog date, as 0 cannot be used).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes-recipe/repository.bbclass | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/meta/classes-recipe/repository.bbclass b/meta/classes-recipe/repository.bbclass
index ba2803ad..a2061100 100644
--- a/meta/classes-recipe/repository.bbclass
+++ b/meta/classes-recipe/repository.bbclass
@@ -10,6 +10,20 @@ def repo_expand_opt_fields(d, var):
         return ''
     return '\n'.join('{}: {}'.format(k, v) for k, v in f.items())
 
+# use a fixed but arbitrary date for the Release file to make it reproducible
+ISAR_APT_RELEASE_DATE = "3600"
+repo_set_release_date() {
+    local dir="$1"
+    local codename="$2"
+    local release="${dir}/dists/${codename}/Release"
+
+    [ "${dir}" = "${REPO_ISAR_DIR}/${DISTRO}" ] || return 0
+    [ -f "${release}" ] || return 0
+
+    sed -i -e "s/^Date: .*/Date: $(LC_ALL=C date -u -R -d @${ISAR_APT_RELEASE_DATE})/" \
+        "${release}"
+}
+
 repo_create() {
     local dir="$1"
     local dbdir="$2"
@@ -40,6 +54,7 @@ repo_create() {
     fi
     if [ ! -d "${dbdir}" ]; then
         reprepro -b "${dir}" --dbdir "${dbdir}" export "${codename}"
+        repo_set_release_date "${dir}" "${codename}"
     fi
 }
 
@@ -55,6 +70,7 @@ repo_add_srcpackage() {
     reprepro -b "${dir}" --dbdir "${dbdir}" -C main -S - -P source \
         includedsc "${codename}" \
         "$@"
+    repo_set_release_date "${dir}" "${codename}"
 }
 
 repo_add_packages() {
@@ -77,6 +93,7 @@ repo_add_packages() {
         fi
         reprepro -b "${dir}" --dbdir "${dbdir}" -C main ${prio_opt} \
             includedeb "${codename}" "${package}"
+        repo_set_release_date "${dir}" "${codename}"
     done
 }
 
@@ -92,6 +109,7 @@ repo_del_srcpackage() {
     reprepro -b "${dir}" --dbdir "${dbdir}" -A source \
         remove "${codename}" \
         "${packagename}"
+    repo_set_release_date "${dir}" "${codename}"
 }
 
 repo_del_package() {
@@ -108,6 +126,7 @@ repo_del_package() {
     reprepro -b "${dir}" --dbdir "${dbdir}" -C main \
         removefilter "${codename}" \
         'Package (= '${p}'), Architecture (= '${a}'), $PackageType (= deb)'
+    repo_set_release_date "${dir}" "${codename}"
 }
 
 repo_contains_package() {
-- 
2.55.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260820083014.3377283-7-felix.moessbauer%40siemens.com.

[PATCH 7/8] make our internal chroots bit-by-bit reproducible

['Felix Moessbauer' via isar-users]

By that, the generated sstate artifacts are also bit identical which
reduces the cache size. It further makes it possible to re-use the
artifacts with the hashequiv logic of bitbake.

For that, we auto-select all rootfs features needed for reproducible
contents.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta-test/recipes-core/images/isar-rootfs-ci.bb | 2 +-
 meta/classes-recipe/image.bbclass               | 4 ----
 meta/classes-recipe/initramfs.bbclass           | 3 +--
 meta/classes-recipe/rootfs.bbclass              | 9 +++++++++
 meta/classes-recipe/sdk.bbclass                 | 2 +-
 5 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/meta-test/recipes-core/images/isar-rootfs-ci.bb b/meta-test/recipes-core/images/isar-rootfs-ci.bb
index 5ac33665..4aeed71f 100644
--- a/meta-test/recipes-core/images/isar-rootfs-ci.bb
+++ b/meta-test/recipes-core/images/isar-rootfs-ci.bb
@@ -7,7 +7,7 @@
 ROOTFS_MANIFEST_DEPLOY_DIR = "${DEPLOY_DIR_IMAGE}"
 
 ROOTFSDIR = "${WORKDIR}/rootfs"
-ROOTFS_FEATURES = "generate-sbom"
+ROOTFS_FEATURES += "generate-sbom"
 
 inherit multiarch
 inherit rootfs
diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass
index 7abda3de..a609cde5 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -94,13 +94,9 @@ inherit essential
 
 ROOTFSDIR = "${IMAGE_ROOTFS}"
 ROOTFS_FEATURES += "\
-    clean-package-cache \
     clean-apt-lists \
-    clean-pycache \
     generate-manifest \
     export-dpkg-status \
-    clean-log-files \
-    clean-debconf-cache \
     populate-systemd-preset \
     generate-sbom \
     clean-apt-credentials \
diff --git a/meta/classes-recipe/initramfs.bbclass b/meta/classes-recipe/initramfs.bbclass
index a647b6ea..45626319 100644
--- a/meta/classes-recipe/initramfs.bbclass
+++ b/meta/classes-recipe/initramfs.bbclass
@@ -26,8 +26,7 @@ SBOM_DISTRO_NAME:append = "-Initramfs"
 DEPENDS += "${INITRAMFS_INSTALL}"
 
 ROOTFSDIR = "${INITRAMFS_ROOTFS}"
-ROOTFS_FEATURES = "generate-manifest generate-sbom"
-ROOTFS_FEATURES += "generate-initrd"
+ROOTFS_FEATURES += "generate-manifest generate-sbom generate-initrd"
 ROOTFS_PACKAGES = "${INITRAMFS_GENERATOR_PKG} ${INITRAMFS_PREINSTALL} ${INITRAMFS_INSTALL}"
 
 # validate whether there are incompatible packages in the installation list
diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index 9a08312e..e5d158ca 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -41,6 +41,15 @@ ROOTFS_BASE_DISTRO ?= "${BASE_DISTRO}"
 # 'populate-systemd-preset' - enable systemd units according to systemd presets
 # 'clean-apt-credentials' - remove apt auth credentials written by ISAR_APT_CREDS
 
+# convenience variable to enable all features needed for a reproducible rootfs build
+ROOTFS_FEATURES_REPRODUCIBLE = " \
+    clean-package-cache \
+    clean-log-files \
+    clean-debconf-cache \
+    clean-pycache \
+"
+ROOTFS_FEATURES += "${ROOTFS_FEATURES_REPRODUCIBLE}"
+
 # only supported from bookworm / jammy on
 ROOTFS_FEATURES:remove:buster = "generate-sbom"
 ROOTFS_FEATURES:remove:bullseye = "generate-sbom"
diff --git a/meta/classes-recipe/sdk.bbclass b/meta/classes-recipe/sdk.bbclass
index d79de2bf..ca38cfc3 100644
--- a/meta/classes-recipe/sdk.bbclass
+++ b/meta/classes-recipe/sdk.bbclass
@@ -49,7 +49,7 @@ ROOTFS_ARCH:class-sdk = "${HOST_ARCH}"
 ROOTFS_DISTRO:class-sdk = "${@get_rootfs_distro(d)}"
 ROOTFS_PACKAGES:class-sdk = "sdk-files ${SDK_TOOLCHAIN} ${SDK_PREINSTALL} ${@isar_multiarch_packages('SDK_INSTALL', d)}"
 ROOTFS_VARDEPS:class-sdk = "SDK_INSTALL SDK_INCLUDE_ISAR_APT"
-ROOTFS_FEATURES:append:class-sdk = " clean-package-cache generate-manifest export-dpkg-status generate-sbom clean-apt-credentials"
+ROOTFS_FEATURES:append:class-sdk = " clean-apt-lists clean-apt-credentials generate-manifest export-dpkg-status generate-sbom"
 ROOTFS_MANIFEST_DEPLOY_DIR:class-sdk = "${DEPLOY_DIR_SDKCHROOT}"
 ROOTFS_DPKGSTATUS_DEPLOY_DIR:class-sdk = "${DEPLOY_DIR_SDKCHROOT}"
 
-- 
2.55.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260820083014.3377283-8-felix.moessbauer%40siemens.com.

[PATCH 8/8] rootfs: make capture of apt state bit-by-bit reproducible

['Felix Moessbauer' via isar-users]

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes-recipe/rootfs.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index e5d158ca..0f2e9d9e 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -426,7 +426,8 @@ rootfs_clear_initrd_symlinks() {
 ROOTFS_INSTALL_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'generate-sbom', 'rootfs_capture_apt_state', '', d)}"
 rootfs_capture_apt_state() {
     ( cd ${ROOTFSDIR} && find usr/share/doc -name copyright -print0 ) | \
-    tar -cf ${WORKDIR}/${ROOTFS_APT_STATE} --zstd -C ${ROOTFSDIR} \
+    tar -cf ${WORKDIR}/${ROOTFS_APT_STATE} --zstd --sort=name \
+        --mtime=@${SOURCE_DATE_EPOCH} -C ${ROOTFSDIR} \
         --exclude=var/lib/apt/lists/partial \
         --exclude=var/lib/apt/lists/lock \
         --exclude=var/lib/apt/lists/auxfiles \
-- 
2.55.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260820083014.3377283-9-felix.moessbauer%40siemens.com.