Re: [PATCH v2 1/1] use debian snapshot mirror if SOURCE_DATE_EPOCH is set

[Jan Kiszka <jan.kiszka@siemens.com>]

On 03.04.24 16:12, Felix Moessbauer wrote:
> In case the SOURCE_DATE_EPOCH variable is set, we switch the debian
> mirror to a snapshot mirror. The used date is derived from the value of
> SOURCE_DATE_EPOCH. Similar to the DISTRO_APT_PREMIRRORS, this mirror is
> only injected temporarily during the build.
> 
> To further control the behavior, we introduce the following variables:
> 
> - ISAR_USE_DEBIAN_SNAPSHOTS: overwrite if a snapshot shall be used
> - ISAR_DEBIAN_SNAPSHOT_MIRROR: The snapshot mirror to use (defaults to
>   snapshot-cloudflare.debian.org)
> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
> Changes since v1:
> 
> - disable valid-until checking for both bootstrapping and schroot
> - conditionally make the SOURCE_DATE_EPOCH a vardep of bootstrap
> - move ISAR_USE_DEBIAN_SNAPSHOTS to the bitbake.conf as it is used
>   both in bootstrap, as well as in rootfs.
> 
> Best regards,
> Felix Moessbauer
> Siemens AG
> 
>  RECIPE-API-CHANGELOG.md                         |  6 ++++++
>  doc/user_manual.md                              |  3 +++
>  meta/classes/rootfs.bbclass                     |  3 +++
>  meta/conf/bitbake.conf                          |  1 +
>  .../isar-bootstrap/isar-bootstrap.inc           | 17 +++++++++++++++--
>  5 files changed, 28 insertions(+), 2 deletions(-)
> 
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index 6653ab43..c146d60c 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -583,3 +583,9 @@ Cross compiling kernel modules for distro kernels is not supported in debian.
>  To simplify downstream kernel module builds, we automatically turn of cross
>  compilation for a user-provided module when building it for a distro kernel.
>  
> +
> +### Build against debian snapshot mirror if SOURCE_DATE_EPOCH is set
> +
> +In case the bitbake variable `SOURCE_DATE_EPOCH` is set, a debian snapshot
> +mirror is used. This can be overwritten with `ISAR_USE_DEBIAN_SNAPSHOTS`.
> +The snapshot to use is specified in `ISAR_DEBIAN_SNAPSHOT_MIRROR`.
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 419d5339..227ce5f9 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -425,12 +425,15 @@ Some other variables include:
>  
>   - `IMAGE_INSTALL` - The list of custom packages to build and install to target image, please refer to relative chapter for more information.
>   - `BB_NUMBER_THREADS` - The number of `bitbake` jobs that can be run in parallel. Please set this option according to your host CPU cores number.
> + - `SOURCE_DATE_EPOCH` - The unix timestamp passed to all tooling to make the results reproducible. This variable is optional.
>   - `HOST_DISTRO` - The distro to use for SDK root filesystem. This variable is optional.
>   - `HOST_ARCH` - The Debian architecture of SDK root filesystem (e.g., `amd64`). By default set to current Debian host architecture. This variable is optional.
>   - `HOST_DISTRO_APT_SOURCES` - List of apt source files for SDK root filesystem. This variable is optional.
>   - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for SDK root filesystem. This variable is optional.
>   - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host.
>   - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES.
> + - `ISAR_USE_DEBIAN_SNAPSHOTS` - Use a frozen debian snapshot instead of the live mirror. Auto-enabled if `SOURCE_DATE_EPOCH` is set. Optional.
> + - `ISAR_DEBIAN_SNAPSHOT_MIRROR` - The snapshot mirror to use. Defaults to `snapshot-cloudflare.debian.org`.
>   - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping.
>   - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS.
>   - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable.
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index fb14f3ca..0d7744f7 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -112,6 +112,9 @@ rootfs_configure_apt() {
>      mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d'
>      {
>          echo 'Acquire::Retries "3";'
> +        if [ "${ISAR_USE_DEBIAN_SNAPSHOTS}" = "1" ]; then
> +            echo 'Acquire::Check-Valid-Until "false";'
> +        fi
>          echo 'APT::Install-Recommends "0";'
>          echo 'APT::Install-Suggests "0";'
>      } > '${ROOTFSDIR}/etc/apt/apt.conf.d/50isar'
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 91c5c815..84c9a9bb 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -68,6 +68,7 @@ KERNEL_FILE ?= "vmlinuz"
>  KERNEL_FILE:mipsel ?= "vmlinux"
>  KERNEL_FILE:riscv64 ?= "vmlinux"
>  KERNEL_FILE:arm64 ?= "vmlinux"
> +ISAR_USE_DEBIAN_SNAPSHOTS ??= "${@'1' if d.getVar('SOURCE_DATE_EPOCH') else '0'}"
>  
>  MACHINEOVERRIDES ?= "${MACHINE}"
>  DISTROOVERRIDES ?= "${DISTRO}"
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index f548e202..d44876a4 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -32,6 +32,8 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR
>  BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}"
>  BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}"
>  FILESEXTRAPATHS:append = ":${BBPATH}"
> +# reproducible builds
> +ISAR_DEBIAN_SNAPSHOT_MIRROR ??= "snapshot-cloudflare.debian.org"
>  
>  inherit deb-dl-dir
>  
> @@ -111,9 +113,15 @@ def parse_aptsources_list_line(source_list_line):
>  
>  def get_apt_source_mirror(d, aptsources_entry_list):
>      import re
> +    import time
>  
>      if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>          premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"
> +    elif bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')):
> +        snapshot_mirror = d.getVar('ISAR_DEBIAN_SNAPSHOT_MIRROR')
> +        source_date_epoch = d.getVar('SOURCE_DATE_EPOCH') or int(time.time())
> +        snapshot_date = time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch)))
> +        premirrors = 'deb.debian.org/(.*) {}/archive/\\1/{}/\n'.format(snapshot_mirror, snapshot_date)
>      else:
>          premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or ""
>      mirror_list = [entry.split()
> @@ -265,6 +273,7 @@ def get_host_release():
>      return rel
>  
>  do_bootstrap[vardeps] += " \
> +    ${@'SOURCE_DATE_EPOCH' if bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')) else ''} \
>      DISTRO_APT_PREMIRRORS \
>      ISAR_ENABLE_COMPAT_ARCH \
>      ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
> @@ -382,8 +391,12 @@ do_bootstrap() {
>              chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH}
>          fi
>  
> -        chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \
> -                                -o APT::Update::Error-Mode=any
> +
> +        APT_UPDATE_OPTS="-o APT::Update::Error-Mode=any"
> +        if [ "${ISAR_USE_DEBIAN_SNAPSHOTS}" = "1" ]; then
> +            APT_UPDATE_OPTS="${APT_UPDATE_OPTS} -o Acquire::Check-Valid-Until=false"
> +        fi
> +        chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y ${APT_UPDATE_OPTS}
>          chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
>          chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
>                                  -o Debug::pkgProblemResolver=yes

This can be a valuable addition but it needs more thoughts first:

How do you plan to handle snapshot.ubuntu.com? And what will you when
the repo list contains a third-party repo for which there is no
snapshot? That should at least raise a warning, rather than silently
suggesting to produce a reproducible image.

BTW, we should likely also double-check how we define the
SOURCE_DATE_EPOCH value, in comparison to OE. See my reply on your kas
patch.

Jan

-- 
Siemens AG, Technology
Linux Expert Center