Re: [PATCH v3 2/5] use apt snapshot mirror if ISAR_USE_APT_SNAPSHOT is set

[Anton Mikanovich <amikan@ilbers.de>]

09/04/2024 18:55, 'Felix Moessbauer' via isar-users wrote:
> This patch adds infrastructure to switch the apt sources to a frozen
> snapshot mirror. To build against a mirror, set ISAR_USE_APT_SNAPSHOT=1.
> As the mirror is distro specific, it is configured in the distro config
> for all supported distros (currently only debian and ubuntu). For
> unsupported distros, a meaningful error message is emitted on enabling
> the snapshot build. Similar to the DISTRO_APT_PREMIRRORS, this mirror is
> only injected temporarily and does not end up in the final apt sources
> list.
>
> To further control the behavior, we introduce the following variables:
>
> - DISTRO_APT_SNAPSHOT_PREMIRROR: The snapshot mirror to use. Syntax
>    identical to DISTRO_APT_PREMIRRORS.
> - ISAR_APT_SNAPSHOT_TIMESTAMP: Unix timestamp of the snapshot. This is
>    automatically derived from the SOURCE_DATE_EPOCH if not set.
>
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>   RECIPE-API-CHANGELOG.md                          |  6 ++++++
>   doc/user_manual.md                               |  3 +++
>   meta-isar/conf/distro/ubuntu-common.inc          |  3 +++
>   meta/conf/bitbake.conf                           |  3 +++
>   meta/conf/distro/debian-common.conf              |  3 +++
>   .../isar-bootstrap/isar-bootstrap.inc            | 16 ++++++++++++++++
>   6 files changed, 34 insertions(+)
>
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index 6653ab43..e6861523 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -583,3 +583,9 @@ Cross compiling kernel modules for distro kernels is not supported in debian.
>   To simplify downstream kernel module builds, we automatically turn of cross
>   compilation for a user-provided module when building it for a distro kernel.
>   
> +
> +### Build against debian snapshot mirror
> +
> +To build against a distributions snapshot mirror, set `ISAR_USE_APT_SNAPSHOT="1"`.
> +The mirror to use is specified in `DISTRO_APT_SNAPSHOT_PREMIRROR` and usually
> +pre-defined in the distro config.
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 419d5339..70741968 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -431,6 +431,9 @@ Some other variables include:
>    - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for SDK root filesystem. This variable is optional.
>    - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host.
>    - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES.
> + - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of the live mirror. Optional.
> + - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported distros.
> + - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Timestamp of the apt snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not overwritten.
>    - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping.
>    - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS.
>    - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable.
> diff --git a/meta-isar/conf/distro/ubuntu-common.inc b/meta-isar/conf/distro/ubuntu-common.inc
> index 9d8a843b..54bb747a 100644
> --- a/meta-isar/conf/distro/ubuntu-common.inc
> +++ b/meta-isar/conf/distro/ubuntu-common.inc
> @@ -32,3 +32,6 @@ IMAGE_PREINSTALL += "init"
>   IMAGE_PREINSTALL += "initramfs-tools"
>   
>   IMAGER_INSTALL:wic += "python3-distutils"
> +
> +# snapshot mirror for reproducible builds
> +DISTRO_APT_SNAPSHOT_PREMIRROR ??= "(http|https)://archive.ubuntu.com/(.*) https://snapshot.ubuntu.com/\2/${APT_SNAPSHOT_DATE}/\n"
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 1da3ecac..4cfa8b10 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -145,6 +145,9 @@ export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
>   # A SOURCE_DATE_EPOCH of '0' might be misinterpreted as no SDE
>   # ISAR: set value to date of latest release
>   SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251"
> +# Debian snapshots
> +ISAR_USE_APT_SNAPSHOT ??= "0"
> +ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}"
>   
>   # Default parallelism and resource usage for xz
>   XZ_MEMLIMIT ?= "50%"
> diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf
> index 1e1dfc83..db538510 100644
> --- a/meta/conf/distro/debian-common.conf
> +++ b/meta/conf/distro/debian-common.conf
> @@ -39,3 +39,6 @@ SYSTEMD_BOOTLOADER_INSTALL:sid = "systemd-boot-efi:${DISTRO_ARCH}"
>   
>   COMPAT_DISTRO_ARCH:amd64 = "i386"
>   COMPAT_DISTRO_ARCH:arm64 = "armhf"
> +
> +# snapshot mirror for reproducible builds
> +DISTRO_APT_SNAPSHOT_PREMIRROR ??= "deb.debian.org/(.*) snapshot-cloudflare.debian.org/archive/\1/${APT_SNAPSHOT_DATE}/\n"
> \ No newline at end of file
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index 17f19fd8..733a23df 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -30,6 +30,9 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR
>   BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}"
>   BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}"
>   FILESEXTRAPATHS:append = ":${BBPATH}"
> +# reproducible builds, only enabled if ISAR_USE_APT_SNAPSHOT
> +ISAR_APT_SNAPSHOT_MIRROR ??= ""
> +APT_SNAPSHOT_DATE = "${@ get_apt_snapshot_date(d)}"
>   
>   inherit deb-dl-dir
>   
> @@ -107,11 +110,21 @@ def parse_aptsources_list_line(source_list_line):
>   
>       return [type, options, source, suite, components]
>   
> +def get_apt_snapshot_date(d):
> +    import time
> +    source_date_epoch = d.getVar('ISAR_APT_SNAPSHOT_TIMESTAMP')
> +    return time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch)))
> +
>   def get_apt_source_mirror(d, aptsources_entry_list):
>       import re
>   
> +    # this is executed during parsing. No error checking possible
> +    use_snapshot = bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT'))
> +    snapshot_mirror = d.getVar('DISTRO_APT_SNAPSHOT_PREMIRROR')
>       if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>           premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"
> +    elif use_snapshot and snapshot_mirror:
> +        premirrors = snapshot_mirror
>       else:
>           premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or ""
>       mirror_list = [entry.split()
> @@ -126,6 +139,8 @@ def get_apt_source_mirror(d, aptsources_entry_list):
>               new_aptsources_entry_list[2] = re.sub(regex, replace,
>                                                     aptsources_entry_list[2],
>                                                     count = 1)
> +            if use_snapshot:
> +                new_aptsources_entry_list[1] = "[check-valid-until=no]"
>               return new_aptsources_entry_list
>   
>       return aptsources_entry_list
> @@ -240,6 +255,7 @@ do_apt_config_prepare[vardeps] += " \
>       APTSRCS \
>       ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
>       DEPLOY_ISAR_BOOTSTRAP \
> +    ${@'DISTRO_APT_SNAPSHOT_PREMIRROR' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''} \
>       "
>   python do_apt_config_prepare() {
>       apt_preferences_out = d.getVar("APTPREFS")

Hello Felix,

How ISAR_APT_SNAPSHOT_MIRROR supposed to be used?