Re: [PATCH 1/1] use debian snapshot mirror if SOURCE_DATE_EPOCH is set

["MOESSBAUER, Felix" <felix.moessbauer@siemens.com>]

On Wed, 2024-04-03 at 13:33 +0200, Felix Moessbauer wrote:
> In case the SOURCE_DATE_EPOCH variable is set, we switch the debian
> mirror to a snapshot mirror. The used date is derived from the value
> of
> SOURCE_DATE_EPOCH. Similar to the DISTRO_APT_PREMIRRORS, this mirror
> is
> only injected temporarily during the build.
> 
> To further control the behavior, we introduce the following
> variables:
> 
> - ISAR_USE_DEBIAN_SNAPSHOTS: overwrite if a snapshot shall be used
> - ISAR_DEBIAN_SNAPSHOT_MIRROR: The snapshot mirror to use (defaults
> to
>   snapshot-cloudflare.debian.org)

I just noticed, that we also need to disable the valid-until check.
Will fix that in a v2.

Best regards,
Felix

> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
> Dear maintainers,
> 
> I'm not quite sure if the introduced variables also need to be added
> to the vardeps of e.g. the bootstrap task. Please double check this.
> 
> Best regards,
> Felix Moessbauer
> Siemens AG
> 
>  RECIPE-API-CHANGELOG.md                             | 6 ++++++
>  doc/user_manual.md                                  | 3 +++
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 9 +++++++++
>  3 files changed, 18 insertions(+)
> 
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index 6653ab43..c146d60c 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -583,3 +583,9 @@ Cross compiling kernel modules for distro kernels
> is not supported in debian.
>  To simplify downstream kernel module builds, we automatically turn
> of cross
>  compilation for a user-provided module when building it for a distro
> kernel.
>  
> +
> +### Build against debian snapshot mirror if SOURCE_DATE_EPOCH is set
> +
> +In case the bitbake variable `SOURCE_DATE_EPOCH` is set, a debian
> snapshot
> +mirror is used. This can be overwritten with
> `ISAR_USE_DEBIAN_SNAPSHOTS`.
> +The snapshot to use is specified in `ISAR_DEBIAN_SNAPSHOT_MIRROR`.
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 419d5339..227ce5f9 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -425,12 +425,15 @@ Some other variables include:
>  
>   - `IMAGE_INSTALL` - The list of custom packages to build and
> install to target image, please refer to relative chapter for more
> information.
>   - `BB_NUMBER_THREADS` - The number of `bitbake` jobs that can be
> run in parallel. Please set this option according to your host CPU
> cores number.
> + - `SOURCE_DATE_EPOCH` - The unix timestamp passed to all tooling to
> make the results reproducible. This variable is optional.
>   - `HOST_DISTRO` - The distro to use for SDK root filesystem. This
> variable is optional.
>   - `HOST_ARCH` - The Debian architecture of SDK root filesystem
> (e.g., `amd64`). By default set to current Debian host architecture.
> This variable is optional.
>   - `HOST_DISTRO_APT_SOURCES` - List of apt source files for SDK root
> filesystem. This variable is optional.
>   - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for
> SDK root filesystem. This variable is optional.
>   - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to
> DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt
> bootstrap repo for the host.
>   - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the
> default URI in the format `ftp.debian.org my.preferred.mirror`. This
> variable is optional. PREMIRRORS will be used only for the build. The
> final images will have the sources list as mentioned in
> DISTRO_APT_SOURCES.
> + - `ISAR_USE_DEBIAN_SNAPSHOTS` - Use a frozen debian snapshot
> instead of the live mirror. Auto-enabled if `SOURCE_DATE_EPOCH` is
> set. Optional.
> + - `ISAR_DEBIAN_SNAPSHOT_MIRROR` - The snapshot mirror to use.
> Defaults to `snapshot-cloudflare.debian.org`.
>   - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt
> repos for apt installation after bootstrapping.
>   - `FILESEXTRAPATHS` - The default directories BitBake uses when it
> processes recipes are initially defined by the FILESPATH variable.
> You can extend FILESPATH variable by using FILESEXTRAPATHS.
>   - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system
> for creating FILESPATH. The FILESOVERRIDES variable uses overrides to
> automatically extend the FILESPATH variable.
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index f548e202..1e5a2911 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -32,6 +32,9 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if
> bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR
>  BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if
> bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}"
>  BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if
> bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else
> 'BASE_DISTRO')}"
>  FILESEXTRAPATHS:append = ":${BBPATH}"
> +# reproducible builds
> +ISAR_USE_DEBIAN_SNAPSHOTS ??= "${@'1' if
> d.getVar('SOURCE_DATE_EPOCH') else '0'}"
> +ISAR_DEBIAN_SNAPSHOT_MIRROR ??= "snapshot-cloudflare.debian.org"
>  
>  inherit deb-dl-dir
>  
> @@ -111,9 +114,15 @@ def
> parse_aptsources_list_line(source_list_line):
>  
>  def get_apt_source_mirror(d, aptsources_entry_list):
>      import re
> +    import time
>  
>      if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>          premirrors = "\S*
> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"
> +    elif bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')):
> +        snapshot_mirror = d.getVar('ISAR_DEBIAN_SNAPSHOT_MIRROR')
> +        source_date_epoch = d.getVar('SOURCE_DATE_EPOCH') or
> int(time.time())
> +        snapshot_date = time.strftime('%Y%m%dT%H%M%SZ',
> time.gmtime(int(source_date_epoch)))
> +        premirrors = 'deb.debian.org/(.*)
> {}/archive/\\1/{}/\n'.format(snapshot_mirror, snapshot_date)
>      else:
>          premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or ""
>      mirror_list = [entry.split()

-- 
Siemens AG, Technology
Linux Expert Center