public inbox for isar-users@googlegroups.com
 help / color / mirror / Atom feed
* [PATCH 1/1] use debian snapshot mirror if SOURCE_DATE_EPOCH is set
@ 2024-04-03 11:33 Felix Moessbauer
  2024-04-03 12:08 ` MOESSBAUER, Felix
  0 siblings, 1 reply; 2+ messages in thread
From: Felix Moessbauer @ 2024-04-03 11:33 UTC (permalink / raw)
  To: isar-users
  Cc: jan.kiszka, venkata.pyla, kazuhiro3.hayashi, dinesh.kumar,
	Felix Moessbauer

In case the SOURCE_DATE_EPOCH variable is set, we switch the debian
mirror to a snapshot mirror. The used date is derived from the value of
SOURCE_DATE_EPOCH. Similar to the DISTRO_APT_PREMIRRORS, this mirror is
only injected temporarily during the build.

To further control the behavior, we introduce the following variables:

- ISAR_USE_DEBIAN_SNAPSHOTS: overwrite if a snapshot shall be used
- ISAR_DEBIAN_SNAPSHOT_MIRROR: The snapshot mirror to use (defaults to
  snapshot-cloudflare.debian.org)

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
Dear maintainers,

I'm not quite sure if the introduced variables also need to be added
to the vardeps of e.g. the bootstrap task. Please double check this.

Best regards,
Felix Moessbauer
Siemens AG

 RECIPE-API-CHANGELOG.md                             | 6 ++++++
 doc/user_manual.md                                  | 3 +++
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 9 +++++++++
 3 files changed, 18 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 6653ab43..c146d60c 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -583,3 +583,9 @@ Cross compiling kernel modules for distro kernels is not supported in debian.
 To simplify downstream kernel module builds, we automatically turn of cross
 compilation for a user-provided module when building it for a distro kernel.
 
+
+### Build against debian snapshot mirror if SOURCE_DATE_EPOCH is set
+
+In case the bitbake variable `SOURCE_DATE_EPOCH` is set, a debian snapshot
+mirror is used. This can be overwritten with `ISAR_USE_DEBIAN_SNAPSHOTS`.
+The snapshot to use is specified in `ISAR_DEBIAN_SNAPSHOT_MIRROR`.
diff --git a/doc/user_manual.md b/doc/user_manual.md
index 419d5339..227ce5f9 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -425,12 +425,15 @@ Some other variables include:
 
  - `IMAGE_INSTALL` - The list of custom packages to build and install to target image, please refer to relative chapter for more information.
  - `BB_NUMBER_THREADS` - The number of `bitbake` jobs that can be run in parallel. Please set this option according to your host CPU cores number.
+ - `SOURCE_DATE_EPOCH` - The unix timestamp passed to all tooling to make the results reproducible. This variable is optional.
  - `HOST_DISTRO` - The distro to use for SDK root filesystem. This variable is optional.
  - `HOST_ARCH` - The Debian architecture of SDK root filesystem (e.g., `amd64`). By default set to current Debian host architecture. This variable is optional.
  - `HOST_DISTRO_APT_SOURCES` - List of apt source files for SDK root filesystem. This variable is optional.
  - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for SDK root filesystem. This variable is optional.
  - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host.
  - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES.
+ - `ISAR_USE_DEBIAN_SNAPSHOTS` - Use a frozen debian snapshot instead of the live mirror. Auto-enabled if `SOURCE_DATE_EPOCH` is set. Optional.
+ - `ISAR_DEBIAN_SNAPSHOT_MIRROR` - The snapshot mirror to use. Defaults to `snapshot-cloudflare.debian.org`.
  - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping.
  - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS.
  - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable.
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index f548e202..1e5a2911 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -32,6 +32,9 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR
 BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}"
 BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}"
 FILESEXTRAPATHS:append = ":${BBPATH}"
+# reproducible builds
+ISAR_USE_DEBIAN_SNAPSHOTS ??= "${@'1' if d.getVar('SOURCE_DATE_EPOCH') else '0'}"
+ISAR_DEBIAN_SNAPSHOT_MIRROR ??= "snapshot-cloudflare.debian.org"
 
 inherit deb-dl-dir
 
@@ -111,9 +114,15 @@ def parse_aptsources_list_line(source_list_line):
 
 def get_apt_source_mirror(d, aptsources_entry_list):
     import re
+    import time
 
     if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
         premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"
+    elif bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')):
+        snapshot_mirror = d.getVar('ISAR_DEBIAN_SNAPSHOT_MIRROR')
+        source_date_epoch = d.getVar('SOURCE_DATE_EPOCH') or int(time.time())
+        snapshot_date = time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch)))
+        premirrors = 'deb.debian.org/(.*) {}/archive/\\1/{}/\n'.format(snapshot_mirror, snapshot_date)
     else:
         premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or ""
     mirror_list = [entry.split()
-- 
2.39.2


^ permalink raw reply	[flat|nested] 2+ messages in thread
end of thread, other threads:[~2024-04-03 12:09 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-04-03 11:33 [PATCH 1/1] use debian snapshot mirror if SOURCE_DATE_EPOCH is set Felix Moessbauer
2024-04-03 12:08 ` MOESSBAUER, Felix

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox