* [PATCH v3 1/5] add reproducible builds infrastructure from oe
2024-04-09 15:55 [PATCH v3 0/5] use debian snapshot mirror if SOURCE_DATE_EPOCH is set Felix Moessbauer
@ 2024-04-09 15:55 ` Felix Moessbauer
2024-04-09 15:55 ` [PATCH v3 2/5] use apt snapshot mirror if ISAR_USE_APT_SNAPSHOT is set Felix Moessbauer
` (5 subsequent siblings)
6 siblings, 0 replies; 10+ messages in thread
From: Felix Moessbauer @ 2024-04-09 15:55 UTC (permalink / raw)
To: isar-users; +Cc: jan.kiszka, quirin.gylstorff, Felix Moessbauer
This patch adds the reproducible builds infrastructure from OE. This
includes python helpers to determine the SOURCE_DATE_EPOCH per component
(if not set). Once determined, these values are written to the SDE_FILE
and picked up on the next build as a fallback.
Both this and the reproducible section of the bitbake.conf are taken 1:1
from OE 6548354 (corresponding commit to bitbake version currently used
by ISAR).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
meta/classes/base.bbclass | 17 +++-
meta/conf/bitbake.conf | 12 +++
meta/lib/oe/reproducible.py | 197 ++++++++++++++++++++++++++++++++++++
3 files changed, 225 insertions(+), 1 deletion(-)
create mode 100644 meta/lib/oe/reproducible.py
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 529811af..b8825bd3 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -21,7 +21,7 @@
THISDIR = "${@os.path.dirname(d.getVar('FILE'))}"
FILESPATH = "${@base_set_filespath(["${FILE_DIRNAME}/${PF}", "${FILE_DIRNAME}/${P}", "${FILE_DIRNAME}/${PN}", "${FILE_DIRNAME}/files", "${FILE_DIRNAME}"], d)}"
-OE_IMPORTS += "os sys time oe.path oe.patch oe.sstatesig oe.utils"
+OE_IMPORTS += "os sys time oe.path oe.patch oe.reproducible oe.sstatesig oe.utils"
OE_IMPORTS[type] = "list"
def oe_import(d):
@@ -318,3 +318,18 @@ def calculate_build_uuid(d):
# Unique ID for this build, used to avoid name clashes on external mountpoints
# When running parallel builds in different PID namespaces
ISAR_BUILD_UUID = "${@ calculate_build_uuid(d)}"
+
+do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
+do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
+addtask do_deploy_source_date_epoch_setscene
+addtask do_deploy_source_date_epoch before do_configure after do_patch
+
+python create_source_date_epoch_stamp() {
+ # Version: 1
+ source_date_epoch = oe.reproducible.get_source_date_epoch(d, d.getVar('S'))
+ oe.reproducible.epochfile_write(source_date_epoch, d.getVar('SDE_FILE'), d)
+}
+do_unpack[postfuncs] += "create_source_date_epoch_stamp"
+
+def get_source_date_epoch_value(d):
+ return oe.reproducible.epochfile_read(d.getVar('SDE_FILE'), d)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 91c5c815..1da3ecac 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -134,6 +134,18 @@ BB_NUMBER_THREADS ?= "${@bb.utils.cpu_count()}"
# Default to setting automatically based on cpu count
PARALLEL_MAKE ?= "-j ${@bb.utils.cpu_count()}"
+# Reproducibility (taken 1:1 from oe)
+SDE_DIR = "${WORKDIR}/source-date-epoch"
+SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt"
+SDE_DEPLOYDIR = "${WORKDIR}/deploy-source-date-epoch"
+
+export PYTHONHASHSEED = "0"
+export PERL_HASH_SEED = "0"
+export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
+# A SOURCE_DATE_EPOCH of '0' might be misinterpreted as no SDE
+# ISAR: set value to date of latest release
+SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251"
+
# Default parallelism and resource usage for xz
XZ_MEMLIMIT ?= "50%"
XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
diff --git a/meta/lib/oe/reproducible.py b/meta/lib/oe/reproducible.py
new file mode 100644
index 00000000..448befce
--- /dev/null
+++ b/meta/lib/oe/reproducible.py
@@ -0,0 +1,197 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+import os
+import subprocess
+import bb
+
+# For reproducible builds, this code sets the default SOURCE_DATE_EPOCH in each
+# component's build environment. The format is number of seconds since the
+# system epoch.
+#
+# Upstream components (generally) respect this environment variable,
+# using it in place of the "current" date and time.
+# See https://reproducible-builds.org/specs/source-date-epoch/
+#
+# The default value of SOURCE_DATE_EPOCH comes from the function
+# get_source_date_epoch_value which reads from the SDE_FILE, or if the file
+# is not available will use the fallback of SOURCE_DATE_EPOCH_FALLBACK.
+#
+# The SDE_FILE is normally constructed from the function
+# create_source_date_epoch_stamp which is typically added as a postfuncs to
+# the do_unpack task. If a recipe does NOT have do_unpack, it should be added
+# to a task that runs after the source is available and before the
+# do_deploy_source_date_epoch task is executed.
+#
+# If a recipe wishes to override the default behavior it should set it's own
+# SOURCE_DATE_EPOCH or override the do_deploy_source_date_epoch_stamp task
+# with recipe-specific functionality to write the appropriate
+# SOURCE_DATE_EPOCH into the SDE_FILE.
+#
+# SOURCE_DATE_EPOCH is intended to be a reproducible value. This value should
+# be reproducible for anyone who builds the same revision from the same
+# sources.
+#
+# There are 4 ways the create_source_date_epoch_stamp function determines what
+# becomes SOURCE_DATE_EPOCH:
+#
+# 1. Use the value from __source_date_epoch.txt file if this file exists.
+# This file was most likely created in the previous build by one of the
+# following methods 2,3,4.
+# Alternatively, it can be provided by a recipe via SRC_URI.
+#
+# If the file does not exist:
+#
+# 2. If there is a git checkout, use the last git commit timestamp.
+# Git does not preserve file timestamps on checkout.
+#
+# 3. Use the mtime of "known" files such as NEWS, CHANGLELOG, ...
+# This works for well-kept repositories distributed via tarball.
+#
+# 4. Use the modification time of the youngest file in the source tree, if
+# there is one.
+# This will be the newest file from the distribution tarball, if any.
+#
+# 5. Fall back to a fixed timestamp (SOURCE_DATE_EPOCH_FALLBACK).
+#
+# Once the value is determined, it is stored in the recipe's SDE_FILE.
+
+def get_source_date_epoch_from_known_files(d, sourcedir):
+ source_date_epoch = None
+ newest_file = None
+ known_files = set(["NEWS", "ChangeLog", "Changelog", "CHANGES"])
+ for file in known_files:
+ filepath = os.path.join(sourcedir, file)
+ if os.path.isfile(filepath):
+ mtime = int(os.lstat(filepath).st_mtime)
+ # There may be more than one "known_file" present, if so, use the youngest one
+ if not source_date_epoch or mtime > source_date_epoch:
+ source_date_epoch = mtime
+ newest_file = filepath
+ if newest_file:
+ bb.debug(1, "SOURCE_DATE_EPOCH taken from: %s" % newest_file)
+ return source_date_epoch
+
+def find_git_folder(d, sourcedir):
+ # First guess: WORKDIR/git
+ # This is the default git fetcher unpack path
+ workdir = d.getVar('WORKDIR')
+ gitpath = os.path.join(workdir, "git/.git")
+ if os.path.isdir(gitpath):
+ return gitpath
+
+ # Second guess: ${S}
+ gitpath = os.path.join(sourcedir, ".git")
+ if os.path.isdir(gitpath):
+ return gitpath
+
+ # Perhaps there was a subpath or destsuffix specified.
+ # Go looking in the WORKDIR
+ exclude = set(["build", "image", "license-destdir", "patches", "pseudo",
+ "recipe-sysroot", "recipe-sysroot-native", "sysroot-destdir", "temp"])
+ for root, dirs, files in os.walk(workdir, topdown=True):
+ dirs[:] = [d for d in dirs if d not in exclude]
+ if '.git' in dirs:
+ return os.path.join(root, ".git")
+
+ bb.warn("Failed to find a git repository in WORKDIR: %s" % workdir)
+ return None
+
+def get_source_date_epoch_from_git(d, sourcedir):
+ if not "git://" in d.getVar('SRC_URI') and not "gitsm://" in d.getVar('SRC_URI'):
+ return None
+
+ gitpath = find_git_folder(d, sourcedir)
+ if not gitpath:
+ return None
+
+ # Check that the repository has a valid HEAD; it may not if subdir is used
+ # in SRC_URI
+ p = subprocess.run(['git', '--git-dir', gitpath, 'rev-parse', 'HEAD'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+ if p.returncode != 0:
+ bb.debug(1, "%s does not have a valid HEAD: %s" % (gitpath, p.stdout.decode('utf-8')))
+ return None
+
+ bb.debug(1, "git repository: %s" % gitpath)
+ p = subprocess.run(['git', '-c', 'log.showSignature=false', '--git-dir', gitpath, 'log', '-1', '--pretty=%ct'],
+ check=True, stdout=subprocess.PIPE)
+ return int(p.stdout.decode('utf-8'))
+
+def get_source_date_epoch_from_youngest_file(d, sourcedir):
+ if sourcedir == d.getVar('WORKDIR'):
+ # These sources are almost certainly not from a tarball
+ return None
+
+ # Do it the hard way: check all files and find the youngest one...
+ source_date_epoch = None
+ newest_file = None
+ for root, dirs, files in os.walk(sourcedir, topdown=True):
+ files = [f for f in files if not f[0] == '.']
+
+ for fname in files:
+ if fname == "singletask.lock":
+ # Ignore externalsrc/devtool lockfile [YOCTO #14921]
+ continue
+ filename = os.path.join(root, fname)
+ try:
+ mtime = int(os.lstat(filename).st_mtime)
+ except ValueError:
+ mtime = 0
+ if not source_date_epoch or mtime > source_date_epoch:
+ source_date_epoch = mtime
+ newest_file = filename
+
+ if newest_file:
+ bb.debug(1, "Newest file found: %s" % newest_file)
+ return source_date_epoch
+
+def fixed_source_date_epoch(d):
+ bb.debug(1, "No tarball or git repo found to determine SOURCE_DATE_EPOCH")
+ source_date_epoch = d.getVar('SOURCE_DATE_EPOCH_FALLBACK')
+ if source_date_epoch:
+ bb.debug(1, "Using SOURCE_DATE_EPOCH_FALLBACK")
+ return int(source_date_epoch)
+ return 0
+
+def get_source_date_epoch(d, sourcedir):
+ return (
+ get_source_date_epoch_from_git(d, sourcedir) or
+ get_source_date_epoch_from_youngest_file(d, sourcedir) or
+ fixed_source_date_epoch(d) # Last resort
+ )
+
+def epochfile_read(epochfile, d):
+ cached, efile = d.getVar('__CACHED_SOURCE_DATE_EPOCH') or (None, None)
+ if cached and efile == epochfile:
+ return cached
+
+ if cached and epochfile != efile:
+ bb.debug(1, "Epoch file changed from %s to %s" % (efile, epochfile))
+
+ source_date_epoch = int(d.getVar('SOURCE_DATE_EPOCH_FALLBACK'))
+ try:
+ with open(epochfile, 'r') as f:
+ s = f.read()
+ try:
+ source_date_epoch = int(s)
+ except ValueError:
+ bb.warn("SOURCE_DATE_EPOCH value '%s' is invalid. Reverting to SOURCE_DATE_EPOCH_FALLBACK" % s)
+ source_date_epoch = int(d.getVar('SOURCE_DATE_EPOCH_FALLBACK'))
+ bb.debug(1, "SOURCE_DATE_EPOCH: %d" % source_date_epoch)
+ except FileNotFoundError:
+ bb.debug(1, "Cannot find %s. SOURCE_DATE_EPOCH will default to %d" % (epochfile, source_date_epoch))
+
+ d.setVar('__CACHED_SOURCE_DATE_EPOCH', (str(source_date_epoch), epochfile))
+ return str(source_date_epoch)
+
+def epochfile_write(source_date_epoch, epochfile, d):
+
+ bb.debug(1, "SOURCE_DATE_EPOCH: %d" % source_date_epoch)
+ bb.utils.mkdirhier(os.path.dirname(epochfile))
+
+ tmp_file = "%s.new" % epochfile
+ with open(tmp_file, 'w') as f:
+ f.write(str(source_date_epoch))
+ os.rename(tmp_file, epochfile)
--
2.39.2
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH v3 2/5] use apt snapshot mirror if ISAR_USE_APT_SNAPSHOT is set
2024-04-09 15:55 [PATCH v3 0/5] use debian snapshot mirror if SOURCE_DATE_EPOCH is set Felix Moessbauer
2024-04-09 15:55 ` [PATCH v3 1/5] add reproducible builds infrastructure from oe Felix Moessbauer
@ 2024-04-09 15:55 ` Felix Moessbauer
2024-06-06 13:15 ` Anton Mikanovich
2024-04-09 15:55 ` [PATCH v3 3/5] add kas menu options to build against snapshots Felix Moessbauer
` (4 subsequent siblings)
6 siblings, 1 reply; 10+ messages in thread
From: Felix Moessbauer @ 2024-04-09 15:55 UTC (permalink / raw)
To: isar-users; +Cc: jan.kiszka, quirin.gylstorff, Felix Moessbauer
This patch adds infrastructure to switch the apt sources to a frozen
snapshot mirror. To build against a mirror, set ISAR_USE_APT_SNAPSHOT=1.
As the mirror is distro specific, it is configured in the distro config
for all supported distros (currently only debian and ubuntu). For
unsupported distros, a meaningful error message is emitted on enabling
the snapshot build. Similar to the DISTRO_APT_PREMIRRORS, this mirror is
only injected temporarily and does not end up in the final apt sources
list.
To further control the behavior, we introduce the following variables:
- DISTRO_APT_SNAPSHOT_PREMIRROR: The snapshot mirror to use. Syntax
identical to DISTRO_APT_PREMIRRORS.
- ISAR_APT_SNAPSHOT_TIMESTAMP: Unix timestamp of the snapshot. This is
automatically derived from the SOURCE_DATE_EPOCH if not set.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
RECIPE-API-CHANGELOG.md | 6 ++++++
doc/user_manual.md | 3 +++
meta-isar/conf/distro/ubuntu-common.inc | 3 +++
meta/conf/bitbake.conf | 3 +++
meta/conf/distro/debian-common.conf | 3 +++
.../isar-bootstrap/isar-bootstrap.inc | 16 ++++++++++++++++
6 files changed, 34 insertions(+)
diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 6653ab43..e6861523 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -583,3 +583,9 @@ Cross compiling kernel modules for distro kernels is not supported in debian.
To simplify downstream kernel module builds, we automatically turn of cross
compilation for a user-provided module when building it for a distro kernel.
+
+### Build against debian snapshot mirror
+
+To build against a distributions snapshot mirror, set `ISAR_USE_APT_SNAPSHOT="1"`.
+The mirror to use is specified in `DISTRO_APT_SNAPSHOT_PREMIRROR` and usually
+pre-defined in the distro config.
diff --git a/doc/user_manual.md b/doc/user_manual.md
index 419d5339..70741968 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -431,6 +431,9 @@ Some other variables include:
- `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for SDK root filesystem. This variable is optional.
- `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host.
- `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES.
+ - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of the live mirror. Optional.
+ - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported distros.
+ - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Timestamp of the apt snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not overwritten.
- `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping.
- `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS.
- `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable.
diff --git a/meta-isar/conf/distro/ubuntu-common.inc b/meta-isar/conf/distro/ubuntu-common.inc
index 9d8a843b..54bb747a 100644
--- a/meta-isar/conf/distro/ubuntu-common.inc
+++ b/meta-isar/conf/distro/ubuntu-common.inc
@@ -32,3 +32,6 @@ IMAGE_PREINSTALL += "init"
IMAGE_PREINSTALL += "initramfs-tools"
IMAGER_INSTALL:wic += "python3-distutils"
+
+# snapshot mirror for reproducible builds
+DISTRO_APT_SNAPSHOT_PREMIRROR ??= "(http|https)://archive.ubuntu.com/(.*) https://snapshot.ubuntu.com/\2/${APT_SNAPSHOT_DATE}/\n"
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 1da3ecac..4cfa8b10 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -145,6 +145,9 @@ export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
# A SOURCE_DATE_EPOCH of '0' might be misinterpreted as no SDE
# ISAR: set value to date of latest release
SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251"
+# Debian snapshots
+ISAR_USE_APT_SNAPSHOT ??= "0"
+ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}"
# Default parallelism and resource usage for xz
XZ_MEMLIMIT ?= "50%"
diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf
index 1e1dfc83..db538510 100644
--- a/meta/conf/distro/debian-common.conf
+++ b/meta/conf/distro/debian-common.conf
@@ -39,3 +39,6 @@ SYSTEMD_BOOTLOADER_INSTALL:sid = "systemd-boot-efi:${DISTRO_ARCH}"
COMPAT_DISTRO_ARCH:amd64 = "i386"
COMPAT_DISTRO_ARCH:arm64 = "armhf"
+
+# snapshot mirror for reproducible builds
+DISTRO_APT_SNAPSHOT_PREMIRROR ??= "deb.debian.org/(.*) snapshot-cloudflare.debian.org/archive/\1/${APT_SNAPSHOT_DATE}/\n"
\ No newline at end of file
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 17f19fd8..733a23df 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -30,6 +30,9 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR
BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}"
BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}"
FILESEXTRAPATHS:append = ":${BBPATH}"
+# reproducible builds, only enabled if ISAR_USE_APT_SNAPSHOT
+ISAR_APT_SNAPSHOT_MIRROR ??= ""
+APT_SNAPSHOT_DATE = "${@ get_apt_snapshot_date(d)}"
inherit deb-dl-dir
@@ -107,11 +110,21 @@ def parse_aptsources_list_line(source_list_line):
return [type, options, source, suite, components]
+def get_apt_snapshot_date(d):
+ import time
+ source_date_epoch = d.getVar('ISAR_APT_SNAPSHOT_TIMESTAMP')
+ return time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch)))
+
def get_apt_source_mirror(d, aptsources_entry_list):
import re
+ # this is executed during parsing. No error checking possible
+ use_snapshot = bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT'))
+ snapshot_mirror = d.getVar('DISTRO_APT_SNAPSHOT_PREMIRROR')
if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"
+ elif use_snapshot and snapshot_mirror:
+ premirrors = snapshot_mirror
else:
premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or ""
mirror_list = [entry.split()
@@ -126,6 +139,8 @@ def get_apt_source_mirror(d, aptsources_entry_list):
new_aptsources_entry_list[2] = re.sub(regex, replace,
aptsources_entry_list[2],
count = 1)
+ if use_snapshot:
+ new_aptsources_entry_list[1] = "[check-valid-until=no]"
return new_aptsources_entry_list
return aptsources_entry_list
@@ -240,6 +255,7 @@ do_apt_config_prepare[vardeps] += " \
APTSRCS \
${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
DEPLOY_ISAR_BOOTSTRAP \
+ ${@'DISTRO_APT_SNAPSHOT_PREMIRROR' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''} \
"
python do_apt_config_prepare() {
apt_preferences_out = d.getVar("APTPREFS")
--
2.39.2
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH v3 2/5] use apt snapshot mirror if ISAR_USE_APT_SNAPSHOT is set
2024-04-09 15:55 ` [PATCH v3 2/5] use apt snapshot mirror if ISAR_USE_APT_SNAPSHOT is set Felix Moessbauer
@ 2024-06-06 13:15 ` Anton Mikanovich
2024-06-10 9:22 ` MOESSBAUER, Felix
0 siblings, 1 reply; 10+ messages in thread
From: Anton Mikanovich @ 2024-06-06 13:15 UTC (permalink / raw)
To: Felix Moessbauer, isar-users
09/04/2024 18:55, 'Felix Moessbauer' via isar-users wrote:
> This patch adds infrastructure to switch the apt sources to a frozen
> snapshot mirror. To build against a mirror, set ISAR_USE_APT_SNAPSHOT=1.
> As the mirror is distro specific, it is configured in the distro config
> for all supported distros (currently only debian and ubuntu). For
> unsupported distros, a meaningful error message is emitted on enabling
> the snapshot build. Similar to the DISTRO_APT_PREMIRRORS, this mirror is
> only injected temporarily and does not end up in the final apt sources
> list.
>
> To further control the behavior, we introduce the following variables:
>
> - DISTRO_APT_SNAPSHOT_PREMIRROR: The snapshot mirror to use. Syntax
> identical to DISTRO_APT_PREMIRRORS.
> - ISAR_APT_SNAPSHOT_TIMESTAMP: Unix timestamp of the snapshot. This is
> automatically derived from the SOURCE_DATE_EPOCH if not set.
>
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
> RECIPE-API-CHANGELOG.md | 6 ++++++
> doc/user_manual.md | 3 +++
> meta-isar/conf/distro/ubuntu-common.inc | 3 +++
> meta/conf/bitbake.conf | 3 +++
> meta/conf/distro/debian-common.conf | 3 +++
> .../isar-bootstrap/isar-bootstrap.inc | 16 ++++++++++++++++
> 6 files changed, 34 insertions(+)
>
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index 6653ab43..e6861523 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -583,3 +583,9 @@ Cross compiling kernel modules for distro kernels is not supported in debian.
> To simplify downstream kernel module builds, we automatically turn of cross
> compilation for a user-provided module when building it for a distro kernel.
>
> +
> +### Build against debian snapshot mirror
> +
> +To build against a distributions snapshot mirror, set `ISAR_USE_APT_SNAPSHOT="1"`.
> +The mirror to use is specified in `DISTRO_APT_SNAPSHOT_PREMIRROR` and usually
> +pre-defined in the distro config.
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 419d5339..70741968 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -431,6 +431,9 @@ Some other variables include:
> - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for SDK root filesystem. This variable is optional.
> - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host.
> - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES.
> + - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of the live mirror. Optional.
> + - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported distros.
> + - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Timestamp of the apt snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not overwritten.
> - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping.
> - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS.
> - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable.
> diff --git a/meta-isar/conf/distro/ubuntu-common.inc b/meta-isar/conf/distro/ubuntu-common.inc
> index 9d8a843b..54bb747a 100644
> --- a/meta-isar/conf/distro/ubuntu-common.inc
> +++ b/meta-isar/conf/distro/ubuntu-common.inc
> @@ -32,3 +32,6 @@ IMAGE_PREINSTALL += "init"
> IMAGE_PREINSTALL += "initramfs-tools"
>
> IMAGER_INSTALL:wic += "python3-distutils"
> +
> +# snapshot mirror for reproducible builds
> +DISTRO_APT_SNAPSHOT_PREMIRROR ??= "(http|https)://archive.ubuntu.com/(.*) https://snapshot.ubuntu.com/\2/${APT_SNAPSHOT_DATE}/\n"
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 1da3ecac..4cfa8b10 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -145,6 +145,9 @@ export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
> # A SOURCE_DATE_EPOCH of '0' might be misinterpreted as no SDE
> # ISAR: set value to date of latest release
> SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251"
> +# Debian snapshots
> +ISAR_USE_APT_SNAPSHOT ??= "0"
> +ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}"
>
> # Default parallelism and resource usage for xz
> XZ_MEMLIMIT ?= "50%"
> diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf
> index 1e1dfc83..db538510 100644
> --- a/meta/conf/distro/debian-common.conf
> +++ b/meta/conf/distro/debian-common.conf
> @@ -39,3 +39,6 @@ SYSTEMD_BOOTLOADER_INSTALL:sid = "systemd-boot-efi:${DISTRO_ARCH}"
>
> COMPAT_DISTRO_ARCH:amd64 = "i386"
> COMPAT_DISTRO_ARCH:arm64 = "armhf"
> +
> +# snapshot mirror for reproducible builds
> +DISTRO_APT_SNAPSHOT_PREMIRROR ??= "deb.debian.org/(.*) snapshot-cloudflare.debian.org/archive/\1/${APT_SNAPSHOT_DATE}/\n"
> \ No newline at end of file
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index 17f19fd8..733a23df 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -30,6 +30,9 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR
> BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}"
> BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}"
> FILESEXTRAPATHS:append = ":${BBPATH}"
> +# reproducible builds, only enabled if ISAR_USE_APT_SNAPSHOT
> +ISAR_APT_SNAPSHOT_MIRROR ??= ""
> +APT_SNAPSHOT_DATE = "${@ get_apt_snapshot_date(d)}"
>
> inherit deb-dl-dir
>
> @@ -107,11 +110,21 @@ def parse_aptsources_list_line(source_list_line):
>
> return [type, options, source, suite, components]
>
> +def get_apt_snapshot_date(d):
> + import time
> + source_date_epoch = d.getVar('ISAR_APT_SNAPSHOT_TIMESTAMP')
> + return time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch)))
> +
> def get_apt_source_mirror(d, aptsources_entry_list):
> import re
>
> + # this is executed during parsing. No error checking possible
> + use_snapshot = bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT'))
> + snapshot_mirror = d.getVar('DISTRO_APT_SNAPSHOT_PREMIRROR')
> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"
> + elif use_snapshot and snapshot_mirror:
> + premirrors = snapshot_mirror
> else:
> premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or ""
> mirror_list = [entry.split()
> @@ -126,6 +139,8 @@ def get_apt_source_mirror(d, aptsources_entry_list):
> new_aptsources_entry_list[2] = re.sub(regex, replace,
> aptsources_entry_list[2],
> count = 1)
> + if use_snapshot:
> + new_aptsources_entry_list[1] = "[check-valid-until=no]"
> return new_aptsources_entry_list
>
> return aptsources_entry_list
> @@ -240,6 +255,7 @@ do_apt_config_prepare[vardeps] += " \
> APTSRCS \
> ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
> DEPLOY_ISAR_BOOTSTRAP \
> + ${@'DISTRO_APT_SNAPSHOT_PREMIRROR' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''} \
> "
> python do_apt_config_prepare() {
> apt_preferences_out = d.getVar("APTPREFS")
Hello Felix,
How ISAR_APT_SNAPSHOT_MIRROR supposed to be used?
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH v3 2/5] use apt snapshot mirror if ISAR_USE_APT_SNAPSHOT is set
2024-06-06 13:15 ` Anton Mikanovich
@ 2024-06-10 9:22 ` MOESSBAUER, Felix
0 siblings, 0 replies; 10+ messages in thread
From: MOESSBAUER, Felix @ 2024-06-10 9:22 UTC (permalink / raw)
To: amikan, isar-users
On Thu, 2024-06-06 at 16:15 +0300, Anton Mikanovich wrote:
> 09/04/2024 18:55, 'Felix Moessbauer' via isar-users wrote:
> > This patch adds infrastructure to switch the apt sources to a
> > frozen
> > snapshot mirror. To build against a mirror, set
> > ISAR_USE_APT_SNAPSHOT=1.
> > As the mirror is distro specific, it is configured in the distro
> > config
> > for all supported distros (currently only debian and ubuntu). For
> > unsupported distros, a meaningful error message is emitted on
> > enabling
> > the snapshot build. Similar to the DISTRO_APT_PREMIRRORS, this
> > mirror is
> > only injected temporarily and does not end up in the final apt
> > sources
> > list.
> >
> > To further control the behavior, we introduce the following
> > variables:
> >
> > - DISTRO_APT_SNAPSHOT_PREMIRROR: The snapshot mirror to use. Syntax
> > identical to DISTRO_APT_PREMIRRORS.
> > - ISAR_APT_SNAPSHOT_TIMESTAMP: Unix timestamp of the snapshot. This
> > is
> > automatically derived from the SOURCE_DATE_EPOCH if not set.
> >
> > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> > ---
>
> Hello Felix,
>
> How ISAR_APT_SNAPSHOT_MIRROR supposed to be used?
Hi,
it looks like this variable is a leftover and now is called
DISTRO_APT_SNAPSHOT_PREMIRROR. The variable ISAR_APT_SNAPSHOT_MIRROR is
nowhere used and can be removed.
I'll prepare a patch.
Best regards,
Felix
--
Siemens AG, Technology
Linux Expert Center
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v3 3/5] add kas menu options to build against snapshots
2024-04-09 15:55 [PATCH v3 0/5] use debian snapshot mirror if SOURCE_DATE_EPOCH is set Felix Moessbauer
2024-04-09 15:55 ` [PATCH v3 1/5] add reproducible builds infrastructure from oe Felix Moessbauer
2024-04-09 15:55 ` [PATCH v3 2/5] use apt snapshot mirror if ISAR_USE_APT_SNAPSHOT is set Felix Moessbauer
@ 2024-04-09 15:55 ` Felix Moessbauer
2024-04-09 15:55 ` [PATCH v3 4/5] change the proposed interface to set the SDE Felix Moessbauer
` (3 subsequent siblings)
6 siblings, 0 replies; 10+ messages in thread
From: Felix Moessbauer @ 2024-04-09 15:55 UTC (permalink / raw)
To: isar-users; +Cc: jan.kiszka, quirin.gylstorff, Felix Moessbauer
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
kas/opt/Kconfig | 11 +++++++++++
kas/opt/mirror-snapshot.yaml | 9 +++++++++
2 files changed, 20 insertions(+)
create mode 100644 kas/opt/mirror-snapshot.yaml
diff --git a/kas/opt/Kconfig b/kas/opt/Kconfig
index 1619a4cf..701bd263 100644
--- a/kas/opt/Kconfig
+++ b/kas/opt/Kconfig
@@ -40,6 +40,17 @@ config KAS_INCLUDE_MIRROR_UBUNTU
default "kas/opt/mirror-ubuntu.yaml"
depends on MIRROR_UBUNTU
+config USE_APT_SNAPSHOT
+ bool "Use the distros snapshot mirror"
+ depends on !MIRROR_DEBIAN && (DEBIAN_BUSTER || DEBIAN_BULLSEYE || DEBIAN_BOOKWORM || DEBIAN_TRIXIE || UBUNTU_FOCAL || UBUNTU_JAMMY)
+ help
+ Use a snapshot mirror for the selected distribution. The date is defined by ISAR_APT_SNAPSHOT_TIMESTAMP.
+
+config KAS_INCLUDE_APT_SNAPSHOT
+ string
+ default "kas/opt/mirror-snapshot.yaml"
+ depends on USE_APT_SNAPSHOT
+
endmenu
diff --git a/kas/opt/mirror-snapshot.yaml b/kas/opt/mirror-snapshot.yaml
new file mode 100644
index 00000000..59fae192
--- /dev/null
+++ b/kas/opt/mirror-snapshot.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (C) 2024 Siemens AG
+
+header:
+ version: 14
+
+local_conf_header:
+ mirror-debian: |
+ ISAR_USE_APT_SNAPSHOT = "1"
--
2.39.2
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH v3 4/5] change the proposed interface to set the SDE
2024-04-09 15:55 [PATCH v3 0/5] use debian snapshot mirror if SOURCE_DATE_EPOCH is set Felix Moessbauer
` (2 preceding siblings ...)
2024-04-09 15:55 ` [PATCH v3 3/5] add kas menu options to build against snapshots Felix Moessbauer
@ 2024-04-09 15:55 ` Felix Moessbauer
2024-04-09 15:55 ` [PATCH v3 5/5] ci: use snapshot in reproducible builds test Felix Moessbauer
` (2 subsequent siblings)
6 siblings, 0 replies; 10+ messages in thread
From: Felix Moessbauer @ 2024-04-09 15:55 UTC (permalink / raw)
To: isar-users; +Cc: jan.kiszka, quirin.gylstorff, Felix Moessbauer
To align the handling of source timestamps with OE, we propose to not
set the SOURCE_DATE_EPOCH variable directly, but via
`SOURCE_DATE_EPOCH_FALLBACK`. By that, we internally can use heuristics
to better estimate the value of the SDE, while still having a global
fallback if it cannot be estimated otherwise.
Please note, that change is backwards compatible. You can still set the
SOURCE_DATE_EPOCH variable directly and get the same behavior as before.
This patch also updates the reproducibility test according to the new
interface.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
RECIPE-API-CHANGELOG.md | 6 ++++++
doc/user_manual.md | 1 +
testsuite/cibuilder.py | 2 +-
3 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index e6861523..4afe8b16 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -589,3 +589,9 @@ compilation for a user-provided module when building it for a distro kernel.
To build against a distributions snapshot mirror, set `ISAR_USE_APT_SNAPSHOT="1"`.
The mirror to use is specified in `DISTRO_APT_SNAPSHOT_PREMIRROR` and usually
pre-defined in the distro config.
+
+### Use OE interface to set timestamp for reproducible builds
+
+The `SOURCE_DATE_EPOCH` (SDE) should not be set globally, but on a per-recipe basis
+and to meaningful values. As a global fallback, set the `SOURCE_DATE_EPOCH_FALLBACK`
+bitbake variable to the desired unix timestamp.
diff --git a/doc/user_manual.md b/doc/user_manual.md
index 70741968..b12c7692 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -425,6 +425,7 @@ Some other variables include:
- `IMAGE_INSTALL` - The list of custom packages to build and install to target image, please refer to relative chapter for more information.
- `BB_NUMBER_THREADS` - The number of `bitbake` jobs that can be run in parallel. Please set this option according to your host CPU cores number.
+ - `SOURCE_DATE_EPOCH_FALLBACK` - The unix timestamp passed to all tooling to make the results reproducible. This variable is optional.
- `HOST_DISTRO` - The distro to use for SDK root filesystem. This variable is optional.
- `HOST_ARCH` - The Debian architecture of SDK root filesystem (e.g., `amd64`). By default set to current Debian host architecture. This variable is optional.
- `HOST_DISTRO_APT_SOURCES` - List of apt source files for SDK root filesystem. This variable is optional.
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index fa30c2f5..7e69dafe 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -159,7 +159,7 @@ class CIBuilder(Test):
f.write('USE_CCACHE = "1"\n')
f.write('CCACHE_TOP_DIR = "%s"\n' % ccache_dir)
if source_date_epoch:
- f.write('SOURCE_DATE_EPOCH = "%s"\n' % source_date_epoch)
+ f.write('SOURCE_DATE_EPOCH_FALLBACK = "%s"\n' % source_date_epoch)
if dl_dir:
f.write('DL_DIR = "%s"\n' % dl_dir)
if sstate_dir:
--
2.39.2
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH v3 5/5] ci: use snapshot in reproducible builds test
2024-04-09 15:55 [PATCH v3 0/5] use debian snapshot mirror if SOURCE_DATE_EPOCH is set Felix Moessbauer
` (3 preceding siblings ...)
2024-04-09 15:55 ` [PATCH v3 4/5] change the proposed interface to set the SDE Felix Moessbauer
@ 2024-04-09 15:55 ` Felix Moessbauer
2024-04-15 9:39 ` [PATCH v3 0/5] use debian snapshot mirror if SOURCE_DATE_EPOCH is set Uladzimir Bely
2024-04-17 8:06 ` Uladzimir Bely
6 siblings, 0 replies; 10+ messages in thread
From: Felix Moessbauer @ 2024-04-09 15:55 UTC (permalink / raw)
To: isar-users; +Cc: jan.kiszka, quirin.gylstorff, Felix Moessbauer
As we now have easy to use snapshot support, we also switch over the
reproducible builds test to use a snapshot. By that, we both test the
snapshot support, as well as make sure to always build against a common
set of packages.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
testsuite/cibuilder.py | 6 +++++-
testsuite/repro-build-test.py | 2 +-
2 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 7e69dafe..039355e5 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -78,7 +78,8 @@ class CIBuilder(Test):
container=False, ccache=False, sstate=False, offline=False,
gpg_pub_key=None, wic_deploy_parts=False, dl_dir=None,
sstate_dir=None, ccache_dir=None,
- source_date_epoch=None, image_install=None, **kwargs):
+ source_date_epoch=None, use_apt_snapshot=False,
+ image_install=None, **kwargs):
# write configuration file and set bitbake_args
# can run multiple times per test case
self.check_init()
@@ -116,6 +117,7 @@ class CIBuilder(Test):
f' gpg_pub_key = {gpg_pub_key}\n'
f' wic_deploy_parts = {wic_deploy_parts}\n'
f' source_date_epoch = {source_date_epoch} \n'
+ f' use_apt_snapshot = {use_apt_snapshot} \n'
f' dl_dir = {dl_dir}\n'
f' sstate_dir = {sstate_dir}\n'
f' ccache_dir = {ccache_dir}\n'
@@ -160,6 +162,8 @@ class CIBuilder(Test):
f.write('CCACHE_TOP_DIR = "%s"\n' % ccache_dir)
if source_date_epoch:
f.write('SOURCE_DATE_EPOCH_FALLBACK = "%s"\n' % source_date_epoch)
+ if use_apt_snapshot:
+ f.write('ISAR_USE_APT_SNAPSHOT = "1"\n')
if dl_dir:
f.write('DL_DIR = "%s"\n' % dl_dir)
if sstate_dir:
diff --git a/testsuite/repro-build-test.py b/testsuite/repro-build-test.py
index 040a844e..843f3bc6 100755
--- a/testsuite/repro-build-test.py
+++ b/testsuite/repro-build-test.py
@@ -46,7 +46,7 @@ class ReproBuild(CIBuilder):
# Build
self.log.info("Started Build " + image_name)
- self.configure(source_date_epoch=source_date_epoch)
+ self.configure(source_date_epoch=source_date_epoch, use_apt_snapshot=True)
self.bitbake(target)
# copy the artifacts image name with given name
--
2.39.2
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH v3 0/5] use debian snapshot mirror if SOURCE_DATE_EPOCH is set
2024-04-09 15:55 [PATCH v3 0/5] use debian snapshot mirror if SOURCE_DATE_EPOCH is set Felix Moessbauer
` (4 preceding siblings ...)
2024-04-09 15:55 ` [PATCH v3 5/5] ci: use snapshot in reproducible builds test Felix Moessbauer
@ 2024-04-15 9:39 ` Uladzimir Bely
2024-04-17 8:06 ` Uladzimir Bely
6 siblings, 0 replies; 10+ messages in thread
From: Uladzimir Bely @ 2024-04-15 9:39 UTC (permalink / raw)
To: Felix Moessbauer, isar-users; +Cc: jan.kiszka, quirin.gylstorff
On Tue, 2024-04-09 at 17:55 +0200, 'Felix Moessbauer' via isar-users
wrote:
> This patch series is a continuation of the idea to add OOTB support
> for apt snapshot mirrors. In contrast to the first two versions, this
> series approaches the topic at a broader scope.
>
> Changes since v2:
>
> - make the reproducible builds support more similar with OE
> - support per-distro snapshots
> - add support for snapshot builds in the kas menu
> - add a test
>
> Best regards,
> Felix Moessbauer
> Siemens AG
>
> Felix Moessbauer (5):
> add reproducible builds infrastructure from oe
> use apt snapshot mirror if ISAR_USE_APT_SNAPSHOT is set
> add kas menu options to build against snapshots
> change the proposed interface to set the SDE
> ci: use snapshot in reproducible builds test
>
> RECIPE-API-CHANGELOG.md | 12 ++
> doc/user_manual.md | 4 +
> kas/opt/Kconfig | 11 +
> kas/opt/mirror-snapshot.yaml | 9 +
> meta-isar/conf/distro/ubuntu-common.inc | 3 +
> meta/classes/base.bbclass | 17 +-
> meta/conf/bitbake.conf | 15 ++
> meta/conf/distro/debian-common.conf | 3 +
> meta/lib/oe/reproducible.py | 197
> ++++++++++++++++++
> .../isar-bootstrap/isar-bootstrap.inc | 16 ++
> testsuite/cibuilder.py | 8 +-
> testsuite/repro-build-test.py | 2 +-
> 12 files changed, 293 insertions(+), 4 deletions(-)
> create mode 100644 kas/opt/mirror-snapshot.yaml
> create mode 100644 meta/lib/oe/reproducible.py
>
> --
> 2.39.2
>
The patchset passes CI. If there are no objections/additions, we are
ready to merge it.
---
Regards,
Uladzimir Bely.
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH v3 0/5] use debian snapshot mirror if SOURCE_DATE_EPOCH is set
2024-04-09 15:55 [PATCH v3 0/5] use debian snapshot mirror if SOURCE_DATE_EPOCH is set Felix Moessbauer
` (5 preceding siblings ...)
2024-04-15 9:39 ` [PATCH v3 0/5] use debian snapshot mirror if SOURCE_DATE_EPOCH is set Uladzimir Bely
@ 2024-04-17 8:06 ` Uladzimir Bely
6 siblings, 0 replies; 10+ messages in thread
From: Uladzimir Bely @ 2024-04-17 8:06 UTC (permalink / raw)
To: Felix Moessbauer, isar-users
On Tue, 2024-04-09 at 17:55 +0200, 'Felix Moessbauer' via isar-users
wrote:
> This patch series is a continuation of the idea to add OOTB support
> for apt snapshot mirrors. In contrast to the first two versions, this
> series approaches the topic at a broader scope.
>
> Changes since v2:
>
> - make the reproducible builds support more similar with OE
> - support per-distro snapshots
> - add support for snapshot builds in the kas menu
> - add a test
>
> Best regards,
> Felix Moessbauer
> Siemens AG
>
> Felix Moessbauer (5):
> add reproducible builds infrastructure from oe
> use apt snapshot mirror if ISAR_USE_APT_SNAPSHOT is set
> add kas menu options to build against snapshots
> change the proposed interface to set the SDE
> ci: use snapshot in reproducible builds test
>
> RECIPE-API-CHANGELOG.md | 12 ++
> doc/user_manual.md | 4 +
> kas/opt/Kconfig | 11 +
> kas/opt/mirror-snapshot.yaml | 9 +
> meta-isar/conf/distro/ubuntu-common.inc | 3 +
> meta/classes/base.bbclass | 17 +-
> meta/conf/bitbake.conf | 15 ++
> meta/conf/distro/debian-common.conf | 3 +
> meta/lib/oe/reproducible.py | 197
> ++++++++++++++++++
> .../isar-bootstrap/isar-bootstrap.inc | 16 ++
> testsuite/cibuilder.py | 8 +-
> testsuite/repro-build-test.py | 2 +-
> 12 files changed, 293 insertions(+), 4 deletions(-)
> create mode 100644 kas/opt/mirror-snapshot.yaml
> create mode 100644 meta/lib/oe/reproducible.py
>
> --
> 2.39.2
>
Aplied to next, thanks.
^ permalink raw reply [flat|nested] 10+ messages in thread