[PATCH v3 0/5] use debian snapshot mirror if SOURCE_DATE_EPOCH is set

[PATCH v3 0/5] use debian snapshot mirror if SOURCE_DATE_EPOCH is set

[Felix Moessbauer]

This patch series is a continuation of the idea to add OOTB support
for apt snapshot mirrors. In contrast to the first two versions, this
series approaches the topic at a broader scope.

Changes since v2:

- make the reproducible builds support more similar with OE
- support per-distro snapshots
- add support for snapshot builds in the kas menu
- add a test

Best regards,
Felix Moessbauer
Siemens AG

Felix Moessbauer (5):
  add reproducible builds infrastructure from oe
  use apt snapshot mirror if ISAR_USE_APT_SNAPSHOT is set
  add kas menu options to build against snapshots
  change the proposed interface to set the SDE
  ci: use snapshot in reproducible builds test

 RECIPE-API-CHANGELOG.md                       |  12 ++
 doc/user_manual.md                            |   4 +
 kas/opt/Kconfig                               |  11 +
 kas/opt/mirror-snapshot.yaml                  |   9 +
 meta-isar/conf/distro/ubuntu-common.inc       |   3 +
 meta/classes/base.bbclass                     |  17 +-
 meta/conf/bitbake.conf                        |  15 ++
 meta/conf/distro/debian-common.conf           |   3 +
 meta/lib/oe/reproducible.py                   | 197 ++++++++++++++++++
 .../isar-bootstrap/isar-bootstrap.inc         |  16 ++
 testsuite/cibuilder.py                        |   8 +-
 testsuite/repro-build-test.py                 |   2 +-
 12 files changed, 293 insertions(+), 4 deletions(-)
 create mode 100644 kas/opt/mirror-snapshot.yaml
 create mode 100644 meta/lib/oe/reproducible.py

-- 
2.39.2

[PATCH v3 1/5] add reproducible builds infrastructure from oe

[Felix Moessbauer]

This patch adds the reproducible builds infrastructure from OE. This
includes python helpers to determine the SOURCE_DATE_EPOCH per component
(if not set). Once determined, these values are written to the SDE_FILE
and picked up on the next build as a fallback.

Both this and the reproducible section of the bitbake.conf are taken 1:1
from OE 6548354 (corresponding commit to bitbake version currently used
by ISAR).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/base.bbclass   |  17 +++-
 meta/conf/bitbake.conf      |  12 +++
 meta/lib/oe/reproducible.py | 197 ++++++++++++++++++++++++++++++++++++
 3 files changed, 225 insertions(+), 1 deletion(-)
 create mode 100644 meta/lib/oe/reproducible.py

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 529811af..b8825bd3 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -21,7 +21,7 @@
 THISDIR = "${@os.path.dirname(d.getVar('FILE'))}"
 FILESPATH = "${@base_set_filespath(["${FILE_DIRNAME}/${PF}", "${FILE_DIRNAME}/${P}", "${FILE_DIRNAME}/${PN}", "${FILE_DIRNAME}/files", "${FILE_DIRNAME}"], d)}"
 
-OE_IMPORTS += "os sys time oe.path oe.patch oe.sstatesig oe.utils"
+OE_IMPORTS += "os sys time oe.path oe.patch oe.reproducible oe.sstatesig oe.utils"
 OE_IMPORTS[type] = "list"
 
 def oe_import(d):
@@ -318,3 +318,18 @@ def calculate_build_uuid(d):
 # Unique ID for this build, used to avoid name clashes on external mountpoints
 # When running parallel builds in different PID namespaces
 ISAR_BUILD_UUID = "${@ calculate_build_uuid(d)}"
+
+do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}"
+do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}"
+addtask do_deploy_source_date_epoch_setscene
+addtask do_deploy_source_date_epoch before do_configure after do_patch
+
+python create_source_date_epoch_stamp() {
+    # Version: 1
+    source_date_epoch = oe.reproducible.get_source_date_epoch(d, d.getVar('S'))
+    oe.reproducible.epochfile_write(source_date_epoch, d.getVar('SDE_FILE'), d)
+}
+do_unpack[postfuncs] += "create_source_date_epoch_stamp"
+
+def get_source_date_epoch_value(d):
+    return oe.reproducible.epochfile_read(d.getVar('SDE_FILE'), d)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 91c5c815..1da3ecac 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -134,6 +134,18 @@ BB_NUMBER_THREADS ?= "${@bb.utils.cpu_count()}"
 # Default to setting automatically based on cpu count
 PARALLEL_MAKE ?= "-j ${@bb.utils.cpu_count()}"
 
+# Reproducibility (taken 1:1 from oe)
+SDE_DIR = "${WORKDIR}/source-date-epoch"
+SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt"
+SDE_DEPLOYDIR = "${WORKDIR}/deploy-source-date-epoch"
+
+export PYTHONHASHSEED = "0"
+export PERL_HASH_SEED = "0"
+export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
+# A SOURCE_DATE_EPOCH of '0' might be misinterpreted as no SDE
+# ISAR: set value to date of latest release
+SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251"
+
 # Default parallelism and resource usage for xz
 XZ_MEMLIMIT ?= "50%"
 XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
diff --git a/meta/lib/oe/reproducible.py b/meta/lib/oe/reproducible.py
new file mode 100644
index 00000000..448befce
--- /dev/null
+++ b/meta/lib/oe/reproducible.py
@@ -0,0 +1,197 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+import os
+import subprocess
+import bb
+
+# For reproducible builds, this code sets the default SOURCE_DATE_EPOCH in each
+# component's build environment. The format is number of seconds since the
+# system epoch.
+#
+# Upstream components (generally) respect this environment variable,
+# using it in place of the "current" date and time.
+# See https://reproducible-builds.org/specs/source-date-epoch/
+#
+# The default value of SOURCE_DATE_EPOCH comes from the function
+# get_source_date_epoch_value which reads from the SDE_FILE, or if the file
+# is not available will use the fallback of SOURCE_DATE_EPOCH_FALLBACK.
+#
+# The SDE_FILE is normally constructed from the function
+# create_source_date_epoch_stamp which is typically added as a postfuncs to
+# the do_unpack task.  If a recipe does NOT have do_unpack, it should be added
+# to a task that runs after the source is available and before the
+# do_deploy_source_date_epoch task is executed.
+#
+# If a recipe wishes to override the default behavior it should set it's own
+# SOURCE_DATE_EPOCH or override the do_deploy_source_date_epoch_stamp task
+# with recipe-specific functionality to write the appropriate
+# SOURCE_DATE_EPOCH into the SDE_FILE.
+#
+# SOURCE_DATE_EPOCH is intended to be a reproducible value.  This value should
+# be reproducible for anyone who builds the same revision from the same
+# sources.
+#
+# There are 4 ways the create_source_date_epoch_stamp function determines what
+# becomes SOURCE_DATE_EPOCH:
+#
+# 1. Use the value from __source_date_epoch.txt file if this file exists.
+#    This file was most likely created in the previous build by one of the
+#    following methods 2,3,4.
+#    Alternatively, it can be provided by a recipe via SRC_URI.
+#
+# If the file does not exist:
+#
+# 2. If there is a git checkout, use the last git commit timestamp.
+#    Git does not preserve file timestamps on checkout.
+#
+# 3. Use the mtime of "known" files such as NEWS, CHANGLELOG, ...
+#    This works for well-kept repositories distributed via tarball.
+#
+# 4. Use the modification time of the youngest file in the source tree, if
+#    there is one.
+#    This will be the newest file from the distribution tarball, if any.
+#
+# 5. Fall back to a fixed timestamp (SOURCE_DATE_EPOCH_FALLBACK).
+#
+# Once the value is determined, it is stored in the recipe's SDE_FILE.
+
+def get_source_date_epoch_from_known_files(d, sourcedir):
+    source_date_epoch = None
+    newest_file = None
+    known_files = set(["NEWS", "ChangeLog", "Changelog", "CHANGES"])
+    for file in known_files:
+        filepath = os.path.join(sourcedir, file)
+        if os.path.isfile(filepath):
+            mtime = int(os.lstat(filepath).st_mtime)
+            # There may be more than one "known_file" present, if so, use the youngest one
+            if not source_date_epoch or mtime > source_date_epoch:
+                source_date_epoch = mtime
+                newest_file = filepath
+    if newest_file:
+        bb.debug(1, "SOURCE_DATE_EPOCH taken from: %s" % newest_file)
+    return source_date_epoch
+
+def find_git_folder(d, sourcedir):
+    # First guess: WORKDIR/git
+    # This is the default git fetcher unpack path
+    workdir = d.getVar('WORKDIR')
+    gitpath = os.path.join(workdir, "git/.git")
+    if os.path.isdir(gitpath):
+        return gitpath
+
+    # Second guess: ${S}
+    gitpath = os.path.join(sourcedir, ".git")
+    if os.path.isdir(gitpath):
+        return gitpath
+
+    # Perhaps there was a subpath or destsuffix specified.
+    # Go looking in the WORKDIR
+    exclude = set(["build", "image", "license-destdir", "patches", "pseudo",
+                   "recipe-sysroot", "recipe-sysroot-native", "sysroot-destdir", "temp"])
+    for root, dirs, files in os.walk(workdir, topdown=True):
+        dirs[:] = [d for d in dirs if d not in exclude]
+        if '.git' in dirs:
+            return os.path.join(root, ".git")
+
+    bb.warn("Failed to find a git repository in WORKDIR: %s" % workdir)
+    return None
+
+def get_source_date_epoch_from_git(d, sourcedir):
+    if not "git://" in d.getVar('SRC_URI') and not "gitsm://" in d.getVar('SRC_URI'):
+        return None
+
+    gitpath = find_git_folder(d, sourcedir)
+    if not gitpath:
+        return None
+
+    # Check that the repository has a valid HEAD; it may not if subdir is used
+    # in SRC_URI
+    p = subprocess.run(['git', '--git-dir', gitpath, 'rev-parse', 'HEAD'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+    if p.returncode != 0:
+        bb.debug(1, "%s does not have a valid HEAD: %s" % (gitpath, p.stdout.decode('utf-8')))
+        return None
+
+    bb.debug(1, "git repository: %s" % gitpath)
+    p = subprocess.run(['git', '-c', 'log.showSignature=false', '--git-dir', gitpath, 'log', '-1', '--pretty=%ct'],
+                       check=True, stdout=subprocess.PIPE)
+    return int(p.stdout.decode('utf-8'))
+
+def get_source_date_epoch_from_youngest_file(d, sourcedir):
+    if sourcedir == d.getVar('WORKDIR'):
+       # These sources are almost certainly not from a tarball
+       return None
+
+    # Do it the hard way: check all files and find the youngest one...
+    source_date_epoch = None
+    newest_file = None
+    for root, dirs, files in os.walk(sourcedir, topdown=True):
+        files = [f for f in files if not f[0] == '.']
+
+        for fname in files:
+            if fname == "singletask.lock":
+                 # Ignore externalsrc/devtool lockfile [YOCTO #14921]
+                 continue
+            filename = os.path.join(root, fname)
+            try:
+                mtime = int(os.lstat(filename).st_mtime)
+            except ValueError:
+                mtime = 0
+            if not source_date_epoch or mtime > source_date_epoch:
+                source_date_epoch = mtime
+                newest_file = filename
+
+    if newest_file:
+        bb.debug(1, "Newest file found: %s" % newest_file)
+    return source_date_epoch
+
+def fixed_source_date_epoch(d):
+    bb.debug(1, "No tarball or git repo found to determine SOURCE_DATE_EPOCH")
+    source_date_epoch = d.getVar('SOURCE_DATE_EPOCH_FALLBACK')
+    if source_date_epoch:
+        bb.debug(1, "Using SOURCE_DATE_EPOCH_FALLBACK")
+        return int(source_date_epoch)
+    return 0
+
+def get_source_date_epoch(d, sourcedir):
+    return (
+        get_source_date_epoch_from_git(d, sourcedir) or
+        get_source_date_epoch_from_youngest_file(d, sourcedir) or
+        fixed_source_date_epoch(d)       # Last resort
+    )
+
+def epochfile_read(epochfile, d):
+    cached, efile = d.getVar('__CACHED_SOURCE_DATE_EPOCH') or (None, None)
+    if cached and efile == epochfile:
+        return cached
+
+    if cached and epochfile != efile:
+        bb.debug(1, "Epoch file changed from %s to %s" % (efile, epochfile))
+
+    source_date_epoch = int(d.getVar('SOURCE_DATE_EPOCH_FALLBACK'))
+    try:
+        with open(epochfile, 'r') as f:
+            s = f.read()
+            try:
+                source_date_epoch = int(s)
+            except ValueError:
+                bb.warn("SOURCE_DATE_EPOCH value '%s' is invalid. Reverting to SOURCE_DATE_EPOCH_FALLBACK" % s)
+                source_date_epoch = int(d.getVar('SOURCE_DATE_EPOCH_FALLBACK'))
+        bb.debug(1, "SOURCE_DATE_EPOCH: %d" % source_date_epoch)
+    except FileNotFoundError:
+        bb.debug(1, "Cannot find %s. SOURCE_DATE_EPOCH will default to %d" % (epochfile, source_date_epoch))
+
+    d.setVar('__CACHED_SOURCE_DATE_EPOCH', (str(source_date_epoch), epochfile))
+    return str(source_date_epoch)
+
+def epochfile_write(source_date_epoch, epochfile, d):
+
+    bb.debug(1, "SOURCE_DATE_EPOCH: %d" % source_date_epoch)
+    bb.utils.mkdirhier(os.path.dirname(epochfile))
+
+    tmp_file = "%s.new" % epochfile
+    with open(tmp_file, 'w') as f:
+        f.write(str(source_date_epoch))
+    os.rename(tmp_file, epochfile)
-- 
2.39.2

[PATCH v3 2/5] use apt snapshot mirror if ISAR_USE_APT_SNAPSHOT is set

[Felix Moessbauer]

This patch adds infrastructure to switch the apt sources to a frozen
snapshot mirror. To build against a mirror, set ISAR_USE_APT_SNAPSHOT=1.
As the mirror is distro specific, it is configured in the distro config
for all supported distros (currently only debian and ubuntu). For
unsupported distros, a meaningful error message is emitted on enabling
the snapshot build. Similar to the DISTRO_APT_PREMIRRORS, this mirror is
only injected temporarily and does not end up in the final apt sources
list.

To further control the behavior, we introduce the following variables:

- DISTRO_APT_SNAPSHOT_PREMIRROR: The snapshot mirror to use. Syntax
  identical to DISTRO_APT_PREMIRRORS.
- ISAR_APT_SNAPSHOT_TIMESTAMP: Unix timestamp of the snapshot. This is
  automatically derived from the SOURCE_DATE_EPOCH if not set.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 RECIPE-API-CHANGELOG.md                          |  6 ++++++
 doc/user_manual.md                               |  3 +++
 meta-isar/conf/distro/ubuntu-common.inc          |  3 +++
 meta/conf/bitbake.conf                           |  3 +++
 meta/conf/distro/debian-common.conf              |  3 +++
 .../isar-bootstrap/isar-bootstrap.inc            | 16 ++++++++++++++++
 6 files changed, 34 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 6653ab43..e6861523 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -583,3 +583,9 @@ Cross compiling kernel modules for distro kernels is not supported in debian.
 To simplify downstream kernel module builds, we automatically turn of cross
 compilation for a user-provided module when building it for a distro kernel.
 
+
+### Build against debian snapshot mirror
+
+To build against a distributions snapshot mirror, set `ISAR_USE_APT_SNAPSHOT="1"`.
+The mirror to use is specified in `DISTRO_APT_SNAPSHOT_PREMIRROR` and usually
+pre-defined in the distro config.
diff --git a/doc/user_manual.md b/doc/user_manual.md
index 419d5339..70741968 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -431,6 +431,9 @@ Some other variables include:
  - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for SDK root filesystem. This variable is optional.
  - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host.
  - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES.
+ - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of the live mirror. Optional.
+ - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported distros.
+ - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Timestamp of the apt snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not overwritten.
  - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping.
  - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS.
  - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable.
diff --git a/meta-isar/conf/distro/ubuntu-common.inc b/meta-isar/conf/distro/ubuntu-common.inc
index 9d8a843b..54bb747a 100644
--- a/meta-isar/conf/distro/ubuntu-common.inc
+++ b/meta-isar/conf/distro/ubuntu-common.inc
@@ -32,3 +32,6 @@ IMAGE_PREINSTALL += "init"
 IMAGE_PREINSTALL += "initramfs-tools"
 
 IMAGER_INSTALL:wic += "python3-distutils"
+
+# snapshot mirror for reproducible builds
+DISTRO_APT_SNAPSHOT_PREMIRROR ??= "(http|https)://archive.ubuntu.com/(.*) https://snapshot.ubuntu.com/\2/${APT_SNAPSHOT_DATE}/\n"
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 1da3ecac..4cfa8b10 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -145,6 +145,9 @@ export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
 # A SOURCE_DATE_EPOCH of '0' might be misinterpreted as no SDE
 # ISAR: set value to date of latest release
 SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251"
+# Debian snapshots
+ISAR_USE_APT_SNAPSHOT ??= "0"
+ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}"
 
 # Default parallelism and resource usage for xz
 XZ_MEMLIMIT ?= "50%"
diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf
index 1e1dfc83..db538510 100644
--- a/meta/conf/distro/debian-common.conf
+++ b/meta/conf/distro/debian-common.conf
@@ -39,3 +39,6 @@ SYSTEMD_BOOTLOADER_INSTALL:sid = "systemd-boot-efi:${DISTRO_ARCH}"
 
 COMPAT_DISTRO_ARCH:amd64 = "i386"
 COMPAT_DISTRO_ARCH:arm64 = "armhf"
+
+# snapshot mirror for reproducible builds
+DISTRO_APT_SNAPSHOT_PREMIRROR ??= "deb.debian.org/(.*) snapshot-cloudflare.debian.org/archive/\1/${APT_SNAPSHOT_DATE}/\n"
\ No newline at end of file
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 17f19fd8..733a23df 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -30,6 +30,9 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR
 BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}"
 BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}"
 FILESEXTRAPATHS:append = ":${BBPATH}"
+# reproducible builds, only enabled if ISAR_USE_APT_SNAPSHOT
+ISAR_APT_SNAPSHOT_MIRROR ??= ""
+APT_SNAPSHOT_DATE = "${@ get_apt_snapshot_date(d)}"
 
 inherit deb-dl-dir
 
@@ -107,11 +110,21 @@ def parse_aptsources_list_line(source_list_line):
 
     return [type, options, source, suite, components]
 
+def get_apt_snapshot_date(d):
+    import time
+    source_date_epoch = d.getVar('ISAR_APT_SNAPSHOT_TIMESTAMP')
+    return time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch)))
+
 def get_apt_source_mirror(d, aptsources_entry_list):
     import re
 
+    # this is executed during parsing. No error checking possible
+    use_snapshot = bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT'))
+    snapshot_mirror = d.getVar('DISTRO_APT_SNAPSHOT_PREMIRROR')
     if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
         premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"
+    elif use_snapshot and snapshot_mirror:
+        premirrors = snapshot_mirror
     else:
         premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or ""
     mirror_list = [entry.split()
@@ -126,6 +139,8 @@ def get_apt_source_mirror(d, aptsources_entry_list):
             new_aptsources_entry_list[2] = re.sub(regex, replace,
                                                   aptsources_entry_list[2],
                                                   count = 1)
+            if use_snapshot:
+                new_aptsources_entry_list[1] = "[check-valid-until=no]"
             return new_aptsources_entry_list
 
     return aptsources_entry_list
@@ -240,6 +255,7 @@ do_apt_config_prepare[vardeps] += " \
     APTSRCS \
     ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
     DEPLOY_ISAR_BOOTSTRAP \
+    ${@'DISTRO_APT_SNAPSHOT_PREMIRROR' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''} \
     "
 python do_apt_config_prepare() {
     apt_preferences_out = d.getVar("APTPREFS")
-- 
2.39.2

[PATCH v3 3/5] add kas menu options to build against snapshots

[Felix Moessbauer]

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 kas/opt/Kconfig              | 11 +++++++++++
 kas/opt/mirror-snapshot.yaml |  9 +++++++++
 2 files changed, 20 insertions(+)
 create mode 100644 kas/opt/mirror-snapshot.yaml

diff --git a/kas/opt/Kconfig b/kas/opt/Kconfig
index 1619a4cf..701bd263 100644
--- a/kas/opt/Kconfig
+++ b/kas/opt/Kconfig
@@ -40,6 +40,17 @@ config KAS_INCLUDE_MIRROR_UBUNTU
 	default "kas/opt/mirror-ubuntu.yaml"
 	depends on MIRROR_UBUNTU
 
+config USE_APT_SNAPSHOT
+	bool "Use the distros snapshot mirror"
+	depends on !MIRROR_DEBIAN && (DEBIAN_BUSTER || DEBIAN_BULLSEYE || DEBIAN_BOOKWORM || DEBIAN_TRIXIE || UBUNTU_FOCAL || UBUNTU_JAMMY)
+	help
+		Use a snapshot mirror for the selected distribution. The date is defined by ISAR_APT_SNAPSHOT_TIMESTAMP.
+
+config KAS_INCLUDE_APT_SNAPSHOT
+	string
+	default "kas/opt/mirror-snapshot.yaml"
+	depends on USE_APT_SNAPSHOT
+
 endmenu
 
 
diff --git a/kas/opt/mirror-snapshot.yaml b/kas/opt/mirror-snapshot.yaml
new file mode 100644
index 00000000..59fae192
--- /dev/null
+++ b/kas/opt/mirror-snapshot.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (C) 2024 Siemens AG
+
+header:
+  version: 14
+
+local_conf_header:
+  mirror-debian: |
+    ISAR_USE_APT_SNAPSHOT = "1"
-- 
2.39.2

[PATCH v3 4/5] change the proposed interface to set the SDE

[Felix Moessbauer]

To align the handling of source timestamps with OE, we propose to not
set the SOURCE_DATE_EPOCH variable directly, but via
`SOURCE_DATE_EPOCH_FALLBACK`. By that, we internally can use heuristics
to better estimate the value of the SDE, while still having a global
fallback if it cannot be estimated otherwise.

Please note, that change is backwards compatible. You can still set the
SOURCE_DATE_EPOCH variable directly and get the same behavior as before.

This patch also updates the reproducibility test according to the new
interface.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 RECIPE-API-CHANGELOG.md | 6 ++++++
 doc/user_manual.md      | 1 +
 testsuite/cibuilder.py  | 2 +-
 3 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index e6861523..4afe8b16 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -589,3 +589,9 @@ compilation for a user-provided module when building it for a distro kernel.
 To build against a distributions snapshot mirror, set `ISAR_USE_APT_SNAPSHOT="1"`.
 The mirror to use is specified in `DISTRO_APT_SNAPSHOT_PREMIRROR` and usually
 pre-defined in the distro config.
+
+### Use OE interface to set timestamp for reproducible builds
+
+The `SOURCE_DATE_EPOCH` (SDE) should not be set globally, but on a per-recipe basis
+and to meaningful values. As a global fallback, set the `SOURCE_DATE_EPOCH_FALLBACK`
+bitbake variable to the desired unix timestamp.
diff --git a/doc/user_manual.md b/doc/user_manual.md
index 70741968..b12c7692 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -425,6 +425,7 @@ Some other variables include:
 
  - `IMAGE_INSTALL` - The list of custom packages to build and install to target image, please refer to relative chapter for more information.
  - `BB_NUMBER_THREADS` - The number of `bitbake` jobs that can be run in parallel. Please set this option according to your host CPU cores number.
+ - `SOURCE_DATE_EPOCH_FALLBACK` - The unix timestamp passed to all tooling to make the results reproducible. This variable is optional.
  - `HOST_DISTRO` - The distro to use for SDK root filesystem. This variable is optional.
  - `HOST_ARCH` - The Debian architecture of SDK root filesystem (e.g., `amd64`). By default set to current Debian host architecture. This variable is optional.
  - `HOST_DISTRO_APT_SOURCES` - List of apt source files for SDK root filesystem. This variable is optional.
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index fa30c2f5..7e69dafe 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -159,7 +159,7 @@ class CIBuilder(Test):
                 f.write('USE_CCACHE = "1"\n')
                 f.write('CCACHE_TOP_DIR = "%s"\n' % ccache_dir)
             if source_date_epoch:
-                f.write('SOURCE_DATE_EPOCH = "%s"\n' % source_date_epoch)
+                f.write('SOURCE_DATE_EPOCH_FALLBACK = "%s"\n' % source_date_epoch)
             if dl_dir:
                 f.write('DL_DIR = "%s"\n' % dl_dir)
             if sstate_dir:
-- 
2.39.2

[PATCH v3 5/5] ci: use snapshot in reproducible builds test

[Felix Moessbauer]

As we now have easy to use snapshot support, we also switch over the
reproducible builds test to use a snapshot. By that, we both test the
snapshot support, as well as make sure to always build against a common
set of packages.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 testsuite/cibuilder.py        | 6 +++++-
 testsuite/repro-build-test.py | 2 +-
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 7e69dafe..039355e5 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -78,7 +78,8 @@ class CIBuilder(Test):
                   container=False, ccache=False, sstate=False, offline=False,
                   gpg_pub_key=None, wic_deploy_parts=False, dl_dir=None,
                   sstate_dir=None, ccache_dir=None,
-                  source_date_epoch=None, image_install=None, **kwargs):
+                  source_date_epoch=None, use_apt_snapshot=False,
+                  image_install=None, **kwargs):
         # write configuration file and set bitbake_args
         # can run multiple times per test case
         self.check_init()
@@ -116,6 +117,7 @@ class CIBuilder(Test):
                       f'  gpg_pub_key = {gpg_pub_key}\n'
                       f'  wic_deploy_parts = {wic_deploy_parts}\n'
                       f'  source_date_epoch = {source_date_epoch} \n'
+                      f'  use_apt_snapshot = {use_apt_snapshot} \n'
                       f'  dl_dir = {dl_dir}\n'
                       f'  sstate_dir = {sstate_dir}\n'
                       f'  ccache_dir = {ccache_dir}\n'
@@ -160,6 +162,8 @@ class CIBuilder(Test):
                 f.write('CCACHE_TOP_DIR = "%s"\n' % ccache_dir)
             if source_date_epoch:
                 f.write('SOURCE_DATE_EPOCH_FALLBACK = "%s"\n' % source_date_epoch)
+            if use_apt_snapshot:
+                f.write('ISAR_USE_APT_SNAPSHOT = "1"\n')
             if dl_dir:
                 f.write('DL_DIR = "%s"\n' % dl_dir)
             if sstate_dir:
diff --git a/testsuite/repro-build-test.py b/testsuite/repro-build-test.py
index 040a844e..843f3bc6 100755
--- a/testsuite/repro-build-test.py
+++ b/testsuite/repro-build-test.py
@@ -46,7 +46,7 @@ class ReproBuild(CIBuilder):
 
         # Build
         self.log.info("Started Build " + image_name)
-        self.configure(source_date_epoch=source_date_epoch)
+        self.configure(source_date_epoch=source_date_epoch, use_apt_snapshot=True)
         self.bitbake(target)
 
         # copy the artifacts image name with given name
-- 
2.39.2

Re: [PATCH v3 0/5] use debian snapshot mirror if SOURCE_DATE_EPOCH is set

[Uladzimir Bely]

On Tue, 2024-04-09 at 17:55 +0200, 'Felix Moessbauer' via isar-users
wrote:
> This patch series is a continuation of the idea to add OOTB support
> for apt snapshot mirrors. In contrast to the first two versions, this
> series approaches the topic at a broader scope.
> 
> Changes since v2:
> 
> - make the reproducible builds support more similar with OE
> - support per-distro snapshots
> - add support for snapshot builds in the kas menu
> - add a test
> 
> Best regards,
> Felix Moessbauer
> Siemens AG
> 
> Felix Moessbauer (5):
>   add reproducible builds infrastructure from oe
>   use apt snapshot mirror if ISAR_USE_APT_SNAPSHOT is set
>   add kas menu options to build against snapshots
>   change the proposed interface to set the SDE
>   ci: use snapshot in reproducible builds test
> 
>  RECIPE-API-CHANGELOG.md                       |  12 ++
>  doc/user_manual.md                            |   4 +
>  kas/opt/Kconfig                               |  11 +
>  kas/opt/mirror-snapshot.yaml                  |   9 +
>  meta-isar/conf/distro/ubuntu-common.inc       |   3 +
>  meta/classes/base.bbclass                     |  17 +-
>  meta/conf/bitbake.conf                        |  15 ++
>  meta/conf/distro/debian-common.conf           |   3 +
>  meta/lib/oe/reproducible.py                   | 197
> ++++++++++++++++++
>  .../isar-bootstrap/isar-bootstrap.inc         |  16 ++
>  testsuite/cibuilder.py                        |   8 +-
>  testsuite/repro-build-test.py                 |   2 +-
>  12 files changed, 293 insertions(+), 4 deletions(-)
>  create mode 100644 kas/opt/mirror-snapshot.yaml
>  create mode 100644 meta/lib/oe/reproducible.py
> 
> -- 
> 2.39.2
> 

The patchset passes CI. If there are no objections/additions, we are
ready to merge it.

---
Regards,
Uladzimir Bely.

Re: [PATCH v3 0/5] use debian snapshot mirror if SOURCE_DATE_EPOCH is set

[Uladzimir Bely]

On Tue, 2024-04-09 at 17:55 +0200, 'Felix Moessbauer' via isar-users
wrote:
> This patch series is a continuation of the idea to add OOTB support
> for apt snapshot mirrors. In contrast to the first two versions, this
> series approaches the topic at a broader scope.
> 
> Changes since v2:
> 
> - make the reproducible builds support more similar with OE
> - support per-distro snapshots
> - add support for snapshot builds in the kas menu
> - add a test
> 
> Best regards,
> Felix Moessbauer
> Siemens AG
> 
> Felix Moessbauer (5):
>   add reproducible builds infrastructure from oe
>   use apt snapshot mirror if ISAR_USE_APT_SNAPSHOT is set
>   add kas menu options to build against snapshots
>   change the proposed interface to set the SDE
>   ci: use snapshot in reproducible builds test
> 
>  RECIPE-API-CHANGELOG.md                       |  12 ++
>  doc/user_manual.md                            |   4 +
>  kas/opt/Kconfig                               |  11 +
>  kas/opt/mirror-snapshot.yaml                  |   9 +
>  meta-isar/conf/distro/ubuntu-common.inc       |   3 +
>  meta/classes/base.bbclass                     |  17 +-
>  meta/conf/bitbake.conf                        |  15 ++
>  meta/conf/distro/debian-common.conf           |   3 +
>  meta/lib/oe/reproducible.py                   | 197
> ++++++++++++++++++
>  .../isar-bootstrap/isar-bootstrap.inc         |  16 ++
>  testsuite/cibuilder.py                        |   8 +-
>  testsuite/repro-build-test.py                 |   2 +-
>  12 files changed, 293 insertions(+), 4 deletions(-)
>  create mode 100644 kas/opt/mirror-snapshot.yaml
>  create mode 100644 meta/lib/oe/reproducible.py
> 
> -- 
> 2.39.2
> 

Aplied to next, thanks.

Re: [PATCH v3 2/5] use apt snapshot mirror if ISAR_USE_APT_SNAPSHOT is set

[Anton Mikanovich]

09/04/2024 18:55, 'Felix Moessbauer' via isar-users wrote:
> This patch adds infrastructure to switch the apt sources to a frozen
> snapshot mirror. To build against a mirror, set ISAR_USE_APT_SNAPSHOT=1.
> As the mirror is distro specific, it is configured in the distro config
> for all supported distros (currently only debian and ubuntu). For
> unsupported distros, a meaningful error message is emitted on enabling
> the snapshot build. Similar to the DISTRO_APT_PREMIRRORS, this mirror is
> only injected temporarily and does not end up in the final apt sources
> list.
>
> To further control the behavior, we introduce the following variables:
>
> - DISTRO_APT_SNAPSHOT_PREMIRROR: The snapshot mirror to use. Syntax
>    identical to DISTRO_APT_PREMIRRORS.
> - ISAR_APT_SNAPSHOT_TIMESTAMP: Unix timestamp of the snapshot. This is
>    automatically derived from the SOURCE_DATE_EPOCH if not set.
>
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>   RECIPE-API-CHANGELOG.md                          |  6 ++++++
>   doc/user_manual.md                               |  3 +++
>   meta-isar/conf/distro/ubuntu-common.inc          |  3 +++
>   meta/conf/bitbake.conf                           |  3 +++
>   meta/conf/distro/debian-common.conf              |  3 +++
>   .../isar-bootstrap/isar-bootstrap.inc            | 16 ++++++++++++++++
>   6 files changed, 34 insertions(+)
>
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index 6653ab43..e6861523 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -583,3 +583,9 @@ Cross compiling kernel modules for distro kernels is not supported in debian.
>   To simplify downstream kernel module builds, we automatically turn of cross
>   compilation for a user-provided module when building it for a distro kernel.
>   
> +
> +### Build against debian snapshot mirror
> +
> +To build against a distributions snapshot mirror, set `ISAR_USE_APT_SNAPSHOT="1"`.
> +The mirror to use is specified in `DISTRO_APT_SNAPSHOT_PREMIRROR` and usually
> +pre-defined in the distro config.
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 419d5339..70741968 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -431,6 +431,9 @@ Some other variables include:
>    - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for SDK root filesystem. This variable is optional.
>    - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host.
>    - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES.
> + - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of the live mirror. Optional.
> + - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported distros.
> + - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Timestamp of the apt snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not overwritten.
>    - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping.
>    - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS.
>    - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable.
> diff --git a/meta-isar/conf/distro/ubuntu-common.inc b/meta-isar/conf/distro/ubuntu-common.inc
> index 9d8a843b..54bb747a 100644
> --- a/meta-isar/conf/distro/ubuntu-common.inc
> +++ b/meta-isar/conf/distro/ubuntu-common.inc
> @@ -32,3 +32,6 @@ IMAGE_PREINSTALL += "init"
>   IMAGE_PREINSTALL += "initramfs-tools"
>   
>   IMAGER_INSTALL:wic += "python3-distutils"
> +
> +# snapshot mirror for reproducible builds
> +DISTRO_APT_SNAPSHOT_PREMIRROR ??= "(http|https)://archive.ubuntu.com/(.*) https://snapshot.ubuntu.com/\2/${APT_SNAPSHOT_DATE}/\n"
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 1da3ecac..4cfa8b10 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -145,6 +145,9 @@ export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
>   # A SOURCE_DATE_EPOCH of '0' might be misinterpreted as no SDE
>   # ISAR: set value to date of latest release
>   SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251"
> +# Debian snapshots
> +ISAR_USE_APT_SNAPSHOT ??= "0"
> +ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}"
>   
>   # Default parallelism and resource usage for xz
>   XZ_MEMLIMIT ?= "50%"
> diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf
> index 1e1dfc83..db538510 100644
> --- a/meta/conf/distro/debian-common.conf
> +++ b/meta/conf/distro/debian-common.conf
> @@ -39,3 +39,6 @@ SYSTEMD_BOOTLOADER_INSTALL:sid = "systemd-boot-efi:${DISTRO_ARCH}"
>   
>   COMPAT_DISTRO_ARCH:amd64 = "i386"
>   COMPAT_DISTRO_ARCH:arm64 = "armhf"
> +
> +# snapshot mirror for reproducible builds
> +DISTRO_APT_SNAPSHOT_PREMIRROR ??= "deb.debian.org/(.*) snapshot-cloudflare.debian.org/archive/\1/${APT_SNAPSHOT_DATE}/\n"
> \ No newline at end of file
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index 17f19fd8..733a23df 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -30,6 +30,9 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR
>   BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}"
>   BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}"
>   FILESEXTRAPATHS:append = ":${BBPATH}"
> +# reproducible builds, only enabled if ISAR_USE_APT_SNAPSHOT
> +ISAR_APT_SNAPSHOT_MIRROR ??= ""
> +APT_SNAPSHOT_DATE = "${@ get_apt_snapshot_date(d)}"
>   
>   inherit deb-dl-dir
>   
> @@ -107,11 +110,21 @@ def parse_aptsources_list_line(source_list_line):
>   
>       return [type, options, source, suite, components]
>   
> +def get_apt_snapshot_date(d):
> +    import time
> +    source_date_epoch = d.getVar('ISAR_APT_SNAPSHOT_TIMESTAMP')
> +    return time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch)))
> +
>   def get_apt_source_mirror(d, aptsources_entry_list):
>       import re
>   
> +    # this is executed during parsing. No error checking possible
> +    use_snapshot = bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT'))
> +    snapshot_mirror = d.getVar('DISTRO_APT_SNAPSHOT_PREMIRROR')
>       if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>           premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"
> +    elif use_snapshot and snapshot_mirror:
> +        premirrors = snapshot_mirror
>       else:
>           premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or ""
>       mirror_list = [entry.split()
> @@ -126,6 +139,8 @@ def get_apt_source_mirror(d, aptsources_entry_list):
>               new_aptsources_entry_list[2] = re.sub(regex, replace,
>                                                     aptsources_entry_list[2],
>                                                     count = 1)
> +            if use_snapshot:
> +                new_aptsources_entry_list[1] = "[check-valid-until=no]"
>               return new_aptsources_entry_list
>   
>       return aptsources_entry_list
> @@ -240,6 +255,7 @@ do_apt_config_prepare[vardeps] += " \
>       APTSRCS \
>       ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
>       DEPLOY_ISAR_BOOTSTRAP \
> +    ${@'DISTRO_APT_SNAPSHOT_PREMIRROR' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''} \
>       "
>   python do_apt_config_prepare() {
>       apt_preferences_out = d.getVar("APTPREFS")

Hello Felix,

How ISAR_APT_SNAPSHOT_MIRROR supposed to be used?

Re: [PATCH v3 2/5] use apt snapshot mirror if ISAR_USE_APT_SNAPSHOT is set

[MOESSBAUER, Felix]

On Thu, 2024-06-06 at 16:15 +0300, Anton Mikanovich wrote:
> 09/04/2024 18:55, 'Felix Moessbauer' via isar-users wrote:
> > This patch adds infrastructure to switch the apt sources to a
> > frozen
> > snapshot mirror. To build against a mirror, set
> > ISAR_USE_APT_SNAPSHOT=1.
> > As the mirror is distro specific, it is configured in the distro
> > config
> > for all supported distros (currently only debian and ubuntu). For
> > unsupported distros, a meaningful error message is emitted on
> > enabling
> > the snapshot build. Similar to the DISTRO_APT_PREMIRRORS, this
> > mirror is
> > only injected temporarily and does not end up in the final apt
> > sources
> > list.
> > 
> > To further control the behavior, we introduce the following
> > variables:
> > 
> > - DISTRO_APT_SNAPSHOT_PREMIRROR: The snapshot mirror to use. Syntax
> >    identical to DISTRO_APT_PREMIRRORS.
> > - ISAR_APT_SNAPSHOT_TIMESTAMP: Unix timestamp of the snapshot. This
> > is
> >    automatically derived from the SOURCE_DATE_EPOCH if not set.
> > 
> > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> > ---
> 
> Hello Felix,
> 
> How ISAR_APT_SNAPSHOT_MIRROR supposed to be used?

Hi,

it looks like this variable is a leftover and now is called
DISTRO_APT_SNAPSHOT_PREMIRROR. The variable ISAR_APT_SNAPSHOT_MIRROR is
nowhere used and can be removed.

I'll prepare a patch.

Best regards,
Felix


-- 
Siemens AG, Technology
Linux Expert Center