From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7355896709558304768 X-Received: by 2002:a05:6870:5a84:b0:22e:7390:da7 with SMTP id dt4-20020a0568705a8400b0022e73900da7mr9380970oab.21.1712678165042; Tue, 09 Apr 2024 08:56:05 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6870:1648:b0:22f:1b54:b7ca with SMTP id c8-20020a056870164800b0022f1b54b7cals3048181oae.0.-pod-prod-07-us; Tue, 09 Apr 2024 08:56:03 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH2L4fJYCS/+f6e9mGYUW1iLvB9pTm7EU5mY8+g4D8aScZsFmHLmu1cHBmJlL7ALES6ujqw X-Received: by 2002:aca:2313:0:b0:3c5:e553:475a with SMTP id e19-20020aca2313000000b003c5e553475amr9001982oie.45.1712678163551; Tue, 09 Apr 2024 08:56:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1712678163; cv=none; d=google.com; s=arc-20160816; b=zBT3ChUirhmxXwAkViAAaLSLAsU2OYPqM/HMVPodwAx4y4W1FtuekmlwapsCRhckXH WBD6EMgFbGTEVLAwpDpnCrkaRzy115X78Ma6f1d+ErR5eELyyL+2g5Ia2H4w0BOxO/Bg PyUbaQJwDjWwCwy8dfD1RuPYtGGcHnLtqjJNGAPuzQfc3RTTIUjSFVZDnsL05pFn6Uc/ IHejJ4XpvbZKHaVlve+j7wIiuPrQrCGfe5so/wPealHINRR/NUPte9Et8EXqjGDcXnWg pEy173/RXMN7dmdWG0GCoQCJcwddrdHOiwS8oXoQZxw5M/8ybjTEIYXW/HvH/Z1LY+fJ xhiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=x74ASns5mdLMMVRYE5JA6iV/n2XbnOrouqPaD7Qxc1g=; fh=WkhL8kaJc+l2wQon1t06Ej3uvBGj9sVhNcE8PaS/XbI=; b=ZJI2xSOYLupthXJc+rhcTTYXYQbawSROBeYcuDlkBMHhnxiT0IhA9diwjX9HqftoGB jCIdCsGa2xGz26SzxTJD7pf1YE3hyxkk0CCBuC3oAS7BpUIP5fPeCmHRMNzfmznxvuv6 00Oe08IjN2cgtXtY26p+bTSFb4tuypDIzc9wTdfUpad65UzcKCJPwWkoRUygJl8LRDz2 IEo9kBmUIiMEia9MCNFSdjAlBCxV057qeqAc3NUfjBIdwvWQPnPgGfnEvT8fXwaaZEkZ nVMlk4p+gY3EMysL9pMlALd9+1130CTR2igAZo4ojF9bBwInAbs+4+OKbVN9c5eYFr8L BbZw==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=o5bvaMdV; spf=pass (google.com: domain of fm-1321639-20240409155601d80ed3c77cbb2655c6-mjy35y@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-1321639-20240409155601d80ed3c77cbb2655c6-mJy35y@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Return-Path: Received: from mta-65-227.siemens.flowmailer.net (mta-65-227.siemens.flowmailer.net. [185.136.65.227]) by gmr-mx.google.com with ESMTPS id bf8-20020a056808190800b003c5f1bfdb6esi385091oib.0.2024.04.09.08.56.02 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2024 08:56:03 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1321639-20240409155601d80ed3c77cbb2655c6-mjy35y@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) client-ip=185.136.65.227; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=o5bvaMdV; spf=pass (google.com: domain of fm-1321639-20240409155601d80ed3c77cbb2655c6-mjy35y@rts-flowmailer.siemens.com designates 185.136.65.227 as permitted sender) smtp.mailfrom=fm-1321639-20240409155601d80ed3c77cbb2655c6-mJy35y@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: by mta-65-227.siemens.flowmailer.net with ESMTPSA id 20240409155601d80ed3c77cbb2655c6 for ; Tue, 09 Apr 2024 17:56:01 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=felix.moessbauer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=x74ASns5mdLMMVRYE5JA6iV/n2XbnOrouqPaD7Qxc1g=; b=o5bvaMdV1f6AD15l+NMzr+AWwSpaAph0D1WlK7TdA30W634C7DHesFfaIst92f2+hnOvLS ZeYMLfLE8aK6nieAnynkt6PUlkYokLIy3xsFfcukEKzMAX4dX2ECeM0kls9uB5kyTg08J72V LtpL3kdN+6l00PpZsE9DPAGj/VsKg=; From: Felix Moessbauer To: isar-users@googlegroups.com Cc: jan.kiszka@siemens.com, quirin.gylstorff@siemens.com, Felix Moessbauer Subject: [PATCH v3 1/5] add reproducible builds infrastructure from oe Date: Tue, 9 Apr 2024 17:55:45 +0200 Message-Id: <20240409155549.826454-2-felix.moessbauer@siemens.com> In-Reply-To: <20240409155549.826454-1-felix.moessbauer@siemens.com> References: <20240409155549.826454-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-TUID: 2QzcTt/Qh26R This patch adds the reproducible builds infrastructure from OE. This includes python helpers to determine the SOURCE_DATE_EPOCH per component (if not set). Once determined, these values are written to the SDE_FILE and picked up on the next build as a fallback. Both this and the reproducible section of the bitbake.conf are taken 1:1 from OE 6548354 (corresponding commit to bitbake version currently used by ISAR). Signed-off-by: Felix Moessbauer --- meta/classes/base.bbclass | 17 +++- meta/conf/bitbake.conf | 12 +++ meta/lib/oe/reproducible.py | 197 ++++++++++++++++++++++++++++++++++++ 3 files changed, 225 insertions(+), 1 deletion(-) create mode 100644 meta/lib/oe/reproducible.py diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass index 529811af..b8825bd3 100644 --- a/meta/classes/base.bbclass +++ b/meta/classes/base.bbclass @@ -21,7 +21,7 @@ THISDIR = "${@os.path.dirname(d.getVar('FILE'))}" FILESPATH = "${@base_set_filespath(["${FILE_DIRNAME}/${PF}", "${FILE_DIRNAME}/${P}", "${FILE_DIRNAME}/${PN}", "${FILE_DIRNAME}/files", "${FILE_DIRNAME}"], d)}" -OE_IMPORTS += "os sys time oe.path oe.patch oe.sstatesig oe.utils" +OE_IMPORTS += "os sys time oe.path oe.patch oe.reproducible oe.sstatesig oe.utils" OE_IMPORTS[type] = "list" def oe_import(d): @@ -318,3 +318,18 @@ def calculate_build_uuid(d): # Unique ID for this build, used to avoid name clashes on external mountpoints # When running parallel builds in different PID namespaces ISAR_BUILD_UUID = "${@ calculate_build_uuid(d)}" + +do_deploy_source_date_epoch[dirs] = "${SDE_DEPLOYDIR}" +do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DEPLOYDIR}" +addtask do_deploy_source_date_epoch_setscene +addtask do_deploy_source_date_epoch before do_configure after do_patch + +python create_source_date_epoch_stamp() { + # Version: 1 + source_date_epoch = oe.reproducible.get_source_date_epoch(d, d.getVar('S')) + oe.reproducible.epochfile_write(source_date_epoch, d.getVar('SDE_FILE'), d) +} +do_unpack[postfuncs] += "create_source_date_epoch_stamp" + +def get_source_date_epoch_value(d): + return oe.reproducible.epochfile_read(d.getVar('SDE_FILE'), d) diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 91c5c815..1da3ecac 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -134,6 +134,18 @@ BB_NUMBER_THREADS ?= "${@bb.utils.cpu_count()}" # Default to setting automatically based on cpu count PARALLEL_MAKE ?= "-j ${@bb.utils.cpu_count()}" +# Reproducibility (taken 1:1 from oe) +SDE_DIR = "${WORKDIR}/source-date-epoch" +SDE_FILE = "${SDE_DIR}/__source_date_epoch.txt" +SDE_DEPLOYDIR = "${WORKDIR}/deploy-source-date-epoch" + +export PYTHONHASHSEED = "0" +export PERL_HASH_SEED = "0" +export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}" +# A SOURCE_DATE_EPOCH of '0' might be misinterpreted as no SDE +# ISAR: set value to date of latest release +SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251" + # Default parallelism and resource usage for xz XZ_MEMLIMIT ?= "50%" XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" diff --git a/meta/lib/oe/reproducible.py b/meta/lib/oe/reproducible.py new file mode 100644 index 00000000..448befce --- /dev/null +++ b/meta/lib/oe/reproducible.py @@ -0,0 +1,197 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: GPL-2.0-only +# +import os +import subprocess +import bb + +# For reproducible builds, this code sets the default SOURCE_DATE_EPOCH in each +# component's build environment. The format is number of seconds since the +# system epoch. +# +# Upstream components (generally) respect this environment variable, +# using it in place of the "current" date and time. +# See https://reproducible-builds.org/specs/source-date-epoch/ +# +# The default value of SOURCE_DATE_EPOCH comes from the function +# get_source_date_epoch_value which reads from the SDE_FILE, or if the file +# is not available will use the fallback of SOURCE_DATE_EPOCH_FALLBACK. +# +# The SDE_FILE is normally constructed from the function +# create_source_date_epoch_stamp which is typically added as a postfuncs to +# the do_unpack task. If a recipe does NOT have do_unpack, it should be added +# to a task that runs after the source is available and before the +# do_deploy_source_date_epoch task is executed. +# +# If a recipe wishes to override the default behavior it should set it's own +# SOURCE_DATE_EPOCH or override the do_deploy_source_date_epoch_stamp task +# with recipe-specific functionality to write the appropriate +# SOURCE_DATE_EPOCH into the SDE_FILE. +# +# SOURCE_DATE_EPOCH is intended to be a reproducible value. This value should +# be reproducible for anyone who builds the same revision from the same +# sources. +# +# There are 4 ways the create_source_date_epoch_stamp function determines what +# becomes SOURCE_DATE_EPOCH: +# +# 1. Use the value from __source_date_epoch.txt file if this file exists. +# This file was most likely created in the previous build by one of the +# following methods 2,3,4. +# Alternatively, it can be provided by a recipe via SRC_URI. +# +# If the file does not exist: +# +# 2. If there is a git checkout, use the last git commit timestamp. +# Git does not preserve file timestamps on checkout. +# +# 3. Use the mtime of "known" files such as NEWS, CHANGLELOG, ... +# This works for well-kept repositories distributed via tarball. +# +# 4. Use the modification time of the youngest file in the source tree, if +# there is one. +# This will be the newest file from the distribution tarball, if any. +# +# 5. Fall back to a fixed timestamp (SOURCE_DATE_EPOCH_FALLBACK). +# +# Once the value is determined, it is stored in the recipe's SDE_FILE. + +def get_source_date_epoch_from_known_files(d, sourcedir): + source_date_epoch = None + newest_file = None + known_files = set(["NEWS", "ChangeLog", "Changelog", "CHANGES"]) + for file in known_files: + filepath = os.path.join(sourcedir, file) + if os.path.isfile(filepath): + mtime = int(os.lstat(filepath).st_mtime) + # There may be more than one "known_file" present, if so, use the youngest one + if not source_date_epoch or mtime > source_date_epoch: + source_date_epoch = mtime + newest_file = filepath + if newest_file: + bb.debug(1, "SOURCE_DATE_EPOCH taken from: %s" % newest_file) + return source_date_epoch + +def find_git_folder(d, sourcedir): + # First guess: WORKDIR/git + # This is the default git fetcher unpack path + workdir = d.getVar('WORKDIR') + gitpath = os.path.join(workdir, "git/.git") + if os.path.isdir(gitpath): + return gitpath + + # Second guess: ${S} + gitpath = os.path.join(sourcedir, ".git") + if os.path.isdir(gitpath): + return gitpath + + # Perhaps there was a subpath or destsuffix specified. + # Go looking in the WORKDIR + exclude = set(["build", "image", "license-destdir", "patches", "pseudo", + "recipe-sysroot", "recipe-sysroot-native", "sysroot-destdir", "temp"]) + for root, dirs, files in os.walk(workdir, topdown=True): + dirs[:] = [d for d in dirs if d not in exclude] + if '.git' in dirs: + return os.path.join(root, ".git") + + bb.warn("Failed to find a git repository in WORKDIR: %s" % workdir) + return None + +def get_source_date_epoch_from_git(d, sourcedir): + if not "git://" in d.getVar('SRC_URI') and not "gitsm://" in d.getVar('SRC_URI'): + return None + + gitpath = find_git_folder(d, sourcedir) + if not gitpath: + return None + + # Check that the repository has a valid HEAD; it may not if subdir is used + # in SRC_URI + p = subprocess.run(['git', '--git-dir', gitpath, 'rev-parse', 'HEAD'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + if p.returncode != 0: + bb.debug(1, "%s does not have a valid HEAD: %s" % (gitpath, p.stdout.decode('utf-8'))) + return None + + bb.debug(1, "git repository: %s" % gitpath) + p = subprocess.run(['git', '-c', 'log.showSignature=false', '--git-dir', gitpath, 'log', '-1', '--pretty=%ct'], + check=True, stdout=subprocess.PIPE) + return int(p.stdout.decode('utf-8')) + +def get_source_date_epoch_from_youngest_file(d, sourcedir): + if sourcedir == d.getVar('WORKDIR'): + # These sources are almost certainly not from a tarball + return None + + # Do it the hard way: check all files and find the youngest one... + source_date_epoch = None + newest_file = None + for root, dirs, files in os.walk(sourcedir, topdown=True): + files = [f for f in files if not f[0] == '.'] + + for fname in files: + if fname == "singletask.lock": + # Ignore externalsrc/devtool lockfile [YOCTO #14921] + continue + filename = os.path.join(root, fname) + try: + mtime = int(os.lstat(filename).st_mtime) + except ValueError: + mtime = 0 + if not source_date_epoch or mtime > source_date_epoch: + source_date_epoch = mtime + newest_file = filename + + if newest_file: + bb.debug(1, "Newest file found: %s" % newest_file) + return source_date_epoch + +def fixed_source_date_epoch(d): + bb.debug(1, "No tarball or git repo found to determine SOURCE_DATE_EPOCH") + source_date_epoch = d.getVar('SOURCE_DATE_EPOCH_FALLBACK') + if source_date_epoch: + bb.debug(1, "Using SOURCE_DATE_EPOCH_FALLBACK") + return int(source_date_epoch) + return 0 + +def get_source_date_epoch(d, sourcedir): + return ( + get_source_date_epoch_from_git(d, sourcedir) or + get_source_date_epoch_from_youngest_file(d, sourcedir) or + fixed_source_date_epoch(d) # Last resort + ) + +def epochfile_read(epochfile, d): + cached, efile = d.getVar('__CACHED_SOURCE_DATE_EPOCH') or (None, None) + if cached and efile == epochfile: + return cached + + if cached and epochfile != efile: + bb.debug(1, "Epoch file changed from %s to %s" % (efile, epochfile)) + + source_date_epoch = int(d.getVar('SOURCE_DATE_EPOCH_FALLBACK')) + try: + with open(epochfile, 'r') as f: + s = f.read() + try: + source_date_epoch = int(s) + except ValueError: + bb.warn("SOURCE_DATE_EPOCH value '%s' is invalid. Reverting to SOURCE_DATE_EPOCH_FALLBACK" % s) + source_date_epoch = int(d.getVar('SOURCE_DATE_EPOCH_FALLBACK')) + bb.debug(1, "SOURCE_DATE_EPOCH: %d" % source_date_epoch) + except FileNotFoundError: + bb.debug(1, "Cannot find %s. SOURCE_DATE_EPOCH will default to %d" % (epochfile, source_date_epoch)) + + d.setVar('__CACHED_SOURCE_DATE_EPOCH', (str(source_date_epoch), epochfile)) + return str(source_date_epoch) + +def epochfile_write(source_date_epoch, epochfile, d): + + bb.debug(1, "SOURCE_DATE_EPOCH: %d" % source_date_epoch) + bb.utils.mkdirhier(os.path.dirname(epochfile)) + + tmp_file = "%s.new" % epochfile + with open(tmp_file, 'w') as f: + f.write(str(source_date_epoch)) + os.rename(tmp_file, epochfile) -- 2.39.2