From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7353602520359370752 X-Received: by 2002:a05:6820:3094:b0:5a5:1fdc:8548 with SMTP id eu20-20020a056820309400b005a51fdc8548mr14798099oob.3.1712144007344; Wed, 03 Apr 2024 04:33:27 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac8:5d92:0:b0:431:3419:79d3 with SMTP id d18-20020ac85d92000000b00431341979d3ls8405810qtx.0.-pod-prod-01-us; Wed, 03 Apr 2024 04:33:26 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFCH4/54j81CPf9/+seW25k9UmwAlnaPDy6w/MVVcgifShsZ31LZ2SqZaxFJrMEH/2TQivs X-Received: by 2002:a05:622a:60c:b0:432:bad8:ba70 with SMTP id z12-20020a05622a060c00b00432bad8ba70mr16335370qta.32.1712144005878; Wed, 03 Apr 2024 04:33:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1712144005; cv=none; d=google.com; s=arc-20160816; b=uP/LXqPz/J9rEL1ZpEHplhrxoNB6zMUn15fUap1KPe2ARBgCmXJD9jd4d2VVj6svEo Dma62oZMNzRBqyEZODAVCkKk/6FinXfHE4kjLO3jn6AkhG9t+BkVVwGjtbeEPPBvTxgH 3GxGZLcEY0LmBP34Jj4aL5pcjVhUGcs6e/3PHmMEWUh1E8aaqTYLGJoz8qaI4y/Y25iK 86iwZGEHJJ04jAIzCFoRwpeYxLOE9G5xI19DkyuuNdtRw8VHw50LHJrU1r7f99reYHWe YTH4wu52MyTf0CMM3VIyRUmDbgTTwmfqve+7AZc07OqxAN40Bxq77/1Jt7KgVa1aiyib Iv9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=qOhpFwKRAwtV47eKubkbS54J351QNBrku6i2nS0TnDc=; fh=bVX03jNaRhKZGJkxT8AB1TU04tIU5gqw6O8qRk+YAyc=; b=tgNtTAam5X8VMJhDRV+7AyZJo7T3eK+i8x0e5NQKGZthT7/HtlOMbDy7BBbLm8sz/b fbQks4YVxPcfm4swXQ+lXjtNzVShR5KBW27hKKvmVYG6AIF7Xm+ITzqYICmE+T3acYbe IEMLiFMfvHAYbHQq+qaJl7cgVqcnWlTMwiW7PWBcpvtv21Un7sEDP1eKsHDg+IOri101 u00COLD2EQyvcrYCsWbXry3GjDLZFcdAGDul3Ob30uINBeTDtHb/r9Nogsrw1J2tn48j NmMclAn9UzAzl95JDS8VbaY/EU+6Iy/QTpndBnWGo6rCW4ylUO5yAghOp3lfFZnMZeOP DtkQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=kJnzafQq; spf=pass (google.com: domain of fm-1321639-20240403113324f382b7e4d6db6c350e-6tehyk@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1321639-20240403113324f382b7e4d6db6c350e-6tEHYK@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Return-Path: Received: from mta-65-226.siemens.flowmailer.net (mta-65-226.siemens.flowmailer.net. [185.136.65.226]) by gmr-mx.google.com with ESMTPS id eq25-20020a05622a5e1900b0043140b1ef8fsi1180513qtb.3.2024.04.03.04.33.25 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2024 04:33:25 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-1321639-20240403113324f382b7e4d6db6c350e-6tehyk@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) client-ip=185.136.65.226; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=kJnzafQq; spf=pass (google.com: domain of fm-1321639-20240403113324f382b7e4d6db6c350e-6tehyk@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-1321639-20240403113324f382b7e4d6db6c350e-6tEHYK@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: by mta-65-226.siemens.flowmailer.net with ESMTPSA id 20240403113324f382b7e4d6db6c350e for ; Wed, 03 Apr 2024 13:33:24 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=felix.moessbauer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc; bh=qOhpFwKRAwtV47eKubkbS54J351QNBrku6i2nS0TnDc=; b=kJnzafQqNpcKwUCOmo9k9kGG+WfUWmz65V7AJ70MkKNTUDgWt13MaMuCnfpC5qianhPeas +tfso1JHM97dNZKMijAWVk11lDP6BLaHNGBq7uSSH5yVhILzcqPYdul3Carikehx98bhQIB0 gO3GtYIUvTwpiEyihdqYK7qiEPVIA=; From: Felix Moessbauer To: isar-users@googlegroups.com Cc: jan.kiszka@siemens.com, venkata.pyla@toshiba-tsip.com, kazuhiro3.hayashi@toshiba.co.jp, dinesh.kumar@toshiba-tsip.com, Felix Moessbauer Subject: [PATCH 1/1] use debian snapshot mirror if SOURCE_DATE_EPOCH is set Date: Wed, 3 Apr 2024 13:33:10 +0200 Message-Id: <20240403113310.135008-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-1321639:519-21489:flowmailer X-TUID: gzU7Strm9EVW In case the SOURCE_DATE_EPOCH variable is set, we switch the debian mirror to a snapshot mirror. The used date is derived from the value of SOURCE_DATE_EPOCH. Similar to the DISTRO_APT_PREMIRRORS, this mirror is only injected temporarily during the build. To further control the behavior, we introduce the following variables: - ISAR_USE_DEBIAN_SNAPSHOTS: overwrite if a snapshot shall be used - ISAR_DEBIAN_SNAPSHOT_MIRROR: The snapshot mirror to use (defaults to snapshot-cloudflare.debian.org) Signed-off-by: Felix Moessbauer --- Dear maintainers, I'm not quite sure if the introduced variables also need to be added to the vardeps of e.g. the bootstrap task. Please double check this. Best regards, Felix Moessbauer Siemens AG RECIPE-API-CHANGELOG.md | 6 ++++++ doc/user_manual.md | 3 +++ meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 9 +++++++++ 3 files changed, 18 insertions(+) diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index 6653ab43..c146d60c 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -583,3 +583,9 @@ Cross compiling kernel modules for distro kernels is not supported in debian. To simplify downstream kernel module builds, we automatically turn of cross compilation for a user-provided module when building it for a distro kernel. + +### Build against debian snapshot mirror if SOURCE_DATE_EPOCH is set + +In case the bitbake variable `SOURCE_DATE_EPOCH` is set, a debian snapshot +mirror is used. This can be overwritten with `ISAR_USE_DEBIAN_SNAPSHOTS`. +The snapshot to use is specified in `ISAR_DEBIAN_SNAPSHOT_MIRROR`. diff --git a/doc/user_manual.md b/doc/user_manual.md index 419d5339..227ce5f9 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -425,12 +425,15 @@ Some other variables include: - `IMAGE_INSTALL` - The list of custom packages to build and install to target image, please refer to relative chapter for more information. - `BB_NUMBER_THREADS` - The number of `bitbake` jobs that can be run in parallel. Please set this option according to your host CPU cores number. + - `SOURCE_DATE_EPOCH` - The unix timestamp passed to all tooling to make the results reproducible. This variable is optional. - `HOST_DISTRO` - The distro to use for SDK root filesystem. This variable is optional. - `HOST_ARCH` - The Debian architecture of SDK root filesystem (e.g., `amd64`). By default set to current Debian host architecture. This variable is optional. - `HOST_DISTRO_APT_SOURCES` - List of apt source files for SDK root filesystem. This variable is optional. - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for SDK root filesystem. This variable is optional. - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host. - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES. + - `ISAR_USE_DEBIAN_SNAPSHOTS` - Use a frozen debian snapshot instead of the live mirror. Auto-enabled if `SOURCE_DATE_EPOCH` is set. Optional. + - `ISAR_DEBIAN_SNAPSHOT_MIRROR` - The snapshot mirror to use. Defaults to `snapshot-cloudflare.debian.org`. - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping. - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS. - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable. diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index f548e202..1e5a2911 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -32,6 +32,9 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}" BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}" FILESEXTRAPATHS:append = ":${BBPATH}" +# reproducible builds +ISAR_USE_DEBIAN_SNAPSHOTS ??= "${@'1' if d.getVar('SOURCE_DATE_EPOCH') else '0'}" +ISAR_DEBIAN_SNAPSHOT_MIRROR ??= "snapshot-cloudflare.debian.org" inherit deb-dl-dir @@ -111,9 +114,15 @@ def parse_aptsources_list_line(source_list_line): def get_apt_source_mirror(d, aptsources_entry_list): import re + import time if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" + elif bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')): + snapshot_mirror = d.getVar('ISAR_DEBIAN_SNAPSHOT_MIRROR') + source_date_epoch = d.getVar('SOURCE_DATE_EPOCH') or int(time.time()) + snapshot_date = time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch))) + premirrors = 'deb.debian.org/(.*) {}/archive/\\1/{}/\n'.format(snapshot_mirror, snapshot_date) else: premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or "" mirror_list = [entry.split() -- 2.39.2