From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7353602520359370752 X-Forwarded-Encrypted: i=2; AJvYcCXcEktZysvbw1N8g9M5IaWUZ8J/3R0FvAjTgKwetma3gQ4M+QRCDX5H2T73BIXQaRvv1HTuTd/A1+zC9qeVGqU8C7BQ38Rdsc/vrQc= X-Received: by 2002:a2e:bb8f:0:b0:2ea:8370:8a86 with SMTP id 38308e7fff4ca-2eac79baa5cmr29539141fa.10.1717679715591; Thu, 06 Jun 2024 06:15:15 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6402:520c:b0:578:61d9:9aa6 with SMTP id 4fb4d7f45d1cf-57c4cb2441bls43278a12.1.-pod-prod-06-eu; Thu, 06 Jun 2024 06:15:13 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCV2i4bksHcKb73kpI0E5RdgUx0K1ph9WIdanDO2hzDfaobmLQ9NVQjRT921/opuaX0uRWcFtlBIkY/gJbuw+JHe24gQ3Ue3Wsk5Q0Q= X-Google-Smtp-Source: AGHT+IG8u7BVWLXBOO3hvQsUZZ8bdWSf3vBx3vfpfUttXDzGFX/mipX9jvblJ5zkEWlKe/7s+F1z X-Received: by 2002:a50:bac8:0:b0:578:6198:d6fa with SMTP id 4fb4d7f45d1cf-57a8b6740a0mr3729819a12.2.1717679713383; Thu, 06 Jun 2024 06:15:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1717679713; cv=none; d=google.com; s=arc-20160816; b=X5n242RRvMdPuUGsgrPnvwvkLnDvegDb0REZzQTZ2IHlVnJjjeOr6BX+doggtsnClA 2b30KUZn59mi7r9aWpUwtsIFBwncmA2ieb/NKsiIQDF6zGwC3fZtdXo0zxCnatBaRBDQ lYfq3Xul+4pKqGLN8yzxbJZqulrKabYWgVB2yksY8ZNqtc+gSOgcNE8yGMHheTWLsSqN DizUmbS75ZGLKMVWMLfSvCvrgp1qKoO9J6rop6XEhRFP2Xy/RxBeGE4IPFtIsjcDesFM aATwdJ32rZgtMEMziU+wK1gZr8YRJDX6K0lsNYg4TvHA54X2RRJl3JvQafs8h2lHbD8v q8JA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:in-reply-to:from:content-language :references:to:subject:user-agent:mime-version:date:message-id; bh=CVE6iei9a1wYVJnE7mSqCNNcbtquc5Oy/XzgE0ak6j4=; fh=ez+yYIN2+t1/A1KtVD1essDQ6+anTTnZTbSXjuK9WaU=; b=iNIlqqSZq3kqDd08DPorAneSmZRSgOqy3gJRs7UkpM6Be9xa8GKyos8zWu5GeBG/HY 8U6mzqyQdUZ0Pi1u3R71cQucsK5yw+56IVEg9GvywvEa7A34D9FT/8b4mTuH0Etr/fOt 3uSk2py5YQxmscicrGeaBysnxS3V8fAUg0pFIrNSmSSb0gZQ7AhzfJN1tynTgzQ5wD/a Qh1oaYcS2jQU1WX+rNZpTjL5HII6cokpTjx6e5k1JhDpoLyJ4ubRFn0zgyFCo+yw1Kcm pZPaYxEvFdsalTNiwBgQDwGnFCLzSfeOyO17xsjKYiOZthQphe8NUgXxTQsDGO11xyk3 J1SA==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 4fb4d7f45d1cf-57aae2272d9si25351a12.4.2024.06.06.06.15.13 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 06 Jun 2024 06:15:13 -0700 (PDT) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from [127.0.0.1] (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 456DFBsG023302 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 6 Jun 2024 15:15:12 +0200 Message-ID: Date: Thu, 6 Jun 2024 16:15:10 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 2/5] use apt snapshot mirror if ISAR_USE_APT_SNAPSHOT is set To: Felix Moessbauer , isar-users@googlegroups.com References: <20240409155549.826454-1-felix.moessbauer@siemens.com> <20240409155549.826454-3-felix.moessbauer@siemens.com> Content-Language: en-US, ru-RU From: Anton Mikanovich In-Reply-To: <20240409155549.826454-3-felix.moessbauer@siemens.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: zfdXrV1S7nl5 09/04/2024 18:55, 'Felix Moessbauer' via isar-users wrote: > This patch adds infrastructure to switch the apt sources to a frozen > snapshot mirror. To build against a mirror, set ISAR_USE_APT_SNAPSHOT=1. > As the mirror is distro specific, it is configured in the distro config > for all supported distros (currently only debian and ubuntu). For > unsupported distros, a meaningful error message is emitted on enabling > the snapshot build. Similar to the DISTRO_APT_PREMIRRORS, this mirror is > only injected temporarily and does not end up in the final apt sources > list. > > To further control the behavior, we introduce the following variables: > > - DISTRO_APT_SNAPSHOT_PREMIRROR: The snapshot mirror to use. Syntax > identical to DISTRO_APT_PREMIRRORS. > - ISAR_APT_SNAPSHOT_TIMESTAMP: Unix timestamp of the snapshot. This is > automatically derived from the SOURCE_DATE_EPOCH if not set. > > Signed-off-by: Felix Moessbauer > --- > RECIPE-API-CHANGELOG.md | 6 ++++++ > doc/user_manual.md | 3 +++ > meta-isar/conf/distro/ubuntu-common.inc | 3 +++ > meta/conf/bitbake.conf | 3 +++ > meta/conf/distro/debian-common.conf | 3 +++ > .../isar-bootstrap/isar-bootstrap.inc | 16 ++++++++++++++++ > 6 files changed, 34 insertions(+) > > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md > index 6653ab43..e6861523 100644 > --- a/RECIPE-API-CHANGELOG.md > +++ b/RECIPE-API-CHANGELOG.md > @@ -583,3 +583,9 @@ Cross compiling kernel modules for distro kernels is not supported in debian. > To simplify downstream kernel module builds, we automatically turn of cross > compilation for a user-provided module when building it for a distro kernel. > > + > +### Build against debian snapshot mirror > + > +To build against a distributions snapshot mirror, set `ISAR_USE_APT_SNAPSHOT="1"`. > +The mirror to use is specified in `DISTRO_APT_SNAPSHOT_PREMIRROR` and usually > +pre-defined in the distro config. > diff --git a/doc/user_manual.md b/doc/user_manual.md > index 419d5339..70741968 100644 > --- a/doc/user_manual.md > +++ b/doc/user_manual.md > @@ -431,6 +431,9 @@ Some other variables include: > - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files for SDK root filesystem. This variable is optional. > - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host. > - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES. > + - `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of the live mirror. Optional. > + - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported distros. > + - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Timestamp of the apt snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not overwritten. > - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping. > - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS. > - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable. > diff --git a/meta-isar/conf/distro/ubuntu-common.inc b/meta-isar/conf/distro/ubuntu-common.inc > index 9d8a843b..54bb747a 100644 > --- a/meta-isar/conf/distro/ubuntu-common.inc > +++ b/meta-isar/conf/distro/ubuntu-common.inc > @@ -32,3 +32,6 @@ IMAGE_PREINSTALL += "init" > IMAGE_PREINSTALL += "initramfs-tools" > > IMAGER_INSTALL:wic += "python3-distutils" > + > +# snapshot mirror for reproducible builds > +DISTRO_APT_SNAPSHOT_PREMIRROR ??= "(http|https)://archive.ubuntu.com/(.*) https://snapshot.ubuntu.com/\2/${APT_SNAPSHOT_DATE}/\n" > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 1da3ecac..4cfa8b10 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -145,6 +145,9 @@ export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}" > # A SOURCE_DATE_EPOCH of '0' might be misinterpreted as no SDE > # ISAR: set value to date of latest release > SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251" > +# Debian snapshots > +ISAR_USE_APT_SNAPSHOT ??= "0" > +ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}" > > # Default parallelism and resource usage for xz > XZ_MEMLIMIT ?= "50%" > diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf > index 1e1dfc83..db538510 100644 > --- a/meta/conf/distro/debian-common.conf > +++ b/meta/conf/distro/debian-common.conf > @@ -39,3 +39,6 @@ SYSTEMD_BOOTLOADER_INSTALL:sid = "systemd-boot-efi:${DISTRO_ARCH}" > > COMPAT_DISTRO_ARCH:amd64 = "i386" > COMPAT_DISTRO_ARCH:arm64 = "armhf" > + > +# snapshot mirror for reproducible builds > +DISTRO_APT_SNAPSHOT_PREMIRROR ??= "deb.debian.org/(.*) snapshot-cloudflare.debian.org/archive/\1/${APT_SNAPSHOT_DATE}/\n" > \ No newline at end of file > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > index 17f19fd8..733a23df 100644 > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > @@ -30,6 +30,9 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR > BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}" > BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}" > FILESEXTRAPATHS:append = ":${BBPATH}" > +# reproducible builds, only enabled if ISAR_USE_APT_SNAPSHOT > +ISAR_APT_SNAPSHOT_MIRROR ??= "" > +APT_SNAPSHOT_DATE = "${@ get_apt_snapshot_date(d)}" > > inherit deb-dl-dir > > @@ -107,11 +110,21 @@ def parse_aptsources_list_line(source_list_line): > > return [type, options, source, suite, components] > > +def get_apt_snapshot_date(d): > + import time > + source_date_epoch = d.getVar('ISAR_APT_SNAPSHOT_TIMESTAMP') > + return time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch))) > + > def get_apt_source_mirror(d, aptsources_entry_list): > import re > > + # this is executed during parsing. No error checking possible > + use_snapshot = bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) > + snapshot_mirror = d.getVar('DISTRO_APT_SNAPSHOT_PREMIRROR') > if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): > premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" > + elif use_snapshot and snapshot_mirror: > + premirrors = snapshot_mirror > else: > premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or "" > mirror_list = [entry.split() > @@ -126,6 +139,8 @@ def get_apt_source_mirror(d, aptsources_entry_list): > new_aptsources_entry_list[2] = re.sub(regex, replace, > aptsources_entry_list[2], > count = 1) > + if use_snapshot: > + new_aptsources_entry_list[1] = "[check-valid-until=no]" > return new_aptsources_entry_list > > return aptsources_entry_list > @@ -240,6 +255,7 @@ do_apt_config_prepare[vardeps] += " \ > APTSRCS \ > ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \ > DEPLOY_ISAR_BOOTSTRAP \ > + ${@'DISTRO_APT_SNAPSHOT_PREMIRROR' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''} \ > " > python do_apt_config_prepare() { > apt_preferences_out = d.getVar("APTPREFS") Hello Felix, How ISAR_APT_SNAPSHOT_MIRROR supposed to be used?