Re: [PATCH v2 1/1] use debian snapshot mirror if SOURCE_DATE_EPOCH is set

["MOESSBAUER, Felix" <felix.moessbauer@siemens.com>]

On Mon, 2024-04-08 at 07:02 +0200, Jan Kiszka wrote:
> On 03.04.24 16:12, Felix Moessbauer wrote:
> > In case the SOURCE_DATE_EPOCH variable is set, we switch the debian
> > mirror to a snapshot mirror. The used date is derived from the
> > value of
> > SOURCE_DATE_EPOCH. Similar to the DISTRO_APT_PREMIRRORS, this
> > mirror is
> > only injected temporarily during the build.
> > 
> > To further control the behavior, we introduce the following
> > variables:
> > 
> > - ISAR_USE_DEBIAN_SNAPSHOTS: overwrite if a snapshot shall be used
> > - ISAR_DEBIAN_SNAPSHOT_MIRROR: The snapshot mirror to use (defaults
> > to
> >   snapshot-cloudflare.debian.org)
> > 
> > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> > ---
> > Changes since v1:
> > 
> > - disable valid-until checking for both bootstrapping and schroot
> > - conditionally make the SOURCE_DATE_EPOCH a vardep of bootstrap
> > - move ISAR_USE_DEBIAN_SNAPSHOTS to the bitbake.conf as it is used
> >   both in bootstrap, as well as in rootfs.
> > 
> > Best regards,
> > Felix Moessbauer
> > Siemens AG
> > 
> >  RECIPE-API-CHANGELOG.md                         |  6 ++++++
> >  doc/user_manual.md                              |  3 +++
> >  meta/classes/rootfs.bbclass                     |  3 +++
> >  meta/conf/bitbake.conf                          |  1 +
> >  .../isar-bootstrap/isar-bootstrap.inc           | 17
> > +++++++++++++++--
> >  5 files changed, 28 insertions(+), 2 deletions(-)
> > 
> > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> > index 6653ab43..c146d60c 100644
> > --- a/RECIPE-API-CHANGELOG.md
> > +++ b/RECIPE-API-CHANGELOG.md
> > @@ -583,3 +583,9 @@ Cross compiling kernel modules for distro
> > kernels is not supported in debian.
> >  To simplify downstream kernel module builds, we automatically turn
> > of cross
> >  compilation for a user-provided module when building it for a
> > distro kernel.
> >  
> > +
> > +### Build against debian snapshot mirror if SOURCE_DATE_EPOCH is
> > set
> > +
> > +In case the bitbake variable `SOURCE_DATE_EPOCH` is set, a debian
> > snapshot
> > +mirror is used. This can be overwritten with
> > `ISAR_USE_DEBIAN_SNAPSHOTS`.
> > +The snapshot to use is specified in `ISAR_DEBIAN_SNAPSHOT_MIRROR`.
> > diff --git a/doc/user_manual.md b/doc/user_manual.md
> > index 419d5339..227ce5f9 100644
> > --- a/doc/user_manual.md
> > +++ b/doc/user_manual.md
> > @@ -425,12 +425,15 @@ Some other variables include:
> >  
> >   - `IMAGE_INSTALL` - The list of custom packages to build and
> > install to target image, please refer to relative chapter for more
> > information.
> >   - `BB_NUMBER_THREADS` - The number of `bitbake` jobs that can be
> > run in parallel. Please set this option according to your host CPU
> > cores number.
> > + - `SOURCE_DATE_EPOCH` - The unix timestamp passed to all tooling
> > to make the results reproducible. This variable is optional.
> >   - `HOST_DISTRO` - The distro to use for SDK root filesystem. This
> > variable is optional.
> >   - `HOST_ARCH` - The Debian architecture of SDK root filesystem
> > (e.g., `amd64`). By default set to current Debian host
> > architecture. This variable is optional.
> >   - `HOST_DISTRO_APT_SOURCES` - List of apt source files for SDK
> > root filesystem. This variable is optional.
> >   - `HOST_DISTRO_APT_PREFERENCES` - List of apt preference files
> > for SDK root filesystem. This variable is optional.
> >   - `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to
> > DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt
> > bootstrap repo for the host.
> >   - `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to
> > the default URI in the format `ftp.debian.org my.preferred.mirror`.
> > This variable is optional. PREMIRRORS will be used only for the
> > build. The final images will have the sources list as mentioned in
> > DISTRO_APT_SOURCES.
> > + - `ISAR_USE_DEBIAN_SNAPSHOTS` - Use a frozen debian snapshot
> > instead of the live mirror. Auto-enabled if `SOURCE_DATE_EPOCH` is
> > set. Optional.
> > + - `ISAR_DEBIAN_SNAPSHOT_MIRROR` - The snapshot mirror to use.
> > Defaults to `snapshot-cloudflare.debian.org`.
> >   - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify
> > apt repos for apt installation after bootstrapping.
> >   - `FILESEXTRAPATHS` - The default directories BitBake uses when
> > it processes recipes are initially defined by the FILESPATH
> > variable. You can extend FILESPATH variable by using
> > FILESEXTRAPATHS.
> >   - `FILESOVERRIDES` - A subset of OVERRIDES used by the build
> > system for creating FILESPATH. The FILESOVERRIDES variable uses
> > overrides to automatically extend the FILESPATH variable.
> > diff --git a/meta/classes/rootfs.bbclass
> > b/meta/classes/rootfs.bbclass
> > index fb14f3ca..0d7744f7 100644
> > --- a/meta/classes/rootfs.bbclass
> > +++ b/meta/classes/rootfs.bbclass
> > @@ -112,6 +112,9 @@ rootfs_configure_apt() {
> >      mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d'
> >      {
> >          echo 'Acquire::Retries "3";'
> > +        if [ "${ISAR_USE_DEBIAN_SNAPSHOTS}" = "1" ]; then
> > +            echo 'Acquire::Check-Valid-Until "false";'
> > +        fi
> >          echo 'APT::Install-Recommends "0";'
> >          echo 'APT::Install-Suggests "0";'
> >      } > '${ROOTFSDIR}/etc/apt/apt.conf.d/50isar'
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > index 91c5c815..84c9a9bb 100644
> > --- a/meta/conf/bitbake.conf
> > +++ b/meta/conf/bitbake.conf
> > @@ -68,6 +68,7 @@ KERNEL_FILE ?= "vmlinuz"
> >  KERNEL_FILE:mipsel ?= "vmlinux"
> >  KERNEL_FILE:riscv64 ?= "vmlinux"
> >  KERNEL_FILE:arm64 ?= "vmlinux"
> > +ISAR_USE_DEBIAN_SNAPSHOTS ??= "${@'1' if
> > d.getVar('SOURCE_DATE_EPOCH') else '0'}"
> >  
> >  MACHINEOVERRIDES ?= "${MACHINE}"
> >  DISTROOVERRIDES ?= "${DISTRO}"
> > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > index f548e202..d44876a4 100644
> > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > @@ -32,6 +32,8 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if
> > bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR
> >  BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if
> > bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else
> > 'DISTRO')}"
> >  BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if
> > bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else
> > 'BASE_DISTRO')}"
> >  FILESEXTRAPATHS:append = ":${BBPATH}"
> > +# reproducible builds
> > +ISAR_DEBIAN_SNAPSHOT_MIRROR ??= "snapshot-cloudflare.debian.org"
> >  
> >  inherit deb-dl-dir
> >  
> > @@ -111,9 +113,15 @@ def
> > parse_aptsources_list_line(source_list_line):
> >  
> >  def get_apt_source_mirror(d, aptsources_entry_list):
> >      import re
> > +    import time
> >  
> >      if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> >          premirrors = "\S*
> > file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"
> > +    elif
> > bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')):
> > +        snapshot_mirror = d.getVar('ISAR_DEBIAN_SNAPSHOT_MIRROR')
> > +        source_date_epoch = d.getVar('SOURCE_DATE_EPOCH') or
> > int(time.time())
> > +        snapshot_date = time.strftime('%Y%m%dT%H%M%SZ',
> > time.gmtime(int(source_date_epoch)))
> > +        premirrors = 'deb.debian.org/(.*)
> > {}/archive/\\1/{}/\n'.format(snapshot_mirror, snapshot_date)
> >      else:
> >          premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or ""
> >      mirror_list = [entry.split()
> > @@ -265,6 +273,7 @@ def get_host_release():
> >      return rel
> >  
> >  do_bootstrap[vardeps] += " \
> > +    ${@'SOURCE_DATE_EPOCH' if
> > bb.utils.to_boolean(d.getVar('ISAR_USE_DEBIAN_SNAPSHOTS')) else ''}
> > \
> >      DISTRO_APT_PREMIRRORS \
> >      ISAR_ENABLE_COMPAT_ARCH \
> >      ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
> > @@ -382,8 +391,12 @@ do_bootstrap() {
> >              chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture
> > ${COMPAT_DISTRO_ARCH}
> >          fi
> >  
> > -        chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \
> > -                                -o APT::Update::Error-Mode=any
> > +
> > +        APT_UPDATE_OPTS="-o APT::Update::Error-Mode=any"
> > +        if [ "${ISAR_USE_DEBIAN_SNAPSHOTS}" = "1" ]; then
> > +            APT_UPDATE_OPTS="${APT_UPDATE_OPTS} -o Acquire::Check-
> > Valid-Until=false"
> > +        fi
> > +        chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
> > ${APT_UPDATE_OPTS}
> >          chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
> >          chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
> >                                  -o Debug::pkgProblemResolver=yes
> 
> This can be a valuable addition but it needs more thoughts first:
> 
> How do you plan to handle snapshot.ubuntu.com? And what will you when
> the repo list contains a third-party repo for which there is no

Good point. How about moving the ISAR_DEBIAN_SNAPSHOT_MIRROR to the
distro.conf?

> snapshot? That should at least raise a warning, rather than silently
> suggesting to produce a reproducible image.

Hmm... I don't want to raise a warning in this case, as this can very
well be a stable source. We simply can't tell by just looking at the
sources. Even having a stable mirror does not mean we can reproduce the
image: It is just another piece in the big puzzle to make it finally
work.

It is currently accepted (across various projects, including docker
buildx), that reproducibility is something that needs to be checked
manually and by that is stated by a user. 

> 
> BTW, we should likely also double-check how we define the
> SOURCE_DATE_EPOCH value, in comparison to OE. See my reply on your
> kas
> patch.

Right. According to [1]->Current Development->Timestamps, [2], the SDE
is injected into the build as an environment variable. However, this
can and should be overwritten on a per-recipe basis by using the
bitbake variable SOURCE_DATE_EPOCH.

Given that, the best way to inject is via env + addition to
BB_ENV_PASSTHROUGH_ADDITIONS.

CC'ing Quirin, as we now have a similar discussion on the CIP ML as
well.

Felix

[1] https://wiki.yoctoproject.org/wiki/Reproducible_Builds
[2]
https://github.com/openembedded/openembedded-core/blob/486ed69939f1b7572f121960c2dde246032686e7/meta/conf/bitbake.conf#L686

> 
> Jan
> 

-- 
Siemens AG, Technology
Linux Expert Center