[PATCH 1/1] snapshots: add option to use separate timestamp for security component

[PATCH 1/1] snapshots: add option to use separate timestamp for security component

['Felix Moessbauer' via isar-users]

Before releasing a product all available security fixes should be
included. However, you might not want to get other proposed updates.
With the previous snapshot logic it was not possible to model this, as a
single timestamp is used for all apt source-list entries.

We change that by adding a "security" flag to snapshot date variables.
By that, dedicated control over the security distribution is possible.

For now, we only add this logic for debian distributions (not ubuntu),
as only there we have a dedicated security distribution.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
For details about the used terms (e.g. "security distribution") please
refer to https://wiki.debian.org/SourcesList.

 doc/user_manual.md                  | 2 ++
 meta/classes/bootstrap.bbclass      | 5 ++++-
 meta/conf/distro/debian-common.conf | 5 ++++-
 3 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 1e505c66..fd4fe249 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -447,7 +447,9 @@ Some other variables include:
  - `ISAR_APT_DELAY_MAX` - Maximum time in seconds apt performs retries. Optional
  - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported distros.
  - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Unix timestamp of the apt snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not overwritten. (Consider `ISAR_APT_SNAPSHOT_DATE` for a more user friendly format)
+ - `ISAR_APT_SNAPSHOT_TIMESTAMP[security]` - Unix timestamp of the security distribution. Optional.
  - `ISAR_APT_SNAPSHOT_DATE` - Timestamp in upstream format (e.g. `20240702T082400Z`) of the apt snapshot. Overrides `ISAR_APT_SNAPSHOT_TIMESTAMP` if set. Otherwise, will be automatically derived from `ISAR_APT_SNAPSHOT_TIMESTAMP`
+ - `ISAR_APT_SNAPSHOT_DATE[security]` - Timestamp in upstream format of the security distribution. Optional.
  - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping.
  - `FILESEXTRAPATHS` - The default directories BitBake uses when it processes recipes are initially defined by the FILESPATH variable. You can extend FILESPATH variable by using FILESEXTRAPATHS.
  - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system for creating FILESPATH. The FILESOVERRIDES variable uses overrides to automatically extend the FILESPATH variable.
diff --git a/meta/classes/bootstrap.bbclass b/meta/classes/bootstrap.bbclass
index f5b92808..c0644acb 100644
--- a/meta/classes/bootstrap.bbclass
+++ b/meta/classes/bootstrap.bbclass
@@ -28,6 +28,7 @@ BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('B
 BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}"
 BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO_ARCH')}"
 ISAR_APT_SNAPSHOT_DATE ?= "${@ get_isar_apt_snapshot_date(d)}"
+ISAR_APT_SNAPSHOT_DATE[security] ?= "${@ get_isar_apt_snapshot_date(d, 'security')}"
 
 python () {
     distro_bootstrap_keys = (d.getVar("DISTRO_BOOTSTRAP_KEYS") or "").split()
@@ -101,9 +102,11 @@ def parse_aptsources_list_line(source_list_line):
 
     return [type, options, source, suite, components]
 
-def get_isar_apt_snapshot_date(d):
+def get_isar_apt_snapshot_date(d, dist=None):
     import time
     source_date_epoch = d.getVar('ISAR_APT_SNAPSHOT_TIMESTAMP')
+    if dist:
+        source_date_epoch = d.getVarFlag('ISAR_APT_SNAPSHOT_TIMESTAMP', dist) or source_date_epoch
     return time.strftime('%Y%m%dT%H%M%SZ', time.gmtime(int(source_date_epoch)))
 
 def get_apt_source_mirror(d, aptsources_entry_list):
diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf
index 92a15404..b5d8aa9a 100644
--- a/meta/conf/distro/debian-common.conf
+++ b/meta/conf/distro/debian-common.conf
@@ -40,4 +40,7 @@ COMPAT_DISTRO_ARCH:amd64 = "i386"
 COMPAT_DISTRO_ARCH:arm64 = "armhf"
 
 # snapshot mirror for reproducible builds
-DISTRO_APT_SNAPSHOT_PREMIRROR ??= "deb.debian.org/(.*) snapshot.debian.org/archive/\1/${ISAR_APT_SNAPSHOT_DATE}\n"
+DISTRO_APT_SNAPSHOT_PREMIRROR ??= " \
+    deb.debian.org/(debian-security)/? snapshot.debian.org/archive/\1/${@d.getVarFlag('ISAR_APT_SNAPSHOT_DATE', 'security')}\n \
+    deb.debian.org/(.*)/? snapshot.debian.org/archive/\1/${ISAR_APT_SNAPSHOT_DATE}\n \
+"
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20241206131702.60476-1-felix.moessbauer%40siemens.com.

Re: [PATCH 1/1] snapshots: add option to use separate timestamp for security component

[Uladzimir Bely]

On Fri, 2024-12-06 at 14:17 +0100, 'Felix Moessbauer' via isar-users
wrote:
> Before releasing a product all available security fixes should be
> included. However, you might not want to get other proposed updates.
> With the previous snapshot logic it was not possible to model this,
> as a
> single timestamp is used for all apt source-list entries.
> 
> We change that by adding a "security" flag to snapshot date
> variables.
> By that, dedicated control over the security distribution is
> possible.
> 
> For now, we only add this logic for debian distributions (not
> ubuntu),
> as only there we have a dedicated security distribution.
> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
> For details about the used terms (e.g. "security distribution")
> please
> refer to https://wiki.debian.org/SourcesList.
> 
>  doc/user_manual.md                  | 2 ++
>  meta/classes/bootstrap.bbclass      | 5 ++++-
>  meta/conf/distro/debian-common.conf | 5 ++++-
>  3 files changed, 10 insertions(+), 2 deletions(-)
> 
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 1e505c66..fd4fe249 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -447,7 +447,9 @@ Some other variables include:
>   - `ISAR_APT_DELAY_MAX` - Maximum time in seconds apt performs
> retries. Optional
>   - `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to
> `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported
> distros.
>   - `ISAR_APT_SNAPSHOT_TIMESTAMP` - Unix timestamp of the apt
> snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not
> overwritten. (Consider `ISAR_APT_SNAPSHOT_DATE` for a more user
> friendly format)
> + - `ISAR_APT_SNAPSHOT_TIMESTAMP[security]` - Unix timestamp of the
> security distribution. Optional.
>   - `ISAR_APT_SNAPSHOT_DATE` - Timestamp in upstream format (e.g.
> `20240702T082400Z`) of the apt snapshot. Overrides
> `ISAR_APT_SNAPSHOT_TIMESTAMP` if set. Otherwise, will be
> automatically derived from `ISAR_APT_SNAPSHOT_TIMESTAMP`
> + - `ISAR_APT_SNAPSHOT_DATE[security]` - Timestamp in upstream format
> of the security distribution. Optional.
>   - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt
> repos for apt installation after bootstrapping.
>   - `FILESEXTRAPATHS` - The default directories BitBake uses when it
> processes recipes are initially defined by the FILESPATH variable.
> You can extend FILESPATH variable by using FILESEXTRAPATHS.
>   - `FILESOVERRIDES` - A subset of OVERRIDES used by the build system
> for creating FILESPATH. The FILESOVERRIDES variable uses overrides to
> automatically extend the FILESPATH variable.
> diff --git a/meta/classes/bootstrap.bbclass
> b/meta/classes/bootstrap.bbclass
> index f5b92808..c0644acb 100644
> --- a/meta/classes/bootstrap.bbclass
> +++ b/meta/classes/bootstrap.bbclass
> @@ -28,6 +28,7 @@ BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if
> bb.utils.to_boolean(d.getVar('B
>  BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if
> bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else
> 'BASE_DISTRO')}"
>  BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if
> bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else
> 'DISTRO_ARCH')}"
>  ISAR_APT_SNAPSHOT_DATE ?= "${@ get_isar_apt_snapshot_date(d)}"
> +ISAR_APT_SNAPSHOT_DATE[security] ?= "${@
> get_isar_apt_snapshot_date(d, 'security')}"
>  
>  python () {
>      distro_bootstrap_keys = (d.getVar("DISTRO_BOOTSTRAP_KEYS") or
> "").split()
> @@ -101,9 +102,11 @@ def
> parse_aptsources_list_line(source_list_line):
>  
>      return [type, options, source, suite, components]
>  
> -def get_isar_apt_snapshot_date(d):
> +def get_isar_apt_snapshot_date(d, dist=None):
>      import time
>      source_date_epoch = d.getVar('ISAR_APT_SNAPSHOT_TIMESTAMP')
> +    if dist:
> +        source_date_epoch =
> d.getVarFlag('ISAR_APT_SNAPSHOT_TIMESTAMP', dist) or
> source_date_epoch
>      return time.strftime('%Y%m%dT%H%M%SZ',
> time.gmtime(int(source_date_epoch)))
>  
>  def get_apt_source_mirror(d, aptsources_entry_list):
> diff --git a/meta/conf/distro/debian-common.conf
> b/meta/conf/distro/debian-common.conf
> index 92a15404..b5d8aa9a 100644
> --- a/meta/conf/distro/debian-common.conf
> +++ b/meta/conf/distro/debian-common.conf
> @@ -40,4 +40,7 @@ COMPAT_DISTRO_ARCH:amd64 = "i386"
>  COMPAT_DISTRO_ARCH:arm64 = "armhf"
>  
>  # snapshot mirror for reproducible builds
> -DISTRO_APT_SNAPSHOT_PREMIRROR ??= "deb.debian.org/(.*)
> snapshot.debian.org/archive/\1/${ISAR_APT_SNAPSHOT_DATE}\n"
> +DISTRO_APT_SNAPSHOT_PREMIRROR ??= " \
> +    deb.debian.org/(debian-security)/?
> snapshot.debian.org/archive/\1/${@d.getVarFlag('ISAR_APT_SNAPSHOT_DAT
> E', 'security')}\n \
> +    deb.debian.org/(.*)/?
> snapshot.debian.org/archive/\1/${ISAR_APT_SNAPSHOT_DATE}\n \
> +"
> -- 
> 2.39.5
> 

Applied to next, thanks.

-- 
Best regards,
Uladzimir.



-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/4c26cf2664e442e44d8437dec49da6f46353a161.camel%40ilbers.de.