[PATCH 1/1] add interface to passthrough env vars to sbuild

[PATCH 1/1] add interface to passthrough env vars to sbuild

[Felix Moessbauer]

This patch introduces the bb variable SBUILD_PASSTHROUGH_ADDITIONS.
All variable names that are added to it are forwarded into the sbuild
environment and also excluded from the bb signatures of the task.

By adding this interface, we provide a mechanism to inject external
mirrors for packages that need to download components at build time.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---

A couple of comments about the patch:

We do not add a test it would require a lot of testing code with little value in it.
Also, we do not yet have examples in ISAR that use go or rust lang, where this would
be relevant. The standard case (no forwards) is already tested in CI.
The forward case will be tested by a lot of ISAR users in China on a daily basis.

Why not simply use templates to address this use case?
In general, templates should be used to pass data into the sbuild environment.
However, these have an influence on the signatures and also the binary content of
the package: It is no longer bit-by-bit identical.
The new infrastructure is especially desigend for mirrors, where changes in a mirror
do not lead to changes in the content of the package (at least should not).
In general, this is ensured by checksum files of the programming languages package manager.

Fetching in the sbuild step is an antipattern:
Yes, true. But while not all dependencies of a component are debianized yet, it is often
not possible to easily avoid internet access during the build.
If we don't provide an infrastructure for that, people start or continue using pre-built
binaries of the components. That is much worse.

Felix


 RECIPE-API-CHANGELOG.md   | 13 +++++++++++++
 meta/classes/dpkg.bbclass | 14 ++++++++++++++
 2 files changed, 27 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index a5f5abee..1b8efa59 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -428,3 +428,16 @@ It was never intended to also deploy those partitions, so that will also not be
 To explicitly deploy the individual partition files (e.g. for swupdate), set `WIC_DEPLOY_PARTITIONS = "1"`.
 
 For compressed wic images `IMAGE_FSTYPES` should simply be extended with a compressed wic format, like "wic.xz".
+
+### Introduce mechanism to forward bitbake variables into sbuild environment
+
+All recipes that inherit from dpkg can use the bitbake variable `SBUILD_PASSTHROUGH_ADDITIONS` to forward
+specific bitbake variables as environment variables into the sbuild environment.
+The motivation behind it is to allow the use of external mirrors for programming languages with builtin
+package managers (like rust and go). By that, the variables are also excluded from the bitbake signatures.
+This helps in areas where default mirrors can either not be reached or provide only little throughput.
+Please note, the forwarded variables do not have to exist. While they are not forwared in case they do not
+exist, empty variables are forwarded.
+
+**Note about reproducability**: the forwarded variables must not have any influence on the generated package.
+This mechanism must also not be used to inject build configurations. For these cases, templates should be used.
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index c92ea7db..fcfbcd7b 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -7,11 +7,24 @@ PACKAGE_ARCH ?= "${DISTRO_ARCH}"
 
 DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
 
+# bitbake variables that should be passed into sbuild env
+# Note: must not have any logical influence on the generated package
+SBUILD_PASSTHROUGH_ADDITIONS ?= ""
+
+def expand_sbuild_pt_additions(d):
+    cmds = ''
+    for var in d.getVar('SBUILD_PASSTHROUGH_ADDITIONS', True).split():
+        varval = d.getVar(var, True)
+        if varval != None:
+            cmds += 'sbuild_export ' + var + ' "' + varval + '"\n'
+    return cmds
+
 do_prepare_build_append() {
     env > ${DPKG_PREBUILD_ENV_FILE}
 }
 
 # Build package from sources using build script
+dpkg_runbuild[vardepsexclude] += "${SBUILD_PASSTHROUGH_ADDITIONS}"
 dpkg_runbuild() {
     E="${@ isar_export_proxies(d)}"
     E="${@ isar_export_ccache(d)}"
@@ -78,6 +91,7 @@ dpkg_runbuild() {
         ftp_proxy FTP_PROXY no_proxy NO_PROXY; do
         sbuild_add_env_filter "$envvar"
     done
+    ${@ expand_sbuild_pt_additions(d)}
 
     echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
 
-- 
2.30.2

Re: [PATCH 1/1] add interface to passthrough env vars to sbuild

[Anton Mikanovich]

20.11.2022 10:37, Felix Moessbauer wrote:
> This patch introduces the bb variable SBUILD_PASSTHROUGH_ADDITIONS.
> All variable names that are added to it are forwarded into the sbuild
> environment and also excluded from the bb signatures of the task.
>
> By adding this interface, we provide a mechanism to inject external
> mirrors for packages that need to download components at build time.
>
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>

Applied to next, thanks.