From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7167999216506109952 X-Received: by 2002:a7b:cbc6:0:b0:3c6:b650:34dd with SMTP id n6-20020a7bcbc6000000b003c6b65034ddmr2613455wmi.45.1668929871846; Sat, 19 Nov 2022 23:37:51 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:5247:0:b0:236:8fa4:71d1 with SMTP id k7-20020a5d5247000000b002368fa471d1ls11026800wrc.1.-pod-prod-gmail; Sat, 19 Nov 2022 23:37:50 -0800 (PST) X-Google-Smtp-Source: AA0mqf41jICUvWOTLJVI003zEYY54qbQ3PagIEpqtJoyqDrrVRLizLFr3bMdML4GDsjqaKNfNEq6 X-Received: by 2002:a5d:6811:0:b0:236:8322:4bf1 with SMTP id w17-20020a5d6811000000b0023683224bf1mr7867922wru.119.1668929870491; Sat, 19 Nov 2022 23:37:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668929870; cv=none; d=google.com; s=arc-20160816; b=onWxia/5XHSy6MpqKFQTSAGsZrSP76IDstWzJq253NFVPRg9OVBAv+4M1SijxgHU9m 2868Z1BK7AsTwkjwsV/iyzxkl6bGgnoKL4Q1/fHCD2OEkZVGJtffb2cwg6n1MtFRetGD feve/VmigNiAB1frjmbLcE7ysGkb7B8BcHb1zdaXg6UFOp/5NgLALa2gZ4sJCUaOcMZy NvhOarVldG/p1Hx7GmhDVSWM08czMJ9PUhWz2jge8twzopvPLcE+L5XDiip0cLId66Da ZCNMn7Qxwrhz1dHuJ6JILLq1NMzHs5n38Xbx4FoF9ycc1Mxe2CRs6DCDnJUvr7qQ09Fi Bsxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=mBVA2E9FAuX1+mWE3hvK7uB2UIRv/SK8QLKATlzYjVQ=; b=qF8ojRWaydpFEKHdwBsm0XL0ebGGV3m+JCMWsBhEgADP4KK7WIZnXrsqWi/CNMFbKQ pyjpVVjmMRfPxI0uCx2JSCV1z9r+5avbcJvKl6K8KCDmiFo4+vumZbd3IYcOxyW/YfuK w8b7TQygCqZGZ1vEDJwKB16Db/quEHyWVtNJChDyRn5u0x+8wu9ktEyusyvAu90c773K hlW7sCtYeVO11WtcfTEbBkrSCwW7qf/lthXPHM9jP4LGmx8fKMuaZcKeP3JRM6oFnr2K cvVgOKHeaoTpp8ymFilSTw9WmflBT5PUyuDvx3a4jlnEJZxAq3XRfbjsiNGPqGv51abN oHrw== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=dM94qFd6; spf=pass (google.com: domain of fm-72506-20221120073749e8d3b43a3dd7fa8802-ewaqno@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-72506-20221120073749e8d3b43a3dd7fa8802-EWAqnO@rts-flowmailer.siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from mta-64-228.siemens.flowmailer.net (mta-64-228.siemens.flowmailer.net. [185.136.64.228]) by gmr-mx.google.com with ESMTPS id f187-20020a1c38c4000000b003cf567afb1fsi480679wma.3.2022.11.19.23.37.50 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 19 Nov 2022 23:37:50 -0800 (PST) Received-SPF: pass (google.com: domain of fm-72506-20221120073749e8d3b43a3dd7fa8802-ewaqno@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) client-ip=185.136.64.228; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=dM94qFd6; spf=pass (google.com: domain of fm-72506-20221120073749e8d3b43a3dd7fa8802-ewaqno@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-72506-20221120073749e8d3b43a3dd7fa8802-EWAqnO@rts-flowmailer.siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: by mta-64-228.siemens.flowmailer.net with ESMTPSA id 20221120073749e8d3b43a3dd7fa8802 for ; Sun, 20 Nov 2022 08:37:49 +0100 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=felix.moessbauer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc; bh=mBVA2E9FAuX1+mWE3hvK7uB2UIRv/SK8QLKATlzYjVQ=; b=dM94qFd6aydVjxyZZP8bJm4uWPQ4oOX186CGWptVVO6rVHAIGj9whNy3NJ5ocZZSJnNftX BpWqY+Jnl4uUGxYc6mjyHJcteR7gtqOgw68tLyuqnuxvoc75OzCo7p9213EjkVvUDc+rR0Xs EkXgbizDdkAKmm3ZCDZBoC1oqXWUE=; From: Felix Moessbauer To: isar-users@googlegroups.com Cc: jan.kiszka@siemens.com, daniel.bovensiepen@siemens.com, Felix Moessbauer Subject: [PATCH 1/1] add interface to passthrough env vars to sbuild Date: Sun, 20 Nov 2022 08:37:24 +0100 Message-Id: <20221120073724.623860-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-72506:519-21489:flowmailer X-TUID: Y/onJgwmRHEi This patch introduces the bb variable SBUILD_PASSTHROUGH_ADDITIONS. All variable names that are added to it are forwarded into the sbuild environment and also excluded from the bb signatures of the task. By adding this interface, we provide a mechanism to inject external mirrors for packages that need to download components at build time. Signed-off-by: Felix Moessbauer --- A couple of comments about the patch: We do not add a test it would require a lot of testing code with little value in it. Also, we do not yet have examples in ISAR that use go or rust lang, where this would be relevant. The standard case (no forwards) is already tested in CI. The forward case will be tested by a lot of ISAR users in China on a daily basis. Why not simply use templates to address this use case? In general, templates should be used to pass data into the sbuild environment. However, these have an influence on the signatures and also the binary content of the package: It is no longer bit-by-bit identical. The new infrastructure is especially desigend for mirrors, where changes in a mirror do not lead to changes in the content of the package (at least should not). In general, this is ensured by checksum files of the programming languages package manager. Fetching in the sbuild step is an antipattern: Yes, true. But while not all dependencies of a component are debianized yet, it is often not possible to easily avoid internet access during the build. If we don't provide an infrastructure for that, people start or continue using pre-built binaries of the components. That is much worse. Felix RECIPE-API-CHANGELOG.md | 13 +++++++++++++ meta/classes/dpkg.bbclass | 14 ++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index a5f5abee..1b8efa59 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -428,3 +428,16 @@ It was never intended to also deploy those partitions, so that will also not be To explicitly deploy the individual partition files (e.g. for swupdate), set `WIC_DEPLOY_PARTITIONS = "1"`. For compressed wic images `IMAGE_FSTYPES` should simply be extended with a compressed wic format, like "wic.xz". + +### Introduce mechanism to forward bitbake variables into sbuild environment + +All recipes that inherit from dpkg can use the bitbake variable `SBUILD_PASSTHROUGH_ADDITIONS` to forward +specific bitbake variables as environment variables into the sbuild environment. +The motivation behind it is to allow the use of external mirrors for programming languages with builtin +package managers (like rust and go). By that, the variables are also excluded from the bitbake signatures. +This helps in areas where default mirrors can either not be reached or provide only little throughput. +Please note, the forwarded variables do not have to exist. While they are not forwared in case they do not +exist, empty variables are forwarded. + +**Note about reproducability**: the forwarded variables must not have any influence on the generated package. +This mechanism must also not be used to inject build configurations. For these cases, templates should be used. diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index c92ea7db..fcfbcd7b 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -7,11 +7,24 @@ PACKAGE_ARCH ?= "${DISTRO_ARCH}" DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env" +# bitbake variables that should be passed into sbuild env +# Note: must not have any logical influence on the generated package +SBUILD_PASSTHROUGH_ADDITIONS ?= "" + +def expand_sbuild_pt_additions(d): + cmds = '' + for var in d.getVar('SBUILD_PASSTHROUGH_ADDITIONS', True).split(): + varval = d.getVar(var, True) + if varval != None: + cmds += 'sbuild_export ' + var + ' "' + varval + '"\n' + return cmds + do_prepare_build_append() { env > ${DPKG_PREBUILD_ENV_FILE} } # Build package from sources using build script +dpkg_runbuild[vardepsexclude] += "${SBUILD_PASSTHROUGH_ADDITIONS}" dpkg_runbuild() { E="${@ isar_export_proxies(d)}" E="${@ isar_export_ccache(d)}" @@ -78,6 +91,7 @@ dpkg_runbuild() { ftp_proxy FTP_PROXY no_proxy NO_PROXY; do sbuild_add_env_filter "$envvar" done + ${@ expand_sbuild_pt_additions(d)} echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG} -- 2.30.2