[PATCH v2 0/1] Make do_rootfs work with proxy settings

[PATCH v2 0/1] Make do_rootfs work with proxy settings

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

Regarding issue #19 on github: Consider testing with http_proxy

Diff to 1st version:
* Remove comments before python inline code.
* Remove -E option from sudo in dev-tools/buildchroot at configure
  step, since a configure step should not download things

If isar is built behind a proxy, multistrap fails to communicate
with the repository.

Usually, the user's environment has *_proxy variables set to configure
tools. However, these values are not passed correctly. With this patch,
multistrap is able to make use of the users proxy config.

The fix is strongly oriented on how the internal fetcher deals with
this problem.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>

Andreas Reichel (1):
  Add proxy support to isar-image-*.bb and buildchroot.bb

 meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++-
 meta/recipes-devtools/buildchroot/buildchroot.bb | 4 +++-
 scripts/isar-buildenv-internal                   | 2 +-
 3 files changed, 8 insertions(+), 3 deletions(-)

-- 
2.14.1

[PATCH v2 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

* BB_ENV_EXTRAWHITE provides a list for variables that are kept in the
environment by bitbake. However, isar init script clears any additional
settings. Thus, add proxy variables to BB_ENV_EXTRAWHITE in
isar-buildenv-internal.

* Bitbake clears environment variables for each task within a recipe.
However, bb.utils.export_proxies function can be used with an
inline-python call to reexport the proxy settings.

* Sudo loses environment variables again, thus call multistrap with sudo
with the -E option to preserve (the already cleaned) environment for the
task's multistrap command.

Note:
Downloads are normally done by the fetcher task, which calls a python
function that in turn uses bb.util.export_proxies. However we have a
non-fetcher task, which needs download capabilities as well.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++-
 meta/recipes-devtools/buildchroot/buildchroot.bb | 4 +++-
 scripts/isar-buildenv-internal                   | 2 +-
 3 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index b679d97..96b5510 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -24,6 +24,8 @@ IMAGE_ROOTFS = "${S}"
 do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
 
 do_rootfs() {
+    E="${@ bb.utils.export_proxies(d)}"
+
     install -d -m 755 ${WORKDIR}/hooks_multistrap
 
     # Copy config file
@@ -46,7 +48,8 @@ do_rootfs() {
     cd ${TOPDIR}
 
     # Create root filesystem
-    sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f "${WORKDIR}/multistrap.conf" || true
+    # We must use sudo -E here to preserve the environment because of proxy settings
+    sudo -E multistrap -a ${DISTRO_ARCH} -d "${S}" -f "${WORKDIR}/multistrap.conf" || true
 
     # Configure root filesystem
     sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL} ${BAUDRATE_TTY} \
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index ccba683..e251a8f 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -26,6 +26,8 @@ WORKDIR = "${TMPDIR}/work/${PF}/${DISTRO}"
 do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 do_build() {
+    E="${@ bb.utils.export_proxies(d)}"
+
     install -d -m 755 ${WORKDIR}/hooks_multistrap
 
     # Copy config files
@@ -48,7 +50,7 @@ do_build() {
     cd ${TOPDIR}
 
     # Create root filesystem
-    sudo multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf" || true
+    sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf" || true
 
     # Install package builder script
     sudo install -m 755 ${THISDIR}/files/build.sh ${BUILDCHROOT_DIR}
diff --git a/scripts/isar-buildenv-internal b/scripts/isar-buildenv-internal
index f14d1ff..94d7eb1 100755
--- a/scripts/isar-buildenv-internal
+++ b/scripts/isar-buildenv-internal
@@ -66,5 +66,5 @@ export PATH
 BBPATH="${BUILDDIR}"
 export BBPATH
 
-BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR"
+BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR http_proxy https_proxy ftp_proxy no_proxy"
 export BB_ENV_EXTRAWHITE
-- 
2.14.1

Re: [PATCH v2 0/1] Make do_rootfs work with proxy settings

[Henning Schild]

Am Mon, 11 Sep 2017 12:55:57 +0200
schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:

> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> Regarding issue #19 on github: Consider testing with http_proxy
> 
> Diff to 1st version:
> * Remove comments before python inline code.
> * Remove -E option from sudo in dev-tools/buildchroot at configure
>   step, since a configure step should not download things

Sorry, my comment for the buildchroot was wrong. We do have an
"apt-get update" in the configure.sh for buildchroot, and for that to
work we need the proxies configured in there.
In the image the configure.sh should not get access to the internet, in
the buildchroot it is fine and currently required. So please add the -E
back in.

Henning

> If isar is built behind a proxy, multistrap fails to communicate
> with the repository.
> 
> Usually, the user's environment has *_proxy variables set to configure
> tools. However, these values are not passed correctly. With this
> patch, multistrap is able to make use of the users proxy config.
> 
> The fix is strongly oriented on how the internal fetcher deals with
> this problem.
> 
> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> Andreas Reichel (1):
>   Add proxy support to isar-image-*.bb and buildchroot.bb
> 
>  meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++-
>  meta/recipes-devtools/buildchroot/buildchroot.bb | 4 +++-
>  scripts/isar-buildenv-internal                   | 2 +-
>  3 files changed, 8 insertions(+), 3 deletions(-)
>