[PATCH v2 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb

[PATCH v2 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

* BB_ENV_EXTRAWHITE provides a list for variables that are kept in the
environment by bitbake. However, isar init script clears any additional
settings. Thus, add proxy variables to BB_ENV_EXTRAWHITE in
isar-buildenv-internal.

* Bitbake clears environment variables for each task within a recipe.
However, bb.utils.export_proxies function can be used with an
inline-python call to reexport the proxy settings.

* Sudo loses environment variables again, thus call multistrap with sudo
with the -E option to preserve (the already cleaned) environment for the
task's multistrap command.

Note:
Downloads are normally done by the fetcher task, which calls a python
function that in turn uses bb.util.export_proxies. However we have a
non-fetcher task, which needs download capabilities as well.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++-
 meta/recipes-devtools/buildchroot/buildchroot.bb | 4 +++-
 scripts/isar-buildenv-internal                   | 2 +-
 3 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index b679d97..96b5510 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -24,6 +24,8 @@ IMAGE_ROOTFS = "${S}"
 do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
 
 do_rootfs() {
+    E="${@ bb.utils.export_proxies(d)}"
+
     install -d -m 755 ${WORKDIR}/hooks_multistrap
 
     # Copy config file
@@ -46,7 +48,8 @@ do_rootfs() {
     cd ${TOPDIR}
 
     # Create root filesystem
-    sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f "${WORKDIR}/multistrap.conf" || true
+    # We must use sudo -E here to preserve the environment because of proxy settings
+    sudo -E multistrap -a ${DISTRO_ARCH} -d "${S}" -f "${WORKDIR}/multistrap.conf" || true
 
     # Configure root filesystem
     sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL} ${BAUDRATE_TTY} \
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index ccba683..e251a8f 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -26,6 +26,8 @@ WORKDIR = "${TMPDIR}/work/${PF}/${DISTRO}"
 do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 do_build() {
+    E="${@ bb.utils.export_proxies(d)}"
+
     install -d -m 755 ${WORKDIR}/hooks_multistrap
 
     # Copy config files
@@ -48,7 +50,7 @@ do_build() {
     cd ${TOPDIR}
 
     # Create root filesystem
-    sudo multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf" || true
+    sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf" || true
 
     # Install package builder script
     sudo install -m 755 ${THISDIR}/files/build.sh ${BUILDCHROOT_DIR}
diff --git a/scripts/isar-buildenv-internal b/scripts/isar-buildenv-internal
index f14d1ff..94d7eb1 100755
--- a/scripts/isar-buildenv-internal
+++ b/scripts/isar-buildenv-internal
@@ -66,5 +66,5 @@ export PATH
 BBPATH="${BUILDDIR}"
 export BBPATH
 
-BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR"
+BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR http_proxy https_proxy ftp_proxy no_proxy"
 export BB_ENV_EXTRAWHITE
-- 
2.14.1

Re: [PATCH v2 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb

[Andreas Reichel]

This patch was somehow unintentionally posted to the isar mailing list.
It had already been posted a while ago. Sorry for duplication.

Kind regards
Andreas

-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: [PATCH v2 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb

[Henning Schild]

Am Wed, 27 Sep 2017 14:02:18 +0200
schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:

> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> * BB_ENV_EXTRAWHITE provides a list for variables that are kept in the
> environment by bitbake. However, isar init script clears any
> additional settings. Thus, add proxy variables to BB_ENV_EXTRAWHITE in
> isar-buildenv-internal.
> 
> * Bitbake clears environment variables for each task within a recipe.
> However, bb.utils.export_proxies function can be used with an
> inline-python call to reexport the proxy settings.
> 
> * Sudo loses environment variables again, thus call multistrap with
> sudo with the -E option to preserve (the already cleaned) environment
> for the task's multistrap command.
> 
> Note:
> Downloads are normally done by the fetcher task, which calls a python
> function that in turn uses bb.util.export_proxies. However we have a
> non-fetcher task, which needs download capabilities as well.
> 
> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> ---
>  meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++-
>  meta/recipes-devtools/buildchroot/buildchroot.bb | 4 +++-
>  scripts/isar-buildenv-internal                   | 2 +-
>  3 files changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
> b/meta-isar/recipes-core/images/isar-image-base.bb index
> b679d97..96b5510 100644 ---
> a/meta-isar/recipes-core/images/isar-image-base.bb +++
> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -24,6 +24,8 @@
> IMAGE_ROOTFS = "${S}" do_rootfs[stamp-extra-info] =
> "${MACHINE}-${DISTRO}" 
>  do_rootfs() {
> +    E="${@ bb.utils.export_proxies(d)}"
> +
>      install -d -m 755 ${WORKDIR}/hooks_multistrap
>  
>      # Copy config file
> @@ -46,7 +48,8 @@ do_rootfs() {
>      cd ${TOPDIR}
>  
>      # Create root filesystem
> -    sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f
> "${WORKDIR}/multistrap.conf" || true
> +    # We must use sudo -E here to preserve the environment because
> of proxy settings
> +    sudo -E multistrap -a ${DISTRO_ARCH} -d "${S}" -f
> "${WORKDIR}/multistrap.conf" || true 
>      # Configure root filesystem
>      sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL}
> ${BAUDRATE_TTY} \ diff --git
> a/meta/recipes-devtools/buildchroot/buildchroot.bb
> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
> ccba683..e251a8f 100644 ---
> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -26,6 +26,8 @@
> WORKDIR = "${TMPDIR}/work/${PF}/${DISTRO}" do_build[stamp-extra-info]
> = "${DISTRO}-${DISTRO_ARCH}" do_build() {
> +    E="${@ bb.utils.export_proxies(d)}"
> +
>      install -d -m 755 ${WORKDIR}/hooks_multistrap
>  
>      # Copy config files
> @@ -48,7 +50,7 @@ do_build() {
>      cd ${TOPDIR}
>  
>      # Create root filesystem
> -    sudo multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f
> "${WORKDIR}/multistrap.conf" || true
> +    sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f
> "${WORKDIR}/multistrap.conf" || true 
>      # Install package builder script
>      sudo install -m 755 ${THISDIR}/files/build.sh ${BUILDCHROOT_DIR}

In the first round i questioned the need for the "-E" in build.sh and
configure.sh in buildchroot.bb. But now i think they both need the -E
as well. One contains "apt-get install" and the other "apt-get update",
one could question the update but the install might fetch and install
more packages.
I already raised that point for v2 and there was no answer, maybe that
is why you patch did not make it to next yet.

Henning

> diff --git a/scripts/isar-buildenv-internal
> b/scripts/isar-buildenv-internal index f14d1ff..94d7eb1 100755
> --- a/scripts/isar-buildenv-internal
> +++ b/scripts/isar-buildenv-internal
> @@ -66,5 +66,5 @@ export PATH
>  BBPATH="${BUILDDIR}"
>  export BBPATH
>  
> -BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR"
> +BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR http_proxy https_proxy ftp_proxy
> no_proxy" export BB_ENV_EXTRAWHITE

Re: [PATCH v2 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb

[Henning Schild]

Am Wed, 27 Sep 2017 14:29:06 +0200
schrieb "[ext] Andreas Reichel" <andreas.reichel.ext@siemens.com>:

> This patch was somehow unintentionally posted to the isar mailing
> list. It had already been posted a while ago. Sorry for duplication.

No worries. It is an important one, could you please look into the
comments and send a v3 for integration into Isar?

Henning

> Kind regards
> Andreas
> 

Re: [PATCH v2 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb

[Henning Schild]

Alex,

could you please merge this patch or comment on it. I had some comments
and after that the process stopped. However i do not have any
objections to this version.

Henning

Am Wed, 27 Sep 2017 14:02:18 +0200
schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:

> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> * BB_ENV_EXTRAWHITE provides a list for variables that are kept in the
> environment by bitbake. However, isar init script clears any
> additional settings. Thus, add proxy variables to BB_ENV_EXTRAWHITE in
> isar-buildenv-internal.
> 
> * Bitbake clears environment variables for each task within a recipe.
> However, bb.utils.export_proxies function can be used with an
> inline-python call to reexport the proxy settings.
> 
> * Sudo loses environment variables again, thus call multistrap with
> sudo with the -E option to preserve (the already cleaned) environment
> for the task's multistrap command.
> 
> Note:
> Downloads are normally done by the fetcher task, which calls a python
> function that in turn uses bb.util.export_proxies. However we have a
> non-fetcher task, which needs download capabilities as well.
> 
> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> ---
>  meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++-
>  meta/recipes-devtools/buildchroot/buildchroot.bb | 4 +++-
>  scripts/isar-buildenv-internal                   | 2 +-
>  3 files changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
> b/meta-isar/recipes-core/images/isar-image-base.bb index
> b679d97..96b5510 100644 ---
> a/meta-isar/recipes-core/images/isar-image-base.bb +++
> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -24,6 +24,8 @@
> IMAGE_ROOTFS = "${S}" do_rootfs[stamp-extra-info] =
> "${MACHINE}-${DISTRO}" 
>  do_rootfs() {
> +    E="${@ bb.utils.export_proxies(d)}"
> +
>      install -d -m 755 ${WORKDIR}/hooks_multistrap
>  
>      # Copy config file
> @@ -46,7 +48,8 @@ do_rootfs() {
>      cd ${TOPDIR}
>  
>      # Create root filesystem
> -    sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f
> "${WORKDIR}/multistrap.conf" || true
> +    # We must use sudo -E here to preserve the environment because
> of proxy settings
> +    sudo -E multistrap -a ${DISTRO_ARCH} -d "${S}" -f
> "${WORKDIR}/multistrap.conf" || true 
>      # Configure root filesystem
>      sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL}
> ${BAUDRATE_TTY} \ diff --git
> a/meta/recipes-devtools/buildchroot/buildchroot.bb
> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
> ccba683..e251a8f 100644 ---
> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -26,6 +26,8 @@
> WORKDIR = "${TMPDIR}/work/${PF}/${DISTRO}" do_build[stamp-extra-info]
> = "${DISTRO}-${DISTRO_ARCH}" do_build() {
> +    E="${@ bb.utils.export_proxies(d)}"
> +
>      install -d -m 755 ${WORKDIR}/hooks_multistrap
>  
>      # Copy config files
> @@ -48,7 +50,7 @@ do_build() {
>      cd ${TOPDIR}
>  
>      # Create root filesystem
> -    sudo multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f
> "${WORKDIR}/multistrap.conf" || true
> +    sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f
> "${WORKDIR}/multistrap.conf" || true 
>      # Install package builder script
>      sudo install -m 755 ${THISDIR}/files/build.sh ${BUILDCHROOT_DIR}
> diff --git a/scripts/isar-buildenv-internal
> b/scripts/isar-buildenv-internal index f14d1ff..94d7eb1 100755
> --- a/scripts/isar-buildenv-internal
> +++ b/scripts/isar-buildenv-internal
> @@ -66,5 +66,5 @@ export PATH
>  BBPATH="${BUILDDIR}"
>  export BBPATH
>  
> -BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR"
> +BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR http_proxy https_proxy ftp_proxy
> no_proxy" export BB_ENV_EXTRAWHITE

Re: [PATCH v2 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb

[Alexander Smirnov]

On 09/27/2017 03:02 PM, Andreas J. Reichel wrote:
> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> * BB_ENV_EXTRAWHITE provides a list for variables that are kept in the
> environment by bitbake. However, isar init script clears any additional
> settings. Thus, add proxy variables to BB_ENV_EXTRAWHITE in
> isar-buildenv-internal.
> 
> * Bitbake clears environment variables for each task within a recipe.
> However, bb.utils.export_proxies function can be used with an
> inline-python call to reexport the proxy settings.
> 
> * Sudo loses environment variables again, thus call multistrap with sudo
> with the -E option to preserve (the already cleaned) environment for the
> task's multistrap command.
> 
> Note:
> Downloads are normally done by the fetcher task, which calls a python
> function that in turn uses bb.util.export_proxies. However we have a
> non-fetcher task, which needs download capabilities as well.
>

Applied to next, thanks!

Alex

[PATCH v2 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

* BB_ENV_EXTRAWHITE provides a list for variables that are kept in the
environment by bitbake. However, isar init script clears any additional
settings. Thus, add proxy variables to BB_ENV_EXTRAWHITE in
isar-buildenv-internal.

* Bitbake clears environment variables for each task within a recipe.
However, bb.utils.export_proxies function can be used with an
inline-python call to reexport the proxy settings.

* Sudo loses environment variables again, thus call multistrap with sudo
with the -E option to preserve (the already cleaned) environment for the
task's multistrap command.

Note:
Downloads are normally done by the fetcher task, which calls a python
function that in turn uses bb.util.export_proxies. However we have a
non-fetcher task, which needs download capabilities as well.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++-
 meta/recipes-devtools/buildchroot/buildchroot.bb | 4 +++-
 scripts/isar-buildenv-internal                   | 2 +-
 3 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index b679d97..96b5510 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -24,6 +24,8 @@ IMAGE_ROOTFS = "${S}"
 do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
 
 do_rootfs() {
+    E="${@ bb.utils.export_proxies(d)}"
+
     install -d -m 755 ${WORKDIR}/hooks_multistrap
 
     # Copy config file
@@ -46,7 +48,8 @@ do_rootfs() {
     cd ${TOPDIR}
 
     # Create root filesystem
-    sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f "${WORKDIR}/multistrap.conf" || true
+    # We must use sudo -E here to preserve the environment because of proxy settings
+    sudo -E multistrap -a ${DISTRO_ARCH} -d "${S}" -f "${WORKDIR}/multistrap.conf" || true
 
     # Configure root filesystem
     sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL} ${BAUDRATE_TTY} \
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index ccba683..e251a8f 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -26,6 +26,8 @@ WORKDIR = "${TMPDIR}/work/${PF}/${DISTRO}"
 do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 do_build() {
+    E="${@ bb.utils.export_proxies(d)}"
+
     install -d -m 755 ${WORKDIR}/hooks_multistrap
 
     # Copy config files
@@ -48,7 +50,7 @@ do_build() {
     cd ${TOPDIR}
 
     # Create root filesystem
-    sudo multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf" || true
+    sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf" || true
 
     # Install package builder script
     sudo install -m 755 ${THISDIR}/files/build.sh ${BUILDCHROOT_DIR}
diff --git a/scripts/isar-buildenv-internal b/scripts/isar-buildenv-internal
index f14d1ff..94d7eb1 100755
--- a/scripts/isar-buildenv-internal
+++ b/scripts/isar-buildenv-internal
@@ -66,5 +66,5 @@ export PATH
 BBPATH="${BUILDDIR}"
 export BBPATH
 
-BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR"
+BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR http_proxy https_proxy ftp_proxy no_proxy"
 export BB_ENV_EXTRAWHITE
-- 
2.14.1