[PATCH v2 0/1] Make do_rootfs work with proxy settings

[PATCH v2 0/1] Make do_rootfs work with proxy settings

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

Regarding issue #19 on github: Consider testing with http_proxy

Diff to 1st version:
* Remove comments before python inline code.
* Remove -E option from sudo in dev-tools/buildchroot at configure
  step, since a configure step should not download things

If isar is built behind a proxy, multistrap fails to communicate
with the repository.

Usually, the user's environment has *_proxy variables set to configure
tools. However, these values are not passed correctly. With this patch,
multistrap is able to make use of the users proxy config.

The fix is strongly oriented on how the internal fetcher deals with
this problem.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>

Andreas Reichel (1):
  Add proxy support to isar-image-*.bb and buildchroot.bb

 meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++-
 meta/recipes-devtools/buildchroot/buildchroot.bb | 4 +++-
 scripts/isar-buildenv-internal                   | 2 +-
 3 files changed, 8 insertions(+), 3 deletions(-)

-- 
2.14.1

[PATCH v2 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

* BB_ENV_EXTRAWHITE provides a list for variables that are kept in the
environment by bitbake. However, isar init script clears any additional
settings. Thus, add proxy variables to BB_ENV_EXTRAWHITE in
isar-buildenv-internal.

* Bitbake clears environment variables for each task within a recipe.
However, bb.utils.export_proxies function can be used with an
inline-python call to reexport the proxy settings.

* Sudo loses environment variables again, thus call multistrap with sudo
with the -E option to preserve (the already cleaned) environment for the
task's multistrap command.

Note:
Downloads are normally done by the fetcher task, which calls a python
function that in turn uses bb.util.export_proxies. However we have a
non-fetcher task, which needs download capabilities as well.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++-
 meta/recipes-devtools/buildchroot/buildchroot.bb | 4 +++-
 scripts/isar-buildenv-internal                   | 2 +-
 3 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index b679d97..96b5510 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -24,6 +24,8 @@ IMAGE_ROOTFS = "${S}"
 do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
 
 do_rootfs() {
+    E="${@ bb.utils.export_proxies(d)}"
+
     install -d -m 755 ${WORKDIR}/hooks_multistrap
 
     # Copy config file
@@ -46,7 +48,8 @@ do_rootfs() {
     cd ${TOPDIR}
 
     # Create root filesystem
-    sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f "${WORKDIR}/multistrap.conf" || true
+    # We must use sudo -E here to preserve the environment because of proxy settings
+    sudo -E multistrap -a ${DISTRO_ARCH} -d "${S}" -f "${WORKDIR}/multistrap.conf" || true
 
     # Configure root filesystem
     sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL} ${BAUDRATE_TTY} \
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index ccba683..e251a8f 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -26,6 +26,8 @@ WORKDIR = "${TMPDIR}/work/${PF}/${DISTRO}"
 do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 do_build() {
+    E="${@ bb.utils.export_proxies(d)}"
+
     install -d -m 755 ${WORKDIR}/hooks_multistrap
 
     # Copy config files
@@ -48,7 +50,7 @@ do_build() {
     cd ${TOPDIR}
 
     # Create root filesystem
-    sudo multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf" || true
+    sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf" || true
 
     # Install package builder script
     sudo install -m 755 ${THISDIR}/files/build.sh ${BUILDCHROOT_DIR}
diff --git a/scripts/isar-buildenv-internal b/scripts/isar-buildenv-internal
index f14d1ff..94d7eb1 100755
--- a/scripts/isar-buildenv-internal
+++ b/scripts/isar-buildenv-internal
@@ -66,5 +66,5 @@ export PATH
 BBPATH="${BUILDDIR}"
 export BBPATH
 
-BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR"
+BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR http_proxy https_proxy ftp_proxy no_proxy"
 export BB_ENV_EXTRAWHITE
-- 
2.14.1

Re: [PATCH v2 0/1] Make do_rootfs work with proxy settings

[Henning Schild]

Am Mon, 11 Sep 2017 12:55:57 +0200
schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:

> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> Regarding issue #19 on github: Consider testing with http_proxy
> 
> Diff to 1st version:
> * Remove comments before python inline code.
> * Remove -E option from sudo in dev-tools/buildchroot at configure
>   step, since a configure step should not download things

Sorry, my comment for the buildchroot was wrong. We do have an
"apt-get update" in the configure.sh for buildchroot, and for that to
work we need the proxies configured in there.
In the image the configure.sh should not get access to the internet, in
the buildchroot it is fine and currently required. So please add the -E
back in.

Henning

> If isar is built behind a proxy, multistrap fails to communicate
> with the repository.
> 
> Usually, the user's environment has *_proxy variables set to configure
> tools. However, these values are not passed correctly. With this
> patch, multistrap is able to make use of the users proxy config.
> 
> The fix is strongly oriented on how the internal fetcher deals with
> this problem.
> 
> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> Andreas Reichel (1):
>   Add proxy support to isar-image-*.bb and buildchroot.bb
> 
>  meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++-
>  meta/recipes-devtools/buildchroot/buildchroot.bb | 4 +++-
>  scripts/isar-buildenv-internal                   | 2 +-
>  3 files changed, 8 insertions(+), 3 deletions(-)
> 

[PATCH v2 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

* BB_ENV_EXTRAWHITE provides a list for variables that are kept in the
environment by bitbake. However, isar init script clears any additional
settings. Thus, add proxy variables to BB_ENV_EXTRAWHITE in
isar-buildenv-internal.

* Bitbake clears environment variables for each task within a recipe.
However, bb.utils.export_proxies function can be used with an
inline-python call to reexport the proxy settings.

* Sudo loses environment variables again, thus call multistrap with sudo
with the -E option to preserve (the already cleaned) environment for the
task's multistrap command.

Note:
Downloads are normally done by the fetcher task, which calls a python
function that in turn uses bb.util.export_proxies. However we have a
non-fetcher task, which needs download capabilities as well.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++-
 meta/recipes-devtools/buildchroot/buildchroot.bb | 4 +++-
 scripts/isar-buildenv-internal                   | 2 +-
 3 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index b679d97..96b5510 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -24,6 +24,8 @@ IMAGE_ROOTFS = "${S}"
 do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
 
 do_rootfs() {
+    E="${@ bb.utils.export_proxies(d)}"
+
     install -d -m 755 ${WORKDIR}/hooks_multistrap
 
     # Copy config file
@@ -46,7 +48,8 @@ do_rootfs() {
     cd ${TOPDIR}
 
     # Create root filesystem
-    sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f "${WORKDIR}/multistrap.conf" || true
+    # We must use sudo -E here to preserve the environment because of proxy settings
+    sudo -E multistrap -a ${DISTRO_ARCH} -d "${S}" -f "${WORKDIR}/multistrap.conf" || true
 
     # Configure root filesystem
     sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL} ${BAUDRATE_TTY} \
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index ccba683..e251a8f 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -26,6 +26,8 @@ WORKDIR = "${TMPDIR}/work/${PF}/${DISTRO}"
 do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 do_build() {
+    E="${@ bb.utils.export_proxies(d)}"
+
     install -d -m 755 ${WORKDIR}/hooks_multistrap
 
     # Copy config files
@@ -48,7 +50,7 @@ do_build() {
     cd ${TOPDIR}
 
     # Create root filesystem
-    sudo multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf" || true
+    sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf" || true
 
     # Install package builder script
     sudo install -m 755 ${THISDIR}/files/build.sh ${BUILDCHROOT_DIR}
diff --git a/scripts/isar-buildenv-internal b/scripts/isar-buildenv-internal
index f14d1ff..94d7eb1 100755
--- a/scripts/isar-buildenv-internal
+++ b/scripts/isar-buildenv-internal
@@ -66,5 +66,5 @@ export PATH
 BBPATH="${BUILDDIR}"
 export BBPATH
 
-BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR"
+BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR http_proxy https_proxy ftp_proxy no_proxy"
 export BB_ENV_EXTRAWHITE
-- 
2.14.1

Re: [PATCH v2 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb

[Andreas Reichel]

This patch was somehow unintentionally posted to the isar mailing list.
It had already been posted a while ago. Sorry for duplication.

Kind regards
Andreas

-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: [PATCH v2 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb

[Henning Schild]

Am Wed, 27 Sep 2017 14:29:06 +0200
schrieb "[ext] Andreas Reichel" <andreas.reichel.ext@siemens.com>:

> This patch was somehow unintentionally posted to the isar mailing
> list. It had already been posted a while ago. Sorry for duplication.

No worries. It is an important one, could you please look into the
comments and send a v3 for integration into Isar?

Henning

> Kind regards
> Andreas
> 

Re: [PATCH v2 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb

[Henning Schild]

Am Wed, 27 Sep 2017 14:02:18 +0200
schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:

> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> * BB_ENV_EXTRAWHITE provides a list for variables that are kept in the
> environment by bitbake. However, isar init script clears any
> additional settings. Thus, add proxy variables to BB_ENV_EXTRAWHITE in
> isar-buildenv-internal.
> 
> * Bitbake clears environment variables for each task within a recipe.
> However, bb.utils.export_proxies function can be used with an
> inline-python call to reexport the proxy settings.
> 
> * Sudo loses environment variables again, thus call multistrap with
> sudo with the -E option to preserve (the already cleaned) environment
> for the task's multistrap command.
> 
> Note:
> Downloads are normally done by the fetcher task, which calls a python
> function that in turn uses bb.util.export_proxies. However we have a
> non-fetcher task, which needs download capabilities as well.
> 
> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> ---
>  meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++-
>  meta/recipes-devtools/buildchroot/buildchroot.bb | 4 +++-
>  scripts/isar-buildenv-internal                   | 2 +-
>  3 files changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
> b/meta-isar/recipes-core/images/isar-image-base.bb index
> b679d97..96b5510 100644 ---
> a/meta-isar/recipes-core/images/isar-image-base.bb +++
> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -24,6 +24,8 @@
> IMAGE_ROOTFS = "${S}" do_rootfs[stamp-extra-info] =
> "${MACHINE}-${DISTRO}" 
>  do_rootfs() {
> +    E="${@ bb.utils.export_proxies(d)}"
> +
>      install -d -m 755 ${WORKDIR}/hooks_multistrap
>  
>      # Copy config file
> @@ -46,7 +48,8 @@ do_rootfs() {
>      cd ${TOPDIR}
>  
>      # Create root filesystem
> -    sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f
> "${WORKDIR}/multistrap.conf" || true
> +    # We must use sudo -E here to preserve the environment because
> of proxy settings
> +    sudo -E multistrap -a ${DISTRO_ARCH} -d "${S}" -f
> "${WORKDIR}/multistrap.conf" || true 
>      # Configure root filesystem
>      sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL}
> ${BAUDRATE_TTY} \ diff --git
> a/meta/recipes-devtools/buildchroot/buildchroot.bb
> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
> ccba683..e251a8f 100644 ---
> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -26,6 +26,8 @@
> WORKDIR = "${TMPDIR}/work/${PF}/${DISTRO}" do_build[stamp-extra-info]
> = "${DISTRO}-${DISTRO_ARCH}" do_build() {
> +    E="${@ bb.utils.export_proxies(d)}"
> +
>      install -d -m 755 ${WORKDIR}/hooks_multistrap
>  
>      # Copy config files
> @@ -48,7 +50,7 @@ do_build() {
>      cd ${TOPDIR}
>  
>      # Create root filesystem
> -    sudo multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f
> "${WORKDIR}/multistrap.conf" || true
> +    sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f
> "${WORKDIR}/multistrap.conf" || true 
>      # Install package builder script
>      sudo install -m 755 ${THISDIR}/files/build.sh ${BUILDCHROOT_DIR}

In the first round i questioned the need for the "-E" in build.sh and
configure.sh in buildchroot.bb. But now i think they both need the -E
as well. One contains "apt-get install" and the other "apt-get update",
one could question the update but the install might fetch and install
more packages.
I already raised that point for v2 and there was no answer, maybe that
is why you patch did not make it to next yet.

Henning

> diff --git a/scripts/isar-buildenv-internal
> b/scripts/isar-buildenv-internal index f14d1ff..94d7eb1 100755
> --- a/scripts/isar-buildenv-internal
> +++ b/scripts/isar-buildenv-internal
> @@ -66,5 +66,5 @@ export PATH
>  BBPATH="${BUILDDIR}"
>  export BBPATH
>  
> -BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR"
> +BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR http_proxy https_proxy ftp_proxy
> no_proxy" export BB_ENV_EXTRAWHITE

Re: [PATCH v2 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb

[Henning Schild]

Alex,

could you please merge this patch or comment on it. I had some comments
and after that the process stopped. However i do not have any
objections to this version.

Henning

Am Wed, 27 Sep 2017 14:02:18 +0200
schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:

> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> * BB_ENV_EXTRAWHITE provides a list for variables that are kept in the
> environment by bitbake. However, isar init script clears any
> additional settings. Thus, add proxy variables to BB_ENV_EXTRAWHITE in
> isar-buildenv-internal.
> 
> * Bitbake clears environment variables for each task within a recipe.
> However, bb.utils.export_proxies function can be used with an
> inline-python call to reexport the proxy settings.
> 
> * Sudo loses environment variables again, thus call multistrap with
> sudo with the -E option to preserve (the already cleaned) environment
> for the task's multistrap command.
> 
> Note:
> Downloads are normally done by the fetcher task, which calls a python
> function that in turn uses bb.util.export_proxies. However we have a
> non-fetcher task, which needs download capabilities as well.
> 
> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> ---
>  meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++-
>  meta/recipes-devtools/buildchroot/buildchroot.bb | 4 +++-
>  scripts/isar-buildenv-internal                   | 2 +-
>  3 files changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
> b/meta-isar/recipes-core/images/isar-image-base.bb index
> b679d97..96b5510 100644 ---
> a/meta-isar/recipes-core/images/isar-image-base.bb +++
> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -24,6 +24,8 @@
> IMAGE_ROOTFS = "${S}" do_rootfs[stamp-extra-info] =
> "${MACHINE}-${DISTRO}" 
>  do_rootfs() {
> +    E="${@ bb.utils.export_proxies(d)}"
> +
>      install -d -m 755 ${WORKDIR}/hooks_multistrap
>  
>      # Copy config file
> @@ -46,7 +48,8 @@ do_rootfs() {
>      cd ${TOPDIR}
>  
>      # Create root filesystem
> -    sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f
> "${WORKDIR}/multistrap.conf" || true
> +    # We must use sudo -E here to preserve the environment because
> of proxy settings
> +    sudo -E multistrap -a ${DISTRO_ARCH} -d "${S}" -f
> "${WORKDIR}/multistrap.conf" || true 
>      # Configure root filesystem
>      sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL}
> ${BAUDRATE_TTY} \ diff --git
> a/meta/recipes-devtools/buildchroot/buildchroot.bb
> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
> ccba683..e251a8f 100644 ---
> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -26,6 +26,8 @@
> WORKDIR = "${TMPDIR}/work/${PF}/${DISTRO}" do_build[stamp-extra-info]
> = "${DISTRO}-${DISTRO_ARCH}" do_build() {
> +    E="${@ bb.utils.export_proxies(d)}"
> +
>      install -d -m 755 ${WORKDIR}/hooks_multistrap
>  
>      # Copy config files
> @@ -48,7 +50,7 @@ do_build() {
>      cd ${TOPDIR}
>  
>      # Create root filesystem
> -    sudo multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f
> "${WORKDIR}/multistrap.conf" || true
> +    sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f
> "${WORKDIR}/multistrap.conf" || true 
>      # Install package builder script
>      sudo install -m 755 ${THISDIR}/files/build.sh ${BUILDCHROOT_DIR}
> diff --git a/scripts/isar-buildenv-internal
> b/scripts/isar-buildenv-internal index f14d1ff..94d7eb1 100755
> --- a/scripts/isar-buildenv-internal
> +++ b/scripts/isar-buildenv-internal
> @@ -66,5 +66,5 @@ export PATH
>  BBPATH="${BUILDDIR}"
>  export BBPATH
>  
> -BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR"
> +BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR http_proxy https_proxy ftp_proxy
> no_proxy" export BB_ENV_EXTRAWHITE

Re: [PATCH v2 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb

[Alexander Smirnov]

On 09/27/2017 03:02 PM, Andreas J. Reichel wrote:
> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> * BB_ENV_EXTRAWHITE provides a list for variables that are kept in the
> environment by bitbake. However, isar init script clears any additional
> settings. Thus, add proxy variables to BB_ENV_EXTRAWHITE in
> isar-buildenv-internal.
> 
> * Bitbake clears environment variables for each task within a recipe.
> However, bb.utils.export_proxies function can be used with an
> inline-python call to reexport the proxy settings.
> 
> * Sudo loses environment variables again, thus call multistrap with sudo
> with the -E option to preserve (the already cleaned) environment for the
> task's multistrap command.
> 
> Note:
> Downloads are normally done by the fetcher task, which calls a python
> function that in turn uses bb.util.export_proxies. However we have a
> non-fetcher task, which needs download capabilities as well.
>

Applied to next, thanks!

Alex