* [PATCH] isar-bootstrap: preserve environment in sudo'ed debootstrap call @ 2018-09-13 12:20 Cedric Hombourger 2018-09-13 12:36 ` Henning Schild 0 siblings, 1 reply; 10+ messages in thread From: Cedric Hombourger @ 2018-09-13 12:20 UTC (permalink / raw) To: isar-users; +Cc: Cedric Hombourger Make bitbake add proxy environment variables and preserve them when calling debootstrap under sudo. This is required to get user-defined proxies used while bootstraping Isar. Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com> --- meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 4010307..a936e7f 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -171,7 +171,8 @@ isar_bootstrap() { esac shift done - sudo flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ + E="${@bb.utils.export_proxies(d)}" + sudo -E flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ set -e if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then if [ -e "${ROOTFSDIR}" ]; then @@ -179,7 +180,6 @@ isar_bootstrap() { umount -l "${ROOTFSDIR}/proc" || true rm -rf "${ROOTFSDIR}" fi - E="${@bb.utils.export_proxies(d)}" if [ ${IS_HOST} ]; then ${DEBOOTSTRAP} --verbose \ --variant=minbase \ -- 2.11.0 ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] isar-bootstrap: preserve environment in sudo'ed debootstrap call 2018-09-13 12:20 [PATCH] isar-bootstrap: preserve environment in sudo'ed debootstrap call Cedric Hombourger @ 2018-09-13 12:36 ` Henning Schild 2018-09-13 12:42 ` Hombourger, Cedric 0 siblings, 1 reply; 10+ messages in thread From: Henning Schild @ 2018-09-13 12:36 UTC (permalink / raw) To: Cedric Hombourger; +Cc: isar-users Am Thu, 13 Sep 2018 14:20:58 +0200 schrieb Cedric Hombourger <Cedric_Hombourger@mentor.com>: > Make bitbake add proxy environment variables and preserve them when > calling debootstrap under sudo. This is required to get user-defined > proxies used while bootstraping Isar. > > Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com> > --- > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > 4010307..a936e7f 100644 --- > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -171,7 > +171,8 @@ isar_bootstrap() { esac > shift > done > - sudo flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ > + E="${@bb.utils.export_proxies(d)}" > + sudo -E flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ > set -e > if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then > if [ -e "${ROOTFSDIR}" ]; then > @@ -179,7 +180,6 @@ isar_bootstrap() { > umount -l "${ROOTFSDIR}/proc" || true > rm -rf "${ROOTFSDIR}" > fi > - E="${@bb.utils.export_proxies(d)}" > if [ ${IS_HOST} ]; then > ${DEBOOTSTRAP} --verbose \ > --variant=minbase \ So you are pulling that thing out of the sudo? How does that change what is happening? One way or the other, the environment should be as expected. The other thing i was wondering, why is this export_proxies expression in the flock-block twice? If there is a difference in doing the export_proxies before "sudo flock", please explain why it should be outside. And remove the second inner export_proxies if possible. Henning ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: [PATCH] isar-bootstrap: preserve environment in sudo'ed debootstrap call 2018-09-13 12:36 ` Henning Schild @ 2018-09-13 12:42 ` Hombourger, Cedric 2018-09-13 12:46 ` [PATCH v2] " Cedric Hombourger 0 siblings, 1 reply; 10+ messages in thread From: Hombourger, Cedric @ 2018-09-13 12:42 UTC (permalink / raw) To: Henning Schild; +Cc: isar-users Hi Henning, I was also curious about this export_proxies() thing and looked at its implementation as I was not understanding how the E variable was being used export_proxies() expands to True if it succeeded. It is a python function that puts http_proxy and friends into the environment. When we execute the shell block passed to sudo, the Python code will evidently have no effect. If unsure, take a look at the generated run.do_bootstrap script. I missed the export_proxies() call we have towards the end of the sudo block. That's a good catch. Will update my patch Cedric -----Original Message----- From: Henning Schild [mailto:henning.schild@siemens.com] Sent: Thursday, September 13, 2018 2:36 PM To: Hombourger, Cedric <Cedric_Hombourger@mentor.com> Cc: isar-users@googlegroups.com Subject: Re: [PATCH] isar-bootstrap: preserve environment in sudo'ed debootstrap call Am Thu, 13 Sep 2018 14:20:58 +0200 schrieb Cedric Hombourger <Cedric_Hombourger@mentor.com>: > Make bitbake add proxy environment variables and preserve them when > calling debootstrap under sudo. This is required to get user-defined > proxies used while bootstraping Isar. > > Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com> > --- > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > 4010307..a936e7f 100644 --- > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -171,7 > +171,8 @@ isar_bootstrap() { esac > shift > done > - sudo flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ > + E="${@bb.utils.export_proxies(d)}" > + sudo -E flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ > set -e > if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then > if [ -e "${ROOTFSDIR}" ]; then @@ -179,7 +180,6 @@ > isar_bootstrap() { > umount -l "${ROOTFSDIR}/proc" || true > rm -rf "${ROOTFSDIR}" > fi > - E="${@bb.utils.export_proxies(d)}" > if [ ${IS_HOST} ]; then > ${DEBOOTSTRAP} --verbose \ > --variant=minbase \ So you are pulling that thing out of the sudo? How does that change what is happening? One way or the other, the environment should be as expected. The other thing i was wondering, why is this export_proxies expression in the flock-block twice? If there is a difference in doing the export_proxies before "sudo flock", please explain why it should be outside. And remove the second inner export_proxies if possible. Henning ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v2] isar-bootstrap: preserve environment in sudo'ed debootstrap call 2018-09-13 12:42 ` Hombourger, Cedric @ 2018-09-13 12:46 ` Cedric Hombourger 2018-09-13 13:09 ` Jan Kiszka 2018-09-25 17:39 ` Maxim Yu. Osipov 0 siblings, 2 replies; 10+ messages in thread From: Cedric Hombourger @ 2018-09-13 12:46 UTC (permalink / raw) To: isar-users; +Cc: Cedric Hombourger Make bitbake add proxy environment variables and preserve them when calling debootstrap under sudo. This is required to get user-defined proxies used while bootstraping Isar. Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com> --- meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 4010307..cfad136 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -171,7 +171,8 @@ isar_bootstrap() { esac shift done - sudo flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ + E="${@bb.utils.export_proxies(d)}" + sudo -E flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ set -e if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then if [ -e "${ROOTFSDIR}" ]; then @@ -179,7 +180,6 @@ isar_bootstrap() { umount -l "${ROOTFSDIR}/proc" || true rm -rf "${ROOTFSDIR}" fi - E="${@bb.utils.export_proxies(d)}" if [ ${IS_HOST} ]; then ${DEBOOTSTRAP} --verbose \ --variant=minbase \ @@ -233,7 +233,6 @@ isar_bootstrap() { mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${ROOTFSDIR}/dev mount -t proc none ${ROOTFSDIR}/proc - E="${@bb.utils.export_proxies(d)}" export DEBIAN_FRONTEND=noninteractive chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ -- 2.11.0 ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v2] isar-bootstrap: preserve environment in sudo'ed debootstrap call 2018-09-13 12:46 ` [PATCH v2] " Cedric Hombourger @ 2018-09-13 13:09 ` Jan Kiszka 2018-09-13 13:15 ` Hombourger, Cedric 2018-09-25 17:39 ` Maxim Yu. Osipov 1 sibling, 1 reply; 10+ messages in thread From: Jan Kiszka @ 2018-09-13 13:09 UTC (permalink / raw) To: Cedric Hombourger, isar-users Hi Cedric, On 13.09.18 14:46, Cedric Hombourger wrote: > Make bitbake add proxy environment variables and preserve them when > calling debootstrap under sudo. This is required to get user-defined > proxies used while bootstraping Isar. Can you specify the error scenario a bit more precisely? We are building with user-defined proxies frequently, and we do not need that change. While I have no problem with saving one line of code, I would be good to understand the setup, specifically as I requested to set up a proxies test scenario in the upstream Q&A to avoid relying on the Siemens-internal Q&A here. Thanks, Jan > > Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com> > --- > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 ++--- > 1 file changed, 2 insertions(+), 3 deletions(-) > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > index 4010307..cfad136 100644 > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > @@ -171,7 +171,8 @@ isar_bootstrap() { > esac > shift > done > - sudo flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ > + E="${@bb.utils.export_proxies(d)}" > + sudo -E flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ > set -e > if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then > if [ -e "${ROOTFSDIR}" ]; then > @@ -179,7 +180,6 @@ isar_bootstrap() { > umount -l "${ROOTFSDIR}/proc" || true > rm -rf "${ROOTFSDIR}" > fi > - E="${@bb.utils.export_proxies(d)}" > if [ ${IS_HOST} ]; then > ${DEBOOTSTRAP} --verbose \ > --variant=minbase \ > @@ -233,7 +233,6 @@ isar_bootstrap() { > mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${ROOTFSDIR}/dev > mount -t proc none ${ROOTFSDIR}/proc > > - E="${@bb.utils.export_proxies(d)}" > export DEBIAN_FRONTEND=noninteractive > chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y > chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ > -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: [PATCH v2] isar-bootstrap: preserve environment in sudo'ed debootstrap call 2018-09-13 13:09 ` Jan Kiszka @ 2018-09-13 13:15 ` Hombourger, Cedric 2018-09-13 13:37 ` Henning Schild 0 siblings, 1 reply; 10+ messages in thread From: Hombourger, Cedric @ 2018-09-13 13:15 UTC (permalink / raw) To: Jan Kiszka; +Cc: isar-users Hi Jan, Are you sure your company mirror of debian repositories wasn't used in your builds? I will repeat the tests that I did both with and without the changes. I was traveling to Nurnberg earlier this week and unfortunately had a very slow and unreliable internet connection at our hotel When checking squid access logs in realtime, I noticed that no requests were received while running do_bootstrap I however agree that we should make sure we are fixing a real problem I will therefore collect additional data Cedric -----Original Message----- From: Jan Kiszka [mailto:jan.kiszka@siemens.com] Sent: Thursday, September 13, 2018 3:09 PM To: Hombourger, Cedric <Cedric_Hombourger@mentor.com>; isar-users@googlegroups.com Subject: Re: [PATCH v2] isar-bootstrap: preserve environment in sudo'ed debootstrap call Hi Cedric, On 13.09.18 14:46, Cedric Hombourger wrote: > Make bitbake add proxy environment variables and preserve them when > calling debootstrap under sudo. This is required to get user-defined > proxies used while bootstraping Isar. Can you specify the error scenario a bit more precisely? We are building with user-defined proxies frequently, and we do not need that change. While I have no problem with saving one line of code, I would be good to understand the setup, specifically as I requested to set up a proxies test scenario in the upstream Q&A to avoid relying on the Siemens-internal Q&A here. Thanks, Jan > > Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com> > --- > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 ++--- > 1 file changed, 2 insertions(+), 3 deletions(-) > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > index 4010307..cfad136 100644 > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > @@ -171,7 +171,8 @@ isar_bootstrap() { > esac > shift > done > - sudo flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ > + E="${@bb.utils.export_proxies(d)}" > + sudo -E flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ > set -e > if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then > if [ -e "${ROOTFSDIR}" ]; then @@ -179,7 +180,6 @@ > isar_bootstrap() { > umount -l "${ROOTFSDIR}/proc" || true > rm -rf "${ROOTFSDIR}" > fi > - E="${@bb.utils.export_proxies(d)}" > if [ ${IS_HOST} ]; then > ${DEBOOTSTRAP} --verbose \ > --variant=minbase \ @@ -233,7 +233,6 > @@ isar_bootstrap() { > mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${ROOTFSDIR}/dev > mount -t proc none ${ROOTFSDIR}/proc > > - E="${@bb.utils.export_proxies(d)}" > export DEBIAN_FRONTEND=noninteractive > chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y > chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ > -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v2] isar-bootstrap: preserve environment in sudo'ed debootstrap call 2018-09-13 13:15 ` Hombourger, Cedric @ 2018-09-13 13:37 ` Henning Schild 2018-09-13 16:17 ` chombourger 0 siblings, 1 reply; 10+ messages in thread From: Henning Schild @ 2018-09-13 13:37 UTC (permalink / raw) To: Hombourger, Cedric; +Cc: Jan Kiszka, isar-users Am Thu, 13 Sep 2018 13:15:03 +0000 schrieb "Hombourger, Cedric" <Cedric_Hombourger@mentor.com>: > Hi Jan, > > Are you sure your company mirror of debian repositories wasn't used > in your builds? I will repeat the tests that I did both with and > without the changes. Yes, i have never seen proxy issues in that step and never use our internal mirror. Henning > I was traveling to Nurnberg earlier this week and unfortunately had a > very slow and unreliable internet connection at our hotel When > checking squid access logs in realtime, I noticed that no requests > were received while running do_bootstrap > > I however agree that we should make sure we are fixing a real problem > I will therefore collect additional data > > Cedric > > -----Original Message----- > From: Jan Kiszka [mailto:jan.kiszka@siemens.com] > Sent: Thursday, September 13, 2018 3:09 PM > To: Hombourger, Cedric <Cedric_Hombourger@mentor.com>; > isar-users@googlegroups.com Subject: Re: [PATCH v2] isar-bootstrap: > preserve environment in sudo'ed debootstrap call > > Hi Cedric, > > On 13.09.18 14:46, Cedric Hombourger wrote: > > Make bitbake add proxy environment variables and preserve them when > > calling debootstrap under sudo. This is required to get > > user-defined proxies used while bootstraping Isar. > > Can you specify the error scenario a bit more precisely? We are > building with user-defined proxies frequently, and we do not need > that change. While I have no problem with saving one line of code, I > would be good to understand the setup, specifically as I requested to > set up a proxies test scenario in the upstream Q&A to avoid relying > on the Siemens-internal Q&A here. > > Thanks, > Jan > > > > > Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com> > > --- > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 ++--- > > 1 file changed, 2 insertions(+), 3 deletions(-) > > > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > index 4010307..cfad136 100644 > > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > @@ -171,7 +171,8 @@ isar_bootstrap() { > > esac > > shift > > done > > - sudo flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ > > + E="${@bb.utils.export_proxies(d)}" > > + sudo -E flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ > > set -e > > if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then > > if [ -e "${ROOTFSDIR}" ]; then @@ -179,7 +180,6 @@ > > isar_bootstrap() { > > umount -l "${ROOTFSDIR}/proc" || true > > rm -rf "${ROOTFSDIR}" > > fi > > - E="${@bb.utils.export_proxies(d)}" > > if [ ${IS_HOST} ]; then > > ${DEBOOTSTRAP} --verbose \ > > --variant=minbase \ @@ -233,7 > > +233,6 @@ isar_bootstrap() { > > mount -t devtmpfs -o mode=0755,nosuid devtmpfs > > ${ROOTFSDIR}/dev mount -t proc none ${ROOTFSDIR}/proc > > > > - E="${@bb.utils.export_proxies(d)}" > > export DEBIAN_FRONTEND=noninteractive > > chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y > > chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y > > \ > > -- > Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate > Competence Center Embedded Linux > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v2] isar-bootstrap: preserve environment in sudo'ed debootstrap call 2018-09-13 13:37 ` Henning Schild @ 2018-09-13 16:17 ` chombourger 2018-09-25 10:05 ` chombourger 0 siblings, 1 reply; 10+ messages in thread From: chombourger @ 2018-09-13 16:17 UTC (permalink / raw) To: isar-users [-- Attachment #1.1: Type: text/plain, Size: 6666 bytes --] Hi Henning, I have reproduced the failure. Here's my setup: * Host: Windows 10 * Guest: Debian 9 (running under VMWare) * Isar: e231e88b447cdad1a233ad29ff23545bc50f398b (next) Steps to reproduce (from the VM): $ sudo route del -net 0.0.0.0 # $ sudo route add -net 172.17.0.0 netmask 255.255.255.0 gw 192.168.20.2 $ ping -c 1 8.8.8.8 connect: network is unreachable # as expected, no direct connection to the Internet $ export http_proxy=http://172.17.0.7:3128 $ export https_proxy=http://172.17.0.7:3128 industrial@packer-debian-9-amd64:~/Projects/upstream/build-test$ bitbake multiconfig:qemuamd64-stretch:isar-image-base ... NOTE: Executing RunQueue Tasks ERROR: mc:qemuamd64-stretch:isar-bootstrap-target-1.0-r0 do_bootstrap: Function failed: do_bootstrap (log file is located at /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd64/isar-bootstrap-target/temp/log.do_bootstrap.2240) ERROR: Logfile of failure stored in: /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd64/isar-bootstrap-target/temp/log.do_bootstrap.2240 Log data follows: | DEBUG: Executing shell function do_bootstrap | umount: | /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd6 | 4/isar-bootstrap-target/rootfs/dev: mountpoint not found | umount: | /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd6 | 4/isar-bootstrap-target/rootfs/proc: mountpoint not found | W: Target architecture is the same as host architecture; disabling | QEMU support | I: Running command: debootstrap --arch amd64 --verbose | --variant=minbase --include=locales --components=main,contrib,non-free | stretch | /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd6 | 4/isar-bootstrap-target/rootfs http://ftp.de.debian.org/debian | I: Retrieving InRelease | I: Retrieving Release | E: Failed getting release file | http://ftp.de.debian.org/debian/dists/stretch/Release | WARNING: exit code 1 from a shell command. | ERROR: Function failed: do_bootstrap (log file is located at | /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd6 | 4/isar-bootstrap-target/temp/log.do_bootstrap.2240) ERROR: Task (multiconfig:qemuamd64-stretch:/home/vmuser/Projects/upstream/isar/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb:do_bootstrap) failed with exit code '1' NOTE: Tasks Summary: Attempted 12 tasks of which 10 didn't need to be rerun and 1 failed. Logs for squid do show that the proxy was used by bitbake to fetch sources such as libhello They did not show any attempts to connect to Debian repositories I then added the previously submitted patch and the build went through (and the squid logs did show a bunch of requests for debian.org) Cedric On Thursday, September 13, 2018 at 3:37:45 PM UTC+2, Henning Schild wrote: > > Am Thu, 13 Sep 2018 13:15:03 +0000 > schrieb "Hombourger, Cedric" <Cedric_H...@mentor.com <javascript:>>: > > > Hi Jan, > > > > Are you sure your company mirror of debian repositories wasn't used > > in your builds? I will repeat the tests that I did both with and > > without the changes. > > Yes, i have never seen proxy issues in that step and never use our > internal mirror. > > Henning > > > I was traveling to Nurnberg earlier this week and unfortunately had a > > very slow and unreliable internet connection at our hotel When > > checking squid access logs in realtime, I noticed that no requests > > were received while running do_bootstrap > > > > I however agree that we should make sure we are fixing a real problem > > I will therefore collect additional data > > > > Cedric > > > > -----Original Message----- > > From: Jan Kiszka [mailto:jan.k...@siemens.com <javascript:>] > > Sent: Thursday, September 13, 2018 3:09 PM > > To: Hombourger, Cedric <Cedric_H...@mentor.com <javascript:>>; > > isar-...@googlegroups.com <javascript:> Subject: Re: [PATCH v2] > isar-bootstrap: > > preserve environment in sudo'ed debootstrap call > > > > Hi Cedric, > > > > On 13.09.18 14:46, Cedric Hombourger wrote: > > > Make bitbake add proxy environment variables and preserve them when > > > calling debootstrap under sudo. This is required to get > > > user-defined proxies used while bootstraping Isar. > > > > Can you specify the error scenario a bit more precisely? We are > > building with user-defined proxies frequently, and we do not need > > that change. While I have no problem with saving one line of code, I > > would be good to understand the setup, specifically as I requested to > > set up a proxies test scenario in the upstream Q&A to avoid relying > > on the Siemens-internal Q&A here. > > > > Thanks, > > Jan > > > > > > > > Signed-off-by: Cedric Hombourger <Cedric_H...@mentor.com <javascript:>> > > > > --- > > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 ++--- > > > 1 file changed, 2 insertions(+), 3 deletions(-) > > > > > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > > index 4010307..cfad136 100644 > > > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > > @@ -171,7 +171,8 @@ isar_bootstrap() { > > > esac > > > shift > > > done > > > - sudo flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ > > > + E="${@bb.utils.export_proxies(d)}" > > > + sudo -E flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ > > > set -e > > > if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then > > > if [ -e "${ROOTFSDIR}" ]; then @@ -179,7 +180,6 @@ > > > isar_bootstrap() { > > > umount -l "${ROOTFSDIR}/proc" || true > > > rm -rf "${ROOTFSDIR}" > > > fi > > > - E="${@bb.utils.export_proxies(d)}" > > > if [ ${IS_HOST} ]; then > > > ${DEBOOTSTRAP} --verbose \ > > > --variant=minbase \ @@ -233,7 > > > +233,6 @@ isar_bootstrap() { > > > mount -t devtmpfs -o mode=0755,nosuid devtmpfs > > > ${ROOTFSDIR}/dev mount -t proc none ${ROOTFSDIR}/proc > > > > > > - E="${@bb.utils.export_proxies(d)}" > > > export DEBIAN_FRONTEND=noninteractive > > > chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y > > > chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y > > > \ > > > > -- > > Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate > > Competence Center Embedded Linux > > > > [-- Attachment #1.2: Type: text/html, Size: 10957 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v2] isar-bootstrap: preserve environment in sudo'ed debootstrap call 2018-09-13 16:17 ` chombourger @ 2018-09-25 10:05 ` chombourger 0 siblings, 0 replies; 10+ messages in thread From: chombourger @ 2018-09-25 10:05 UTC (permalink / raw) To: isar-users [-- Attachment #1.1: Type: text/plain, Size: 6952 bytes --] ping On Thursday, September 13, 2018 at 6:17:33 PM UTC+2, chomb...@gmail.com wrote: > > Hi Henning, > > > > I have reproduced the failure. Here's my setup: > > > > * Host: Windows 10 > > * Guest: Debian 9 (running under VMWare) > > * Isar: e231e88b447cdad1a233ad29ff23545bc50f398b (next) > > > > Steps to reproduce (from the VM): > > > > $ sudo route del -net 0.0.0.0 # > > $ sudo route add -net 172.17.0.0 netmask 255.255.255.0 gw 192.168.20.2 $ > ping -c 1 8.8.8.8 > > connect: network is unreachable # as expected, no direct connection to the > Internet $ export http_proxy=http://172.17.0.7:3128 $ export https_proxy= > http://172.17.0.7:3128 > > > > industrial@packer-debian-9-amd64:~/Projects/upstream/build-test$ bitbake > multiconfig:qemuamd64-stretch:isar-image-base > > ... > > NOTE: Executing RunQueue Tasks > > ERROR: mc:qemuamd64-stretch:isar-bootstrap-target-1.0-r0 do_bootstrap: > Function failed: do_bootstrap (log file is located at > /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd64/isar-bootstrap-target/temp/log.do_bootstrap.2240) > > ERROR: Logfile of failure stored in: > /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd64/isar-bootstrap-target/temp/log.do_bootstrap.2240 > > Log data follows: > > | DEBUG: Executing shell function do_bootstrap > > | umount: > > | /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd6 > > | 4/isar-bootstrap-target/rootfs/dev: mountpoint not found > > | umount: > > | /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd6 > > | 4/isar-bootstrap-target/rootfs/proc: mountpoint not found > > | W: Target architecture is the same as host architecture; disabling > > | QEMU support > > | I: Running command: debootstrap --arch amd64 --verbose > > | --variant=minbase --include=locales --components=main,contrib,non-free > > | stretch > > | /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd6 > > | 4/isar-bootstrap-target/rootfs http://ftp.de.debian.org/debian > > | I: Retrieving InRelease > > | I: Retrieving Release > > | E: Failed getting release file > > | http://ftp.de.debian.org/debian/dists/stretch/Release > > | WARNING: exit code 1 from a shell command. > > | ERROR: Function failed: do_bootstrap (log file is located at > > | /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd6 > > | 4/isar-bootstrap-target/temp/log.do_bootstrap.2240) > > ERROR: Task > (multiconfig:qemuamd64-stretch:/home/vmuser/Projects/upstream/isar/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb:do_bootstrap) > failed with exit code '1' > > NOTE: Tasks Summary: Attempted 12 tasks of which 10 didn't need to be > rerun and 1 failed. > > > > Logs for squid do show that the proxy was used by bitbake to fetch sources > such as libhello They did not show any attempts to connect to Debian > repositories > > > > I then added the previously submitted patch and the build went through > (and the squid logs did show a bunch of requests for debian.org) > > > > Cedric > > > > On Thursday, September 13, 2018 at 3:37:45 PM UTC+2, Henning Schild wrote: >> >> Am Thu, 13 Sep 2018 13:15:03 +0000 >> schrieb "Hombourger, Cedric" <Cedric_H...@mentor.com>: >> >> > Hi Jan, >> > >> > Are you sure your company mirror of debian repositories wasn't used >> > in your builds? I will repeat the tests that I did both with and >> > without the changes. >> >> Yes, i have never seen proxy issues in that step and never use our >> internal mirror. >> >> Henning >> >> > I was traveling to Nurnberg earlier this week and unfortunately had a >> > very slow and unreliable internet connection at our hotel When >> > checking squid access logs in realtime, I noticed that no requests >> > were received while running do_bootstrap >> > >> > I however agree that we should make sure we are fixing a real problem >> > I will therefore collect additional data >> > >> > Cedric >> > >> > -----Original Message----- >> > From: Jan Kiszka [mailto:jan.k...@siemens.com] >> > Sent: Thursday, September 13, 2018 3:09 PM >> > To: Hombourger, Cedric <Cedric_H...@mentor.com>; >> > isar-...@googlegroups.com Subject: Re: [PATCH v2] isar-bootstrap: >> > preserve environment in sudo'ed debootstrap call >> > >> > Hi Cedric, >> > >> > On 13.09.18 14:46, Cedric Hombourger wrote: >> > > Make bitbake add proxy environment variables and preserve them when >> > > calling debootstrap under sudo. This is required to get >> > > user-defined proxies used while bootstraping Isar. >> > >> > Can you specify the error scenario a bit more precisely? We are >> > building with user-defined proxies frequently, and we do not need >> > that change. While I have no problem with saving one line of code, I >> > would be good to understand the setup, specifically as I requested to >> > set up a proxies test scenario in the upstream Q&A to avoid relying >> > on the Siemens-internal Q&A here. >> > >> > Thanks, >> > Jan >> > >> > > >> > > Signed-off-by: Cedric Hombourger <Cedric_H...@mentor.com> >> > > --- >> > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 ++--- >> > > 1 file changed, 2 insertions(+), 3 deletions(-) >> > > >> > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >> > > index 4010307..cfad136 100644 >> > > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >> > > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >> > > @@ -171,7 +171,8 @@ isar_bootstrap() { >> > > esac >> > > shift >> > > done >> > > - sudo flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ >> > > + E="${@bb.utils.export_proxies(d)}" >> > > + sudo -E flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ >> > > set -e >> > > if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then >> > > if [ -e "${ROOTFSDIR}" ]; then @@ -179,7 +180,6 @@ >> > > isar_bootstrap() { >> > > umount -l "${ROOTFSDIR}/proc" || true >> > > rm -rf "${ROOTFSDIR}" >> > > fi >> > > - E="${@bb.utils.export_proxies(d)}" >> > > if [ ${IS_HOST} ]; then >> > > ${DEBOOTSTRAP} --verbose \ >> > > --variant=minbase \ @@ -233,7 >> > > +233,6 @@ isar_bootstrap() { >> > > mount -t devtmpfs -o mode=0755,nosuid devtmpfs >> > > ${ROOTFSDIR}/dev mount -t proc none ${ROOTFSDIR}/proc >> > > >> > > - E="${@bb.utils.export_proxies(d)}" >> > > export DEBIAN_FRONTEND=noninteractive >> > > chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y >> > > chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y >> > > \ >> > >> > -- >> > Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate >> > Competence Center Embedded Linux >> > >> >> [-- Attachment #1.2: Type: text/html, Size: 10884 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v2] isar-bootstrap: preserve environment in sudo'ed debootstrap call 2018-09-13 12:46 ` [PATCH v2] " Cedric Hombourger 2018-09-13 13:09 ` Jan Kiszka @ 2018-09-25 17:39 ` Maxim Yu. Osipov 1 sibling, 0 replies; 10+ messages in thread From: Maxim Yu. Osipov @ 2018-09-25 17:39 UTC (permalink / raw) To: Cedric Hombourger, isar-users On 9/13/18 3:46 PM, Cedric Hombourger wrote: > Make bitbake add proxy environment variables and preserve them when > calling debootstrap under sudo. This is required to get user-defined > proxies used while bootstraping Isar. Applied to the 'next', Thanks, Maxim. > Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com> > --- > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 ++--- > 1 file changed, 2 insertions(+), 3 deletions(-) > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > index 4010307..cfad136 100644 > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > @@ -171,7 +171,8 @@ isar_bootstrap() { > esac > shift > done > - sudo flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ > + E="${@bb.utils.export_proxies(d)}" > + sudo -E flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ > set -e > if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then > if [ -e "${ROOTFSDIR}" ]; then > @@ -179,7 +180,6 @@ isar_bootstrap() { > umount -l "${ROOTFSDIR}/proc" || true > rm -rf "${ROOTFSDIR}" > fi > - E="${@bb.utils.export_proxies(d)}" > if [ ${IS_HOST} ]; then > ${DEBOOTSTRAP} --verbose \ > --variant=minbase \ > @@ -233,7 +233,6 @@ isar_bootstrap() { > mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${ROOTFSDIR}/dev > mount -t proc none ${ROOTFSDIR}/proc > > - E="${@bb.utils.export_proxies(d)}" > export DEBIAN_FRONTEND=noninteractive > chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y > chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ > -- Maxim Osipov ilbers GmbH Maria-Merian-Str. 8 85521 Ottobrunn Germany +49 (151) 6517 6917 mosipov@ilbers.de http://ilbers.de/ Commercial register Munich, HRB 214197 General Manager: Baurzhan Ismagulov ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2018-09-25 17:39 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-09-13 12:20 [PATCH] isar-bootstrap: preserve environment in sudo'ed debootstrap call Cedric Hombourger 2018-09-13 12:36 ` Henning Schild 2018-09-13 12:42 ` Hombourger, Cedric 2018-09-13 12:46 ` [PATCH v2] " Cedric Hombourger 2018-09-13 13:09 ` Jan Kiszka 2018-09-13 13:15 ` Hombourger, Cedric 2018-09-13 13:37 ` Henning Schild 2018-09-13 16:17 ` chombourger 2018-09-25 10:05 ` chombourger 2018-09-25 17:39 ` Maxim Yu. Osipov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox