From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6600683098303627264 Date: Tue, 25 Sep 2018 03:05:07 -0700 (PDT) From: chombourger@gmail.com To: isar-users Message-Id: <2454394a-be4f-4ea5-9208-e3094cd50934@googlegroups.com> In-Reply-To: References: <1536842780-221-1-git-send-email-Cedric_Hombourger@mentor.com> <838acea7-e98d-6f48-8497-d3c13871a8c8@siemens.com> <20180913153743.6f6d72e7@md1pvb1c.ad001.siemens.net> Subject: Re: [PATCH v2] isar-bootstrap: preserve environment in sudo'ed debootstrap call MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_2014_494991773.1537869907695" X-Google-Token: ENOQqN0FImWyp4_1kK40 X-Google-IP: 192.94.31.2 X-TUID: uqXXIBAEHRIL ------=_Part_2014_494991773.1537869907695 Content-Type: multipart/alternative; boundary="----=_Part_2015_1617600335.1537869907695" ------=_Part_2015_1617600335.1537869907695 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit ping On Thursday, September 13, 2018 at 6:17:33 PM UTC+2, chomb...@gmail.com wrote: > > Hi Henning, > > > > I have reproduced the failure. Here's my setup: > > > > * Host: Windows 10 > > * Guest: Debian 9 (running under VMWare) > > * Isar: e231e88b447cdad1a233ad29ff23545bc50f398b (next) > > > > Steps to reproduce (from the VM): > > > > $ sudo route del -net 0.0.0.0 # > > $ sudo route add -net 172.17.0.0 netmask 255.255.255.0 gw 192.168.20.2 $ > ping -c 1 8.8.8.8 > > connect: network is unreachable # as expected, no direct connection to the > Internet $ export http_proxy=http://172.17.0.7:3128 $ export https_proxy= > http://172.17.0.7:3128 > > > > industrial@packer-debian-9-amd64:~/Projects/upstream/build-test$ bitbake > multiconfig:qemuamd64-stretch:isar-image-base > > ... > > NOTE: Executing RunQueue Tasks > > ERROR: mc:qemuamd64-stretch:isar-bootstrap-target-1.0-r0 do_bootstrap: > Function failed: do_bootstrap (log file is located at > /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd64/isar-bootstrap-target/temp/log.do_bootstrap.2240) > > ERROR: Logfile of failure stored in: > /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd64/isar-bootstrap-target/temp/log.do_bootstrap.2240 > > Log data follows: > > | DEBUG: Executing shell function do_bootstrap > > | umount: > > | /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd6 > > | 4/isar-bootstrap-target/rootfs/dev: mountpoint not found > > | umount: > > | /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd6 > > | 4/isar-bootstrap-target/rootfs/proc: mountpoint not found > > | W: Target architecture is the same as host architecture; disabling > > | QEMU support > > | I: Running command: debootstrap --arch amd64 --verbose > > | --variant=minbase --include=locales --components=main,contrib,non-free > > | stretch > > | /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd6 > > | 4/isar-bootstrap-target/rootfs http://ftp.de.debian.org/debian > > | I: Retrieving InRelease > > | I: Retrieving Release > > | E: Failed getting release file > > | http://ftp.de.debian.org/debian/dists/stretch/Release > > | WARNING: exit code 1 from a shell command. > > | ERROR: Function failed: do_bootstrap (log file is located at > > | /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd6 > > | 4/isar-bootstrap-target/temp/log.do_bootstrap.2240) > > ERROR: Task > (multiconfig:qemuamd64-stretch:/home/vmuser/Projects/upstream/isar/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb:do_bootstrap) > failed with exit code '1' > > NOTE: Tasks Summary: Attempted 12 tasks of which 10 didn't need to be > rerun and 1 failed. > > > > Logs for squid do show that the proxy was used by bitbake to fetch sources > such as libhello They did not show any attempts to connect to Debian > repositories > > > > I then added the previously submitted patch and the build went through > (and the squid logs did show a bunch of requests for debian.org) > > > > Cedric > > > > On Thursday, September 13, 2018 at 3:37:45 PM UTC+2, Henning Schild wrote: >> >> Am Thu, 13 Sep 2018 13:15:03 +0000 >> schrieb "Hombourger, Cedric" : >> >> > Hi Jan, >> > >> > Are you sure your company mirror of debian repositories wasn't used >> > in your builds? I will repeat the tests that I did both with and >> > without the changes. >> >> Yes, i have never seen proxy issues in that step and never use our >> internal mirror. >> >> Henning >> >> > I was traveling to Nurnberg earlier this week and unfortunately had a >> > very slow and unreliable internet connection at our hotel When >> > checking squid access logs in realtime, I noticed that no requests >> > were received while running do_bootstrap >> > >> > I however agree that we should make sure we are fixing a real problem >> > I will therefore collect additional data >> > >> > Cedric >> > >> > -----Original Message----- >> > From: Jan Kiszka [mailto:jan.k...@siemens.com] >> > Sent: Thursday, September 13, 2018 3:09 PM >> > To: Hombourger, Cedric ; >> > isar-...@googlegroups.com Subject: Re: [PATCH v2] isar-bootstrap: >> > preserve environment in sudo'ed debootstrap call >> > >> > Hi Cedric, >> > >> > On 13.09.18 14:46, Cedric Hombourger wrote: >> > > Make bitbake add proxy environment variables and preserve them when >> > > calling debootstrap under sudo. This is required to get >> > > user-defined proxies used while bootstraping Isar. >> > >> > Can you specify the error scenario a bit more precisely? We are >> > building with user-defined proxies frequently, and we do not need >> > that change. While I have no problem with saving one line of code, I >> > would be good to understand the setup, specifically as I requested to >> > set up a proxies test scenario in the upstream Q&A to avoid relying >> > on the Siemens-internal Q&A here. >> > >> > Thanks, >> > Jan >> > >> > > >> > > Signed-off-by: Cedric Hombourger >> > > --- >> > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 ++--- >> > > 1 file changed, 2 insertions(+), 3 deletions(-) >> > > >> > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >> > > index 4010307..cfad136 100644 >> > > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >> > > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >> > > @@ -171,7 +171,8 @@ isar_bootstrap() { >> > > esac >> > > shift >> > > done >> > > - sudo flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ >> > > + E="${@bb.utils.export_proxies(d)}" >> > > + sudo -E flock "${ISAR_BOOTSTRAP_LOCK}" -c "\ >> > > set -e >> > > if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then >> > > if [ -e "${ROOTFSDIR}" ]; then @@ -179,7 +180,6 @@ >> > > isar_bootstrap() { >> > > umount -l "${ROOTFSDIR}/proc" || true >> > > rm -rf "${ROOTFSDIR}" >> > > fi >> > > - E="${@bb.utils.export_proxies(d)}" >> > > if [ ${IS_HOST} ]; then >> > > ${DEBOOTSTRAP} --verbose \ >> > > --variant=minbase \ @@ -233,7 >> > > +233,6 @@ isar_bootstrap() { >> > > mount -t devtmpfs -o mode=0755,nosuid devtmpfs >> > > ${ROOTFSDIR}/dev mount -t proc none ${ROOTFSDIR}/proc >> > > >> > > - E="${@bb.utils.export_proxies(d)}" >> > > export DEBIAN_FRONTEND=noninteractive >> > > chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y >> > > chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y >> > > \ >> > >> > -- >> > Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate >> > Competence Center Embedded Linux >> > >> >> ------=_Part_2015_1617600335.1537869907695 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
ping

On Thursday, September 13, 2018 at 6:17:33 PM = UTC+2, chomb...@gmail.com wrote:

Hi Henning,

=C2=A0

I have reproduced the failure. Here's my setup:

=C2=A0

=C2=A0=C2=A0 * Host: Windows 10

=C2=A0=C2=A0 * Guest: Debian 9 (running under VMWare)

=C2=A0=C2=A0 * Isar: e231e88b447cdad1a233ad29ff23545bc50f398b (next)

=C2=A0

Steps to reproduce (from the VM):

=C2=A0

$ sudo route del -net 0.0.0.0 #

$ sudo route add -net 172.17.0.0 netmask 255.255.255.0 gw 192.168.20.2 $ ping -c 1 8.8.8.8

connect: network is unreachable # as expected, no direct connection to the Internet $ export http_proxy=3Dhttp://172.17.0.7:3128 $ export https_proxy=3Dhttp:/= /172.17.0.7:3128

=C2=A0

industrial@packer-debian-9-amd64:~/Projects/upstream/build-tes= t$ bitbake multiconfig:qemuamd64-stretch:isar-image-base

...

NOTE: Executing RunQueue Tasks

ERROR: mc:qemuamd64-stretch:isar-bootstrap-target-1.0-r0 do_bootstrap: Function failed: do_bootstrap (log file is located at /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch= -amd64/isar-bootstrap-target/temp/log.do_bootstrap.2240)

ERROR: Logfile of failure stored in: /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch= -amd64/isar-bootstrap-target/temp/log.do_bootstrap.2240

Log data follows:

| DEBUG: Executing shell function do_bootstrap

| umount:

| /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch= -amd6

| 4/isar-bootstrap-target/rootfs/dev: mountpoint not found

| umount:

| /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch= -amd6

| 4/isar-bootstrap-target/rootfs/proc: mountpoint not found

| W: Target architecture is the same as host architecture; disabling

| QEMU support

| I: Running command: debootstrap --arch amd64 --verbose

| --variant=3Dminbase --include=3Dlocales --components=3Dmain,contrib,non-free

| stretch

| /home/vmuser/Projects/upstream/build-test/tmp/work/debian-st= retch-amd6

| 4/isar-bootstrap-target/rootfs http://ftp.de.debian.org/de= bian

| I: Retrieving InRelease

| I: Retrieving Release

| E: Failed getting release file

| http://ftp.de.debian.org/debian/dists/stretch/Release

| WARNING: exit code 1 from a shell command.

| ERROR: Function failed: do_bootstrap (log file is located at

| /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch= -amd6

| 4/isar-bootstrap-target/temp/log.do_bootstrap.2240)

ERROR: Task (multiconfig:qemuamd64-stretch:/home/vmuser/Projects/upstream/isa= r/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb:do_bootstrap) failed with exit code '1'

NOTE: Tasks Summary: Attempted 12 tasks of which 10 didn't need to be rerun and 1 failed.

=C2=A0

Logs for squid do show that the proxy was used by bitbake to fetch sources such as libhello They did not show any attempts to connect= to Debian repositories

=C2=A0

I then added the previously submitted patch and the build went through (and the squid logs did show a bunch of requests for debian.org)

=C2=A0

Cedric

=C2=A0


On Thursday, September 13, 2018 at 3:37:45 PM UTC+2, Henni= ng Schild wrote:
Am Thu, 13 Sep 2018= 13:15:03 +0000
schrieb "Hombourger, Cedric" <Cedric_H= ...@mentor.com>:

> Hi Jan,
>=20
> Are you sure your company mirror of debian repositories wasn't= used
> in your builds? I will repeat the tests that I did both with and
> without the changes.

Yes, i have never seen proxy issues in that step and never use our
internal mirror.

Henning

> I was traveling to Nurnberg earlier this week and unfortunately ha= d a
> very slow and unreliable internet connection at our hotel When
> checking squid access logs in realtime, I noticed that no requests
> were received while running do_bootstrap
>=20
> I however agree that we should make sure we are fixing a real prob= lem
> I will therefore collect additional data
>=20
> Cedric
>=20
> -----Original Message-----
> From: Jan Kiszka [mailto:jan.k...@siemens.com<= /a>]=20
> Sent: Thursday, September 13, 2018 3:09 PM
> To: Hombourger, Cedric <Cedric_H...@mentor.= com>;
> isar-...@googlegroups.com Subject: Re: [PA= TCH v2] isar-bootstrap:
> preserve environment in sudo'ed debootstrap call
>=20
> Hi Cedric,
>=20
> On 13.09.18 14:46, Cedric Hombourger wrote:
> > Make bitbake add proxy environment variables and preserve the= m when=20
> > calling debootstrap under sudo. This is required to get
> > user-defined proxies used while bootstraping Isar. =C2=A0
>=20
> Can you specify the error scenario a bit more precisely? We are
> building with user-defined proxies frequently, and we do not need
> that change. While I have no problem with saving one line of code,= I
> would be good to understand the setup, specifically as I requested= to
> set up a proxies test scenario in the upstream Q&A to avoid re= lying
> on the Siemens-internal Q&A here.
>=20
> Thanks,
> Jan
>=20
> >=20
> > Signed-off-by: Cedric Hombourger <Cedr= ic_H...@mentor.com>
> > ---
> > =C2=A0meta/recipes-core/isar-bootstrap/isar-bootstrap.in= c | 5 ++---
> > =C2=A01 file changed, 2 insertions(+), 3 deletions(-)
> >=20
> > diff --git a/meta/recipes-core/isar-bootstrap/isar-boots= trap.inc=20
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > index 4010307..cfad136 100644
> > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.in= c
> > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.in= c
> > @@ -171,7 +171,8 @@ isar_bootstrap() {
> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0esac
> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0shift
> > =C2=A0 =C2=A0 =C2=A0done
> > - =C2=A0 =C2=A0sudo flock "${ISAR_BOOTSTRAP_LOCK}" = -c "\
> > + =C2=A0 =C2=A0E=3D"${@bb.utils.export_proxies(d)}&= quot;
> > + =C2=A0 =C2=A0sudo -E flock "${ISAR_BOOTSTRAP_LOCK}&quo= t; -c "\
> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0set -e
> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if [ ! -e "${DEPLOY_IS= AR_BOOTSTRAP}" ]; then
> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if [ -e "= ;${ROOTFSDIR}" ]; then @@ -179,7 +180,6 @@=20
> > isar_bootstrap() {
> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 umoun= t -l "${ROOTFSDIR}/proc" || true
> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 rm -r= f "${ROOTFSDIR}"
> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0fi
> > - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0E=3D"${@bb.ut= ils.export_proxies(d)}"
> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if [ ${IS_HOS= T} ]; then
> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0${DEBOOTSTRAP} --verbose \
> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 --variant=3Dminbase \ = @@ -233,7
> > +233,6 @@ isar_bootstrap() {
> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mount -t devt= mpfs -o mode=3D0755,nosuid devtmpfs
> > ${ROOTFSDIR}/dev mount -t proc none ${ROOTFSDIR}/proc
> > =C2=A0
> > - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0E=3D"${@bb.ut= ils.export_proxies(d)}"
> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0export DEBIAN= _FRONTEND=3Dnoninteractive
> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0chroot "= ${ROOTFSDIR}" /usr/bin/apt-get update -y
> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0chroot "= ${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y
> > \=20
>=20
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate
> Competence Center Embedded Linux
>=20

------=_Part_2015_1617600335.1537869907695-- ------=_Part_2014_494991773.1537869907695--