From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6463052406736289792 X-Received: by 10.25.202.17 with SMTP id a17mr388830lfg.6.1505121918158; Mon, 11 Sep 2017 02:25:18 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.100.130 with SMTP id y124ls1082098wmb.9.gmail; Mon, 11 Sep 2017 02:25:17 -0700 (PDT) X-Google-Smtp-Source: AOwi7QBHfuHo8IzZH0PVYhynZ9XZlQB7kXer0P90c6a99RQCtqvkP7Xw2Rds57inLayy3e8btHwP X-Received: by 10.28.136.143 with SMTP id k137mr303287wmd.3.1505121917730; Mon, 11 Sep 2017 02:25:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1505121917; cv=none; d=google.com; s=arc-20160816; b=Do5l1Iqdvt/vFYJQaMgIFs+XY14i3CNhe1VroMOvhtam3rfnW04dDSuHRSAyAdfwcK lDCwwpS1RJ9OwqsqFo7wglPL2glR3PPgvjMOg7Dr4UCsIdlLJP3iAg8aBXmelxjLiL2k sF3o1xkdi08v0GevuQyc5IklNUqGw2tdcr41c7h/sKIdZPc20YfhHErj+fBMw2IkaOaV Gr1u/7OBi+6QKcnzCZQvoAbnPEaqopWzbtE81ChArbOP8r5EMyM1oK/wtDbd89OplDIN /c7CidqzywZkfUVwilORldWdjeu/zqNAf0+xO0XZCSwZz+aIRQQUSV+EO98z7dP0VHUp khMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:in-reply-to:content-disposition:content-description :mime-version:references:message-id:subject:cc:to:from:date :arc-authentication-results; bh=T99oVWnoVXXy0HE4FQcSuz0+ebBOfqp3LmZ5IwiqjZY=; b=Ggooky9xMfbuID1O0XTOdI3ILzQZuRPs7EyaDHLCvBMR2Pf9Z4ORhegRrn5dcvIccF xpLzZEesr9tY7ORPg0+pbMh6yNytWsWAVfAzFH0vw+LLwukGhAYC5mAfKGk024SErMDq sfJV4TyceuyRGfQ2x5UjzStXXTlq1va37AzLH0x5AN/Drke1zKbWV4eTOvPBTAL3Gb4V tOIzhyIdv0wNiJh7ejFs7JYziH7o9FsjNdNMBj2kuByueTsritm8hFmYNvVbkoKwa1rw wYIuDSco+gMDs1koYQQCUKBlLdyPVI3/T3omg2blTX2ku+5NEbldHMU/11H+jeM+w2wd De5w== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 192.35.17.14 is neither permitted nor denied by best guess record for domain of andreas.reichel.ext@siemens.com) smtp.mailfrom=andreas.reichel.ext@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id 7si329384wme.7.2017.09.11.02.25.17 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 11 Sep 2017 02:25:17 -0700 (PDT) Received-SPF: neutral (google.com: 192.35.17.14 is neither permitted nor denied by best guess record for domain of andreas.reichel.ext@siemens.com) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 192.35.17.14 is neither permitted nor denied by best guess record for domain of andreas.reichel.ext@siemens.com) smtp.mailfrom=andreas.reichel.ext@siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id v8B9PHZ3010624 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Mon, 11 Sep 2017 11:25:17 +0200 Received: from iiotirae (golem.ppmd.siemens.net [139.25.69.162]) by mail1.sbs.de (8.15.2/8.15.2) with SMTP id v8B9PGaV018147; Mon, 11 Sep 2017 11:25:17 +0200 Date: Mon, 11 Sep 2017 11:24:59 +0200 From: Andreas Reichel To: Henning Schild Cc: isar-users@googlegroups.com Subject: Re: [PATCH 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb Message-ID: <20170911092458.GA32679@iiotirae> References: <20170907150335.30970-1-andreas.reichel.ext@siemens.com> <20170907150335.30970-2-andreas.reichel.ext@siemens.com> <20170908093738.609fe0df@md1em3qc> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Description: message Content-Disposition: inline In-Reply-To: <20170908093738.609fe0df@md1em3qc> User-Agent: Mutt/1.8.3 (2017-05-23) X-TUID: A6UWc6PTOCDU On Fri, Sep 08, 2017 at 09:37:38AM +0200, Henning Schild wrote: > Thanks for looking into this and finally finding a solution. More > comments inline. > > Am Thu, 7 Sep 2017 17:03:35 +0200 > schrieb "Andreas J. Reichel" : > > > From: Andreas Reichel > > > > * BB_ENV_EXTRAWHITE provides a list for variables that are kept in the > > environment by bitbake. However, isar init script clears any > > additional settings. Thus, add proxy variables to BB_ENV_EXTRAWHITE in > > isar-buildenv-internal. > > > > * Bitbake clears environment variables for each task within a recipe. > > However, bb.utils.export_proxies function can be used with an > > inline-python call to reexport the proxy settings. > > > > * Sudo loses environment variables again, thus call multistrap with > > sudo with the -E option to preserve (the already cleaned) environment > > for the task's multistrap command. > > > > Note: > > Downloads are normally done by the fetcher task, which calls a python > > function that in turn uses bb.util.export_proxies. However we have a > > non-fetcher task, which needs download capabilities as well. > > > > Signed-off-by: Andreas Reichel > > --- > > meta-isar/recipes-core/images/isar-image-base.bb | 8 +++++++- > > meta/recipes-devtools/buildchroot/buildchroot.bb | 9 +++++++-- > > scripts/isar-buildenv-internal | 2 +- > > 3 files changed, 15 insertions(+), 4 deletions(-) > > > > diff --git a/meta-isar/recipes-core/images/isar-image-base.bb > > b/meta-isar/recipes-core/images/isar-image-base.bb index > > b679d97..a826b88 100644 --- > > a/meta-isar/recipes-core/images/isar-image-base.bb +++ > > b/meta-isar/recipes-core/images/isar-image-base.bb @@ -24,6 +24,11 @@ > > IMAGE_ROOTFS = "${S}" do_rootfs[stamp-extra-info] = > > "${MACHINE}-${DISTRO}" > > do_rootfs() { > > + # Bitbake clears environment for all task functions, but we need > > the proxy > > + # settings in this task so do an inline python call which > > exports them > > + # again to the environment > > + E="${@ bb.utils.export_proxies(d)}" > > + > > I think the commit message is already verbose enough and the > function-name tells people that it is about proxies. IMHO no need for a > comment. > > > install -d -m 755 ${WORKDIR}/hooks_multistrap > > > > # Copy config file > > @@ -46,7 +51,8 @@ do_rootfs() { > > cd ${TOPDIR} > > > > # Create root filesystem > > - sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f > > "${WORKDIR}/multistrap.conf" || true > > + # We must use sudo -E here to preserve the environment because > > of proxy settings > > + sudo -E multistrap -a ${DISTRO_ARCH} -d "${S}" -f > > "${WORKDIR}/multistrap.conf" || true > > I know that the env was already cleared and that it should be safe to > use "sudo -E". What about the following? > > sudo http_proxy=$http_proxy ... no_proxy=$no_proxy multistrap ... > > It makes truly clear which variables should be set. There is not risk > to keep anything in addition and the comment can go away. Note, if the > variables end up empty because they where not set in the env everything > should work as if they where not set in the first place. I just tested > that with wget. > > > # Configure root filesystem > > sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL} > > ${BAUDRATE_TTY} \ diff --git > > a/meta/recipes-devtools/buildchroot/buildchroot.bb > > b/meta/recipes-devtools/buildchroot/buildchroot.bb index > > ccba683..7627015 100644 --- > > a/meta/recipes-devtools/buildchroot/buildchroot.bb +++ > > b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -26,6 +26,11 @@ > > WORKDIR = "${TMPDIR}/work/${PF}/${DISTRO}" do_build[stamp-extra-info] > > = "${DISTRO}-${DISTRO_ARCH}" do_build() { > > + # Bitbake clears environment for all task functions, but we need > > the proxy > > + # settings in this task so do an inline python call which > > exports them > > + # again to the environment > > + E="${@ bb.utils.export_proxies(d)}" > > + > > Again, comment can probably go. > > > install -d -m 755 ${WORKDIR}/hooks_multistrap > > > > # Copy config files > > @@ -48,11 +53,11 @@ do_build() { > > cd ${TOPDIR} > > > > # Create root filesystem > > - sudo multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f > > "${WORKDIR}/multistrap.conf" || true > > + sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f > > "${WORKDIR}/multistrap.conf" || true > > # Install package builder script > > sudo install -m 755 ${THISDIR}/files/build.sh ${BUILDCHROOT_DIR} > > > > # Configure root filesystem > > - sudo chroot ${BUILDCHROOT_DIR} /configscript.sh > > + sudo -E chroot ${BUILDCHROOT_DIR} /configscript.sh > > Consider the explicit export of the vars for those two sudos. Whatever > you decide it should be consistent between the 3 sudos. > > With the package hooks in place the configscript will probably shrink > or disappear. So if it does not access the internet today this step > should not gain "permission" to do so. Please consider dropping the > "-E" from the third sudo. > > > } > > diff --git a/scripts/isar-buildenv-internal > > b/scripts/isar-buildenv-internal index f14d1ff..94d7eb1 100755 > > --- a/scripts/isar-buildenv-internal > > +++ b/scripts/isar-buildenv-internal > > @@ -66,5 +66,5 @@ export PATH > > BBPATH="${BUILDDIR}" > > export BBPATH > > > > -BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR" > > +BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR http_proxy https_proxy ftp_proxy > > no_proxy" export BB_ENV_EXTRAWHITE > > I do not fully understand that change. As far as i understood the > problem, the fetcher was so far always able to deal with proxies > and _all_ the magic would be in bb.utils.export_proxies. Why is > export_proxies not enough for the other tasks? > > Henning I have just tested this without adding those variable names and again isar cannot connect to official debian mirrors. To me it seems obvious, that the whole environment is cleared besides 'BASEDIR' and 'BUILDDIR'. At least I cannot reproduce any other behavior. If you look in bitbake/lib/bb/utils.py, line 635, BB_ENV_EXTRAWHITE contains a list of approved variables, that remain in the environment. In line 629, approved is set to empty list if BB_PRESERVE_ENV is not set, which is the case. Line 641 defines clean_environment, which filters everything out, that is not mentioned in approved_variables(), except BB_PRESERVE_ENV is set, which is not the case. in bitbake's main.py, line 445, clean_environment gets called, which starts the before-mentioned mechanism. Thus, I do not believe, it has ever worked without extending BB_ENV_EXTRAWHITE. In official poky, we find BB_ENV_EXTRAWHITE_OE="MACHINE DISTRO TCMODE TCLIBC HTTP_PROXY http_proxy \ HTTPS_PROXY https_proxy FTP_PROXY ftp_proxy FTPS_PROXY ftps_proxy ALL_PROXY \ all_proxy NO_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY \ SDKMACHINE BB_NUMBER_THREADS BB_NO_NETWORK PARALLEL_MAKE GIT_PROXY_COMMAND \ SOCKS5_PASSWD SOCKS5_USER SCREENDIR STAMPS_DIR BBPATH_EXTRA BB_SETSCENE_ENFORCE" Oh what great wonder... we have the proxy variables there... Andreas -- Andreas Reichel Dipl.-Phys. (Univ.) Software Consultant Andreas.Reichel@tngtech.com, +49-174-3180074 TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082