From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6463052406736289792 X-Received: by 10.46.34.66 with SMTP id i63mr148718lji.36.1504857715712; Fri, 08 Sep 2017 01:01:55 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.229.14 with SMTP id c14ls215327wmh.3.canary-gmail; Fri, 08 Sep 2017 01:01:55 -0700 (PDT) X-Google-Smtp-Source: AOwi7QD69tg4P+6WPBaGTgt+sD1ElsasnHln04H5/QrWlR3DICHBE4oO9nP1NPigI52yFDu6ngaY X-Received: by 10.28.87.132 with SMTP id l126mr124460wmb.23.1504857715372; Fri, 08 Sep 2017 01:01:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1504857715; cv=none; d=google.com; s=arc-20160816; b=gOi2Qbx+Z7tW5oF+0pdeI0i0Riee4iaXya2slObA9qaPHNlypcRetNdJZ1Cwk10whp bQGxL6E2iaUv7ey+Zyc4gSHEGQDFtcrmKtib/G6GPDDmvmDTkpF/QIqeyL42kvWfZ7FB tTkX939vjwxKMcxR9gd386jBrnssJpqZg9RIwLsnFdBZd4BnduIeSiUE5Vw8zQW6oWfe Qk70P9eZoG87fFuBtTCfQqW1n/c104AfoMeaYkcCjPQeH3AkwSMg5PXAEYYGveaTF3E/ PIA2+OguZsdsG3FlFzVqvfyHtsSFBtOsZ3HPoIUY6ZR9Ytp8x3d8JbbAlXSTtoHAPs0h lj8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:arc-authentication-results; bh=YXwpvjGja6Jso0PXw9ehBccq2zsSZuOB7XG9/g4i2zY=; b=HcPp3hB64VpaDQHjst1GVsCF/PlqbMNViA/eyJ8lADr/kBoSA3WwlSNqu6o0h7ozDj mSd3mnfi8b/ph43BmM7HfgK5/qf1EGmje4hzLDNVGMxIvLMCqYHsJIszjH5BfjoXdUSl RpjfdwP5NRqvp+rGK5JVwXtyiJk68TBIDzXugjT9Xtq54sDhH07aJCSdHAiJMuZRoIFt ZbREjFLss0z5BLCOh8NkUsB94xXY39Z8LuB8x7+WRA4UWXXA/L5Jm6jXG9v9jE/pwHqJ x7KJCV7Y4ir8jP8wz9O+qLqRP/BbWtedk+EEutZzXuJh7yMHtf1YZoGxVx0LyqWk/3wq Jvsw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id 7si62840wme.7.2017.09.08.01.01.55 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 08 Sep 2017 01:01:55 -0700 (PDT) Received-SPF: neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) smtp.mailfrom=henning.schild@siemens.com Received: from mail1.siemens.de (mail1.siemens.de [139.23.33.14]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id v8881sif005766 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Fri, 8 Sep 2017 10:01:55 +0200 Received: from md1em3qc ([139.25.68.40]) by mail1.siemens.de (8.15.2/8.15.2) with ESMTP id v8881sIg026913; Fri, 8 Sep 2017 10:01:54 +0200 Date: Fri, 8 Sep 2017 10:02:04 +0200 From: Henning Schild To: "Andreas J. Reichel" Cc: Subject: Re: [PATCH 1/1] Add proxy support to isar-image-*.bb and buildchroot.bb Message-ID: <20170908100204.3ba1f0e3@md1em3qc> In-Reply-To: <20170908093738.609fe0df@md1em3qc> References: <20170907150335.30970-1-andreas.reichel.ext@siemens.com> <20170907150335.30970-2-andreas.reichel.ext@siemens.com> <20170908093738.609fe0df@md1em3qc> X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: FEGAhem64/Lw Am Fri, 8 Sep 2017 09:37:38 +0200 schrieb "[ext] Henning Schild" : > Thanks for looking into this and finally finding a solution. More > comments inline. > > Am Thu, 7 Sep 2017 17:03:35 +0200 > schrieb "Andreas J. Reichel" : > > > From: Andreas Reichel > > > > * BB_ENV_EXTRAWHITE provides a list for variables that are kept in > > the environment by bitbake. However, isar init script clears any > > additional settings. Thus, add proxy variables to BB_ENV_EXTRAWHITE > > in isar-buildenv-internal. > > > > * Bitbake clears environment variables for each task within a > > recipe. However, bb.utils.export_proxies function can be used with > > an inline-python call to reexport the proxy settings. > > > > * Sudo loses environment variables again, thus call multistrap with > > sudo with the -E option to preserve (the already cleaned) > > environment for the task's multistrap command. > > > > Note: > > Downloads are normally done by the fetcher task, which calls a > > python function that in turn uses bb.util.export_proxies. However > > we have a non-fetcher task, which needs download capabilities as > > well. > > > > Signed-off-by: Andreas Reichel > > --- > > meta-isar/recipes-core/images/isar-image-base.bb | 8 +++++++- > > meta/recipes-devtools/buildchroot/buildchroot.bb | 9 +++++++-- > > scripts/isar-buildenv-internal | 2 +- > > 3 files changed, 15 insertions(+), 4 deletions(-) > > > > diff --git a/meta-isar/recipes-core/images/isar-image-base.bb > > b/meta-isar/recipes-core/images/isar-image-base.bb index > > b679d97..a826b88 100644 --- > > a/meta-isar/recipes-core/images/isar-image-base.bb +++ > > b/meta-isar/recipes-core/images/isar-image-base.bb @@ -24,6 +24,11 > > @@ IMAGE_ROOTFS = "${S}" do_rootfs[stamp-extra-info] = > > "${MACHINE}-${DISTRO}" > > do_rootfs() { > > + # Bitbake clears environment for all task functions, but we > > need the proxy > > + # settings in this task so do an inline python call which > > exports them > > + # again to the environment > > + E="${@ bb.utils.export_proxies(d)}" > > + > > I think the commit message is already verbose enough and the > function-name tells people that it is about proxies. IMHO no need for > a comment. > > > install -d -m 755 ${WORKDIR}/hooks_multistrap > > > > # Copy config file > > @@ -46,7 +51,8 @@ do_rootfs() { > > cd ${TOPDIR} > > > > # Create root filesystem > > - sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f > > "${WORKDIR}/multistrap.conf" || true > > + # We must use sudo -E here to preserve the environment because > > of proxy settings > > + sudo -E multistrap -a ${DISTRO_ARCH} -d "${S}" -f > > "${WORKDIR}/multistrap.conf" || true > > I know that the env was already cleared and that it should be safe to > use "sudo -E". What about the following? > > sudo http_proxy=$http_proxy ... no_proxy=$no_proxy multistrap ... Well this would actually be pretty dangerous. The right side of the 4 assignments needs to get quoted and we should be safe. sudo http_proxy="$http_proxy" ... no_proxy="$no_proxy" multistrap ... Problem without the quotes, one could put commands into the variables and execute them with sudo: export foo="bla ls" sudo foo=$foo env | grep foo ls: cannot access 'env': No such file or directory The quotes solve that problem. Quote removal always comes last in posix shell Word Expansion. http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_06 Henning > It makes truly clear which variables should be set. There is not risk > to keep anything in addition and the comment can go away. Note, if the > variables end up empty because they where not set in the env > everything should work as if they where not set in the first place. I > just tested that with wget. > > > # Configure root filesystem > > sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL} > > ${BAUDRATE_TTY} \ diff --git > > a/meta/recipes-devtools/buildchroot/buildchroot.bb > > b/meta/recipes-devtools/buildchroot/buildchroot.bb index > > ccba683..7627015 100644 --- > > a/meta/recipes-devtools/buildchroot/buildchroot.bb +++ > > b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -26,6 +26,11 > > @@ WORKDIR = "${TMPDIR}/work/${PF}/${DISTRO}" > > do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" do_build() { > > + # Bitbake clears environment for all task functions, but we > > need the proxy > > + # settings in this task so do an inline python call which > > exports them > > + # again to the environment > > + E="${@ bb.utils.export_proxies(d)}" > > + > > Again, comment can probably go. > > > install -d -m 755 ${WORKDIR}/hooks_multistrap > > > > # Copy config files > > @@ -48,11 +53,11 @@ do_build() { > > cd ${TOPDIR} > > > > # Create root filesystem > > - sudo multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f > > "${WORKDIR}/multistrap.conf" || true > > + sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f > > "${WORKDIR}/multistrap.conf" || true > > # Install package builder script > > sudo install -m 755 ${THISDIR}/files/build.sh > > ${BUILDCHROOT_DIR} > > # Configure root filesystem > > - sudo chroot ${BUILDCHROOT_DIR} /configscript.sh > > + sudo -E chroot ${BUILDCHROOT_DIR} /configscript.sh > > Consider the explicit export of the vars for those two sudos. Whatever > you decide it should be consistent between the 3 sudos. > > With the package hooks in place the configscript will probably shrink > or disappear. So if it does not access the internet today this step > should not gain "permission" to do so. Please consider dropping the > "-E" from the third sudo. > > > } > > diff --git a/scripts/isar-buildenv-internal > > b/scripts/isar-buildenv-internal index f14d1ff..94d7eb1 100755 > > --- a/scripts/isar-buildenv-internal > > +++ b/scripts/isar-buildenv-internal > > @@ -66,5 +66,5 @@ export PATH > > BBPATH="${BUILDDIR}" > > export BBPATH > > > > -BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR" > > +BB_ENV_EXTRAWHITE="BASEDIR BUILDDIR http_proxy https_proxy > > ftp_proxy no_proxy" export BB_ENV_EXTRAWHITE > > I do not fully understand that change. As far as i understood the > problem, the fetcher was so far always able to deal with proxies > and _all_ the magic would be in bb.utils.export_proxies. Why is > export_proxies not enough for the other tasks? > > Henning >