From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6600694396621946880 X-Received: by 2002:a1c:ecb:: with SMTP id 194-v6mr793091wmo.9.1536850519358; Thu, 13 Sep 2018 07:55:19 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a50:c8c9:: with SMTP id k9-v6ls3793794edh.1.gmail; Thu, 13 Sep 2018 07:55:18 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZtD37lf/FVnMgty+iR5qZhFRV6ab7GdJ+Pgus6udAYsHZVQYiJVvfCZM6/0dFpiOvj1B8G X-Received: by 2002:aa7:d5ca:: with SMTP id d10-v6mr2238509eds.2.1536850518937; Thu, 13 Sep 2018 07:55:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536850518; cv=none; d=google.com; s=arc-20160816; b=ISypZv3CNmwY+Bq/w+9xGFnlum1DH63V9muOZWBy8qq03BHuvIRUxqjfaMEq17PSJ0 ay9NcnIfi3f1D4pJQ3DYQ98hlTBsW753Hv9fjgmCf9ye55mL/bF90BoZkqkhiUwcdTbX 0EQyWPBJ9bBagcerHJtymCBLvzgdRuY7i7PIHcXEhhpDTigKD/K/1ZOk0bTY+6Qx1fQp HHHa7M5nquVZBqVziD0BXYkHf6Kb8pPKpVA98RflMrfywcsFeZKnCv/h0ybwUGxKFlky zCAXp4hxj3YAbSMEIihigFKG3C599E8BnAnywwfiayVo/AOqgbUDUXmzCQdqBj15+Es3 uN+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:content-language :accept-language:in-reply-to:references:message-id:date:thread-index :thread-topic:subject:cc:to:from; bh=ojZsVa7b/38y09q14k9XvWhQtLh1Qn6Ah2d8G/IS+iI=; b=rR+zcDYLZIU6SozgiY7avmcVUWFZR1rP7AurKs6ri0BO/VmItCGYzeIvhPmzJ5BH5w sQb8bmK2DfIwl/ncihqH9OJnsFRNNe0QNTONTolYwhaihp7bKCOERqHpV/3OvQ4ctLIE WmXlSVJ0FqnT1BZgDgwTLzFpD/Vrp3DvRZxnUATAw/98EPIz5FjMtztOR1Yq03Hsllro +QFNB7oEM2OMNr/6/R6YBhaK9qK6Ru2iQOwdDZSbFAS6QbceMq3qCC0WLEvyk4FB2Ma5 uuXwlvDlcZnBzDo1HCLqcNCJuY/Vu4tY1HCAPLwZYuFD5xZsd2lmiBWGNQ2ueTM48CKU biog== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of cedric_hombourger@mentor.com designates 192.94.38.131 as permitted sender) smtp.mailfrom=Cedric_Hombourger@mentor.com Return-Path: Received: from relay1.mentorg.com (relay1.mentorg.com. [192.94.38.131]) by gmr-mx.google.com with ESMTPS id y4-v6si235739edh.4.2018.09.13.07.55.18 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Sep 2018 07:55:18 -0700 (PDT) Received-SPF: pass (google.com: domain of cedric_hombourger@mentor.com designates 192.94.38.131 as permitted sender) client-ip=192.94.38.131; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of cedric_hombourger@mentor.com designates 192.94.38.131 as permitted sender) smtp.mailfrom=Cedric_Hombourger@mentor.com Received: from nat-ies.mentorg.com ([192.94.31.2] helo=SVR-IES-MBX-04.mgc.mentorg.com) by relay1.mentorg.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-SHA384:256) id 1g0T1Z-0006zI-A1 from Cedric_Hombourger@mentor.com ; Thu, 13 Sep 2018 07:55:17 -0700 Received: from svr-ies-mbx-02.mgc.mentorg.com (139.181.222.2) by SVR-IES-MBX-04.mgc.mentorg.com (139.181.222.4) with Microsoft SMTP Server (TLS) id 15.0.1320.4; Thu, 13 Sep 2018 15:55:13 +0100 Received: from svr-ies-mbx-02.mgc.mentorg.com ([fe80::a01f:51c9:5b6c:e0c]) by svr-ies-mbx-02.mgc.mentorg.com ([fe80::a01f:51c9:5b6c:e0c%22]) with mapi id 15.00.1320.000; Thu, 13 Sep 2018 15:55:13 +0100 From: "Hombourger, Cedric" To: Henning Schild , "[ext] Jan Kiszka" CC: "isar-users@googlegroups.com" Subject: RE: RFC: need to support package builds ala pbuilder? Thread-Topic: RFC: need to support package builds ala pbuilder? Thread-Index: AdRLYWVhkUVS5utPRG2QlS8TNAnyBf//8/4AgAALCID//+CggA== Date: Thu, 13 Sep 2018 14:55:13 +0000 Message-ID: <8ff33e1510644084a73e72a9468b338e@svr-ies-mbx-02.mgc.mentorg.com> References: <5469172f38754fd6b432249a3bd1bd8d@svr-ies-mbx-02.mgc.mentorg.com> <20180913155429.52daa5b6@md1pvb1c.ad001.siemens.net> In-Reply-To: <20180913155429.52daa5b6@md1pvb1c.ad001.siemens.net> Accept-Language: en-US, en-IE Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [137.202.0.90] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-TUID: cmKOtpGl6X/I Hi Henning, I have reproduced the failure. Here's my setup: * Host: Windows 10 * Guest: Debian 9 (running under VMWare) * Isar: e231e88b447cdad1a233ad29ff23545bc50f398b (next) Steps to reproduce (from the VM): $ sudo route del -net 0.0.0.0 #=20 $ sudo route add -net 172.17.0.0 netmask 255.255.255.0 gw 192.168.20.2 $ ping -c 1 8.8.8.8 connect: network is unreachable # as expected, no direct connection to the = Internet $ export http_proxy=3Dhttp://172.17.0.7:3128 $ export https_proxy=3Dhttp://172.17.0.7:3128 industrial@packer-debian-9-amd64:~/Projects/upstream/build-test$ bitbake mu= lticonfig:qemuamd64-stretch:isar-image-base ... NOTE: Executing RunQueue Tasks ERROR: mc:qemuamd64-stretch:isar-bootstrap-target-1.0-r0 do_bootstrap: Func= tion failed: do_bootstrap (log file is located at /home/vmuser/Projects/ups= tream/build-test/tmp/work/debian-stretch-amd64/isar-bootstrap-target/temp/l= og.do_bootstrap.2240) ERROR: Logfile of failure stored in: /home/vmuser/Projects/upstream/build-t= est/tmp/work/debian-stretch-amd64/isar-bootstrap-target/temp/log.do_bootstr= ap.2240 Log data follows: | DEBUG: Executing shell function do_bootstrap | umount: /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch= -amd64/isar-bootstrap-target/rootfs/dev: mountpoint not found | umount: /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch= -amd64/isar-bootstrap-target/rootfs/proc: mountpoint not found | W: Target architecture is the same as host architecture; disabling QEMU s= upport | I: Running command: debootstrap --arch amd64 --verbose --variant=3Dminbas= e --include=3Dlocales --components=3Dmain,contrib,non-free stretch /home/vm= user/Projects/upstream/build-test/tmp/work/debian-stretch-amd64/isar-bootst= rap-target/rootfs http://ftp.de.debian.org/debian | I: Retrieving InRelease | I: Retrieving Release | E: Failed getting release file http://ftp.de.debian.org/debian/dists/stre= tch/Release | WARNING: exit code 1 from a shell command. | ERROR: Function failed: do_bootstrap (log file is located at /home/vmuser= /Projects/upstream/build-test/tmp/work/debian-stretch-amd64/isar-bootstrap-= target/temp/log.do_bootstrap.2240) ERROR: Task (multiconfig:qemuamd64-stretch:/home/vmuser/Projects/upstream/i= sar/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb:do_bootstrap)= failed with exit code '1' NOTE: Tasks Summary: Attempted 12 tasks of which 10 didn't need to be rerun= and 1 failed. Logs for squid do show that the proxy was used by bitbake to fetch sources = such as libhello They did not show any attempts to connect to Debian repositories I then added the previously submitted patch and the build went through (and the squid logs did show a bunch of requests for debian.org) Cedric -----Original Message----- From: Henning Schild [mailto:henning.schild@siemens.com]=20 Sent: Thursday, September 13, 2018 3:54 PM To: [ext] Jan Kiszka Cc: Hombourger, Cedric ; isar-users@googlegro= ups.com Subject: Re: RFC: need to support package builds ala pbuilder? Am Thu, 13 Sep 2018 15:15:00 +0200 schrieb "[ext] Jan Kiszka" : > On 13.09.18 15:05, Hombourger, Cedric wrote: > > Hello all, > >=20 > > I recently came across an interesting case that may require us=20 > > providing a mechanism to support building packages in their own=20 > > private buildchroot Let me first describe the issue: > >=20 > > # Isar defines two tasks to build Debian packages: (1) do_prepare=20 > > and (2) do_build # The former installs build dependencies while the=20 > > latter does the actual build # # The Isar lock is acquired for=20 > > do_prepare_build to serialize access to the package # database.=20 > > While this looks ok, we may have builds fail in the following=20 > > scenario # > > # core 1 core 2 > > # -------------------------------- --------------------------- > > # > > # recipe1:do_prepare_build > > # | download dependencies > > # | install libssl-dev > > # | task completes > > # recipe2:do_prepare_build > > # recipe2:do_build | download dependencies > > # | autoconf | remove libssl-dev > > # | make | install libssl1.0-dev > > # > > # Running "autoconf" or "make" while libssl-dev gets removed to=20 > > allow installation of # libssl1.0-dev may cause either to fail since=20 > > OpenSSL headers / libraries will be # temporarily removed >=20 > If recipe2 (or did you rather meant recipe3?) depends on libssl-dev,=20 > and that is built without a proper dependency encoded, that's a recipe=20 > bug. that build step must not start prio to the deploy_deb step of the=20 > producing recipe is done. No they just build two packages against different openssl versions, where t= he -dev packages can not be installed at the same time. I guess that is a special case and i would serialize such builds with DEPEN= DS statements, maybe in .bbappend files. > > To keep locking simple and avoid introducing a big fat lock for the=20 > > entire package build (do_prepare_build + do_build), adding an option=20 > > for a recipe to instruct bitbake to create a buildchroot for a=20 > > specific package (e.g. > > tmp/work/my-distro-amd64/my-package/buildchroot) may be necessary. > > This would result in something conceptually similar to pbuilder but=20 > > integrated into Isar. The downside for those packages is that we=20 > > would likely end-up downloading the same packages again (not a big=20 > > deal for those of you using a local caching proxy such as > > apt-cacher-ng) > >=20 > > I therefore wanted to seek your opinion in either the need for such=20 > > a mechanism or alternate solutions you may have in mind? >=20 > The above thing aside, having "clean-room" package built rather than=20 > sharing the buildchroot would be a valuable feature, to catch recipe=20 > bugs early and more reliably. Agreed. We can even reuse the debootstrap output, which might reduce the "t= ime"-cost to acceptable. Either keep a copy or go for union mounting with a= ufs/overlayfs. Henning > Jan >=20