RFC: need to support package builds ala pbuilder?

RFC: need to support package builds ala pbuilder?

[Hombourger, Cedric]

Hello all,

I recently came across an interesting case that may require us providing a mechanism to support building packages in their own private buildchroot
Let me first describe the issue:

# Isar defines two tasks to build Debian packages: (1) do_prepare and (2) do_build
# The former installs build dependencies while the latter does the actual build
#
# The Isar lock is acquired for do_prepare_build to serialize access to the package
# database. While this looks ok, we may have builds fail in the following scenario
#
#     core 1                           core 2
#     -------------------------------- ---------------------------
#
#     recipe1:do_prepare_build
#      |  download dependencies
#      |  install libssl-dev
#      |  task completes
#                                       recipe2:do_prepare_build
#     recipe2:do_build                   | download dependencies
#      | autoconf                        | remove libssl-dev
#      | make                            | install libssl1.0-dev
#
# Running "autoconf" or "make" while libssl-dev gets removed to allow installation of
# libssl1.0-dev may cause either to fail since OpenSSL headers / libraries will be
# temporarily removed

To keep locking simple and avoid introducing a big fat lock for the entire package build
(do_prepare_build + do_build), adding an option for a recipe to instruct bitbake to create
a buildchroot for a specific package (e.g. tmp/work/my-distro-amd64/my-package/buildchroot)
may be necessary. This would result in something conceptually similar to pbuilder but
integrated into Isar. The downside for those packages is that we would likely end-up
downloading the same packages again (not a big deal for those of you using a local
caching proxy such as apt-cacher-ng)

I therefore wanted to seek your opinion in either the need for such a mechanism or alternate solutions you may have in mind?

Thanks
Cedric

Re: RFC: need to support package builds ala pbuilder?

[Jan Kiszka]

On 13.09.18 15:05, Hombourger, Cedric wrote:
> Hello all,
> 
> I recently came across an interesting case that may require us providing a mechanism to support building packages in their own private buildchroot
> Let me first describe the issue:
> 
> # Isar defines two tasks to build Debian packages: (1) do_prepare and (2) do_build
> # The former installs build dependencies while the latter does the actual build
> #
> # The Isar lock is acquired for do_prepare_build to serialize access to the package
> # database. While this looks ok, we may have builds fail in the following scenario
> #
> #     core 1                           core 2
> #     -------------------------------- ---------------------------
> #
> #     recipe1:do_prepare_build
> #      |  download dependencies
> #      |  install libssl-dev
> #      |  task completes
> #                                       recipe2:do_prepare_build
> #     recipe2:do_build                   | download dependencies
> #      | autoconf                        | remove libssl-dev
> #      | make                            | install libssl1.0-dev
> #
> # Running "autoconf" or "make" while libssl-dev gets removed to allow installation of
> # libssl1.0-dev may cause either to fail since OpenSSL headers / libraries will be
> # temporarily removed

If recipe2 (or did you rather meant recipe3?) depends on libssl-dev, and that is
built without a proper dependency encoded, that's a recipe bug. that build step
must not start prio to the deploy_deb step of the producing recipe is done.

> 
> To keep locking simple and avoid introducing a big fat lock for the entire package build
> (do_prepare_build + do_build), adding an option for a recipe to instruct bitbake to create
> a buildchroot for a specific package (e.g. tmp/work/my-distro-amd64/my-package/buildchroot)
> may be necessary. This would result in something conceptually similar to pbuilder but
> integrated into Isar. The downside for those packages is that we would likely end-up
> downloading the same packages again (not a big deal for those of you using a local
> caching proxy such as apt-cacher-ng)
> 
> I therefore wanted to seek your opinion in either the need for such a mechanism or alternate solutions you may have in mind?

The above thing aside, having "clean-room" package built rather than sharing the
buildchroot would be a valuable feature, to catch recipe bugs early and more
reliably.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: RFC: need to support package builds ala pbuilder?

[Henning Schild]

Am Thu, 13 Sep 2018 15:15:00 +0200
schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:

> On 13.09.18 15:05, Hombourger, Cedric wrote:
> > Hello all,
> > 
> > I recently came across an interesting case that may require us
> > providing a mechanism to support building packages in their own
> > private buildchroot Let me first describe the issue:
> > 
> > # Isar defines two tasks to build Debian packages: (1) do_prepare
> > and (2) do_build # The former installs build dependencies while the
> > latter does the actual build #
> > # The Isar lock is acquired for do_prepare_build to serialize
> > access to the package # database. While this looks ok, we may have
> > builds fail in the following scenario #
> > #     core 1                           core 2
> > #     -------------------------------- ---------------------------
> > #
> > #     recipe1:do_prepare_build
> > #      |  download dependencies
> > #      |  install libssl-dev
> > #      |  task completes
> > #                                       recipe2:do_prepare_build
> > #     recipe2:do_build                   | download dependencies
> > #      | autoconf                        | remove libssl-dev
> > #      | make                            | install libssl1.0-dev
> > #
> > # Running "autoconf" or "make" while libssl-dev gets removed to
> > allow installation of # libssl1.0-dev may cause either to fail
> > since OpenSSL headers / libraries will be # temporarily removed  
> 
> If recipe2 (or did you rather meant recipe3?) depends on libssl-dev,
> and that is built without a proper dependency encoded, that's a
> recipe bug. that build step must not start prio to the deploy_deb
> step of the producing recipe is done.

No they just build two packages against different openssl versions,
where the -dev packages can not be installed at the same time.

I guess that is a special case and i would serialize such builds with
DEPENDS statements, maybe in .bbappend files.

> > To keep locking simple and avoid introducing a big fat lock for the
> > entire package build (do_prepare_build + do_build), adding an
> > option for a recipe to instruct bitbake to create a buildchroot for
> > a specific package (e.g.
> > tmp/work/my-distro-amd64/my-package/buildchroot) may be necessary.
> > This would result in something conceptually similar to pbuilder but
> > integrated into Isar. The downside for those packages is that we
> > would likely end-up downloading the same packages again (not a big
> > deal for those of you using a local caching proxy such as
> > apt-cacher-ng)
> > 
> > I therefore wanted to seek your opinion in either the need for such
> > a mechanism or alternate solutions you may have in mind?  
> 
> The above thing aside, having "clean-room" package built rather than
> sharing the buildchroot would be a valuable feature, to catch recipe
> bugs early and more reliably.

Agreed. We can even reuse the debootstrap output, which might reduce
the "time"-cost to acceptable. Either keep a copy or go for union
mounting with aufs/overlayfs.

Henning

> Jan
> 

RE: RFC: need to support package builds ala pbuilder?

[Hombourger, Cedric]

Hi Henning,

I have reproduced the failure. Here's my setup:

   * Host: Windows 10
   * Guest: Debian 9 (running under VMWare)
   * Isar: e231e88b447cdad1a233ad29ff23545bc50f398b (next)

Steps to reproduce (from the VM):

$ sudo route del -net 0.0.0.0 # 
$ sudo route add -net 172.17.0.0 netmask 255.255.255.0 gw 192.168.20.2
$ ping -c 1 8.8.8.8
connect: network is unreachable # as expected, no direct connection to the Internet
$ export http_proxy=http://172.17.0.7:3128
$ export https_proxy=http://172.17.0.7:3128

industrial@packer-debian-9-amd64:~/Projects/upstream/build-test$ bitbake multiconfig:qemuamd64-stretch:isar-image-base
...
NOTE: Executing RunQueue Tasks
ERROR: mc:qemuamd64-stretch:isar-bootstrap-target-1.0-r0 do_bootstrap: Function failed: do_bootstrap (log file is located at /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd64/isar-bootstrap-target/temp/log.do_bootstrap.2240)
ERROR: Logfile of failure stored in: /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd64/isar-bootstrap-target/temp/log.do_bootstrap.2240
Log data follows:
| DEBUG: Executing shell function do_bootstrap
| umount: /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd64/isar-bootstrap-target/rootfs/dev: mountpoint not found
| umount: /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd64/isar-bootstrap-target/rootfs/proc: mountpoint not found
| W: Target architecture is the same as host architecture; disabling QEMU support
| I: Running command: debootstrap --arch amd64 --verbose --variant=minbase --include=locales --components=main,contrib,non-free stretch /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd64/isar-bootstrap-target/rootfs http://ftp.de.debian.org/debian
| I: Retrieving InRelease
| I: Retrieving Release
| E: Failed getting release file http://ftp.de.debian.org/debian/dists/stretch/Release
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_bootstrap (log file is located at /home/vmuser/Projects/upstream/build-test/tmp/work/debian-stretch-amd64/isar-bootstrap-target/temp/log.do_bootstrap.2240)
ERROR: Task (multiconfig:qemuamd64-stretch:/home/vmuser/Projects/upstream/isar/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb:do_bootstrap) failed with exit code '1'
NOTE: Tasks Summary: Attempted 12 tasks of which 10 didn't need to be rerun and 1 failed.

Logs for squid do show that the proxy was used by bitbake to fetch sources such as libhello
They did not show any attempts to connect to Debian repositories

I then added the previously submitted patch and the build went through
(and the squid logs did show a bunch of requests for debian.org)

Cedric

-----Original Message-----
From: Henning Schild [mailto:henning.schild@siemens.com] 
Sent: Thursday, September 13, 2018 3:54 PM
To: [ext] Jan Kiszka <jan.kiszka@siemens.com>
Cc: Hombourger, Cedric <Cedric_Hombourger@mentor.com>; isar-users@googlegroups.com
Subject: Re: RFC: need to support package builds ala pbuilder?

Am Thu, 13 Sep 2018 15:15:00 +0200
schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:

> On 13.09.18 15:05, Hombourger, Cedric wrote:
> > Hello all,
> > 
> > I recently came across an interesting case that may require us 
> > providing a mechanism to support building packages in their own 
> > private buildchroot Let me first describe the issue:
> > 
> > # Isar defines two tasks to build Debian packages: (1) do_prepare 
> > and (2) do_build # The former installs build dependencies while the 
> > latter does the actual build # # The Isar lock is acquired for 
> > do_prepare_build to serialize access to the package # database. 
> > While this looks ok, we may have builds fail in the following 
> > scenario #
> > #     core 1                           core 2
> > #     -------------------------------- ---------------------------
> > #
> > #     recipe1:do_prepare_build
> > #      |  download dependencies
> > #      |  install libssl-dev
> > #      |  task completes
> > #                                       recipe2:do_prepare_build
> > #     recipe2:do_build                   | download dependencies
> > #      | autoconf                        | remove libssl-dev
> > #      | make                            | install libssl1.0-dev
> > #
> > # Running "autoconf" or "make" while libssl-dev gets removed to 
> > allow installation of # libssl1.0-dev may cause either to fail since 
> > OpenSSL headers / libraries will be # temporarily removed
> 
> If recipe2 (or did you rather meant recipe3?) depends on libssl-dev, 
> and that is built without a proper dependency encoded, that's a recipe 
> bug. that build step must not start prio to the deploy_deb step of the 
> producing recipe is done.

No they just build two packages against different openssl versions, where the -dev packages can not be installed at the same time.

I guess that is a special case and i would serialize such builds with DEPENDS statements, maybe in .bbappend files.

> > To keep locking simple and avoid introducing a big fat lock for the 
> > entire package build (do_prepare_build + do_build), adding an option 
> > for a recipe to instruct bitbake to create a buildchroot for a 
> > specific package (e.g.
> > tmp/work/my-distro-amd64/my-package/buildchroot) may be necessary.
> > This would result in something conceptually similar to pbuilder but 
> > integrated into Isar. The downside for those packages is that we 
> > would likely end-up downloading the same packages again (not a big 
> > deal for those of you using a local caching proxy such as
> > apt-cacher-ng)
> > 
> > I therefore wanted to seek your opinion in either the need for such 
> > a mechanism or alternate solutions you may have in mind?
> 
> The above thing aside, having "clean-room" package built rather than 
> sharing the buildchroot would be a valuable feature, to catch recipe 
> bugs early and more reliably.

Agreed. We can even reuse the debootstrap output, which might reduce the "time"-cost to acceptable. Either keep a copy or go for union mounting with aufs/overlayfs.

Henning

> Jan
>