Re: [PATCH 3/3] classes: image: remove populate and replace it with a custom repo

[Henning Schild <henning.schild@siemens.com>]

Hey,

Hans pointed out that this one effectively doubles build-times. That is
because we do multistrap two times. Once for the buildchroot and one
for the rootfs. Without this patch these two run in parallel, with it
they are serialized because the rootfs can only start once all .debs
are ready.

He suggested building a repo, very much like mine, but replace "dpdk
-i" in do_populate with "apt-get install". This way only do_populate of
rootfs has to wait for buildchroot doing its stuff and multistrap can
already run in the background.

But i still think we should feed the debs into multistrap
- we introduce an important policy "the only way into the rootfs is
  multistrap"
  - that is very likely to help with the reproducible builds
  - that reduces all fetching/internet access to multistrap and the
    recipes build in chroot
- build times will still be very short compared to Yocto

To address the performance problem we should first look at what is
taking multistrap so long, the downloading or the installing. And
then look into caching:
- make the second multistrap use the files downloaded by the first
- keep the buildchroot across builds
- keep all remote .debs acroos builds

Henning

Am Tue, 1 Aug 2017 17:24:08 +0200
schrieb Henning Schild <henning.schild@siemens.com>:

> Create a custom Debian repository and give that to multistrap. That
> way multistrap can take care of our dependencies and we do not force
> in packages with "dpdk -i" anymore.
> The only way stuff gets into the rootfs is debian packages installed
> by multistrap. Plus whatever the configscripts do.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>  .../recipes-core/images/files/multistrap.conf.in   | 10 +++++-
>  meta/classes/ext4-img.bbclass                      |  2 +-
>  meta/classes/image.bbclass                         | 37
> ++++++++++------------ 3 files changed, 26 insertions(+), 23
> deletions(-)
> 
> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in
> b/meta-isar/recipes-core/images/files/multistrap.conf.in index
> 2d08c36..fd24baa 100644 ---
> a/meta-isar/recipes-core/images/files/multistrap.conf.in +++
> b/meta-isar/recipes-core/images/files/multistrap.conf.in @@ -4,7 +4,7
> @@ [General]
>  noauth=true
>  unpack=true
> -bootstrap=##DISTRO##
> +bootstrap=##DISTRO## Isar
>  aptsources=##DISTRO##
>  configscript=##CONFIG_SCRIPT##
>  setupscript=##SETUP_SCRIPT##
> @@ -16,3 +16,11 @@ suite=##DISTRO_SUITE##
>  components=##DISTRO_COMPONENTS##
>  packages=##IMAGE_PREINSTALL##
>  omitdebsrc=true
> +
> +[Isar]
> +packages=##IMAGE_INSTALL##
> +source=##DEPLOY_DIR_DEB##
> +suite=./
> +components=
> +omitdebsrc=true
> +
> diff --git a/meta/classes/ext4-img.bbclass
> b/meta/classes/ext4-img.bbclass index 61699b8..430032c 100644
> --- a/meta/classes/ext4-img.bbclass
> +++ b/meta/classes/ext4-img.bbclass
> @@ -36,4 +36,4 @@ do_ext4_image() {
>      fi
>  }
>  
> -addtask ext4_image before do_build after do_populate
> +addtask ext4_image before do_build after do_configure
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index ab3d1ff..d9bd879 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -9,6 +9,18 @@ IMAGE_INSTALL ?= ""
>  WORKDIR = "${TMPDIR}/work/${PN}/${MACHINE}"
>  S = "${WORKDIR}/rootfs"
>  
> +do_repository() {
> +    if [ -n "${IMAGE_INSTALL}" ]; then
> +	cd ${DEPLOY_DIR_DEB}
> +	sudo dpkg-scanpackages . > Packages
> +	sudo gzip Packages
> +    fi
> +}
> +
> +addtask repository before do_build
> +do_repository[deptask] = "do_build"
> +do_repository[stamp-extra-info] = "${MACHINE}"
> +
>  do_multistrap_conf() {
>      # Copy config file
>      install -m 644 ${FILESDIR}/multistrap.conf.in
> ${WORKDIR}/multistrap.conf @@ -33,9 +45,11 @@ do_multistrap_conf() {
>          ${WORKDIR}/multistrap.conf
>      sed -i
> 's|##DIR_HOOKS##|./tmp/work/${PN}/${MACHINE}/hooks_multistrap|' \
> ${WORKDIR}/multistrap.conf
> +    sed -i 's|##DEPLOY_DIR_DEB##|copy:///${DEPLOY_DIR_DEB}|'
> ${WORKDIR}/multistrap.conf
> +    sed -i 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|'
> ${WORKDIR}/multistrap.conf }
>  
> -addtask multistrap_conf before do_build
> +addtask multistrap_conf after do_repository before do_build
>  do_multistrap_conf[stamp-extra-info] = "${MACHINE}"
>  do_multistrap_conf[dirs] = "${WORKDIR}/hooks_multistrap"
>  
> @@ -44,7 +58,7 @@ do_multistrap() {
>  
>      # Create root filesystem
>      sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f \
> -        "${WORKDIR}/multistrap.conf" || true
> +        "${WORKDIR}/multistrap.conf"
>  }
>  
>  addtask multistrap after do_multistrap_conf before do_build
> @@ -60,22 +74,3 @@ do_configure() {
>  
>  addtask configure after do_multistrap before do_build
>  do_configure[stamp-extra-info] = "${MACHINE}"
> -
> -# Install Debian packages, that were built from sources
> -do_populate() {
> -    if [ -n "${IMAGE_INSTALL}" ]; then
> -        sudo mkdir -p ${S}/deb
> -
> -        for p in ${IMAGE_INSTALL}; do
> -            sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb
> -        done
> -
> -        sudo chroot ${S} /usr/bin/dpkg -i -R /deb
> -
> -        sudo rm -rf ${S}/deb
> -    fi
> -}
> -
> -addtask populate after do_configure before do_build
> -do_populate[deptask] = "do_build"
> -do_populate[stamp-extra-info] = "${MACHINE}"