[PATCH 0/3] use local repo for multistrap and drop "dpkg -i"

[PATCH 0/3] use local repo for multistrap and drop "dpkg -i"

[Henning Schild]

This series introduces support for a local repository to be used instead
of "dpkg -i" after bootstrap. The do_populate task was dropped.

It is based on the previous series i sent but it should not be too hard to
rebase it. Please let me know if it should be rebased.

Henning Schild (3):
  classes: images: remove cyclic "inherit"
  classes: move rootfs into image class and split into tasks
  classes: image: remove populate and replace it with a custom repo

 .../recipes-core/images/files/multistrap.conf.in   | 10 ++-
 meta-isar/recipes-core/images/isar-image-base.bb   | 43 +-----------
 meta/classes/ext4-img.bbclass                      |  4 +-
 meta/classes/image.bbclass                         | 76 +++++++++++++++++-----
 4 files changed, 75 insertions(+), 58 deletions(-)

-- 
2.13.0

[PATCH 1/3] classes: images: remove cyclic "inherit"

[Henning Schild]

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/recipes-core/images/isar-image-base.bb | 4 +++-
 meta/classes/ext4-img.bbclass                    | 2 +-
 meta/classes/image.bbclass                       | 3 ---
 3 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index 0810de0..25682ed 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -10,7 +10,9 @@ LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260
 
 PV = "1.0"
 
-inherit image
+IMAGE_TYPE    ?= "ext4-img"
+
+inherit ${IMAGE_TYPE}
 
 DEPENDS += "${IMAGE_INSTALL}"
 
diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass
index 5125d8e..61699b8 100644
--- a/meta/classes/ext4-img.bbclass
+++ b/meta/classes/ext4-img.bbclass
@@ -1,7 +1,7 @@
 # This software is a part of ISAR.
 # Copyright (C) 2015-2016 ilbers GmbH
 
-inherit isar-base-image
+inherit image
 
 # Extra space for rootfs in MB
 ROOTFS_EXTRA ?= "64"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 8db3352..3faa609 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -5,9 +5,6 @@ KERNEL_IMAGE ?= ""
 INITRD_IMAGE ?= ""
 
 IMAGE_INSTALL ?= ""
-IMAGE_TYPE    ?= "ext4-img"
-
-inherit ${IMAGE_TYPE}
 
 do_populate[stamp-extra-info] = "${MACHINE}"
 
-- 
2.13.0

[PATCH 2/3] classes: move rootfs into image class and split into tasks

[Henning Schild]

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/recipes-core/images/isar-image-base.bb | 39 ----------------
 meta/classes/image.bbclass                       | 58 +++++++++++++++++++++++-
 2 files changed, 56 insertions(+), 41 deletions(-)

diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index 25682ed..5de30ac 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -18,42 +18,3 @@ DEPENDS += "${IMAGE_INSTALL}"
 
 IMAGE_PREINSTALL += "apt \
                      dbus"
-
-WORKDIR = "${TMPDIR}/work/${PN}/${MACHINE}"
-S = "${WORKDIR}/rootfs"
-IMAGE_ROOTFS = "${S}"
-
-do_rootfs[stamp-extra-info] = "${MACHINE}"
-
-do_rootfs() {
-    install -d -m 755 ${WORKDIR}/hooks_multistrap
-
-    # Copy config file
-    install -m 644 ${FILESDIR}/multistrap.conf.in ${WORKDIR}/multistrap.conf
-    install -m 755 ${FILESDIR}/${DISTRO_CONFIG_SCRIPT} ${WORKDIR}/configscript.sh
-    install -m 755 ${FILESDIR}/setup.sh ${WORKDIR}
-    install -m 755 ${FILESDIR}/download_dev-random ${WORKDIR}/hooks_multistrap/
-
-    # Adjust multistrap config
-    sed -i 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|' ${WORKDIR}/multistrap.conf
-    sed -i 's|##DISTRO##|${DISTRO}|' ${WORKDIR}/multistrap.conf
-    sed -i 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|' ${WORKDIR}/multistrap.conf
-    sed -i 's|##DISTRO_SUITE##|${DISTRO_SUITE}|' ${WORKDIR}/multistrap.conf
-    sed -i 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|' ${WORKDIR}/multistrap.conf
-    sed -i 's|##CONFIG_SCRIPT##|./tmp/work/${PN}/${MACHINE}/configscript.sh|' ${WORKDIR}/multistrap.conf
-    sed -i 's|##SETUP_SCRIPT##|./tmp/work/${PN}/${MACHINE}/setup.sh|' ${WORKDIR}/multistrap.conf
-    sed -i 's|##DIR_HOOKS##|./tmp/work/${PN}/${MACHINE}/hooks_multistrap|' ${WORKDIR}/multistrap.conf
-
-    # Multistrap config use relative paths, so ensure that we are in the right folder
-    cd ${TOPDIR}
-
-    # Create root filesystem
-    sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f "${WORKDIR}/multistrap.conf" || true
-
-    # Configure root filesystem
-    sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL} ${BAUDRATE_TTY} \
-        ${ROOTFS_DEV}
-    sudo rm ${S}/configscript.sh
-}
-
-addtask rootfs before do_populate
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 3faa609..ab3d1ff 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -6,7 +6,60 @@ INITRD_IMAGE ?= ""
 
 IMAGE_INSTALL ?= ""
 
-do_populate[stamp-extra-info] = "${MACHINE}"
+WORKDIR = "${TMPDIR}/work/${PN}/${MACHINE}"
+S = "${WORKDIR}/rootfs"
+
+do_multistrap_conf() {
+    # Copy config file
+    install -m 644 ${FILESDIR}/multistrap.conf.in ${WORKDIR}/multistrap.conf
+    install -m 755 ${FILESDIR}/${DISTRO_CONFIG_SCRIPT} \
+        ${WORKDIR}/configscript.sh
+    install -m 755 ${FILESDIR}/setup.sh ${WORKDIR}
+    install -m 755 ${FILESDIR}/download_dev-random \
+        ${WORKDIR}/hooks_multistrap/
+
+    # Adjust multistrap config
+    sed -i 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|' \
+        ${WORKDIR}/multistrap.conf
+    sed -i 's|##DISTRO##|${DISTRO}|' ${WORKDIR}/multistrap.conf
+    sed -i 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|' \
+        ${WORKDIR}/multistrap.conf
+    sed -i 's|##DISTRO_SUITE##|${DISTRO_SUITE}|' ${WORKDIR}/multistrap.conf
+    sed -i 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|' \
+        ${WORKDIR}/multistrap.conf
+    sed -i 's|##CONFIG_SCRIPT##|./tmp/work/${PN}/${MACHINE}/configscript.sh|' \
+        ${WORKDIR}/multistrap.conf
+    sed -i 's|##SETUP_SCRIPT##|./tmp/work/${PN}/${MACHINE}/setup.sh|' \
+        ${WORKDIR}/multistrap.conf
+    sed -i 's|##DIR_HOOKS##|./tmp/work/${PN}/${MACHINE}/hooks_multistrap|' \
+        ${WORKDIR}/multistrap.conf
+}
+
+addtask multistrap_conf before do_build
+do_multistrap_conf[stamp-extra-info] = "${MACHINE}"
+do_multistrap_conf[dirs] = "${WORKDIR}/hooks_multistrap"
+
+do_multistrap() {
+    cd ${TOPDIR}
+
+    # Create root filesystem
+    sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f \
+        "${WORKDIR}/multistrap.conf" || true
+}
+
+addtask multistrap after do_multistrap_conf before do_build
+do_multistrap[stamp-extra-info] = "${MACHINE}"
+
+do_configure() {
+    cd ${TOPDIR}
+    # Configure root filesystem
+    sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL} \
+        ${BAUDRATE_TTY} ${ROOTFS_DEV}
+    sudo rm ${S}/configscript.sh
+}
+
+addtask configure after do_multistrap before do_build
+do_configure[stamp-extra-info] = "${MACHINE}"
 
 # Install Debian packages, that were built from sources
 do_populate() {
@@ -23,5 +76,6 @@ do_populate() {
     fi
 }
 
-addtask populate before do_build
+addtask populate after do_configure before do_build
 do_populate[deptask] = "do_build"
+do_populate[stamp-extra-info] = "${MACHINE}"
-- 
2.13.0

[PATCH 3/3] classes: image: remove populate and replace it with a custom repo

[Henning Schild]

Create a custom Debian repository and give that to multistrap. That way
multistrap can take care of our dependencies and we do not force in
packages with "dpdk -i" anymore.
The only way stuff gets into the rootfs is debian packages installed by
multistrap. Plus whatever the configscripts do.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 .../recipes-core/images/files/multistrap.conf.in   | 10 +++++-
 meta/classes/ext4-img.bbclass                      |  2 +-
 meta/classes/image.bbclass                         | 37 ++++++++++------------
 3 files changed, 26 insertions(+), 23 deletions(-)

diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in
index 2d08c36..fd24baa 100644
--- a/meta-isar/recipes-core/images/files/multistrap.conf.in
+++ b/meta-isar/recipes-core/images/files/multistrap.conf.in
@@ -4,7 +4,7 @@
 [General]
 noauth=true
 unpack=true
-bootstrap=##DISTRO##
+bootstrap=##DISTRO## Isar
 aptsources=##DISTRO##
 configscript=##CONFIG_SCRIPT##
 setupscript=##SETUP_SCRIPT##
@@ -16,3 +16,11 @@ suite=##DISTRO_SUITE##
 components=##DISTRO_COMPONENTS##
 packages=##IMAGE_PREINSTALL##
 omitdebsrc=true
+
+[Isar]
+packages=##IMAGE_INSTALL##
+source=##DEPLOY_DIR_DEB##
+suite=./
+components=
+omitdebsrc=true
+
diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass
index 61699b8..430032c 100644
--- a/meta/classes/ext4-img.bbclass
+++ b/meta/classes/ext4-img.bbclass
@@ -36,4 +36,4 @@ do_ext4_image() {
     fi
 }
 
-addtask ext4_image before do_build after do_populate
+addtask ext4_image before do_build after do_configure
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index ab3d1ff..d9bd879 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -9,6 +9,18 @@ IMAGE_INSTALL ?= ""
 WORKDIR = "${TMPDIR}/work/${PN}/${MACHINE}"
 S = "${WORKDIR}/rootfs"
 
+do_repository() {
+    if [ -n "${IMAGE_INSTALL}" ]; then
+	cd ${DEPLOY_DIR_DEB}
+	sudo dpkg-scanpackages . > Packages
+	sudo gzip Packages
+    fi
+}
+
+addtask repository before do_build
+do_repository[deptask] = "do_build"
+do_repository[stamp-extra-info] = "${MACHINE}"
+
 do_multistrap_conf() {
     # Copy config file
     install -m 644 ${FILESDIR}/multistrap.conf.in ${WORKDIR}/multistrap.conf
@@ -33,9 +45,11 @@ do_multistrap_conf() {
         ${WORKDIR}/multistrap.conf
     sed -i 's|##DIR_HOOKS##|./tmp/work/${PN}/${MACHINE}/hooks_multistrap|' \
         ${WORKDIR}/multistrap.conf
+    sed -i 's|##DEPLOY_DIR_DEB##|copy:///${DEPLOY_DIR_DEB}|' ${WORKDIR}/multistrap.conf
+    sed -i 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|' ${WORKDIR}/multistrap.conf
 }
 
-addtask multistrap_conf before do_build
+addtask multistrap_conf after do_repository before do_build
 do_multistrap_conf[stamp-extra-info] = "${MACHINE}"
 do_multistrap_conf[dirs] = "${WORKDIR}/hooks_multistrap"
 
@@ -44,7 +58,7 @@ do_multistrap() {
 
     # Create root filesystem
     sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f \
-        "${WORKDIR}/multistrap.conf" || true
+        "${WORKDIR}/multistrap.conf"
 }
 
 addtask multistrap after do_multistrap_conf before do_build
@@ -60,22 +74,3 @@ do_configure() {
 
 addtask configure after do_multistrap before do_build
 do_configure[stamp-extra-info] = "${MACHINE}"
-
-# Install Debian packages, that were built from sources
-do_populate() {
-    if [ -n "${IMAGE_INSTALL}" ]; then
-        sudo mkdir -p ${S}/deb
-
-        for p in ${IMAGE_INSTALL}; do
-            sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb
-        done
-
-        sudo chroot ${S} /usr/bin/dpkg -i -R /deb
-
-        sudo rm -rf ${S}/deb
-    fi
-}
-
-addtask populate after do_configure before do_build
-do_populate[deptask] = "do_build"
-do_populate[stamp-extra-info] = "${MACHINE}"
-- 
2.13.0

Re: [PATCH 0/3] use local repo for multistrap and drop "dpkg -i"

[Jan Kiszka]

On 2017-08-01 17:24, [ext] Henning Schild wrote:
> This series introduces support for a local repository to be used instead
> of "dpkg -i" after bootstrap. The do_populate task was dropped.
> 

I think this also provides a nice path to extract all packages for
archiving purposes and replay installations from them.

What would be a strategy to enable multiple Debian repo sources on top
of that?

Jan

> It is based on the previous series i sent but it should not be too hard to
> rebase it. Please let me know if it should be rebased.
> 
> Henning Schild (3):
>   classes: images: remove cyclic "inherit"
>   classes: move rootfs into image class and split into tasks
>   classes: image: remove populate and replace it with a custom repo
> 
>  .../recipes-core/images/files/multistrap.conf.in   | 10 ++-
>  meta-isar/recipes-core/images/isar-image-base.bb   | 43 +-----------
>  meta/classes/ext4-img.bbclass                      |  4 +-
>  meta/classes/image.bbclass                         | 76 +++++++++++++++++-----
>  4 files changed, 75 insertions(+), 58 deletions(-)
> 

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 0/3] use local repo for multistrap and drop "dpkg -i"

[Claudius Heine]

On 08/02/2017 08:16 AM, [ext] Jan Kiszka wrote:
> On 2017-08-01 17:24, [ext] Henning Schild wrote:
>> This series introduces support for a local repository to be used instead
>> of "dpkg -i" after bootstrap. The do_populate task was dropped.
>>
> 
> I think this also provides a nice path to extract all packages for
> archiving purposes and replay installations from them.

That would just archive your own packages though.
As we discussed offline, it would be nice to fetch all required packages 
into an local repo first and then use just this repo to install 
everything to the rootfs/buildchroot. This way you would not depend on 
the package archiving from upstream. But I currently don't know how to 
do that. Maybe we should look into what is possible with [1] and 
integrate it into isar.

Cheers,
Claudius

[1] https://launchpad.net/python-apt/

> 
> What would be a strategy to enable multiple Debian repo sources on top
> of that?
> 
> Jan
> 
>> It is based on the previous series i sent but it should not be too hard to
>> rebase it. Please let me know if it should be rebased.
>>
>> Henning Schild (3):
>>    classes: images: remove cyclic "inherit"
>>    classes: move rootfs into image class and split into tasks
>>    classes: image: remove populate and replace it with a custom repo
>>
>>   .../recipes-core/images/files/multistrap.conf.in   | 10 ++-
>>   meta-isar/recipes-core/images/isar-image-base.bb   | 43 +-----------
>>   meta/classes/ext4-img.bbclass                      |  4 +-
>>   meta/classes/image.bbclass                         | 76 +++++++++++++++++-----
>>   4 files changed, 75 insertions(+), 58 deletions(-)
>>
> 

Re: [PATCH 0/3] use local repo for multistrap and drop "dpkg -i"

[Henning Schild]

Am Wed, 2 Aug 2017 08:16:46 +0200
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 2017-08-01 17:24, [ext] Henning Schild wrote:
> > This series introduces support for a local repository to be used
> > instead of "dpkg -i" after bootstrap. The do_populate task was
> > dropped. 
> 
> I think this also provides a nice path to extract all packages for
> archiving purposes and replay installations from them.

Something like that could be done on top. After multistrap is done we
probably have all the used debs lying around somewhere.

> What would be a strategy to enable multiple Debian repo sources on top
> of that?

That would require adding sections to multistrap.conf and enabling them
in the global section. Splitting multistrap into a configure_multistrap
and multistrap is heading in this direction where one could hook in
and manipulate the config before the run.

> Jan
> 
> > It is based on the previous series i sent but it should not be too
> > hard to rebase it. Please let me know if it should be rebased.
> > 
> > Henning Schild (3):
> >   classes: images: remove cyclic "inherit"
> >   classes: move rootfs into image class and split into tasks
> >   classes: image: remove populate and replace it with a custom repo
> > 
> >  .../recipes-core/images/files/multistrap.conf.in   | 10 ++-
> >  meta-isar/recipes-core/images/isar-image-base.bb   | 43
> > +----------- meta/classes/ext4-img.bbclass                      |
> > 4 +- meta/classes/image.bbclass                         | 76
> > +++++++++++++++++----- 4 files changed, 75 insertions(+), 58
> > deletions(-) 
> 

Re: [PATCH 0/3] use local repo for multistrap and drop "dpkg -i"

[Henning Schild]

This series breaks multiconfig, moving the pieces around results in the
configs not beeing used anymore. I did not debug that yet, but just
see this series as an idea where we could go.

Henning

Am Tue, 1 Aug 2017 17:24:05 +0200
schrieb Henning Schild <henning.schild@siemens.com>:

> This series introduces support for a local repository to be used
> instead of "dpkg -i" after bootstrap. The do_populate task was
> dropped.
> 
> It is based on the previous series i sent but it should not be too
> hard to rebase it. Please let me know if it should be rebased.
> 
> Henning Schild (3):
>   classes: images: remove cyclic "inherit"
>   classes: move rootfs into image class and split into tasks
>   classes: image: remove populate and replace it with a custom repo
> 
>  .../recipes-core/images/files/multistrap.conf.in   | 10 ++-
>  meta-isar/recipes-core/images/isar-image-base.bb   | 43 +-----------
>  meta/classes/ext4-img.bbclass                      |  4 +-
>  meta/classes/image.bbclass                         | 76
> +++++++++++++++++----- 4 files changed, 75 insertions(+), 58
> deletions(-)
> 

Re: [PATCH 0/3] use local repo for multistrap and drop "dpkg -i"

[Henning Schild]

Am Wed, 2 Aug 2017 09:04:36 +0200
schrieb Claudius Heine <claudius.heine.ext@siemens.com>:

> On 08/02/2017 08:16 AM, [ext] Jan Kiszka wrote:
> > On 2017-08-01 17:24, [ext] Henning Schild wrote:  
> >> This series introduces support for a local repository to be used
> >> instead of "dpkg -i" after bootstrap. The do_populate task was
> >> dropped. 
> > 
> > I think this also provides a nice path to extract all packages for
> > archiving purposes and replay installations from them.  
> 
> That would just archive your own packages though.
> As we discussed offline, it would be nice to fetch all required
> packages into an local repo first and then use just this repo to
> install everything to the rootfs/buildchroot. This way you would not
> depend on the package archiving from upstream. But I currently don't
> know how to do that. Maybe we should look into what is possible with
> [1] and integrate it into isar.

Multistrap has something called retainsources i am hoping that will
help once we do everything with multistrap and drop any rootfs-changes
after it.

Henning

> Cheers,
> Claudius
> 
> [1] https://launchpad.net/python-apt/
> 
> > 
> > What would be a strategy to enable multiple Debian repo sources on
> > top of that?
> > 
> > Jan
> >   
> >> It is based on the previous series i sent but it should not be too
> >> hard to rebase it. Please let me know if it should be rebased.
> >>
> >> Henning Schild (3):
> >>    classes: images: remove cyclic "inherit"
> >>    classes: move rootfs into image class and split into tasks
> >>    classes: image: remove populate and replace it with a custom
> >> repo
> >>
> >>   .../recipes-core/images/files/multistrap.conf.in   | 10 ++-
> >>   meta-isar/recipes-core/images/isar-image-base.bb   | 43
> >> +----------- meta/classes/ext4-img.bbclass                      |
> >> 4 +- meta/classes/image.bbclass                         | 76
> >> +++++++++++++++++----- 4 files changed, 75 insertions(+), 58
> >> deletions(-) 
> >   

Re: [PATCH 0/3] use local repo for multistrap and drop "dpkg -i"

[Henning Schild]

Am Wed, 2 Aug 2017 10:12:53 +0200
schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:

> This series breaks multiconfig, moving the pieces around results in
> the configs not beeing used anymore. I did not debug that yet, but
> just see this series as an idea where we could go.

Turns out this issue in in current master, maybe i am building it
wrongly. But my sign for it being broken is that the rootfs does not
contain a kernel after build.

That means this series is ready for a review round!

Henning

> Henning
> 
> Am Tue, 1 Aug 2017 17:24:05 +0200
> schrieb Henning Schild <henning.schild@siemens.com>:
> 
> > This series introduces support for a local repository to be used
> > instead of "dpkg -i" after bootstrap. The do_populate task was
> > dropped.
> > 
> > It is based on the previous series i sent but it should not be too
> > hard to rebase it. Please let me know if it should be rebased.
> > 
> > Henning Schild (3):
> >   classes: images: remove cyclic "inherit"
> >   classes: move rootfs into image class and split into tasks
> >   classes: image: remove populate and replace it with a custom repo
> > 
> >  .../recipes-core/images/files/multistrap.conf.in   | 10 ++-
> >  meta-isar/recipes-core/images/isar-image-base.bb   | 43
> > +----------- meta/classes/ext4-img.bbclass                      |
> > 4 +- meta/classes/image.bbclass                         | 76
> > +++++++++++++++++----- 4 files changed, 75 insertions(+), 58
> > deletions(-)
> >   
> 

Re: [PATCH 3/3] classes: image: remove populate and replace it with a custom repo

[Henning Schild]

Hey,

Hans pointed out that this one effectively doubles build-times. That is
because we do multistrap two times. Once for the buildchroot and one
for the rootfs. Without this patch these two run in parallel, with it
they are serialized because the rootfs can only start once all .debs
are ready.

He suggested building a repo, very much like mine, but replace "dpdk
-i" in do_populate with "apt-get install". This way only do_populate of
rootfs has to wait for buildchroot doing its stuff and multistrap can
already run in the background.

But i still think we should feed the debs into multistrap
- we introduce an important policy "the only way into the rootfs is
  multistrap"
  - that is very likely to help with the reproducible builds
  - that reduces all fetching/internet access to multistrap and the
    recipes build in chroot
- build times will still be very short compared to Yocto

To address the performance problem we should first look at what is
taking multistrap so long, the downloading or the installing. And
then look into caching:
- make the second multistrap use the files downloaded by the first
- keep the buildchroot across builds
- keep all remote .debs acroos builds

Henning

Am Tue, 1 Aug 2017 17:24:08 +0200
schrieb Henning Schild <henning.schild@siemens.com>:

> Create a custom Debian repository and give that to multistrap. That
> way multistrap can take care of our dependencies and we do not force
> in packages with "dpdk -i" anymore.
> The only way stuff gets into the rootfs is debian packages installed
> by multistrap. Plus whatever the configscripts do.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>  .../recipes-core/images/files/multistrap.conf.in   | 10 +++++-
>  meta/classes/ext4-img.bbclass                      |  2 +-
>  meta/classes/image.bbclass                         | 37
> ++++++++++------------ 3 files changed, 26 insertions(+), 23
> deletions(-)
> 
> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in
> b/meta-isar/recipes-core/images/files/multistrap.conf.in index
> 2d08c36..fd24baa 100644 ---
> a/meta-isar/recipes-core/images/files/multistrap.conf.in +++
> b/meta-isar/recipes-core/images/files/multistrap.conf.in @@ -4,7 +4,7
> @@ [General]
>  noauth=true
>  unpack=true
> -bootstrap=##DISTRO##
> +bootstrap=##DISTRO## Isar
>  aptsources=##DISTRO##
>  configscript=##CONFIG_SCRIPT##
>  setupscript=##SETUP_SCRIPT##
> @@ -16,3 +16,11 @@ suite=##DISTRO_SUITE##
>  components=##DISTRO_COMPONENTS##
>  packages=##IMAGE_PREINSTALL##
>  omitdebsrc=true
> +
> +[Isar]
> +packages=##IMAGE_INSTALL##
> +source=##DEPLOY_DIR_DEB##
> +suite=./
> +components=
> +omitdebsrc=true
> +
> diff --git a/meta/classes/ext4-img.bbclass
> b/meta/classes/ext4-img.bbclass index 61699b8..430032c 100644
> --- a/meta/classes/ext4-img.bbclass
> +++ b/meta/classes/ext4-img.bbclass
> @@ -36,4 +36,4 @@ do_ext4_image() {
>      fi
>  }
>  
> -addtask ext4_image before do_build after do_populate
> +addtask ext4_image before do_build after do_configure
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index ab3d1ff..d9bd879 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -9,6 +9,18 @@ IMAGE_INSTALL ?= ""
>  WORKDIR = "${TMPDIR}/work/${PN}/${MACHINE}"
>  S = "${WORKDIR}/rootfs"
>  
> +do_repository() {
> +    if [ -n "${IMAGE_INSTALL}" ]; then
> +	cd ${DEPLOY_DIR_DEB}
> +	sudo dpkg-scanpackages . > Packages
> +	sudo gzip Packages
> +    fi
> +}
> +
> +addtask repository before do_build
> +do_repository[deptask] = "do_build"
> +do_repository[stamp-extra-info] = "${MACHINE}"
> +
>  do_multistrap_conf() {
>      # Copy config file
>      install -m 644 ${FILESDIR}/multistrap.conf.in
> ${WORKDIR}/multistrap.conf @@ -33,9 +45,11 @@ do_multistrap_conf() {
>          ${WORKDIR}/multistrap.conf
>      sed -i
> 's|##DIR_HOOKS##|./tmp/work/${PN}/${MACHINE}/hooks_multistrap|' \
> ${WORKDIR}/multistrap.conf
> +    sed -i 's|##DEPLOY_DIR_DEB##|copy:///${DEPLOY_DIR_DEB}|'
> ${WORKDIR}/multistrap.conf
> +    sed -i 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|'
> ${WORKDIR}/multistrap.conf }
>  
> -addtask multistrap_conf before do_build
> +addtask multistrap_conf after do_repository before do_build
>  do_multistrap_conf[stamp-extra-info] = "${MACHINE}"
>  do_multistrap_conf[dirs] = "${WORKDIR}/hooks_multistrap"
>  
> @@ -44,7 +58,7 @@ do_multistrap() {
>  
>      # Create root filesystem
>      sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f \
> -        "${WORKDIR}/multistrap.conf" || true
> +        "${WORKDIR}/multistrap.conf"
>  }
>  
>  addtask multistrap after do_multistrap_conf before do_build
> @@ -60,22 +74,3 @@ do_configure() {
>  
>  addtask configure after do_multistrap before do_build
>  do_configure[stamp-extra-info] = "${MACHINE}"
> -
> -# Install Debian packages, that were built from sources
> -do_populate() {
> -    if [ -n "${IMAGE_INSTALL}" ]; then
> -        sudo mkdir -p ${S}/deb
> -
> -        for p in ${IMAGE_INSTALL}; do
> -            sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb
> -        done
> -
> -        sudo chroot ${S} /usr/bin/dpkg -i -R /deb
> -
> -        sudo rm -rf ${S}/deb
> -    fi
> -}
> -
> -addtask populate after do_configure before do_build
> -do_populate[deptask] = "do_build"
> -do_populate[stamp-extra-info] = "${MACHINE}"

Re: [PATCH 0/3] use local repo for multistrap and drop "dpkg -i"

[Baurzhan Ismagulov]

Hello Henning,

On Tue, Aug 01, 2017 at 05:24:05PM +0200, Henning Schild wrote:
> This series introduces support for a local repository to be used instead
> of "dpkg -i" after bootstrap.

Before we discuss it in detail: We have an implementation of apt generation
that is also used for binary package caching across bitbake runs in the
lenormf/develop-l20170602-dpkg-cross branch provided to Hans and mentioned on
the list. Which advantages does your implementation bring?

I like your dpkg-scanpackages approach compared to reprepro in the existing
implementation. Analyzing the rest and choosing / combining the optimal
strategy will take some time. To speed up the process, could you perhaps sketch
the changes in the workflow in a few sentences?


Regarding the kernel: Which arch and suite are you building for? We don't
install kernels in QEMU images because they are not necessary.


Regarding the increase of build time, I'd really like to avoid that. Building
all configurations in one run already takes up to 40 min. The problem should be
easily mitigatable by splitting rootfs multistrap and apt-getting the built
packages into separate recipes. At least I'm not aware of any increase in the
existing implementation.

Re: [PATCH 0/3] use local repo for multistrap and drop "dpkg -i"

[Henning Schild]

Am Thu, 10 Aug 2017 14:46:27 +0200
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> Hello Henning,
> 
> On Tue, Aug 01, 2017 at 05:24:05PM +0200, Henning Schild wrote:
> > This series introduces support for a local repository to be used
> > instead of "dpkg -i" after bootstrap.  
> 
> Before we discuss it in detail: We have an implementation of apt
> generation that is also used for binary package caching across
> bitbake runs in the lenormf/develop-l20170602-dpkg-cross branch
> provided to Hans and mentioned on the list. Which advantages does
> your implementation bring?

As i said before there are too many branches and forks around. I would
like to keep the discussion on master and patches under review. A
comparison makes sense after someone sketched the changes in a few
sentences.

> I like your dpkg-scanpackages approach compared to reprepro in the
> existing implementation. Analyzing the rest and choosing / combining
> the optimal strategy will take some time. 

I am not sure i like it because it is a plain directory. Reprepro
probably has its advantages but is more complicated.

> To speed up the process,
> could you perhaps sketch the changes in the workflow in a few
> sentences?

The idea behind it is to introduce one policy. Everything that goes
into the rootfs comes out of debian packages and their hooks.
Multistrap is the only entrance to the rootfs.

- solves dependency tracking compared to "dpdk -i"
- limits fetching of debs to multistrap
 - if you fetch with "apt-get" in the chroot you get proxy problems
   again
 - you loose track of what you used for the rootfs
 - your rootfs needs apt-get installed
- makes multistrap the valid candidate to tell us exactly what we used
  and allow us to create a partial mirror for reproducible builds
 - evaluate "retainsources" feature

Changes in the workflow:
- create a repo as the first image-task
 - this one has to wait for all .debs to be built and copied
- drop do_populate which messes with the rootfs after our new
  gatekeeper multistrap

> Regarding the kernel: Which arch and suite are you building for? We
> don't install kernels in QEMU images because they are not necessary.

That was just me building incorrectly, i did not know about the
"multiconfig:" thing.
 
> Regarding the increase of build time, I'd really like to avoid that.

Me too, that is why i started the discussion on caching. Giving up on
the gatekeeper idea would IMHO not be the way to go.

> Building all configurations in one run already takes up to 40 min.

CI is patient and for basic testing one does not need to build all of
them.

> The problem should be easily mitigatable by splitting rootfs
> multistrap and apt-getting the built packages into separate recipes.

Yes but that is what i mean with "giving up the gatekeeper". We could
implement both ways controlled by a variable in a config. That would
speed up the builds during development and allow switching to strict
gatekeeper mode for releases.

Henning

> At least I'm not aware of any increase in the existing implementation.
 

Re: [PATCH 0/3] use local repo for multistrap and drop "dpkg -i"

[Baurzhan Ismagulov]

On Fri, Aug 11, 2017 at 10:13:46AM +0200, Henning Schild wrote:
> > Regarding the increase of build time, I'd really like to avoid that.
> 
> Me too, that is why i started the discussion on caching. Giving up on
> the gatekeeper idea would IMHO not be the way to go.
> 
> > Building all configurations in one run already takes up to 40 min.
> 
> CI is patient and for basic testing one does not need to build all of
> them.

I'm not patient ;) , since I test every commit during every rebase... We should
automate that, too. But even then, build time will still be developer time.
Performance is the last but not the least requirement.


> > The problem should be easily mitigatable by splitting rootfs
> > multistrap and apt-getting the built packages into separate recipes.
> 
> Yes but that is what i mean with "giving up the gatekeeper". We could
> implement both ways controlled by a variable in a config. That would
> speed up the builds during development and allow switching to strict
> gatekeeper mode for releases.

What is the disadvantage of splitting into two recipes? An image would still
require apt-get install, so this step would not be omitted. Implemented in this
way, the second recipe would be your gatekeeper, no?

Adding one boolean variable doubles the amount of testing required. It's
configuration permutations that kill. That is why I'd prefer settling on one
way.


With kind regards,
Baurzhan.