[PATCH 0/4] Various fixes

[PATCH 0/4] Various fixes

[Alexander Smirnov]

Hi all,

this series contains various fixes intended to have possibility to run parallel
builds. Now Isar supports parallel build for all the configuration by single
command:

$ bitbake multiconfig:qemuarm-wheezy:isar-image-base multiconfig:qemuarm-jessie:isar-image-base multiconfig:qemuarm-stretch:isar-image-base multiconfig:qemui386-jessie:isar-image-base multiconfig:qemui386-stretch:isar-image-base multiconfig:qemuamd64-jessie:isar-image-base multiconfig:qemuamd64-stretch:isar-image-base multiconfig:rpi-jessie:isar-image-base

Also this series updates 'start_vm' script to have possibility to test each image.

Alexander Smirnov (2):
  isar: Fix mutli-distro builds in parallel
  scripts: Update start_vm

Frank Lenormand (2):
  buildchroot: Fix dependency installation in buildchroot
  buildchroot: Update build script.

 meta-isar/recipes-core/images/isar-image-base.bb | 10 ++--
 meta/classes/dpkg.bbclass                        |  4 +-
 meta/classes/ext4-img.bbclass                    |  2 +-
 meta/classes/image.bbclass                       |  2 +-
 meta/conf/isar-bitbake.conf                      |  3 +-
 meta/recipes-devtools/buildchroot/buildchroot.bb |  2 +-
 meta/recipes-devtools/buildchroot/files/build.sh |  4 +-
 scripts/start_vm                                 | 60 ++++++++++++++++++------
 8 files changed, 58 insertions(+), 29 deletions(-)

-- 
2.1.4

[PATCH 1/4] isar: Fix mutli-distro builds in parallel

[Alexander Smirnov]

Extend stamp suffix to distinguish the same distro for various
architectures. This fixes parallel build for all possible
configurations.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar/recipes-core/images/isar-image-base.bb | 10 +++++-----
 meta/classes/dpkg.bbclass                        |  4 ++--
 meta/classes/ext4-img.bbclass                    |  2 +-
 meta/classes/image.bbclass                       |  2 +-
 meta/conf/isar-bitbake.conf                      |  3 +--
 meta/recipes-devtools/buildchroot/buildchroot.bb |  2 +-
 6 files changed, 11 insertions(+), 12 deletions(-)

diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index 337c329..b679d97 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -17,11 +17,11 @@ DEPENDS += "${IMAGE_INSTALL}"
 IMAGE_PREINSTALL += "apt \
                      dbus"
 
-WORKDIR = "${TMPDIR}/work/${PN}/${MACHINE}"
+WORKDIR = "${TMPDIR}/work/${PN}/${MACHINE}/${DISTRO}"
 S = "${WORKDIR}/rootfs"
 IMAGE_ROOTFS = "${S}"
 
-do_rootfs[stamp-extra-info] = "${MACHINE}"
+do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
 
 do_rootfs() {
     install -d -m 755 ${WORKDIR}/hooks_multistrap
@@ -38,9 +38,9 @@ do_rootfs() {
     sed -i 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|' ${WORKDIR}/multistrap.conf
     sed -i 's|##DISTRO_SUITE##|${DISTRO_SUITE}|' ${WORKDIR}/multistrap.conf
     sed -i 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|' ${WORKDIR}/multistrap.conf
-    sed -i 's|##CONFIG_SCRIPT##|./tmp/work/${PN}/${MACHINE}/configscript.sh|' ${WORKDIR}/multistrap.conf
-    sed -i 's|##SETUP_SCRIPT##|./tmp/work/${PN}/${MACHINE}/setup.sh|' ${WORKDIR}/multistrap.conf
-    sed -i 's|##DIR_HOOKS##|./tmp/work/${PN}/${MACHINE}/hooks_multistrap|' ${WORKDIR}/multistrap.conf
+    sed -i 's|##CONFIG_SCRIPT##|./tmp/work/${PN}/${MACHINE}/${DISTRO}/configscript.sh|' ${WORKDIR}/multistrap.conf
+    sed -i 's|##SETUP_SCRIPT##|./tmp/work/${PN}/${MACHINE}/${DISTRO}/setup.sh|' ${WORKDIR}/multistrap.conf
+    sed -i 's|##DIR_HOOKS##|./tmp/work/${PN}/${MACHINE}/${DISTRO}/hooks_multistrap|' ${WORKDIR}/multistrap.conf
 
     # Multistrap config use relative paths, so ensure that we are in the right folder
     cd ${TOPDIR}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index ea422f9..7c39c68 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -11,7 +11,7 @@ PP = "/home/builder/${PN}"
 BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
 
 do_unpack[dirs] = "${BUILDROOT}"
-do_unpack[stamp-extra-info] = "${DISTRO}"
+do_unpack[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 S ?= "${BUILDROOT}"
 
 # Unpack package and put it into working directory in buildchroot
@@ -31,7 +31,7 @@ python do_unpack() {
 
 addtask unpack after do_fetch before do_build
 
-do_build[stamp-extra-info] = "${DISTRO}"
+do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 # Build package from sources using build script
 do_build() {
diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass
index 65d4c11..8588626 100644
--- a/meta/classes/ext4-img.bbclass
+++ b/meta/classes/ext4-img.bbclass
@@ -6,7 +6,7 @@ ROOTFS_EXTRA ?= "64"
 
 EXT4_IMAGE_FILE = "${DEPLOY_DIR_IMAGE}/${PN}-${MACHINE}-${DISTRO}.ext4.img"
 
-do_ext4_image[stamp-extra-info] = "${MACHINE}"
+do_ext4_image[stamp-extra-info] = "${MACHINE}-${DISTRO}"
 
 # Generate ext4 filesystem image
 do_ext4_image() {
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 3e4877c..a7f0d74 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -9,7 +9,7 @@ IMAGE_TYPE    ?= "ext4-img"
 
 inherit ${IMAGE_TYPE}
 
-do_populate[stamp-extra-info] = "${MACHINE}"
+do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}"
 
 # Install Debian packages, that were built from sources
 do_populate() {
diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
index ce35a22..f85f5cc 100644
--- a/meta/conf/isar-bitbake.conf
+++ b/meta/conf/isar-bitbake.conf
@@ -18,10 +18,9 @@
 # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
 # OTHER DEALINGS IN THE SOFTWARE.
 
-
 DEPLOY_DIR_DEB = "${TMPDIR}/deploy/deb/${MACHINE}"
 SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
-BUILDCHROOT_DIR = "${TOPDIR}/tmp/work/buildchroot/${DISTRO}/rootfs"
+BUILDCHROOT_DIR = "${TOPDIR}/tmp/work/buildchroot/${DISTRO}-${DISTRO_ARCH}/rootfs"
 
 # Setup our default hash policy
 BB_SIGNATURE_HANDLER ?= "noop"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index ba1bc66..ccba683 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -23,7 +23,7 @@ BUILDCHROOT_PREINSTALL ?= "gcc \
 
 WORKDIR = "${TMPDIR}/work/${PF}/${DISTRO}"
 
-do_build[stamp-extra-info] = "${DISTRO}"
+do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 do_build() {
     install -d -m 755 ${WORKDIR}/hooks_multistrap
-- 
2.1.4

[PATCH 2/4] scripts: Update start_vm

[Alexander Smirnov]

Add support for user command line parameters which can specify:
 - Target architecture
 - Debian distribution
 - Path to build folder

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 scripts/start_vm | 60 ++++++++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 45 insertions(+), 15 deletions(-)

diff --git a/scripts/start_vm b/scripts/start_vm
index a043a52..32d4864 100755
--- a/scripts/start_vm
+++ b/scripts/start_vm
@@ -8,10 +8,6 @@ set -e
 ES_OK=0
 ES_BUG=3
 
-# TODO: Get the target from the command line or from the build environment
-ARCH=arm
-DISTRO=stretch
-
 # Convert bitbake assignments to shell ones
 # a = b         a=b
 # a ?= b        a=b
@@ -56,19 +52,53 @@ show_help() {
     echo "is used."
     echo
     echo "Parameters:"
-    echo "    --help        display this message and exit."
+    echo "    -a, --arch ARCH       set isar machine architecture."
+    echo "                          Supported: arm, i386, amd64."
+    echo "    -b, --build BUILD     set path to build directory."
+    echo "    -d, --distro DISTRO   set isar Debian distribution."
+    echo "                          Supported: jessie, stretch."
+    echo "    --help                display this message and exit."
+    echo
+    echo "Exit status:"
+    echo " 0  if OK,"
+    echo " 3  if invalid parameters are passed."
 }
 
-if [ "$#" == "1" -a "$1" == "--help" ]; then
-    show_help
-    exit 0
-fi
-
-if [ "$#" == "0" ]; then
-    BUILD_DIR=$PWD
-else
-    BUILD_DIR=$1
-fi
+# Set default values, that can be overwritten from command line
+ARCH=arm
+DISTRO=stretch
+BUILD_DIR=$PWD
+
+# Parse command line to get user configuration
+while [[ $# -gt 0 ]]
+do
+    key="$1"
+
+    case $key in
+    -h|--help)
+        show_help
+        exit 0
+        ;;
+    -a|--arch)
+        ARCH=$2
+        shift
+        ;;
+    -b|--build)
+        BUILD_DIR=$2
+        shift
+        ;;
+    -d|--distro)
+        DISTRO=$2
+        shift
+        ;;
+    *)
+        echo "error: invalid parameter '$key', please try '--help' to get list of supported parameters"
+        exit $ES_BUG
+        ;;
+    esac
+
+    shift
+done
 
 readonly IMAGE_DIR=$BUILD_DIR/tmp/deploy/images
 
-- 
2.1.4

[PATCH 3/4] buildchroot: Fix dependency installation in buildchroot

[Alexander Smirnov]

From: Frank Lenormand <flenormand@ilbers.de>

Use '-y' parameter for apt to resolve command prompt dialogs.

Signed-off-by: Frank Lenormand <flenormand@ilbers.de>
---
 meta/recipes-devtools/buildchroot/files/build.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
index 9268b05..ba443bc 100644
--- a/meta/recipes-devtools/buildchroot/files/build.sh
+++ b/meta/recipes-devtools/buildchroot/files/build.sh
@@ -12,7 +12,7 @@ cd $1
 DEPS=`perl -ne 'next if /^#/; $p=(s/^Build-Depends:\s*/ / or (/^ / and $p)); s/,|\n|\([^)]+\)//mg; print if $p' < debian/control`
 
 # Install deps
-apt-get install $DEPS
+apt-get install -y $DEPS
 
 # If autotools files have been created, update their timestamp to
 # prevent them from being regenerated
-- 
2.1.4

[PATCH 4/4] buildchroot: Update build script.

[Alexander Smirnov]

From: Frank Lenormand <flenormand@ilbers.de>

Handle the `package [arch]` syntax in Debian's `Build-Depends` field.

Signed-off-by: Frank Lenormand <flenormand@ilbers.de>
---
 meta/recipes-devtools/buildchroot/files/build.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh
index ba443bc..19d554e 100644
--- a/meta/recipes-devtools/buildchroot/files/build.sh
+++ b/meta/recipes-devtools/buildchroot/files/build.sh
@@ -9,7 +9,7 @@ cd $1
 # Get list of dependencies manually. The package is not in apt, so no apt-get
 # build-dep. dpkg-checkbuilddeps output contains version information and isn't
 # directly suitable for apt-get install.
-DEPS=`perl -ne 'next if /^#/; $p=(s/^Build-Depends:\s*/ / or (/^ / and $p)); s/,|\n|\([^)]+\)//mg; print if $p' < debian/control`
+DEPS=`perl -ne 'next if /^#/; $p=(s/^Build-Depends:\s*/ / or (/^ / and $p)); s/,|\n|\([^)]+\)|\[[^]]+\]//mg; print if $p' < debian/control`
 
 # Install deps
 apt-get install -y $DEPS
-- 
2.1.4

Re: [PATCH 1/4] isar: Fix mutli-distro builds in parallel

[Henning Schild]

Am Tue, 8 Aug 2017 13:04:21 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Extend stamp suffix to distinguish the same distro for various
> architectures. This fixes parallel build for all possible
> configurations.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta-isar/recipes-core/images/isar-image-base.bb | 10 +++++-----
>  meta/classes/dpkg.bbclass                        |  4 ++--
>  meta/classes/ext4-img.bbclass                    |  2 +-
>  meta/classes/image.bbclass                       |  2 +-
>  meta/conf/isar-bitbake.conf                      |  3 +--
>  meta/recipes-devtools/buildchroot/buildchroot.bb |  2 +-
>  6 files changed, 11 insertions(+), 12 deletions(-)
> 
> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
> b/meta-isar/recipes-core/images/isar-image-base.bb index
> 337c329..b679d97 100644 ---
> a/meta-isar/recipes-core/images/isar-image-base.bb +++
> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -17,11 +17,11
> @@ DEPENDS += "${IMAGE_INSTALL}" IMAGE_PREINSTALL += "apt \
>                       dbus"
>  
> -WORKDIR = "${TMPDIR}/work/${PN}/${MACHINE}"
> +WORKDIR = "${TMPDIR}/work/${PN}/${MACHINE}/${DISTRO}"

This should extend to original WORKDIR where you would also have PV and
PR included and do not copy its definition.

WORKDIR =+ "${MACHINE}/${DISTRO}"

The scheme is different to BUILDCHROOT_DIR where we use
/${DISTRO}-${DISTRO_ARCH}/ for the directory structure.

I still think BUILDCHROOT_DIR should be somewhere under the original
WORKDIR.

Henning

>  S = "${WORKDIR}/rootfs"
>  IMAGE_ROOTFS = "${S}"
>  
> -do_rootfs[stamp-extra-info] = "${MACHINE}"
> +do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>  
>  do_rootfs() {
>      install -d -m 755 ${WORKDIR}/hooks_multistrap
> @@ -38,9 +38,9 @@ do_rootfs() {
>      sed -i 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|'
> ${WORKDIR}/multistrap.conf sed -i
> 's|##DISTRO_SUITE##|${DISTRO_SUITE}|' ${WORKDIR}/multistrap.conf sed
> -i 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|'
> ${WORKDIR}/multistrap.conf
> -    sed -i
> 's|##CONFIG_SCRIPT##|./tmp/work/${PN}/${MACHINE}/configscript.sh|'
> ${WORKDIR}/multistrap.conf
> -    sed -i
> 's|##SETUP_SCRIPT##|./tmp/work/${PN}/${MACHINE}/setup.sh|'
> ${WORKDIR}/multistrap.conf
> -    sed -i
> 's|##DIR_HOOKS##|./tmp/work/${PN}/${MACHINE}/hooks_multistrap|'
> ${WORKDIR}/multistrap.conf
> +    sed -i
> 's|##CONFIG_SCRIPT##|./tmp/work/${PN}/${MACHINE}/${DISTRO}/configscript.sh|'
> ${WORKDIR}/multistrap.conf
> +    sed -i
> 's|##SETUP_SCRIPT##|./tmp/work/${PN}/${MACHINE}/${DISTRO}/setup.sh|'
> ${WORKDIR}/multistrap.conf
> +    sed -i
> 's|##DIR_HOOKS##|./tmp/work/${PN}/${MACHINE}/${DISTRO}/hooks_multistrap|'
> ${WORKDIR}/multistrap.conf # Multistrap config use relative paths, so
> ensure that we are in the right folder cd ${TOPDIR}
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index ea422f9..7c39c68 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -11,7 +11,7 @@ PP = "/home/builder/${PN}"
>  BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
>  
>  do_unpack[dirs] = "${BUILDROOT}"
> -do_unpack[stamp-extra-info] = "${DISTRO}"
> +do_unpack[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>  S ?= "${BUILDROOT}"
>  
>  # Unpack package and put it into working directory in buildchroot
> @@ -31,7 +31,7 @@ python do_unpack() {
>  
>  addtask unpack after do_fetch before do_build
>  
> -do_build[stamp-extra-info] = "${DISTRO}"
> +do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>  
>  # Build package from sources using build script
>  do_build() {
> diff --git a/meta/classes/ext4-img.bbclass
> b/meta/classes/ext4-img.bbclass index 65d4c11..8588626 100644
> --- a/meta/classes/ext4-img.bbclass
> +++ b/meta/classes/ext4-img.bbclass
> @@ -6,7 +6,7 @@ ROOTFS_EXTRA ?= "64"
>  
>  EXT4_IMAGE_FILE =
> "${DEPLOY_DIR_IMAGE}/${PN}-${MACHINE}-${DISTRO}.ext4.img" 
> -do_ext4_image[stamp-extra-info] = "${MACHINE}"
> +do_ext4_image[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>  
>  # Generate ext4 filesystem image
>  do_ext4_image() {
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 3e4877c..a7f0d74 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -9,7 +9,7 @@ IMAGE_TYPE    ?= "ext4-img"
>  
>  inherit ${IMAGE_TYPE}
>  
> -do_populate[stamp-extra-info] = "${MACHINE}"
> +do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>  
>  # Install Debian packages, that were built from sources
>  do_populate() {
> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
> index ce35a22..f85f5cc 100644
> --- a/meta/conf/isar-bitbake.conf
> +++ b/meta/conf/isar-bitbake.conf
> @@ -18,10 +18,9 @@
>  # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE
> OR # OTHER DEALINGS IN THE SOFTWARE.
>  
> -
>  DEPLOY_DIR_DEB = "${TMPDIR}/deploy/deb/${MACHINE}"
>  SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
> -BUILDCHROOT_DIR = "${TOPDIR}/tmp/work/buildchroot/${DISTRO}/rootfs"
> +BUILDCHROOT_DIR =
> "${TOPDIR}/tmp/work/buildchroot/${DISTRO}-${DISTRO_ARCH}/rootfs" 
>  # Setup our default hash policy
>  BB_SIGNATURE_HANDLER ?= "noop"
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
> ba1bc66..ccba683 100644 ---
> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -23,7 +23,7 @@
> BUILDCHROOT_PREINSTALL ?= "gcc \ 
>  WORKDIR = "${TMPDIR}/work/${PF}/${DISTRO}"
>  
> -do_build[stamp-extra-info] = "${DISTRO}"
> +do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>  
>  do_build() {
>      install -d -m 755 ${WORKDIR}/hooks_multistrap

Re: [PATCH 4/4] buildchroot: Update build script.

[Henning Schild]

Looks good, but this whole script should go into dpkg-src.bbclass into
the task where it gets called. Less indirection, files and confusion.

Henning

Am Tue, 8 Aug 2017 13:04:24 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> From: Frank Lenormand <flenormand@ilbers.de>
> 
> Handle the `package [arch]` syntax in Debian's `Build-Depends` field.
> 
> Signed-off-by: Frank Lenormand <flenormand@ilbers.de>
> ---
>  meta/recipes-devtools/buildchroot/files/build.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/recipes-devtools/buildchroot/files/build.sh
> b/meta/recipes-devtools/buildchroot/files/build.sh index
> ba443bc..19d554e 100644 ---
> a/meta/recipes-devtools/buildchroot/files/build.sh +++
> b/meta/recipes-devtools/buildchroot/files/build.sh @@ -9,7 +9,7 @@ cd
> $1 # Get list of dependencies manually. The package is not in apt, so
> no apt-get # build-dep. dpkg-checkbuilddeps output contains version
> information and isn't # directly suitable for apt-get install.
> -DEPS=`perl -ne 'next if /^#/; $p=(s/^Build-Depends:\s*/ / or (/^ /
> and $p)); s/,|\n|\([^)]+\)//mg; print if $p' < debian/control`
> +DEPS=`perl -ne 'next if /^#/; $p=(s/^Build-Depends:\s*/ / or (/^ /
> and $p)); s/,|\n|\([^)]+\)|\[[^]]+\]//mg; print if $p' <
> debian/control` # Install deps apt-get install -y $DEPS

Re: [PATCH 1/4] isar: Fix mutli-distro builds in parallel

[Alexander Smirnov]

On 08/08/2017 01:58 PM, Henning Schild wrote:
> Am Tue, 8 Aug 2017 13:04:21 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Extend stamp suffix to distinguish the same distro for various
>> architectures. This fixes parallel build for all possible
>> configurations.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta-isar/recipes-core/images/isar-image-base.bb | 10 +++++-----
>>   meta/classes/dpkg.bbclass                        |  4 ++--
>>   meta/classes/ext4-img.bbclass                    |  2 +-
>>   meta/classes/image.bbclass                       |  2 +-
>>   meta/conf/isar-bitbake.conf                      |  3 +--
>>   meta/recipes-devtools/buildchroot/buildchroot.bb |  2 +-
>>   6 files changed, 11 insertions(+), 12 deletions(-)
>>
>> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
>> b/meta-isar/recipes-core/images/isar-image-base.bb index
>> 337c329..b679d97 100644 ---
>> a/meta-isar/recipes-core/images/isar-image-base.bb +++
>> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -17,11 +17,11
>> @@ DEPENDS += "${IMAGE_INSTALL}" IMAGE_PREINSTALL += "apt \
>>                        dbus"
>>   
>> -WORKDIR = "${TMPDIR}/work/${PN}/${MACHINE}"
>> +WORKDIR = "${TMPDIR}/work/${PN}/${MACHINE}/${DISTRO}"
> 
> This should extend to original WORKDIR where you would also have PV and
> PR included and do not copy its definition.
> 
> WORKDIR =+ "${MACHINE}/${DISTRO}"
> 
> The scheme is different to BUILDCHROOT_DIR where we use
> /${DISTRO}-${DISTRO_ARCH}/ for the directory structure.
> 
> I still think BUILDCHROOT_DIR should be somewhere under the original
> WORKDIR.

That's the main difference between buildchroot and machine stuff:
1. Buildchroot is specified by Debian *distro* and *architecture*.
2. Images/packages are specified by Debian *distro* and *machine*.
3. Machine doesn't equal to architecture. The same buildchroot can be 
used for building package for different machines.

Regarding combining of buildchroot and workdir, I'd suggest to start 
separate thread and discuss benefits/drawback. This patch doesn't 
influence on this topic, it just fixes existing build.

Alex

> 
> Henning
> 
>>   S = "${WORKDIR}/rootfs"
>>   IMAGE_ROOTFS = "${S}"
>>   
>> -do_rootfs[stamp-extra-info] = "${MACHINE}"
>> +do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>>   
>>   do_rootfs() {
>>       install -d -m 755 ${WORKDIR}/hooks_multistrap
>> @@ -38,9 +38,9 @@ do_rootfs() {
>>       sed -i 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|'
>> ${WORKDIR}/multistrap.conf sed -i
>> 's|##DISTRO_SUITE##|${DISTRO_SUITE}|' ${WORKDIR}/multistrap.conf sed
>> -i 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|'
>> ${WORKDIR}/multistrap.conf
>> -    sed -i
>> 's|##CONFIG_SCRIPT##|./tmp/work/${PN}/${MACHINE}/configscript.sh|'
>> ${WORKDIR}/multistrap.conf
>> -    sed -i
>> 's|##SETUP_SCRIPT##|./tmp/work/${PN}/${MACHINE}/setup.sh|'
>> ${WORKDIR}/multistrap.conf
>> -    sed -i
>> 's|##DIR_HOOKS##|./tmp/work/${PN}/${MACHINE}/hooks_multistrap|'
>> ${WORKDIR}/multistrap.conf
>> +    sed -i
>> 's|##CONFIG_SCRIPT##|./tmp/work/${PN}/${MACHINE}/${DISTRO}/configscript.sh|'
>> ${WORKDIR}/multistrap.conf
>> +    sed -i
>> 's|##SETUP_SCRIPT##|./tmp/work/${PN}/${MACHINE}/${DISTRO}/setup.sh|'
>> ${WORKDIR}/multistrap.conf
>> +    sed -i
>> 's|##DIR_HOOKS##|./tmp/work/${PN}/${MACHINE}/${DISTRO}/hooks_multistrap|'
>> ${WORKDIR}/multistrap.conf # Multistrap config use relative paths, so
>> ensure that we are in the right folder cd ${TOPDIR}
>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>> index ea422f9..7c39c68 100644
>> --- a/meta/classes/dpkg.bbclass
>> +++ b/meta/classes/dpkg.bbclass
>> @@ -11,7 +11,7 @@ PP = "/home/builder/${PN}"
>>   BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
>>   
>>   do_unpack[dirs] = "${BUILDROOT}"
>> -do_unpack[stamp-extra-info] = "${DISTRO}"
>> +do_unpack[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   S ?= "${BUILDROOT}"
>>   
>>   # Unpack package and put it into working directory in buildchroot
>> @@ -31,7 +31,7 @@ python do_unpack() {
>>   
>>   addtask unpack after do_fetch before do_build
>>   
>> -do_build[stamp-extra-info] = "${DISTRO}"
>> +do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   
>>   # Build package from sources using build script
>>   do_build() {
>> diff --git a/meta/classes/ext4-img.bbclass
>> b/meta/classes/ext4-img.bbclass index 65d4c11..8588626 100644
>> --- a/meta/classes/ext4-img.bbclass
>> +++ b/meta/classes/ext4-img.bbclass
>> @@ -6,7 +6,7 @@ ROOTFS_EXTRA ?= "64"
>>   
>>   EXT4_IMAGE_FILE =
>> "${DEPLOY_DIR_IMAGE}/${PN}-${MACHINE}-${DISTRO}.ext4.img"
>> -do_ext4_image[stamp-extra-info] = "${MACHINE}"
>> +do_ext4_image[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>>   
>>   # Generate ext4 filesystem image
>>   do_ext4_image() {
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index 3e4877c..a7f0d74 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -9,7 +9,7 @@ IMAGE_TYPE    ?= "ext4-img"
>>   
>>   inherit ${IMAGE_TYPE}
>>   
>> -do_populate[stamp-extra-info] = "${MACHINE}"
>> +do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>>   
>>   # Install Debian packages, that were built from sources
>>   do_populate() {
>> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
>> index ce35a22..f85f5cc 100644
>> --- a/meta/conf/isar-bitbake.conf
>> +++ b/meta/conf/isar-bitbake.conf
>> @@ -18,10 +18,9 @@
>>   # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE
>> OR # OTHER DEALINGS IN THE SOFTWARE.
>>   
>> -
>>   DEPLOY_DIR_DEB = "${TMPDIR}/deploy/deb/${MACHINE}"
>>   SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
>> -BUILDCHROOT_DIR = "${TOPDIR}/tmp/work/buildchroot/${DISTRO}/rootfs"
>> +BUILDCHROOT_DIR =
>> "${TOPDIR}/tmp/work/buildchroot/${DISTRO}-${DISTRO_ARCH}/rootfs"
>>   # Setup our default hash policy
>>   BB_SIGNATURE_HANDLER ?= "noop"
>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
>> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
>> ba1bc66..ccba683 100644 ---
>> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
>> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -23,7 +23,7 @@
>> BUILDCHROOT_PREINSTALL ?= "gcc \
>>   WORKDIR = "${TMPDIR}/work/${PF}/${DISTRO}"
>>   
>> -do_build[stamp-extra-info] = "${DISTRO}"
>> +do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   
>>   do_build() {
>>       install -d -m 755 ${WORKDIR}/hooks_multistrap
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH 1/4] isar: Fix mutli-distro builds in parallel

[Henning Schild]

Am Tue, 8 Aug 2017 14:21:18 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 08/08/2017 01:58 PM, Henning Schild wrote:
> > Am Tue, 8 Aug 2017 13:04:21 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> Extend stamp suffix to distinguish the same distro for various
> >> architectures. This fixes parallel build for all possible
> >> configurations.
> >>
> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >> ---
> >>   meta-isar/recipes-core/images/isar-image-base.bb | 10 +++++-----
> >>   meta/classes/dpkg.bbclass                        |  4 ++--
> >>   meta/classes/ext4-img.bbclass                    |  2 +-
> >>   meta/classes/image.bbclass                       |  2 +-
> >>   meta/conf/isar-bitbake.conf                      |  3 +--
> >>   meta/recipes-devtools/buildchroot/buildchroot.bb |  2 +-
> >>   6 files changed, 11 insertions(+), 12 deletions(-)
> >>
> >> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
> >> b/meta-isar/recipes-core/images/isar-image-base.bb index
> >> 337c329..b679d97 100644 ---
> >> a/meta-isar/recipes-core/images/isar-image-base.bb +++
> >> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -17,11 +17,11
> >> @@ DEPENDS += "${IMAGE_INSTALL}" IMAGE_PREINSTALL += "apt \
> >>                        dbus"
> >>   
> >> -WORKDIR = "${TMPDIR}/work/${PN}/${MACHINE}"
> >> +WORKDIR = "${TMPDIR}/work/${PN}/${MACHINE}/${DISTRO}"  
> > 
> > This should extend to original WORKDIR where you would also have PV
> > and PR included and do not copy its definition.
> > 
> > WORKDIR =+ "${MACHINE}/${DISTRO}"

How about this one?

> > 
> > The scheme is different to BUILDCHROOT_DIR where we use
> > /${DISTRO}-${DISTRO_ARCH}/ for the directory structure.
> > 
> > I still think BUILDCHROOT_DIR should be somewhere under the original
> > WORKDIR.  
> 
> That's the main difference between buildchroot and machine stuff:
> 1. Buildchroot is specified by Debian *distro* and *architecture*.
> 2. Images/packages are specified by Debian *distro* and *machine*.
> 3. Machine doesn't equal to architecture. The same buildchroot can be 
> used for building package for different machines.
> 
> Regarding combining of buildchroot and workdir, I'd suggest to start 
> separate thread and discuss benefits/drawback. This patch doesn't 
> influence on this topic, it just fixes existing build.

Ok.

Henning

> Alex
> 
> > 
> > Henning
> >   
> >>   S = "${WORKDIR}/rootfs"
> >>   IMAGE_ROOTFS = "${S}"
> >>   
> >> -do_rootfs[stamp-extra-info] = "${MACHINE}"
> >> +do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
> >>   
> >>   do_rootfs() {
> >>       install -d -m 755 ${WORKDIR}/hooks_multistrap
> >> @@ -38,9 +38,9 @@ do_rootfs() {
> >>       sed -i 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|'
> >> ${WORKDIR}/multistrap.conf sed -i
> >> 's|##DISTRO_SUITE##|${DISTRO_SUITE}|' ${WORKDIR}/multistrap.conf
> >> sed -i 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|'
> >> ${WORKDIR}/multistrap.conf
> >> -    sed -i
> >> 's|##CONFIG_SCRIPT##|./tmp/work/${PN}/${MACHINE}/configscript.sh|'
> >> ${WORKDIR}/multistrap.conf
> >> -    sed -i
> >> 's|##SETUP_SCRIPT##|./tmp/work/${PN}/${MACHINE}/setup.sh|'
> >> ${WORKDIR}/multistrap.conf
> >> -    sed -i
> >> 's|##DIR_HOOKS##|./tmp/work/${PN}/${MACHINE}/hooks_multistrap|'
> >> ${WORKDIR}/multistrap.conf
> >> +    sed -i
> >> 's|##CONFIG_SCRIPT##|./tmp/work/${PN}/${MACHINE}/${DISTRO}/configscript.sh|'
> >> ${WORKDIR}/multistrap.conf
> >> +    sed -i
> >> 's|##SETUP_SCRIPT##|./tmp/work/${PN}/${MACHINE}/${DISTRO}/setup.sh|'
> >> ${WORKDIR}/multistrap.conf
> >> +    sed -i
> >> 's|##DIR_HOOKS##|./tmp/work/${PN}/${MACHINE}/${DISTRO}/hooks_multistrap|'
> >> ${WORKDIR}/multistrap.conf # Multistrap config use relative paths,
> >> so ensure that we are in the right folder cd ${TOPDIR}
> >> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> >> index ea422f9..7c39c68 100644
> >> --- a/meta/classes/dpkg.bbclass
> >> +++ b/meta/classes/dpkg.bbclass
> >> @@ -11,7 +11,7 @@ PP = "/home/builder/${PN}"
> >>   BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
> >>   
> >>   do_unpack[dirs] = "${BUILDROOT}"
> >> -do_unpack[stamp-extra-info] = "${DISTRO}"
> >> +do_unpack[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> >>   S ?= "${BUILDROOT}"
> >>   
> >>   # Unpack package and put it into working directory in buildchroot
> >> @@ -31,7 +31,7 @@ python do_unpack() {
> >>   
> >>   addtask unpack after do_fetch before do_build
> >>   
> >> -do_build[stamp-extra-info] = "${DISTRO}"
> >> +do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> >>   
> >>   # Build package from sources using build script
> >>   do_build() {
> >> diff --git a/meta/classes/ext4-img.bbclass
> >> b/meta/classes/ext4-img.bbclass index 65d4c11..8588626 100644
> >> --- a/meta/classes/ext4-img.bbclass
> >> +++ b/meta/classes/ext4-img.bbclass
> >> @@ -6,7 +6,7 @@ ROOTFS_EXTRA ?= "64"
> >>   
> >>   EXT4_IMAGE_FILE =
> >> "${DEPLOY_DIR_IMAGE}/${PN}-${MACHINE}-${DISTRO}.ext4.img"
> >> -do_ext4_image[stamp-extra-info] = "${MACHINE}"
> >> +do_ext4_image[stamp-extra-info] = "${MACHINE}-${DISTRO}"
> >>   
> >>   # Generate ext4 filesystem image
> >>   do_ext4_image() {
> >> diff --git a/meta/classes/image.bbclass
> >> b/meta/classes/image.bbclass index 3e4877c..a7f0d74 100644
> >> --- a/meta/classes/image.bbclass
> >> +++ b/meta/classes/image.bbclass
> >> @@ -9,7 +9,7 @@ IMAGE_TYPE    ?= "ext4-img"
> >>   
> >>   inherit ${IMAGE_TYPE}
> >>   
> >> -do_populate[stamp-extra-info] = "${MACHINE}"
> >> +do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}"
> >>   
> >>   # Install Debian packages, that were built from sources
> >>   do_populate() {
> >> diff --git a/meta/conf/isar-bitbake.conf
> >> b/meta/conf/isar-bitbake.conf index ce35a22..f85f5cc 100644
> >> --- a/meta/conf/isar-bitbake.conf
> >> +++ b/meta/conf/isar-bitbake.conf
> >> @@ -18,10 +18,9 @@
> >>   # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
> >> USE OR # OTHER DEALINGS IN THE SOFTWARE.
> >>   
> >> -
> >>   DEPLOY_DIR_DEB = "${TMPDIR}/deploy/deb/${MACHINE}"
> >>   SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
> >> -BUILDCHROOT_DIR =
> >> "${TOPDIR}/tmp/work/buildchroot/${DISTRO}/rootfs" +BUILDCHROOT_DIR
> >> = "${TOPDIR}/tmp/work/buildchroot/${DISTRO}-${DISTRO_ARCH}/rootfs"
> >>   # Setup our default hash policy
> >>   BB_SIGNATURE_HANDLER ?= "noop"
> >> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
> >> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
> >> ba1bc66..ccba683 100644 ---
> >> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
> >> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -23,7 +23,7
> >> @@ BUILDCHROOT_PREINSTALL ?= "gcc \
> >>   WORKDIR = "${TMPDIR}/work/${PF}/${DISTRO}"
> >>   
> >> -do_build[stamp-extra-info] = "${DISTRO}"
> >> +do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> >>   
> >>   do_build() {
> >>       install -d -m 755 ${WORKDIR}/hooks_multistrap  
> >   
> 

Re: [PATCH 1/4] isar: Fix mutli-distro builds in parallel

[Baurzhan Ismagulov]

On Tue, Aug 08, 2017 at 01:04:21PM +0300, Alexander Smirnov wrote:
> Extend stamp suffix to distinguish the same distro for various
> architectures. This fixes parallel build for all possible
> configurations.

Applied with comment amendments, thanks.

With kind regards,
Baurzhan.

Re: [PATCH 3/4] buildchroot: Fix dependency installation in buildchroot

[Baurzhan Ismagulov]

On Tue, Aug 08, 2017 at 01:04:23PM +0300, Alexander Smirnov wrote:
> Use '-y' parameter for apt to resolve command prompt dialogs.

Applied with comment amendments, thanks.

With kind regards,
Baurzhan.

Re: [PATCH 4/4] buildchroot: Update build script.

[Baurzhan Ismagulov]

On Tue, Aug 08, 2017 at 01:04:24PM +0300, Alexander Smirnov wrote:
> Handle the `package [arch]` syntax in Debian's `Build-Depends` field.

Applied with comment amendments, thanks.

With kind regards,
Baurzhan.

Re: [PATCH 4/4] buildchroot: Update build script.

[Baurzhan Ismagulov]

On Tue, Aug 08, 2017 at 01:05:16PM +0200, Henning Schild wrote:
> Looks good, but this whole script should go into dpkg-src.bbclass into
> the task where it gets called. Less indirection, files and confusion.

How exactly?

As I see it, build.sh cannot be executed in the same shell process as the task,
since it requires chrooting into buildchroot (possibly under qemu). The only
way I spontaneously see is to generate it on the fly with e.g. a here document
-- my rating for that particular implementation: More indirection, the same
number of files, more confusion.

With kind regards,
Baurzhan.