[PATCH 0/4 v2] Buildchroot prepare/cleanup

[PATCH 0/4 v2] Buildchroot prepare/cleanup

[Alexander Smirnov]

Hi all,

the goal of this series is to intrioduce 'prepare' and 'cleanup' tasks
for buildchroot. Here they are used to mount/umount git downloads to
buildchroot. In parallel several issue were fixed.

With best regards,
Alex

Alexander Smirnov (4):
  dpkg-base: Fix buildchroot dependancy
  image: Add do_rootfs template to image class
  buildchroot: Add prepare and cleanup tasks
  dpkg-base: Update git alternates

 meta-isar/recipes-core/images/isar-image-base.bb |  3 ---
 meta/classes/dpkg-base.bbclass                   | 10 ++++++++--
 meta/classes/image.bbclass                       |  9 +++++++++
 meta/recipes-devtools/buildchroot/buildchroot.bb | 21 +++++++++++++++++++++
 4 files changed, 38 insertions(+), 5 deletions(-)

-- 
2.9.5

[PATCH 1/4 v2] dpkg-base: Fix buildchroot dependancy

[Alexander Smirnov]

Existing code forces recipe to depends from 'do_build' task for all
the items listed in DEPENDS, what actually makes no sense. This task
only produces binary deb package and it can't satisfy any dependancies
for another task.

Original idea was to add dependancy from 'buildchroot:do_build' task,
so this patch fixes this.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 35af6d5..4941f9b 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -2,8 +2,7 @@
 # Copyright (C) 2017 Siemens AG
 
 # Add dependency from buildchroot creation
-DEPENDS += "buildchroot"
-do_build[deptask] = "do_build"
+do_build[depends] = "buildchroot:do_build"
 
 # Each package should have its own unique build folder, so use
 # recipe name as identifier
-- 
2.9.5

[PATCH 2/4 v2] image: Add do_rootfs template to image class

[Alexander Smirnov]

Add 'do_rootfs' task template to the image class due to it's
mandatory task. Moreover image class already has deps from
'do_rootfs' what makes it inconsistent for now.

Also this patch removes trailing empty line.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar/recipes-core/images/isar-image-base.bb | 3 ---
 meta/classes/image.bbclass                       | 8 ++++++++
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index c2150b1..ad776ed 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -26,7 +26,6 @@ IMAGE_PREINSTALL += "apt \
 
 WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
 
-do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
 do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap"
 
 do_rootfs() {
@@ -70,5 +69,3 @@ do_rootfs() {
     sudo rm "${IMAGE_ROOTFS}/${DISTRO_CONFIG_SCRIPT}"
     _do_rootfs_cleanup
 }
-
-addtask rootfs before do_build after do_populate
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 2c2bafc..7813b16 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -84,6 +84,14 @@ do_populate() {
 addtask populate before do_build after do_unpack
 do_populate[deptask] = "do_deploy_deb"
 
+do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
+
+do_rootfs() {
+    die "No root filesystem function defined, please implement in your recipe"
+}
+
+addtask rootfs before do_build after do_populate
+
 do_copy_boot_files() {
     KERNEL_IMAGE=${@get_image_name(d, 'vmlinuz')}
     if [ -n "${KERNEL_IMAGE}" ]; then
-- 
2.9.5

[PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Alexander Smirnov]

Some packages could require builchroot filesystem tunning, for
example share Isar downloads folder with chroot filesystem.

This patch adds two tasks for buildchroot:

1. 'do_prepare': This task is executed after buildchroot rootfs
generation and before 'do_build' from packages to be built.
Some notes:
 - An individual task is required for the case when you want to rebuild
   some package, this would require to run this preparation stuff for
   existing buildchroot rootfs.
 - This task should not have stamp. This means that anytime you want to
   build something, 'do_prepare' should be executed.

2. 'do_cleanup': This task is executed after all the packages are deployed.
Some notes:
 - This task also should not have stamp.
 - This task depends from the recipes listed in IMAGE_INSTALL.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/dpkg-base.bbclass                   |  2 +-
 meta/classes/image.bbclass                       |  1 +
 meta/recipes-devtools/buildchroot/buildchroot.bb | 21 +++++++++++++++++++++
 3 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 4941f9b..21ae88d 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -2,7 +2,7 @@
 # Copyright (C) 2017 Siemens AG
 
 # Add dependency from buildchroot creation
-do_build[depends] = "buildchroot:do_build"
+do_build[depends] = "buildchroot:do_prepare"
 
 # Each package should have its own unique build folder, so use
 # recipe name as identifier
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 7813b16..b0f645c 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -83,6 +83,7 @@ do_populate() {
 
 addtask populate before do_build after do_unpack
 do_populate[deptask] = "do_deploy_deb"
+do_populate[depends] = "buildchroot:do_cleanup"
 
 do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
 
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 6a94733..fe88e98 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -67,7 +67,28 @@ do_build() {
     # Install package builder script
     sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
 
+    # Create share point for downloads
+    sudo install -d ${BUILDCHROOT_DIR}/git
+
     # Configure root filesystem
     sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
     _do_build_cleanup
 }
+
+do_prepare[nostamp] = "1"
+
+do_prepare() {
+    sudo mount --bind ${GITDIR} ${BUILDCHROOT_DIR}/git
+}
+
+addtask prepare after do_build
+
+DEPENDS += "${IMAGE_INSTALL}"
+do_cleanup[deptask] = "do_deploy_deb"
+do_cleanup[nostamp] = "1"
+
+do_cleanup() {
+    sudo umount ${BUILDCHROOT_DIR}/git
+}
+
+addtask cleanup after do_prepare
-- 
2.9.5

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Henning Schild]

Will the cleanup be run if a build fails? I can not see code that takes
care of the umount in that case.

Henning

Am Thu, 23 Nov 2017 13:25:31 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Some packages could require builchroot filesystem tunning, for
> example share Isar downloads folder with chroot filesystem.
> 
> This patch adds two tasks for buildchroot:
> 
> 1. 'do_prepare': This task is executed after buildchroot rootfs
> generation and before 'do_build' from packages to be built.
> Some notes:
>  - An individual task is required for the case when you want to
> rebuild some package, this would require to run this preparation
> stuff for existing buildchroot rootfs.
>  - This task should not have stamp. This means that anytime you want
> to build something, 'do_prepare' should be executed.
> 
> 2. 'do_cleanup': This task is executed after all the packages are
> deployed. Some notes:
>  - This task also should not have stamp.
>  - This task depends from the recipes listed in IMAGE_INSTALL.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/classes/dpkg-base.bbclass                   |  2 +-
>  meta/classes/image.bbclass                       |  1 +
>  meta/recipes-devtools/buildchroot/buildchroot.bb | 21
> +++++++++++++++++++++ 3 files changed, 23 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/dpkg-base.bbclass
> b/meta/classes/dpkg-base.bbclass index 4941f9b..21ae88d 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -2,7 +2,7 @@
>  # Copyright (C) 2017 Siemens AG
>  
>  # Add dependency from buildchroot creation
> -do_build[depends] = "buildchroot:do_build"
> +do_build[depends] = "buildchroot:do_prepare"
>  
>  # Each package should have its own unique build folder, so use
>  # recipe name as identifier
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 7813b16..b0f645c 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -83,6 +83,7 @@ do_populate() {
>  
>  addtask populate before do_build after do_unpack
>  do_populate[deptask] = "do_deploy_deb"
> +do_populate[depends] = "buildchroot:do_cleanup"
>  
>  do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>  
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
> 6a94733..fe88e98 100644 ---
> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -67,7 +67,28 @@
> do_build() { # Install package builder script
>      sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
>  
> +    # Create share point for downloads
> +    sudo install -d ${BUILDCHROOT_DIR}/git
> +
>      # Configure root filesystem
>      sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>      _do_build_cleanup
>  }
> +
> +do_prepare[nostamp] = "1"
> +
> +do_prepare() {
> +    sudo mount --bind ${GITDIR} ${BUILDCHROOT_DIR}/git
> +}
> +
> +addtask prepare after do_build
> +
> +DEPENDS += "${IMAGE_INSTALL}"
> +do_cleanup[deptask] = "do_deploy_deb"
> +do_cleanup[nostamp] = "1"
> +
> +do_cleanup() {
> +    sudo umount ${BUILDCHROOT_DIR}/git
> +}
> +
> +addtask cleanup after do_prepare

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Alexander Smirnov]

On 11/24/2017 05:55 PM, Henning Schild wrote:
> Will the cleanup be run if a build fails? I can not see code that takes
> care of the umount in that case.
> 

Christian promised to think about general approach for clean up hooks, 
so I assumed to use it for clean up. But for now I could add some clean 
ups to dpkg-base class.

Alex

> Henning
> 
> Am Thu, 23 Nov 2017 13:25:31 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Some packages could require builchroot filesystem tunning, for
>> example share Isar downloads folder with chroot filesystem.
>>
>> This patch adds two tasks for buildchroot:
>>
>> 1. 'do_prepare': This task is executed after buildchroot rootfs
>> generation and before 'do_build' from packages to be built.
>> Some notes:
>>   - An individual task is required for the case when you want to
>> rebuild some package, this would require to run this preparation
>> stuff for existing buildchroot rootfs.
>>   - This task should not have stamp. This means that anytime you want
>> to build something, 'do_prepare' should be executed.
>>
>> 2. 'do_cleanup': This task is executed after all the packages are
>> deployed. Some notes:
>>   - This task also should not have stamp.
>>   - This task depends from the recipes listed in IMAGE_INSTALL.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta/classes/dpkg-base.bbclass                   |  2 +-
>>   meta/classes/image.bbclass                       |  1 +
>>   meta/recipes-devtools/buildchroot/buildchroot.bb | 21
>> +++++++++++++++++++++ 3 files changed, 23 insertions(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/dpkg-base.bbclass
>> b/meta/classes/dpkg-base.bbclass index 4941f9b..21ae88d 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -2,7 +2,7 @@
>>   # Copyright (C) 2017 Siemens AG
>>   
>>   # Add dependency from buildchroot creation
>> -do_build[depends] = "buildchroot:do_build"
>> +do_build[depends] = "buildchroot:do_prepare"
>>   
>>   # Each package should have its own unique build folder, so use
>>   # recipe name as identifier
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index 7813b16..b0f645c 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -83,6 +83,7 @@ do_populate() {
>>   
>>   addtask populate before do_build after do_unpack
>>   do_populate[deptask] = "do_deploy_deb"
>> +do_populate[depends] = "buildchroot:do_cleanup"
>>   
>>   do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>>   
>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
>> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
>> 6a94733..fe88e98 100644 ---
>> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
>> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -67,7 +67,28 @@
>> do_build() { # Install package builder script
>>       sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
>>   
>> +    # Create share point for downloads
>> +    sudo install -d ${BUILDCHROOT_DIR}/git
>> +
>>       # Configure root filesystem
>>       sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>>       _do_build_cleanup
>>   }
>> +
>> +do_prepare[nostamp] = "1"
>> +
>> +do_prepare() {
>> +    sudo mount --bind ${GITDIR} ${BUILDCHROOT_DIR}/git
>> +}
>> +
>> +addtask prepare after do_build
>> +
>> +DEPENDS += "${IMAGE_INSTALL}"
>> +do_cleanup[deptask] = "do_deploy_deb"
>> +do_cleanup[nostamp] = "1"
>> +
>> +do_cleanup() {
>> +    sudo umount ${BUILDCHROOT_DIR}/git
>> +}
>> +
>> +addtask cleanup after do_prepare
>

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Henning Schild]

Am Thu, 23 Nov 2017 13:25:31 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Some packages could require builchroot filesystem tunning, for
> example share Isar downloads folder with chroot filesystem.
> 
> This patch adds two tasks for buildchroot:
> 
> 1. 'do_prepare': This task is executed after buildchroot rootfs
> generation and before 'do_build' from packages to be built.
> Some notes:
>  - An individual task is required for the case when you want to
> rebuild some package, this would require to run this preparation
> stuff for existing buildchroot rootfs.
>  - This task should not have stamp. This means that anytime you want
> to build something, 'do_prepare' should be executed.
> 
> 2. 'do_cleanup': This task is executed after all the packages are
> deployed. Some notes:
>  - This task also should not have stamp.
>  - This task depends from the recipes listed in IMAGE_INSTALL.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/classes/dpkg-base.bbclass                   |  2 +-
>  meta/classes/image.bbclass                       |  1 +
>  meta/recipes-devtools/buildchroot/buildchroot.bb | 21
> +++++++++++++++++++++ 3 files changed, 23 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/dpkg-base.bbclass
> b/meta/classes/dpkg-base.bbclass index 4941f9b..21ae88d 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -2,7 +2,7 @@
>  # Copyright (C) 2017 Siemens AG
>  
>  # Add dependency from buildchroot creation
> -do_build[depends] = "buildchroot:do_build"
> +do_build[depends] = "buildchroot:do_prepare"
>  
>  # Each package should have its own unique build folder, so use
>  # recipe name as identifier
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 7813b16..b0f645c 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -83,6 +83,7 @@ do_populate() {
>  
>  addtask populate before do_build after do_unpack
>  do_populate[deptask] = "do_deploy_deb"
> +do_populate[depends] = "buildchroot:do_cleanup"

Why do we need that? To me that looks redundant.

Henning

>  do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>  
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
> 6a94733..fe88e98 100644 ---
> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -67,7 +67,28 @@
> do_build() { # Install package builder script
>      sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
>  
> +    # Create share point for downloads
> +    sudo install -d ${BUILDCHROOT_DIR}/git
> +
>      # Configure root filesystem
>      sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>      _do_build_cleanup
>  }
> +
> +do_prepare[nostamp] = "1"
> +
> +do_prepare() {
> +    sudo mount --bind ${GITDIR} ${BUILDCHROOT_DIR}/git
> +}
> +
> +addtask prepare after do_build
> +
> +DEPENDS += "${IMAGE_INSTALL}"
> +do_cleanup[deptask] = "do_deploy_deb"
> +do_cleanup[nostamp] = "1"
> +
> +do_cleanup() {
> +    sudo umount ${BUILDCHROOT_DIR}/git
> +}
> +
> +addtask cleanup after do_prepare

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Alexander Smirnov]

On 11/24/2017 07:34 PM, Henning Schild wrote:
> Am Thu, 23 Nov 2017 13:25:31 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Some packages could require builchroot filesystem tunning, for
>> example share Isar downloads folder with chroot filesystem.
>>
>> This patch adds two tasks for buildchroot:
>>
>> 1. 'do_prepare': This task is executed after buildchroot rootfs
>> generation and before 'do_build' from packages to be built.
>> Some notes:
>>   - An individual task is required for the case when you want to
>> rebuild some package, this would require to run this preparation
>> stuff for existing buildchroot rootfs.
>>   - This task should not have stamp. This means that anytime you want
>> to build something, 'do_prepare' should be executed.
>>
>> 2. 'do_cleanup': This task is executed after all the packages are
>> deployed. Some notes:
>>   - This task also should not have stamp.
>>   - This task depends from the recipes listed in IMAGE_INSTALL.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta/classes/dpkg-base.bbclass                   |  2 +-
>>   meta/classes/image.bbclass                       |  1 +
>>   meta/recipes-devtools/buildchroot/buildchroot.bb | 21
>> +++++++++++++++++++++ 3 files changed, 23 insertions(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/dpkg-base.bbclass
>> b/meta/classes/dpkg-base.bbclass index 4941f9b..21ae88d 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -2,7 +2,7 @@
>>   # Copyright (C) 2017 Siemens AG
>>   
>>   # Add dependency from buildchroot creation
>> -do_build[depends] = "buildchroot:do_build"
>> +do_build[depends] = "buildchroot:do_prepare"
>>   
>>   # Each package should have its own unique build folder, so use
>>   # recipe name as identifier
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index 7813b16..b0f645c 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -83,6 +83,7 @@ do_populate() {
>>   
>>   addtask populate before do_build after do_unpack
>>   do_populate[deptask] = "do_deploy_deb"
>> +do_populate[depends] = "buildchroot:do_cleanup"
> 
> Why do we need that? To me that looks redundant.

1. buildchroot:do_cleanup task should be executed after all the packages 
are build, so that's why the following is used:

   DEPENDS += "${IMAGE_INSTALL}"
   do_cleanup[deptask] = "do_deploy_deb"

2. buildchroot:do_cleanup should be somehow integrated to Isar pipeline, 
so some dependency from late task (which runs after packages building) 
should be added. I chose image:do_populate, so the following line forces 
Isar to run do_cleanup during build:

   do_populate[depends] = "buildchroot:do_cleanup"

Alex

> 
> Henning
> 
>>   do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>>   
>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
>> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
>> 6a94733..fe88e98 100644 ---
>> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
>> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -67,7 +67,28 @@
>> do_build() { # Install package builder script
>>       sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
>>   
>> +    # Create share point for downloads
>> +    sudo install -d ${BUILDCHROOT_DIR}/git
>> +
>>       # Configure root filesystem
>>       sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>>       _do_build_cleanup
>>   }
>> +
>> +do_prepare[nostamp] = "1"
>> +
>> +do_prepare() {
>> +    sudo mount --bind ${GITDIR} ${BUILDCHROOT_DIR}/git
>> +}
>> +
>> +addtask prepare after do_build
>> +
>> +DEPENDS += "${IMAGE_INSTALL}"
>> +do_cleanup[deptask] = "do_deploy_deb"
>> +do_cleanup[nostamp] = "1"
>> +
>> +do_cleanup() {
>> +    sudo umount ${BUILDCHROOT_DIR}/git
>> +}
>> +
>> +addtask cleanup after do_prepare
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Henning Schild]

Am Fri, 24 Nov 2017 19:51:26 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 11/24/2017 07:34 PM, Henning Schild wrote:
> > Am Thu, 23 Nov 2017 13:25:31 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> Some packages could require builchroot filesystem tunning, for
> >> example share Isar downloads folder with chroot filesystem.
> >>
> >> This patch adds two tasks for buildchroot:
> >>
> >> 1. 'do_prepare': This task is executed after buildchroot rootfs
> >> generation and before 'do_build' from packages to be built.
> >> Some notes:
> >>   - An individual task is required for the case when you want to
> >> rebuild some package, this would require to run this preparation
> >> stuff for existing buildchroot rootfs.
> >>   - This task should not have stamp. This means that anytime you
> >> want to build something, 'do_prepare' should be executed.
> >>
> >> 2. 'do_cleanup': This task is executed after all the packages are
> >> deployed. Some notes:
> >>   - This task also should not have stamp.
> >>   - This task depends from the recipes listed in IMAGE_INSTALL.
> >>
> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >> ---
> >>   meta/classes/dpkg-base.bbclass                   |  2 +-
> >>   meta/classes/image.bbclass                       |  1 +
> >>   meta/recipes-devtools/buildchroot/buildchroot.bb | 21
> >> +++++++++++++++++++++ 3 files changed, 23 insertions(+), 1
> >> deletion(-)
> >>
> >> diff --git a/meta/classes/dpkg-base.bbclass
> >> b/meta/classes/dpkg-base.bbclass index 4941f9b..21ae88d 100644
> >> --- a/meta/classes/dpkg-base.bbclass
> >> +++ b/meta/classes/dpkg-base.bbclass
> >> @@ -2,7 +2,7 @@
> >>   # Copyright (C) 2017 Siemens AG
> >>   
> >>   # Add dependency from buildchroot creation
> >> -do_build[depends] = "buildchroot:do_build"
> >> +do_build[depends] = "buildchroot:do_prepare"
> >>   
> >>   # Each package should have its own unique build folder, so use
> >>   # recipe name as identifier
> >> diff --git a/meta/classes/image.bbclass
> >> b/meta/classes/image.bbclass index 7813b16..b0f645c 100644
> >> --- a/meta/classes/image.bbclass
> >> +++ b/meta/classes/image.bbclass
> >> @@ -83,6 +83,7 @@ do_populate() {
> >>   
> >>   addtask populate before do_build after do_unpack
> >>   do_populate[deptask] = "do_deploy_deb"
> >> +do_populate[depends] = "buildchroot:do_cleanup"  
> > 
> > Why do we need that? To me that looks redundant.  
> 
> 1. buildchroot:do_cleanup task should be executed after all the
> packages are build, so that's why the following is used:
> 
>    DEPENDS += "${IMAGE_INSTALL}"
>    do_cleanup[deptask] = "do_deploy_deb"

That one is clear.

> 2. buildchroot:do_cleanup should be somehow integrated to Isar
> pipeline, so some dependency from late task (which runs after
> packages building) should be added. I chose image:do_populate, so the
> following line forces Isar to run do_cleanup during build:
> 
>    do_populate[depends] = "buildchroot:do_cleanup"

If you do not depend on it in the image-recipe, will it not be executed?
Please remember that we discussed dropping do_populate, so it might be
a good idea to integrate that step somewhere else. Maybe give the image
its own do_cleanup.

https://groups.google.com/d/msg/isar-users/efer3RF989o/mB5Fce9WBgAJ

Henning

> Alex
> 
> > 
> > Henning
> >   
> >>   do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
> >>   
> >> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
> >> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
> >> 6a94733..fe88e98 100644 ---
> >> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
> >> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -67,7 +67,28
> >> @@ do_build() { # Install package builder script
> >>       sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
> >>   
> >> +    # Create share point for downloads
> >> +    sudo install -d ${BUILDCHROOT_DIR}/git
> >> +
> >>       # Configure root filesystem
> >>       sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
> >>       _do_build_cleanup
> >>   }
> >> +
> >> +do_prepare[nostamp] = "1"
> >> +
> >> +do_prepare() {
> >> +    sudo mount --bind ${GITDIR} ${BUILDCHROOT_DIR}/git
> >> +}
> >> +
> >> +addtask prepare after do_build
> >> +
> >> +DEPENDS += "${IMAGE_INSTALL}"
> >> +do_cleanup[deptask] = "do_deploy_deb"
> >> +do_cleanup[nostamp] = "1"
> >> +
> >> +do_cleanup() {
> >> +    sudo umount ${BUILDCHROOT_DIR}/git
> >> +}
> >> +
> >> +addtask cleanup after do_prepare  
> >   
> 

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Alexander Smirnov]

On 11/27/2017 10:56 AM, Henning Schild wrote:
> Am Fri, 24 Nov 2017 19:51:26 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 11/24/2017 07:34 PM, Henning Schild wrote:
>>> Am Thu, 23 Nov 2017 13:25:31 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>    
>>>> Some packages could require builchroot filesystem tunning, for
>>>> example share Isar downloads folder with chroot filesystem.
>>>>
>>>> This patch adds two tasks for buildchroot:
>>>>
>>>> 1. 'do_prepare': This task is executed after buildchroot rootfs
>>>> generation and before 'do_build' from packages to be built.
>>>> Some notes:
>>>>    - An individual task is required for the case when you want to
>>>> rebuild some package, this would require to run this preparation
>>>> stuff for existing buildchroot rootfs.
>>>>    - This task should not have stamp. This means that anytime you
>>>> want to build something, 'do_prepare' should be executed.
>>>>
>>>> 2. 'do_cleanup': This task is executed after all the packages are
>>>> deployed. Some notes:
>>>>    - This task also should not have stamp.
>>>>    - This task depends from the recipes listed in IMAGE_INSTALL.
>>>>
>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>> ---
>>>>    meta/classes/dpkg-base.bbclass                   |  2 +-
>>>>    meta/classes/image.bbclass                       |  1 +
>>>>    meta/recipes-devtools/buildchroot/buildchroot.bb | 21
>>>> +++++++++++++++++++++ 3 files changed, 23 insertions(+), 1
>>>> deletion(-)
>>>>
>>>> diff --git a/meta/classes/dpkg-base.bbclass
>>>> b/meta/classes/dpkg-base.bbclass index 4941f9b..21ae88d 100644
>>>> --- a/meta/classes/dpkg-base.bbclass
>>>> +++ b/meta/classes/dpkg-base.bbclass
>>>> @@ -2,7 +2,7 @@
>>>>    # Copyright (C) 2017 Siemens AG
>>>>    
>>>>    # Add dependency from buildchroot creation
>>>> -do_build[depends] = "buildchroot:do_build"
>>>> +do_build[depends] = "buildchroot:do_prepare"
>>>>    
>>>>    # Each package should have its own unique build folder, so use
>>>>    # recipe name as identifier
>>>> diff --git a/meta/classes/image.bbclass
>>>> b/meta/classes/image.bbclass index 7813b16..b0f645c 100644
>>>> --- a/meta/classes/image.bbclass
>>>> +++ b/meta/classes/image.bbclass
>>>> @@ -83,6 +83,7 @@ do_populate() {
>>>>    
>>>>    addtask populate before do_build after do_unpack
>>>>    do_populate[deptask] = "do_deploy_deb"
>>>> +do_populate[depends] = "buildchroot:do_cleanup"
>>>
>>> Why do we need that? To me that looks redundant.
>>
>> 1. buildchroot:do_cleanup task should be executed after all the
>> packages are build, so that's why the following is used:
>>
>>     DEPENDS += "${IMAGE_INSTALL}"
>>     do_cleanup[deptask] = "do_deploy_deb"
> 
> That one is clear.
> 
>> 2. buildchroot:do_cleanup should be somehow integrated to Isar
>> pipeline, so some dependency from late task (which runs after
>> packages building) should be added. I chose image:do_populate, so the
>> following line forces Isar to run do_cleanup during build:
>>
>>     do_populate[depends] = "buildchroot:do_cleanup"
> 
> If you do not depend on it in the image-recipe, will it not be executed?

It will not, buildchroot is joint to pipeline only by 'do_build' task, 
so all the tasks after are silently ignored. For sure, I've tested this.

> Please remember that we discussed dropping do_populate, so it might be
> a good idea to integrate that step somewhere else. Maybe give the image
> its own do_cleanup.
> 
> https://groups.google.com/d/msg/isar-users/efer3RF989o/mB5Fce9WBgAJ
> 

I remember this discussion, but at the moment I have no clear answer 
whether do_populate will go or not.

The concept that was discussed contains the following steps:

1. Buildchroot generation.
2. Building all the packages and put them to apt.
3. Provide this apt as the input for image's multistrap.

Thinking on this I've realized that this approach has one issue - it 
serializes buildchroot and image generation, so significantly increases 
Isar build time. Possible solution could be:

1. Buildchroot and base image rootfs generation.
2. Building all the packages and put them to apt.
3. Populate image rootfs via apt-get using Isar apt.

I'd like to discuss this in more details when I return to apt topic. So 
for now I'd rely on existing pipeline.

Alex

> Henning
> 
>> Alex
>>
>>>
>>> Henning
>>>    
>>>>    do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>>>>    
>>>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
>>>> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
>>>> 6a94733..fe88e98 100644 ---
>>>> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
>>>> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -67,7 +67,28
>>>> @@ do_build() { # Install package builder script
>>>>        sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
>>>>    
>>>> +    # Create share point for downloads
>>>> +    sudo install -d ${BUILDCHROOT_DIR}/git
>>>> +
>>>>        # Configure root filesystem
>>>>        sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>>>>        _do_build_cleanup
>>>>    }
>>>> +
>>>> +do_prepare[nostamp] = "1"
>>>> +
>>>> +do_prepare() {
>>>> +    sudo mount --bind ${GITDIR} ${BUILDCHROOT_DIR}/git
>>>> +}
>>>> +
>>>> +addtask prepare after do_build
>>>> +
>>>> +DEPENDS += "${IMAGE_INSTALL}"
>>>> +do_cleanup[deptask] = "do_deploy_deb"
>>>> +do_cleanup[nostamp] = "1"
>>>> +
>>>> +do_cleanup() {
>>>> +    sudo umount ${BUILDCHROOT_DIR}/git
>>>> +}
>>>> +
>>>> +addtask cleanup after do_prepare

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Christian Storm]

> 2. 'do_cleanup': This task is executed after all the packages are deployed.
> Some notes:
>  - This task also should not have stamp.
>  - This task depends from the recipes listed in IMAGE_INSTALL.

This breaks build dependencies when building two packages from source
(same example I've given in other posts as well):

Consider A-dev required by B, both to be built from source. B cannot
install the A-dev.deb since B depending on A-dev's deploy_deb() and
installing it the ugly way via dpkg -i (as no other option is currently
available in Isar) puts B as late in the task order so that it's git
repo is not mounted anymore. Failure.

If A-dev installs it's produced .deb as last step in its dpkg_runbuild(),
which is also ugly, then there's a potential race in dpkg. Failure.


   Christian

-- 
Dr. Christian Storm
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Otto-Hahn-Ring 6, 81739 M�nchen, Germany

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Alexander Smirnov]

Hi Christian,

On 12/08/2017 10:58 AM, Christian Storm wrote:
> 
>> 2. 'do_cleanup': This task is executed after all the packages are deployed.
>> Some notes:
>>   - This task also should not have stamp.
>>   - This task depends from the recipes listed in IMAGE_INSTALL.
> 
> This breaks build dependencies when building two packages from source
> (same example I've given in other posts as well):
> 
> Consider A-dev required by B, both to be built from source. B cannot
> install the A-dev.deb since B depending on A-dev's deploy_deb() and
> installing it the ugly way via dpkg -i (as no other option is currently
> available in Isar) puts B as late in the task order so that it's git
> repo is not mounted anymore. Failure.
> 
> If A-dev installs it's produced .deb as last step in its dpkg_runbuild(),
> which is also ugly, then there's a potential race in dpkg. Failure.

Seems I don't fully understand your usecase, could you please comment if 
it's correct:

1. You have two Isar recipes A and B that build packages from source code.

2. After building of package A, apart from binary A, A-dev also provided.

3. Package B depends on A-dev.

Also, have you added dependency from A in B recipe?

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Henning Schild]

Am Mon, 11 Dec 2017 12:23:40 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Hi Christian,
> 
> On 12/08/2017 10:58 AM, Christian Storm wrote:
> >   
> >> 2. 'do_cleanup': This task is executed after all the packages are
> >> deployed. Some notes:
> >>   - This task also should not have stamp.
> >>   - This task depends from the recipes listed in IMAGE_INSTALL.  
> > 
> > This breaks build dependencies when building two packages from
> > source (same example I've given in other posts as well):
> > 
> > Consider A-dev required by B, both to be built from source. B cannot
> > install the A-dev.deb since B depending on A-dev's deploy_deb() and
> > installing it the ugly way via dpkg -i (as no other option is
> > currently available in Isar) puts B as late in the task order so
> > that it's git repo is not mounted anymore. Failure.
> > 
> > If A-dev installs it's produced .deb as last step in its
> > dpkg_runbuild(), which is also ugly, then there's a potential race
> > in dpkg. Failure.  
> 
> Seems I don't fully understand your usecase, could you please comment
> if it's correct:
> 
> 1. You have two Isar recipes A and B that build packages from source
> code.
> 
> 2. After building of package A, apart from binary A, A-dev also
> provided.
> 
> 3. Package B depends on A-dev.
> 
> Also, have you added dependency from A in B recipe?

Dependencies will not help here, Isar simply does not support that
case ... yet.
You have B depending on A both at runtime and on build-time. So you need
a buildchroot to build A in. And for B you need a buildchroot which has
A installed.

To get the buildchroot for B you either have to install A into its own
buildchroot or multistrap again.

I think Christian currently installs A, but with "dpkg -i". I think
multistrapping again would be overkill. IMHO the best solution would be
to install such packages in buildchroot with apt-get, after they have
been repreproed to the "Isar" repo.

I suggested improvements on "[PATCH 0/4 v5] Isar apt deployment" but
the feedback was delayed until a v2.

Essentially deploy_deb needs to reprepro, do_populate needs to go away,
builchroot needs aptsources=Isar in its multistrap.conf. Now the
apt-get in build.sh should work.

Henning

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Storm, Christian (CT RDA ITP SES-DE)]

> > >> 2. 'do_cleanup': This task is executed after all the packages are
> > >> deployed. Some notes:
> > >>   - This task also should not have stamp.
> > >>   - This task depends from the recipes listed in IMAGE_INSTALL.  
> > > 
> > > This breaks build dependencies when building two packages from
> > > source (same example I've given in other posts as well):
> > > 
> > > Consider A-dev required by B, both to be built from source. B cannot
> > > install the A-dev.deb since B depending on A-dev's deploy_deb() and
> > > installing it the ugly way via dpkg -i (as no other option is
> > > currently available in Isar) puts B as late in the task order so
> > > that it's git repo is not mounted anymore. Failure.
> > > 
> > > If A-dev installs it's produced .deb as last step in its
> > > dpkg_runbuild(), which is also ugly, then there's a potential race
> > > in dpkg. Failure.  
> > 
> > Seems I don't fully understand your usecase, could you please comment
> > if it's correct:
> > 
> > 1. You have two Isar recipes A and B that build packages from source
> > code.
> > 
> > 2. After building of package A, apart from binary A, A-dev also
> > provided.
> > 
> > 3. Package B depends on A-dev.
> > 
> > Also, have you added dependency from A in B recipe?
> 
> Dependencies will not help here, Isar simply does not support that
> case ... yet.
> You have B depending on A both at runtime and on build-time. So you need
> a buildchroot to build A in. And for B you need a buildchroot which has
> A installed.
> 
> To get the buildchroot for B you either have to install A into its own
> buildchroot or multistrap again.
> 
> I think Christian currently installs A, but with "dpkg -i". I think
> multistrapping again would be overkill. IMHO the best solution would be
> to install such packages in buildchroot with apt-get, after they have
> been repreproed to the "Isar" repo.

Yes, right, ugly, I know, but currently the only way to reflect my use
case -- which Henning described very well -- in Isar. Without this hack
or preferably a proper solution to this, I cannot use Isar for my use
case. Since this change, in fact, I can't.

So, in some sense I relied on a workaround for a missing feature that's
now broken and hence you may rightfully argue that nothing can break a
missing feature. Nonetheless, it did so for me and my hack :)


> I suggested improvements on "[PATCH 0/4 v5] Isar apt deployment" but
> the feedback was delayed until a v2.
> 
> Essentially deploy_deb needs to reprepro, do_populate needs to go away,
> builchroot needs aptsources=Isar in its multistrap.conf. Now the
> apt-get in build.sh should work.

I very much appreciate a proper solution to this to resume using Isar
for my use case.


   Christian

-- 
Dr. Christian Storm
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Otto-Hahn-Ring 6, 81739 M�nchen, Germany

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Alexander Smirnov]

On 12/11/2017 04:37 PM, Storm, Christian (CT RDA ITP SES-DE) wrote:
>>>>> 2. 'do_cleanup': This task is executed after all the packages are
>>>>> deployed. Some notes:
>>>>>    - This task also should not have stamp.
>>>>>    - This task depends from the recipes listed in IMAGE_INSTALL.
>>>>
>>>> This breaks build dependencies when building two packages from
>>>> source (same example I've given in other posts as well):
>>>>
>>>> Consider A-dev required by B, both to be built from source. B cannot
>>>> install the A-dev.deb since B depending on A-dev's deploy_deb() and
>>>> installing it the ugly way via dpkg -i (as no other option is
>>>> currently available in Isar) puts B as late in the task order so
>>>> that it's git repo is not mounted anymore. Failure.
>>>>
>>>> If A-dev installs it's produced .deb as last step in its
>>>> dpkg_runbuild(), which is also ugly, then there's a potential race
>>>> in dpkg. Failure.
>>>
>>> Seems I don't fully understand your usecase, could you please comment
>>> if it's correct:
>>>
>>> 1. You have two Isar recipes A and B that build packages from source
>>> code.
>>>
>>> 2. After building of package A, apart from binary A, A-dev also
>>> provided.
>>>
>>> 3. Package B depends on A-dev.
>>>
>>> Also, have you added dependency from A in B recipe?
>>
>> Dependencies will not help here, Isar simply does not support that
>> case ... yet.
>> You have B depending on A both at runtime and on build-time. So you need
>> a buildchroot to build A in. And for B you need a buildchroot which has
>> A installed.
>>
>> To get the buildchroot for B you either have to install A into its own
>> buildchroot or multistrap again.
>>
>> I think Christian currently installs A, but with "dpkg -i". I think
>> multistrapping again would be overkill. IMHO the best solution would be
>> to install such packages in buildchroot with apt-get, after they have
>> been repreproed to the "Isar" repo.
> 
> Yes, right, ugly, I know, but currently the only way to reflect my use
> case -- which Henning described very well -- in Isar. Without this hack
> or preferably a proper solution to this, I cannot use Isar for my use
> case. Since this change, in fact, I can't.
> 

Sorry, but I still don't understand how this patch brakes your build 
because it doesn't influence on dependencies at all. If you need strict 
order for packages to be built, then anyway you have to specify this in 
recipe:

In recipe B:

DEPENDS += "A"
do_build[deptask] = "do_build"

It would be very helpful for me if you could provide recipes and workaround.

> So, in some sense I relied on a workaround for a missing feature that's
> now broken and hence you may rightfully argue that nothing can break a
> missing feature. Nonetheless, it did so for me and my hack :)
> 
> 
>> I suggested improvements on "[PATCH 0/4 v5] Isar apt deployment" but
>> the feedback was delayed until a v2.
>>
>> Essentially deploy_deb needs to reprepro, do_populate needs to go away,
>> builchroot needs aptsources=Isar in its multistrap.conf. Now the
>> apt-get in build.sh should work.
> 
> I very much appreciate a proper solution to this to resume using Isar
> for my use case.

Could handle this after your top-prio issues.

Alex

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Christian Storm]

> Sorry, but I still don't understand how this patch brakes your build 
> because it doesn't influence on dependencies at all. 
> If you need strict order for packages to be built, then anyway you
> have to specify this in recipe:
> 
> In recipe B:
> 
> DEPENDS += "A"
> do_build[deptask] = "do_build"
> 
> It would be very helpful for me if you could provide recipes and workaround.


OK, here's one example, anonymized, that used to work:

```meta-mylayer/recipes-core/A/A.bb
...
SRC_URI = "git://..."
SRC_DIR = "git"

PROVIDES = "${PN}-dev"
inherit dpkg

dpkg_runbuild() {
  ...
  sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
  ...
}
```

This is a pretty standard recipe whose only purpose is to build two .deb
packages, one containing the binaries, one containing build-time
dependencies for B.bb below:
The resulting A-dev.deb package then contains, e.g., 
  /usr/lib/x86_64-linux-gnu/libA.a
and this is used as build-time dependency from B. Consider 
  /usr/lib/x86_64-linux-gnu/libA.a
to be linked into the software binary compiled by B.B.

```meta-mylayer/recipes-core/B/B.bb
...
SRC_URI = "git://..."
SRC_DIR = "git"

DEPENDS = "A-dev"
inherit dpkg

dpkg_runbuild() {
  ...
  ## UGLY hack to install build-time dependency A-dev required for the 
  ## software to be compiled here
  for item in $(echo ${DEPENDS} | sed 's/buildchroot//g'); do
    sudo dpkg --root=${BUILDCHROOT_DIR} -i ${DEPLOY_DIR_DEB}/${item}_*
  done

  sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
  ...
}
```

This does not work on current -next. Even if you do the 
  do_build[deptask] = "do_build"
because do_cleanup() kicks in and unmounts ${BUILDCHROOT_DIR}/git
and the task reordering of deploy_deb().


Hope this example helps?



   Christian

-- 
Dr. Christian Storm
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Otto-Hahn-Ring 6, 81739 M�nchen, Germany

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Alexander Smirnov]

Hi,

On 12/11/2017 06:14 PM, Christian Storm wrote:
>> Sorry, but I still don't understand how this patch brakes your build
>> because it doesn't influence on dependencies at all.
>> If you need strict order for packages to be built, then anyway you
>> have to specify this in recipe:
>>
>> In recipe B:
>>
>> DEPENDS += "A"
>> do_build[deptask] = "do_build"
>>
>> It would be very helpful for me if you could provide recipes and workaround.
> 
> 
> OK, here's one example, anonymized, that used to work:
> 
> ```meta-mylayer/recipes-core/A/A.bb
> ...
> SRC_URI = "git://..."
> SRC_DIR = "git"
> 
> PROVIDES = "${PN}-dev"
> inherit dpkg
> 
> dpkg_runbuild() {
>    ...
>    sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
>    ...
> }
> ```
> 
> This is a pretty standard recipe whose only purpose is to build two .deb
> packages, one containing the binaries, one containing build-time
> dependencies for B.bb below:
> The resulting A-dev.deb package then contains, e.g.,
>    /usr/lib/x86_64-linux-gnu/libA.a
> and this is used as build-time dependency from B. Consider
>    /usr/lib/x86_64-linux-gnu/libA.a
> to be linked into the software binary compiled by B.B.
> 
> ```meta-mylayer/recipes-core/B/B.bb
> ...
> SRC_URI = "git://..."
> SRC_DIR = "git"
> 
> DEPENDS = "A-dev"
> inherit dpkg
> 
> dpkg_runbuild() {
>    ...
>    ## UGLY hack to install build-time dependency A-dev required for the
>    ## software to be compiled here
>    for item in $(echo ${DEPENDS} | sed 's/buildchroot//g'); do
>      sudo dpkg --root=${BUILDCHROOT_DIR} -i ${DEPLOY_DIR_DEB}/${item}_*
>    done
> 

BTW: why can't you add this hack to 'build.sh', after 
'dpkg-buildpackage' complete, just run 'dpkg -i' on newly baked 
packages. Also ugly, but no headache with deps parsing. :-)

>    sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
>    ...
> }
> ```
> 
> This does not work on current -next. Even if you do the
>    do_build[deptask] = "do_build"
> because do_cleanup() kicks in and unmounts ${BUILDCHROOT_DIR}/git
> and the task reordering of deploy_deb().
> 
> 
> Hope this example helps?

Probably I get the possible issue. How do you add packages A and B to 
the build chain, via IMAGE_INSTALL? If so, where do you set 
IMAGE_INSTALL variable?

Alex

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Christian Storm]

> [...] 
> BTW: why can't you add this hack to 'build.sh', after 
> 'dpkg-buildpackage' complete, just run 'dpkg -i' on newly baked 
> packages. Also ugly, but no headache with deps parsing. :-)

Yes, could do that, for the time being. But this installs each and every
package into the buildchroot, I just need one. However, I think I go for
this to not loose contact to -next's HEAD.

> >    sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
> >    ...
> > }
> > ```
> > 
> > This does not work on current -next. Even if you do the
> >    do_build[deptask] = "do_build"
> > because do_cleanup() kicks in and unmounts ${BUILDCHROOT_DIR}/git
> > and the task reordering of deploy_deb().
> > 
> > 
> > Hope this example helps?
> 
> Probably I get the possible issue. How do you add packages A and B to 
> the build chain, via IMAGE_INSTALL? If so, where do you set 
> IMAGE_INSTALL variable?

I do have a meta-mylayer/recipes-core/images/image-mybox.bb in which I
do IMAGE_INSTALL_append = " B"


Besten Gru�,
   Christian

-- 
Dr. Christian Storm
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Otto-Hahn-Ring 6, 81739 M�nchen, Germany

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Alexander Smirnov]

On 12/12/2017 11:01 AM, Christian Storm wrote:
>> [...]
>> BTW: why can't you add this hack to 'build.sh', after
>> 'dpkg-buildpackage' complete, just run 'dpkg -i' on newly baked
>> packages. Also ugly, but no headache with deps parsing. :-)
> 
> Yes, could do that, for the time being. But this installs each and every
> package into the buildchroot, I just need one. However, I think I go for
> this to not loose contact to -next's HEAD.
> 
>>>     sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
>>>     ...
>>> }
>>> ```
>>>
>>> This does not work on current -next. Even if you do the
>>>     do_build[deptask] = "do_build"
>>> because do_cleanup() kicks in and unmounts ${BUILDCHROOT_DIR}/git
>>> and the task reordering of deploy_deb().
>>>
>>>
>>> Hope this example helps?
>>
>> Probably I get the possible issue. How do you add packages A and B to
>> the build chain, via IMAGE_INSTALL? If so, where do you set
>> IMAGE_INSTALL variable?
> 
> I do have a meta-mylayer/recipes-core/images/image-mybox.bb in which I
> do IMAGE_INSTALL_append = " B"
>

I think that's the problem with current series. I assume that 
IMAGE_INSTALL is global variable, like in pure Isar where it's set in 
'local.conf' file. If you append IMAGE_INSTALL locally in recipe, this 
changes won't be seen in buildchroot recipe, so

DEPENDS += "${IMAGE_INSTALL}"

makes no sense.

Ok, will drop it.

Alex

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Henning Schild]

Am Fri, 8 Dec 2017 08:58:06 +0100
schrieb "[ext] Christian Storm" <christian.storm@siemens.com>:

> > 2. 'do_cleanup': This task is executed after all the packages are
> > deployed. Some notes:
> >  - This task also should not have stamp.
> >  - This task depends from the recipes listed in IMAGE_INSTALL.  
> 
> This breaks build dependencies when building two packages from source
> (same example I've given in other posts as well):

These kind of deps are a known missing feature in Isar. Nothing can
break a missing feature.

> Consider A-dev required by B, both to be built from source. B cannot
> install the A-dev.deb since B depending on A-dev's deploy_deb() and
> installing it the ugly way via dpkg -i (as no other option is
> currently available in Isar) puts B as late in the task order so that
> it's git repo is not mounted anymore. Failure.

I think that is what your local hack is based on?

Henning

> If A-dev installs it's produced .deb as last step in its
> dpkg_runbuild(), which is also ugly, then there's a potential race in
> dpkg. Failure.
> 
> 
>    Christian
> 

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Henning Schild]

Just found another issue with this one. Building something on current
next.

>> ERROR: Nothing PROVIDES '${IMAGE_INSTALL}' (but
>> multiconfig:qemuamd64-stretch:/builder/isar/meta/recipes-devtools/buildchroot/buildchroot.bb
>> DEPENDS on or otherwise requires it)

a 'IMAGE_INSTALL ?= ""' fixes that.

Henning

Am Thu, 23 Nov 2017 13:25:31 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Some packages could require builchroot filesystem tunning, for
> example share Isar downloads folder with chroot filesystem.
> 
> This patch adds two tasks for buildchroot:
> 
> 1. 'do_prepare': This task is executed after buildchroot rootfs
> generation and before 'do_build' from packages to be built.
> Some notes:
>  - An individual task is required for the case when you want to
> rebuild some package, this would require to run this preparation
> stuff for existing buildchroot rootfs.
>  - This task should not have stamp. This means that anytime you want
> to build something, 'do_prepare' should be executed.
> 
> 2. 'do_cleanup': This task is executed after all the packages are
> deployed. Some notes:
>  - This task also should not have stamp.
>  - This task depends from the recipes listed in IMAGE_INSTALL.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/classes/dpkg-base.bbclass                   |  2 +-
>  meta/classes/image.bbclass                       |  1 +
>  meta/recipes-devtools/buildchroot/buildchroot.bb | 21
> +++++++++++++++++++++ 3 files changed, 23 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/dpkg-base.bbclass
> b/meta/classes/dpkg-base.bbclass index 4941f9b..21ae88d 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -2,7 +2,7 @@
>  # Copyright (C) 2017 Siemens AG
>  
>  # Add dependency from buildchroot creation
> -do_build[depends] = "buildchroot:do_build"
> +do_build[depends] = "buildchroot:do_prepare"
>  
>  # Each package should have its own unique build folder, so use
>  # recipe name as identifier
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 7813b16..b0f645c 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -83,6 +83,7 @@ do_populate() {
>  
>  addtask populate before do_build after do_unpack
>  do_populate[deptask] = "do_deploy_deb"
> +do_populate[depends] = "buildchroot:do_cleanup"
>  
>  do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>  
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
> 6a94733..fe88e98 100644 ---
> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -67,7 +67,28 @@
> do_build() { # Install package builder script
>      sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
>  
> +    # Create share point for downloads
> +    sudo install -d ${BUILDCHROOT_DIR}/git
> +
>      # Configure root filesystem
>      sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>      _do_build_cleanup
>  }
> +
> +do_prepare[nostamp] = "1"
> +
> +do_prepare() {
> +    sudo mount --bind ${GITDIR} ${BUILDCHROOT_DIR}/git
> +}
> +
> +addtask prepare after do_build
> +
> +DEPENDS += "${IMAGE_INSTALL}"
> +do_cleanup[deptask] = "do_deploy_deb"
> +do_cleanup[nostamp] = "1"
> +
> +do_cleanup() {
> +    sudo umount ${BUILDCHROOT_DIR}/git
> +}
> +
> +addtask cleanup after do_prepare

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Alexander Smirnov]

Hi,

On 12/11/2017 09:29 PM, Henning Schild wrote:
> Just found another issue with this one. Building something on current
> next.
> 
>>> ERROR: Nothing PROVIDES '${IMAGE_INSTALL}' (but
>>> multiconfig:qemuamd64-stretch:/builder/isar/meta/recipes-devtools/buildchroot/buildchroot.bb
>>> DEPENDS on or otherwise requires it)
> 
> a 'IMAGE_INSTALL ?= ""' fixes that.
> 

Thanks for this catch.

It would be helpful if you describe your changes from upstream Isar that 
leads to this issue. Am I right, that you do not set IMAGE_INSTALL 
variable in local.conf file? Also an ideas how to handle such cases in 
pure Isar are welcome.

Alex

> Henning
> 
> Am Thu, 23 Nov 2017 13:25:31 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Some packages could require builchroot filesystem tunning, for
>> example share Isar downloads folder with chroot filesystem.
>>
>> This patch adds two tasks for buildchroot:
>>
>> 1. 'do_prepare': This task is executed after buildchroot rootfs
>> generation and before 'do_build' from packages to be built.
>> Some notes:
>>   - An individual task is required for the case when you want to
>> rebuild some package, this would require to run this preparation
>> stuff for existing buildchroot rootfs.
>>   - This task should not have stamp. This means that anytime you want
>> to build something, 'do_prepare' should be executed.
>>
>> 2. 'do_cleanup': This task is executed after all the packages are
>> deployed. Some notes:
>>   - This task also should not have stamp.
>>   - This task depends from the recipes listed in IMAGE_INSTALL.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta/classes/dpkg-base.bbclass                   |  2 +-
>>   meta/classes/image.bbclass                       |  1 +
>>   meta/recipes-devtools/buildchroot/buildchroot.bb | 21
>> +++++++++++++++++++++ 3 files changed, 23 insertions(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/dpkg-base.bbclass
>> b/meta/classes/dpkg-base.bbclass index 4941f9b..21ae88d 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -2,7 +2,7 @@
>>   # Copyright (C) 2017 Siemens AG
>>   
>>   # Add dependency from buildchroot creation
>> -do_build[depends] = "buildchroot:do_build"
>> +do_build[depends] = "buildchroot:do_prepare"
>>   
>>   # Each package should have its own unique build folder, so use
>>   # recipe name as identifier
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index 7813b16..b0f645c 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -83,6 +83,7 @@ do_populate() {
>>   
>>   addtask populate before do_build after do_unpack
>>   do_populate[deptask] = "do_deploy_deb"
>> +do_populate[depends] = "buildchroot:do_cleanup"
>>   
>>   do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>>   
>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
>> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
>> 6a94733..fe88e98 100644 ---
>> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
>> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -67,7 +67,28 @@
>> do_build() { # Install package builder script
>>       sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
>>   
>> +    # Create share point for downloads
>> +    sudo install -d ${BUILDCHROOT_DIR}/git
>> +
>>       # Configure root filesystem
>>       sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>>       _do_build_cleanup
>>   }
>> +
>> +do_prepare[nostamp] = "1"
>> +
>> +do_prepare() {
>> +    sudo mount --bind ${GITDIR} ${BUILDCHROOT_DIR}/git
>> +}
>> +
>> +addtask prepare after do_build
>> +
>> +DEPENDS += "${IMAGE_INSTALL}"
>> +do_cleanup[deptask] = "do_deploy_deb"
>> +do_cleanup[nostamp] = "1"
>> +
>> +do_cleanup() {
>> +    sudo umount ${BUILDCHROOT_DIR}/git
>> +}
>> +
>> +addtask cleanup after do_prepare
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Christian Storm]

> >>> ERROR: Nothing PROVIDES '${IMAGE_INSTALL}' (but
> >>> multiconfig:qemuamd64-stretch:/builder/isar/meta/recipes-devtools/buildchroot/buildchroot.bb
> >>> DEPENDS on or otherwise requires it)
> > 
> > a 'IMAGE_INSTALL ?= ""' fixes that.
> > 
> 
> Thanks for this catch.
> 
> It would be helpful if you describe your changes from upstream Isar that 
> leads to this issue. Am I right, that you do not set IMAGE_INSTALL 
> variable in local.conf file? Also an ideas how to handle such cases in 
> pure Isar are welcome.

I sent a patch for exactly this on Thu, 7 Dec 2017 14:17:48 +0100 with
the subject "[PATCH] buildchroot: soft-assign default IMAGE_INSTALL".


Besten Gru�,
   Christian

-- 
Dr. Christian Storm
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Otto-Hahn-Ring 6, 81739 M�nchen, Germany

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Henning Schild]

Am Mon, 11 Dec 2017 22:56:42 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Hi,
> 
> On 12/11/2017 09:29 PM, Henning Schild wrote:
> > Just found another issue with this one. Building something on
> > current next.
> >   
> >>> ERROR: Nothing PROVIDES '${IMAGE_INSTALL}' (but
> >>> multiconfig:qemuamd64-stretch:/builder/isar/meta/recipes-devtools/buildchroot/buildchroot.bb
> >>> DEPENDS on or otherwise requires it)  
> > 
> > a 'IMAGE_INSTALL ?= ""' fixes that.
> >   
> 
> Thanks for this catch.
> 
> It would be helpful if you describe your changes from upstream Isar
> that leads to this issue. Am I right, that you do not set
> IMAGE_INSTALL variable in local.conf file?

Yes it is a layer with IMAGE_INSTALL not set in its local.conf, no
changes to Isar.

> Also an ideas how to handle such cases in pure Isar are welcome.

If we really need a VARIABLE to be initialized in a local.conf that
should probably be documented.
It might make sense to extend the testing to actually testing the
layering a well. Such a layer could be in the tree and we should build
targets with and without the layer.

But before that we should probably sort meta and meta-isar. As far as i
see it "meta" should be the core of Isar while "meta-isar" could
eventually be the layer using that core. At the moment i do not really
have an idea what the idea behind the current split is, and whether
that cut is clean.

I am already thinking about patches for more layering features,
i.e. customization of multistrap.conf.

Henning

> Alex
> 
> > Henning
> > 
> > Am Thu, 23 Nov 2017 13:25:31 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> Some packages could require builchroot filesystem tunning, for
> >> example share Isar downloads folder with chroot filesystem.
> >>
> >> This patch adds two tasks for buildchroot:
> >>
> >> 1. 'do_prepare': This task is executed after buildchroot rootfs
> >> generation and before 'do_build' from packages to be built.
> >> Some notes:
> >>   - An individual task is required for the case when you want to
> >> rebuild some package, this would require to run this preparation
> >> stuff for existing buildchroot rootfs.
> >>   - This task should not have stamp. This means that anytime you
> >> want to build something, 'do_prepare' should be executed.
> >>
> >> 2. 'do_cleanup': This task is executed after all the packages are
> >> deployed. Some notes:
> >>   - This task also should not have stamp.
> >>   - This task depends from the recipes listed in IMAGE_INSTALL.
> >>
> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >> ---
> >>   meta/classes/dpkg-base.bbclass                   |  2 +-
> >>   meta/classes/image.bbclass                       |  1 +
> >>   meta/recipes-devtools/buildchroot/buildchroot.bb | 21
> >> +++++++++++++++++++++ 3 files changed, 23 insertions(+), 1
> >> deletion(-)
> >>
> >> diff --git a/meta/classes/dpkg-base.bbclass
> >> b/meta/classes/dpkg-base.bbclass index 4941f9b..21ae88d 100644
> >> --- a/meta/classes/dpkg-base.bbclass
> >> +++ b/meta/classes/dpkg-base.bbclass
> >> @@ -2,7 +2,7 @@
> >>   # Copyright (C) 2017 Siemens AG
> >>   
> >>   # Add dependency from buildchroot creation
> >> -do_build[depends] = "buildchroot:do_build"
> >> +do_build[depends] = "buildchroot:do_prepare"
> >>   
> >>   # Each package should have its own unique build folder, so use
> >>   # recipe name as identifier
> >> diff --git a/meta/classes/image.bbclass
> >> b/meta/classes/image.bbclass index 7813b16..b0f645c 100644
> >> --- a/meta/classes/image.bbclass
> >> +++ b/meta/classes/image.bbclass
> >> @@ -83,6 +83,7 @@ do_populate() {
> >>   
> >>   addtask populate before do_build after do_unpack
> >>   do_populate[deptask] = "do_deploy_deb"
> >> +do_populate[depends] = "buildchroot:do_cleanup"
> >>   
> >>   do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
> >>   
> >> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
> >> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
> >> 6a94733..fe88e98 100644 ---
> >> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
> >> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -67,7 +67,28
> >> @@ do_build() { # Install package builder script
> >>       sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
> >>   
> >> +    # Create share point for downloads
> >> +    sudo install -d ${BUILDCHROOT_DIR}/git
> >> +
> >>       # Configure root filesystem
> >>       sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
> >>       _do_build_cleanup
> >>   }
> >> +
> >> +do_prepare[nostamp] = "1"
> >> +
> >> +do_prepare() {
> >> +    sudo mount --bind ${GITDIR} ${BUILDCHROOT_DIR}/git
> >> +}
> >> +
> >> +addtask prepare after do_build
> >> +
> >> +DEPENDS += "${IMAGE_INSTALL}"
> >> +do_cleanup[deptask] = "do_deploy_deb"
> >> +do_cleanup[nostamp] = "1"
> >> +
> >> +do_cleanup() {
> >> +    sudo umount ${BUILDCHROOT_DIR}/git
> >> +}
> >> +
> >> +addtask cleanup after do_prepare  
> >   
> 

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Jan Kiszka]

On 2017-12-12 13:05, [ext] Henning Schild wrote:
> Am Mon, 11 Dec 2017 22:56:42 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Hi,
>>
>> On 12/11/2017 09:29 PM, Henning Schild wrote:
>>> Just found another issue with this one. Building something on
>>> current next.
>>>   
>>>>> ERROR: Nothing PROVIDES '${IMAGE_INSTALL}' (but
>>>>> multiconfig:qemuamd64-stretch:/builder/isar/meta/recipes-devtools/buildchroot/buildchroot.bb
>>>>> DEPENDS on or otherwise requires it)  
>>>
>>> a 'IMAGE_INSTALL ?= ""' fixes that.
>>>   
>>
>> Thanks for this catch.
>>
>> It would be helpful if you describe your changes from upstream Isar
>> that leads to this issue. Am I right, that you do not set
>> IMAGE_INSTALL variable in local.conf file?
> 
> Yes it is a layer with IMAGE_INSTALL not set in its local.conf, no
> changes to Isar.
> 
>> Also an ideas how to handle such cases in pure Isar are welcome.
> 
> If we really need a VARIABLE to be initialized in a local.conf that
> should probably be documented.

If that was required, it would be a bug in the design. Such default
initializations belong into a central conf file in the Isar repo that is
of the required scope so that local.conf does not require any special
additions.

Jan

> It might make sense to extend the testing to actually testing the
> layering a well. Such a layer could be in the tree and we should build
> targets with and without the layer.
> 
> But before that we should probably sort meta and meta-isar. As far as i
> see it "meta" should be the core of Isar while "meta-isar" could
> eventually be the layer using that core. At the moment i do not really
> have an idea what the idea behind the current split is, and whether
> that cut is clean.
> 
> I am already thinking about patches for more layering features,
> i.e. customization of multistrap.conf.
> 
> Henning
> 
>> Alex
>>
>>> Henning
>>>
>>> Am Thu, 23 Nov 2017 13:25:31 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>   
>>>> Some packages could require builchroot filesystem tunning, for
>>>> example share Isar downloads folder with chroot filesystem.
>>>>
>>>> This patch adds two tasks for buildchroot:
>>>>
>>>> 1. 'do_prepare': This task is executed after buildchroot rootfs
>>>> generation and before 'do_build' from packages to be built.
>>>> Some notes:
>>>>   - An individual task is required for the case when you want to
>>>> rebuild some package, this would require to run this preparation
>>>> stuff for existing buildchroot rootfs.
>>>>   - This task should not have stamp. This means that anytime you
>>>> want to build something, 'do_prepare' should be executed.
>>>>
>>>> 2. 'do_cleanup': This task is executed after all the packages are
>>>> deployed. Some notes:
>>>>   - This task also should not have stamp.
>>>>   - This task depends from the recipes listed in IMAGE_INSTALL.
>>>>
>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>> ---
>>>>   meta/classes/dpkg-base.bbclass                   |  2 +-
>>>>   meta/classes/image.bbclass                       |  1 +
>>>>   meta/recipes-devtools/buildchroot/buildchroot.bb | 21
>>>> +++++++++++++++++++++ 3 files changed, 23 insertions(+), 1
>>>> deletion(-)
>>>>
>>>> diff --git a/meta/classes/dpkg-base.bbclass
>>>> b/meta/classes/dpkg-base.bbclass index 4941f9b..21ae88d 100644
>>>> --- a/meta/classes/dpkg-base.bbclass
>>>> +++ b/meta/classes/dpkg-base.bbclass
>>>> @@ -2,7 +2,7 @@
>>>>   # Copyright (C) 2017 Siemens AG
>>>>   
>>>>   # Add dependency from buildchroot creation
>>>> -do_build[depends] = "buildchroot:do_build"
>>>> +do_build[depends] = "buildchroot:do_prepare"
>>>>   
>>>>   # Each package should have its own unique build folder, so use
>>>>   # recipe name as identifier
>>>> diff --git a/meta/classes/image.bbclass
>>>> b/meta/classes/image.bbclass index 7813b16..b0f645c 100644
>>>> --- a/meta/classes/image.bbclass
>>>> +++ b/meta/classes/image.bbclass
>>>> @@ -83,6 +83,7 @@ do_populate() {
>>>>   
>>>>   addtask populate before do_build after do_unpack
>>>>   do_populate[deptask] = "do_deploy_deb"
>>>> +do_populate[depends] = "buildchroot:do_cleanup"
>>>>   
>>>>   do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>>>>   
>>>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
>>>> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
>>>> 6a94733..fe88e98 100644 ---
>>>> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
>>>> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -67,7 +67,28
>>>> @@ do_build() { # Install package builder script
>>>>       sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
>>>>   
>>>> +    # Create share point for downloads
>>>> +    sudo install -d ${BUILDCHROOT_DIR}/git
>>>> +
>>>>       # Configure root filesystem
>>>>       sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>>>>       _do_build_cleanup
>>>>   }
>>>> +
>>>> +do_prepare[nostamp] = "1"
>>>> +
>>>> +do_prepare() {
>>>> +    sudo mount --bind ${GITDIR} ${BUILDCHROOT_DIR}/git
>>>> +}
>>>> +
>>>> +addtask prepare after do_build
>>>> +
>>>> +DEPENDS += "${IMAGE_INSTALL}"
>>>> +do_cleanup[deptask] = "do_deploy_deb"
>>>> +do_cleanup[nostamp] = "1"
>>>> +
>>>> +do_cleanup() {
>>>> +    sudo umount ${BUILDCHROOT_DIR}/git
>>>> +}
>>>> +
>>>> +addtask cleanup after do_prepare  
>>>   
>>
> 


-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 3/4 v2] buildchroot: Add prepare and cleanup tasks

[Henning Schild]

Am Thu, 23 Nov 2017 13:25:31 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Some packages could require builchroot filesystem tunning, for
> example share Isar downloads folder with chroot filesystem.
> 
> This patch adds two tasks for buildchroot:
> 
> 1. 'do_prepare': This task is executed after buildchroot rootfs
> generation and before 'do_build' from packages to be built.
> Some notes:
>  - An individual task is required for the case when you want to
> rebuild some package, this would require to run this preparation
> stuff for existing buildchroot rootfs.
>  - This task should not have stamp. This means that anytime you want
> to build something, 'do_prepare' should be executed.
> 
> 2. 'do_cleanup': This task is executed after all the packages are
> deployed. Some notes:
>  - This task also should not have stamp.
>  - This task depends from the recipes listed in IMAGE_INSTALL.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/classes/dpkg-base.bbclass                   |  2 +-
>  meta/classes/image.bbclass                       |  1 +
>  meta/recipes-devtools/buildchroot/buildchroot.bb | 21
> +++++++++++++++++++++ 3 files changed, 23 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/dpkg-base.bbclass
> b/meta/classes/dpkg-base.bbclass index 4941f9b..21ae88d 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -2,7 +2,7 @@
>  # Copyright (C) 2017 Siemens AG
>  
>  # Add dependency from buildchroot creation
> -do_build[depends] = "buildchroot:do_build"
> +do_build[depends] = "buildchroot:do_prepare"
>  
>  # Each package should have its own unique build folder, so use
>  # recipe name as identifier
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 7813b16..b0f645c 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -83,6 +83,7 @@ do_populate() {
>  
>  addtask populate before do_build after do_unpack
>  do_populate[deptask] = "do_deploy_deb"
> +do_populate[depends] = "buildchroot:do_cleanup"
>  
>  do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>  
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
> 6a94733..fe88e98 100644 ---
> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -67,7 +67,28 @@
> do_build() { # Install package builder script
>      sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
>  
> +    # Create share point for downloads
> +    sudo install -d ${BUILDCHROOT_DIR}/git
> +
>      # Configure root filesystem
>      sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>      _do_build_cleanup
>  }
> +
> +do_prepare[nostamp] = "1"
> +
> +do_prepare() {
> +    sudo mount --bind ${GITDIR} ${BUILDCHROOT_DIR}/git
> +}

And another one ...

mount: special device /builder/build/tmp/downloads/git does not exist

Henning

> +
> +addtask prepare after do_build
> +
> +DEPENDS += "${IMAGE_INSTALL}"
> +do_cleanup[deptask] = "do_deploy_deb"
> +do_cleanup[nostamp] = "1"
> +
> +do_cleanup() {
> +    sudo umount ${BUILDCHROOT_DIR}/git
> +}
> +
> +addtask cleanup after do_prepare

[PATCH 4/4 v2] dpkg-base: Update git alternates

[Alexander Smirnov]

Update git alternates to work in buildchroot.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 21ae88d..d51a709 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -18,6 +18,13 @@ dpkg_runbuild() {
 
 # Wrap the function dpkg_runbuild with the bind mount for buildroot
 do_build() {
+    if [ -d ${WORKDIR}/git/.git ]; then
+        OBJ_PATH=$(cat ${WORKDIR}/git/.git/objects/info/alternates)
+        REPO_PATH=$(dirname $OBJ_PATH)
+        REPO_NAME=$(basename $REPO_PATH)
+        echo "/git/$REPO_NAME/objects" > ${WORKDIR}/git/.git/objects/info/alternates
+    fi
+
     mkdir -p ${BUILDROOT}
     sudo mount --bind ${WORKDIR} ${BUILDROOT}
     _do_build_cleanup() {
-- 
2.9.5

Re: [PATCH 4/4 v2] dpkg-base: Update git alternates

[Henning Schild]

Am Thu, 23 Nov 2017 13:25:32 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Update git alternates to work in buildchroot.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/classes/dpkg-base.bbclass | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/meta/classes/dpkg-base.bbclass
> b/meta/classes/dpkg-base.bbclass index 21ae88d..d51a709 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -18,6 +18,13 @@ dpkg_runbuild() {
>  
>  # Wrap the function dpkg_runbuild with the bind mount for buildroot
>  do_build() {
> +    if [ -d ${WORKDIR}/git/.git ]; then
> +        OBJ_PATH=$(cat ${WORKDIR}/git/.git/objects/info/alternates)
> +        REPO_PATH=$(dirname $OBJ_PATH)
> +        REPO_NAME=$(basename $REPO_PATH)
> +        echo "/git/$REPO_NAME/objects" >
> ${WORKDIR}/git/.git/objects/info/alternates
> +    fi

Does that not overwrite a file so that the repo can now work only in
buildchroot and will be broken on the host?

Henning

>      mkdir -p ${BUILDROOT}
>      sudo mount --bind ${WORKDIR} ${BUILDROOT}
>      _do_build_cleanup() {

Re: [PATCH 4/4 v2] dpkg-base: Update git alternates

[Alexander Smirnov]

On 11/24/2017 05:57 PM, Henning Schild wrote:
> Am Thu, 23 Nov 2017 13:25:32 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Update git alternates to work in buildchroot.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta/classes/dpkg-base.bbclass | 7 +++++++
>>   1 file changed, 7 insertions(+)
>>
>> diff --git a/meta/classes/dpkg-base.bbclass
>> b/meta/classes/dpkg-base.bbclass index 21ae88d..d51a709 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -18,6 +18,13 @@ dpkg_runbuild() {
>>   
>>   # Wrap the function dpkg_runbuild with the bind mount for buildroot
>>   do_build() {
>> +    if [ -d ${WORKDIR}/git/.git ]; then
>> +        OBJ_PATH=$(cat ${WORKDIR}/git/.git/objects/info/alternates)
>> +        REPO_PATH=$(dirname $OBJ_PATH)
>> +        REPO_NAME=$(basename $REPO_PATH)
>> +        echo "/git/$REPO_NAME/objects" >
>> ${WORKDIR}/git/.git/objects/info/alternates
>> +    fi
> 
> Does that not overwrite a file so that the repo can now work only in
> buildchroot and will be broken on the host?
> 

It does, but in general there is no use-case for having working repo 
clone on the host. I could restore original alternates in cleanup hook, 
but it really makes no sense.

Alex

> Henning
> 
>>       mkdir -p ${BUILDROOT}
>>       sudo mount --bind ${WORKDIR} ${BUILDROOT}
>>       _do_build_cleanup() {

Re: [PATCH 4/4 v2] dpkg-base: Update git alternates

[Henning Schild]

Am Fri, 24 Nov 2017 18:12:43 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 11/24/2017 05:57 PM, Henning Schild wrote:
> > Am Thu, 23 Nov 2017 13:25:32 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> Update git alternates to work in buildchroot.
> >>
> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >> ---
> >>   meta/classes/dpkg-base.bbclass | 7 +++++++
> >>   1 file changed, 7 insertions(+)
> >>
> >> diff --git a/meta/classes/dpkg-base.bbclass
> >> b/meta/classes/dpkg-base.bbclass index 21ae88d..d51a709 100644
> >> --- a/meta/classes/dpkg-base.bbclass
> >> +++ b/meta/classes/dpkg-base.bbclass
> >> @@ -18,6 +18,13 @@ dpkg_runbuild() {
> >>   
> >>   # Wrap the function dpkg_runbuild with the bind mount for
> >> buildroot do_build() {
> >> +    if [ -d ${WORKDIR}/git/.git ]; then
> >> +        OBJ_PATH=$(cat
> >> ${WORKDIR}/git/.git/objects/info/alternates)
> >> +        REPO_PATH=$(dirname $OBJ_PATH)
> >> +        REPO_NAME=$(basename $REPO_PATH)
> >> +        echo "/git/$REPO_NAME/objects" >
> >> ${WORKDIR}/git/.git/objects/info/alternates
> >> +    fi  
> > 
> > Does that not overwrite a file so that the repo can now work only in
> > buildchroot and will be broken on the host?
> >   
> 
> It does, but in general there is no use-case for having working repo 
> clone on the host. I could restore original alternates in cleanup
> hook, but it really makes no sense.

I do not like breaking that clone, even for a short while and repairing
it later.
The host-clone might be looked at while debugging.
And one could imagine the fetcher/unpacker supporting deduplication if
multiple recipes use the same repo. Even if such a feature is not
present in bitbake today, if it is implemented Isar will break. Think
one recipe being in do_build and another one in do_fetch at the same
time.

Henning

> Alex
> 
> > Henning
> >   
> >>       mkdir -p ${BUILDROOT}
> >>       sudo mount --bind ${WORKDIR} ${BUILDROOT}
> >>       _do_build_cleanup() {  

Re: [PATCH 4/4 v2] dpkg-base: Update git alternates

[Alexander Smirnov]

On 11/27/2017 10:44 AM, Henning Schild wrote:
> Am Fri, 24 Nov 2017 18:12:43 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 11/24/2017 05:57 PM, Henning Schild wrote:
>>> Am Thu, 23 Nov 2017 13:25:32 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>    
>>>> Update git alternates to work in buildchroot.
>>>>
>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>> ---
>>>>    meta/classes/dpkg-base.bbclass | 7 +++++++
>>>>    1 file changed, 7 insertions(+)
>>>>
>>>> diff --git a/meta/classes/dpkg-base.bbclass
>>>> b/meta/classes/dpkg-base.bbclass index 21ae88d..d51a709 100644
>>>> --- a/meta/classes/dpkg-base.bbclass
>>>> +++ b/meta/classes/dpkg-base.bbclass
>>>> @@ -18,6 +18,13 @@ dpkg_runbuild() {
>>>>    
>>>>    # Wrap the function dpkg_runbuild with the bind mount for
>>>> buildroot do_build() {
>>>> +    if [ -d ${WORKDIR}/git/.git ]; then
>>>> +        OBJ_PATH=$(cat
>>>> ${WORKDIR}/git/.git/objects/info/alternates)
>>>> +        REPO_PATH=$(dirname $OBJ_PATH)
>>>> +        REPO_NAME=$(basename $REPO_PATH)
>>>> +        echo "/git/$REPO_NAME/objects" >
>>>> ${WORKDIR}/git/.git/objects/info/alternates
>>>> +    fi
>>>
>>> Does that not overwrite a file so that the repo can now work only in
>>> buildchroot and will be broken on the host?
>>>    
>>
>> It does, but in general there is no use-case for having working repo
>> clone on the host. I could restore original alternates in cleanup
>> hook, but it really makes no sense.
> 
> I do not like breaking that clone, even for a short while and repairing
> it later.
> The host-clone might be looked at while debugging.
> And one could imagine the fetcher/unpacker supporting deduplication if
> multiple recipes use the same repo. Even if such a feature is not
> present in bitbake today, if it is implemented Isar will break. Think
> one recipe being in do_build and another one in do_fetch at the same
> time.

Don't understand what you mean, but probably the following items could 
better describe what happens in this patch:

1. There is Isar downloads folder 'tmp/downloads', which contains git 
clones from origins.
2. There is workdir folder per recipe, which contains git clone from 
Isar downloads folder.
3. The patch modifies clone in workdir, so the only current recipe is 
user for this repo.
4. The downloads folder content stays unmodified for all the time.

Alex

Re: [PATCH 4/4 v2] dpkg-base: Update git alternates

[Henning Schild]

Am Mon, 27 Nov 2017 11:03:47 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 11/27/2017 10:44 AM, Henning Schild wrote:
> > Am Fri, 24 Nov 2017 18:12:43 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> On 11/24/2017 05:57 PM, Henning Schild wrote:  
> >>> Am Thu, 23 Nov 2017 13:25:32 +0300
> >>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >>>      
> >>>> Update git alternates to work in buildchroot.
> >>>>
> >>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >>>> ---
> >>>>    meta/classes/dpkg-base.bbclass | 7 +++++++
> >>>>    1 file changed, 7 insertions(+)
> >>>>
> >>>> diff --git a/meta/classes/dpkg-base.bbclass
> >>>> b/meta/classes/dpkg-base.bbclass index 21ae88d..d51a709 100644
> >>>> --- a/meta/classes/dpkg-base.bbclass
> >>>> +++ b/meta/classes/dpkg-base.bbclass
> >>>> @@ -18,6 +18,13 @@ dpkg_runbuild() {
> >>>>    
> >>>>    # Wrap the function dpkg_runbuild with the bind mount for
> >>>> buildroot do_build() {
> >>>> +    if [ -d ${WORKDIR}/git/.git ]; then
> >>>> +        OBJ_PATH=$(cat
> >>>> ${WORKDIR}/git/.git/objects/info/alternates)
> >>>> +        REPO_PATH=$(dirname $OBJ_PATH)
> >>>> +        REPO_NAME=$(basename $REPO_PATH)
> >>>> +        echo "/git/$REPO_NAME/objects" >
> >>>> ${WORKDIR}/git/.git/objects/info/alternates
> >>>> +    fi  
> >>>
> >>> Does that not overwrite a file so that the repo can now work only
> >>> in buildchroot and will be broken on the host?
> >>>      
> >>
> >> It does, but in general there is no use-case for having working
> >> repo clone on the host. I could restore original alternates in
> >> cleanup hook, but it really makes no sense.  
> > 
> > I do not like breaking that clone, even for a short while and
> > repairing it later.
> > The host-clone might be looked at while debugging.
> > And one could imagine the fetcher/unpacker supporting deduplication
> > if multiple recipes use the same repo. Even if such a feature is not
> > present in bitbake today, if it is implemented Isar will break.
> > Think one recipe being in do_build and another one in do_fetch at
> > the same time.  
> 
> Don't understand what you mean, but probably the following items
> could better describe what happens in this patch:
> 
> 1. There is Isar downloads folder 'tmp/downloads', which contains git 
> clones from origins.
> 2. There is workdir folder per recipe, which contains git clone from 
> Isar downloads folder.
> 3. The patch modifies clone in workdir, so the only current recipe is 
> user for this repo.
> 4. The downloads folder content stays unmodified for all the time.

I see, in that case i have no objections.


Thanks,
Henning

> Alex