[PATCH 0/3] Improvements

[PATCH 0/3] Improvements

[Alexander Smirnov]

Hi all,

the goal of this series is to intrioduce 'prepare' and 'cleanup' tasks
for buildchroot. In parallel several issue were fixed.

Any suggestions regarding architecture are welcome!

With best regards,
Alex

Alexander Smirnov (3):
  dpkg-base: Fix buildchroot dependancy
  image: Add do_rootfs template to image class
  buildchroot: Add prepare and cleanup tasks

 meta-isar/recipes-core/images/isar-image-base.bb |  3 ---
 meta/classes/dpkg-base.bbclass                   |  3 +--
 meta/classes/image.bbclass                       |  9 +++++++++
 meta/recipes-devtools/buildchroot/buildchroot.bb | 18 ++++++++++++++++++
 4 files changed, 28 insertions(+), 5 deletions(-)

-- 
2.9.5

[PATCH 1/3] dpkg-base: Fix buildchroot dependency

[Alexander Smirnov]

Existing code forces recipe to depends from 'do_build' task for all
the items listed in DEPENDS, what actually makes no sense. This task
only produces binary deb package and it can't satisfy any dependencies
for another task.

Original idea was to add dependency from 'buildchroot:do_build' task,
so this patch fixes this.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 35af6d5..4941f9b 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -2,8 +2,7 @@
 # Copyright (C) 2017 Siemens AG
 
 # Add dependency from buildchroot creation
-DEPENDS += "buildchroot"
-do_build[deptask] = "do_build"
+do_build[depends] = "buildchroot:do_build"
 
 # Each package should have its own unique build folder, so use
 # recipe name as identifier
-- 
2.9.5

Re: [PATCH 1/3] dpkg-base: Fix buildchroot dependency

[Henning Schild]

Am Tue, 21 Nov 2017 21:49:38 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Existing code forces recipe to depends from 'do_build' task for all
> the items listed in DEPENDS, what actually makes no sense. This task
> only produces binary deb package and it can't satisfy any dependencies
> for another task.

With DEPENDS you would also say that application1 that you build in
buildchroot needs lib1 and lib2 and buildtool3, and you have recipes
for all of those (they are not upstream debian).
Now requiring .debs makes some sense because now the "apt-get install
-y $DEPS" could actually install them in buildchroot. I do not think
that works at the moment, because the Isar repo is not in buildchroot
multistrap. 

Technically both versions are wrong, so applying that patch does not
seem to improve the situation.

We need an example of two packages from Isar that actually DEPEND on
each other in buildchroot. I think Christian has patches for that
problem.

Henning

> Original idea was to add dependency from 'buildchroot:do_build' task,
> so this patch fixes this.
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/classes/dpkg-base.bbclass | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/meta/classes/dpkg-base.bbclass
> b/meta/classes/dpkg-base.bbclass index 35af6d5..4941f9b 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -2,8 +2,7 @@
>  # Copyright (C) 2017 Siemens AG
>  
>  # Add dependency from buildchroot creation
> -DEPENDS += "buildchroot"
> -do_build[deptask] = "do_build"
> +do_build[depends] = "buildchroot:do_build"
>  
>  # Each package should have its own unique build folder, so use
>  # recipe name as identifier

Re: [PATCH 1/3] dpkg-base: Fix buildchroot dependency

[Alexander Smirnov]

On 11/21/2017 10:10 PM, Henning Schild wrote:
> Am Tue, 21 Nov 2017 21:49:38 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Existing code forces recipe to depends from 'do_build' task for all
>> the items listed in DEPENDS, what actually makes no sense. This task
>> only produces binary deb package and it can't satisfy any dependencies
>> for another task.
> 
> With DEPENDS you would also say that application1 that you build in
> buildchroot needs lib1 and lib2 and buildtool3, and you have recipes
> for all of those (they are not upstream debian).
> Now requiring .debs makes some sense because now the "apt-get install
> -y $DEPS" could actually install them in buildchroot. I do not think
> that works at the moment, because the Isar repo is not in buildchroot
> multistrap.
> 
> Technically both versions are wrong, so applying that patch does not
> seem to improve the situation.

IMHO not so bad. Current usage of DEPENDS is invalid, so the patch drops 
this from Isar. I should come back with complete dependency implementation.

Also I'd like to split deps from buildchroot and deps between packages. 
They are different. Also packages possibly will use different task to 
implement interpackages deps, for example:

  - buildchroot:do_build
  - package_X:do_populate_buildchroot

So anyway it needs to explicitly set dependency from buildchroot:do_build.

> 
> We need an example of two packages from Isar that actually DEPEND on

That's true.

Alex

> each other in buildchroot. I think Christian has patches for that
> problem.
> 
> Henning
> 
>> Original idea was to add dependency from 'buildchroot:do_build' task,
>> so this patch fixes this.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta/classes/dpkg-base.bbclass | 3 +--
>>   1 file changed, 1 insertion(+), 2 deletions(-)
>>
>> diff --git a/meta/classes/dpkg-base.bbclass
>> b/meta/classes/dpkg-base.bbclass index 35af6d5..4941f9b 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -2,8 +2,7 @@
>>   # Copyright (C) 2017 Siemens AG
>>   
>>   # Add dependency from buildchroot creation
>> -DEPENDS += "buildchroot"
>> -do_build[deptask] = "do_build"
>> +do_build[depends] = "buildchroot:do_build"
>>   
>>   # Each package should have its own unique build folder, so use
>>   # recipe name as identifier

Re: [PATCH 1/3] dpkg-base: Fix buildchroot dependency

[Christian Storm]

> [...]
> We need an example of two packages from Isar that actually DEPEND on
> each other in buildchroot. I think Christian has patches for that
> problem.

Yes, this is exactly what I posted yesterday in "Re: [PATCH 0/4 v5] Isar
apt deployment" [1]: A (real) use case (for me), however, described in a
general manner.


Kind regards,
   Christian

[1] https://groups.google.com/d/msg/isar-users/efer3RF989o/r-zfUNNDAgAJ

-- 
Dr. Christian Storm
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Otto-Hahn-Ring 6, 81739 M�nchen, Germany

[PATCH 2/3] image: Add do_rootfs template to image class

[Alexander Smirnov]

Add 'do_rootfs' task template to the image class because it's
mandatory task for image generation. Moreover image class already
has dependencies from 'do_rootfs' what makes it inconsistent for
now. Also this patch moves generic task parameters to the class from
image recipe.

Also this patch removes trailing empty line.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar/recipes-core/images/isar-image-base.bb | 3 ---
 meta/classes/image.bbclass                       | 8 ++++++++
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index c2150b1..ad776ed 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -26,7 +26,6 @@ IMAGE_PREINSTALL += "apt \
 
 WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
 
-do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
 do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap"
 
 do_rootfs() {
@@ -70,5 +69,3 @@ do_rootfs() {
     sudo rm "${IMAGE_ROOTFS}/${DISTRO_CONFIG_SCRIPT}"
     _do_rootfs_cleanup
 }
-
-addtask rootfs before do_build after do_populate
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 2c2bafc..7813b16 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -84,6 +84,14 @@ do_populate() {
 addtask populate before do_build after do_unpack
 do_populate[deptask] = "do_deploy_deb"
 
+do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
+
+do_rootfs() {
+    die "No root filesystem function defined, please implement in your recipe"
+}
+
+addtask rootfs before do_build after do_populate
+
 do_copy_boot_files() {
     KERNEL_IMAGE=${@get_image_name(d, 'vmlinuz')}
     if [ -n "${KERNEL_IMAGE}" ]; then
-- 
2.9.5

[PATCH 3/3] buildchroot: Add prepare and cleanup tasks

[Alexander Smirnov]

Some packages could require builchroot filesystem tunning, for
example share Isar downloads folder with chroot filesystem.

This patch adds two tasks for buildchroot:

1. 'do_prepare': This task is executed after buildchroot rootfs
generation and before 'do_build' from packages to be built.
Some notes:
 - An individual task is required for the case when you want to rebuild
   some package, this would require to run this preparation stuff for
   existing buildchroot rootfs.
 - This task should not have stamp. This means that anytime you want to
   build something, 'do_prepare' should be executed.

2. 'do_cleanup': This task is executed after all the packages are deployed.
Some notes:
 - This task also should not have stamp.
 - This task depends from the recipes listed in IMAGE_INSTALL.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/dpkg-base.bbclass                   |  2 +-
 meta/classes/image.bbclass                       |  1 +
 meta/recipes-devtools/buildchroot/buildchroot.bb | 18 ++++++++++++++++++
 3 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 4941f9b..21ae88d 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -2,7 +2,7 @@
 # Copyright (C) 2017 Siemens AG
 
 # Add dependency from buildchroot creation
-do_build[depends] = "buildchroot:do_build"
+do_build[depends] = "buildchroot:do_prepare"
 
 # Each package should have its own unique build folder, so use
 # recipe name as identifier
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 7813b16..b0f645c 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -83,6 +83,7 @@ do_populate() {
 
 addtask populate before do_build after do_unpack
 do_populate[deptask] = "do_deploy_deb"
+do_populate[depends] = "buildchroot:do_cleanup"
 
 do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
 
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 6a94733..4be094c 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -71,3 +71,21 @@ do_build() {
     sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
     _do_build_cleanup
 }
+
+do_prepare[nostamp] = "1"
+
+do_prepare() {
+    bbnote "Prepare buildchroot for packages building"
+}
+
+addtask prepare after do_build
+
+DEPENDS += "${IMAGE_INSTALL}"
+do_cleanup[deptask] = "do_deploy_deb"
+do_cleanup[nostamp] = "1"
+
+do_cleanup() {
+    bbnote "Cleanup buildchroot after packages building"
+}
+
+addtask cleanup after do_prepare
-- 
2.9.5

Re: [PATCH 3/3] buildchroot: Add prepare and cleanup tasks

[Henning Schild]

Please repost this with a use-case. At the moment i can only guess what
that could be used for. What is the Issue?

Henning

Am Tue, 21 Nov 2017 21:49:40 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Some packages could require builchroot filesystem tunning, for
> example share Isar downloads folder with chroot filesystem.
> 
> This patch adds two tasks for buildchroot:
> 
> 1. 'do_prepare': This task is executed after buildchroot rootfs
> generation and before 'do_build' from packages to be built.
> Some notes:
>  - An individual task is required for the case when you want to
> rebuild some package, this would require to run this preparation
> stuff for existing buildchroot rootfs.
>  - This task should not have stamp. This means that anytime you want
> to build something, 'do_prepare' should be executed.
> 
> 2. 'do_cleanup': This task is executed after all the packages are
> deployed. Some notes:
>  - This task also should not have stamp.
>  - This task depends from the recipes listed in IMAGE_INSTALL.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/classes/dpkg-base.bbclass                   |  2 +-
>  meta/classes/image.bbclass                       |  1 +
>  meta/recipes-devtools/buildchroot/buildchroot.bb | 18
> ++++++++++++++++++ 3 files changed, 20 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/dpkg-base.bbclass
> b/meta/classes/dpkg-base.bbclass index 4941f9b..21ae88d 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -2,7 +2,7 @@
>  # Copyright (C) 2017 Siemens AG
>  
>  # Add dependency from buildchroot creation
> -do_build[depends] = "buildchroot:do_build"
> +do_build[depends] = "buildchroot:do_prepare"
>  
>  # Each package should have its own unique build folder, so use
>  # recipe name as identifier
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 7813b16..b0f645c 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -83,6 +83,7 @@ do_populate() {
>  
>  addtask populate before do_build after do_unpack
>  do_populate[deptask] = "do_deploy_deb"
> +do_populate[depends] = "buildchroot:do_cleanup"
>  
>  do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>  
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
> 6a94733..4be094c 100644 ---
> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -71,3 +71,21 @@
> do_build() { sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>      _do_build_cleanup
>  }
> +
> +do_prepare[nostamp] = "1"
> +
> +do_prepare() {
> +    bbnote "Prepare buildchroot for packages building"
> +}
> +
> +addtask prepare after do_build
> +
> +DEPENDS += "${IMAGE_INSTALL}"
> +do_cleanup[deptask] = "do_deploy_deb"
> +do_cleanup[nostamp] = "1"
> +
> +do_cleanup() {
> +    bbnote "Cleanup buildchroot after packages building"
> +}
> +
> +addtask cleanup after do_prepare

Re: [PATCH 3/3] buildchroot: Add prepare and cleanup tasks

[Alexander Smirnov]

On 11/21/2017 10:15 PM, Henning Schild wrote:
> Please repost this with a use-case. At the moment i can only guess what
> that could be used for. What is the Issue?
> 

The issue is the same:
  - mount downloads to buildchroot
  - build packages
  - umount downloads from buildchroot

It would be nice to discuss the overall approach before implementing 
specific usecase to avoid unnecessary effort. The series to apply will 
include use-case for sure.

Alex

> Henning
> 
> Am Tue, 21 Nov 2017 21:49:40 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Some packages could require builchroot filesystem tunning, for
>> example share Isar downloads folder with chroot filesystem.
>>
>> This patch adds two tasks for buildchroot:
>>
>> 1. 'do_prepare': This task is executed after buildchroot rootfs
>> generation and before 'do_build' from packages to be built.
>> Some notes:
>>   - An individual task is required for the case when you want to
>> rebuild some package, this would require to run this preparation
>> stuff for existing buildchroot rootfs.
>>   - This task should not have stamp. This means that anytime you want
>> to build something, 'do_prepare' should be executed.
>>
>> 2. 'do_cleanup': This task is executed after all the packages are
>> deployed. Some notes:
>>   - This task also should not have stamp.
>>   - This task depends from the recipes listed in IMAGE_INSTALL.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta/classes/dpkg-base.bbclass                   |  2 +-
>>   meta/classes/image.bbclass                       |  1 +
>>   meta/recipes-devtools/buildchroot/buildchroot.bb | 18
>> ++++++++++++++++++ 3 files changed, 20 insertions(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/dpkg-base.bbclass
>> b/meta/classes/dpkg-base.bbclass index 4941f9b..21ae88d 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -2,7 +2,7 @@
>>   # Copyright (C) 2017 Siemens AG
>>   
>>   # Add dependency from buildchroot creation
>> -do_build[depends] = "buildchroot:do_build"
>> +do_build[depends] = "buildchroot:do_prepare"
>>   
>>   # Each package should have its own unique build folder, so use
>>   # recipe name as identifier
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index 7813b16..b0f645c 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -83,6 +83,7 @@ do_populate() {
>>   
>>   addtask populate before do_build after do_unpack
>>   do_populate[deptask] = "do_deploy_deb"
>> +do_populate[depends] = "buildchroot:do_cleanup"
>>   
>>   do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>>   
>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
>> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
>> 6a94733..4be094c 100644 ---
>> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
>> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -71,3 +71,21 @@
>> do_build() { sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>>       _do_build_cleanup
>>   }
>> +
>> +do_prepare[nostamp] = "1"
>> +
>> +do_prepare() {
>> +    bbnote "Prepare buildchroot for packages building"
>> +}
>> +
>> +addtask prepare after do_build
>> +
>> +DEPENDS += "${IMAGE_INSTALL}"
>> +do_cleanup[deptask] = "do_deploy_deb"
>> +do_cleanup[nostamp] = "1"
>> +
>> +do_cleanup() {
>> +    bbnote "Cleanup buildchroot after packages building"
>> +}
>> +
>> +addtask cleanup after do_prepare
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov