[PATCH] doc: Add technical overview

[PATCH] doc: Add technical overview

[Alexander Smirnov]

Add initial Isar technical overview.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 doc/technical_overview.md | 110 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 110 insertions(+)
 create mode 100644 doc/technical_overview.md

diff --git a/doc/technical_overview.md b/doc/technical_overview.md
new file mode 100644
index 0000000..f65fc08
--- /dev/null
+++ b/doc/technical_overview.md
@@ -0,0 +1,110 @@
+# 1 Introduction
+
+Isar can be shortly described as a set of bitbake recipes that implement main build system logic. To simplify overall Isar understanding, this document is split into two main parts:
+ - Isar logical components
+ - Isar internal processes
+
+# 2 Isar Logical Components
+
+In this chapter the most important Isar components are considered in details. In this text component doesn't especially mean some self-contained single entity of something, it's just an attempt to split Isar internals by various criteria.
+
+### 2.1 Bitbake and Recipes
+
+All the processes in Isar are started by bitbake, so it manages general build executing process. Recipes in Isar can be split in two categories:
+
+ - System recipes and classes: they are responsible for setting up Debian-like infrastructure, manages Debian tools execution for package building and installation, generation root file system images
+ - User recipes: custom user applications, that should be builе from sources
+
+There are two types of dependencies in Isar:
+ - Dependency between bitbake recipes
+ - Dependencies in Debian filesystem
+
+**NOTE:** Isar doesn't manage dependencies in Debian file systems. User can only list specific Debian dependencies in recipe, so they eventually will be passed to apt-get or multistrap. Dependency installation is managed by Debian tools.
+
+### 2.2 Stamps
+
+Each task managed by bitbake uses stamp to notify that it has been completed. Due to Isar supports various Debian distributions and parallel builds for multiple machines and architectures, the stamps for tasks use special suffixes that include:
+ - Debian distro name
+ - Architecture
+ - Machine
+
+Typical example, when Isar builds the following configurations:
+ - Debian Jessie, amd64
+ - Debian Jessie, i386
+ - Debian Stretch, i386
+
+In this case there will be 3 different buildchroots, so standard hello demo application should be processed 3 times for each environment. Three different sets of stamps should be used for correct bitbake operating.
+
+### 2.3 Buildchroot
+
+One of the key aspect of Debian philosophy claims the fact, that everything in Debian should be built within Debian environment. Moreover native compilation is more preferable than cross-compilation. To follow this rules, Isar introduces the new component - buildchroot. Bulidchroot is typical Debian filesystem that is created using standard Debian tools: multistrap, apt. The source of packages can be either official Debian repositories or custom repositories created by user.
+
+Buildchroot lifecycle can be described as following:
+ - Buildchroot has initial configuration file which is passed to multistrap tool. This configuration file is generated by bitbake recipe from patterns and values defined by user. Based on this configuration file, multistrap generates initial filesystem.
+ - During building custom Debian package, list of its build dependencies is installed to buildchroot.
+ - When package has been built, it's installed to current buildchroot to satisfy further packages build dependencies.
+
+### 2.4 Target Root Filesystem
+
+Target filesystem is quite similar to buildchroot. The only difference is that it doesn't have development packages installed.
+
+Target filesystem lifecycle can be described as following:
+ - Target filesystem has initial configuration file which is passed to multistrap tool. This configuration file is generated by bitbake recipe from patterns and values defined by user. Based on this configuration file, multistrap generates initial filesystem.
+ - According to the list of custom packages in bitbake recipes, the initial filesystem will be populated by successfully built packages.
+
+# 3 Isar Internal Processes
+
+### 3.1 General Overview
+
+Whole Isar build process can be split into the following steps:
+ - Generation of initial buildchroots for each configuration (Debian distro, machine and architecture) requested by user.
+ - Generation of initial target filesystems for each configuration.
+ - Building custom packages.
+ - Populate target filesystems.
+ - Generate bootable images.
+
+All these steps are described in details below.
+
+### 3.2 Initial Buildchroot Generation
+
+As mentioned above, initial buildchroot is generated using multistrap. The bitbake recipe which is responsible for buildchroot can be found here: `meta/recipes-devtools/buildchroot/buildchroot.bb`
+
+This recipe implementes `do_build` task which performs the following:
+1. Generates multistrap config from template: `meta/recipes-devtools/buildchroot/files/multistrap.conf.in`
+2. Install pre/post scripts for multistrap: `meta/recipes-devtools/buildchroot/files/configscript.sh` and `meta/recipes-devtools/buildchroot/files/setup.sh`
+3. Run multistrap
+4. Install script for building custom Debian packages: `meta/recipes-devtools/buildchroot/files/build.sh`
+
+The single stamp is created for each user buildchroot configuration.
+
+### 3.3 Initial Target Filesystem Generation
+
+Initial target filesystem generation process is very similar to buildchroot creating, the difference is only in initial packages list.
+
+Target image recipes are the part of Isar core. There is a sample of typical Isar image that can be customized according to the user requirements: `meta-isar/recipes-core/images/isar-image-base.bb`
+Like for buildchroot, the multistrap configuration files for image can be found here: `meta-isar/recipes-core/images/files`, and it implements `do_build` task.
+
+### 3.4 Building Custom Packages
+
+Isar provides possibility to build Debian packages from sources. This features works with Debian-like source packages, i.e. the source code tree should contain debian folder. The build process is implemented in `meta/classes/dpkg.bbclass` and consists from the following steps:
+1. Task `do_fetch`: fetch source code from external link
+2. Task `do_unpack`: unpack source code to `${BUILDCHROOT_DIR}/home/build/${PN}`
+3. Task `do_build`: switch to buildchroot using chroot command and run `build.sh` script. The `build.sh` script performs the following:
+   1. Go to `/home/build/${PN}`
+   2. Get list of dependencies from debian/control and install them using apt.
+   3. Run dpkg-buildpackage
+4. Task `do_install`: install successfully built packages `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory `${DEPLOY_DIR_DEB}`
+
+### 3.5 Populate Target Filesystem
+
+Each target image can be extended by custom packages listed in IMAGE_INSTALL variable. Task `do_populate` performs the following:
+1. Parse IMAGE_INSTALL variable
+2. Find respective packages in `${DEPLOY_DIR_DEB}`
+3. Copy them to deb folder in dedicated target filesystem
+4. Execute dpkg command in chroot for all the copied packages
+
+### 3.6 Generate Bootable Image
+
+This process contains the following steps:
+1. Task `do_ext4_image`: target filesystem is packed to extfs image
+2. wic tool generates bootable image for dedicated platform
-- 
2.1.4

Re: [PATCH] doc: Add technical overview

[Henning Schild]

I did not really read this yet. At the moment i think we have too many
changes in the q to add such a document to the repository. Any change
would just mean heaving to keep it up to date.
We talked about the need for such a document for the possible OE
integration. I think it is valuable for that already.

That should be added after we all are happy with the feature-set of
Isar, until then it would just be another file to update when changing
how things work.

Henning

Am Tue, 8 Aug 2017 13:05:59 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Add initial Isar technical overview.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  doc/technical_overview.md | 110
> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110
> insertions(+) create mode 100644 doc/technical_overview.md
> 
> diff --git a/doc/technical_overview.md b/doc/technical_overview.md
> new file mode 100644
> index 0000000..f65fc08
> --- /dev/null
> +++ b/doc/technical_overview.md
> @@ -0,0 +1,110 @@
> +# 1 Introduction
> +
> +Isar can be shortly described as a set of bitbake recipes that
> implement main build system logic. To simplify overall Isar
> understanding, this document is split into two main parts:
> + - Isar logical components
> + - Isar internal processes
> +
> +# 2 Isar Logical Components
> +
> +In this chapter the most important Isar components are considered in
> details. In this text component doesn't especially mean some
> self-contained single entity of something, it's just an attempt to
> split Isar internals by various criteria. + +### 2.1 Bitbake and
> Recipes + +All the processes in Isar are started by bitbake, so it
> manages general build executing process. Recipes in Isar can be split
> in two categories: +
> + - System recipes and classes: they are responsible for setting up
> Debian-like infrastructure, manages Debian tools execution for
> package building and installation, generation root file system images
> + - User recipes: custom user applications, that should be builе from
> sources +
> +There are two types of dependencies in Isar:
> + - Dependency between bitbake recipes
> + - Dependencies in Debian filesystem
> +
> +**NOTE:** Isar doesn't manage dependencies in Debian file systems.
> User can only list specific Debian dependencies in recipe, so they
> eventually will be passed to apt-get or multistrap. Dependency
> installation is managed by Debian tools. + +### 2.2 Stamps +
> +Each task managed by bitbake uses stamp to notify that it has been
> completed. Due to Isar supports various Debian distributions and
> parallel builds for multiple machines and architectures, the stamps
> for tasks use special suffixes that include:
> + - Debian distro name
> + - Architecture
> + - Machine
> +
> +Typical example, when Isar builds the following configurations:
> + - Debian Jessie, amd64
> + - Debian Jessie, i386
> + - Debian Stretch, i386
> +
> +In this case there will be 3 different buildchroots, so standard
> hello demo application should be processed 3 times for each
> environment. Three different sets of stamps should be used for
> correct bitbake operating. + +### 2.3 Buildchroot +
> +One of the key aspect of Debian philosophy claims the fact, that
> everything in Debian should be built within Debian environment.
> Moreover native compilation is more preferable than
> cross-compilation. To follow this rules, Isar introduces the new
> component - buildchroot. Bulidchroot is typical Debian filesystem
> that is created using standard Debian tools: multistrap, apt. The
> source of packages can be either official Debian repositories or
> custom repositories created by user. + +Buildchroot lifecycle can be
> described as following:
> + - Buildchroot has initial configuration file which is passed to
> multistrap tool. This configuration file is generated by bitbake
> recipe from patterns and values defined by user. Based on this
> configuration file, multistrap generates initial filesystem.
> + - During building custom Debian package, list of its build
> dependencies is installed to buildchroot.
> + - When package has been built, it's installed to current
> buildchroot to satisfy further packages build dependencies. +
> +### 2.4 Target Root Filesystem
> +
> +Target filesystem is quite similar to buildchroot. The only
> difference is that it doesn't have development packages installed. +
> +Target filesystem lifecycle can be described as following:
> + - Target filesystem has initial configuration file which is passed
> to multistrap tool. This configuration file is generated by bitbake
> recipe from patterns and values defined by user. Based on this
> configuration file, multistrap generates initial filesystem.
> + - According to the list of custom packages in bitbake recipes, the
> initial filesystem will be populated by successfully built packages. +
> +# 3 Isar Internal Processes
> +
> +### 3.1 General Overview
> +
> +Whole Isar build process can be split into the following steps:
> + - Generation of initial buildchroots for each configuration (Debian
> distro, machine and architecture) requested by user.
> + - Generation of initial target filesystems for each configuration.
> + - Building custom packages.
> + - Populate target filesystems.
> + - Generate bootable images.
> +
> +All these steps are described in details below.
> +
> +### 3.2 Initial Buildchroot Generation
> +
> +As mentioned above, initial buildchroot is generated using
> multistrap. The bitbake recipe which is responsible for buildchroot
> can be found here: `meta/recipes-devtools/buildchroot/buildchroot.bb`
> + +This recipe implementes `do_build` task which performs the
> following: +1. Generates multistrap config from template:
> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in` +2.
> Install pre/post scripts for multistrap:
> `meta/recipes-devtools/buildchroot/files/configscript.sh` and
> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run multistrap
> +4. Install script for building custom Debian packages:
> `meta/recipes-devtools/buildchroot/files/build.sh` + +The single
> stamp is created for each user buildchroot configuration. + +### 3.3
> Initial Target Filesystem Generation + +Initial target filesystem
> generation process is very similar to buildchroot creating, the
> difference is only in initial packages list. + +Target image recipes
> are the part of Isar core. There is a sample of typical Isar image
> that can be customized according to the user requirements:
> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for
> buildchroot, the multistrap configuration files for image can be
> found here: `meta-isar/recipes-core/images/files`, and it implements
> `do_build` task. + +### 3.4 Building Custom Packages + +Isar provides
> possibility to build Debian packages from sources. This features
> works with Debian-like source packages, i.e. the source code tree
> should contain debian folder. The build process is implemented in
> `meta/classes/dpkg.bbclass` and consists from the following steps:
> +1. Task `do_fetch`: fetch source code from external link +2. Task
> `do_unpack`: unpack source code to
> `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task `do_build`: switch to
> buildchroot using chroot command and run `build.sh` script. The
> `build.sh` script performs the following:
> +   1. Go to `/home/build/${PN}`
> +   2. Get list of dependencies from debian/control and install them
> using apt.
> +   3. Run dpkg-buildpackage
> +4. Task `do_install`: install successfully built packages
> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory
> `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target Filesystem
> +
> +Each target image can be extended by custom packages listed in
> IMAGE_INSTALL variable. Task `do_populate` performs the following:
> +1. Parse IMAGE_INSTALL variable +2. Find respective packages in
> `${DEPLOY_DIR_DEB}` +3. Copy them to deb folder in dedicated target
> filesystem +4. Execute dpkg command in chroot for all the copied
> packages +
> +### 3.6 Generate Bootable Image
> +
> +This process contains the following steps:
> +1. Task `do_ext4_image`: target filesystem is packed to extfs image
> +2. wic tool generates bootable image for dedicated platform

Re: [PATCH] doc: Add technical overview

[Alexander Smirnov]

Hi,

On 08/08/2017 02:11 PM, Henning Schild wrote:
> I did not really read this yet. At the moment i think we have too many
> changes in the q to add such a document to the repository. Any change
> would just mean heaving to keep it up to date.
> We talked about the need for such a document for the possible OE
> integration. I think it is valuable for that already.
> 

Briefly speaking, this document describes initial motivation and 
high-level current overview.

> That should be added after we all are happy with the feature-set of
> Isar, until then it would just be another file to update when changing
> how things work.

Hmm, this document doesn't affect any discussions about Isar feature 
set. It's intended to be as technical background for hot-plug joining to 
this project. During discussions last month I see that we have gap in 
our understandings how Isar works, so this document should be start 
point to eliminate such gaps.

For sure, it's *not* a specification, which explicitly defines specific 
feature set and nothing more will be added. It's just an explanation of 
current state. I see no problems to regularly update it, moreover I 
highly appreciate this. Lack of documentation is the most common problem 
of open source projects, so as soon we start documenting the things, the 
easier will be documentation maintenance in future.

Alex

> 
> Am Tue, 8 Aug 2017 13:05:59 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Add initial Isar technical overview.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   doc/technical_overview.md | 110
>> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110
>> insertions(+) create mode 100644 doc/technical_overview.md
>>
>> diff --git a/doc/technical_overview.md b/doc/technical_overview.md
>> new file mode 100644
>> index 0000000..f65fc08
>> --- /dev/null
>> +++ b/doc/technical_overview.md
>> @@ -0,0 +1,110 @@
>> +# 1 Introduction
>> +
>> +Isar can be shortly described as a set of bitbake recipes that
>> implement main build system logic. To simplify overall Isar
>> understanding, this document is split into two main parts:
>> + - Isar logical components
>> + - Isar internal processes
>> +
>> +# 2 Isar Logical Components
>> +
>> +In this chapter the most important Isar components are considered in
>> details. In this text component doesn't especially mean some
>> self-contained single entity of something, it's just an attempt to
>> split Isar internals by various criteria. + +### 2.1 Bitbake and
>> Recipes + +All the processes in Isar are started by bitbake, so it
>> manages general build executing process. Recipes in Isar can be split
>> in two categories: +
>> + - System recipes and classes: they are responsible for setting up
>> Debian-like infrastructure, manages Debian tools execution for
>> package building and installation, generation root file system images
>> + - User recipes: custom user applications, that should be builе from
>> sources +
>> +There are two types of dependencies in Isar:
>> + - Dependency between bitbake recipes
>> + - Dependencies in Debian filesystem
>> +
>> +**NOTE:** Isar doesn't manage dependencies in Debian file systems.
>> User can only list specific Debian dependencies in recipe, so they
>> eventually will be passed to apt-get or multistrap. Dependency
>> installation is managed by Debian tools. + +### 2.2 Stamps +
>> +Each task managed by bitbake uses stamp to notify that it has been
>> completed. Due to Isar supports various Debian distributions and
>> parallel builds for multiple machines and architectures, the stamps
>> for tasks use special suffixes that include:
>> + - Debian distro name
>> + - Architecture
>> + - Machine
>> +
>> +Typical example, when Isar builds the following configurations:
>> + - Debian Jessie, amd64
>> + - Debian Jessie, i386
>> + - Debian Stretch, i386
>> +
>> +In this case there will be 3 different buildchroots, so standard
>> hello demo application should be processed 3 times for each
>> environment. Three different sets of stamps should be used for
>> correct bitbake operating. + +### 2.3 Buildchroot +
>> +One of the key aspect of Debian philosophy claims the fact, that
>> everything in Debian should be built within Debian environment.
>> Moreover native compilation is more preferable than
>> cross-compilation. To follow this rules, Isar introduces the new
>> component - buildchroot. Bulidchroot is typical Debian filesystem
>> that is created using standard Debian tools: multistrap, apt. The
>> source of packages can be either official Debian repositories or
>> custom repositories created by user. + +Buildchroot lifecycle can be
>> described as following:
>> + - Buildchroot has initial configuration file which is passed to
>> multistrap tool. This configuration file is generated by bitbake
>> recipe from patterns and values defined by user. Based on this
>> configuration file, multistrap generates initial filesystem.
>> + - During building custom Debian package, list of its build
>> dependencies is installed to buildchroot.
>> + - When package has been built, it's installed to current
>> buildchroot to satisfy further packages build dependencies. +
>> +### 2.4 Target Root Filesystem
>> +
>> +Target filesystem is quite similar to buildchroot. The only
>> difference is that it doesn't have development packages installed. +
>> +Target filesystem lifecycle can be described as following:
>> + - Target filesystem has initial configuration file which is passed
>> to multistrap tool. This configuration file is generated by bitbake
>> recipe from patterns and values defined by user. Based on this
>> configuration file, multistrap generates initial filesystem.
>> + - According to the list of custom packages in bitbake recipes, the
>> initial filesystem will be populated by successfully built packages. +
>> +# 3 Isar Internal Processes
>> +
>> +### 3.1 General Overview
>> +
>> +Whole Isar build process can be split into the following steps:
>> + - Generation of initial buildchroots for each configuration (Debian
>> distro, machine and architecture) requested by user.
>> + - Generation of initial target filesystems for each configuration.
>> + - Building custom packages.
>> + - Populate target filesystems.
>> + - Generate bootable images.
>> +
>> +All these steps are described in details below.
>> +
>> +### 3.2 Initial Buildchroot Generation
>> +
>> +As mentioned above, initial buildchroot is generated using
>> multistrap. The bitbake recipe which is responsible for buildchroot
>> can be found here: `meta/recipes-devtools/buildchroot/buildchroot.bb`
>> + +This recipe implementes `do_build` task which performs the
>> following: +1. Generates multistrap config from template:
>> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in` +2.
>> Install pre/post scripts for multistrap:
>> `meta/recipes-devtools/buildchroot/files/configscript.sh` and
>> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run multistrap
>> +4. Install script for building custom Debian packages:
>> `meta/recipes-devtools/buildchroot/files/build.sh` + +The single
>> stamp is created for each user buildchroot configuration. + +### 3.3
>> Initial Target Filesystem Generation + +Initial target filesystem
>> generation process is very similar to buildchroot creating, the
>> difference is only in initial packages list. + +Target image recipes
>> are the part of Isar core. There is a sample of typical Isar image
>> that can be customized according to the user requirements:
>> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for
>> buildchroot, the multistrap configuration files for image can be
>> found here: `meta-isar/recipes-core/images/files`, and it implements
>> `do_build` task. + +### 3.4 Building Custom Packages + +Isar provides
>> possibility to build Debian packages from sources. This features
>> works with Debian-like source packages, i.e. the source code tree
>> should contain debian folder. The build process is implemented in
>> `meta/classes/dpkg.bbclass` and consists from the following steps:
>> +1. Task `do_fetch`: fetch source code from external link +2. Task
>> `do_unpack`: unpack source code to
>> `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task `do_build`: switch to
>> buildchroot using chroot command and run `build.sh` script. The
>> `build.sh` script performs the following:
>> +   1. Go to `/home/build/${PN}`
>> +   2. Get list of dependencies from debian/control and install them
>> using apt.
>> +   3. Run dpkg-buildpackage
>> +4. Task `do_install`: install successfully built packages
>> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory
>> `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target Filesystem
>> +
>> +Each target image can be extended by custom packages listed in
>> IMAGE_INSTALL variable. Task `do_populate` performs the following:
>> +1. Parse IMAGE_INSTALL variable +2. Find respective packages in
>> `${DEPLOY_DIR_DEB}` +3. Copy them to deb folder in dedicated target
>> filesystem +4. Execute dpkg command in chroot for all the copied
>> packages +
>> +### 3.6 Generate Bootable Image
>> +
>> +This process contains the following steps:
>> +1. Task `do_ext4_image`: target filesystem is packed to extfs image
>> +2. wic tool generates bootable image for dedicated platform
> 

Re: [PATCH] doc: Add technical overview

[Baurzhan Ismagulov]

On Tue, Aug 08, 2017 at 01:11:02PM +0200, Henning Schild wrote:
> I did not really read this yet. At the moment i think we have too many
> changes in the q to add such a document to the repository. Any change
> would just mean heaving to keep it up to date.
> We talked about the need for such a document for the possible OE
> integration. I think it is valuable for that already.
> 
> That should be added after we all are happy with the feature-set of
> Isar, until then it would just be another file to update when changing
> how things work.

This is exactly one of the goals of documenting this at this stage. It is
important to document architecture changes, in small steps along the way.
Otherwise, later it will be difficult to understand the big picture and the
motivation for the changes from the many patches. I think that changing a
couple of paragraphs in the 110 lines of a really high-level overview shouldn't
be that difficult to do and is well worth it.

On a related note, my problem with some of the patches at this list is that I
see what the code does, but don't see the context and motivation. So, if a
patch requires a change in the document, providing it will only speed up the
review process.

With kind regards,
Baurzhan.

Re: [PATCH] doc: Add technical overview

[Henning Schild]

Am Tue, 8 Aug 2017 14:32:00 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Hi,
> 
> On 08/08/2017 02:11 PM, Henning Schild wrote:
> > I did not really read this yet. At the moment i think we have too
> > many changes in the q to add such a document to the repository. Any
> > change would just mean heaving to keep it up to date.
> > We talked about the need for such a document for the possible OE
> > integration. I think it is valuable for that already.
> >   
> 
> Briefly speaking, this document describes initial motivation and 
> high-level current overview.
> 
> > That should be added after we all are happy with the feature-set of
> > Isar, until then it would just be another file to update when
> > changing how things work.  
> 
> Hmm, this document doesn't affect any discussions about Isar feature 
> set. It's intended to be as technical background for hot-plug joining
> to this project. During discussions last month I see that we have gap
> in our understandings how Isar works, so this document should be
> start point to eliminate such gaps.
> 
> For sure, it's *not* a specification, which explicitly defines
> specific feature set and nothing more will be added. It's just an
> explanation of current state. I see no problems to regularly update
> it, moreover I highly appreciate this. Lack of documentation is the
> most common problem of open source projects, so as soon we start
> documenting the things, the easier will be documentation maintenance
> in future.

For sure we need that! But especially section 3 contains lots of
interna that will change if my patches get accepted. That is why i
would like to postpone adding the file after my patches went in. 

Adding that file before them would mean another rebase round for
documentation. I would be happy to rebase this one for you on top of a
master that contains my stuff, but not too happy about the other way
around.

Henning

> Alex
> 
> > 
> > Am Tue, 8 Aug 2017 13:05:59 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> Add initial Isar technical overview.
> >>
> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >> ---
> >>   doc/technical_overview.md | 110
> >> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110
> >> insertions(+) create mode 100644 doc/technical_overview.md
> >>
> >> diff --git a/doc/technical_overview.md b/doc/technical_overview.md
> >> new file mode 100644
> >> index 0000000..f65fc08
> >> --- /dev/null
> >> +++ b/doc/technical_overview.md
> >> @@ -0,0 +1,110 @@
> >> +# 1 Introduction
> >> +
> >> +Isar can be shortly described as a set of bitbake recipes that
> >> implement main build system logic. To simplify overall Isar
> >> understanding, this document is split into two main parts:
> >> + - Isar logical components
> >> + - Isar internal processes
> >> +
> >> +# 2 Isar Logical Components
> >> +
> >> +In this chapter the most important Isar components are considered
> >> in details. In this text component doesn't especially mean some
> >> self-contained single entity of something, it's just an attempt to
> >> split Isar internals by various criteria. + +### 2.1 Bitbake and
> >> Recipes + +All the processes in Isar are started by bitbake, so it
> >> manages general build executing process. Recipes in Isar can be
> >> split in two categories: +
> >> + - System recipes and classes: they are responsible for setting up
> >> Debian-like infrastructure, manages Debian tools execution for
> >> package building and installation, generation root file system
> >> images
> >> + - User recipes: custom user applications, that should be builе
> >> from sources +
> >> +There are two types of dependencies in Isar:
> >> + - Dependency between bitbake recipes
> >> + - Dependencies in Debian filesystem
> >> +
> >> +**NOTE:** Isar doesn't manage dependencies in Debian file systems.
> >> User can only list specific Debian dependencies in recipe, so they
> >> eventually will be passed to apt-get or multistrap. Dependency
> >> installation is managed by Debian tools. + +### 2.2 Stamps +
> >> +Each task managed by bitbake uses stamp to notify that it has been
> >> completed. Due to Isar supports various Debian distributions and
> >> parallel builds for multiple machines and architectures, the stamps
> >> for tasks use special suffixes that include:
> >> + - Debian distro name
> >> + - Architecture
> >> + - Machine
> >> +
> >> +Typical example, when Isar builds the following configurations:
> >> + - Debian Jessie, amd64
> >> + - Debian Jessie, i386
> >> + - Debian Stretch, i386
> >> +
> >> +In this case there will be 3 different buildchroots, so standard
> >> hello demo application should be processed 3 times for each
> >> environment. Three different sets of stamps should be used for
> >> correct bitbake operating. + +### 2.3 Buildchroot +
> >> +One of the key aspect of Debian philosophy claims the fact, that
> >> everything in Debian should be built within Debian environment.
> >> Moreover native compilation is more preferable than
> >> cross-compilation. To follow this rules, Isar introduces the new
> >> component - buildchroot. Bulidchroot is typical Debian filesystem
> >> that is created using standard Debian tools: multistrap, apt. The
> >> source of packages can be either official Debian repositories or
> >> custom repositories created by user. + +Buildchroot lifecycle can
> >> be described as following:
> >> + - Buildchroot has initial configuration file which is passed to
> >> multistrap tool. This configuration file is generated by bitbake
> >> recipe from patterns and values defined by user. Based on this
> >> configuration file, multistrap generates initial filesystem.
> >> + - During building custom Debian package, list of its build
> >> dependencies is installed to buildchroot.
> >> + - When package has been built, it's installed to current
> >> buildchroot to satisfy further packages build dependencies. +
> >> +### 2.4 Target Root Filesystem
> >> +
> >> +Target filesystem is quite similar to buildchroot. The only
> >> difference is that it doesn't have development packages installed.
> >> + +Target filesystem lifecycle can be described as following:
> >> + - Target filesystem has initial configuration file which is
> >> passed to multistrap tool. This configuration file is generated by
> >> bitbake recipe from patterns and values defined by user. Based on
> >> this configuration file, multistrap generates initial filesystem.
> >> + - According to the list of custom packages in bitbake recipes,
> >> the initial filesystem will be populated by successfully built
> >> packages. + +# 3 Isar Internal Processes
> >> +
> >> +### 3.1 General Overview
> >> +
> >> +Whole Isar build process can be split into the following steps:
> >> + - Generation of initial buildchroots for each configuration
> >> (Debian distro, machine and architecture) requested by user.
> >> + - Generation of initial target filesystems for each
> >> configuration.
> >> + - Building custom packages.
> >> + - Populate target filesystems.
> >> + - Generate bootable images.
> >> +
> >> +All these steps are described in details below.
> >> +
> >> +### 3.2 Initial Buildchroot Generation
> >> +
> >> +As mentioned above, initial buildchroot is generated using
> >> multistrap. The bitbake recipe which is responsible for buildchroot
> >> can be found here:
> >> `meta/recipes-devtools/buildchroot/buildchroot.bb`
> >> + +This recipe implementes `do_build` task which performs the
> >> following: +1. Generates multistrap config from template:
> >> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in` +2.
> >> Install pre/post scripts for multistrap:
> >> `meta/recipes-devtools/buildchroot/files/configscript.sh` and
> >> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run
> >> multistrap +4. Install script for building custom Debian packages:
> >> `meta/recipes-devtools/buildchroot/files/build.sh` + +The single
> >> stamp is created for each user buildchroot configuration. + +###
> >> 3.3 Initial Target Filesystem Generation + +Initial target
> >> filesystem generation process is very similar to buildchroot
> >> creating, the difference is only in initial packages list. +
> >> +Target image recipes are the part of Isar core. There is a sample
> >> of typical Isar image that can be customized according to the user
> >> requirements: `meta-isar/recipes-core/images/isar-image-base.bb`
> >> +Like for buildchroot, the multistrap configuration files for
> >> image can be found here: `meta-isar/recipes-core/images/files`,
> >> and it implements `do_build` task. + +### 3.4 Building Custom
> >> Packages + +Isar provides possibility to build Debian packages
> >> from sources. This features works with Debian-like source
> >> packages, i.e. the source code tree should contain debian folder.
> >> The build process is implemented in `meta/classes/dpkg.bbclass`
> >> and consists from the following steps: +1. Task `do_fetch`: fetch
> >> source code from external link +2. Task `do_unpack`: unpack source
> >> code to `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task `do_build`:
> >> switch to buildchroot using chroot command and run `build.sh`
> >> script. The `build.sh` script performs the following:
> >> +   1. Go to `/home/build/${PN}`
> >> +   2. Get list of dependencies from debian/control and install
> >> them using apt.
> >> +   3. Run dpkg-buildpackage
> >> +4. Task `do_install`: install successfully built packages
> >> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory
> >> `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target Filesystem
> >> +
> >> +Each target image can be extended by custom packages listed in
> >> IMAGE_INSTALL variable. Task `do_populate` performs the following:
> >> +1. Parse IMAGE_INSTALL variable +2. Find respective packages in
> >> `${DEPLOY_DIR_DEB}` +3. Copy them to deb folder in dedicated target
> >> filesystem +4. Execute dpkg command in chroot for all the copied
> >> packages +
> >> +### 3.6 Generate Bootable Image
> >> +
> >> +This process contains the following steps:
> >> +1. Task `do_ext4_image`: target filesystem is packed to extfs
> >> image +2. wic tool generates bootable image for dedicated
> >> platform  
> >   

Re: [PATCH] doc: Add technical overview

[Baurzhan Ismagulov]

On Tue, Aug 08, 2017 at 01:05:59PM +0300, Alexander Smirnov wrote:
> Add initial Isar technical overview.

I'd suggest to break lines at column 79 for plain text readers.


> +# 2 Isar Logical Components
...
> +### 2.1 Bitbake and Recipes

Why ### after #? I'd suggest using ## for right semantics, even if GitHub's
rendering is suboptimal here.


With kind regards,
Baurzhan.

Re: [PATCH] doc: Add technical overview

[Alexander Smirnov]

On 08/08/2017 02:54 PM, Henning Schild wrote:
> Am Tue, 8 Aug 2017 14:32:00 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Hi,
>>
>> On 08/08/2017 02:11 PM, Henning Schild wrote:
>>> I did not really read this yet. At the moment i think we have too
>>> many changes in the q to add such a document to the repository. Any
>>> change would just mean heaving to keep it up to date.
>>> We talked about the need for such a document for the possible OE
>>> integration. I think it is valuable for that already.
>>>    
>>
>> Briefly speaking, this document describes initial motivation and
>> high-level current overview.
>>
>>> That should be added after we all are happy with the feature-set of
>>> Isar, until then it would just be another file to update when
>>> changing how things work.
>>
>> Hmm, this document doesn't affect any discussions about Isar feature
>> set. It's intended to be as technical background for hot-plug joining
>> to this project. During discussions last month I see that we have gap
>> in our understandings how Isar works, so this document should be
>> start point to eliminate such gaps.
>>
>> For sure, it's *not* a specification, which explicitly defines
>> specific feature set and nothing more will be added. It's just an
>> explanation of current state. I see no problems to regularly update
>> it, moreover I highly appreciate this. Lack of documentation is the
>> most common problem of open source projects, so as soon we start
>> documenting the things, the easier will be documentation maintenance
>> in future.
> 
> For sure we need that! But especially section 3 contains lots of
> interna that will change if my patches get accepted. That is why i
> would like to postpone adding the file after my patches went in.

As I said, this document describes current project state, so I don't 
think that upcoming changes should block publishing of documentation.


> 
> Adding that file before them would mean another rebase round for
> documentation. I would be happy to rebase this one for you on top of a
> master that contains my stuff, but not too happy about the other way
> around.

The patch creates document from the scratch, so it can't conflict with 
any upcoming patches.

Also I expect, that the author of new features will update the 
documentation, so some reference document should be published anyway. To 
be honest I'm not happy with the idea that I'll document all the new 
features in future.

Alex

>>>
>>> Am Tue, 8 Aug 2017 13:05:59 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>    
>>>> Add initial Isar technical overview.
>>>>
>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>> ---
>>>>    doc/technical_overview.md | 110
>>>> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110
>>>> insertions(+) create mode 100644 doc/technical_overview.md
>>>>
>>>> diff --git a/doc/technical_overview.md b/doc/technical_overview.md
>>>> new file mode 100644
>>>> index 0000000..f65fc08
>>>> --- /dev/null
>>>> +++ b/doc/technical_overview.md
>>>> @@ -0,0 +1,110 @@
>>>> +# 1 Introduction
>>>> +
>>>> +Isar can be shortly described as a set of bitbake recipes that
>>>> implement main build system logic. To simplify overall Isar
>>>> understanding, this document is split into two main parts:
>>>> + - Isar logical components
>>>> + - Isar internal processes
>>>> +
>>>> +# 2 Isar Logical Components
>>>> +
>>>> +In this chapter the most important Isar components are considered
>>>> in details. In this text component doesn't especially mean some
>>>> self-contained single entity of something, it's just an attempt to
>>>> split Isar internals by various criteria. + +### 2.1 Bitbake and
>>>> Recipes + +All the processes in Isar are started by bitbake, so it
>>>> manages general build executing process. Recipes in Isar can be
>>>> split in two categories: +
>>>> + - System recipes and classes: they are responsible for setting up
>>>> Debian-like infrastructure, manages Debian tools execution for
>>>> package building and installation, generation root file system
>>>> images
>>>> + - User recipes: custom user applications, that should be builе
>>>> from sources +
>>>> +There are two types of dependencies in Isar:
>>>> + - Dependency between bitbake recipes
>>>> + - Dependencies in Debian filesystem
>>>> +
>>>> +**NOTE:** Isar doesn't manage dependencies in Debian file systems.
>>>> User can only list specific Debian dependencies in recipe, so they
>>>> eventually will be passed to apt-get or multistrap. Dependency
>>>> installation is managed by Debian tools. + +### 2.2 Stamps +
>>>> +Each task managed by bitbake uses stamp to notify that it has been
>>>> completed. Due to Isar supports various Debian distributions and
>>>> parallel builds for multiple machines and architectures, the stamps
>>>> for tasks use special suffixes that include:
>>>> + - Debian distro name
>>>> + - Architecture
>>>> + - Machine
>>>> +
>>>> +Typical example, when Isar builds the following configurations:
>>>> + - Debian Jessie, amd64
>>>> + - Debian Jessie, i386
>>>> + - Debian Stretch, i386
>>>> +
>>>> +In this case there will be 3 different buildchroots, so standard
>>>> hello demo application should be processed 3 times for each
>>>> environment. Three different sets of stamps should be used for
>>>> correct bitbake operating. + +### 2.3 Buildchroot +
>>>> +One of the key aspect of Debian philosophy claims the fact, that
>>>> everything in Debian should be built within Debian environment.
>>>> Moreover native compilation is more preferable than
>>>> cross-compilation. To follow this rules, Isar introduces the new
>>>> component - buildchroot. Bulidchroot is typical Debian filesystem
>>>> that is created using standard Debian tools: multistrap, apt. The
>>>> source of packages can be either official Debian repositories or
>>>> custom repositories created by user. + +Buildchroot lifecycle can
>>>> be described as following:
>>>> + - Buildchroot has initial configuration file which is passed to
>>>> multistrap tool. This configuration file is generated by bitbake
>>>> recipe from patterns and values defined by user. Based on this
>>>> configuration file, multistrap generates initial filesystem.
>>>> + - During building custom Debian package, list of its build
>>>> dependencies is installed to buildchroot.
>>>> + - When package has been built, it's installed to current
>>>> buildchroot to satisfy further packages build dependencies. +
>>>> +### 2.4 Target Root Filesystem
>>>> +
>>>> +Target filesystem is quite similar to buildchroot. The only
>>>> difference is that it doesn't have development packages installed.
>>>> + +Target filesystem lifecycle can be described as following:
>>>> + - Target filesystem has initial configuration file which is
>>>> passed to multistrap tool. This configuration file is generated by
>>>> bitbake recipe from patterns and values defined by user. Based on
>>>> this configuration file, multistrap generates initial filesystem.
>>>> + - According to the list of custom packages in bitbake recipes,
>>>> the initial filesystem will be populated by successfully built
>>>> packages. + +# 3 Isar Internal Processes
>>>> +
>>>> +### 3.1 General Overview
>>>> +
>>>> +Whole Isar build process can be split into the following steps:
>>>> + - Generation of initial buildchroots for each configuration
>>>> (Debian distro, machine and architecture) requested by user.
>>>> + - Generation of initial target filesystems for each
>>>> configuration.
>>>> + - Building custom packages.
>>>> + - Populate target filesystems.
>>>> + - Generate bootable images.
>>>> +
>>>> +All these steps are described in details below.
>>>> +
>>>> +### 3.2 Initial Buildchroot Generation
>>>> +
>>>> +As mentioned above, initial buildchroot is generated using
>>>> multistrap. The bitbake recipe which is responsible for buildchroot
>>>> can be found here:
>>>> `meta/recipes-devtools/buildchroot/buildchroot.bb`
>>>> + +This recipe implementes `do_build` task which performs the
>>>> following: +1. Generates multistrap config from template:
>>>> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in` +2.
>>>> Install pre/post scripts for multistrap:
>>>> `meta/recipes-devtools/buildchroot/files/configscript.sh` and
>>>> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run
>>>> multistrap +4. Install script for building custom Debian packages:
>>>> `meta/recipes-devtools/buildchroot/files/build.sh` + +The single
>>>> stamp is created for each user buildchroot configuration. + +###
>>>> 3.3 Initial Target Filesystem Generation + +Initial target
>>>> filesystem generation process is very similar to buildchroot
>>>> creating, the difference is only in initial packages list. +
>>>> +Target image recipes are the part of Isar core. There is a sample
>>>> of typical Isar image that can be customized according to the user
>>>> requirements: `meta-isar/recipes-core/images/isar-image-base.bb`
>>>> +Like for buildchroot, the multistrap configuration files for
>>>> image can be found here: `meta-isar/recipes-core/images/files`,
>>>> and it implements `do_build` task. + +### 3.4 Building Custom
>>>> Packages + +Isar provides possibility to build Debian packages
>>>> from sources. This features works with Debian-like source
>>>> packages, i.e. the source code tree should contain debian folder.
>>>> The build process is implemented in `meta/classes/dpkg.bbclass`
>>>> and consists from the following steps: +1. Task `do_fetch`: fetch
>>>> source code from external link +2. Task `do_unpack`: unpack source
>>>> code to `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task `do_build`:
>>>> switch to buildchroot using chroot command and run `build.sh`
>>>> script. The `build.sh` script performs the following:
>>>> +   1. Go to `/home/build/${PN}`
>>>> +   2. Get list of dependencies from debian/control and install
>>>> them using apt.
>>>> +   3. Run dpkg-buildpackage
>>>> +4. Task `do_install`: install successfully built packages
>>>> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory
>>>> `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target Filesystem
>>>> +
>>>> +Each target image can be extended by custom packages listed in
>>>> IMAGE_INSTALL variable. Task `do_populate` performs the following:
>>>> +1. Parse IMAGE_INSTALL variable +2. Find respective packages in
>>>> `${DEPLOY_DIR_DEB}` +3. Copy them to deb folder in dedicated target
>>>> filesystem +4. Execute dpkg command in chroot for all the copied
>>>> packages +
>>>> +### 3.6 Generate Bootable Image
>>>> +
>>>> +This process contains the following steps:
>>>> +1. Task `do_ext4_image`: target filesystem is packed to extfs
>>>> image +2. wic tool generates bootable image for dedicated
>>>> platform
>>>    
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH] doc: Add technical overview

[Henning Schild]

Am Tue, 8 Aug 2017 15:41:10 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 08/08/2017 02:54 PM, Henning Schild wrote:
> > Am Tue, 8 Aug 2017 14:32:00 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> Hi,
> >>
> >> On 08/08/2017 02:11 PM, Henning Schild wrote:  
> >>> I did not really read this yet. At the moment i think we have too
> >>> many changes in the q to add such a document to the repository.
> >>> Any change would just mean heaving to keep it up to date.
> >>> We talked about the need for such a document for the possible OE
> >>> integration. I think it is valuable for that already.
> >>>      
> >>
> >> Briefly speaking, this document describes initial motivation and
> >> high-level current overview.
> >>  
> >>> That should be added after we all are happy with the feature-set
> >>> of Isar, until then it would just be another file to update when
> >>> changing how things work.  
> >>
> >> Hmm, this document doesn't affect any discussions about Isar
> >> feature set. It's intended to be as technical background for
> >> hot-plug joining to this project. During discussions last month I
> >> see that we have gap in our understandings how Isar works, so this
> >> document should be start point to eliminate such gaps.
> >>
> >> For sure, it's *not* a specification, which explicitly defines
> >> specific feature set and nothing more will be added. It's just an
> >> explanation of current state. I see no problems to regularly update
> >> it, moreover I highly appreciate this. Lack of documentation is the
> >> most common problem of open source projects, so as soon we start
> >> documenting the things, the easier will be documentation
> >> maintenance in future.  
> > 
> > For sure we need that! But especially section 3 contains lots of
> > interna that will change if my patches get accepted. That is why i
> > would like to postpone adding the file after my patches went in.  
> 
> As I said, this document describes current project state, so I don't 
> think that upcoming changes should block publishing of documentation.
> 
> 
> > 
> > Adding that file before them would mean another rebase round for
> > documentation. I would be happy to rebase this one for you on top
> > of a master that contains my stuff, but not too happy about the
> > other way around.  
> 
> The patch creates document from the scratch, so it can't conflict
> with any upcoming patches.

It conflicts with other patches that are under review right now. All i
am asking is to merge conflicting patches in the order the patches
initially showed up for review, in that case this one comes last
anyways. And when its time has come it will be outdated, and i will
update it for you.

> Also I expect, that the author of new features will update the 
> documentation, so some reference document should be published anyway.

Sure.

> To be honest I'm not happy with the idea that I'll document all the
> new features in future.

Missing or inconsistent documentation are a valid reason to not accept
a patch so it will not come to that.

Henning

> Alex
> 
> >>>
> >>> Am Tue, 8 Aug 2017 13:05:59 +0300
> >>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >>>      
> >>>> Add initial Isar technical overview.
> >>>>
> >>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >>>> ---
> >>>>    doc/technical_overview.md | 110
> >>>> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed,
> >>>> 110 insertions(+) create mode 100644 doc/technical_overview.md
> >>>>
> >>>> diff --git a/doc/technical_overview.md
> >>>> b/doc/technical_overview.md new file mode 100644
> >>>> index 0000000..f65fc08
> >>>> --- /dev/null
> >>>> +++ b/doc/technical_overview.md
> >>>> @@ -0,0 +1,110 @@
> >>>> +# 1 Introduction
> >>>> +
> >>>> +Isar can be shortly described as a set of bitbake recipes that
> >>>> implement main build system logic. To simplify overall Isar
> >>>> understanding, this document is split into two main parts:
> >>>> + - Isar logical components
> >>>> + - Isar internal processes
> >>>> +
> >>>> +# 2 Isar Logical Components
> >>>> +
> >>>> +In this chapter the most important Isar components are
> >>>> considered in details. In this text component doesn't especially
> >>>> mean some self-contained single entity of something, it's just
> >>>> an attempt to split Isar internals by various criteria. + +###
> >>>> 2.1 Bitbake and Recipes + +All the processes in Isar are started
> >>>> by bitbake, so it manages general build executing process.
> >>>> Recipes in Isar can be split in two categories: +
> >>>> + - System recipes and classes: they are responsible for setting
> >>>> up Debian-like infrastructure, manages Debian tools execution for
> >>>> package building and installation, generation root file system
> >>>> images
> >>>> + - User recipes: custom user applications, that should be builе
> >>>> from sources +
> >>>> +There are two types of dependencies in Isar:
> >>>> + - Dependency between bitbake recipes
> >>>> + - Dependencies in Debian filesystem
> >>>> +
> >>>> +**NOTE:** Isar doesn't manage dependencies in Debian file
> >>>> systems. User can only list specific Debian dependencies in
> >>>> recipe, so they eventually will be passed to apt-get or
> >>>> multistrap. Dependency installation is managed by Debian tools.
> >>>> + +### 2.2 Stamps + +Each task managed by bitbake uses stamp to
> >>>> notify that it has been completed. Due to Isar supports various
> >>>> Debian distributions and parallel builds for multiple machines
> >>>> and architectures, the stamps for tasks use special suffixes
> >>>> that include:
> >>>> + - Debian distro name
> >>>> + - Architecture
> >>>> + - Machine
> >>>> +
> >>>> +Typical example, when Isar builds the following configurations:
> >>>> + - Debian Jessie, amd64
> >>>> + - Debian Jessie, i386
> >>>> + - Debian Stretch, i386
> >>>> +
> >>>> +In this case there will be 3 different buildchroots, so standard
> >>>> hello demo application should be processed 3 times for each
> >>>> environment. Three different sets of stamps should be used for
> >>>> correct bitbake operating. + +### 2.3 Buildchroot +
> >>>> +One of the key aspect of Debian philosophy claims the fact, that
> >>>> everything in Debian should be built within Debian environment.
> >>>> Moreover native compilation is more preferable than
> >>>> cross-compilation. To follow this rules, Isar introduces the new
> >>>> component - buildchroot. Bulidchroot is typical Debian filesystem
> >>>> that is created using standard Debian tools: multistrap, apt. The
> >>>> source of packages can be either official Debian repositories or
> >>>> custom repositories created by user. + +Buildchroot lifecycle can
> >>>> be described as following:
> >>>> + - Buildchroot has initial configuration file which is passed to
> >>>> multistrap tool. This configuration file is generated by bitbake
> >>>> recipe from patterns and values defined by user. Based on this
> >>>> configuration file, multistrap generates initial filesystem.
> >>>> + - During building custom Debian package, list of its build
> >>>> dependencies is installed to buildchroot.
> >>>> + - When package has been built, it's installed to current
> >>>> buildchroot to satisfy further packages build dependencies. +
> >>>> +### 2.4 Target Root Filesystem
> >>>> +
> >>>> +Target filesystem is quite similar to buildchroot. The only
> >>>> difference is that it doesn't have development packages
> >>>> installed.
> >>>> + +Target filesystem lifecycle can be described as following:
> >>>> + - Target filesystem has initial configuration file which is
> >>>> passed to multistrap tool. This configuration file is generated
> >>>> by bitbake recipe from patterns and values defined by user.
> >>>> Based on this configuration file, multistrap generates initial
> >>>> filesystem.
> >>>> + - According to the list of custom packages in bitbake recipes,
> >>>> the initial filesystem will be populated by successfully built
> >>>> packages. + +# 3 Isar Internal Processes
> >>>> +
> >>>> +### 3.1 General Overview
> >>>> +
> >>>> +Whole Isar build process can be split into the following steps:
> >>>> + - Generation of initial buildchroots for each configuration
> >>>> (Debian distro, machine and architecture) requested by user.
> >>>> + - Generation of initial target filesystems for each
> >>>> configuration.
> >>>> + - Building custom packages.
> >>>> + - Populate target filesystems.
> >>>> + - Generate bootable images.
> >>>> +
> >>>> +All these steps are described in details below.
> >>>> +
> >>>> +### 3.2 Initial Buildchroot Generation
> >>>> +
> >>>> +As mentioned above, initial buildchroot is generated using
> >>>> multistrap. The bitbake recipe which is responsible for
> >>>> buildchroot can be found here:
> >>>> `meta/recipes-devtools/buildchroot/buildchroot.bb`
> >>>> + +This recipe implementes `do_build` task which performs the
> >>>> following: +1. Generates multistrap config from template:
> >>>> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in` +2.
> >>>> Install pre/post scripts for multistrap:
> >>>> `meta/recipes-devtools/buildchroot/files/configscript.sh` and
> >>>> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run
> >>>> multistrap +4. Install script for building custom Debian
> >>>> packages: `meta/recipes-devtools/buildchroot/files/build.sh` +
> >>>> +The single stamp is created for each user buildchroot
> >>>> configuration. + +### 3.3 Initial Target Filesystem Generation +
> >>>> +Initial target filesystem generation process is very similar to
> >>>> buildchroot creating, the difference is only in initial packages
> >>>> list. + +Target image recipes are the part of Isar core. There
> >>>> is a sample of typical Isar image that can be customized
> >>>> according to the user requirements:
> >>>> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for
> >>>> buildchroot, the multistrap configuration files for image can be
> >>>> found here: `meta-isar/recipes-core/images/files`, and it
> >>>> implements `do_build` task. + +### 3.4 Building Custom Packages
> >>>> + +Isar provides possibility to build Debian packages from
> >>>> sources. This features works with Debian-like source packages,
> >>>> i.e. the source code tree should contain debian folder. The
> >>>> build process is implemented in `meta/classes/dpkg.bbclass` and
> >>>> consists from the following steps: +1. Task `do_fetch`: fetch
> >>>> source code from external link +2. Task `do_unpack`: unpack
> >>>> source code to `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task
> >>>> `do_build`: switch to buildchroot using chroot command and run
> >>>> `build.sh` script. The `build.sh` script performs the following:
> >>>> +   1. Go to `/home/build/${PN}`
> >>>> +   2. Get list of dependencies from debian/control and install
> >>>> them using apt.
> >>>> +   3. Run dpkg-buildpackage
> >>>> +4. Task `do_install`: install successfully built packages
> >>>> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory
> >>>> `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target Filesystem
> >>>> +
> >>>> +Each target image can be extended by custom packages listed in
> >>>> IMAGE_INSTALL variable. Task `do_populate` performs the
> >>>> following: +1. Parse IMAGE_INSTALL variable +2. Find respective
> >>>> packages in `${DEPLOY_DIR_DEB}` +3. Copy them to deb folder in
> >>>> dedicated target filesystem +4. Execute dpkg command in chroot
> >>>> for all the copied packages +
> >>>> +### 3.6 Generate Bootable Image
> >>>> +
> >>>> +This process contains the following steps:
> >>>> +1. Task `do_ext4_image`: target filesystem is packed to extfs
> >>>> image +2. wic tool generates bootable image for dedicated
> >>>> platform  
> >>>      
> >   
> 

Re: [PATCH] doc: Add technical overview

[Alexander Smirnov]

On 08/08/2017 04:03 PM, Henning Schild wrote:
> Am Tue, 8 Aug 2017 15:41:10 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 08/08/2017 02:54 PM, Henning Schild wrote:
>>> Am Tue, 8 Aug 2017 14:32:00 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>    
>>>> Hi,
>>>>
>>>> On 08/08/2017 02:11 PM, Henning Schild wrote:
>>>>> I did not really read this yet. At the moment i think we have too
>>>>> many changes in the q to add such a document to the repository.
>>>>> Any change would just mean heaving to keep it up to date.
>>>>> We talked about the need for such a document for the possible OE
>>>>> integration. I think it is valuable for that already.
>>>>>       
>>>>
>>>> Briefly speaking, this document describes initial motivation and
>>>> high-level current overview.
>>>>   
>>>>> That should be added after we all are happy with the feature-set
>>>>> of Isar, until then it would just be another file to update when
>>>>> changing how things work.
>>>>
>>>> Hmm, this document doesn't affect any discussions about Isar
>>>> feature set. It's intended to be as technical background for
>>>> hot-plug joining to this project. During discussions last month I
>>>> see that we have gap in our understandings how Isar works, so this
>>>> document should be start point to eliminate such gaps.
>>>>
>>>> For sure, it's *not* a specification, which explicitly defines
>>>> specific feature set and nothing more will be added. It's just an
>>>> explanation of current state. I see no problems to regularly update
>>>> it, moreover I highly appreciate this. Lack of documentation is the
>>>> most common problem of open source projects, so as soon we start
>>>> documenting the things, the easier will be documentation
>>>> maintenance in future.
>>>
>>> For sure we need that! But especially section 3 contains lots of
>>> interna that will change if my patches get accepted. That is why i
>>> would like to postpone adding the file after my patches went in.
>>
>> As I said, this document describes current project state, so I don't
>> think that upcoming changes should block publishing of documentation.
>>
>>
>>>
>>> Adding that file before them would mean another rebase round for
>>> documentation. I would be happy to rebase this one for you on top
>>> of a master that contains my stuff, but not too happy about the
>>> other way around.
>>
>> The patch creates document from the scratch, so it can't conflict
>> with any upcoming patches.
> 
> It conflicts with other patches that are under review right now. All i
> am asking is to merge conflicting patches in the order the patches
> initially showed up for review, in that case this one comes last
> anyways. And when its time has come it will be outdated, and i will
> update it for you.
> 

Hmm, are you speaking about exactly this patch or about previous series?

Alex

>> Also I expect, that the author of new features will update the
>> documentation, so some reference document should be published anyway.
> 
> Sure.
> 
>> To be honest I'm not happy with the idea that I'll document all the
>> new features in future.
> 
> Missing or inconsistent documentation are a valid reason to not accept
> a patch so it will not come to that.
> 
> Henning
> 
>> Alex
>>
>>>>>
>>>>> Am Tue, 8 Aug 2017 13:05:59 +0300
>>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>>>       
>>>>>> Add initial Isar technical overview.
>>>>>>
>>>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>>>> ---
>>>>>>     doc/technical_overview.md | 110
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed,
>>>>>> 110 insertions(+) create mode 100644 doc/technical_overview.md
>>>>>>
>>>>>> diff --git a/doc/technical_overview.md
>>>>>> b/doc/technical_overview.md new file mode 100644
>>>>>> index 0000000..f65fc08
>>>>>> --- /dev/null
>>>>>> +++ b/doc/technical_overview.md
>>>>>> @@ -0,0 +1,110 @@
>>>>>> +# 1 Introduction
>>>>>> +
>>>>>> +Isar can be shortly described as a set of bitbake recipes that
>>>>>> implement main build system logic. To simplify overall Isar
>>>>>> understanding, this document is split into two main parts:
>>>>>> + - Isar logical components
>>>>>> + - Isar internal processes
>>>>>> +
>>>>>> +# 2 Isar Logical Components
>>>>>> +
>>>>>> +In this chapter the most important Isar components are
>>>>>> considered in details. In this text component doesn't especially
>>>>>> mean some self-contained single entity of something, it's just
>>>>>> an attempt to split Isar internals by various criteria. + +###
>>>>>> 2.1 Bitbake and Recipes + +All the processes in Isar are started
>>>>>> by bitbake, so it manages general build executing process.
>>>>>> Recipes in Isar can be split in two categories: +
>>>>>> + - System recipes and classes: they are responsible for setting
>>>>>> up Debian-like infrastructure, manages Debian tools execution for
>>>>>> package building and installation, generation root file system
>>>>>> images
>>>>>> + - User recipes: custom user applications, that should be builе
>>>>>> from sources +
>>>>>> +There are two types of dependencies in Isar:
>>>>>> + - Dependency between bitbake recipes
>>>>>> + - Dependencies in Debian filesystem
>>>>>> +
>>>>>> +**NOTE:** Isar doesn't manage dependencies in Debian file
>>>>>> systems. User can only list specific Debian dependencies in
>>>>>> recipe, so they eventually will be passed to apt-get or
>>>>>> multistrap. Dependency installation is managed by Debian tools.
>>>>>> + +### 2.2 Stamps + +Each task managed by bitbake uses stamp to
>>>>>> notify that it has been completed. Due to Isar supports various
>>>>>> Debian distributions and parallel builds for multiple machines
>>>>>> and architectures, the stamps for tasks use special suffixes
>>>>>> that include:
>>>>>> + - Debian distro name
>>>>>> + - Architecture
>>>>>> + - Machine
>>>>>> +
>>>>>> +Typical example, when Isar builds the following configurations:
>>>>>> + - Debian Jessie, amd64
>>>>>> + - Debian Jessie, i386
>>>>>> + - Debian Stretch, i386
>>>>>> +
>>>>>> +In this case there will be 3 different buildchroots, so standard
>>>>>> hello demo application should be processed 3 times for each
>>>>>> environment. Three different sets of stamps should be used for
>>>>>> correct bitbake operating. + +### 2.3 Buildchroot +
>>>>>> +One of the key aspect of Debian philosophy claims the fact, that
>>>>>> everything in Debian should be built within Debian environment.
>>>>>> Moreover native compilation is more preferable than
>>>>>> cross-compilation. To follow this rules, Isar introduces the new
>>>>>> component - buildchroot. Bulidchroot is typical Debian filesystem
>>>>>> that is created using standard Debian tools: multistrap, apt. The
>>>>>> source of packages can be either official Debian repositories or
>>>>>> custom repositories created by user. + +Buildchroot lifecycle can
>>>>>> be described as following:
>>>>>> + - Buildchroot has initial configuration file which is passed to
>>>>>> multistrap tool. This configuration file is generated by bitbake
>>>>>> recipe from patterns and values defined by user. Based on this
>>>>>> configuration file, multistrap generates initial filesystem.
>>>>>> + - During building custom Debian package, list of its build
>>>>>> dependencies is installed to buildchroot.
>>>>>> + - When package has been built, it's installed to current
>>>>>> buildchroot to satisfy further packages build dependencies. +
>>>>>> +### 2.4 Target Root Filesystem
>>>>>> +
>>>>>> +Target filesystem is quite similar to buildchroot. The only
>>>>>> difference is that it doesn't have development packages
>>>>>> installed.
>>>>>> + +Target filesystem lifecycle can be described as following:
>>>>>> + - Target filesystem has initial configuration file which is
>>>>>> passed to multistrap tool. This configuration file is generated
>>>>>> by bitbake recipe from patterns and values defined by user.
>>>>>> Based on this configuration file, multistrap generates initial
>>>>>> filesystem.
>>>>>> + - According to the list of custom packages in bitbake recipes,
>>>>>> the initial filesystem will be populated by successfully built
>>>>>> packages. + +# 3 Isar Internal Processes
>>>>>> +
>>>>>> +### 3.1 General Overview
>>>>>> +
>>>>>> +Whole Isar build process can be split into the following steps:
>>>>>> + - Generation of initial buildchroots for each configuration
>>>>>> (Debian distro, machine and architecture) requested by user.
>>>>>> + - Generation of initial target filesystems for each
>>>>>> configuration.
>>>>>> + - Building custom packages.
>>>>>> + - Populate target filesystems.
>>>>>> + - Generate bootable images.
>>>>>> +
>>>>>> +All these steps are described in details below.
>>>>>> +
>>>>>> +### 3.2 Initial Buildchroot Generation
>>>>>> +
>>>>>> +As mentioned above, initial buildchroot is generated using
>>>>>> multistrap. The bitbake recipe which is responsible for
>>>>>> buildchroot can be found here:
>>>>>> `meta/recipes-devtools/buildchroot/buildchroot.bb`
>>>>>> + +This recipe implementes `do_build` task which performs the
>>>>>> following: +1. Generates multistrap config from template:
>>>>>> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in` +2.
>>>>>> Install pre/post scripts for multistrap:
>>>>>> `meta/recipes-devtools/buildchroot/files/configscript.sh` and
>>>>>> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run
>>>>>> multistrap +4. Install script for building custom Debian
>>>>>> packages: `meta/recipes-devtools/buildchroot/files/build.sh` +
>>>>>> +The single stamp is created for each user buildchroot
>>>>>> configuration. + +### 3.3 Initial Target Filesystem Generation +
>>>>>> +Initial target filesystem generation process is very similar to
>>>>>> buildchroot creating, the difference is only in initial packages
>>>>>> list. + +Target image recipes are the part of Isar core. There
>>>>>> is a sample of typical Isar image that can be customized
>>>>>> according to the user requirements:
>>>>>> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for
>>>>>> buildchroot, the multistrap configuration files for image can be
>>>>>> found here: `meta-isar/recipes-core/images/files`, and it
>>>>>> implements `do_build` task. + +### 3.4 Building Custom Packages
>>>>>> + +Isar provides possibility to build Debian packages from
>>>>>> sources. This features works with Debian-like source packages,
>>>>>> i.e. the source code tree should contain debian folder. The
>>>>>> build process is implemented in `meta/classes/dpkg.bbclass` and
>>>>>> consists from the following steps: +1. Task `do_fetch`: fetch
>>>>>> source code from external link +2. Task `do_unpack`: unpack
>>>>>> source code to `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task
>>>>>> `do_build`: switch to buildchroot using chroot command and run
>>>>>> `build.sh` script. The `build.sh` script performs the following:
>>>>>> +   1. Go to `/home/build/${PN}`
>>>>>> +   2. Get list of dependencies from debian/control and install
>>>>>> them using apt.
>>>>>> +   3. Run dpkg-buildpackage
>>>>>> +4. Task `do_install`: install successfully built packages
>>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory
>>>>>> `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target Filesystem
>>>>>> +
>>>>>> +Each target image can be extended by custom packages listed in
>>>>>> IMAGE_INSTALL variable. Task `do_populate` performs the
>>>>>> following: +1. Parse IMAGE_INSTALL variable +2. Find respective
>>>>>> packages in `${DEPLOY_DIR_DEB}` +3. Copy them to deb folder in
>>>>>> dedicated target filesystem +4. Execute dpkg command in chroot
>>>>>> for all the copied packages +
>>>>>> +### 3.6 Generate Bootable Image
>>>>>> +
>>>>>> +This process contains the following steps:
>>>>>> +1. Task `do_ext4_image`: target filesystem is packed to extfs
>>>>>> image +2. wic tool generates bootable image for dedicated
>>>>>> platform
>>>>>       
>>>    
>>
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH] doc: Add technical overview

[Henning Schild]

Am Tue, 8 Aug 2017 16:12:02 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 08/08/2017 04:03 PM, Henning Schild wrote:
> > Am Tue, 8 Aug 2017 15:41:10 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> On 08/08/2017 02:54 PM, Henning Schild wrote:  
> >>> Am Tue, 8 Aug 2017 14:32:00 +0300
> >>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >>>      
> >>>> Hi,
> >>>>
> >>>> On 08/08/2017 02:11 PM, Henning Schild wrote:  
> >>>>> I did not really read this yet. At the moment i think we have
> >>>>> too many changes in the q to add such a document to the
> >>>>> repository. Any change would just mean heaving to keep it up to
> >>>>> date. We talked about the need for such a document for the
> >>>>> possible OE integration. I think it is valuable for that
> >>>>> already. 
> >>>>
> >>>> Briefly speaking, this document describes initial motivation and
> >>>> high-level current overview.
> >>>>     
> >>>>> That should be added after we all are happy with the feature-set
> >>>>> of Isar, until then it would just be another file to update when
> >>>>> changing how things work.  
> >>>>
> >>>> Hmm, this document doesn't affect any discussions about Isar
> >>>> feature set. It's intended to be as technical background for
> >>>> hot-plug joining to this project. During discussions last month I
> >>>> see that we have gap in our understandings how Isar works, so
> >>>> this document should be start point to eliminate such gaps.
> >>>>
> >>>> For sure, it's *not* a specification, which explicitly defines
> >>>> specific feature set and nothing more will be added. It's just an
> >>>> explanation of current state. I see no problems to regularly
> >>>> update it, moreover I highly appreciate this. Lack of
> >>>> documentation is the most common problem of open source
> >>>> projects, so as soon we start documenting the things, the easier
> >>>> will be documentation maintenance in future.  
> >>>
> >>> For sure we need that! But especially section 3 contains lots of
> >>> interna that will change if my patches get accepted. That is why i
> >>> would like to postpone adding the file after my patches went in.  
> >>
> >> As I said, this document describes current project state, so I
> >> don't think that upcoming changes should block publishing of
> >> documentation.
> >>
> >>  
> >>>
> >>> Adding that file before them would mean another rebase round for
> >>> documentation. I would be happy to rebase this one for you on top
> >>> of a master that contains my stuff, but not too happy about the
> >>> other way around.  
> >>
> >> The patch creates document from the scratch, so it can't conflict
> >> with any upcoming patches.  
> > 
> > It conflicts with other patches that are under review right now.
> > All i am asking is to merge conflicting patches in the order the
> > patches initially showed up for review, in that case this one comes
> > last anyways. And when its time has come it will be outdated, and i
> > will update it for you.
> >   
> 
> Hmm, are you speaking about exactly this patch or about previous
> series?

Section 3.4 and 3.5 conflict with my changes, they will have to be
changed. If your patch comes first i would have to rebase.

i.e.
- 3.4 talks about dpkg.bbclass which i renamed
[PATCH 11-16 of 16 v2 4/6]
- 3.5 will not happen anymore if everything enters the rootfs through
  multistrap
[PATCH 3/3] classes: image: remove populate and replace it with a
custom repo


Henning


> Alex
> 
> >> Also I expect, that the author of new features will update the
> >> documentation, so some reference document should be published
> >> anyway.  
> > 
> > Sure.
> >   
> >> To be honest I'm not happy with the idea that I'll document all the
> >> new features in future.  
> > 
> > Missing or inconsistent documentation are a valid reason to not
> > accept a patch so it will not come to that.
> > 
> > Henning
> >   
> >> Alex
> >>  
> >>>>>
> >>>>> Am Tue, 8 Aug 2017 13:05:59 +0300
> >>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >>>>>         
> >>>>>> Add initial Isar technical overview.
> >>>>>>
> >>>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >>>>>> ---
> >>>>>>     doc/technical_overview.md | 110
> >>>>>> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed,
> >>>>>> 110 insertions(+) create mode 100644 doc/technical_overview.md
> >>>>>>
> >>>>>> diff --git a/doc/technical_overview.md
> >>>>>> b/doc/technical_overview.md new file mode 100644
> >>>>>> index 0000000..f65fc08
> >>>>>> --- /dev/null
> >>>>>> +++ b/doc/technical_overview.md
> >>>>>> @@ -0,0 +1,110 @@
> >>>>>> +# 1 Introduction
> >>>>>> +
> >>>>>> +Isar can be shortly described as a set of bitbake recipes that
> >>>>>> implement main build system logic. To simplify overall Isar
> >>>>>> understanding, this document is split into two main parts:
> >>>>>> + - Isar logical components
> >>>>>> + - Isar internal processes
> >>>>>> +
> >>>>>> +# 2 Isar Logical Components
> >>>>>> +
> >>>>>> +In this chapter the most important Isar components are
> >>>>>> considered in details. In this text component doesn't
> >>>>>> especially mean some self-contained single entity of
> >>>>>> something, it's just an attempt to split Isar internals by
> >>>>>> various criteria. + +### 2.1 Bitbake and Recipes + +All the
> >>>>>> processes in Isar are started by bitbake, so it manages
> >>>>>> general build executing process. Recipes in Isar can be split
> >>>>>> in two categories: +
> >>>>>> + - System recipes and classes: they are responsible for
> >>>>>> setting up Debian-like infrastructure, manages Debian tools
> >>>>>> execution for package building and installation, generation
> >>>>>> root file system images
> >>>>>> + - User recipes: custom user applications, that should be
> >>>>>> builе from sources +
> >>>>>> +There are two types of dependencies in Isar:
> >>>>>> + - Dependency between bitbake recipes
> >>>>>> + - Dependencies in Debian filesystem
> >>>>>> +
> >>>>>> +**NOTE:** Isar doesn't manage dependencies in Debian file
> >>>>>> systems. User can only list specific Debian dependencies in
> >>>>>> recipe, so they eventually will be passed to apt-get or
> >>>>>> multistrap. Dependency installation is managed by Debian tools.
> >>>>>> + +### 2.2 Stamps + +Each task managed by bitbake uses stamp to
> >>>>>> notify that it has been completed. Due to Isar supports various
> >>>>>> Debian distributions and parallel builds for multiple machines
> >>>>>> and architectures, the stamps for tasks use special suffixes
> >>>>>> that include:
> >>>>>> + - Debian distro name
> >>>>>> + - Architecture
> >>>>>> + - Machine
> >>>>>> +
> >>>>>> +Typical example, when Isar builds the following
> >>>>>> configurations:
> >>>>>> + - Debian Jessie, amd64
> >>>>>> + - Debian Jessie, i386
> >>>>>> + - Debian Stretch, i386
> >>>>>> +
> >>>>>> +In this case there will be 3 different buildchroots, so
> >>>>>> standard hello demo application should be processed 3 times
> >>>>>> for each environment. Three different sets of stamps should be
> >>>>>> used for correct bitbake operating. + +### 2.3 Buildchroot +
> >>>>>> +One of the key aspect of Debian philosophy claims the fact,
> >>>>>> that everything in Debian should be built within Debian
> >>>>>> environment. Moreover native compilation is more preferable
> >>>>>> than cross-compilation. To follow this rules, Isar introduces
> >>>>>> the new component - buildchroot. Bulidchroot is typical Debian
> >>>>>> filesystem that is created using standard Debian tools:
> >>>>>> multistrap, apt. The source of packages can be either official
> >>>>>> Debian repositories or custom repositories created by user. +
> >>>>>> +Buildchroot lifecycle can be described as following:
> >>>>>> + - Buildchroot has initial configuration file which is passed
> >>>>>> to multistrap tool. This configuration file is generated by
> >>>>>> bitbake recipe from patterns and values defined by user. Based
> >>>>>> on this configuration file, multistrap generates initial
> >>>>>> filesystem.
> >>>>>> + - During building custom Debian package, list of its build
> >>>>>> dependencies is installed to buildchroot.
> >>>>>> + - When package has been built, it's installed to current
> >>>>>> buildchroot to satisfy further packages build dependencies. +
> >>>>>> +### 2.4 Target Root Filesystem
> >>>>>> +
> >>>>>> +Target filesystem is quite similar to buildchroot. The only
> >>>>>> difference is that it doesn't have development packages
> >>>>>> installed.
> >>>>>> + +Target filesystem lifecycle can be described as following:
> >>>>>> + - Target filesystem has initial configuration file which is
> >>>>>> passed to multistrap tool. This configuration file is generated
> >>>>>> by bitbake recipe from patterns and values defined by user.
> >>>>>> Based on this configuration file, multistrap generates initial
> >>>>>> filesystem.
> >>>>>> + - According to the list of custom packages in bitbake
> >>>>>> recipes, the initial filesystem will be populated by
> >>>>>> successfully built packages. + +# 3 Isar Internal Processes
> >>>>>> +
> >>>>>> +### 3.1 General Overview
> >>>>>> +
> >>>>>> +Whole Isar build process can be split into the following
> >>>>>> steps:
> >>>>>> + - Generation of initial buildchroots for each configuration
> >>>>>> (Debian distro, machine and architecture) requested by user.
> >>>>>> + - Generation of initial target filesystems for each
> >>>>>> configuration.
> >>>>>> + - Building custom packages.
> >>>>>> + - Populate target filesystems.
> >>>>>> + - Generate bootable images.
> >>>>>> +
> >>>>>> +All these steps are described in details below.
> >>>>>> +
> >>>>>> +### 3.2 Initial Buildchroot Generation
> >>>>>> +
> >>>>>> +As mentioned above, initial buildchroot is generated using
> >>>>>> multistrap. The bitbake recipe which is responsible for
> >>>>>> buildchroot can be found here:
> >>>>>> `meta/recipes-devtools/buildchroot/buildchroot.bb`
> >>>>>> + +This recipe implementes `do_build` task which performs the
> >>>>>> following: +1. Generates multistrap config from template:
> >>>>>> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in`
> >>>>>> +2. Install pre/post scripts for multistrap:
> >>>>>> `meta/recipes-devtools/buildchroot/files/configscript.sh` and
> >>>>>> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run
> >>>>>> multistrap +4. Install script for building custom Debian
> >>>>>> packages: `meta/recipes-devtools/buildchroot/files/build.sh` +
> >>>>>> +The single stamp is created for each user buildchroot
> >>>>>> configuration. + +### 3.3 Initial Target Filesystem Generation
> >>>>>> + +Initial target filesystem generation process is very
> >>>>>> similar to buildchroot creating, the difference is only in
> >>>>>> initial packages list. + +Target image recipes are the part of
> >>>>>> Isar core. There is a sample of typical Isar image that can be
> >>>>>> customized according to the user requirements:
> >>>>>> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for
> >>>>>> buildchroot, the multistrap configuration files for image can
> >>>>>> be found here: `meta-isar/recipes-core/images/files`, and it
> >>>>>> implements `do_build` task. + +### 3.4 Building Custom Packages
> >>>>>> + +Isar provides possibility to build Debian packages from
> >>>>>> sources. This features works with Debian-like source packages,
> >>>>>> i.e. the source code tree should contain debian folder. The
> >>>>>> build process is implemented in `meta/classes/dpkg.bbclass` and
> >>>>>> consists from the following steps: +1. Task `do_fetch`: fetch
> >>>>>> source code from external link +2. Task `do_unpack`: unpack
> >>>>>> source code to `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task
> >>>>>> `do_build`: switch to buildchroot using chroot command and run
> >>>>>> `build.sh` script. The `build.sh` script performs the
> >>>>>> following:
> >>>>>> +   1. Go to `/home/build/${PN}`
> >>>>>> +   2. Get list of dependencies from debian/control and install
> >>>>>> them using apt.
> >>>>>> +   3. Run dpkg-buildpackage
> >>>>>> +4. Task `do_install`: install successfully built packages
> >>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory
> >>>>>> `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target Filesystem
> >>>>>> +
> >>>>>> +Each target image can be extended by custom packages listed in
> >>>>>> IMAGE_INSTALL variable. Task `do_populate` performs the
> >>>>>> following: +1. Parse IMAGE_INSTALL variable +2. Find respective
> >>>>>> packages in `${DEPLOY_DIR_DEB}` +3. Copy them to deb folder in
> >>>>>> dedicated target filesystem +4. Execute dpkg command in chroot
> >>>>>> for all the copied packages +
> >>>>>> +### 3.6 Generate Bootable Image
> >>>>>> +
> >>>>>> +This process contains the following steps:
> >>>>>> +1. Task `do_ext4_image`: target filesystem is packed to extfs
> >>>>>> image +2. wic tool generates bootable image for dedicated
> >>>>>> platform  
> >>>>>         
> >>>      
> >>  
> >   
> 

Re: [PATCH] doc: Add technical overview

[Alexander Smirnov]

On 08/08/2017 05:38 PM, Henning Schild wrote:
> Am Tue, 8 Aug 2017 16:12:02 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 08/08/2017 04:03 PM, Henning Schild wrote:
>>> Am Tue, 8 Aug 2017 15:41:10 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>    
>>>> On 08/08/2017 02:54 PM, Henning Schild wrote:
>>>>> Am Tue, 8 Aug 2017 14:32:00 +0300
>>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>>>       
>>>>>> Hi,
>>>>>>
>>>>>> On 08/08/2017 02:11 PM, Henning Schild wrote:
>>>>>>> I did not really read this yet. At the moment i think we have
>>>>>>> too many changes in the q to add such a document to the
>>>>>>> repository. Any change would just mean heaving to keep it up to
>>>>>>> date. We talked about the need for such a document for the
>>>>>>> possible OE integration. I think it is valuable for that
>>>>>>> already.
>>>>>>
>>>>>> Briefly speaking, this document describes initial motivation and
>>>>>> high-level current overview.
>>>>>>      
>>>>>>> That should be added after we all are happy with the feature-set
>>>>>>> of Isar, until then it would just be another file to update when
>>>>>>> changing how things work.
>>>>>>
>>>>>> Hmm, this document doesn't affect any discussions about Isar
>>>>>> feature set. It's intended to be as technical background for
>>>>>> hot-plug joining to this project. During discussions last month I
>>>>>> see that we have gap in our understandings how Isar works, so
>>>>>> this document should be start point to eliminate such gaps.
>>>>>>
>>>>>> For sure, it's *not* a specification, which explicitly defines
>>>>>> specific feature set and nothing more will be added. It's just an
>>>>>> explanation of current state. I see no problems to regularly
>>>>>> update it, moreover I highly appreciate this. Lack of
>>>>>> documentation is the most common problem of open source
>>>>>> projects, so as soon we start documenting the things, the easier
>>>>>> will be documentation maintenance in future.
>>>>>
>>>>> For sure we need that! But especially section 3 contains lots of
>>>>> interna that will change if my patches get accepted. That is why i
>>>>> would like to postpone adding the file after my patches went in.
>>>>
>>>> As I said, this document describes current project state, so I
>>>> don't think that upcoming changes should block publishing of
>>>> documentation.
>>>>
>>>>   
>>>>>
>>>>> Adding that file before them would mean another rebase round for
>>>>> documentation. I would be happy to rebase this one for you on top
>>>>> of a master that contains my stuff, but not too happy about the
>>>>> other way around.
>>>>
>>>> The patch creates document from the scratch, so it can't conflict
>>>> with any upcoming patches.
>>>
>>> It conflicts with other patches that are under review right now.
>>> All i am asking is to merge conflicting patches in the order the
>>> patches initially showed up for review, in that case this one comes
>>> last anyways. And when its time has come it will be outdated, and i
>>> will update it for you.
>>>    
>>
>> Hmm, are you speaking about exactly this patch or about previous
>> series?
> 
> Section 3.4 and 3.5 conflict with my changes, they will have to be
> changed. If your patch comes first i would have to rebase.
> 
> i.e.
> - 3.4 talks about dpkg.bbclass which i renamed
> [PATCH 11-16 of 16 v2 4/6]
> - 3.5 will not happen anymore if everything enters the rootfs through
>    multistrap
> [PATCH 3/3] classes: image: remove populate and replace it with a
> custom repo

I didn't get the idea, how this affects your patches? After your 
features will be applied, you can update this document. In this thread 
you've mentioned, that this patch breaks your series and force you to 
perform rebase, this point is not clear for me.

> 
> 
> Henning
> 
> 
>> Alex
>>
>>>> Also I expect, that the author of new features will update the
>>>> documentation, so some reference document should be published
>>>> anyway.
>>>
>>> Sure.
>>>    
>>>> To be honest I'm not happy with the idea that I'll document all the
>>>> new features in future.
>>>
>>> Missing or inconsistent documentation are a valid reason to not
>>> accept a patch so it will not come to that.
>>>
>>> Henning
>>>    
>>>> Alex
>>>>   
>>>>>>>
>>>>>>> Am Tue, 8 Aug 2017 13:05:59 +0300
>>>>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>>>>>          
>>>>>>>> Add initial Isar technical overview.
>>>>>>>>
>>>>>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>>>>>> ---
>>>>>>>>      doc/technical_overview.md | 110
>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed,
>>>>>>>> 110 insertions(+) create mode 100644 doc/technical_overview.md
>>>>>>>>
>>>>>>>> diff --git a/doc/technical_overview.md
>>>>>>>> b/doc/technical_overview.md new file mode 100644
>>>>>>>> index 0000000..f65fc08
>>>>>>>> --- /dev/null
>>>>>>>> +++ b/doc/technical_overview.md
>>>>>>>> @@ -0,0 +1,110 @@
>>>>>>>> +# 1 Introduction
>>>>>>>> +
>>>>>>>> +Isar can be shortly described as a set of bitbake recipes that
>>>>>>>> implement main build system logic. To simplify overall Isar
>>>>>>>> understanding, this document is split into two main parts:
>>>>>>>> + - Isar logical components
>>>>>>>> + - Isar internal processes
>>>>>>>> +
>>>>>>>> +# 2 Isar Logical Components
>>>>>>>> +
>>>>>>>> +In this chapter the most important Isar components are
>>>>>>>> considered in details. In this text component doesn't
>>>>>>>> especially mean some self-contained single entity of
>>>>>>>> something, it's just an attempt to split Isar internals by
>>>>>>>> various criteria. + +### 2.1 Bitbake and Recipes + +All the
>>>>>>>> processes in Isar are started by bitbake, so it manages
>>>>>>>> general build executing process. Recipes in Isar can be split
>>>>>>>> in two categories: +
>>>>>>>> + - System recipes and classes: they are responsible for
>>>>>>>> setting up Debian-like infrastructure, manages Debian tools
>>>>>>>> execution for package building and installation, generation
>>>>>>>> root file system images
>>>>>>>> + - User recipes: custom user applications, that should be
>>>>>>>> builе from sources +
>>>>>>>> +There are two types of dependencies in Isar:
>>>>>>>> + - Dependency between bitbake recipes
>>>>>>>> + - Dependencies in Debian filesystem
>>>>>>>> +
>>>>>>>> +**NOTE:** Isar doesn't manage dependencies in Debian file
>>>>>>>> systems. User can only list specific Debian dependencies in
>>>>>>>> recipe, so they eventually will be passed to apt-get or
>>>>>>>> multistrap. Dependency installation is managed by Debian tools.
>>>>>>>> + +### 2.2 Stamps + +Each task managed by bitbake uses stamp to
>>>>>>>> notify that it has been completed. Due to Isar supports various
>>>>>>>> Debian distributions and parallel builds for multiple machines
>>>>>>>> and architectures, the stamps for tasks use special suffixes
>>>>>>>> that include:
>>>>>>>> + - Debian distro name
>>>>>>>> + - Architecture
>>>>>>>> + - Machine
>>>>>>>> +
>>>>>>>> +Typical example, when Isar builds the following
>>>>>>>> configurations:
>>>>>>>> + - Debian Jessie, amd64
>>>>>>>> + - Debian Jessie, i386
>>>>>>>> + - Debian Stretch, i386
>>>>>>>> +
>>>>>>>> +In this case there will be 3 different buildchroots, so
>>>>>>>> standard hello demo application should be processed 3 times
>>>>>>>> for each environment. Three different sets of stamps should be
>>>>>>>> used for correct bitbake operating. + +### 2.3 Buildchroot +
>>>>>>>> +One of the key aspect of Debian philosophy claims the fact,
>>>>>>>> that everything in Debian should be built within Debian
>>>>>>>> environment. Moreover native compilation is more preferable
>>>>>>>> than cross-compilation. To follow this rules, Isar introduces
>>>>>>>> the new component - buildchroot. Bulidchroot is typical Debian
>>>>>>>> filesystem that is created using standard Debian tools:
>>>>>>>> multistrap, apt. The source of packages can be either official
>>>>>>>> Debian repositories or custom repositories created by user. +
>>>>>>>> +Buildchroot lifecycle can be described as following:
>>>>>>>> + - Buildchroot has initial configuration file which is passed
>>>>>>>> to multistrap tool. This configuration file is generated by
>>>>>>>> bitbake recipe from patterns and values defined by user. Based
>>>>>>>> on this configuration file, multistrap generates initial
>>>>>>>> filesystem.
>>>>>>>> + - During building custom Debian package, list of its build
>>>>>>>> dependencies is installed to buildchroot.
>>>>>>>> + - When package has been built, it's installed to current
>>>>>>>> buildchroot to satisfy further packages build dependencies. +
>>>>>>>> +### 2.4 Target Root Filesystem
>>>>>>>> +
>>>>>>>> +Target filesystem is quite similar to buildchroot. The only
>>>>>>>> difference is that it doesn't have development packages
>>>>>>>> installed.
>>>>>>>> + +Target filesystem lifecycle can be described as following:
>>>>>>>> + - Target filesystem has initial configuration file which is
>>>>>>>> passed to multistrap tool. This configuration file is generated
>>>>>>>> by bitbake recipe from patterns and values defined by user.
>>>>>>>> Based on this configuration file, multistrap generates initial
>>>>>>>> filesystem.
>>>>>>>> + - According to the list of custom packages in bitbake
>>>>>>>> recipes, the initial filesystem will be populated by
>>>>>>>> successfully built packages. + +# 3 Isar Internal Processes
>>>>>>>> +
>>>>>>>> +### 3.1 General Overview
>>>>>>>> +
>>>>>>>> +Whole Isar build process can be split into the following
>>>>>>>> steps:
>>>>>>>> + - Generation of initial buildchroots for each configuration
>>>>>>>> (Debian distro, machine and architecture) requested by user.
>>>>>>>> + - Generation of initial target filesystems for each
>>>>>>>> configuration.
>>>>>>>> + - Building custom packages.
>>>>>>>> + - Populate target filesystems.
>>>>>>>> + - Generate bootable images.
>>>>>>>> +
>>>>>>>> +All these steps are described in details below.
>>>>>>>> +
>>>>>>>> +### 3.2 Initial Buildchroot Generation
>>>>>>>> +
>>>>>>>> +As mentioned above, initial buildchroot is generated using
>>>>>>>> multistrap. The bitbake recipe which is responsible for
>>>>>>>> buildchroot can be found here:
>>>>>>>> `meta/recipes-devtools/buildchroot/buildchroot.bb`
>>>>>>>> + +This recipe implementes `do_build` task which performs the
>>>>>>>> following: +1. Generates multistrap config from template:
>>>>>>>> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in`
>>>>>>>> +2. Install pre/post scripts for multistrap:
>>>>>>>> `meta/recipes-devtools/buildchroot/files/configscript.sh` and
>>>>>>>> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run
>>>>>>>> multistrap +4. Install script for building custom Debian
>>>>>>>> packages: `meta/recipes-devtools/buildchroot/files/build.sh` +
>>>>>>>> +The single stamp is created for each user buildchroot
>>>>>>>> configuration. + +### 3.3 Initial Target Filesystem Generation
>>>>>>>> + +Initial target filesystem generation process is very
>>>>>>>> similar to buildchroot creating, the difference is only in
>>>>>>>> initial packages list. + +Target image recipes are the part of
>>>>>>>> Isar core. There is a sample of typical Isar image that can be
>>>>>>>> customized according to the user requirements:
>>>>>>>> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for
>>>>>>>> buildchroot, the multistrap configuration files for image can
>>>>>>>> be found here: `meta-isar/recipes-core/images/files`, and it
>>>>>>>> implements `do_build` task. + +### 3.4 Building Custom Packages
>>>>>>>> + +Isar provides possibility to build Debian packages from
>>>>>>>> sources. This features works with Debian-like source packages,
>>>>>>>> i.e. the source code tree should contain debian folder. The
>>>>>>>> build process is implemented in `meta/classes/dpkg.bbclass` and
>>>>>>>> consists from the following steps: +1. Task `do_fetch`: fetch
>>>>>>>> source code from external link +2. Task `do_unpack`: unpack
>>>>>>>> source code to `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task
>>>>>>>> `do_build`: switch to buildchroot using chroot command and run
>>>>>>>> `build.sh` script. The `build.sh` script performs the
>>>>>>>> following:
>>>>>>>> +   1. Go to `/home/build/${PN}`
>>>>>>>> +   2. Get list of dependencies from debian/control and install
>>>>>>>> them using apt.
>>>>>>>> +   3. Run dpkg-buildpackage
>>>>>>>> +4. Task `do_install`: install successfully built packages
>>>>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory
>>>>>>>> `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target Filesystem
>>>>>>>> +
>>>>>>>> +Each target image can be extended by custom packages listed in
>>>>>>>> IMAGE_INSTALL variable. Task `do_populate` performs the
>>>>>>>> following: +1. Parse IMAGE_INSTALL variable +2. Find respective
>>>>>>>> packages in `${DEPLOY_DIR_DEB}` +3. Copy them to deb folder in
>>>>>>>> dedicated target filesystem +4. Execute dpkg command in chroot
>>>>>>>> for all the copied packages +
>>>>>>>> +### 3.6 Generate Bootable Image
>>>>>>>> +
>>>>>>>> +This process contains the following steps:
>>>>>>>> +1. Task `do_ext4_image`: target filesystem is packed to extfs
>>>>>>>> image +2. wic tool generates bootable image for dedicated
>>>>>>>> platform
>>>>>>>          
>>>>>       
>>>>   
>>>    
>>
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH] doc: Add technical overview

[Henning Schild]

Am Tue, 8 Aug 2017 17:44:30 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 08/08/2017 05:38 PM, Henning Schild wrote:
> > Am Tue, 8 Aug 2017 16:12:02 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> On 08/08/2017 04:03 PM, Henning Schild wrote:  
> >>> Am Tue, 8 Aug 2017 15:41:10 +0300
> >>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >>>      
> >>>> On 08/08/2017 02:54 PM, Henning Schild wrote:  
> >>>>> Am Tue, 8 Aug 2017 14:32:00 +0300
> >>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >>>>>         
> >>>>>> Hi,
> >>>>>>
> >>>>>> On 08/08/2017 02:11 PM, Henning Schild wrote:  
> >>>>>>> I did not really read this yet. At the moment i think we have
> >>>>>>> too many changes in the q to add such a document to the
> >>>>>>> repository. Any change would just mean heaving to keep it up
> >>>>>>> to date. We talked about the need for such a document for the
> >>>>>>> possible OE integration. I think it is valuable for that
> >>>>>>> already.  
> >>>>>>
> >>>>>> Briefly speaking, this document describes initial motivation
> >>>>>> and high-level current overview.
> >>>>>>        
> >>>>>>> That should be added after we all are happy with the
> >>>>>>> feature-set of Isar, until then it would just be another file
> >>>>>>> to update when changing how things work.  
> >>>>>>
> >>>>>> Hmm, this document doesn't affect any discussions about Isar
> >>>>>> feature set. It's intended to be as technical background for
> >>>>>> hot-plug joining to this project. During discussions last
> >>>>>> month I see that we have gap in our understandings how Isar
> >>>>>> works, so this document should be start point to eliminate
> >>>>>> such gaps.
> >>>>>>
> >>>>>> For sure, it's *not* a specification, which explicitly defines
> >>>>>> specific feature set and nothing more will be added. It's just
> >>>>>> an explanation of current state. I see no problems to regularly
> >>>>>> update it, moreover I highly appreciate this. Lack of
> >>>>>> documentation is the most common problem of open source
> >>>>>> projects, so as soon we start documenting the things, the
> >>>>>> easier will be documentation maintenance in future.  
> >>>>>
> >>>>> For sure we need that! But especially section 3 contains lots of
> >>>>> interna that will change if my patches get accepted. That is
> >>>>> why i would like to postpone adding the file after my patches
> >>>>> went in.  
> >>>>
> >>>> As I said, this document describes current project state, so I
> >>>> don't think that upcoming changes should block publishing of
> >>>> documentation.
> >>>>
> >>>>     
> >>>>>
> >>>>> Adding that file before them would mean another rebase round for
> >>>>> documentation. I would be happy to rebase this one for you on
> >>>>> top of a master that contains my stuff, but not too happy about
> >>>>> the other way around.  
> >>>>
> >>>> The patch creates document from the scratch, so it can't conflict
> >>>> with any upcoming patches.  
> >>>
> >>> It conflicts with other patches that are under review right now.
> >>> All i am asking is to merge conflicting patches in the order the
> >>> patches initially showed up for review, in that case this one
> >>> comes last anyways. And when its time has come it will be
> >>> outdated, and i will update it for you.
> >>>      
> >>
> >> Hmm, are you speaking about exactly this patch or about previous
> >> series?  
> > 
> > Section 3.4 and 3.5 conflict with my changes, they will have to be
> > changed. If your patch comes first i would have to rebase.
> > 
> > i.e.
> > - 3.4 talks about dpkg.bbclass which i renamed
> > [PATCH 11-16 of 16 v2 4/6]
> > - 3.5 will not happen anymore if everything enters the rootfs
> > through multistrap
> > [PATCH 3/3] classes: image: remove populate and replace it with a
> > custom repo  
> 
> I didn't get the idea, how this affects your patches? After your 
> features will be applied, you can update this document. In this
> thread you've mentioned, that this patch breaks your series and force
> you to perform rebase, this point is not clear for me.

A consistent patch updates the code along with the documentation. If
the code i am touching suddenly gets documented my commits will become
inconsistent.

Henning

> > 
> > 
> > Henning
> > 
> >   
> >> Alex
> >>  
> >>>> Also I expect, that the author of new features will update the
> >>>> documentation, so some reference document should be published
> >>>> anyway.  
> >>>
> >>> Sure.
> >>>      
> >>>> To be honest I'm not happy with the idea that I'll document all
> >>>> the new features in future.  
> >>>
> >>> Missing or inconsistent documentation are a valid reason to not
> >>> accept a patch so it will not come to that.
> >>>
> >>> Henning
> >>>      
> >>>> Alex
> >>>>     
> >>>>>>>
> >>>>>>> Am Tue, 8 Aug 2017 13:05:59 +0300
> >>>>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >>>>>>>            
> >>>>>>>> Add initial Isar technical overview.
> >>>>>>>>
> >>>>>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >>>>>>>> ---
> >>>>>>>>      doc/technical_overview.md | 110
> >>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file
> >>>>>>>> changed, 110 insertions(+) create mode 100644
> >>>>>>>> doc/technical_overview.md
> >>>>>>>>
> >>>>>>>> diff --git a/doc/technical_overview.md
> >>>>>>>> b/doc/technical_overview.md new file mode 100644
> >>>>>>>> index 0000000..f65fc08
> >>>>>>>> --- /dev/null
> >>>>>>>> +++ b/doc/technical_overview.md
> >>>>>>>> @@ -0,0 +1,110 @@
> >>>>>>>> +# 1 Introduction
> >>>>>>>> +
> >>>>>>>> +Isar can be shortly described as a set of bitbake recipes
> >>>>>>>> that implement main build system logic. To simplify overall
> >>>>>>>> Isar understanding, this document is split into two main
> >>>>>>>> parts:
> >>>>>>>> + - Isar logical components
> >>>>>>>> + - Isar internal processes
> >>>>>>>> +
> >>>>>>>> +# 2 Isar Logical Components
> >>>>>>>> +
> >>>>>>>> +In this chapter the most important Isar components are
> >>>>>>>> considered in details. In this text component doesn't
> >>>>>>>> especially mean some self-contained single entity of
> >>>>>>>> something, it's just an attempt to split Isar internals by
> >>>>>>>> various criteria. + +### 2.1 Bitbake and Recipes + +All the
> >>>>>>>> processes in Isar are started by bitbake, so it manages
> >>>>>>>> general build executing process. Recipes in Isar can be split
> >>>>>>>> in two categories: +
> >>>>>>>> + - System recipes and classes: they are responsible for
> >>>>>>>> setting up Debian-like infrastructure, manages Debian tools
> >>>>>>>> execution for package building and installation, generation
> >>>>>>>> root file system images
> >>>>>>>> + - User recipes: custom user applications, that should be
> >>>>>>>> builе from sources +
> >>>>>>>> +There are two types of dependencies in Isar:
> >>>>>>>> + - Dependency between bitbake recipes
> >>>>>>>> + - Dependencies in Debian filesystem
> >>>>>>>> +
> >>>>>>>> +**NOTE:** Isar doesn't manage dependencies in Debian file
> >>>>>>>> systems. User can only list specific Debian dependencies in
> >>>>>>>> recipe, so they eventually will be passed to apt-get or
> >>>>>>>> multistrap. Dependency installation is managed by Debian
> >>>>>>>> tools.
> >>>>>>>> + +### 2.2 Stamps + +Each task managed by bitbake uses stamp
> >>>>>>>> to notify that it has been completed. Due to Isar supports
> >>>>>>>> various Debian distributions and parallel builds for
> >>>>>>>> multiple machines and architectures, the stamps for tasks
> >>>>>>>> use special suffixes that include:
> >>>>>>>> + - Debian distro name
> >>>>>>>> + - Architecture
> >>>>>>>> + - Machine
> >>>>>>>> +
> >>>>>>>> +Typical example, when Isar builds the following
> >>>>>>>> configurations:
> >>>>>>>> + - Debian Jessie, amd64
> >>>>>>>> + - Debian Jessie, i386
> >>>>>>>> + - Debian Stretch, i386
> >>>>>>>> +
> >>>>>>>> +In this case there will be 3 different buildchroots, so
> >>>>>>>> standard hello demo application should be processed 3 times
> >>>>>>>> for each environment. Three different sets of stamps should
> >>>>>>>> be used for correct bitbake operating. + +### 2.3
> >>>>>>>> Buildchroot + +One of the key aspect of Debian philosophy
> >>>>>>>> claims the fact, that everything in Debian should be built
> >>>>>>>> within Debian environment. Moreover native compilation is
> >>>>>>>> more preferable than cross-compilation. To follow this
> >>>>>>>> rules, Isar introduces the new component - buildchroot.
> >>>>>>>> Bulidchroot is typical Debian filesystem that is created
> >>>>>>>> using standard Debian tools: multistrap, apt. The source of
> >>>>>>>> packages can be either official Debian repositories or
> >>>>>>>> custom repositories created by user. + +Buildchroot
> >>>>>>>> lifecycle can be described as following:
> >>>>>>>> + - Buildchroot has initial configuration file which is
> >>>>>>>> passed to multistrap tool. This configuration file is
> >>>>>>>> generated by bitbake recipe from patterns and values defined
> >>>>>>>> by user. Based on this configuration file, multistrap
> >>>>>>>> generates initial filesystem.
> >>>>>>>> + - During building custom Debian package, list of its build
> >>>>>>>> dependencies is installed to buildchroot.
> >>>>>>>> + - When package has been built, it's installed to current
> >>>>>>>> buildchroot to satisfy further packages build dependencies. +
> >>>>>>>> +### 2.4 Target Root Filesystem
> >>>>>>>> +
> >>>>>>>> +Target filesystem is quite similar to buildchroot. The only
> >>>>>>>> difference is that it doesn't have development packages
> >>>>>>>> installed.
> >>>>>>>> + +Target filesystem lifecycle can be described as following:
> >>>>>>>> + - Target filesystem has initial configuration file which is
> >>>>>>>> passed to multistrap tool. This configuration file is
> >>>>>>>> generated by bitbake recipe from patterns and values defined
> >>>>>>>> by user. Based on this configuration file, multistrap
> >>>>>>>> generates initial filesystem.
> >>>>>>>> + - According to the list of custom packages in bitbake
> >>>>>>>> recipes, the initial filesystem will be populated by
> >>>>>>>> successfully built packages. + +# 3 Isar Internal Processes
> >>>>>>>> +
> >>>>>>>> +### 3.1 General Overview
> >>>>>>>> +
> >>>>>>>> +Whole Isar build process can be split into the following
> >>>>>>>> steps:
> >>>>>>>> + - Generation of initial buildchroots for each configuration
> >>>>>>>> (Debian distro, machine and architecture) requested by user.
> >>>>>>>> + - Generation of initial target filesystems for each
> >>>>>>>> configuration.
> >>>>>>>> + - Building custom packages.
> >>>>>>>> + - Populate target filesystems.
> >>>>>>>> + - Generate bootable images.
> >>>>>>>> +
> >>>>>>>> +All these steps are described in details below.
> >>>>>>>> +
> >>>>>>>> +### 3.2 Initial Buildchroot Generation
> >>>>>>>> +
> >>>>>>>> +As mentioned above, initial buildchroot is generated using
> >>>>>>>> multistrap. The bitbake recipe which is responsible for
> >>>>>>>> buildchroot can be found here:
> >>>>>>>> `meta/recipes-devtools/buildchroot/buildchroot.bb`
> >>>>>>>> + +This recipe implementes `do_build` task which performs the
> >>>>>>>> following: +1. Generates multistrap config from template:
> >>>>>>>> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in`
> >>>>>>>> +2. Install pre/post scripts for multistrap:
> >>>>>>>> `meta/recipes-devtools/buildchroot/files/configscript.sh` and
> >>>>>>>> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run
> >>>>>>>> multistrap +4. Install script for building custom Debian
> >>>>>>>> packages: `meta/recipes-devtools/buildchroot/files/build.sh`
> >>>>>>>> + +The single stamp is created for each user buildchroot
> >>>>>>>> configuration. + +### 3.3 Initial Target Filesystem
> >>>>>>>> Generation
> >>>>>>>> + +Initial target filesystem generation process is very
> >>>>>>>> similar to buildchroot creating, the difference is only in
> >>>>>>>> initial packages list. + +Target image recipes are the part
> >>>>>>>> of Isar core. There is a sample of typical Isar image that
> >>>>>>>> can be customized according to the user requirements:
> >>>>>>>> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for
> >>>>>>>> buildchroot, the multistrap configuration files for image can
> >>>>>>>> be found here: `meta-isar/recipes-core/images/files`, and it
> >>>>>>>> implements `do_build` task. + +### 3.4 Building Custom
> >>>>>>>> Packages
> >>>>>>>> + +Isar provides possibility to build Debian packages from
> >>>>>>>> sources. This features works with Debian-like source
> >>>>>>>> packages, i.e. the source code tree should contain debian
> >>>>>>>> folder. The build process is implemented in
> >>>>>>>> `meta/classes/dpkg.bbclass` and consists from the following
> >>>>>>>> steps: +1. Task `do_fetch`: fetch source code from external
> >>>>>>>> link +2. Task `do_unpack`: unpack source code to
> >>>>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task `do_build`:
> >>>>>>>> switch to buildchroot using chroot command and run
> >>>>>>>> `build.sh` script. The `build.sh` script performs the
> >>>>>>>> following:
> >>>>>>>> +   1. Go to `/home/build/${PN}`
> >>>>>>>> +   2. Get list of dependencies from debian/control and
> >>>>>>>> install them using apt.
> >>>>>>>> +   3. Run dpkg-buildpackage
> >>>>>>>> +4. Task `do_install`: install successfully built packages
> >>>>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy
> >>>>>>>> directory `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target
> >>>>>>>> Filesystem +
> >>>>>>>> +Each target image can be extended by custom packages listed
> >>>>>>>> in IMAGE_INSTALL variable. Task `do_populate` performs the
> >>>>>>>> following: +1. Parse IMAGE_INSTALL variable +2. Find
> >>>>>>>> respective packages in `${DEPLOY_DIR_DEB}` +3. Copy them to
> >>>>>>>> deb folder in dedicated target filesystem +4. Execute dpkg
> >>>>>>>> command in chroot for all the copied packages +
> >>>>>>>> +### 3.6 Generate Bootable Image
> >>>>>>>> +
> >>>>>>>> +This process contains the following steps:
> >>>>>>>> +1. Task `do_ext4_image`: target filesystem is packed to
> >>>>>>>> extfs image +2. wic tool generates bootable image for
> >>>>>>>> dedicated platform  
> >>>>>>>            
> >>>>>         
> >>>>     
> >>>      
> >>  
> >   
> 

Re: [PATCH] doc: Add technical overview

[Alexander Smirnov]

On 08/08/2017 05:53 PM, Henning Schild wrote:
> Am Tue, 8 Aug 2017 17:44:30 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 08/08/2017 05:38 PM, Henning Schild wrote:
>>> Am Tue, 8 Aug 2017 16:12:02 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>    
>>>> On 08/08/2017 04:03 PM, Henning Schild wrote:
>>>>> Am Tue, 8 Aug 2017 15:41:10 +0300
>>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>>>       
>>>>>> On 08/08/2017 02:54 PM, Henning Schild wrote:
>>>>>>> Am Tue, 8 Aug 2017 14:32:00 +0300
>>>>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>>>>>          
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On 08/08/2017 02:11 PM, Henning Schild wrote:
>>>>>>>>> I did not really read this yet. At the moment i think we have
>>>>>>>>> too many changes in the q to add such a document to the
>>>>>>>>> repository. Any change would just mean heaving to keep it up
>>>>>>>>> to date. We talked about the need for such a document for the
>>>>>>>>> possible OE integration. I think it is valuable for that
>>>>>>>>> already.
>>>>>>>>
>>>>>>>> Briefly speaking, this document describes initial motivation
>>>>>>>> and high-level current overview.
>>>>>>>>         
>>>>>>>>> That should be added after we all are happy with the
>>>>>>>>> feature-set of Isar, until then it would just be another file
>>>>>>>>> to update when changing how things work.
>>>>>>>>
>>>>>>>> Hmm, this document doesn't affect any discussions about Isar
>>>>>>>> feature set. It's intended to be as technical background for
>>>>>>>> hot-plug joining to this project. During discussions last
>>>>>>>> month I see that we have gap in our understandings how Isar
>>>>>>>> works, so this document should be start point to eliminate
>>>>>>>> such gaps.
>>>>>>>>
>>>>>>>> For sure, it's *not* a specification, which explicitly defines
>>>>>>>> specific feature set and nothing more will be added. It's just
>>>>>>>> an explanation of current state. I see no problems to regularly
>>>>>>>> update it, moreover I highly appreciate this. Lack of
>>>>>>>> documentation is the most common problem of open source
>>>>>>>> projects, so as soon we start documenting the things, the
>>>>>>>> easier will be documentation maintenance in future.
>>>>>>>
>>>>>>> For sure we need that! But especially section 3 contains lots of
>>>>>>> interna that will change if my patches get accepted. That is
>>>>>>> why i would like to postpone adding the file after my patches
>>>>>>> went in.
>>>>>>
>>>>>> As I said, this document describes current project state, so I
>>>>>> don't think that upcoming changes should block publishing of
>>>>>> documentation.
>>>>>>
>>>>>>      
>>>>>>>
>>>>>>> Adding that file before them would mean another rebase round for
>>>>>>> documentation. I would be happy to rebase this one for you on
>>>>>>> top of a master that contains my stuff, but not too happy about
>>>>>>> the other way around.
>>>>>>
>>>>>> The patch creates document from the scratch, so it can't conflict
>>>>>> with any upcoming patches.
>>>>>
>>>>> It conflicts with other patches that are under review right now.
>>>>> All i am asking is to merge conflicting patches in the order the
>>>>> patches initially showed up for review, in that case this one
>>>>> comes last anyways. And when its time has come it will be
>>>>> outdated, and i will update it for you.
>>>>>       
>>>>
>>>> Hmm, are you speaking about exactly this patch or about previous
>>>> series?
>>>
>>> Section 3.4 and 3.5 conflict with my changes, they will have to be
>>> changed. If your patch comes first i would have to rebase.
>>>
>>> i.e.
>>> - 3.4 talks about dpkg.bbclass which i renamed
>>> [PATCH 11-16 of 16 v2 4/6]
>>> - 3.5 will not happen anymore if everything enters the rootfs
>>> through multistrap
>>> [PATCH 3/3] classes: image: remove populate and replace it with a
>>> custom repo
>>
>> I didn't get the idea, how this affects your patches? After your
>> features will be applied, you can update this document. In this
>> thread you've mentioned, that this patch breaks your series and force
>> you to perform rebase, this point is not clear for me.
> 
> A consistent patch updates the code along with the documentation. If
> the code i am touching suddenly gets documented my commits will become
> inconsistent.

You can update the documentation later after your series will be applied.

Alex

>>>
>>> Henning
>>>
>>>    
>>>> Alex
>>>>   
>>>>>> Also I expect, that the author of new features will update the
>>>>>> documentation, so some reference document should be published
>>>>>> anyway.
>>>>>
>>>>> Sure.
>>>>>       
>>>>>> To be honest I'm not happy with the idea that I'll document all
>>>>>> the new features in future.
>>>>>
>>>>> Missing or inconsistent documentation are a valid reason to not
>>>>> accept a patch so it will not come to that.
>>>>>
>>>>> Henning
>>>>>       
>>>>>> Alex
>>>>>>      
>>>>>>>>>
>>>>>>>>> Am Tue, 8 Aug 2017 13:05:59 +0300
>>>>>>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>>>>>>>             
>>>>>>>>>> Add initial Isar technical overview.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>>>>>>>> ---
>>>>>>>>>>       doc/technical_overview.md | 110
>>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file
>>>>>>>>>> changed, 110 insertions(+) create mode 100644
>>>>>>>>>> doc/technical_overview.md
>>>>>>>>>>
>>>>>>>>>> diff --git a/doc/technical_overview.md
>>>>>>>>>> b/doc/technical_overview.md new file mode 100644
>>>>>>>>>> index 0000000..f65fc08
>>>>>>>>>> --- /dev/null
>>>>>>>>>> +++ b/doc/technical_overview.md
>>>>>>>>>> @@ -0,0 +1,110 @@
>>>>>>>>>> +# 1 Introduction
>>>>>>>>>> +
>>>>>>>>>> +Isar can be shortly described as a set of bitbake recipes
>>>>>>>>>> that implement main build system logic. To simplify overall
>>>>>>>>>> Isar understanding, this document is split into two main
>>>>>>>>>> parts:
>>>>>>>>>> + - Isar logical components
>>>>>>>>>> + - Isar internal processes
>>>>>>>>>> +
>>>>>>>>>> +# 2 Isar Logical Components
>>>>>>>>>> +
>>>>>>>>>> +In this chapter the most important Isar components are
>>>>>>>>>> considered in details. In this text component doesn't
>>>>>>>>>> especially mean some self-contained single entity of
>>>>>>>>>> something, it's just an attempt to split Isar internals by
>>>>>>>>>> various criteria. + +### 2.1 Bitbake and Recipes + +All the
>>>>>>>>>> processes in Isar are started by bitbake, so it manages
>>>>>>>>>> general build executing process. Recipes in Isar can be split
>>>>>>>>>> in two categories: +
>>>>>>>>>> + - System recipes and classes: they are responsible for
>>>>>>>>>> setting up Debian-like infrastructure, manages Debian tools
>>>>>>>>>> execution for package building and installation, generation
>>>>>>>>>> root file system images
>>>>>>>>>> + - User recipes: custom user applications, that should be
>>>>>>>>>> builе from sources +
>>>>>>>>>> +There are two types of dependencies in Isar:
>>>>>>>>>> + - Dependency between bitbake recipes
>>>>>>>>>> + - Dependencies in Debian filesystem
>>>>>>>>>> +
>>>>>>>>>> +**NOTE:** Isar doesn't manage dependencies in Debian file
>>>>>>>>>> systems. User can only list specific Debian dependencies in
>>>>>>>>>> recipe, so they eventually will be passed to apt-get or
>>>>>>>>>> multistrap. Dependency installation is managed by Debian
>>>>>>>>>> tools.
>>>>>>>>>> + +### 2.2 Stamps + +Each task managed by bitbake uses stamp
>>>>>>>>>> to notify that it has been completed. Due to Isar supports
>>>>>>>>>> various Debian distributions and parallel builds for
>>>>>>>>>> multiple machines and architectures, the stamps for tasks
>>>>>>>>>> use special suffixes that include:
>>>>>>>>>> + - Debian distro name
>>>>>>>>>> + - Architecture
>>>>>>>>>> + - Machine
>>>>>>>>>> +
>>>>>>>>>> +Typical example, when Isar builds the following
>>>>>>>>>> configurations:
>>>>>>>>>> + - Debian Jessie, amd64
>>>>>>>>>> + - Debian Jessie, i386
>>>>>>>>>> + - Debian Stretch, i386
>>>>>>>>>> +
>>>>>>>>>> +In this case there will be 3 different buildchroots, so
>>>>>>>>>> standard hello demo application should be processed 3 times
>>>>>>>>>> for each environment. Three different sets of stamps should
>>>>>>>>>> be used for correct bitbake operating. + +### 2.3
>>>>>>>>>> Buildchroot + +One of the key aspect of Debian philosophy
>>>>>>>>>> claims the fact, that everything in Debian should be built
>>>>>>>>>> within Debian environment. Moreover native compilation is
>>>>>>>>>> more preferable than cross-compilation. To follow this
>>>>>>>>>> rules, Isar introduces the new component - buildchroot.
>>>>>>>>>> Bulidchroot is typical Debian filesystem that is created
>>>>>>>>>> using standard Debian tools: multistrap, apt. The source of
>>>>>>>>>> packages can be either official Debian repositories or
>>>>>>>>>> custom repositories created by user. + +Buildchroot
>>>>>>>>>> lifecycle can be described as following:
>>>>>>>>>> + - Buildchroot has initial configuration file which is
>>>>>>>>>> passed to multistrap tool. This configuration file is
>>>>>>>>>> generated by bitbake recipe from patterns and values defined
>>>>>>>>>> by user. Based on this configuration file, multistrap
>>>>>>>>>> generates initial filesystem.
>>>>>>>>>> + - During building custom Debian package, list of its build
>>>>>>>>>> dependencies is installed to buildchroot.
>>>>>>>>>> + - When package has been built, it's installed to current
>>>>>>>>>> buildchroot to satisfy further packages build dependencies. +
>>>>>>>>>> +### 2.4 Target Root Filesystem
>>>>>>>>>> +
>>>>>>>>>> +Target filesystem is quite similar to buildchroot. The only
>>>>>>>>>> difference is that it doesn't have development packages
>>>>>>>>>> installed.
>>>>>>>>>> + +Target filesystem lifecycle can be described as following:
>>>>>>>>>> + - Target filesystem has initial configuration file which is
>>>>>>>>>> passed to multistrap tool. This configuration file is
>>>>>>>>>> generated by bitbake recipe from patterns and values defined
>>>>>>>>>> by user. Based on this configuration file, multistrap
>>>>>>>>>> generates initial filesystem.
>>>>>>>>>> + - According to the list of custom packages in bitbake
>>>>>>>>>> recipes, the initial filesystem will be populated by
>>>>>>>>>> successfully built packages. + +# 3 Isar Internal Processes
>>>>>>>>>> +
>>>>>>>>>> +### 3.1 General Overview
>>>>>>>>>> +
>>>>>>>>>> +Whole Isar build process can be split into the following
>>>>>>>>>> steps:
>>>>>>>>>> + - Generation of initial buildchroots for each configuration
>>>>>>>>>> (Debian distro, machine and architecture) requested by user.
>>>>>>>>>> + - Generation of initial target filesystems for each
>>>>>>>>>> configuration.
>>>>>>>>>> + - Building custom packages.
>>>>>>>>>> + - Populate target filesystems.
>>>>>>>>>> + - Generate bootable images.
>>>>>>>>>> +
>>>>>>>>>> +All these steps are described in details below.
>>>>>>>>>> +
>>>>>>>>>> +### 3.2 Initial Buildchroot Generation
>>>>>>>>>> +
>>>>>>>>>> +As mentioned above, initial buildchroot is generated using
>>>>>>>>>> multistrap. The bitbake recipe which is responsible for
>>>>>>>>>> buildchroot can be found here:
>>>>>>>>>> `meta/recipes-devtools/buildchroot/buildchroot.bb`
>>>>>>>>>> + +This recipe implementes `do_build` task which performs the
>>>>>>>>>> following: +1. Generates multistrap config from template:
>>>>>>>>>> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in`
>>>>>>>>>> +2. Install pre/post scripts for multistrap:
>>>>>>>>>> `meta/recipes-devtools/buildchroot/files/configscript.sh` and
>>>>>>>>>> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run
>>>>>>>>>> multistrap +4. Install script for building custom Debian
>>>>>>>>>> packages: `meta/recipes-devtools/buildchroot/files/build.sh`
>>>>>>>>>> + +The single stamp is created for each user buildchroot
>>>>>>>>>> configuration. + +### 3.3 Initial Target Filesystem
>>>>>>>>>> Generation
>>>>>>>>>> + +Initial target filesystem generation process is very
>>>>>>>>>> similar to buildchroot creating, the difference is only in
>>>>>>>>>> initial packages list. + +Target image recipes are the part
>>>>>>>>>> of Isar core. There is a sample of typical Isar image that
>>>>>>>>>> can be customized according to the user requirements:
>>>>>>>>>> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for
>>>>>>>>>> buildchroot, the multistrap configuration files for image can
>>>>>>>>>> be found here: `meta-isar/recipes-core/images/files`, and it
>>>>>>>>>> implements `do_build` task. + +### 3.4 Building Custom
>>>>>>>>>> Packages
>>>>>>>>>> + +Isar provides possibility to build Debian packages from
>>>>>>>>>> sources. This features works with Debian-like source
>>>>>>>>>> packages, i.e. the source code tree should contain debian
>>>>>>>>>> folder. The build process is implemented in
>>>>>>>>>> `meta/classes/dpkg.bbclass` and consists from the following
>>>>>>>>>> steps: +1. Task `do_fetch`: fetch source code from external
>>>>>>>>>> link +2. Task `do_unpack`: unpack source code to
>>>>>>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task `do_build`:
>>>>>>>>>> switch to buildchroot using chroot command and run
>>>>>>>>>> `build.sh` script. The `build.sh` script performs the
>>>>>>>>>> following:
>>>>>>>>>> +   1. Go to `/home/build/${PN}`
>>>>>>>>>> +   2. Get list of dependencies from debian/control and
>>>>>>>>>> install them using apt.
>>>>>>>>>> +   3. Run dpkg-buildpackage
>>>>>>>>>> +4. Task `do_install`: install successfully built packages
>>>>>>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy
>>>>>>>>>> directory `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target
>>>>>>>>>> Filesystem +
>>>>>>>>>> +Each target image can be extended by custom packages listed
>>>>>>>>>> in IMAGE_INSTALL variable. Task `do_populate` performs the
>>>>>>>>>> following: +1. Parse IMAGE_INSTALL variable +2. Find
>>>>>>>>>> respective packages in `${DEPLOY_DIR_DEB}` +3. Copy them to
>>>>>>>>>> deb folder in dedicated target filesystem +4. Execute dpkg
>>>>>>>>>> command in chroot for all the copied packages +
>>>>>>>>>> +### 3.6 Generate Bootable Image
>>>>>>>>>> +
>>>>>>>>>> +This process contains the following steps:
>>>>>>>>>> +1. Task `do_ext4_image`: target filesystem is packed to
>>>>>>>>>> extfs image +2. wic tool generates bootable image for
>>>>>>>>>> dedicated platform
>>>>>>>>>             
>>>>>>>          
>>>>>>      
>>>>>       
>>>>   
>>>    
>>
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH] doc: Add technical overview

[Baurzhan Ismagulov]

On Tue, Aug 08, 2017 at 01:05:59PM +0300, Alexander Smirnov wrote:
> Add initial Isar technical overview.

Applied with line wrapping at column 79, thanks.

With kind regards,
Baurzhan.