Re: [PATCH] doc: Add technical overview

[Henning Schild <henning.schild@siemens.com>]

I did not really read this yet. At the moment i think we have too many
changes in the q to add such a document to the repository. Any change
would just mean heaving to keep it up to date.
We talked about the need for such a document for the possible OE
integration. I think it is valuable for that already.

That should be added after we all are happy with the feature-set of
Isar, until then it would just be another file to update when changing
how things work.

Henning

Am Tue, 8 Aug 2017 13:05:59 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Add initial Isar technical overview.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  doc/technical_overview.md | 110
> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110
> insertions(+) create mode 100644 doc/technical_overview.md
> 
> diff --git a/doc/technical_overview.md b/doc/technical_overview.md
> new file mode 100644
> index 0000000..f65fc08
> --- /dev/null
> +++ b/doc/technical_overview.md
> @@ -0,0 +1,110 @@
> +# 1 Introduction
> +
> +Isar can be shortly described as a set of bitbake recipes that
> implement main build system logic. To simplify overall Isar
> understanding, this document is split into two main parts:
> + - Isar logical components
> + - Isar internal processes
> +
> +# 2 Isar Logical Components
> +
> +In this chapter the most important Isar components are considered in
> details. In this text component doesn't especially mean some
> self-contained single entity of something, it's just an attempt to
> split Isar internals by various criteria. + +### 2.1 Bitbake and
> Recipes + +All the processes in Isar are started by bitbake, so it
> manages general build executing process. Recipes in Isar can be split
> in two categories: +
> + - System recipes and classes: they are responsible for setting up
> Debian-like infrastructure, manages Debian tools execution for
> package building and installation, generation root file system images
> + - User recipes: custom user applications, that should be builе from
> sources +
> +There are two types of dependencies in Isar:
> + - Dependency between bitbake recipes
> + - Dependencies in Debian filesystem
> +
> +**NOTE:** Isar doesn't manage dependencies in Debian file systems.
> User can only list specific Debian dependencies in recipe, so they
> eventually will be passed to apt-get or multistrap. Dependency
> installation is managed by Debian tools. + +### 2.2 Stamps +
> +Each task managed by bitbake uses stamp to notify that it has been
> completed. Due to Isar supports various Debian distributions and
> parallel builds for multiple machines and architectures, the stamps
> for tasks use special suffixes that include:
> + - Debian distro name
> + - Architecture
> + - Machine
> +
> +Typical example, when Isar builds the following configurations:
> + - Debian Jessie, amd64
> + - Debian Jessie, i386
> + - Debian Stretch, i386
> +
> +In this case there will be 3 different buildchroots, so standard
> hello demo application should be processed 3 times for each
> environment. Three different sets of stamps should be used for
> correct bitbake operating. + +### 2.3 Buildchroot +
> +One of the key aspect of Debian philosophy claims the fact, that
> everything in Debian should be built within Debian environment.
> Moreover native compilation is more preferable than
> cross-compilation. To follow this rules, Isar introduces the new
> component - buildchroot. Bulidchroot is typical Debian filesystem
> that is created using standard Debian tools: multistrap, apt. The
> source of packages can be either official Debian repositories or
> custom repositories created by user. + +Buildchroot lifecycle can be
> described as following:
> + - Buildchroot has initial configuration file which is passed to
> multistrap tool. This configuration file is generated by bitbake
> recipe from patterns and values defined by user. Based on this
> configuration file, multistrap generates initial filesystem.
> + - During building custom Debian package, list of its build
> dependencies is installed to buildchroot.
> + - When package has been built, it's installed to current
> buildchroot to satisfy further packages build dependencies. +
> +### 2.4 Target Root Filesystem
> +
> +Target filesystem is quite similar to buildchroot. The only
> difference is that it doesn't have development packages installed. +
> +Target filesystem lifecycle can be described as following:
> + - Target filesystem has initial configuration file which is passed
> to multistrap tool. This configuration file is generated by bitbake
> recipe from patterns and values defined by user. Based on this
> configuration file, multistrap generates initial filesystem.
> + - According to the list of custom packages in bitbake recipes, the
> initial filesystem will be populated by successfully built packages. +
> +# 3 Isar Internal Processes
> +
> +### 3.1 General Overview
> +
> +Whole Isar build process can be split into the following steps:
> + - Generation of initial buildchroots for each configuration (Debian
> distro, machine and architecture) requested by user.
> + - Generation of initial target filesystems for each configuration.
> + - Building custom packages.
> + - Populate target filesystems.
> + - Generate bootable images.
> +
> +All these steps are described in details below.
> +
> +### 3.2 Initial Buildchroot Generation
> +
> +As mentioned above, initial buildchroot is generated using
> multistrap. The bitbake recipe which is responsible for buildchroot
> can be found here: `meta/recipes-devtools/buildchroot/buildchroot.bb`
> + +This recipe implementes `do_build` task which performs the
> following: +1. Generates multistrap config from template:
> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in` +2.
> Install pre/post scripts for multistrap:
> `meta/recipes-devtools/buildchroot/files/configscript.sh` and
> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run multistrap
> +4. Install script for building custom Debian packages:
> `meta/recipes-devtools/buildchroot/files/build.sh` + +The single
> stamp is created for each user buildchroot configuration. + +### 3.3
> Initial Target Filesystem Generation + +Initial target filesystem
> generation process is very similar to buildchroot creating, the
> difference is only in initial packages list. + +Target image recipes
> are the part of Isar core. There is a sample of typical Isar image
> that can be customized according to the user requirements:
> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for
> buildchroot, the multistrap configuration files for image can be
> found here: `meta-isar/recipes-core/images/files`, and it implements
> `do_build` task. + +### 3.4 Building Custom Packages + +Isar provides
> possibility to build Debian packages from sources. This features
> works with Debian-like source packages, i.e. the source code tree
> should contain debian folder. The build process is implemented in
> `meta/classes/dpkg.bbclass` and consists from the following steps:
> +1. Task `do_fetch`: fetch source code from external link +2. Task
> `do_unpack`: unpack source code to
> `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task `do_build`: switch to
> buildchroot using chroot command and run `build.sh` script. The
> `build.sh` script performs the following:
> +   1. Go to `/home/build/${PN}`
> +   2. Get list of dependencies from debian/control and install them
> using apt.
> +   3. Run dpkg-buildpackage
> +4. Task `do_install`: install successfully built packages
> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory
> `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target Filesystem
> +
> +Each target image can be extended by custom packages listed in
> IMAGE_INSTALL variable. Task `do_populate` performs the following:
> +1. Parse IMAGE_INSTALL variable +2. Find respective packages in
> `${DEPLOY_DIR_DEB}` +3. Copy them to deb folder in dedicated target
> filesystem +4. Execute dpkg command in chroot for all the copied
> packages +
> +### 3.6 Generate Bootable Image
> +
> +This process contains the following steps:
> +1. Task `do_ext4_image`: target filesystem is packed to extfs image
> +2. wic tool generates bootable image for dedicated platform