Re: [PATCH] doc: Add technical overview

[Henning Schild <henning.schild@siemens.com>]

Am Tue, 8 Aug 2017 14:32:00 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Hi,
> 
> On 08/08/2017 02:11 PM, Henning Schild wrote:
> > I did not really read this yet. At the moment i think we have too
> > many changes in the q to add such a document to the repository. Any
> > change would just mean heaving to keep it up to date.
> > We talked about the need for such a document for the possible OE
> > integration. I think it is valuable for that already.
> >   
> 
> Briefly speaking, this document describes initial motivation and 
> high-level current overview.
> 
> > That should be added after we all are happy with the feature-set of
> > Isar, until then it would just be another file to update when
> > changing how things work.  
> 
> Hmm, this document doesn't affect any discussions about Isar feature 
> set. It's intended to be as technical background for hot-plug joining
> to this project. During discussions last month I see that we have gap
> in our understandings how Isar works, so this document should be
> start point to eliminate such gaps.
> 
> For sure, it's *not* a specification, which explicitly defines
> specific feature set and nothing more will be added. It's just an
> explanation of current state. I see no problems to regularly update
> it, moreover I highly appreciate this. Lack of documentation is the
> most common problem of open source projects, so as soon we start
> documenting the things, the easier will be documentation maintenance
> in future.

For sure we need that! But especially section 3 contains lots of
interna that will change if my patches get accepted. That is why i
would like to postpone adding the file after my patches went in. 

Adding that file before them would mean another rebase round for
documentation. I would be happy to rebase this one for you on top of a
master that contains my stuff, but not too happy about the other way
around.

Henning

> Alex
> 
> > 
> > Am Tue, 8 Aug 2017 13:05:59 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> Add initial Isar technical overview.
> >>
> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >> ---
> >>   doc/technical_overview.md | 110
> >> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110
> >> insertions(+) create mode 100644 doc/technical_overview.md
> >>
> >> diff --git a/doc/technical_overview.md b/doc/technical_overview.md
> >> new file mode 100644
> >> index 0000000..f65fc08
> >> --- /dev/null
> >> +++ b/doc/technical_overview.md
> >> @@ -0,0 +1,110 @@
> >> +# 1 Introduction
> >> +
> >> +Isar can be shortly described as a set of bitbake recipes that
> >> implement main build system logic. To simplify overall Isar
> >> understanding, this document is split into two main parts:
> >> + - Isar logical components
> >> + - Isar internal processes
> >> +
> >> +# 2 Isar Logical Components
> >> +
> >> +In this chapter the most important Isar components are considered
> >> in details. In this text component doesn't especially mean some
> >> self-contained single entity of something, it's just an attempt to
> >> split Isar internals by various criteria. + +### 2.1 Bitbake and
> >> Recipes + +All the processes in Isar are started by bitbake, so it
> >> manages general build executing process. Recipes in Isar can be
> >> split in two categories: +
> >> + - System recipes and classes: they are responsible for setting up
> >> Debian-like infrastructure, manages Debian tools execution for
> >> package building and installation, generation root file system
> >> images
> >> + - User recipes: custom user applications, that should be builе
> >> from sources +
> >> +There are two types of dependencies in Isar:
> >> + - Dependency between bitbake recipes
> >> + - Dependencies in Debian filesystem
> >> +
> >> +**NOTE:** Isar doesn't manage dependencies in Debian file systems.
> >> User can only list specific Debian dependencies in recipe, so they
> >> eventually will be passed to apt-get or multistrap. Dependency
> >> installation is managed by Debian tools. + +### 2.2 Stamps +
> >> +Each task managed by bitbake uses stamp to notify that it has been
> >> completed. Due to Isar supports various Debian distributions and
> >> parallel builds for multiple machines and architectures, the stamps
> >> for tasks use special suffixes that include:
> >> + - Debian distro name
> >> + - Architecture
> >> + - Machine
> >> +
> >> +Typical example, when Isar builds the following configurations:
> >> + - Debian Jessie, amd64
> >> + - Debian Jessie, i386
> >> + - Debian Stretch, i386
> >> +
> >> +In this case there will be 3 different buildchroots, so standard
> >> hello demo application should be processed 3 times for each
> >> environment. Three different sets of stamps should be used for
> >> correct bitbake operating. + +### 2.3 Buildchroot +
> >> +One of the key aspect of Debian philosophy claims the fact, that
> >> everything in Debian should be built within Debian environment.
> >> Moreover native compilation is more preferable than
> >> cross-compilation. To follow this rules, Isar introduces the new
> >> component - buildchroot. Bulidchroot is typical Debian filesystem
> >> that is created using standard Debian tools: multistrap, apt. The
> >> source of packages can be either official Debian repositories or
> >> custom repositories created by user. + +Buildchroot lifecycle can
> >> be described as following:
> >> + - Buildchroot has initial configuration file which is passed to
> >> multistrap tool. This configuration file is generated by bitbake
> >> recipe from patterns and values defined by user. Based on this
> >> configuration file, multistrap generates initial filesystem.
> >> + - During building custom Debian package, list of its build
> >> dependencies is installed to buildchroot.
> >> + - When package has been built, it's installed to current
> >> buildchroot to satisfy further packages build dependencies. +
> >> +### 2.4 Target Root Filesystem
> >> +
> >> +Target filesystem is quite similar to buildchroot. The only
> >> difference is that it doesn't have development packages installed.
> >> + +Target filesystem lifecycle can be described as following:
> >> + - Target filesystem has initial configuration file which is
> >> passed to multistrap tool. This configuration file is generated by
> >> bitbake recipe from patterns and values defined by user. Based on
> >> this configuration file, multistrap generates initial filesystem.
> >> + - According to the list of custom packages in bitbake recipes,
> >> the initial filesystem will be populated by successfully built
> >> packages. + +# 3 Isar Internal Processes
> >> +
> >> +### 3.1 General Overview
> >> +
> >> +Whole Isar build process can be split into the following steps:
> >> + - Generation of initial buildchroots for each configuration
> >> (Debian distro, machine and architecture) requested by user.
> >> + - Generation of initial target filesystems for each
> >> configuration.
> >> + - Building custom packages.
> >> + - Populate target filesystems.
> >> + - Generate bootable images.
> >> +
> >> +All these steps are described in details below.
> >> +
> >> +### 3.2 Initial Buildchroot Generation
> >> +
> >> +As mentioned above, initial buildchroot is generated using
> >> multistrap. The bitbake recipe which is responsible for buildchroot
> >> can be found here:
> >> `meta/recipes-devtools/buildchroot/buildchroot.bb`
> >> + +This recipe implementes `do_build` task which performs the
> >> following: +1. Generates multistrap config from template:
> >> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in` +2.
> >> Install pre/post scripts for multistrap:
> >> `meta/recipes-devtools/buildchroot/files/configscript.sh` and
> >> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run
> >> multistrap +4. Install script for building custom Debian packages:
> >> `meta/recipes-devtools/buildchroot/files/build.sh` + +The single
> >> stamp is created for each user buildchroot configuration. + +###
> >> 3.3 Initial Target Filesystem Generation + +Initial target
> >> filesystem generation process is very similar to buildchroot
> >> creating, the difference is only in initial packages list. +
> >> +Target image recipes are the part of Isar core. There is a sample
> >> of typical Isar image that can be customized according to the user
> >> requirements: `meta-isar/recipes-core/images/isar-image-base.bb`
> >> +Like for buildchroot, the multistrap configuration files for
> >> image can be found here: `meta-isar/recipes-core/images/files`,
> >> and it implements `do_build` task. + +### 3.4 Building Custom
> >> Packages + +Isar provides possibility to build Debian packages
> >> from sources. This features works with Debian-like source
> >> packages, i.e. the source code tree should contain debian folder.
> >> The build process is implemented in `meta/classes/dpkg.bbclass`
> >> and consists from the following steps: +1. Task `do_fetch`: fetch
> >> source code from external link +2. Task `do_unpack`: unpack source
> >> code to `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task `do_build`:
> >> switch to buildchroot using chroot command and run `build.sh`
> >> script. The `build.sh` script performs the following:
> >> +   1. Go to `/home/build/${PN}`
> >> +   2. Get list of dependencies from debian/control and install
> >> them using apt.
> >> +   3. Run dpkg-buildpackage
> >> +4. Task `do_install`: install successfully built packages
> >> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory
> >> `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target Filesystem
> >> +
> >> +Each target image can be extended by custom packages listed in
> >> IMAGE_INSTALL variable. Task `do_populate` performs the following:
> >> +1. Parse IMAGE_INSTALL variable +2. Find respective packages in
> >> `${DEPLOY_DIR_DEB}` +3. Copy them to deb folder in dedicated target
> >> filesystem +4. Execute dpkg command in chroot for all the copied
> >> packages +
> >> +### 3.6 Generate Bootable Image
> >> +
> >> +This process contains the following steps:
> >> +1. Task `do_ext4_image`: target filesystem is packed to extfs
> >> image +2. wic tool generates bootable image for dedicated
> >> platform  
> >