Re: [PATCH] doc: Add technical overview

[Henning Schild <henning.schild@siemens.com>]

Am Tue, 8 Aug 2017 17:44:30 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 08/08/2017 05:38 PM, Henning Schild wrote:
> > Am Tue, 8 Aug 2017 16:12:02 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> On 08/08/2017 04:03 PM, Henning Schild wrote:  
> >>> Am Tue, 8 Aug 2017 15:41:10 +0300
> >>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >>>      
> >>>> On 08/08/2017 02:54 PM, Henning Schild wrote:  
> >>>>> Am Tue, 8 Aug 2017 14:32:00 +0300
> >>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >>>>>         
> >>>>>> Hi,
> >>>>>>
> >>>>>> On 08/08/2017 02:11 PM, Henning Schild wrote:  
> >>>>>>> I did not really read this yet. At the moment i think we have
> >>>>>>> too many changes in the q to add such a document to the
> >>>>>>> repository. Any change would just mean heaving to keep it up
> >>>>>>> to date. We talked about the need for such a document for the
> >>>>>>> possible OE integration. I think it is valuable for that
> >>>>>>> already.  
> >>>>>>
> >>>>>> Briefly speaking, this document describes initial motivation
> >>>>>> and high-level current overview.
> >>>>>>        
> >>>>>>> That should be added after we all are happy with the
> >>>>>>> feature-set of Isar, until then it would just be another file
> >>>>>>> to update when changing how things work.  
> >>>>>>
> >>>>>> Hmm, this document doesn't affect any discussions about Isar
> >>>>>> feature set. It's intended to be as technical background for
> >>>>>> hot-plug joining to this project. During discussions last
> >>>>>> month I see that we have gap in our understandings how Isar
> >>>>>> works, so this document should be start point to eliminate
> >>>>>> such gaps.
> >>>>>>
> >>>>>> For sure, it's *not* a specification, which explicitly defines
> >>>>>> specific feature set and nothing more will be added. It's just
> >>>>>> an explanation of current state. I see no problems to regularly
> >>>>>> update it, moreover I highly appreciate this. Lack of
> >>>>>> documentation is the most common problem of open source
> >>>>>> projects, so as soon we start documenting the things, the
> >>>>>> easier will be documentation maintenance in future.  
> >>>>>
> >>>>> For sure we need that! But especially section 3 contains lots of
> >>>>> interna that will change if my patches get accepted. That is
> >>>>> why i would like to postpone adding the file after my patches
> >>>>> went in.  
> >>>>
> >>>> As I said, this document describes current project state, so I
> >>>> don't think that upcoming changes should block publishing of
> >>>> documentation.
> >>>>
> >>>>     
> >>>>>
> >>>>> Adding that file before them would mean another rebase round for
> >>>>> documentation. I would be happy to rebase this one for you on
> >>>>> top of a master that contains my stuff, but not too happy about
> >>>>> the other way around.  
> >>>>
> >>>> The patch creates document from the scratch, so it can't conflict
> >>>> with any upcoming patches.  
> >>>
> >>> It conflicts with other patches that are under review right now.
> >>> All i am asking is to merge conflicting patches in the order the
> >>> patches initially showed up for review, in that case this one
> >>> comes last anyways. And when its time has come it will be
> >>> outdated, and i will update it for you.
> >>>      
> >>
> >> Hmm, are you speaking about exactly this patch or about previous
> >> series?  
> > 
> > Section 3.4 and 3.5 conflict with my changes, they will have to be
> > changed. If your patch comes first i would have to rebase.
> > 
> > i.e.
> > - 3.4 talks about dpkg.bbclass which i renamed
> > [PATCH 11-16 of 16 v2 4/6]
> > - 3.5 will not happen anymore if everything enters the rootfs
> > through multistrap
> > [PATCH 3/3] classes: image: remove populate and replace it with a
> > custom repo  
> 
> I didn't get the idea, how this affects your patches? After your 
> features will be applied, you can update this document. In this
> thread you've mentioned, that this patch breaks your series and force
> you to perform rebase, this point is not clear for me.

A consistent patch updates the code along with the documentation. If
the code i am touching suddenly gets documented my commits will become
inconsistent.

Henning

> > 
> > 
> > Henning
> > 
> >   
> >> Alex
> >>  
> >>>> Also I expect, that the author of new features will update the
> >>>> documentation, so some reference document should be published
> >>>> anyway.  
> >>>
> >>> Sure.
> >>>      
> >>>> To be honest I'm not happy with the idea that I'll document all
> >>>> the new features in future.  
> >>>
> >>> Missing or inconsistent documentation are a valid reason to not
> >>> accept a patch so it will not come to that.
> >>>
> >>> Henning
> >>>      
> >>>> Alex
> >>>>     
> >>>>>>>
> >>>>>>> Am Tue, 8 Aug 2017 13:05:59 +0300
> >>>>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >>>>>>>            
> >>>>>>>> Add initial Isar technical overview.
> >>>>>>>>
> >>>>>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >>>>>>>> ---
> >>>>>>>>      doc/technical_overview.md | 110
> >>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file
> >>>>>>>> changed, 110 insertions(+) create mode 100644
> >>>>>>>> doc/technical_overview.md
> >>>>>>>>
> >>>>>>>> diff --git a/doc/technical_overview.md
> >>>>>>>> b/doc/technical_overview.md new file mode 100644
> >>>>>>>> index 0000000..f65fc08
> >>>>>>>> --- /dev/null
> >>>>>>>> +++ b/doc/technical_overview.md
> >>>>>>>> @@ -0,0 +1,110 @@
> >>>>>>>> +# 1 Introduction
> >>>>>>>> +
> >>>>>>>> +Isar can be shortly described as a set of bitbake recipes
> >>>>>>>> that implement main build system logic. To simplify overall
> >>>>>>>> Isar understanding, this document is split into two main
> >>>>>>>> parts:
> >>>>>>>> + - Isar logical components
> >>>>>>>> + - Isar internal processes
> >>>>>>>> +
> >>>>>>>> +# 2 Isar Logical Components
> >>>>>>>> +
> >>>>>>>> +In this chapter the most important Isar components are
> >>>>>>>> considered in details. In this text component doesn't
> >>>>>>>> especially mean some self-contained single entity of
> >>>>>>>> something, it's just an attempt to split Isar internals by
> >>>>>>>> various criteria. + +### 2.1 Bitbake and Recipes + +All the
> >>>>>>>> processes in Isar are started by bitbake, so it manages
> >>>>>>>> general build executing process. Recipes in Isar can be split
> >>>>>>>> in two categories: +
> >>>>>>>> + - System recipes and classes: they are responsible for
> >>>>>>>> setting up Debian-like infrastructure, manages Debian tools
> >>>>>>>> execution for package building and installation, generation
> >>>>>>>> root file system images
> >>>>>>>> + - User recipes: custom user applications, that should be
> >>>>>>>> builе from sources +
> >>>>>>>> +There are two types of dependencies in Isar:
> >>>>>>>> + - Dependency between bitbake recipes
> >>>>>>>> + - Dependencies in Debian filesystem
> >>>>>>>> +
> >>>>>>>> +**NOTE:** Isar doesn't manage dependencies in Debian file
> >>>>>>>> systems. User can only list specific Debian dependencies in
> >>>>>>>> recipe, so they eventually will be passed to apt-get or
> >>>>>>>> multistrap. Dependency installation is managed by Debian
> >>>>>>>> tools.
> >>>>>>>> + +### 2.2 Stamps + +Each task managed by bitbake uses stamp
> >>>>>>>> to notify that it has been completed. Due to Isar supports
> >>>>>>>> various Debian distributions and parallel builds for
> >>>>>>>> multiple machines and architectures, the stamps for tasks
> >>>>>>>> use special suffixes that include:
> >>>>>>>> + - Debian distro name
> >>>>>>>> + - Architecture
> >>>>>>>> + - Machine
> >>>>>>>> +
> >>>>>>>> +Typical example, when Isar builds the following
> >>>>>>>> configurations:
> >>>>>>>> + - Debian Jessie, amd64
> >>>>>>>> + - Debian Jessie, i386
> >>>>>>>> + - Debian Stretch, i386
> >>>>>>>> +
> >>>>>>>> +In this case there will be 3 different buildchroots, so
> >>>>>>>> standard hello demo application should be processed 3 times
> >>>>>>>> for each environment. Three different sets of stamps should
> >>>>>>>> be used for correct bitbake operating. + +### 2.3
> >>>>>>>> Buildchroot + +One of the key aspect of Debian philosophy
> >>>>>>>> claims the fact, that everything in Debian should be built
> >>>>>>>> within Debian environment. Moreover native compilation is
> >>>>>>>> more preferable than cross-compilation. To follow this
> >>>>>>>> rules, Isar introduces the new component - buildchroot.
> >>>>>>>> Bulidchroot is typical Debian filesystem that is created
> >>>>>>>> using standard Debian tools: multistrap, apt. The source of
> >>>>>>>> packages can be either official Debian repositories or
> >>>>>>>> custom repositories created by user. + +Buildchroot
> >>>>>>>> lifecycle can be described as following:
> >>>>>>>> + - Buildchroot has initial configuration file which is
> >>>>>>>> passed to multistrap tool. This configuration file is
> >>>>>>>> generated by bitbake recipe from patterns and values defined
> >>>>>>>> by user. Based on this configuration file, multistrap
> >>>>>>>> generates initial filesystem.
> >>>>>>>> + - During building custom Debian package, list of its build
> >>>>>>>> dependencies is installed to buildchroot.
> >>>>>>>> + - When package has been built, it's installed to current
> >>>>>>>> buildchroot to satisfy further packages build dependencies. +
> >>>>>>>> +### 2.4 Target Root Filesystem
> >>>>>>>> +
> >>>>>>>> +Target filesystem is quite similar to buildchroot. The only
> >>>>>>>> difference is that it doesn't have development packages
> >>>>>>>> installed.
> >>>>>>>> + +Target filesystem lifecycle can be described as following:
> >>>>>>>> + - Target filesystem has initial configuration file which is
> >>>>>>>> passed to multistrap tool. This configuration file is
> >>>>>>>> generated by bitbake recipe from patterns and values defined
> >>>>>>>> by user. Based on this configuration file, multistrap
> >>>>>>>> generates initial filesystem.
> >>>>>>>> + - According to the list of custom packages in bitbake
> >>>>>>>> recipes, the initial filesystem will be populated by
> >>>>>>>> successfully built packages. + +# 3 Isar Internal Processes
> >>>>>>>> +
> >>>>>>>> +### 3.1 General Overview
> >>>>>>>> +
> >>>>>>>> +Whole Isar build process can be split into the following
> >>>>>>>> steps:
> >>>>>>>> + - Generation of initial buildchroots for each configuration
> >>>>>>>> (Debian distro, machine and architecture) requested by user.
> >>>>>>>> + - Generation of initial target filesystems for each
> >>>>>>>> configuration.
> >>>>>>>> + - Building custom packages.
> >>>>>>>> + - Populate target filesystems.
> >>>>>>>> + - Generate bootable images.
> >>>>>>>> +
> >>>>>>>> +All these steps are described in details below.
> >>>>>>>> +
> >>>>>>>> +### 3.2 Initial Buildchroot Generation
> >>>>>>>> +
> >>>>>>>> +As mentioned above, initial buildchroot is generated using
> >>>>>>>> multistrap. The bitbake recipe which is responsible for
> >>>>>>>> buildchroot can be found here:
> >>>>>>>> `meta/recipes-devtools/buildchroot/buildchroot.bb`
> >>>>>>>> + +This recipe implementes `do_build` task which performs the
> >>>>>>>> following: +1. Generates multistrap config from template:
> >>>>>>>> `meta/recipes-devtools/buildchroot/files/multistrap.conf.in`
> >>>>>>>> +2. Install pre/post scripts for multistrap:
> >>>>>>>> `meta/recipes-devtools/buildchroot/files/configscript.sh` and
> >>>>>>>> `meta/recipes-devtools/buildchroot/files/setup.sh` +3. Run
> >>>>>>>> multistrap +4. Install script for building custom Debian
> >>>>>>>> packages: `meta/recipes-devtools/buildchroot/files/build.sh`
> >>>>>>>> + +The single stamp is created for each user buildchroot
> >>>>>>>> configuration. + +### 3.3 Initial Target Filesystem
> >>>>>>>> Generation
> >>>>>>>> + +Initial target filesystem generation process is very
> >>>>>>>> similar to buildchroot creating, the difference is only in
> >>>>>>>> initial packages list. + +Target image recipes are the part
> >>>>>>>> of Isar core. There is a sample of typical Isar image that
> >>>>>>>> can be customized according to the user requirements:
> >>>>>>>> `meta-isar/recipes-core/images/isar-image-base.bb` +Like for
> >>>>>>>> buildchroot, the multistrap configuration files for image can
> >>>>>>>> be found here: `meta-isar/recipes-core/images/files`, and it
> >>>>>>>> implements `do_build` task. + +### 3.4 Building Custom
> >>>>>>>> Packages
> >>>>>>>> + +Isar provides possibility to build Debian packages from
> >>>>>>>> sources. This features works with Debian-like source
> >>>>>>>> packages, i.e. the source code tree should contain debian
> >>>>>>>> folder. The build process is implemented in
> >>>>>>>> `meta/classes/dpkg.bbclass` and consists from the following
> >>>>>>>> steps: +1. Task `do_fetch`: fetch source code from external
> >>>>>>>> link +2. Task `do_unpack`: unpack source code to
> >>>>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}` +3. Task `do_build`:
> >>>>>>>> switch to buildchroot using chroot command and run
> >>>>>>>> `build.sh` script. The `build.sh` script performs the
> >>>>>>>> following:
> >>>>>>>> +   1. Go to `/home/build/${PN}`
> >>>>>>>> +   2. Get list of dependencies from debian/control and
> >>>>>>>> install them using apt.
> >>>>>>>> +   3. Run dpkg-buildpackage
> >>>>>>>> +4. Task `do_install`: install successfully built packages
> >>>>>>>> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy
> >>>>>>>> directory `${DEPLOY_DIR_DEB}` + +### 3.5 Populate Target
> >>>>>>>> Filesystem +
> >>>>>>>> +Each target image can be extended by custom packages listed
> >>>>>>>> in IMAGE_INSTALL variable. Task `do_populate` performs the
> >>>>>>>> following: +1. Parse IMAGE_INSTALL variable +2. Find
> >>>>>>>> respective packages in `${DEPLOY_DIR_DEB}` +3. Copy them to
> >>>>>>>> deb folder in dedicated target filesystem +4. Execute dpkg
> >>>>>>>> command in chroot for all the copied packages +
> >>>>>>>> +### 3.6 Generate Bootable Image
> >>>>>>>> +
> >>>>>>>> +This process contains the following steps:
> >>>>>>>> +1. Task `do_ext4_image`: target filesystem is packed to
> >>>>>>>> extfs image +2. wic tool generates bootable image for
> >>>>>>>> dedicated platform  
> >>>>>>>            
> >>>>>         
> >>>>     
> >>>      
> >>  
> >   
>